Shallow Thoughts : tags : linux

Open the X selection in a browser window, from any desktop

Like most Linux users, I use virtual desktops. Normally my browser window is on a desktop of its own.

Naturally, it often happens that I encounter a link I'd like to visit while I'm on a desktop where the browser isn't visible. From some apps, I can click on the link and have it show up. But sometimes, the link is just text, and I have to select it, change to the browser desktop, paste the link into firefox, then change desktops again to do something else while the link loads.

So I set up a way to load whatever's in the X selection in firefox no matter what desktop I'm on.

In most browsers, including firefox, you can tell your existing browser window to open a new link from the command line: firefox http://example.com/ opens that link in your existing browser window if you already have one up, rather than starting another browser. So the trick is to get the text you've selected.

At first, I used a program called xclip. You can run this command: firefox `xclip -o` to open the selection. That worked okay at first -- until I hit my first URL in weechat that was so long that it was wrapped to the next line. It turns out xclip does odd things with multi-line output; depending on whether it thinks the output is a terminal or not, it may replace the newline with a space, or delete whatever follows the newline. In any case, I couldn't find a way to make it work reliably when pasted into firefox.

After futzing with xclip for a little too long, trying to reverse-engineer its undocumented newline behavior, I decided it would be easier just to write my own X clipboard app in Python. I already knew how to do that, and it's super easy once you know the trick:

mport gtk
primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)
if primary.wait_is_text_available() :
    print primary.wait_for_text()

That just prints it directly, including any newlines or spaces. But as long as I was writing my own app, why not handle that too?

It's not entirely necessary on Firefox: on Linux, Firefox has some special code to deal with pasting multi-line URLs, so you can copy a URL that spans multiple lines, middleclick in the content area and things will work. On other platforms, that's disabled, and some Linux distros disable it as well; you can enable it by going to about:config and searching for single, then setting the preference editor.singlelinepaste.pasteNewlines to 2.

However, it was easy enough to make my Python clipboard app do the right thing so it would work in any browser. I used Python's re (regular expressions) module:

#!/usr/bin/env python

import gtk
import re

primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)

if not primary.wait_is_text_available() :
    sys.exit(0)
s = primary.wait_for_text()

# eliminate newlines, and any spaces immediately following a newline:
print re.sub(r'[\r\n]+ *', '', s)

That seemed to work fine, even on long URLs pasted from weechat with newlines and spaces, like that looked like

http://example.com/long-
    url.html

All that was left was binding it so I could access it from anywhere. Of course, that varies depending on your desktop/window manager. In Openbox, I added two items to my desktop menu in menu.xml:

  <item label="open selection in Firefox">
    <action name="Execute"><execute>sh -c 'firefox `xclip -o`'</execute></action>
  </item>
  <item label="open selection in new tab">
    <action name="Execute"><execute>sh -c 'firefox -new-tab `xclip -o`'</execute></action>
  </item>

I also added some code in rc.xml inside <context name="Desktop">, so I can middle-click or control-middle-click on the desktop to open a link in the browser:

      <mousebind button="Middle" action="Press">
        <action name="Execute">
          <execute>sh -c 'firefox `pyclip`'</execute>
        </action>
      </mousebind>
      <mousebind button="C-Middle" action="Press">
        <action name="Execute">
          <execute>sh -c -new-tab 'firefox `pyclip`'</execute>
        </action>
      </mousebind>

I set this up maybe two hours ago and I've probably used it ten or fifteen times already. This is something I should have done long ago!

Taming the loud system beep

One of the distros I'm trying on my Dell Latitude 2120 laptop is Arch Linux. I like a lot of things about Arch; I had to stop using it on my previous laptop because something broke in the wi-fi drivers, but I always regretted giving it up. And it seems to work quite well on the Dell, with one exception: the system beep (which other distros don't support at all) was ear-shatteringly loud, far too loud to consider being able to use this laptop in a public space. Clearly that needed to be fixed.

The usual approach to system beeps, unloading or blacklisting the pcspkr module, had no effect. xset b off turned the beep off in X; I could also set its pitch and duration to change it to a nice quiet click, though I wasn't able to change the volume that way. I actually do like having a system beep, as long as it's fairly quiet and won't disturb people nearby. Unfortunately, xset b only affects the bell while in X; it didn't have any effect on the deafening sound Arch gave upon shutdown or reboot.

It turns out that on some laptops, including this Dell, the system beep goes not through the old-style pcspkr driver, but through the normal sound card. And the sound card has a separate channel for the system beep, so even if you have your volume turned down, the beep may still be at 100%. All I needed to do was run alsamixer and find out what the channel was called: "Beep".

Given that, I could use the amixer program to ensure the beep volume will be sane when I log in. I added the following to .zlogin (for zsh; obviously, adjust for your own shell):

amixer -q -c 0 set Beep 5

That gave me a nice quiet beep. If I need to turn it off completely, amixer can do that too: amixer -q -c 0 set Beep mute (curiously, amixer -q -c 0 set Beep 0 doesn't actually set the volume to zero, just sets it very low).

That volume setting applies to the shutdown beep, too, fortunately. Though what I'd really like is to have quiet beeps while I'm running, but no shutdown beep at all. I don't understand the purpose of the shutdown beep; obviously I know when I've told my machine to shut down or reboot, so why do I need an audible reminder? But I've been unable to find anything explaining what's causing this beep. I tried adding a amixer -q -c 0 set Beep mute to /etc/rc.local.shutdown, but it didn't help; apparently the shutdown beep is called before that file is run. Which strongly suggests it is being run by Arch Linux, not by something in the BIOS. But nobody I've asked had any suggestions as to its source, or how to change it. Another enduring Linux mystery ...

Update: I mentioned xset b as a way to adjust beeps inside X -- in addition to turning beeps totally off, you can also set pitch, duration, and sometimes volume though the volume part didn't work for me. But outside X, you can make similar adjustments with setterm, e.g. setterm -bfreq 400 -blength 50. Thanks to Mikachu for the tip!

Configuring extlinux's auto-update on Debian

A few days ago, I wrote about how to set up and configure extlinux (syslinux) as a bootloader. But on Debian or Ubuntu, if you make changes to files like /boot/extlinux/extlinux.conf directly, they'll be overwritten.

The configuration files are regenerated by a program called extlinux-update, which runs automatically every time you update your kernel. (Specifically, it runs from the postinst script of the linux-base package: you can see it in /var/lib/dpkg/info/linux-base.postinst.)

So what's a Debian user to do if she wants to customize the menus, add a splash image or boot other operating systems?

First, if you decide you really don't want Debian overwriting your configuration files, you can change disable updates by editing /etc/default/extlinux. Just be aware you won't get your boot menu updated when you install new kernels -- you'll have to remember to update them by hand.

It might be worth it: the automatic update is nearly as annoying as the grub2 updater: it creates two automatic entries for every kernel you have installed. So if you have several distros installed, each with a kernel or two in your shared /boot, you'll get an entry to boot Debian Squeeze with the Ubuntu Oneiric kernel, one for Squeeze with the Natty kernel, one for Squeeze with the Fedora 16 kernel ... as well as entries for every kernel you have that's actually owned by Debian. And then for each of these, you'll also get a second entry, to boot in recovery mode. If you have several distros installed, it makes for a very long and confusing boot menu!

It's a shame that the auto-updater doesn't restrict itself to kernels managed by the packaging system, which would be easy enough to do. (Wonder if they would accept a patch?) You might be able to fudge something that works right by setting up symlinks so that the only readable kernels actually live on the root partition, so Debian can't read the kernels from the other distros. Sounds a bit complicated and I haven't tried it. For now, I've turned off automatic updating on my system.

But if your setup is simpler -- perhaps just one Debian or one Ubuntu partition plus some non-Linux entries such as BSD or Windows -- here's how to set up Debian-style automatic updating and still keep all your non-Linux boot entries and your nice menu customizations.

Debian automatic updates and themes

First, take a quick look at /etc/default/extlinux and customize anything there you might need, like the names of the kernels, kernel boot parameters or timeout. See man extlinux-update for details.

For configuring menu colors, image backgrounds and such, you'll need to make a theme. You can see a sample theme by installing the package syslinux-themes-debian -- but watch out. If you haven't configured apt not to pull in suggested packages, that may bring back grub or grub-legacy, which you probably don't want.

You can make a theme without needing that package, though. Create a directory /usr/share/syslinux/themes/mythemename (the extlinux-update man page claims you can put a theme anywhere and specify it by its full path, but it lies). Create a directory called extlinux inside it, and make a file with everything you want from extlinux.conf. For example:

default 0
prompt 1
timeout 50

ui vesamenu.c32
menu title Welcome to my Linux machine!
menu background mysplash.png
menu color title 1;36 #ffff8888 #00000000 std
menu color unsel 0    #ffffffff #00000000 none
menu color sel   7    #ff000000 #ffffff00 none

include linux.cfg
menu separator
include themes/mythemename/other.cfg

Note that last line: you can include other files from your theme. For instance, you can create a file called other.cfg with entries for other partitions you want to boot:

label oneiric
menu label Ubuntu Oneiric Ocelot
kernel /vmlinuz-3.0.0-12-generic
append initrd=/initrd.img-3.0.0-12-generic root=UUID=c332b3e2-5c38-4c50-982a-680af82c00ab ro quiet

label fedora
menu label Fedora 16
kernel /vmlinuz-3.1.0-7.fc16.i686
append initrd=/initramfs-3.1.0-7.fc16.i686.img root=UUID=47f6b1fa-eb5d-4254-9fe0-79c8b106f0d9 ro quiet

menu separator

LABEL Windows
KERNEL chain.c32
APPEND hd0 1

Of course, you could have a debian.cfg, an ubuntu.cfg, a fedora.cfg etc. if you wanted to have multiple distros all keeping their kernels up-to-date. Or you can keep the whole thing in one file, theme.cfg. You can make a theme as complex or as simple as you like.

How to install extlinux (syslinux) as a bootloader

When my new netbook arrived, I chose Debian Squeeze as the first Linux distro to install, because I was under the impression it still used grub1, and I wanted to avoid grub2. I was wrong -- Squeeze uses grub2. Uninstalling grub2, installing grub-legacy and running grub-install and update-grub didn't help; it turns out even in Debian's grub-legacy package, those programs come from grub2's grub-common package.

What a hassle! But maybe it was a blessing in disguise -- I'd been looking for an excuse to explore extlinux as a bootloader as a way out of the grub mess.

Extlinux is one of the many spinoffs of syslinux -- the bootloader used for live CDs and many other applications. It's not as commonly used as a bootloader for desktops and laptops, but it's perfectly capable of that. It's simple, well tested and has been around for years. And it supports the few things I want out of a bootloader: it has a simple configuration file that lives on the /boot partition; it can chain-load Windows, on machines with a Windows partition; it even offers pretty graphical menus with image backgrounds.

Since there isn't much written about how to use extlinux, I wrote up my experiences along with some tips for configuring it. It came out too long for a blog article, so instead I've made it its own page: How to install extlinux (syslinux) as a bootloader.

Synaptics horizontal scrolling (and other touchpad configurations)

My new netbook (about which, more later) has a trackpad with areas set aside for both vertical and horizontal scrolling. Vertical scrolling worked out of the box on Squeeze without needing any extra fiddling. My old Vaio had that too, and I loved it.

Horizontal scrolling took some extra research. I was able to turn it on with a command:

synclient HorizEdgeScroll=1

But it took a lot more searching to find out the right place to set it permanently. Nearly every page you find on trackpad settings tells you to edit /etc/X11/xorg.conf. Distros haven't normally used xorg.conf in over three years! Sure, you can generate one, then edit it -- but surely there's a better way.

And there is. At least in Debian Squeeze and Ubuntu Natty, you can edit /usr/share/X11/xorg.conf.d/50-synaptics.conf and add this options line inside the "InputClass" section:

        options "HorizEdgeScroll" "1"

In theory, you can use this for any of the Synaptics driver options -- tap rate and sensitivity, even multitouch. Your mileage may vary -- horizontal scroll is the only one I've tested so far. But at least it's easier than generating and maintaining an xorg.conf file!

Making an Ubuntu live USB stick persistent

I wrote a few days ago about my multi-distro Linux live USB stick. Very handy!

But one thing that bugs me about live distros: they're set up with default settings and don't have a lot of the programs I want to use. Even getting a terminal takes quite a lot of clicks on most distros. If only they would save their settings!

It's possible to make a live USB stick "persistent", but not much is written about it. Most of what's written tells you to create the USB stick with usb-creator -- a GUI app that I've tried periodically for the past two years without ever once succeeding in creating a bootable USB stick.

Even if usb-creator did work, it wouldn't work with a multi-boot stick like this one, because it would want to overwrite the whole drive. So how does persistence really work? What is usb-creator doing, anyway?

How persistence works: Casper

The best howto I've found on Ubuntu persistence is LiveCD Persistence. But it's long and you have to wade through a lot of fdisk commands and similar arcana. So here's how to take your multi-distro stick and make at least one of the installs persistent.

Ubuntu persistence uses a package called casper which overlays the live filesystem with the contents of another filesystem. Figuring out where it looks for that filesystem is the key.

Casper looks for its persistent storage in two possible places: a partition with the label "casper-rw", and a file named "casper-rw" at the root of its mounted partitions.

So you could make a separate partition labeled "casper-rw", using your favorite partitioning tool, such as gparted or fdisk. But if you already have your multi-distro stick set up as one big partition, it's just as easy to create a file. You'll have to decide how big to make the file, based on the size of your USB stick.

I'm using a 4G stick, and I chose 512M for my persistent partition:

$ dd if=/dev/zero of=/path/to/casper-rw bs=1M count=512

Next, create a filesystem inside that file. I'm not sure what the tradeoffs are among various filesystem types -- no filesystem is optimized for being run as a loopback file read from a vfat USB stick that was also the boot device. So I flipped a coin and used ext3:

$ mkfs.ext3 /path/to/casper-rw
/path/to/casper-rw is not a block special device.
Proceed anyway? (y,n) y

One more step: you need to add the persistent flag to your boot options. If you're following the multi-distro USB stick tutorial I linked to earlier, that means you should edit boot/grub/grub.cfg on the USB stick, find the boot stanza you're using for Ubuntu, and make the line starting with linux look something like this:

    linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile quiet splash noprompt persistent --

Now write the stick, unmount it, and try booting your live install.

Testing: did it work?

The LiveCD/Persistence page says persistent settings aren't necessarily saved for the default "ubuntu" user, so it's a good idea to make a new user. I did so.

Oops -- about that Ubuntu new user thing

But at least in Ubuntu Oneiric: there's a problem with that. If you create a user, even as class Administrator (and of course you do want to be an Administrator), it doesn't ask you for a password. If you now log out or reboot, your new user should be saved -- but you won't be able to do anything with the system, because anything that requires sudo will prompt you for your nonexistent password. Even attempting to set a password will prompt you for the nonexistent password.

Apparently you can "unlock" the user at the time you create it, and then maybe it'll let you set a password. I didn't know this beforehand, so here's how to set a password on a locked user from a terminal:

$ sudo passwd username

For some reason, sudo will let you do this without prompting for a password, even though you can't do anything administrative through the GUI.

Testing redux

Once you're logged in as your new user, try making some changes. Add and remove some items from the unity taskbar. Install a couple of packages. Change the background.

Now try rebooting. If your casper-rw file worked, it should remember your changes.

When you're not booted from your live USB stick, you can poke around in the filesystem it uses by mounting it in "loopback" mode. Plug the stick into a running Linux machine, mount it the usb stick, then mount it with

$ sudo mount -o loop /path/to/casper-rw /mnt

/path/to is wherever you mounted your usb stick -- e.g. /media/whatever. With the file mounted in loopback mode, you should be able to adjust settings or add new files without needing to boot the live install -- and they should show up the next time you use the live install.

My live Ubuntu Oneiric install is so much more fun to use now!

Creating a multi-distro Linux Live USB stick

Linux live USB sticks (flash drivers) are awesome. You can carry them anywhere and give a demo of Linux on anyone's computer, any time. But how do you keep track of them? Especially since USB sticks don't have any place to write a label. How do you remember that the shiny blue stick is the one with Ubuntu Oneiric, the black one has Ubuntu Lucid, the other blue one that's missing its top is Debian ... and so forth. It's impossible! Plus, such a waste -- you can hardly buy a flash drive smaller than 4G these days, and then you go and devote it to a 700Mb ISO designed to fit on a CD. Silly.

The answer: get one big USB stick and put lots of distros on it, using grub to let you choose at boot time.

To create my stick, I followed the easy instructions at HOWTO: Booting LiveCD ISOs from USB flash drive with Grub2. I found that tutorial quite simple, so I'm not going to duplicate the instructions there. I used the non-LUA version, since my grub on Ubuntu Natty didn't seem to support LUA. Basically you run grub-install to the stick, create a directory called iso where you stick all your ISO files, then create a grub.cfg with magic incantations to boot each ISO.

Ah, wait ... magic incantations? The tutorial is missing one important part: what if you want to use an ISO that isn't already mentioned in the tutorial? If Ubuntu's entry is
linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile quiet splash noprompt -- and Parted Magic's is
linux (loop)/pmagic/bzImage iso_filename=$isofile edd=off noapic load_ramdisk=1 prompt_ramdisk=0 rwnomce sleep=10 loglevel=0 then you know there's some magic going on there.

I knew I needed at least the Ubuntu "alternate installer", since it allows installing a command-line system without the Unity desktop, and Debian Squeeze, since that's currently the most power-efficient Linux for laptops, in addition to the distros mentioned in the tutorial. How do you figure out what to put in those grub.cfg lines? Here's how to figure it out from the ISO file. I'll use the Debian Squeeze ISO as an example.

Step 1: mount the ISO file.

$ sudo mount -o loop /pix/boot/isos/debian-6.0.0-i386-netinst.iso /mnt

Step 2: find the kernel

$ ls /mnt/*/vmlinuz /mnt/*/bzImage
/mnt/install.386/vmlinuz

Step 3: find the initrd. It might have various names, and might or might not be compressed, but the name will almost always start with init.

$ ls /mnt/*/vmlinuz /mnt/*/init*
/mnt/install.386/initrd.gz

Unmount the ISO file.

$ umount /mnt

The trick in steps 2 and 3 is that nearly all live ISO images put the kernel and initrd a single directory below the root. If you're using an ISO that doesn't, you may have to search more deeply (try /mnt/*/*).

In the case of Debian Squeeze, now I have the two filenames: /install.386/vmlinuz and /install.386/initrd.gz. (I've removed the /mnt part since that won't be there when I'm booting from the USB stick.) Now I can edit boot/grub/grub.cfg and make a boot stanza for Debian:

menuentry "Debian Squeeze" {
    set isofile="/boot/isos/debian-6.0.0-i386-netinst.iso"

    loopback loop $isofile 
    linux (loop)/install.386/vmlinuz iso_filename=$isofile quiet splash noprompt --
    initrd (loop)/install.386/initrd.gz
}

Here's the entry for the Ubuntu alternate installer:

menuentry "Oneiric 11.10 alternate" {
    set isofile="/boot/isos/ubuntu-11.10-alternate-i386.iso"
 
    loopback loop $isofile 
    linux (loop)/install/vmlinuz iso_filename=$isofile
    initrd (loop)/install/initrd.gz
}

It sounds a little convoluted, I know -- but you only have to do it once, and then you have this amazing keychain drive with every Linux distro on it you can think of. Amaze your friends!

List only directories

Fairly often, I want a list of subdirectories inside a particular directory. For instance, when posting blog entries, I may need to decide whether an entry belongs under "linux" or some sub-category, like "linux/cmdline" -- so I need to remind myself what categories I have under linux.

But strangely, Linux offers no straightforward way to ask that question. The ls command lists directories -- along with the files. There's no way to list just the directories. You can list the directories first, with the --group-directories-first option. Or you can flag the directories specially: ls -F appends a slash to each directory name, so instead of linux you'd see linux/. But you still have to pick the directories out of a long list of files. You can do that with grep, of course:

ls -1F ~/web/blog/linux | grep /

Or you can use the find program, which has an option -type d that lists only directories. Perfect, right?

find ~/web/blog/linux -maxdepth 1 -type d

Except that lists everything with full pathnames: /home/akkana/web/blog/linux, /home/akkana/web/blog/linux/editors, /home/akkana/web/blog/linux/cmdline and so forth. Way too much noise to read quickly.

What I'd really like is to have just a list of directory names -- no slashes, no newlines. How do we get from ls or find output to that? Either we can start with find and strip off all the path information, either in a loop with basename or with a sed command; or start with ls -F, pick only the lines with slashes, then strip off those slashes. The latter sounds easier.

So let's go back to that ls -1F ~/web/blog/linux | grep / command. To strip off the slashes, you can use sed's s (substitute) command. Normally the syntax is sed 's/oldpat/newpat/'. But since slashes are the pattern we're substituting, it's better to use something else as the separator character. I'll use an underscore.

The old pattern, the one I want to replace, is / -- but I only want to replace the last slash on the line, so I'll add a $ after it, representing end-of-line. The new pattern I want instead of the slash is -- nothing.

So my sed argument is 's_/$__' and the command becomes:

ls -1F ~/web/blog/linux | grep / | sed 's_/$__'

That does what I want. If I don't want them listed one per line, I can fudge that using backquotes to pass the output of the whole command to the shell's echo command:

echo `ls -1F ~/web/blog/linux | grep / | sed 's_/$__'`

If you have a lot of directories to list and you want ls's nice columnar format, that's a little harder. You can ls the list of directories (the names inside the backquotes), ls `your long command` -- except that now that you've stripped off the path information, ls won't know where to find the files. So you'd have to change directory first:

cd ~/web/blog/linux; ls -d `ls -1F | grep / | sed 's_/$__'`

That's not so good, though, because now you've changed directories from wherever you were before. To get around that, use parentheses to run the commands inside a subshell:

(cd ~/web/blog/linux; ls -d `ls -1F | grep / | sed 's_/$__'`)

Now the cd only applies within the subshell, and when the command finishes, your own shell will still be wherever you started.

Finally, I don't want to have to go through this discovery process every time I want a list of directories. So I turned it into a couple of shell functions, where $* represents all the arguments I pass to the command, and $1 is just the first argument.

lsdirs() { 
  (cd $1; /bin/ls -d `/bin/ls -1F | grep / | sed 's_/$__'`)
}

lsdirs2() { 
  echo `/bin/ls -1F $* | grep / | sed 's_/$__'` 
}

Now I can type lsdirs ~/web/blog/linux and get a nice list of directories.

echo ~/web/blog/linux/*(/:t)

Vaio tips for Debian Squeeze

I switched to the current Debian release, "Squeeze", quite a few months ago on my Sony Vaio laptop. I've found that Squeeze, with its older kernel and good attention to power management (compared to the power management regressions in more recent kernels), gets much better battery life than either Arch Linux or Ubuntu on this machine. I'm using Squeeze as the primary OS at least until the other distros get their kernel power management sorted out.

I did have to solve a couple of minor problems when switching over, though.

Suspend/Resume quirks

The first problem was that my Vaio TX650 would freeze on resuming from suspend -- something that every other Linux distro has handled out of the box on this machine.

The solution turned out to be simple though non-obvious, apparently a problem with controlling power to the display:

sudo pm-suspend --quirk-dpms-on

That wasn't easy to find, but ever since then the machine has been suspending without a single glitch. And it's a true suspend, unlike Ubuntu Natty, which on this machine will use up a full battery if I leave it suspended all day -- Natty uses nearly as much power when suspended as it does running.

Adjusting screen brightness: debugging ACPI

Of course, once I got that sorted out, there were the usual collection of little changes I needed to make. Number one was that it didn't automatically handle brightness adjustment with the Fn-F5 and Fn-F6 keys.

It turned out my previous technique for handling the brightness keys didn't work, because the names of the ACPI events in /etc/acpi/events had changed. Previously, /etc/acpi/events/sony-brightness-down had contained references to the Sony I/O Control, or SPIC:

event=sony/hotkey SPIC 00000001 00000010
action=/etc/acpi/sonybright.sh down

event=video/brightnessdown BRTDN 00000087 00000000
action=/etc/acpi/sonybright.sh down

It's probably a good thing, changing to be less Sony-specific ... but as a user it's one of those niggling annoyances that I have to go chase down every time I upgrade to a new Linux version.

Three SSH tips

Clearing ssh control sockets

We had a network failure recently while I had a few ssh connections open. After the network came back up, when I tried to ssh to one host, it always complained Control socket connect(/home/username/ssh-username@example.com:port): Connection refused -- but then it proceeded to connect anyway. Another server simply failed to connect.

Here's how to fix that: on the local machine -- not the remote one -- there's a file named /home/username/ssh-username@example.com:port. Remove that file, and ssh will work normally again.

Connection Sharing

I think the stuck control socket happened because I was using ssh connection sharing, a nifty feature introduced a few years back that lets ssh-based commands re-use an existing connection without re-authenticating.

Suppose you have an interactive ssh session to a remote host, and you need to copy some files over with a program like scp. Normally, each scp command needs to authenticate with remotehost, sometimes more than once per command. Depending on your setup and whether you're running a setup like ssh-agent, that might mean you have to retype your password several times, or wait while it verifies your host key.

But you can make those scps re-use your existing connection. Add this to .ssh/config:

Host *
  ControlMaster auto
  ControlPath ~/ssh-%r@%h:%p

Eliminating strict host key checking

The final tip is for my biggest ssh pet peeve: strict host key checking. That's the one where you ssh to a machine you use all the time and you get:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

• that machine has re-installed its OS, or rebooted into a different distro; or
• the DHCP server has given that IP address to a different machine from the one that had it last time.

The really frustrating thing is that there's no flag you can pass to ssh to tell it "look, it's fine, just let me in." The only solution is to edit ~/.ssh/known_hosts, find the line corresponding with that host (not so easy if you've forgotten to add HashKnownHosts no) so you can actually see the hostnames) and delete it -- or delete the whole ~/.ssh/known_hosts file. ssh does have an option for StrictHostKeyChecking no, and the documentation implies it might help; but it doesn't get you past this error, and it doesn't prevent ssh asking for confirmation when it sees a new host, either. I'm not sure what it actually does.

What does get you past the error? Here's a fun trick. Add a stanza like this to .ssh/config:

Host 192.168.1.*
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

Translation: for every host on your local net (assuming it's 192.168.1), pretend /dev/null has the list you'd normally get from ~/.ssh/known_hosts. Since there's definitely no host line there matching your intended remote host, it will ask you whether it's okay, then connect.

I wish I could find a way to eliminate the prompt too (I thought that was what StrictHostKeyChecking no was supposed to do); but at least you can just hit return, which is a lot easier than editing known_hosts every time.

And yes, this all makes ssh less secure. You know what? For hosts on my local network, that are sitting in the same room with me, I'm really not too concerned about that.

Making static devices in UDEV

For the first time after installing Ubuntu Natty, I needed to upload some photos from my camera -- and realized with a sinking feeling that I now had the new UDEV, which no longer lets you use the udev all_partitions directive, so that cards inserted into a multi flash card reader will show up as /dev/sdb1 or whatever the appropriate device name is.

Without all_partitions, you get the initial sdb, sdc, sdd and sde for the various slots in the card reader, but since there's no card there when the machine boots, and the reader doesn't send an event when you insert a card later, you never get a mountable /dev/sdb1 device.

But the udev developers in their infinite wisdom removed all_partitions some time last year, apparently without providing any replacement for it. So you can no longer solve this problem through udev rules.

Static udev devices

To begin, first find out what your base devices are. Look through dmesg | more for your card reader. Mine looks something like this:

[    3.304938] scsi 4:0:0:0: Direct-Access     Generic  USB SD Reader    1.00 PQ
: 0 ANSI: 0
[    3.305440] scsi 4:0:0:1: Direct-Access     Generic  USB CF Reader    1.01 PQ
: 0 ANSI: 0
[    3.305939] scsi 4:0:0:2: Direct-Access     Generic  USB xD/SM Reader 1.02 PQ
: 0 ANSI: 0
[    3.306438] scsi 4:0:0:3: Direct-Access     Generic  USB MS Reader    1.03 PQ
: 0 ANSI: 0
[    3.306876] sd 4:0:0:0: Attached scsi generic sg1 type 0
[    3.307020] sd 4:0:0:1: Attached scsi generic sg2 type 0
[    3.307165] sd 4:0:0:2: Attached scsi generic sg3 type 0
[    3.307293] sd 4:0:0:3: Attached scsi generic sg4 type 0
[    3.313181] sd 4:0:0:1: [sdc] Attached SCSI removable disk
[    3.313806] sd 4:0:0:0: [sdb] Attached SCSI removable disk
[    3.314430] sd 4:0:0:2: [sdd] Attached SCSI removable disk
[    3.315055] sd 4:0:0:3: [sde] Attached SCSI removable disk

Notice that the SD reader is scsi 4:0:0:0, and a few lines later, 4:0:0:0 is mapped to sdb. They're out of order, so make sure you match those scsi numbers. If I want to read SD cards, /dev/sdb is where to look.

(Note: sd in "sdb" stands for "SCSI disk", while SD in "SD card" stands for "Secure Digital". Two completely different meanings for the same abbreviation -- just an unfortunate coincidence to make this all extra confusing.)

To create static devices, I'll need the major and minor device numbers for the devices I want to create. Since I know the SD card slot is sdb, I can get those with ls:

$ ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 2011-04-20 09:43 /dev/sdb

Now you can create new static block devices, as root, using mknod in the /lib/udev/devices directory:

$ sudo mknod b 8 17 /lib/udev/devices/sdb1
$ sudo mknod b 8 18 /lib/udev/devices/sdb2
$ sudo mknod b 8 19 /lib/udev/devices/sdb3

Although my camera only uses one partition, sdb1, I created devices for a couple of extra partitions because I sometimes partition cards that way. If you only use flash cards for cameras and MP3 players, you may not need anything beyond sdb1.

You can make devices for the other slots in the card reader the same way. The memory stick reader showed up as scsi 4:0:0:3 or sde, and /dev/sde has device numbers 8, 64 ... so to read the memory stick from Dave's Sony camera, I'd need:

$ sudo mknod b 8 65 /lib/udev/devices/sde1

You don't have to call the devices sdb1, either. You can call them sdcard1 or whatever you like. However, the base device will still be named sdb (unless you write a udev rule to change that).

fstab entry

I like to use fstab entries and keep control over what's mounted, rather than letting the system automatically mount everything it sees. I can do that with this entry in /etc/fstab:

/dev/sdb1 /sdcard vfat user,noauto,exec,fmask=111,shortname=lower 0 0

Now, whenever I insert an SD card and want to mount it, I type mount /sdcard as myself. There's no need for sudo because of the user directive.

Installing Debian Squeeze

I used a "net install" CD, the one that installs only the bare minimum then goes to the net for anything else. I used Expert mode, because I needed to set a static IP address and keep it from overwriting my grub configuration.

Most of the install went smoothly -- until I got to the last big step near the end, "Select and install software", where it froze at 1%.

A little web searching (on another machine) gave me the hint that the Debian installer prints a log on the fourth console, Ctrl-Alt-F4. Checking that log made the problem clear: aptitude was complaining about packages without a proper GPG signature -- type Yes to continue without verifying signatures. But since this was running inside the installer, there's no place to type Yes -- that Ctrl-Alt-F4 console is merely displaying messages, not accepting input, and the installer doesn't accept any input for aptitude.

Fortunately, "Select and install software" isn't crucial to the net install process. I don't actually know what software it would have installed -- it never asked me to choose any -- but without it, you should still have a working minimal Debian on the disk. So I made another console on Ctrl-Alt-F2, ran ps aux, found that aptitude was the highest numbered process running, and killed it. Upon returning to the installer (Ctrl-Alt-F1), I was able to skip "Select and install software", finish the install process and reboot.

Upon rebooting, I logged in as root and ran apt-get update. It complained about GPG errors; but now I could do something about it. I ran apt-get upgrade and confirmed that I wanted to proceed even without verifying package signatures. When that was over, the problem was fixed: a subsequent apt-get update ran without errors.

This ISO was downloaded (from the kernel.org mirror, I believe) a few days after the official release. I'm told that Debian changes the keys at the last minute before a release; perhaps the new keys don't make it into the ISO images on all the mirrors. Or maybe they just messed up with the Squeeze release.

Anyway, it was fairly easily solved, but seemed like a disappointing and silly problem. A web search found lots of people people hitting this problem; it's a shame that the installer can't run aptitude in a mode where it won't prompt and hang up the whole install.

Alas, it's probably all academic anyway, since suspend/resume doesn't work. It freezes on resume, with a black screen -- another common Debian problem, judging by what I see on the net. I'm a bit surprised, since every other distro I've tried has suspended the Vaio beautifully. But after hours of messing with it over the weekend, I ran out of time and conceded defeat.

Making CapsLock equal Control in Debian Squeeze

It's time for another installment of "Where have the control/capslock adjustments migrated to?" This time it's for the latest Debian release, "Squeeze".

Ever since they stopped making keyboards with the control key to the left of the A, I've remapped my CapsLock key to be another Control key. I never need CapsLock, but I use Control constantly all day while editing text. Some people prefer to swap Control and CapsLock.

But the right way to do that changes periodically. For the last few years, since Ubuntu Intrepid, you could set XKbOptions for Control and Capslock in /etc/default/console-setup. But that no longer works in Squeeze.

It turns out Squeeze introduced a new file, /etc/default/keyboard, so any keyboard options previously had in console-setup need to move to keyboard. For me, that's these lines:

XKBMODEL="pc104"
XKBLAYOUT="us"
XKBVARIANT=""
XKBOPTIONS="ctrl:nocaps,compose:menu,terminate:ctrl_alt_bksp"

This wasn't well covered on the web. There aren't many howtos covering Squeeze yet, but I found the hint I needed in a terse Debian IRCbot factoid: Factoid capslock says

For console-setup, append ",ctrl:nocaps" to the value of XKBOPTIONS within /etc/default/console-setup (/etc/default/keyboard on Squeeze).

That factoid assumes you already have XKBOPTIONS set; as shipped, it's empty, so skip that initial comma.

I was going to conclude with a link to the documentation on XKBOPTIONS, or XKbOptions as it was capitalized in xorg.conf ... but there doesn't seem to be any. It's not in any of the Xorg man pages like xorg.conf(5) where I expected to find it; nor can I find anything on the web beyond howtos like this one from people who have figured out a few specific options. Anyone know?

Using grep to solve another Cartalk puzzler

Last week's puzzler was A Seven-Letter Vacation Curiosity. Basically, one couple hiking in Northern California and another couple carousing in Florida both see something described by a seven-letter word containing all five vowels -- but the two things they saw were very different. What's the word?

That's an easy one to solve using basic Linux command-line skills -- assuming the word is in the standard dictionary. If it's some esoteric word, all bets are off. But let's try it and see. It's a good beginning exercise in regular expressions and how to use the command line.

There's a handy word list in /usr/share/dict/words, one word per line. Depending on what packages you have installed, you may have bigger dictionaries handy, but you can usually count on /usr/share/dict/words being there on any Linux system. Some older Unix systems may have it in /usr/dict/words instead.

We need a way to choose all seven letter words. That's easy. In a regular expression, . (a dot) matches one letter. So ....... (seven dots) matches any seven letters.

(There's a more direct way to do that: the expression .\{7\} will also match 7 letters, and is really a better way. But personally, I find it harder both to remember and to type than the seven dots. Still, if you ever need to match 43 characters, or 114, it's good to know the "right" syntax.)

Fine, but if you grep ....... /usr/share/dict/words you get a list of words with seven or more letters. See why? It's because grep prints any line where it finds a match -- and a word with nine letters certainly contains seven letters within it.

The pattern you need to search for is '^.......$' -- the up-caret ^ matches the beginning of a line, and the dollar sign $ matches the end. Put single quotes around the pattern so the shell won't try to interpret the caret or dollar sign as special characters. (When in doubt, it's always safest to put single quotes around grep patterns.)

So now we can view all seven-letter words: grep '^.......$' /usr/share/dict/words
How do we choose only the ones that contain all the letters a e i o and u?

That's easy enough to build up using pipelines, using the pipe character | to pipe the output of one grep into a different grep. grep '^.......$' /usr/share/dict/words | grep a sends that list of 7-letter words through another grep command to make sure you only see words containing an a.

Now tack a grep for each of the other letters on the end, the same way:
grep '^.......$' /usr/share/dict/words | grep a | grep e | grep i | grep o | grep u

Voilà! I won't spoil the puzzler, but there are two words that match, and one of them is obviously the answer.

The power of the Unix command line to the rescue!

SCALE9x

Talks

A few standout talks:

Leigh Honeywell's keynote was a lively and enjoyable discussion of hackerspaces -- from the history of the movement to a discussion of some of the coolest and most innovative hackerspaces around today. She had plenty of stories and examples that left everyone in the audience itching to get involved with a local hackerspace, or start one if necessary.

John Wise and Eugene Clement of LinuxAstronomy.com presented the entertaining "A Reflection on Classroom Robotics with Linux Robots in classrooms". They've taught kids to build and program robots that follow lines, solve mazes, and avoid obstacles. The students have to figure out how to solve problems, details like when and how far to back up. What a fantastic class! I can't decide if I'd rather teach a class like that or take it myself ... but either way, I enjoyed the presentation. They also had a booth in the exhibit hall where they and several of their students presented their Arduino-based robots exploring simulated Martian terrain.

Jonathan Thomas spoke about his OpenShot video editor and the development community behind it, with lots of video samples of what OpenShot can do. Sounds like a great program and a great community as well: I'll definitely be checking out OpenShot next time I need to edit a video.

It's worth mentioning that both the robotics talk and the OpenShot one were full of video clips that ran smoothly without errors. That's rare at conferences -- videos so often cause problems in presentations (OpenOffice is particularly bad at them). These presenters made it look effortless, which most likely points to a lot of preparation and practice work beforehand. Good job, guys!

Larry Bushey's "Produce An Audio Podcast Using Linux" was clear and informative, managing to cover the technology, both hardware and software, and the social factors like how often to broadcast, where to host, and how to get the word out and gain and keep listeners while still leaving plenty of time for questions.

The Exhibit Hall

In between talks I tried to see some of the exhibit hall, which was tough, with two big rooms jam-packed with interesting stuff.

Aside from LinuxAstronomy and their robots, there were several other great projects for getting technology into schools: Partimus from the bay area, and Computers4Kids more local to LA, both doing excellent work.

The distro booths all looked lively. Ubuntu California's booth was always so packed that it was tough getting near to say hi, Fedora was well attended and well stocked with CDs, and SuSE had a huge array of givaways and prizes. Debian, Gentoo, Tiny Core and NetBSD were there as well.

Distro Dilemma and "the Hallway track"

Late in the game I discovered even Arch Linux had a booth hidden off in a corner. I spent some time there hoping I might get help for my ongoing Arch font rendering problem, but ended up waiting a long time for nothing. That left me with a dilemma for my talk later that day: Arch works well on my laptop except that fonts sometimes render with chunks missing, making them ugly and hard to read; but a recent update of Ubuntu Lucid pulled in some weird X change that keeps killing my window manager at unpredictable times. What a choice! In the end I went with Ubuntu, and indeed X did go on the fritz, so I had to do without my live demo and stick to my prepared slides. Not a tragedy, but annoying. The talk went well otherwise.

I had a great conversation with Asheesh from the OpenHatch project about how to make open source projects more welcoming to new contributors. It's something I've always felt strongly about, but I feel powerless to change existing projects so I don't do anything. Well, OpenHatch is doing something about it, and I hope I'll be able to help.

The Venue

Not everything was perfect. The Hilton is a new venue for SCALE, and there were some issues. On Saturday, every room was full, with people lining the walls and sitting on floors. This mostly was not a room size problem, merely a lack of chairs. Made me wonder if we should go all opensource on them and everybody bring their own lawn chair if the hotel can't provide enough.

Parking was a problem too. The Hilton's parking garage fills up early, so plan on driving for ten minutes through exhaust-choked tunnels hoping to find a space to squeeze into. We got lucky, so I didn't find out if you have to pay if you give up and exit without finding a spot.

Then Sunday afternoon they ran short of validation tickets (the ones that reduce the cost from $22 to $9), and it wasn't clear if there was any hope of more showing up (eventually some did).

To top it off, when we finally left on Sunday the payment machine at the exit swallowed my credit card, requiring another 15 minutes of waiting for someone to answer the buzzer. Eventually the parking manager came down to do a magic reset rite.

So I didn't come away with a great impression of the Hilton. But it didn't detract much from a wonderful conference full of interesting people -- I had a great time, and would (and do) recommend SCALE to everyone with any interest in Linux. But it left me musing about the pros and cons of different venues ... a topic I will discuss in a separate post.

Article: Building kernels for plug computers

It covers how to find and install a cross-compiler (sadly, your Linux distro probably doesn't include one), configuring the Linux kernel build, and a few gotchas and things that I've found not to work reliably.

It took me a lot of trial and error to figure out some of this -- there are lots of howtos on the web but a lot of them skip the basic steps, like the syntax for the kernel's CROSS_COMPILE argument -- but you'll need it if you want to enable any unusual drivers like GPIO. So good luck, and have fun!

Unbricking Plug Computers

Here it is: Un-Bricking Linux Plug Computers: uBoot, iBoot, We All Boot for uBoot.

If you want more exhaustive detail, particular on those uBoot scripts and how they work, I go through some of the details in a brain-dump I wrote after two weeks of struggling to unbrick my first GuruPlug: Building and installing a new kernel for a SheevaPlug. But don't worry if that page isn't too clear; I'll cover the kernel-building part more clearly in my next LinuxPlanet article on Feb. 24.

Getting Started with Plug Computers

Although they run standard Linux (usually Debian or Ubuntu), there are some gotchas to choosing and installing plug computers. So this week's article starts with the basics of choosing a model and connecting to it; part II, in two weeks, will address more difficult issues like how to talk to uBoot, flash a new kernel or recover if things go wrong.

Read part I here: Tiny Linux Plug Computers: Wall Wart Linux Servers.

X Terminal Colors (and dark and light backgrounds)

How, then, can I set things up so that both dark- and light-backgrounded people can use the account? I could set up a script that would set up a different set of aliases and configuration files, like when I changed my vim colors. Better, I could fix all of them at once by changing my terminal's idea of colors -- so when the remote machine thinks it's feeding me a light color, I see a dark one.

I use xterm, which has an easy way of setting colors: it has a list of 16 colors defined in X resources. So I can change them in ~/.Xdefaults.

That's all very well. But first I needed a way of seeing the existing colors, so I knew what needed changing, and of testing my changes.

Script to show all terminal colors

I thought I remembered once seeing a program to display terminal colors, but now that I needed one, I couldn't find it. Surely it should be trivial to write. Just find the escape sequences and write a script to substitute 0 through 15, right?

Except finding the escape sequences turned out to be harder than I expected. Sure, I found them -- lots of them, pages that conflicted with each other, most giving sequences that didn't do anything visible in my xterm.

Eventually I used script to capture output from a vim session to see what it used. It used <ESC>[38;5;Nm to set color N, and <ESC>[m to reset to the default color. This more or less agreed Wikipedia's ANSI escape code page, which says <ESC>[38;5; does "Set xterm-256 text coloor" with a note "Dubious - discuss". The discussion says this isn't very standard. That page also mentions the simpler sequence <ESC>[0;Nm to set the first 8 colors.

Okay, so why not write a script that shows both? Like this:

#! /usr/bin/env python

# Display the colors available in a terminal.

print "16-color mode:"
for color in range(0, 16) :
    for i in range(0, 3) :
        print "\033[0;%sm%02s\033[m" % (str(color + 30), str(color)),
    print

# Programs like ls and vim use the first 16 colors of the 256-color palette.
print "256-color mode:"
for color in range(0, 256) :
    for i in range(0, 3) :
        print "\033[38;5;%sm%03s\033[m" % (str(color), str(color)),
    print

Voilà! That shows the 8 colors I needed to see what vim and ls were doing, plus a lovely rainbow of other possible colors in case I ever want to do any serious ASCII graphics in my terminal.

Changing the X resources

The next step was to change the X resources. I started by looking for where the current resources were set, and found them in /etc/X11/app-defaults/XTerm-color:

$ grep color /etc/X11/app-defaults/XTerm-color
irrelevant stuff snipped
*VT100*color0: black
*VT100*color1: red3
*VT100*color2: green3
*VT100*color3: yellow3
*VT100*color4: blue2
*VT100*color5: magenta3
*VT100*color6: cyan3
*VT100*color7: gray90
*VT100*color8: gray50
*VT100*color9: red
*VT100*color10: green
*VT100*color11: yellow
*VT100*color12: rgb:5c/5c/ff
*VT100*color13: magenta
*VT100*color14: cyan
*VT100*color15: white
! Disclaimer: there are no standard colors used in terminal emulation.
! The choice for color4 and color12 is a tradeoff between contrast, depending
! on whether they are used for text or backgrounds.  Note that either color4 or
! color12 would be used for text, while only color4 would be used for a
! Originally color4/color12 were set to the names blue3/blue
!*VT100*color4: blue3
!*VT100*color12: blue
!*VT100*color4: DodgerBlue1
!*VT100*color12: SteelBlue1

So all I needed to do was take the ones that don't show up well -- yellow, green and so forth -- and change them to colors that work better, choosing from the color names in /etc/X11/rgb.txt or my own RGB values. So I added lines like this to my ~/.Xdefaults:

!! color2 was green3
*VT100*color2: green4
!! color8 was gray50
*VT100*color8: gray30
!! color10 was green
*VT100*color10: rgb:00/aa/00
!! color11 was yellow
*VT100*color11: dark orange
!! color14 was cyan
*VT100*color14: dark cyan

Now I can share accounts, and I no longer have to curse at those default ls and vim settings!

Update: Tip from Mikachu: ctlseqs.txt is an excellent reference on terminal control sequences.

Article: Arch Linux

I've been quite favorably impressed with Arch. It's a good, solid, straightforward distro that's very well suited to folks who like to administer their systems via the command-line -- or who want to learn how to do that.

I've been running it on my laptop for a few months, because it has excellent performance, without a lot of the bloatware you see in a lot of other distros, and it boots fast.

The only real problem I've had involves fonts. I see nasty font artifacts -- sometimes subtle, a line or a few pixels missing from certain letters -- but sometimes severe, as in this screenshot or this one. In the article I talk about some solutions I've found that make the problems less bad, but I haven't found any way to make them go away entirely.

Unfortunately, since the font problems are worst inside browsers and I use my laptop for presentations at conferences, this may eventually drive me off Arch. I hope not -- I hope I can find a solution -- because otherwise, Arch has been nothing short of a pleasure.

Article: Troubleshooting, part II: /proc and Python

My article this week on Linux Planet concerns Advanced Linux Server Troubleshooting (part 2).

It's two loosely related topics: exploring the /proc filesystem, and how to use it to find information on a running process; and several ways to get stack traces from Python programs.

This (as well as Troubleshooting part I) arose from a problem we had at work, where we use Linux plug computers (ARM-based Linux appliances) running Python scripts. It's not uncommon for Python networking scripts to go into never-never-land, waiting forever on a network connection without timing out. Since plug computers tend not to be outfitted with the latest and greatest tools like gdb and debug versions of libraries, we've needed to find more creative ways of figuring out what processes are doing to make sure our programs are ready for anything.

Article: Troubleshooting running processes, part I

This is part I, covering basics like top, strace and gdb. Part II will get into hairier stuff and tips for debugging Python applications.

Creating and mounting a LUKS encrypted disk

But modern Gnome systems can't read cryptoloop. Or, rather, they can, but you have to fiddle with them as root -- they won't recognize and mount the filesystem automatically.

It turns out that's because the "new way", instead of cryptoloop, is to use a system called LUKS. But it has a few pitfalls, and there's no documentation about how to use it on a system that doesn't recognize it automatically. So here's some.

Creating a LUKS filesystem

The easiest way is to use a program called palimpsest, available on Ubuntu in the gnome-disk-utility package. Run palimpsest with no arguments; click on the appropriate storage device, then click the obvious buttons to create partitions, label and format them. Click on the box to encrypt the partition, type your password, then sit back and wait while it creates the partition.

The label you give the partition is important: it will be used later to mount it.

All straightforward, right? Except for the one part that isn't: there's a button for safely removing the device after the busy cursor has stopped, and it never works. It always says the device is busy. Running a sync from a terminal doesn't work; waiting ten minutes doesn't help. So just shrug, quit palimpsest and eject the device. If you're lucky it created everything okay.

Mounting a LUKS filesystem from Gnome

In theory, you should be able to plug in the device and after a few seconds Gnome will prompt for your password. If it doesn't, which sometimes happens, maybe you killed palimpsest too early; try again and wait longer this time. If it still doesn't work, maybe the commandline will.

Mounting a LUKS filesystem from the commandline

Assuming you used the partition label "secret" when you created the LUKS encrypted partition, the physical partition is on /dev/sdb2, and you want to mount it on /media/secret (which already exists), these two commands (as root) will mount it:

sudo cryptsetup luksOpen /dev/sdb2 secret
  (prompts for sudo password)
  (prompts for LUKS password)
sudo mount /dev/mapper/secret /media/secret

Easy -- yet a bit frustrating. There seems to be no way to do this purely through /etc/fstab, so you have to remember the cryptsetup command, or write an alias or script to do the two steps for you. And you always have to type your sudo password as well as the password for the filesystem, whereas with cryptoloop you only needed the filesystem's password.

In the end, I'm not convinced LUKS is a win. But since it's so hard to manage cryptoloop filesystems from a Gnome desktop, it's probably worth hassling with LUKS if you need to be able to interoperate with Gnome.

Update: I wrote that yesterday. Today, maybe three weeks after I started using the card on a fairly regular basis to transfer personal files between home and laptop, I had a filesystem failure: I wrote to the card from the desktop, synced, unmounted, put it in the laptop -- and got I/O errors and "You must specify filesystem type" trying to mount it. I was able to fsck, and it apparently restored from an old journal -- including old data.

No loss here, because the card is just a copy of what was on the desktop machine. But the lesson here is: these encrypted cards are great for emergency backups. But you probably don't want to rely on one as your main storage for anything important.

"Who am I?" Maybe nobody!

He was using the command who am i -- reasonable, since it's been a staple since the early days of Unix. For those not familiar with the command, /usr/bin/who, if given two arguments, regardless of what those arguments are, will print information about the current logged-in user. It also offers a -m option to do the same thing. So who am i, who a b, and who -m should all print a line like:

$ who am i
akkana   pts/1        2010-09-29 09:33 (:0.0)

Except they don't. For me, they printed nothing at all -- which broke my colleague's install script.

A quick poll among friends on IRC showed that who am i worked for some people, failed for others, with no obvious logic to it.

It's the terminal

It took some digging to find out what was going on, but the difference turned out to be the terminal being used. The who program -- with or without -m -- gets its info from /var/run/utmp, a file that maintains a record of who's logged in to the system. And it turns out some terminals create a utmp entry, while others don't. So:

Program	Creates utmp entry?
gnome-terminal	yes
konsole	yes
xterm	no
xfterm4	yes
terminator	no
rxvt	no
roxterm	yes

I use xterm myself. Xterm is documented (in its man page) to modify the utmp entry, and it has a command-line flat, +ut, plus two X resources, ptyHandshake and utmpInhibit. None of the three work: setting

XTerm*ptyHandshake: true
XTerm*utmpInhibit: false

How do you get the real user?

Okay, so who am i clearly isn't a reliable way of getting the user ID. What can you use instead?

Several people suggested the id program. It has a -r option which supposedly prints the real UID. Unfortunately, what it really does is print:

$ id -r
id: cannot print only names or real IDs in default format

Update: people keep suggesting id -ru to me. Evidently I wasn't very clear in this article: the goal is to get the real id of the login user. In other words, if you're logged in as mary and using sudo, you want mary, not root.

Alas, adding -u to id's flags gets only the effective user id: -u wins over -r. This is very easy to test: sudo id -ru prints 0, as does id -ru inside su.

But elly on Freenode had a great suggestion:

stat -c '%U' `readlink /proc/self/fd/0`

/proc/self is a symlink to /proc/pid, a directory where you can find out all sorts of information about a process.

One of the things you can find out about a process is open file descriptors: in particular, standard input, output and error. So /proc/self/fd/0 corresponds to standard input of the current process -- which in the example above is readlink.

What is readlink? Well, /proc/self/fd/0, in the normal case, is actually a symlink to the terminal controlling the process. readlink prints the file to which that link points -- for instance, /dev/pts/1. That's the terminal being used.

Now that we know the name of the terminal, all we need to do is find out who owns it. (This is the information who am i would have gotten from utmp, had there been a utmp entry.) ls -l /dev/pts/1 will show you that it's you, even if you run it as sudo ls -l /dev/pts/1. You could take that and strip off fields to get the username, but stat, as elly suggested, is a much better way of doing that.

Put it all together, and stat -c '%U' `readlink /proc/self/fd/0 gets standard input for the current process, follows the link to get the controlling terminal, then finds out who owns that terminal.

That's you!

A similar but slightly shorter solution suggested by Mikachu: stat -c %u `tty`

Of Laptops, Docking Stations and Changing Resolution

At the office I have a Lenovo docking station, attached to a monitor and keyboard. The monitor, naturally, has a different resolution from the laptop's own monitor.

Under Gnome and compiz, when I plugged in the monitor, I could either let the monitor mirror the laptop display -- in which case X would refuse to work at greater than 1024x768, much smaller than the native resolution of either the laptop screen or the external monitor -- or I could call up the classic two-monitor configuration dialog, where I could configure the external monitor to be its correct size and sit alongside the computer's monitor. I had to do this every time I plugged in.

If I wanted to work on the big screen, then when I undocked, I had to drag all the windows on all desktops back to the built-in LCD first, or they'd be lost. Using just the external monitor and turning off the laptop screen didn't seem to be an allowed option.

That all lasted for about two days. Gnome and I just don't get along. Pretty soon gdm was mysteriously refusing to let me log in (probably didn't like my under-1000 user id), and after wasting half a day fighting it I gave up and reverted with relief to my familiar Openbox desktop.

But now I'm in the Openbox world and don't have that dialog anyway. What are my options?

xrandr monitor detection

Fortunately, I already knew about using xrandr to send to a projector; it was only a little more complicated using it for the monitor in the docking station. Running xrandr with no arguments prints all the displays it currently sees, so you can tell whether an external display or projector is connected and even what resolutions it supports.

I used that for a code snippet in my .xinitrc:

# Check whether the external monitor is connected: returns 0 on success
xrandr | grep VGA | grep " connected "
if [ $? -eq 0 ]; then
  xrandr --output VGA1 --mode 1600x900;
  xrandr --output LVDS1 --off
else
  xrandr --output VGA1 --off
  xrandr --output LVDS1 --mode 1280x800
fi

That worked nicely. When I start X it checks for an external monitor, and if it finds one it turns off the laptop display (so it's off when the laptop is sitting closed in the docking station) and sets the screen size for the external monitor.

Making it automatic

All well and good. I worked happily all day in the docking station, suspended the laptop and un-docked it, brought it home, woke it up -- and of course the display was still off. Oops.

Okay, so it also needs the same check when resuming from suspend. That used to be in /etc/acpi/resume.d, but in Lucid they've moved it (because we definitely wouldn't want users to get complacent and think they know how to configure things!) and now it lives in /etc/pm/sleep.d. I created a new file, /etc/pm/sleep.d/20_enable_display which looks like this:

#!/bin/sh

case "$1" in
    resume)
        # Check whether the external monitor is connected:
        # returns 0 on success
        xrandr | grep VGA | grep " connected "
        if [ $? -eq 0 ]; then
            xrandr --output VGA1 --mode 1600x900
            xrandr --output LVDS1 --off
        else
            xrandr --output VGA1 --off
            xrandr --output LVDS1 --mode 1280x800
        fi
        hsetroot -center `find -L $HOME/Backgrounds -name "*.*" | $HOME/bin/randomline`

        ;;
    suspend|hibernate)
        ;;
    *)
        ;;
esac
exit $?

Neat! Now, every time I wake from suspend, the laptop checks whether an external monitor is connected and sets the resolution accordingly. And it re-sets the background (using my random wallpaper method) so I don't get a tiled background on the big monitor.

Update: hsetroot -fill works better than -center given that I'm displaying background images on two different resolutions. Of course, if I wanted to get fancy I could make separate background sets, one for each monitor, and choose images from the appropriate set.

We're almost done. Two more possible adjustments.

Detecting undocking

First, while poking around in /etc/acpi I noticed a script named undock.sh. In theory, I can put the same code snippet in there, and then if I un-dock the laptop without suspending it first, it will immediately change resolution. I haven't actually tried that yet.

Projectors

Second, this business of turning off the built-in display if there's anything plugged into the VGA port is going to break if I use this laptop for presentations, since a projector will also show up as VGA1. So the code may need to be a little smarter. For example:

xrandr | grep VGA | grep " connected " | grep 16.0x

The theory here is that an external monitor will be able to do 1680 or 1600, so it will have a line like VGA1 connected 1680x1050+0+0 (normal left inverted right x axis y axis) 434mm x 270mm. The 1680x matches the 16.0x pattern in grep. A projector isn't likely to do more than 1280, so it won't match the pattern '16.0x'. However, that isn't very robust; it will probably fail for one of those fancy new 1920x1080 monitors. You could extend it with

xrandr | grep VGA | grep " connected " | egrep '16.0x|19.0x'

Which doubtless I'll do if I ever get a 1920x1080 monitor.

Hugin part 2: Rescuing Difficult Panoramas

Hugin is an amazing program, but if you get outside the bounds of the normal "Assistant" steps, the user interface can be a bit confusing -- and sometimes it does things that are Just Plain Weird. But with help from some folks on IRC, I found out that a newer version of Hugin can fix those problems, and worked out how to do it (as well as lots of ways that seemed like they should work, but didn't).

Read the gory details in: Hugin part 2: Rescuing Difficult Panoramas.

There will be a Hugin Part 3, and possibly even a Part 4, discussing things Hugin can do beyond panoramas.

Painless Panorama Stitching with Hugin

But for panoramas bigger than a couple of images, you're much better off using the Linux panorama app: Hugin.

Hugin is very impressive, and much too capable to be summarized in a single short article, so I'm planning three. This week's article is a basic introduction: Painless Panorama Stitching with Hugin.

Fotoxx: the Greatest Little Linux Photo Editor You've Never Heard Of

So I wrote about it in this week's Linux Planet article: Fotoxx, the Greatest Little Linux Photo Editor You've Never Heard Of.

At first, I was most impressed by the Warp tool -- much easier to use than GIMP's IWarp, though it's rather slow and not quite as flexible as IWarp. But once I got to writing the article, I was blown away by two additional features: it has an automatic panorama stitcher and an HDR tool. GIMP doesn't have either of these features, at all.

Now, panorama stitching used to be a big deal, but it isn't so much any more now that Hugin has gotten much easier to use. (My article in two weeks will be about Hugin.) Fotoxx isn't quite that flexible: it can only stitch two images at a time, and can't handle images with a lot of overlap. (But Hugin has some limitations too.)

But HDR -- wow! I've been meaning to learn more about making HDR images in GIMP -- although it has no HDR tool, there are plug-ins to make it a bit easier to assemble one, just like my Pandora plug-in makes it a little easier to assemble panoramas. But now I don't need to -- fotoxx handles it automatically.

I won't be switching from GIMP any time soon for regular photo editing, of course -- GIMP is still much more flexible. But fotoxx is definitely worth a look, and I'll be keeping it installed to make HDR images, if nothing else.

Helpful CUPS error messages, decoded

Dave's printer was failing to print on Ubuntu Lucid. The only failure mode: pages came out with the single printed line,

Unable to open the initial device, quitting.

What a great, helpful error message! Thanks, CUPS!

Web searching found lots of people using HP printers, using various versions of the HPLIP software to installer newer drivers and otherwise reconfigure their HP setups.

Only problem: this printer isn't an HP. It's a Brother HL2070N, which has given very good service and, until now, worked flawlessly with every OS we've tried including Linux.

The solution? It turns out the problem really is HPLIP -- even if the printer isn't an HP. What this message really meant was:

HPLIP isn't installed, and Ubuntu's CUPS refuses to work without it.

apt-get install hplip hplip-data got the printer working again.

Wouldn't it be nice if CUPS bothered to print error messages that gave some hint of the real problem? Ideally, visible on the computer to the user, rather than on the printed page, so you don't have to waste 25 pages of dead tree while you try to narrow down the problem?

Update: After further testing, we have established that a standard Gnome-based Ubuntu Lucid machine needs hplip and hplip-data installed, while a Lucid machine without Gnome needs those two plus hpijs. Or you can get around needing any of them by ignoring CUPS' recommendation for which driver to use, and choosing the Gutenprint driver in the CUPS configuration screens.

A reader asked me if we had checked the CUPS error log. On one machine, the log file was virtually empty; on another, there actually were some lines about hpijs (nothing about hplip), intermixed with a lot of debug chatter and a large number of errors that were fairly clearly unrelated to anything we were doing.

Use "date" to show time abroad

But I suggested he could do the same thing with the TZ environment variable. It's not documented well anywhere I could find, but if you set TZ to the name of a time zone, date will print out the time for that zone rather than your current one.

So, for bash:

$ TZ=Europe/Paris date  # time in Paris
$ TZ=GB date            # time in Great Britain
$ TZ=GMT-02 date        # time two timezones east of GMT

% ( setenv TZ Europe/Paris; date)
% ( setenv TZ GB; date)
% ( setenv TZ GMT-02; date)

That's all very well. But when I tried

% ( setenv TZ UK; date)
% ( setenv TZ FR; date)

The trick seems to be that setting TZ only works for abbreviations in /usr/share/zoneinfo/, or maybe in /usr/share/zoneinfo/posix/. If you give an abbreviation, like UK or FR or America/San_Francisco, it won't give you an error, it'll just print GMT as if that was what you had asked for.

So this trick is useful for printing times abroad -- but if you want to be safe, either stick to syntaxes like GMT-2, or make a script that checks whether your abbreviation exists in the directory before calling date, and warns you rather than just printing the wrong time.

Kdenlive part 2: adding transitions, sounds and titles to your movies

Making Movies in Linux with Kdenlive, part 2: Spice up Those Kdenlive Videos.

Kdenlive -- the easy way to make movies

But then ... what to do with a bunch of disconnected video clips? I've uploaded short clips to youtube before, but never extracted the good parts and edited them together. And most video editing programs look pretty complex.

The answer turned out to be kdenlive, which was surprisingly easy to use -- once I got past one initial bug. So I wrote up the details. Part I, covering the basics of how to get started and combine clips, is on Linux Planet: Making Movies in Linux with Kdenlive.

Watch for part II in a couple of weeks, where I'll cover transition effects, music and titles.

The new udev in Lucid

But like every release, they made some pointless random undocumented changes that broke stuff. The most frustrating has been getting my front-panel flash card reader to work under Lucid's new udev, so I could read SD cards from my camera and PDA.

The SD card slot shows up as /dev/sdb, but unless there's a card plugged in at boot time, there's no /dev/sdb1 that you can actually mount.

hal vs udisks

Prior to Lucid, the "approved" way of creating sdb1 was to let hald-addons-storage poll every USB device every so often, to see if anyone has plugged in a card and if so, check its partition table and create appropriate devices.

That's a lot of polling -- and in any case, hald isn't standard on Lucid, and even when it's installed, it sometimes runs and sometimes doesn't. (I haven't figured out what controls whether it decides to run). Hal isn't even supposed to be needed on Lucid -- it's supposed to use devicekit (renamed to) udisks for that.

Except I guess they couldn't quite figure out how to get udisks working in time, so they patched things together so that on Gnome systems, hald does the same old polling stuff -- and on non Gnome systems, well, maybe it does and maybe it doesn't. And maybe you can't read your camera cards. Oh well!

udev rules

But on systems prior to Lucid there was another way: make a udev rule to create sdb1 through sdb15 every time. I have an older article on setting up udev rules for multicard readers, but none of my old udev rules worked on Lucid.

After many rounds of udevadm info -a -p /block/sdb and udevadm test /block/sdb, service udev restart, and many reboots, I finally found a rule that worked.

Create a /etc/udev/rules.d/71-multicard-reader.rules file containing the following:

# Create all devices for multicard reader:
KERNEL=="sd[b-g]", SUBSYSTEMS=="usb", ATTRS{idVendor}=="1d6b", ATTRS{idProduct}=="0002", OPTIONS+="all_partitions,last_rule"

Replace the 1d6b and 0002 with the vendor and product of your own device, as determined with udevadm info -a -p /block/sdb ... and don't be tempted to use the vendor and device ID you get from lsusb, because those are different.

What didn't work that used to? String matches. Some of them. For example, this worked:

KERNEL=="sd[b-g]", SUBSYSTEMS=="scsi", ATTRS{model}=="*SD*", NAME{all_partitions}="sdcard"

KERNEL=="sd[b-g]", SUBSYSTEMS=="scsi", ATTRS{model}=="*SD*Reader*", NAME{all_partitions}="sdcard"
KERNEL=="sd[a-g]", SUBSYSTEMS=="scsi", ATTRS{model}=="USB SD Reader   ", NAME{all_partitions}="cardsd"

Update: The first of those two lines does indeed work now, whereas it didn't when I was testing. It's possible that this has something to do with saving hardware states and needing an extra udevadm trigger, as suggested in Alex's Changes in Ubuntu Lucid to udev.

According to udevadm info, the model is "USB SD Reader " (three spaces at the end). But somehow "*SD*" matches this while "*SD*Reader*" and the exact string do not. Go figure.

Numeric order

I'd like to have this rule run earlier, so it runs before /lib/udev/rules.d/60-persistent-storage.rules and could use OPTIONS+="last_rule" to keep the persistent storage rules from firing (they run a lot of unnecessary external programs for each device). But if I rename the rule from 71-multicard-reader.rules to 59-, it doesn't run at all. Why? Shrug. It's not like udevadm test will tell me.

Other things I love (not) about the new udev

• I love how if you give the udevadm info arguments in the wrong order, -p -a, it means something else and gives an error message.
• I love how udevadm test doesn't actually test the same rules udev will use, so it's completely unrelated to anything.
• I love the complete lack of documentation on things like string matching and how the numeric order is handled.
• I love how you can't match both the device name (a string) and the USB IDs in the same rule, because one is SUBSYSTEMS=="scsi" and the other is SUBSYSTEMS=="usb".
• Finally, I love how there's no longer any way to test udev rules on a running system -- if you want it to actually create new devices, you have to reboot for each new test. service udev restart and udevadm control --reload-rules don't touch existing devices. Gives me that warm feeling like maybe I'm not missing out on the full Windows experience by using Linux.

The luxury of understanding the problem

On a Linux list, someone was having trouble with wireless networking, and someone else said he'd had a similar problem and solved it by reinstalling Kubuntu from scratch. Another poster then criticised him for that: "if the answer is reinstall, you might as well downgrade to Windows.", and later added, "if "we should understand a problem, and *then* choose a remedy to match."

As someone who spends quite a lot of time trying to track down root causes of problems so that I can come up with a fix that doesn't involve reinstalling, I thought that was unfair. Here is how I replied on the list (or you can go straight to the mailing list version):

I'm a big fan of understanding the root cause of a problem and solving it on that basis. Because I am, I waste many days chasing down problems that ought to "just work", and probably would "just work" if I gave in and installed a bone stock Ubuntu Gnome desktop with no customizations. Modern Linux distros (except maybe Gentoo) are written with the assumption that you aren't going to change anything -- so reverting to the original (reinstalling) will often fix a problem.

Understanding this stuff *shouldn't* take days of wasted time -- but it does, because none of this crap has decent documentation. With a lot of the underlying processes in Linux -- networking, fonts, sound, external storage -- there are plenty of "Click on the System Settings menu, then click on ... here's a screenshot" howtos, but not much "Then the foo daemon runs the /etc/acpi/bar.sh script, which calls ifconfig with these arguments". Mostly you have to reverse-engineer it by running experiments, or read the source code.

Sometimes I wonder why I bother. It may be sort of obsessive-compulsive disorder, but I guess it's better than washing my hands 'til they bleed, or hoarding 100 cats. At least I end up with a nice customized system and more knowledge about how Linux works. And no cat food expenses.

But don't get on someone's case because he doesn't have days to waste chasing down deep understanding of a system problem. If you're going to get on someone's case, go after the people who write these systems and then don't document how they actually work, so people could debug them.

Article: /etc/fstab basics

Function keys on an Apple keyboard, on Linux

On Linux, for some reason Apple keyboards' function keys don't work by default. Most of them try to run special functions instead, like volume up/down or play/pause.

But you can get normal function keys by talking to the kernel module that drives the keyboard:

echo 2 >  /sys/module/hid_apple/parameters/fnmode

This will only last until shutdown, so put that line in /etc/rc.local or a similar place so it runs every time you boot.

Here's an Ubuntu help page on Apple Keyboards with more information and other tricks.

Article: Linux Boot Camp, part 2: Upstart

The Upstart Startup Manager (Linux Boot Camp part 2)

Article: Making Bash Error Messages Friendlier

Linux Planet gave me the chance to write it up in more detail, explaining a bit more about how it works: Making Bash Error Messages Friendlier.

Quick tip: User-mounted filesystems with "exec"

Trying to run debootstrap to install the new Ubuntu beta, I kept hitting a snag right at the beginning: debootstrap kept saying the filesystem was mounted with the wrong permissions, and it couldn't create an executable file or a device.

I have lines for each of my spare filesystems in /etc/fstab, like this:

/dev/sda4  /lucid  ext3  relatime,user,noauto  0   0

Not being able to create executable files means it's mounted with the noexec flag. I checked another machine and saw that it was using lines like

/dev/sda4  /lucid  ext3  exec,user,noauto  0   0

I added the "exec," to fstab, unmounted and remounted ... and it was still mounted with noexec.

Turns out on some Linux versions, making a filesystem user mountable turns off exec even if you've specified it explicitly. You have to add the exec after the user.

But that still didn't make debootstrap happy, because it couldn't create a device. That's a separate fstab option, dev, and user implies nodev.

So here's the fstab entry that finally worked:

/dev/sda4  /lucid  ext3  relatime,user,exec,dev,noauto  0   0

Creating a Linux Live USB stick: Big win for Fedora

There are tons of howtos on the web for creating live USB sticks for Ubuntu. Almost all of them start with "First, download the CD image and burn it to a CD. Now, boot off the CD and ..."

The few that don't discuss apps like usb-creator-gtk or unetbootin tha work great if you're burning the current Ubuntu Live CD image from a reasonably current Ubuntu machine, but which fail miserably in every other case (wildly pathological cases like burning the current Ubuntu alternate installer CD from the last long-term-support version of Ubuntu. I mean, really, should that be so unusual?)

Tonight, I wanted a bootable USB of Fedora 12. I tried the Ubuntu tools already mentioned, but usb-creator-gtk won't even try with an image that isn't Ubuntu, and unetbootin wrote something but the resulting stick didn't boot.

I asked on the Fedora IRC channel, where a helpful person pointed me to this paragraph on copying an ISO image with dd.

Holy mackerel! One command:

dd if=Fedora-12-i686-Live.iso of=/dev/sdf bs=8M

Really, Ubuntu, you should take a look at Fedora now and then. For machines that are new enough, USB boot is much faster and easier than CD burning -- so give people an easy way to get a bootable USB version of your operating system. Or they might give up and try a distro that does make it easy.

Article: Linux Boot Camp (part 1: SysV Init)

It describes the boot sequence, from grub to kernel loading to init scripts to starting X. Part I covers the classic "SysV Init" model still used to some extent by every distro; part II will cover Upstart, the version that's gradually working its way into some of the newer Linux releases.

Grub2 Tutorial, Part 3

Part 3 and final of my series on configuring Ubuntu's new grub2 boot menu. I translate a couple of commonly-seen error messages, but most of the article is devoted to multi-boot machines. If you have several different operating systems or Linux distros installed on separate disk partitions, grub2 has some unpleasant surprises, so see my article for some (unfortunately very hacky) workarounds for its limitations.

Why use Grub2? Good question!
(Let me note that I didn't write the title, though I don't disagree with it.)

Making those Fn- laptop keys do something useful

A friend was trying to get some of her laptop's function keys working under Ubuntu, and that reminded me that I'd been meaning to do the same on my Vaio TX 650P.

My brightness keys worked automagically -- I suspected via the scripts in /etc/acpi -- and that was helpful in tracking down the rest of the information I needed. But it still took a bit of fiddling since (surprise!) how this stuff works isn't documented.

Update: That "isn't documented" remark applies to the ACPI system. Matt Zimmerman points out that there is some good documentation on the rest of the key-handling system, and pointed me to two really excellent pages: Hotkeys architecture and Hotkeys Troubleshooting. Recommended reading!

Here's the procedure I found.

First, use acpi_listen to find out what events are generated by the key you care about. Not all keys generate ACPI events. I haven't get figured out what controls this -- possibly the kernel. When you type the key, you're looking for something like this:

sony/hotkey SPIC 00000001 00000012

Once you know the code for your key, it's time to make it do something. Create a new file in /etc/acpi/events -- I called mine sony-lcd-btn. It doesn't matter what you call it -- acpid will read all of them. (Yes, that means every time you start up it's reading all those toshiba and asus files even if you have a Lenovo or Sony. Looks like a nice place to shave off a little boot time.)

The file is very simple and should look something like this:

# /etc/acpi/events/sony-lcd-btn

event=sony/hotkey SPIC 00000001 00000012
action=/etc/acpi/sonylcd.sh

Now create a script for the action you specified in the event file. I created a script /etc/acpi/sonylcd.sh that looks like this:

#! /bin/bash
# temporary, for testing:
echo "LCD button!" >/dev/console

Now restart acpid: service acpid restart if you're on karmic, or /etc/init.d/acpid restart on earlier releases. Press the button. If you're running from the console (or using a tool like xconsole), and you got all the codes right, you should be able to see the echo from your script.

Now you can do anything you want. For instance, when I press the LCD button I generally want to run this:

xrandr --output VGA --mode 1024x768

Or to make it toggle, I could write a slightly smarter script using xrandr --query to find out the current mode and behave accordingly. I'll probably do that at some point when I have a projector handy.

Grub2 Tutorial, Part 2

Part 2 of my 3-parter on configuring Ubuntu's new grub2 boot menu covers cleaning up all the bogus menu entries (if you have a multiple-boot system) and some tricks on setting color and image backgrounds:

Cleaning up your boot menu (Grub2 part 2).

SCALE 8x

I'm finally getting caught up after SCALE 8x, this year's Southern CA Linux Expo.

A few highlights (not even close to a comprehensive list):

Friday:

The UbuCon and Women in Open Source (WIOS) were both great successes, with a great speaker list and good attendance. It was hard to choose between them.

Malakai Wade, Mirano Cafiero, and Saskia Wade, two 12-year-olds and an 8-year-old, presenting on "Ultimate Randomness - Girl voices in open source". Great stuff! They sang, they discussed their favorite apps, they showed an animated video made with open source tools of dolls in a dollhouse. Lots of energy, confidence and fun. Loved it! I hope to see more of these girls.

I liked Nathan Haines demo of "Quickly", an app for rapid development of python-gtk apps. It looks like a great app, especially for beginning programmers, though his demo did also illustrate the problems with complex UIs filled with a zillion similar toolbuttons. (I'm not criticising Nathan; I find UIs like that very difficult to use, especially under pressure like a live demo in front of an audience.)

Happily, the UbuCon and WIOS scheduled their lightning talks at different times (though UbuCon's conflicted with WIOS's "How to give a Lightning Talk" session). So lightning talk junkies enjoyed two hours of talks back to back, plus the chance to give two different talks to different audiences. Hectic but a lot of fun.

Saturday

I was a little disappointed with the Git Tips & Tricks panel; I wanted more git tips and less discussion of projects that happen to use Git. I liked Don Marti's section on IkiWiki; it looks like a great tool and I wish Don had had more time to present.

I liked Emma Jane Hogbin's useful and interesting talk on "Looking Beautiful in Print", full of practical tips for how to design good flyers and brochures using tools like OpenOffice.

Diana Chen, who got introduced to open source only a year ago at SCALE 7x, gets the award for courage: she gave a talk on "Learning python for non-programmers" using a borrowed laptop that I'm not sure she'd even seen before the presentation. Unfortunately, the laptop turned out to be poorly suited to the task (no Python installed? Dvorak keymap?) so Diana struggled to show what she'd planned, but she came through and her demos eventually worked great. I hope she wasn't too discouraged by the difficulties, and keeps presenting -- preferably with more time to practice ahead of time. The room was absolutely packed -- they had to bring in lots more chairs and there were still a lot of people standing. There's obviously a huge amount of interest in beginner programming talks at this conference!

Shawn Powers' talk, "Linux is for Smart People, and You're Not as Dumb as You Think", was as entertaining as the title suggested -- an excellent beginner-track talk that I think everyone enjoyed.

Sunday

I'm not going to review Sunday's program, because I was busy obsessing over my own "Featherweight Linux" talk. I'll just say that SCALE is a great place to give a talk -- the audience was great, with excellent questions and no heckling and, most important, they laughed when I hoped they would. :-)

Exhibitors

I didn't get to spend much time on the show floor, but it looked active and fun.

The Linux Astronomy folks had a fantastic display, with a big table with a simulated Martian landscape and a couple of robotic rovers exploring it and a robotic telescope driven by a milling machine program, as well as computers exhibiting a selection of Linux astronomy, science and math-teaching software.

ZaReason had a booth, and my mom was able to get info on how to get a spare battery for her laptop. (Can I take a moment to say how cool it is to be wandering around a Linux conference with my mom, who's carrying her own Linux netbook?)

An Ubuntu/Canonical table was testing people's laptops for compatibility with the next Ubuntu release. (There may have been other distros tested as well; I wasn't clear on that.)

Engineers Without Borders, Orange County looked really interesting and assured me that not all of them were in Orange County, and there's activity up here in the Bay Area as well. Definitely on my list to learn more.

Linux Pro magazine was giving out copies of Linux Pro and Ubuntu User, both fantastic magazines packed with good articles.

Beginners and Hobbyists

One notable feature of SCALE is the low price. This conference is very affordable, which means there are a lot of hobbyists, beginners and even people just considering trying Linux. They've offered a "Beginner track" for several years, though not all the talks in that track are really accessible to beginners (speakers: here's your chance to propose that great beginner talk the other conferences aren't interested in! Help some new folks!) There's a lot of energy and diversity and a wide range of interests and knowledge -- yet there's still plenty of depth for hardcore Linux geeks.

Overall, a fantastic conference. The SCALE organizers do a great job of organizing everything, and if there were any glitches they weren't evident from the outside.

Grub2 lightning talk at SCALE 8x Ubucon

I gave a lightning talk at the Ubucon -- the Ubuntu miniconf -- at the SCALE 8x, Southern California Linux Expo yesterday. I've been writing about grub2 for Linux Planet but it left me with some, well, opinions that I wanted to share.

A lightning talk is an informal very short talk, anywhere from 2 to 5 minutes. Typically a conference will have a session of lightning talks, where anyone can get up to plug a project, tell a story or flame about an annoyance. Anything goes. I'm a lightning talk junkie -- I love giving them, and I love hearing what everyone else has to say.

I had some simple slides for this particular talk. Generally I've used bold or other set-offs to indicate terms I showed on a slide.

SCALE 8x, by the way, is awesome so far, and I'm looking forward to the next two days.

Grub2 3-minute lightning talk

What's a grub? A soft wriggly worm.

But it's also the Ubuntu Bootloader. And in Karmic, we have a brand new grub: grub2!

Well, sort of. Karmic uses Grub 2 version 1.97 beta4. Aside from the fact that it's a beta -- nuff said about that -- what's this business of grub TWO being version ONE point something? Are you hearing alarm bells go off yet?

But it must be better, right? Like, they say it cleans up partition numbering.

Yay! So that confusing syntax in grub1, where you have to say [SLIDE] (hd0,0) that doesn't look like anything else on Linux, and you're always wanting to put the parenthesis in the wrong place -- they finally fixed that?

Well, no. Now it looks like this: (hd0,1) THEY KEPT THE CONFUSING SYNTAX BUT CHANGED THE NUMBER! Gee, guys, thanks for making things simpler!

But at least grub2 is better at graphics, right? Like what if you want to add a background image under that boring boot screen? A dark image, because the text is white.

Except now Ubuntu changes the text color to black. So you look in the config file to find out why ...

if background_image `make_system_path_relative...
  set color_normal=black/black

... there it is! But why are there two blacks? Of course, there's no documentation. They can't be fg/bg -- black on black wouldn't make any sense, right?

Well, it turns out it DOES mean foreground and background -- but the second "black" doesn't mean black. It's a special grub2 code for "transparent". That's right, they wrote this brand new program from scratch, but they couldn't make a parser that understands "none" or "transparent".

What if you actually want text with a black background? I have no idea. I guess you're out of luck.

Okay, what about dual booting? grub's great at that, right? I have three distros installed on this laptop. There's a shared /boot partition. When I change something, all I have to do is edit a file in /boot/grub. It's great -- so much better than lilo! Anybody remember what a pain lilo was?

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by /usr/sbin/grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

Oops, wait -- not with grub2. Now I'm not supposed to edit that file. Instead, I edit files in TWO places, /etc/grub.d and /etc/default/grub.conf, and then run a program in a third place, /usr/bin/update-grub. All this has to be done from the same machine where you installed grub2 -- if you're booted into one of your other distros, you're out of luck.

grub2 takes us back to the bad old days of lilo. FAIL

Grub2 really is a soft slimy worm after all.

But I have some ideas for workarounds. If you care, watch my next few articles on LinuxPlanet.com.

Update: links to Linux Planet articles:
Part 1: Grub2 worms into Ubuntu
Part 2: Cleaning up your boot menu
Part 3: Why use Grub2? Good question!

Grub2 Tutorial, Part 1

Upgraded to Ubuntu 9.10 Karmic and wondering how to configure your boot menu or set it up for multiple boots?

Grub2 Worms Into Ubuntu (part 1) is an introductory tutorial -- just enough to get you started. More details will follow in parts 2 and 3.

How to print text files in Ubuntu Karmic

Ever since I upgraded to Ubuntu 9.10 "Karmic koala", printing text files has been a problem. They print out with normal line height, but in a super-wide font so I only get about 48 ugly characters per line.

Various people have reported the problem -- for instance, bug 447961 and this post -- but no one seemed to have an answer.

I don't have an answer either, but I do have a workaround. The problem is that Ubuntu is scaling incorrectly. When it thinks it's putting 10 characters per inch (cpi) on a line, it's actually using a font that only fits 6 characters. But if you tell it to fit 17 characters per inch, that comes out pretty close to the 10cpi that's supposed to be the default:

lpr -o cpi=17 filename

As long as you have to specify the cpi, try different settings for it. cpi=20 gives a nice crisp looking font with about 11.8 characters per inch. If needed, you can adjust line spacing with lpi=NN as well.

Update: The ever-vigilant Till Kamppeter has tracked the problem down to the font used by texttopdf for lp/lpr printing. Interesting details in bug 447961.

Print Beautiful Custom Calendars in Linux With Photo Calendar

Didn't get the calendar you wanted for Christmas this year? Print your own, with your choice of photos and holidas. My Linux Planet Photo Calendar article shows how.

Fetchmail without Postfix

I run procmail to filter my mail into folders -- Linuxchix mail into one folder, GIMP mailing lists into another, and so forth -- and it seemed like it ought to be possible for fetchmail to call procmail directly, without going through postfix.

I found several suggestions on the web -- for instance, fetchmail-procmail-sendmail -- but they didn't work for me. fetchmail downloaded each message, passed it to procmail, and procmail appended it to the relevant mailbox without the appropriate "From " header that mail programs need to tell when each new message starts.

Finally, on a tip from bma on #linuxchix and after a little experimentation, I added this line to ~/.fetchmailrc:

mda /usr/bin/procmail -f %F -m /home/username/.procmailrc

HP Deskjet F4280

HP looked like the best solution, since they're the only printer manufacturer that supports Linux directly. I new wasn't going to buy Canon, because their closed protocols mean that every Linux driver has to be reverse engineered, and I certainly didn't want a Lexmark (see our last experience with Lexmark in Cracking the Lexmark Code).

But which HP? Their array of models is baffling, and no one seems to know the difference between Deskjets, Officejets and Photosmarts, or whether the inks fade, or whether the nozzles are built into the ink cartridges (so a clogged nozzle doesn't mean a dead printer like it does with Epson). And there's no way to get print samples.

So I dithered and stalled -- until Fry's put the HP Deskjet F4280 on sale for $20. The online reviews were fairly positive. And for that price, and with Linux support, how bad could it be?

Answer: not bad at all. It set up pretty easily in CUPS, though the CUPS test page didn't work even after several tries. Fortunately, I don't need to print CUPS test pages. Printing worked fine from GIMP, Firefox and OpenOffice.

The print quality is surprisingly good. (Note: the F4280 is not the same printer as the Photosmart C4280, which caused some confusion at Fry's when I tried to actually buy one). Text and web page on regular paper come out crisp and sharp. "High quality" on good photo paper looks like a photo as long as you don't examine it too closely. It'll be fine for my holiday greeting cards, business cards and most other tasks involving photos. "Photo quality" takes a lot longer, and is indeed better than "High" if you examine it with a loupe. Nobody's going to confuse it with a real photo print under magnification, but it'll look fine on the wall.

Here's the part that impressed me most: it can print all the way to the edge of the paper with no hassle. I could never do that with the C86: though the hardware was supposedly capable of it, the Gutenprint drivers -- the reason I'd been sticking with Epson all those years -- never could handle it (and tended to print yellow smears on the borders if you tried it). Good job, HP!

It's an "All in one" so it has a built-in scanner too (no fax). SANE (on Ubuntu 9.10) doesn't see the scanner, and I haven't tried to track that down since I already have a good scanner. I wouldn't have bought an "all in one" except that dedicated printers are quite a bit more expensive.

Update: it's the usual Ubuntu permissions problem, combined with new udev rules. Root sees the scanner, users don't, unless you add lines to two different udev rules files. In /lib/udev/rules.d/40-libsane.rules, add:
ATTRS{idVendor}=="03f0", ATTRS{idProduct}=="2504", ENV{libsane_matched}="yes"
then create a new file /etc/udev/rules.d/45-libsane.rules and put in it:
SYSFS{idVendor}=="03f0", SYSFS{idProduct}=="2504", MODE="664", GROUP="scanner"
More details are in bug 121082.

And wow, the scanner output is really bad. I mean really, really bad. I'm happy with the printer but I'll definitely keep my old Epson scanner.

Reviews complain that the F4280 is rather ink-hungry and the ink cartridges are overpriced; but every inkjet printer review says that (probably with good reason). I don't print that much, so I'm not too worried. And of course I know nothing about long term reliability or how fade-resistant the prints will be. Ask me in six months. But so far I'm quite pleased. A nice printer with excellent Linux drivers.

Character Sets and Encodings in Linux, part 2

Continuing the discussion of those funny characters you sometimes see in email or on web pages, today's Linux Planet article discusses how to convert and handle encoding errors, using Python or the command-line tool recode:

Mastering Characters Sets in Linux (Weird Characters, part 2).

More on kernel options needed for new X servers

But all was still not right with X. I could boot into a console, start X, and everything was fine -- but once X was running, I couldn't go back to console mode. Switching to a console, e.g. Ctrl-Alt-F2, did nothing except make the mouse cursor disappear, and any attempt to quit X to go back to my login shell put the monitor in sleep mode permanently -- the machine was still up, and I could ssh in or ctrl-alt-Delete to reboot, but nothing else I did would bring my screen back.

It wasn't strictly an Ubuntu problem, though this showed up with Karmic; I have a Gentoo install on another partition and it had the same problem. And I knew it was a kernel problem, because the Ubuntu kernel did let me quit X.

I sought in vain among the kernel's various Graphics settings. You might think that "enable modesetting" would be related to, you know, being unable to change video modes ... but it wasn't. I tried different DRM options and switching framebuffer on and off. Though, oddly, enabling framebuffer didn't actually seem to enable the framebuffer.

Finally I stepped through the Graphics section of make menuconfig comparing my settings with a working kernel, and saw a couple of differences that didn't look at all important: "Select compiled-in fonts" and "VGA 8x16 font". Silly, but what could they hurt? I switched them on and rebuilt.

And on the next boot, I had a framebuffer, and mode switching.

So be warned: those compiled-in fonts are not optional if you want a framebuffer; and you'd better want a framebuffer, because that isn't optional either if you want to be able to get out of X once you start it.

Article: Character Sets and Encodings in Linux

or: Why do I See All Those Those Weird Characters?

Today's Linux Planet article concerns those funny characters you sometimes see in email or on web pages, like when somebody puts “random squiggles’ around a phrase when they probably meant “double quotes”:

Character Sets in Linux or: Why do I See Those Weird Characters?.

Today's article covers only what users need to know. A followup article will discuss character encoding from a programmer's point of view.

Mouse failures with 2.6.31, Karmic and Intel

It wasn't purely an X problem; if I enabled gpm, the mouse failed in the console as well as in X. And it wasn't hardware, because if I used Ubuntu 9.10's standard kernel, my mouse worked every time.

After much poking around with kernel options, I discovered that if I tunred off the Direct Rendering manager ("Intel 830M, 845G, 852GM, 855GM, 865G (i915 driver)"), my mouse would work. But that wasn't a satisfactory solution; aside from not being able to run Google Earth, it seems that Intel graphics needs DRM even to get reasonable performance redrawing windows. Without it, every desktop switch means watching windows slowly redraw over two or three seconds.

(Aside: why is it that Intel cards with shared CPU memory need DRM to draw basic 2-D windows, when my ancient ATI Radeon cards without shared memory had no such problems?)

But I think I finally have it nailed. In the kernel's Direct Rendering Manager options (under Graphics), the "Intel 830M, 845G, 852GM, 855GM, 865G (i915 driver)" using its "i915 driver" option has a new sub-option: "Enable modesetting on intel by default".

The help says:

CONFIG_DRM_I915_KMS:
Choose this option if you want kernel modesetting enabled by default, and you have a new enough userspace to support this. Running old userspaces with this enabled will cause pain. Note that this causes the driver to bind to PCI devices, which precludes loading things like intelfb.

Sounds optional, right? Sounds like, if I want to build a kernel that will work on both karmic and jaunty, I should leave that off so as not to "cause pain".

But no. It turns out it's actually mandatory on karmic. Without it, there's a race condition where about 80-90% of the time, hal won't see a mouse device at all, so the mouse won't work either in X or even on the console with gpm.

It's sort of the opposite of the "Remove sysfs features which may confuse old userspace tools" in General Setup, where the name implies that it's optional on new distros like Karmic, but in fact, if you leave it on, the kernel won't work reliably.

So be warned when configuring a kernel for brand-new distros. There are some new pitfalls, and options that worked in the past may not work any longer!

Update: see also the followup post for two more non-optional options.

Friendlier error messages for shell newbies

1. Typing a file path without a slash, like etc/fstab
2. typing just a filename, without a command in front of it

The first boils down to a misunderstanding of how the Linux file system hierarchy works. (For a refresher, you might want to check out my Linux Planet article Navigating the Linux Filesystem.)

The second problem is due to forgetting the rules of shell grammar. Every shell sentence needs a verb, just like every sentence in English. In the shell, the command is the verb: what do you want to do? The arguments, if any, are the verb's direct object: What do you want to do it to?

(For grammar geeks, there's no noun phrase for a subject because shell commands are imperative. And yes, I ended a sentence with a preposition, so go ahead and feel superior if you believe that's incorrect.)

The thing is, both mistakes are easy to make, especially when you're new to the shell, perhaps coming from a "double-click on the file and let the computer decide what you should do with it" model. The shell model is a lot more flexible and (in my opinion) better -- you, not the computer, gets to decide what you should do with each file -- but it does take some getting used to.

But as a newbie, all you know is that you type a command and get some message like "Permission denied." Why was permission denied? How are you to figure out what the real problem was? And why can't the shell help you with that?

And a few days ago I realized ... it can! Bash, zsh and similar shells have a fairly flexible error handling mechanism. Ubuntu users have seen one part of this, where if you type a command you don't have installed, Ubuntu gives you a fancy error message suggesting what you might have meant and/or what package you might be missing:

$ catt /etc/fstab
No command 'catt' found, did you mean:
 Command 'cat' from package 'coreutils' (main)
 Command 'cant' from package 'swap-cwm' (universe)
catt: command not found

What if I tapped into that same mechanism and wrote a more general handler that could offer helpful suggestions when it looked like the user forgot the command or the leading slash?

It turns out that Ubuntu's error handler uses a ridiculously specific function called command_not_found_handle that can't be used for other errors. Some helpful folks I chatted with on #bash felt, as I did, that such a specific mechanism was silly. But they pointed me to a more general error trapping mechanism that turned out to work fine for my purposes.

It took some fussing and fighting with bash syntax, but I have a basic proof-of-concept. Of course it could be expanded to cover a lot more types of error cases -- and more types of files the user might want to open.

Here are some sample errors it catches:

$ schedule.html
bash: ./schedule.html: Permission denied

schedule.html is an HTML file. Did you want to run: firefox schedule.html

$ screenshot.jpg
bash: ./screenshot.jpg: Permission denied

screenshot.jpg is an image file. Did you want to run:
    pho screenshot.jpg
    gimp screenshot.jpg

$ .bashrc
bash: ./.bashrc: Permission denied

.bashrc is a text file. Did you want to run:
    less .bashrc
    vim .bashrc

$ ls etc/fstab
/bin/ls: cannot access etc/fstab: No such file or directory

Did you forget the leading slash?
etc/fstab doesn't exist, but /etc/fstab does.

You can find the code here: Friendly shell errors and of course I'm happy to take suggestions or contributions for how to make it friendlier to new shell users.

Autologin changes again for Ubuntu Karmic

Create /usr/bin/loginscript if you haven't already, containing something like this:

#! /bin/sh
/bin/login -f yourusername

Then edit /etc/init/tty1.conf and look for the respawn line, and replace the line after it, exec /sbin/getty -8 38400 tty1, with this:

exec /sbin/getty -n -l /usr/bin/loginscript 38400 tty1

As far as I know, it's safe to delete /etc/event.d since it's now unused. I haven't verified that yet. Better rename it first, and see if anything breaks.

Controlling brightness on a Sony laptop in Linux 2.6.31

I found that there were lots of bugs filed on this problem -- kernel.org bug 12816, Ubuntu bug 414810, Fedora bug 519105 and so on. A few of them had terse comments like "Can you try this patched binary release?" but none of them had a source patch, or any hints as to the actual nature of the problem.

But one of them did point me to /sys/class/backlight. In my working 2.6.28.3 kernel, that directory contained a sony subdirectory containing useful files that let me query or change the brightness: echo 1 >/sys/class/backlight/sony/brightness
On my nonworking 2.6.31.5 kernel, I had /sys/class/backlight but there was no sony subdirectory there.

grep SONY .config in the two kernels revealed that my working kernel had SONY_LAPTOP set, while the nonworking one did not. No problem! Just figure out where SONY_LAPTOP is in the configuration (it turns out to be at the very end of "device drivers" under "X86 Platform Specific Device Drivers"), make menuconfig, set SONY_LAPTOP and rebuild ... right?

Well, no. make menuconfig showed me lots of laptop manufacturers in the "Platform Specific" category, but Sony wasn't one of them. Of course, since it didn't show the option it also didn't offer a chance to read the help for the option either, which might have told me what its dependencies were.

Time for a recursive grep of kernel source: grep -r SONY_LAPTOP .
arch/x86/configs/i386_defconfig told me the option did indeed still exist, and where to find it. drivers/platform/x86/Kconfig lists the option itself, and says it depends on INPUT and RFKILL.

RFKILL? A bit more poking around located that one near the end of "Networking support", with the name "RF switch subsystem support". (I love how user-visible names for kernel options only marginally resemble the CONFIG names.) It's apparently intended for "control over RF switches found on many WiFi and Bluetooth cards," something I don't seem to need on this laptop (my WiFi works fine without it) -- except that the kernel for some reason won't let me build the ACPI brightness control without it.

So that's the secret. Go to "Networking support", set "RF switch subsystem support", then back up to "Device drivers", scroll down to the end to find "X86 Platform Specific Device Drivers" and inside it you'll now see "Sony Laptop Extras". Enable that, and the Fn-F5/F6 keys (as well as /sys/class/backlight/sony/brightness) will work again.

Turning off the Gnome Keyring prompt

It comes preinstalled with Ubuntu (currently 9.04 Jaunty) with a full Gnome desktop. I've never been much of a Gnome fan, but this time we thought we'd try keeping it for a while and see how Mom likes it. We can always switch to something faster, like Openbox, later.

Of course, a lot of things needed configuration, like getting rid of one of the two toolbars. (In this age of cinema-width screens, why is it that the major desktops, like Gnome and even Apple, insist on sucking away vertical space with multiple menubars/toolbars?)

(And don't get me started on Evolution's preferences panes that are too big to fit on a netbook screen, yet have no scrollbars; and although the preference window is resizable, Gnome won't let you drag a window past the top of the screen so you can resize it taller.)

What stymied us, though, was the Gnome keyring and the way it prompts you for a password -- even if you've already typed in a login password -- whenever it tries to connect to the wireless network.

Web searches revealed that we were far from the only people who found this annoying and wanted to turn it off. There are lots of howtos. Unfortunately, every howto is different -- apparently gnome-keyring changes its user interface with every release, but somehow none of these UI changes ever make it easier to find your way to the place where you can turn off the password prompting. So here's one for Jaunty.

Howto turn off the Gnome-keyring master password in Ubuntu Jaunty

The key is a program called "seahorse", which you can get to via Applications->Accessories->Password and Encryption Keys. Click on the Passwords tab: you'll probably see two lines, login password and master password. According to some of the earlier howtos, these two passwords need to be the same in order for the following steps to work.

Right-click on login password and choose Unlock (I'm not sure if that step is necessary, but we did it).

Then from the same right-click menu, choose Change Password and make the new password empty. Of course, it will warn you about this horribly insecure behavior and how you're an idiot to want to do this. Your choice!

Article: Building Your Own Linux Kernel, Part III

Meanwhile, I'm still jazzed about the great howto that Nathan Willis of Worldlabel.com wrote a few days ago for my Gimp labels scripts: Fast labels and Card layout with Gimplabels.

Article: Building Your Own Linux Kernel, Part II

Part III, in two weeks, will tour some specific kernel options and what they do.

Article: Building Your Own Linux Kernel, Part I

"But aren't there a gazillion howtos already on the web on kernel building?" I thought so too. But when someone showed up on LinuxChix recently asking for help building her kernel, I went looking -- and all the howtos I could find were out of date (even the README in the very latest kernel gives instructions based on LILO, not GRUB).

More important, none of them offered help in that all-important question: How do I start with a configuration file I know will work?

My quick-and-dirty howto shows you how to take your distro's configuration file and build the latest mainstream kernel based on it. The next article will cover how to change that configuration and tune it for your own machine.

Installing Flash on Linux, for Newbies

Dear Adobe: Please update your instructions when you update your install packages

It gave me more insight into why people say "Linux is hard to use" ... which has little to do with Linux, and everything to do with outside forces that seem to go out of their way to make things hard for Linux users.

See, Mom's Firefox auto-updated to a new version, which started whining about her flash version being insecure and telling her to update it. It pointed her to Adobe's site, get.adobe.com/flashplayer.

She went there and was presented with a long list of options for different types of download. She's on Ubuntu, so the Ubuntu deb might have worked -- but it might not, since she's running a Firefox from Mozilla.org rather than the one from Ubuntu. (Ubuntu's Firefox on Hardy was notoriously crashy, and she has enough problems with the Mozilla version crashing.)

I told her I usually use the tarball, and install it as myself, not as root. In the past, the flash installer has always been very good about noticing I'm not root and installing to ~/.mozilla/plugins. I didn't expect problems.

So she downloaded the tarball and tried to follow their instructions, which look like this:

1. Click the download link to begin installation. A dialog box will appear asking you where to save the file.
2. Save the .tar.gz file to your desktop and wait for the file to download completely.
3. Unpackage the file. A directory called install_flash_player_10_linux will be created.
4. In terminal, navigate to this directory and type ./flashplayer-installer to run the installer. Click Enter. The installer will instruct you to shut down your browser(s).
5. Once the installation is complete, the plug-in will be installed in your Mozilla browser. To verify, launch Mozilla and choose Help > About Plug-ins from the browser menu.

The first problem is "Unpackage the file." Honestly, how hard is it to give people a hint that "unpackage" means "type tar xf install_flash_player_10_linux.tar.gz"? As long as you're writing instructions anyway, why not tell people the actual command instead of expecting them to figure it out somehow?

"In terminal, navigate to this directory" -- if you know your user will be typing shell commands in a terminal, why not tell them to cd rather than expecting them to figure that out from "navigate"? (Mom figured that one out -- go Mom! -- but a lot of users wouldn't.)

Except -- OOPS! try following the instructions and you can't cd ... because it turns out the flash 10 "installer" doesn't contain a directory, or indeed an installer, at all. It's a tarball containing one file, libflashplayer.so.

Now, setting aside the question of why anyone would use tar to package a single file -- why not just make the file available for download and tell users where to put it? -- they give you no hint as to where this libflashplayer.so file is supposed to go. If you don't happen to know how Firefox sets up its plugins, you're out of luck.

Fortunately, I happen to know where the file goes. I told Mom to mv libflashplayer.so ~/.mozilla/plugins/ and all was well. But ... sheesh! With instructions like this on something as (unfortunately) widely needed as the Flash plugin, how can a newbie ever expect to get anywhere?

For newbies reading this, the real instructions for installing Adobe's flash 10 tarball are:

1. Download their file, which is named install_flash_player_10_linux.tar.gz
2. Open a terminal and cd to wherever you downloaded it, e.g. cd ~/Desktop
3. tar xf install_flash_player_10_linux.tar.gz
4. mv libflashplayer.so ~/.mozilla/plugins/
5. Restart firefox, make sure flash works, and (once you're sure, at your option)
   rm install_flash_player_10_linux.tar.gz

Article: Linux on Multicore Processors

Get the Most Out of Your Multicore Processor: Two heads are better than one!

Using apt-file to track down build errors

The solution -- apt-file -- is so simple and easy to use, and yet a lot of people don't know about it. So here's how it works.

The poster reported getting these compiler errors:

ar rc libz.a adler32.o compress.o crc32.o uncompr.o deflate.o trees.o zutil.o inflate.o inftrees.o inffast.o
ranlib libz.a
make[1]: Leaving directory `/home/gregs/xephem-3.7.4/libz'
gcc -I../../libastro -I../../libip -I../../liblilxml -I../../libjpegd -I../../libpng -I../../libz -g -O2 -Wall -I../../libXm/linux86 -I/usr/X11R6/include   -c -o aavso.o aavso.c
In file included from aavso.c:12:
../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:57:23: error: X11/Shell.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:58:23: error: X11/Xatom.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:59:34: error: X11/extensions/Print.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:1373: error: expected `=', `,', `;', `asm' or `__attribute__' before `char'
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:5439:28: error: X11/StringDefs.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:61,
                 from aavso.c:12:
../../libXm/linux86/Xm/VirtKeys.h:108: error: expected `)' before `*' token
In file included from ../../libXm/linux86/Xm/Display.h:49,
                 from ../../libXm/linux86/Xm/DragC.h:48,
                 from ../../libXm/linux86/Xm/Transfer.h:44,
                 from ../../libXm/linux86/Xm/Xm.h:62,
                 from aavso.c:12:
../../libXm/linux86/Xm/DropSMgr.h:88: error: expected specifier-qualifier-list before `XEvent'
../../libXm/linux86/Xm/DropSMgr.h:100: error: expected specifier-qualifier-list before `XEvent'

When interpreting compiler errors, usually what matters is the *first* error. So try to find that. In the transcript above, the first line saying "error:" is this one:

../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory

So the first problem is that the compiler is trying to find a file called Intrinsic.h that isn't installed.

On Debian-based systems, there's a great program you can use to find files available for install: apt-file. It's not installed by default, so install it, then update it, like this (the update will take a long time):

$ sudo apt-get install apt-file
$ sudo apt-file update

$  apt-file search Intrinsic.h
libxt-dev: /usr/include/X11/Intrinsic.h
tendra: /usr/lib/TenDRA/lib/include/x5/t.api/X11/Intrinsic.h

In this case two two packages could install a file by that name. You can usually figure out from looking which one is the "real" one (usually the one with the shorter name, or the one where the package name sounds related to what you're trying to do). If you're stil not sure, try something like apt-cache show libxt-dev tendra to find out more about the packages involved.

In this case, it's pretty clear that tendra is a red herring, and the problem is likely that the libxt-dev package is missing. So apt-get install libxt-dev and try the build again.

Repeat the process until you have everything you need for the build.

Remember apt-file if you're not already using it. It's tremendously useful in tracking down build dependencies.

Reading Palm Datebook files on Linux

Not so fast. It seems that it's been a year or more since I last crunched any of this data -- and in the time since then, pilot-link has bumped its version numbers and is now shipping libpisock.so.9 instead of .8.

So what? Well, the problem is that Linux hasn't offered any way to read Palm Datebook files for years. The pilot-link package offered on most distros used to include a program called pilot-datebook, but it was deleted from the source several years ago. Apparently it was hard to maintain.

Back when it first disappeared, I built the previous version of the source, stuck the pilot-datebook binary in ~/bin/linux and have been using it ever since. Which worked fine -- until libpisock.so.8 was no longer there. (Linking .9 to .8 didn't work either.) This is all the more ironic because I don't need pilot-datebook to talk to the PDA with libpisock -- all I want to do is parse the format of a file I've already uploaded.

Off to hunt for an old version of the source. I started at pilot-link.org, but gave up after a while -- they don't seem to have source there except for the latest couple of versions, nor do they have any documentation. Ironically, in their FAQ the very first question is "How can I read the databook entries from a Palm backup?" but the FAQ page is broken and the "answer" is actually another unrelated FAQ question.

Anyway, no help there. I tried googling for old tarballs but there doesn't seem to be anything like archive.org for source code. All I found was the original pilot-datebook page, with a tarball that you insert into a copy of pilot-link 0.9.5 then modify the Makefile. Might work but that's really old.

So I fell back on old distributions. I guessed that Ubuntu Dapper was old enough that it might still have pilot-datebook. So I went to the Dapper pilot-link source and downloaded the source tarball (curiously, they don't offer src debs -- you have to download the tarball and patches separately).

Of course, it doesn't build on Ubuntu Jaunty. It had various entertaining errors ranging from wanting a mysterious tcl.m4 file not present in the code ... to not being able to find <iostream.h< because all the C++ stdlib files have recently been renamed to remove the .h ... to a change in the open() system call where I needed to add permissions argument for O_CREAT.

But I did get it working! So now I have a pilot-datebook program that builds and runs on Ubuntu Jaunty, and parses my DatebookDB.pdb file.

Since I bet I'm not the only one in the world who occasionally wants to read a Palm Datebook file, I've put my working version of the source here: pilot-link_0.11.8.jaunty.tar.gz.

After the usual configure and make, if all you want is pilot-datebook, cd src/pilot-datebook then copy both pilot-datebook and the directory .libs to wherever you want to install them.

And yeah, it would be better to write a standalone program that just parsed the format. But it's hard to justify that for what's essentially a dead platform. The real solution is to quit using a Palm for this, import the data into some common format and keep it on my Linux workstation from now on.

Linux Bloat 102

Fedora 11 not encouraging so far

I downloaded and burned the latest netinst CD ISO and booted it on the Atom machine. It greeted me with this prompt:

    1.
    2.
Select CD-ROM boot type: 

I am not getting warm fuzzies about Fedora being the solution to the Linux distro problem.

What do you think? Should I choose 1. or 2. ?

Crikey 0.8.3: bug fixes and several new syntaxes

We corresponded a bit, figured out exactly how things should work and some test cases, went through a couple iterations of changes where I got lots of detailed and thoughtful feedback and more test cases, and now Crikey can do an assortment of new useful stuff.

New features: crikey now handles number codes like \27, modifier keys like \A for alt, does a better job with symbols like \(Return\), and handles a couple of new special characters like \e for escape. It also works better at sending window manager commands, like "\A\t" to change the active window.

I've added some better documentation on all the syntaxes it understands, both on the web page and in the -h and -l (longhelp) command-line arguments, and made a release: crikey 0.8.3.

Plus: a list of great regression tests that I can use when testing future updates (in the file TESTING in the tarball).

Linux Bloat 101

This one just covers the basics. The followup article, in two weeks, will dive into more detail on how to analyze what resources programs are really using.

Tracking down performance hogs

I'm posting this while sitting in an excellent OSCON tutorial on Linux System and Network Performance Monitoring, by Darren Hoch. It's full of great information and I'm sure his web site is equally useful.

And it's a great extension to topic that's been occupying me over the past few months: performance tracking to slim down software that might be slowing a Linux system down. That's the topic of one of my two OSCON talks this Wednesday: "Featherweight Linux: How to turn a netbook or older laptop into a Ferrari." Although I don't go into anywhere near the detail Darren does, a lot of the principles are the same, and I know I'll find a use for a lot of his techniques. The talk also includes a free bonus tourist tip for San Jose visitors.

Today's Linux Planet article is related to my Featherweight talk: What's Bogging Down Your Linux PC? Tracking Down Resource Hogs. Usually they publish my articles on Thursdays, but I asked for an early release since it's related to tomorrow's talk.

For anyone at OSCON in San Jose, I hope you can come to Featherweight late Wednesday afternoon, or to my other talk, Wednesday just after lunch, "Bug Fixing for Everyone (even non-programmers!)" where I'll go over the steps programmers use while fixing bugs, and show that anyone can fix simple bugs even without any prior knowledge of programming.

Disabling mouse/keyboard wakeup

I've been through all the BIOS screens looking for a setting to flip, but there's nothing there. Some web searching told me that under Windows, there's a setting you can change that will affect this, but I couldn't find anything similar for Linux, until finally drc clued me in to /proc/acpi/wakeup.

cat /proc/acpi/wakeup

Unfortunately, they're also obscurely coded. Here are mine:

Device  S-state   Status   Sysfs node
SLPB      S4    *enabled  
P32       S4     disabled  pci:0000:00:1e.0
UAR1      S4     enabled   pnp:00:0a
PEX0      S4     disabled  pci:0000:00:1c.0
PEX1      S4     disabled  
PEX2      S4     disabled  pci:0000:00:1c.2
PEX3      S4     disabled  pci:0000:00:1c.3
PEX4      S4     disabled  
PEX5      S4     disabled  
UHC1      S3     disabled  pci:0000:00:1d.0
UHC2      S3     disabled  pci:0000:00:1d.1
UHC3      S3     disabled  pci:0000:00:1d.2
UHC4      S3     disabled  pci:0000:00:1d.3
EHCI      S3     disabled  pci:0000:00:1d.7
AC9M      S4     disabled  
AZAL      S4     disabled  pci:0000:00:1b.0

What do all those symbols mean? I have no clue. Apparently the codes come from the BIOS's DSDT code, and since it varies from board to board, nobody has published tables of likely translations.

The only two wakeups that were enabled for me were SLPB and UAR1. SLPB apparently stands for SLeeP Button, and Rik suggested UAR probably stood for Universal Asynchronous Receiver (the more familiar term UART both receives and Transmits.) Some of the other devices in the list can possibly be identified by comparing their pci: codes against lspci, but not those two.

Time for some experimentation. You can toggle any of these by writing to the wakeup device:

echo UAR1 >/proc/acpi/wakeup

It turned out that to disable mouse and keyboard wakeup, I had to disable both SLPB and UAR1. With both disabled, the machine wakes up when I press the power button. (What the SLeeP Button is, if it's not the power button, I don't know.)

My mouse and keyboard are PS/2. For a USB mouse and keyboard, look for something like USB0, UHC0, USB1.

The UAR1 setting is remembered even across boots: there's no need to do anything to make sure the setting is remembered. But the SLPB setting resets every time I boot. So I edited /etc/rc.local and added this line:

echo SLPB >/proc/acpi/wakeup

Keeping external kernel modules from being deleted

Why not? Well, I updated my kernel to tweak some ACPI parameters (fruitlessly, as it turns out; I'm trying to get powertop to give me more information but I haven't found the magic combination of kernel parameters it wants on this machine) and so I did a make modules_install. And it seems that make modules_install starts out by doing rm -rf /lib/modules/VERSION/kernel which removed my externally built beep module along with everything else.

I couldn't find documentation on this, but I did find Intel Wireless bug 556 which talks about the issue. Apparently somewhere along the way 2.6 started doing this rm -rf, but you can get around it by installing outside the kernel directory.

In other words, instead of

cp beep.ko /lib/modules/2.6.29.4/kernel/drivers/input/misc/

cp beep.ko /lib/modules/2.6.29.4/drivers/input/misc/

I've let the maintainer of Fancy Beeper know about this, so it won't be a problem for that module, but it's a good tip to know about in general -- I'm sure there are lots of modules that hit this problem.

Random beeps

But the Intel dual-Atom board doesn't seem to support a system beep -- there's no obvious place on the motherboard to plug in the connector going to the case speaker. How odd!

With the alternative being no beep at all, I dusted off my old blog post and went to see if Fancy Beeper Daemon kernel module still existed. Happily, it does, and it's up-to-date for current kernels, so all I had to do was download the latest and build it. Easy! Then I added "beep" to the list of automatically loaded modules in /etc/modules, blacklisted the pcspkr module using the /etc/modprobe.d/00local technique, and I was all set.

Except for the really important question: what sound to choose? I did a little web searching for free sounds and downloaded some samples to try out. Then I added a few bird calls from my Stokes Field Guide to Western Bird Songs CD, editing them in audacity to make them shorter and more appropriate for system beeps.

But I still couldn't decide on just one ... and why should I? I've really been enjoying my random wallpaper: every time I log in, I get a different desktop background. It's fun to see a new picture every day. Why not do the same for my system beep?

That's no problem, using the same randomline script I use for wallpaper. I just put this in my .xinitrc:

$HOME/bin/mybeepd `find $HOME/Music/beeps -name "*.wav" | randomline` &

Yesterday it was a loon. Today it's a cow mooing.

No X acceleration (DRI) in Jaunty: solved

The really puzzling thing, though, wasn't the crashes, but the fact that X acceleration didn't work at all. Programs like tuxracer (etracer) and Google earth would display at something like one frame update every two seconds, and glxinfo | grep renderer said

OpenGL renderer string: Software Rasterizer

But that was all on my old desktop machine, with an ATI Radeon 9000 card that I know no one cares about much. I have a new machine now! An Intel dual Atom D945GCLF2D board with 945 graphics. Finally, a graphics chip that's supported! Now everything would work!

Well, not quite -- there were major teething pains, including returning the first nonworking motherboard, but that's a separate article. Eventually I got it running nicely with Intrepid. DRI worked! Tuxracer worked! Even Google Earth worked! Unbelievable!

I copied the Jaunty install from my old machine to a partition on the new machine. Booted into it and -- no DRI. Just like on the Radeon.

Now, there's a huge pile of bugs in Ubuntu's bug system on problems with video on Jaunty, all grouped by graphics card manufacturer even though everybody seems to be seeing pretty much the same problems on every chipset. But hardly any of the bugs talk about not getting any DRI at all -- they're all about whether EXA acceleration works better or worse than XAA and whether it's worth trying UXA. I tried them all: EXA and UXA both gave me no DRI, while XAA crashed/rebooted the machine every time. Clearly, there was something about my install that was disabling DRI, regardless of graphics card. But I poked and prodded and couldn't figure out what it was.

The breakthrough came when, purely by accident, I ran that same glxinfo | grep renderer from a root shell. Guess what?

OpenGL renderer string: Mesa DRI Intel(R) 945G GEM 20090326 2009Q1 RC2 x86/MMX/SSE2

As me (non-root), it still said "Software Rasterizer." It was a simple permissions problem! But wait ... doesn't X run as root?

Well, it does, but the DRI part doesn't, as it turns out. (This is actually a good thing, sort of, in the long term: eventually the hope is to get X not to need root permissions either.)

Armed with the keyword "permissions" I went back to the web, and the Troubleshooting Intel Performance page on the Ubuntu wiki, and found the solution right away. (I'd looked at that page before but never got past the part right at the beginning that says it's for problems involving EXA vs. UXA vs. XAA, which mine clearly wasn't).

The Solution

In Jaunty, the user has to be in group video to use DRI in X. But if you've upgraded from an Ubuntu version prior to Jaunty, where this wasn't required, you're probably not in that group. The upgrader (I used do-release-upgrade) doesn't check for this or warn you that you have desktop users who aren't in the video group, so you're on your own to find out about the problem. Fixing it is easy, though: edit /etc/group as root and add your user(s) to the group.

You might think this would have been an error worth reporting, say, at X startup, or in glxinfo, or even in /var/log/Xorg.0.log. You'd think wrong. Xorg.0.log blithely claims that DRI is enabled and everything is fine, and there's no indication of an error anywhere else.

I hope this article makes it easier for other people with this problem to find the solution.

Bullfrogs in stereo

I didn't have the SLR along, but some of the frogs were close enough (and calm enough not to submerge when we got near them) that I was able to get a few decent shots.

But I really wanted to capture that sound. So I put the camera in video mode and shot a series of videos hoping to catch some of the music ... and did. They sound like this: bullfrog (mp3, 24kb).

Despite the title of this entry, the recording doesn't have any interesting stereo effects; the only microphone was the one built in to my Canon A540. It did okay, though! You'll just have to use your imagination to place two frogs as you listen, one 20 feet to the left and the other 15 feet to the right.

How to extract the audio from a camera video

Extracting the audio was a little tricky. I found lots of pages ostensibly telling me how to do it with mencoder, but none of them seemed to work. This did:

mplayer -vc null -af volume=15 -vo null -ao pcm -benchmark mvi_8992.avi

I added that -af volume=15 argument to make the sound louder, since it was a bit quiet as it came from the camera.

That produced a file named audiodump.wav, which I turned into an mp3 like this:

lame audiodump.wav bullfrogs.mp3

Upgrading without risk

On my disks, I make several relatively small partitions, maybe 15G or so (pause to laugh about what I would have thought ten or even five years ago if someone told me I'd be referring to 15G as "small"), one small shared /boot partition, a swap partition, and use the rest of the disk for /home or other shared data.

Now you can install a new release, like 9.04, onto a new partition without risking your existing install.

If you prefer upgrading rather than running the installer, you can do that too. I needed a jaunty (9.04) install to test whether a bug was fixed. But my intrepid (8.10) is working fine and I know there are some issues with jaunty, so I didn't want to risk the working install. So from Intrepid, I copied the whole root partition over to one of my spare root partitions, sda5:

mkfs.ext3 /dev/sda5
mkdir /jaunty
mount /dev/sda5 /jaunty
cp -ax / /jaunty

Now there are a couple of things you have to do to make that /jaunty partition work as a bootable install:

1. /dev on an ubuntu system isn't a real file system, but something magically created by the kernel and udev. But to boot, you need some basic stuff there. When you're up and running, that's stored in /dev/.static, so you can copy it like this: cp -ax /dev/.static/dev/ /jaunty/

Note: it used to work to copy it to /jaunty/dev/. The exact semantics of copying directories in cp and rsync, and where you need slashes, seem to vary with every release. The important thing is that you want /jaunty/dev to end up containing a lot of devices, not a directory called dev or a directory called .static. So fiddle with it after the cp -ax if you need to.

Note 2: Doesn't it just figure? A couple of days after I posted this, I found out that the latest udev has removed /dev/.static so this doesn't work at all any more. What you can do instead is:
cd /jaunty/dev
/dev/MAKEDEV generic

Note 3: If you're running MAKEDEV from Fedora, it will target /dev instead of the current directory, so you need MAKEDEV -d /whatever/dev generic
. However, caution: on Debian and Ubuntu -d deletes the devices. Check man MAKEDEV first to be sure. Ain't consistency wonderful?

2. /etc/fstab on the system you just created points to the wrong root partition, so you have to fix that. As root, edit /etc/fstab in your favorite editor (e.g. sudo vim /etc/fstab or whatever) and find the line for the root filesystem -- the one where the second entry on the line is /. It'll look something like this:

# /dev/sda1
UUID=f7djaac8-fd44-672b-3432-5afd759bc561  /  ext3  relatime,errors=remount-ro  0 1

The easy fix is to change that to point to your new disk partition:

# jaunty is now on /dev/sda5
/dev/sda5  /  ext3  relatime,errors=remount-ro  0 1

If you want to do it the "right", ubuntu-approved way, with UUIDs, you can get the UUID of your disk this way:

ls -l /dev/disk/by-uuid/ | grep sda5

Take the UUID (that's the big long hex number with the dashes) and put it after the UUID= in the original fstab line.

While you're editing /etc/fstab, be sure to look for any lines that might mount /dev/sda5 as something other than root and delete them or comment them out.

Now you should have a partition that you can boot into and upgrade. Now you just need to tell grub about it. As root, edit /boot/grub/menu.lst and find the line that's booting your current kernel. If you haven't changed the file yourself, that's probably right after a line that says:

## ## End Default Options ##

title           Ubuntu 8.10, kernel 2.6.27-11-generic
uuid            f7djaac8-fd44-672b-3432-5afd759bc561
kernel          /vmlinuz-2.6.27-11-generic root=UUID=f7djaac8-fd44-672b-3432-5afd759bc561 ro
initrd          /initrd.img-2.6.27-11-generic

Make a copy of this whole stanza, so you have two identical copies, and edit one of them. (If you edit the first of them, the new OS it will be the default when you boot; if you're not that confident, edit the second copy.) Change the two UUIDs to point to your new disk partition (the same UUID you just put into /etc/fstab) and change the Title to say 9.04 or Jaunty or My Copy or whatever you want the title to be (this is the title that shows up in the grub menu when you first boot the machine).

Now you should be able to boot into your new partition. Most things should basically work -- certainly enough to start a do-release-upgrade without risking your original install.

Making desktop recordings with recordmydesktop

I've never made videos (except for putting a digital camera in video mode) so it's been an interesting learning experience, and I was surprised at how easy it was in the end.

My Apress contact suggested XVidCap and Wink as possible options. XVidCap looked quite interesting but didn't seem to work in its Ubuntu incarnation. Wink worked very nicely and produced flash videos that worked in a browser with no extra fiddling ... but unfortunately when I tried plugging in a microphone, Wink didn't seem able to read it. And it seemed to slow down all my cursor movements, rather than following my actions in real-time.

But while working with those two, I stumbled across recordmydesktop. It records mouse movements in real-time and it handles the microphone too. It has several front ends available (such as gtk-recordmydesktop) but I found the basic command-line version easiest to use.

To make a video in the upper left 1024x768 section of my screen:
recordmydesktop -width 1024 -height 768 -o layermask.ogv

Since I need to make sure all the action happens within that rectangle, I made a special desktop background that has a nice, not too distracting image in just that 1024x768 rectangle. Any other windows I'm using in that desktop (such the terminal window I'm using to control recordmydesktop) stay outside that area.

recordmydesktop starts recording right away. I run through my tutorial steps, narrating as I go, and when I'm done, I move the mouse back to the terminal window where I started the recording and hit Ctrl-C, and recordmydesktop stops recording and encodes the video (which takes a while).

It saves to ogg format, .ogv. Of course, most web surfers can't view that, and youtube doesn't accept it either (at least, ogg isn't on its list of allowed formats) so I needed to translate it into something else. Youtube suggests mpeg4, so that's what I used. Luckily, I already had a mencoder incantation that some helpful person gave me a long time ago:
mencoder movie.ogv -oac pcm -ovc lavc -lavcopts vcodec=mpeg4:vqmin=2:vlelim=-4:vcelim=9:lumi_mask=0.05:dark_mask=0.01:vhq -o movie.mp4

Only one problem: the audio came out very faint and difficult to hear. I'm sure that's a problem with the microphone I'm using, a cheap OEM model that came with some computer or other many years ago -- it's been sitting in a box since I normally have no use for a microphone. But it turns out mencoder can amplify the volume, with -af volume=X where X is decibels. A little experimentation with mplayer -af volume=X on the original ogg suggested a value around 15 or 20, so the final encoding was:
mencoder movie.ogv -af volume=19 -oac pcm -ovc lavc -lavcopts vcodec=mpeg4:vqmin=2:vlelim=-4:vcelim=9:lumi_mask=0.05:dark_mask=0.01:vhq -o movie.mp4

But Apress tells me that their Windows boxen had trouble with the mp4 and they had to run it through something called "Handbrake", so maybe some other format would have worked better. Here are two other mencoder incantations I know (without the sound amplification):
mencoder movie.ogv -oac pcm -ovc lavc -o movie.divx (divx -- Windows sometimes has trouble with this too)
mencoder movie.ogv -oac pcm -ovc lavc -lavcopts vcodec=mpeg1video -o movie.mpeg (mpeg1)

It's not great cinema, but the end result is up on the Apress page for the GIMP book.

Update on writing udev rules for flash card readers

This always evokes horrified exclaimations from people in the Ubuntu project -- "You shouldn't need to do that!" But there are several reasons for wanting special udev rules for multi-card readers. You might want your SD card to show up in the same place every time (is it /dev/sdb1 or /dev/sdc1 today?); or you might be trying to reduce polling to cut down your CPU and battery use.

But my older article referred to a script that no longer exists, and as I recently had to update my udev rules on a fairly fresh Intrepid install, I needed something more up-to-date and less dependent on Ubuntu's specific udev scripts (which change frequently).

I found a wonderful forum article, Create your own udev rules to control removable devices, that explains exactly how to find out the names of your devices and make rules for them. Another excellent article with essentially the same information is Linux Format's Connect your devices with udev.

Start by guessing at the current device name: for example, in this particular session, my SD card reader showed up on /dev/sdd. Find out the corresponding /block device name for it, like this:

udevinfo -q path -n /dev/sdd

In my case, the SD reader was /block/sdd. Now pass that into udevinfo -a, like so:

udevinfo -a -p /block/sdd

For my card reader, I chose

 KERNEL=="sdd"
 SUBSYSTEMS=="scsi"
 ATTRS{model}=="CardReader SD   "

Note that SUBSYSTEM was scsi: usb-storage devices (handled by the scsi system) sometimes show up as usb and sometimes as scsi.

Now you're ready to create some udev rules. In your favorite text editor, create a new file named /etc/udev/rules.d/59-multicard-reader.rules. You can name it whatever you want, but make sure the number at the beginning is lower than the number of the udev rule that would otherwise create the device's name -- in this case, 60-persistent-storage.rules.

Now write your udev rule. Include the identifying lines you picked out from udevinfo -a:

KERNEL=="sd[a-g]", SUBSYSTEMS=="scsi", ATTRS{vendor}=="USB2.0  ", ATTRS{model}=="CardReader SD   ", NAME{all_partitions}="card-sd", group=plugdev

A few things to notice. First, I used KERNEL=="sd[a-g]" instead of just sdd, in case the devices might some day show up in a different order.

The NAME field can be whatever you choose. NAME{all_partitions}="card-sd" will make the device show up as /dev/card-sd, so to mount the first partition I'll use /dev/card-sd1. The {all_partitions} part tells the kernel to create partitions like /dev/card-sd1 even if there's no SD card inserted in the slot when you boot. Otherwise, you have to run touch /dev/card-sd after inserting a card to get the device created -- or run a daemon like hald-addons-storage that polls the device a few times every second checking to see if anything has been inserted (as Ubuntu normally prefers to do).

GROUP="plugdev" ensures the devices will be owned by the group named "plugdev". This isn't particularly important since you'll probably be mounting the cards using /etc/fstab lines or some sort of automount daemon.

Pause and reflect sadly on the confusing coincidence of "scsi disk" and "secure digital" both having the same abbreviation, so that you need context to tell what each of these "sd"s means.

Test your new udev line by restarting udev:

/etc/init.d/udev restart

Reading the temperature using /sys rather than /proc

cat /proc/acpi/thermal_zone/ATF0/temperature

Though this doesn't work on all machines -- on my AMD desktop it always returns the same number, which, I'm told, means that the BIOS probably has some code that looks something like this:

if (OS == "Win95" || OS == "Win98") {
  return get_win9x_temp();
}
else if (OS == "WinNT" || OS == "WinXP" || OS == "Vista") {
  return get_nt_temp();
}
else {
  return 40;
}

There wasn't anything in /proc/acpi/thermal_zone on the Mac, but /proc is deprecated and we're all supposed to be moving to /sys, right? But nobody writes about the new way to get the temperature from /sys; most people are still using the old /proc way.

Took some digging, but I found it:

cat /sys/class/thermal/thermal_zone0/temp

And on the Mini? Nope, it's not there either. If Dave needs the temperature he needs to stick to OS X, or else figure out lm_sensors.

Update: Matthew Garrett has an excellent blog article on the OS entries reported to ACPI. Apparently Linux since 2.6.29 has claimed to be "Microsoft Windows NT" to avoid just the sort of problem I mentioned. Though that leaves me confused about why my desktop machine always reports 40C. Thanks to JanC for pointing me to that article!

Helpful Error Messages from ALSA (not!)

My desktop machine has been getting flakier for a week or two. Strange messages at boot, CDROM drive unable to burn reliably or verify after burning, and finally it culminated in a morning where it wouldn't boot at all. Turned out (after much experimentation) to be not one but two bad IDE cables -- and these were the snazzy expensive heavy-duty cables, not the cheap ribbon cables, in a box that hadn't been opened for months. Weird.

Anyway, since I had the system disk out anyway (to recover data from it) I left it out, migrated my data to the newer, bigger disk and installed a new Ubuntu Intrepid. Been meaning to do that anyway -- running two disks just adds to the noise, heat and power usage and doesn't really add that much speed.

It took a couple of hours to get the system working the way I want it -- installing things I need, like tcsh, vim, emacs, plucker, vlc, sox etc. and cleaning up some of the longstanding Ubuntu udev and kernel configuration bugs that keep various hardware from working. I thought I had everything ready when I noticed I wasn't getting any sound alerts, so I tried playing a sample .wav file, and got a rather unusual error:

(clavius)- play sample.wav
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
ALSA lib conf.c:3513:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:3513:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
ALSA lib conf.c:3513:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:3985:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2196:(snd_pcm_open_noupdate) Unknown PCM default
play soxio: Can't open output file `default': cannot open audio device

What does that mean? Well, it turns out what it means is ... my user wasn't in the "audio" group, so I didn't have write permission on the sound device. I added myself to "audio" in /etc/groups and sound worked fine in my next session.

Now, I've seen some fairly obscure error messages in my time, but this one may just win my all-time obscurity award. 9 lines and 744 characters to say "Can't open $device."

And with all that, it still managed to omit the one piece of information that might have been helpful: the name of the device it was trying to open (so that an ls -l would have told me the problem right away).

Impressive!

The new gtk file selector fstab viewer

In the place where the bookmarks used to be, instead was a list of ... what are those things? Oh, I see ... they're all the filesystems listed with "noauto" in my /etc/fstab --the filesystems that aren't mounted unless somebody asks for them, typically by plugging in some piece of hardware.

There are a lot of these. Of course there's one for the CDROM drive (I never use floppies so at some point I dropped that entry). I have another entry for Windows-formatted partitions that show up on USB, like when I plug in a digital camera or a thumb drive. I also have one of those front panel flash card readers with 4 slots, for reading SD cards, memory sticks, compact flash, smart media etc. Each of those shows up as a different device, so I treat them separately and mount SD cards as /sdcard, memory sticks as /stick and so on. In addition, there are entries corresponding to other operating systems installed on this multi-boot machine, and to several different partitions on my external USB backup drive. These are all listed in /etc/fstab with entries like this:

/dev/hdd   /cdrom  udf,iso9660  user,noauto               0  0
/dev/sde1  /pix    vfat         rw,user,fmask=133,noauto  0  0

The GTK developers, in their wisdom, have realized that what the file selector really needs to be. I mean, I was just thinking while opening a file in GIMP the other day,

"Browsing image files on filesystems that are actually mounted is so tedious. I wish I could do something else instead, like view my /etc/fstab file to see a list of unmounted filesystems for which I might decide to plug in an external device."

Clicking on one of the unmounted filesystems (even right-clicking!) gives an error:

Could not mount sdcard
mount: special device /dev/sdb1 does not exist

(And before you ask, yes, for the time being I have dbus and hal and fam and gamin and all that crap running.)

The best part

But I haven't even told you the best part yet. Here it is:

If you mount a filesystem manually, e.g. mount /dev/sdb1 /mnt ... it doesn't show up in the list!

So this enormous list of filesystems that's keeping me from seeing my file selector bookmarks ... doesn't even include filesystems that are really there!

Upgraded to Intrepid: X keyboard options and losing network after suspend

The upgrade went smoothly, though with the usual amount of babysitting, watching messages scroll by for a couple of hours so that I could answer the questions that popped up every five or ten minutes. Question: Why, after all these years of software installs, hasn't anyone come up with a way to ask all the questions at the beginning, or at the end, so the user can go have dinner or watch a movie or sleep or do anything besides sit there for hours watching messages scroll by?

XKbOptions: getting Ctrl/Capslock back

The upgrade finished, I rebooted, everything seemed to work ... except my capslock key wasn't doing ctrl as it should. I checked /etc/X11/xorg.conf, where that's set ... and found the whole file commented out, preceded by the comment:

# commented out by update-manager, HAL is now used

Google led me to a forum thread on Intrepid xorg.conf - input section. The official recommendation is to run sudo dpkg-reconfigure console-setup ... but of course it doesn't allow for options like ctrl/capslock. (It does let you specify which key will act as the Compose key, which is thoughtful.)

Fortunately, the release notes give the crucial file name: /etc/default/console-setup. The XKBOPTIONS= line in that file is what I needed.

It also had the useful XKBOPTIONS="compose:menu" option left over from my dpkg-configure run. I hadn't known about that before; I'd been using xmodmap to set my multi key. So my XKBOPTIONS now looks like: "ctrl:nocaps,compose:menu".

Fixing the network after resume from suspend

Another problem I hit was suspending on my desktop machine. It still suspended, but after resuming, there was no network. The problem turned out to lie in /etc/acpi/suspend.d/55-down-interfaces.sh. It makes a list of interfaces which should be brought up and down like this:

IFDOWN_INTERFACES="`cat /etc/network/run/ifstate | sed 's/=.*//'`"
IFUP_INTERFACES="`cat /etc/network/run/ifstate`"

Google to the rescue again. The bad thing about Ubuntu is that they change random stuff so things break from release to release. The good thing about Ubuntu is a zillion other people run it too, so whatever problem you find, someone has already written about. Turns out ifstate is actually in /var/run/network/ifstate now, so making that change in /etc/acpi/suspend.d/55-down-interfaces.sh fixes suspend/resume. It's bug 295544, fixed in Jaunty and nominated for Intrepid (I just learned about the "Nominate for release" button, which I'd completely missed in the past -- very useful!) Should be interesting to see if the fix gets pushed to Intrepid, since networking after resume is completely broken without it.

Otherwise, it was a very clean upgrade -- and now I can build the GIMP trunk again, which was really the point of the exercise.

Quickie tutorial: Making an encrypted USB key or SD card

But that convenient small size also means it's easy to lose your USB key, or leave it somewhere. Someone could pick it up and have access to anything you put there. Wouldn't it be nice to encrypt it?

There are lots of howtos on the net, like this and this, but most of them are out of date and need a bit of fiddling to get working. So here's one that works on Ubuntu hardy and a 2.6.28 kernel. I'll assume that the drive is on /dev/sdb.

First, consider making two partitions on the flash drive. The first partition is a normal unencrypted vfat partition, so you can use it to transfer files to other machines, even Windows and Mac. The second partition will be your encrypted file system. Use fdisk, gparted or your favorite partitioning tool to make the partitions, then create a filesystem on the first:

mkfs.vfat /dev/sdb1

Update: On many distros you'll probably need to load the "cryptoloop" module before proceeding with the following steps. Mount it like this (as root):

modprobe cyptoloop

Easy, moderate security version

Now create the encrypted partition (you'll need to be root). I'll start with a relatively low security version -- good enough to keep out most casual snoops who happen to pick up your flash drive.

losetup -e aes /dev/loop0 /dev/sdb2
[type a password or pass phrase]
mkfs.ext2 /dev/loop0
losetup -d /dev/loop0

I used ext2 as the filesystem type. I want a Linux filesystem, not a Windows one, so I can store dot-filenames like .gnupg/ and so I can make symbolic links. I chose ext2 rather than a journalled filesystem like ext3 because of kernel configuration warnings: to get encrypted mounts working I had to enable loopback and cryptoloop support under drivers/block devices, and the cryptoloop help says:

WARNING: This device is not safe for journaled file systems like ext3 or Reiserfs. Please use the Device Mapper crypto module instead, which can be configured to be on-disk compatible with the cryptoloop device.

Now the filesystem is ready to use. Mount it with:

mount -o loop,encryption=aes /dev/sdb2 /mnt
[type your password/phrase]

Higher security version

There are several ways you can increase security. Of course, you can choose other encryption algorithms than aes -- that choice is up to you.

You can also use a random key, rather than a password you type in. Save the random key to a file on your system (obviously, you'll want to back that up somewhere else). This has both advantages and disadvantages: anyone who has access to your computer has the key and can read your encrypted disk, but on the other hand, someone who finds your flash drive somewhere will be very, very unlikely to be able to use it. Set up a random key like this:

dd if=/dev/random > /etc/loop.key bs=1 count=60
mkfs.ext2 /dev/loop0
losetup -e aes -p 0 /dev/loop0 /dev/sdb2 < /etc/loop.key

(Update: skipped mkfs.ext2 step originally.)

Then mount it with:

cat /etc/loop.key | mount -p0 -o loop,encryption=aes /dev/sdb2 /crypt

Finally, most file systems write predictable data, like superblock backups, at predictable places. This can make it easier for someone to break your encryption. In theory, you can foil this by specifying an offset so those locations are no longer so predictable. Add -o 123456 (of course, use your own offset, not that one) to the losetup line, and ,offset=123456 in the options of the mount command. In practice, though, offset doesn't work for me: I get

ioctl: LOOP_SET_STATUS: Invalid argument

Categorize your fonts with fontypython

I tried fontmatrix and found it complex and inscrutable. But it made me look for smaller font-tagging projects, and that led me to FontyPython. It's fairly small, it's Python, it's already included in Ubuntu ... and how can you not like a project with a name like that?

When you start up, you need to choose a font folder to view any fonts. Unfortunately, the standard place to put fonts on a modern Linux system, ~/.fonts, is not an option: fontypython won't look in directories starting with a dot. The only way to view fonts installed in .fonts is to specify it on the command line: fontypython .fonts

I talked to the author, and it turns out the intent is quite different: you're intended to keep your font list somewhere else (say, ~/myfonts) and use fontypython to move fonts in and out of ~/.fonts, with the Install button. That model doesn't quite match my workflow -- I'd have to keep telling apps like GIMP to rescan the font list as I added and removed fonts (and other apps besides GIMP mostly need to be restarted to see new fonts) -- but it's probably ideal for some people.

When you first start up fontypython it displays the first page of your fonts. Instead of a scrollbar, you page through using the Back/Forward buttons or the option menu down below the font list. By default, fonts are displayed quite large; you can change the size in File->Settings if you want to see more at once.

It's time to start categorizing! To do that, you need to create some pogs, a silly term taken from tyPOGraphy. Pogs are just categories of font. Click on New Pog in the buttons at the bottom right of the window and choose a name for your first pog -- you might want pogs for "script", or "handwriting", or "gothic", or "outline".

Once you've created some pogs, select one in the Target list along the right edge of the window. That's your active pog. Add fonts to the pog by clicking on them in the list (a big red checkmark will appear over the font). Mark as many as you want to move, then click "Put fonts into pogname" at the bottom of the window. Those fonts will grey out, to indicate that they're members of the current pog.

To view fonts by pog -- to view all your handwriting or script fonts -- use the Pogs tab near the upper left of the window.

Nothing to it! Unfortunately, the python routines fontypython uses fails on a few fonts; but that's true of most font viewers (like gtkfontsel), and fontypython does better than many. It does offer a way to screen out bad fonts that can't display, in case you have any fonts that cause serious problems like crashes (I didn't).

Quite a useful program if you're a font junkie like I am! I'm looking forward to using it for real projects.

Updated Network Schemes page

netscheme home

Last week for SCALE I decided it was silly to have to su and create a new scheme file for conferences where all I really needed was the name of the network (the essid), so I added a quick hack to my netscheme script so that typing netscheme foo, where there's no existing scheme by that name, will switch to a scheme using foo as the essid. Worked nicely, and that inspired me to update the "documentation".

I wrote an elaborate page on my network schemes back around 2003, but of course it's all changed since then and I haven't done much toward updating the page. So I've rewritten it completely, taking out most of the old cruft that doesn't seem to apply any more. It's here: Howto Set Up Multiple Network Schemes on a Linux Laptop.

Widescreen laptop for presentations

(What was wrong with my beloved old Vaio? Nothing, really, except that the continued march of software bloat means that a machine that can't use more than 256M RAM is hurting when trying to run programs (*cough* Firefox *cough) that start life by grabbing about 90M and goes steadily up from there. I can find lightweight alternatives for nearly everything else, but not for the browser -- Dillo just doesn't cut it.)

Ebay turned out to be the answer: there are lots of subnotebooks there, nice used machines with full displays at netbook prices. And so a month before LCA I landed a nice Vaio TX650 with 1.5G RAM, Pentium M, Intel 915GM graphics and Centrino networking. All nice Linux-supported hardware.

But that raised another issue: how do widescreen laptops (the TX650 is 1366x768) talk to a projector? I knew it was possible -- I see people presenting from widescreen machines all the time -- but nobody ever writes about how it works.

The first step was to get it talking to an external monitor at all. I ran a VGA cable to my monitor, plugged the other end into the Vaio (it's so nice not to need a video dongle!) and booted. Nothing. Hmm.

But after some poking and googling, I learned that with Intel graphics, xrandr is the answer:

xrandr --output VGA --mode 1024x768

xrandr --auto

Update, April 2010: With Ubuntu Lucid, this has changed and now it's
xrandr --output VGA1 --mode 1024x768
-- in other words, VGA changed to VGA1. You can run xrandr with no arguments to get a list of possible output devices and find out whether X sees the external projector or screen correctly.

Well, mostly. Sometimes it doesn't work -- like, unfortunately, at the lightning talk session, so I had to give my talk without visuals. I haven't figured that out yet. Does the projector have to be connected before I run xrandr? Should it not be connected until after I've already run xrandr? Once it's failed, it doesn't help to run xrandr again ... but a lot of fiddling and re-plugging the cable and power cycling the projector can sometimes fix the problem, which obviously isn't helpful in a lightning talk situation.

Eventually I'll figure that out and blog it (ideas, anyone?) but the real point of today's article is resolution. What I wanted to know was: what happened to that wide 1366-pixel screen when I was projecting 1024 pixels? Would it show me some horrible elongated interpolated screen? Would it display on the left part of the laptop screen, or the middle part?

The answer, I was happy to learn, is that it does the best thing possible: it sends the leftmost 1024 pixels to the projector, while still showing me all 1366 pixels on the laptop screen.

Why ... that means ... I can write notes for myself, to display in the rightmost 342 screen pixels! All it took was a little bit of CSS hacking in my HTML slide presentation package, and it worked fine. Now I have notes just like my Mac friends with their Powerpoint and their dual-head video cards, only I get to use Linux and HTML. How marvellous! I could get used to this widescreen stuff.

Debian/Ubuntu repositories for Pho

But I got mail from a pho user who really wanted that, and even suggested a howto. That howto didn't quite do it, but it got me moving to look for a better one, which I eventually found in the Debian Repository Howto.

It wasn't complete either, alas, so it took some trial-and-error before it actually worked. Here's what finally worked:

I created two web-accessible directories, called hardy and etch. I copied all the files created by dpgk-buildpkg on each distro -- .deb, .dsc, .tar.gz, and .changes (I don't think this last file is used by anything) -- into each directory (renaming them to add -etch and -hardy as appropriate). Then:

% cd hardy/
% dpkg-scanpackages . /dev/null | gzip > Packages.gz
% dpkg-scansources . /dev/null | gzip > Sources.gz
% cd ../etch/
% dpkg-scanpackages . /dev/null | gzip > Packages.gz
% dpkg-scansources . /dev/null | gzip > Sources.gz

Now you can use one of the following /etc/apt/sources.list lines:
deb http://shallowsky.com/apt/hardy ./
deb http://shallowsky.com/apt/etch ./

After an apt-get update, it saw pho, but it warned me

WARNING: The following packages cannot be authenticated!
  pho
Install these packages without verification [y/N]?

This page on Release check of non Debian sources was a little more helpful, and told me how to create the Release and Release.gpg file -- but then I just get a different error,

 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY

Going back to the SecureApt page, under Setting up a secure apt repository they give the two steps the other page gave for creating Release and Release.gpg, with a third step: "Publish the key fingerprint, that way your users will know what key they need to import in order to authenticate the files in the archive."

So apparently if users don't take steps to import the key manually, they can't update at all. Whereas if I leave out the Release and Release.gpg files, all they have to do is type y when they see the warning. Sounds like it's better to leave off the key. I wish, though, that there was a middle ground, where I could offer the key for those who wanted it without making it harder for those who don't care.

Garmin Vista Cx on Ubuntu "Hardy"

I've been resisting the GPS siren song for years -- mostly because I knew it would be a huge time sink involving months of futzing with drivers and software trying to get it to do something useful.

But my experience at an OpenStreetMap mapping party got me fired up about it, and I ordered a Garmin Vista Cx.

Shopping for a handheld GPS is confusing. I was fairly convinced I wanted a Garmin, just because it's the brand used by most people in the open source mapping community so I knew they were likely to work. I wanted one with a barometric altimeter, because I wanted that data from my hikes and bike rides (and besides, it's fun to know how much you've climbed on an outing; I used to have a bike computer with an altimeter and it was a surprisingly good motivator for working harder and getting in better shape).

But Garmin has a bazillion models and I never found any comparison page explaining the differences among the various hiking eTrex models. Eventually I worked it out:

Garmin eTrex models, decoded

C

Color display. This generally also implies USB connectivity instead of serial, just because the color models are newer.

H

High precision (a more sensitive satellite receiver).

x

Takes micro-SD cards. This may not be important for storing tracks and waypoints (you can store quite a long track with the built-in memory) but they mean that you can load extra base maps, like topographic data or other useful features.

Vista, Summit

These models have barometric altimeters and magnetic compasses. (I never did figure out the difference between a Vista and a Summit, except that in the color models (C), Vistas take micro-SD cards (x) while Summits don't, so there's a Summit C and HC while Vistas come in Cx and HCx. I don't know what the difference is between a monochrome Summit and Vista.)

Legend, Venture

These have no altimeter or compass. A Venture is a Legend that comes without the bundled extras like SD card, USB cable and base maps, so it's cheaper.

For me, the price/performance curve pointed to the Vista Cx.

Loading maps

Loading base maps was simplicity itself, and I found lots of howtos on how to use downloadable maps. Just mount the micro-SD card on any computer, make a directory called Garmin, and name the file gmapsupp.img. I used the CloudMade map for California, and it worked great. There are lots of howtos on generating your own maps, too, and I'm looking forward to making some with topographic data (which the CloudMade maps don't have). The most promising howtos I've found so far are the OSM Map On Garmin page on the OSM wiki and the much more difficult, but gorgeous, Hiking Biking Mapswiki page.

Uploading tracks and waypoints

But the real goal was to be able to take this toy out on a hike, then come back and upload the track and waypoint files.

I already knew, from the mapping party, that Garmins have an odd misfeature: you can connect them in usb-storage mode, where they look like an external disk and don't need any special software ... but then you can't upload any waypoints. (In fact, when I tried it with my Vista Cx I didn't even see the track file.) To upload tracks and waypoints, you need to use something that speaks Garmin protocol: namely, the excellent GPSBabel.

So far so good. How do you call GPSbabel? Luckily for me, just before my GPS arrived, Iván Sánchez Ortega posted a useful little gpsbabel script to the OSM newbies list and I thought I was all set.

But once I actually had the Vista in hand, complete with track and waypoints from a walk around the block, it turned out it wasn't quite that simple -- because Ubuntu didn't create the /dev/ttyUSB0 that Iván's script used. A web search found tons of people having that problem on Ubuntu and talking about various workarounds, involving making sure the garmin_usb driver is blacklisted in /etc/modprobe.d/blacklist (it was already), adding a /etc/udev/rules.d/45-garmin.rules file that changes permissions and ownership of ... um, I guess of the file that isn't being created? That didn't make much sense. Anyway, none of it helped.

But finally I found the fix: keep the garmin_usb driver blacklisted use "usb:" as the device to pass to GPSBabel rather than "/dev/ttyUSB0". So the commands are:

gpsbabel -t -i garmin -f usb: -o gpx -F tracks.gpx
gpsbabel -i garmin -f usb: -o gpx -F waypoints.gpx

Like so many other things, it's easy once you know the secret! Viewing tracklogs works great in Merkaartor, though I haven't yet found an app that does anything useful with the elevation data. I may have to write one.

Software brightness control in X11

It has the usual baffling "push buttons at random trying to figure out how to navigate the menu system" brightness control -- which dims the monitor's perceived brightness by about .003% if I take it all the way to the bottom of the scale. (This is apparently a bug that some of these Dells have and others don't.)

It has contrast, too -- but the monitor won't change contrast when running through the DVI cable (this is even documented in the monitor's manual). I have no idea why. It makes me wonder whether there's normally a way of changing brightness over a DVI cable; but lots of googling hasn't brought enlightenment on that score.

I tried the VGA cable. The display was very noticeably less sharp, though pressing the monitor's "auto adjust" improved it a lot. Contrast adjustment did work (and helped) using the VGA cable, but it also turned everything green. I was able to improve the color cast a bit with
xgamma -ggamma .75 -bgamma .9
but this was all looking like quite a hassle. I wanted an easier way to change brightness. xgamma wasn't it: it works well for fine-tuning but its brightness curve is way off if you try to depart by much from full brightness.

Enter xbrightness and xbrightness-gui (Mikael Magnusson to the rescue again! He knew about these excellent programs, and perhaps equally important, he had a copy of xbrightness-gui, which seems to have vanished from the web.)

xbrightness is an excellent little command-line program that sets the X gamma curve to appropriate values. Just run xbrightness BRIGHTNESS passing it a value between 0 and 65535. xbrightness-gui is an interactive program that lets you drag curves around for each of the three color curves, or the combined image, with a user interface very similar to GIMP's Curves tool. You can even save and load curves.

xbrightness-gui's coolness notwithstanding, the simple xbrightness was really all I needed. It does a fine job of adjusting the monitor brightness while keeping colors neutral. The version I was using was Mikael's version, to which he'd added the ability to adjust colors too (much like xgamma does, but using more useful curves). It turns out I don't need the color correction, but it's nice to know it's there.

But what I did need was a way to query the current brightness, and, more important, a way to bump the brightness a little bit up and down. So I added those features. Getting the current brightness isn't actually something you can do, since the whole gamma curve for the three channels is what you perceive as brightness. I didn't try to estimate perceived brightness based on the whole curve; I just took the value of the highest value for each color, and their average or maximum.

Then I tied my new increment/decrement into key bindings in Openbox. I bound W-F5 (the Windows key plus F5) to xbrightness -2560, and W-F6 to xbrightness +2560, so I can go up or down in brightness by pressing keys without having to type any five-digit numbers.

I've made available the old xbrightness-gui, since it's no longer available anywhere else; a patch that integrates my changes and Mika's into xbrightness-0.3; and the patched xbrightness tarball. They're all at http://shallowsky.com/software/xbrightness/.

One other fun thing about using X gamma settings to adjust brightness. The first night I used it, I noticed at some point that my cursor looked very different -- it had become blindingly white. It turns out that the cursor is implemented at a lower level and doesn't go through the X gamma system. So turning the brightness down via gamma curves doesn't affect the cursor, which remains always at full brightness. It's quite a nice side effect -- the cursor is much more visible than it normally is.

Using (or not) an Apple Cinema Display on a non-Apple

I know what you're thinking (if you're not a Mac user): surely Akkana's not lustful of Apple's vastly overpriced monitors when brand-new monitors that size are selling for under $200!

Indeed, I thought that until fairly recently. But there actually is a reason the Apple Cinema displays cost so much more than seemingly equivalent monitors -- and it's not the color and shape of the bezel.

The difference is that Apple cinema displays are a technology called S-IPS, while normal consumer LCD monitors -- those ones you see at Fry's going for around $200 for a 22-inch 1680x1050 -- are a technology called TN. (There's a third technology in between the two called S-PVA, but it's rare.)

The main differences are color range and viewing angle. The TN monitors can't display full color: they're only 6 bits per channel. They simulate colors outside that range by cycling very rapidly between two similar colors (this is called "dithering" but it's not the usual use of the term). Modern TN monitors are astoundingly fast, so they can do this dithering faster than the eye can follow, but many people say they can still see the color difference. S-IPS monitors show a true 8 bits per color channel.

The viewing angle difference is much easier to see. The published numbers are similar, something like 160 degrees for TN monitors versus 180 degrees for S-IPS, but that doesn't begin to tell the story. Align yourself in front of a TN monitor, so the colors look right. Now stand up, if you're sitting down, or squat down if you're standing. See how the image suddenly goes all inverse-video, like a photographic negative only worse? Try that with an S-IPS monitor, and no matter where you stand, all that happens is that the image gets a little less bright.

(For those wanting more background, read TN Film, MVA, PVA and IPS – Which one's for you?, the articles on TFT Central, and the wikipedia article on LCD technology.)

Now, the comparison isn't entirely one-sided. TN monitors have their advantages too. They're outrageously inexpensive. They're blindingly fast -- gamers like them because they don't leave "ghosts" behind fast-moving images. And they're very power efficient (S-IPS monitors, are only a little better than a CRT). But clearly, if you spend a lot of time editing photos and an S-IPS monitor falls into your possession, it's worth at least trying out.

But how? The old Apple Cinema display has a nonstandard connector, called ADC, which provides video, power and USB1 all at once. It turns out the only adaptor from a PC video card with DVI output (forget about using an older card that supports only VGA) to an ADC monitor is the $99 adaptor from the Apple store. It comes with a power brick and USB plug.

Okay, that's a lot for an adaptor, but it's the only game in town, so off I went to the Apple store, and a very short time later I had the monitor plugged in to my machine and showing an image. (On Ubuntu Hardy, simply removing xorg.conf was all I needed, and X automatically detected the correct resolution. But eventually I put back one section from my old xorg.conf, the keyboard section that specifies "XkbOptions" to be "ctrl:nocaps".)

And oh, the image was beautiful. So sharp, clear, bright and colorful. And I got it working so easily!

Of course, things weren't as good as they seemed (they never are, with computers, are they?) Over the next few days I collected a list of things that weren't working quite right:

• The Apple display had no brightness/contrast controls; I got a pretty bad headache the first day sitting in front of that full-brightness screen.
• Suspend didn't work. And here when I'd made so much progress getting suspend to work on my desktop machine!
• While X worked great, the text console didn't.

The brightness problem was the easiest. A little web searching led me to acdcontrol, a commandline program to control brightness on Apple monitors. It turns out that it works via the USB plug of the ADC connector, which I initially hadn't connected (having not much use for another USB 1.1 hub). Naturally, Ubuntu's udev/hal setup created the device in a nonstandard place and with permissions that only worked for root, so I had to figure out that I needed to edit /etc/udev/rules.d/20-names.rules and change the hiddev line to read:

KERNEL=="hiddev[0-9]*", NAME="usb/%k", GROUP="video", MODE="0660"

On the second problem, I never did figure out why suspending with the Apple monitor always locked up the machine, either during suspend or resume. I guess I could live without suspend on a desktop, though I sure like having it.

The third problem was the killer. Big deal, who needs text consoles, right? Well, I use them for debugging, but what was more important, also broken were the grub screen (I could no longer choose kernels or boot options) and the BIOS screen (not something I need very often, but when you need it you really need it).

In fact, the text console itself wasn't a problem. It turns out the problem is that the Apple display won't take a 640x480 signal. I tried building a kernel with framebuffer enabled, and indeed, that gave me back my boot messages and text consoles (at 1280x1024), but still no grub or BIOS screens. It might be possible to hack a grub that could display at 1280x1024. But never being able to change BIOS parameters would be a drag.

The problems were mounting up. Some had solutions; some required further hacking; some didn't have solutions at all. Was this monitor worth the hassle? But the display was so beautiful ...

That was when Dave discovered TFT Central's search page -- and we learned that the Dell 2005FPW uses the exact same Philips tube as the Apple, and there are lots of them for sale used,. That sealed it -- Dave took the Apple monitor (he has a Mac, though he'll need a solution for his Linux box too) and I bought a Dell. Its image is just as beautiful as the Apple (and the bezel is nicer) and it works with DVI or VGA, works at resolutions down to 640x480 and even has a powered speaker bar attached.

Maybe it's possible to make an old Apple Cinema display work on a Mac. But it's way too much work. On a PC, the Dell is a much better bet.

Spamassassin false positives: obsolete rules on Etch

A little analysis of the X-Spam-Status: headers showed that all of the misfiled messages (and lots of messages that didn't quite make it over the threshold) were hitting a rule called DNS_FROM_SECURITYSAGE.

It turned out that this rule is obsolete and has been removed from Spam Assassin, but it hasn't yet been removed from Debian, at least not from Etch.

So I filed a Debian bug. Or at least I think I did -- I got an email acknowledgement from submit@bugs.debian.org but it didn't include a bug number and Debian's HyperEstraier based search engine linked off the bug page doesn't find it (I used reportbug).

Anyway, if you're getting lots of SECURITYSAGE false hits, edit /usr/share/spamassassin/20_dnsbl_tests.cf and comment out the lines for DNS_FROM_SECURITYSAGE and, while you're at it, the lines for RCVD_IN_DSBL, which is also obsolete. Just to be safe, you might also want to add
score DNS_FROM_SECURITYSAGE 0
in your .spamassassin/user_prefs (or equivalent systemwide file) as well.

Now if only I could figure out why it was setting FORGED_RCVD_HELO and UNPARSEABLE_RELAY on messages from what seems to be perfectly legitimate senders ...

Linux Planet: Why Firefox Rocks on Linux

More fun with regexps: Adding "[no output]" in shell logs

I don't know of a way to do that in any shell (the shell would have to capture the output of every command; emacs' shell-mode does that but I don't think any real shells do) but it seemed like it ought to be straightforward enough to do as a regular expression substitute in vi. You're looking for lines where a line beginning with a prompt is followed immediately by another line beginning with a prompt; the goal is to insert a new line consisting only of "[no output]" between the two lines.

It turned out to be pretty easy in vim. Here it is:

:%s/\(^% .*$\n\)\(% \)/\1[no results]\r\2/

Explanation:

:

starts a command

%

do the following command on every line of this file

s/

start a global substitute command

\(

start a "capture group" -- you'll see what it does soon

^

match only patterns starting at the beginning of a line

%

look for a % followed by a space (your prompt)

.*

after the prompt, match any other characters until...

$

the end of the line, after which...

\n

there should be a newline character

\)

end the capture group after the newline character

\(

start a second capture group

%

look for another prompt. In other words, this whole

expression will only match when a line starting with a prompt

is followed immediately by another line starting with a prompt.

\)

end the second capture group

/

We're finally done with the mattern to match!

Now we'll start the replacement pattern.

\1

Insert the full content of the first capture group

(this is also called a "backreference" if you want

to google for a more detailed explanation).

So insert the whole first command up to the newline

after it.

[no results]

After the newline, insert your desired string.

\r

insert a carriage return here (I thought this should be

\n for a newline, but that made vim insert a null instead)

\2

insert the second capture group (that's just the second prompt)

/

end of the substitute pattern

Of course, if you have a different prompt, substitute it for "% ". If you have a complicated prompt that includes time of day or something, you'll have to use a slightly more complicated match pattern to match it.

Getting rid of .sudo_as_admin_successful

Here's what's creating it (credit goes to Dave North for figuring out most of this).

It's there because you're in the group admin, and it's there to turn off a silly bash warning. It's specific to Ubuntu (at least, Fedora doesn't do it). Whenever you log in under bash, if bash sees that you're in the admin group in /etc/groups, it prints this warning:

To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

Once you sudo to root, if you're in the admin group, sudo creates an empty file named .sudo_as_admin_successful in your home directory. That tells bash, the next time you log in, not to print the stupid warning any more. Sudo creates the file even if your login shell isn't bash and so you would never have seen the stupid warning. Hey, you might some day go back to bash, right?

If you want to reclaim your ls columns and get rid of the file forever, it's easy: just edit /etc/group and remove yourself from the admin group. If you were doing anything that required being in the admin group, substitute another group with a different name.

Console Setup in Ubuntu

I explained how Ubuntu always starts the boot process with a perfectly fine font, then about 80% of the way through boot it deliberately changes it to a garbled, difficult to read that was clearly not designed for 1024x761. Been meaning for ages to figure out how to fix it, never spent the time ... Okay, it said "Setting up console font and keymap" just before it changes the font. That message should be easy to find. Maybe I should take a few minutes now and look into it.

The message comes from /etc/init.d/console-setup, which runs a program called setupcons, which has a man page. setupcons uses /etc/default/console-setup which includes the following section:

# Valid font faces are: VGA (sizes 8, 14 and 16), Terminus (sizes
# 12x6, 14, 16, 20x10, 24x12, 28x14 and 32x16), TerminusBold (sizes
# 14, 16, 20x10, 24x12, 28x14 and 32x16), TerminusBoldVGA (sizes 14
# and 16), Fixed (sizes 13, 14, 15, 16 and 18), Goha (sizes 12, 14 and
# 16), GohaClassic (sizes 12, 14 and 16).
FONTFACE="Fixed"
FONTSIZE="16"

The hard part of changing the console font in the past has always been finding out what console fonts are available. So having a list right there in the comment is a big help. Okay, let's try changing it to Terminus and running setupcons again. Nope, error message. How about VGA? Success, looks fine. That was easy!

But while I was in that file, what about the keymap? That's another thing I've been meaning to fix for ages ... under Debian, Redhat and earlier Ubuntu versions I had a .kmap.gz console map that turned my capslock key into a Control key (the way God intended). But Ubuntu changed things all around so the old fix didn't work any more.

I found a thread from December from someone who wanted to make the exact same change, for the same reason, but the only real advice in the thread involved an elaborate ritual involving defining keymaps for X and Gnome then applying them to the console. Surely there was a better way.

It seemed pretty clear that /etc/console-setup/boottime.kmap.gz was the keymap it was using. I tried substituting my old keymap, but since I'd written it to inherit from other keymaps that no longer existed, loadkeys can't use it. Eventually I just gunzipped boottime.kmap.gz, found the Caps Lock key (keycode 29), replaced all the Caps_Locks with Controls and gzipped it back up again. And it worked!

Gary Vollink has a more detailed description, and the process hasn't changed much since his page on Getting "Control" on the "Caps Lock".

Another gem linked to from the Ubuntu thread was this excellent article on keyboard layouts under X by Daniel Paul O'Donnell. It's not relevant to the problem of setting the console keymap, but it looks like a very useful reference on how various international character input methods work under X.

Linux Planet: Linux Astronomy part III: Stellarium and Celestia

Writing this one was somewhat tricky because the current Ubuntu, "Hardy", has a bug in its Radeon handling and both these apps lock my machine up pretty quickly, so I went through a lot of reboot cycles getting the screenshots. (I found lots of bug reports and comments on the web, so I know it's not just me.) Fortunately I was able to test both apps and grab a few screenshots on Fedora 8 and Ubuntu "Feisty" without encountering crashes. (Ubuntu sure has been having a lot of trouble with their X support lately! I'm going to start keeping current Fedora and Suse installs around for times like this.)

Linux Planet: Linux Astronomy part II: XEphem

Useful shell pipeline: Who checks in how much?

2008-08-26  Jane Hacker  <hacker@domain.org>

        * src/app/print.c: make sure the Portrait and Landscape
        * buttons update according to the current setting.

I wanted to take each entry, save the name of the developer checking in, then eventually count the number of times each name occurs (the number of times that developer checked in) and print them in order from most check-ins to least.

Getting the names is easy: for check-ins in the last 9 years, I just want the lines that start with "200". (Of course, if I wanted earlier check-ins I could make the match more general.)

grep "^200" ChangeLog

But now I want to trim the line so it includes only the contributor's name. A bit of sed geekery can do that: the date is a fixed format (four characters, a dash, two, dash, two, then two spaces, so "^....-..-.. " matches that pattern.

But I want to remove the email address part too (sometimes people use different email addresses when they check in). So I want a sed pattern that will match something at the front (to discard), something in the middle (keep that part) and something at the end (discard).

Here's how to do that in sed:

grep "^200" ChangeLog | sed 's/^....-..-..  \(.*\)<.*$/\1/'

That works pretty well! But it's not quite right: it includes the two spaces after the name as part of the name. In sed, \s matches any space character (like space or tab). So you'd think this should work:

grep "^200" ChangeLog | sed 's/^....-..-..  \(.*\)\s+<.*$/\1/'

The way around that is to make the name expression specify that it can't end with a space. \S is the term for "anything that's not a space character"; so the expression becomes

grep "^200" ChangeLog | sed 's/^....-..-..  \(.*\S\)\s\+<.*$/\1/'

We have the list of names! Add a | sort on the end to sort them alphabetically -- that will make sure you get all the "Jane Hacker" lines listed together. But how to count them? The Unix program most frequently invoked after sort is uniq, which gets rid of all the repeated lines. On a hunch, I checked out the man page, man uniq, and found the -c option: "prefix lines by the number of occurrences". Perfect! Then just sort them by the number, from largest to smallest:

grep "^200" ChangeLog | sed 's/^....-..-..  \(.*\S\)\s+<.*$/\1/' | sort | uniq -c | sort -rn

Now, this isn't perfect since it doesn't catch "Checking in patch contributed by susan@otherhost.com" attributions -- but those aren't in a standard format in most projects, so they have to be handled by hand.

Disclaimer: Of course, number of check-ins is not a good measure of how important or productive someone is. You can check in a lot of one-line fixes, or you can write an important new module and submit it for someone else to merge in. The point here wasn't to rank developers, but just to get an idea who was checking into the tree and how often.

Well, that ... and an excuse to play with nifty Linux shell pipelines.

Writing for Linux Planet: Stargazing with KStars

I have an article on Linux Planet! The first of many, I hope. At least the first of a short series on Linux astronomy programs, starting with the one that's easiest to use: KStars. It's oriented toward binocular observing, with suggestions for good targets for beginners.

Viewing the Night Sky with Linux, Part I: KStars

Back from OSCON

A few notes about OSCON:

It was a good conference -- lots of good speakers, interesting topics and interesting people. Best talks: anything by Paul Fenwick, anything by Damian Conway.

The Arduino tutorial was fun too. It's a little embedded processor with a breadboard and sockets to control arbitrary electronic devices, all programmed over a USB plug using a Java app. I'm not a hardware person at all (what do those resistor color codes mean again?) but even I, even after coming in late, managed to catch up and build the basic circuits they demonstrated, including programming them with my laptop. Very cool! I'm looking forward to playing more with the Arduino when I get a spare few moments.

The conference's wi-fi network was slow and sometimes flaky (what else is new?) but they had a nice touch I haven't seen at any other conference: Wired connections, lots of them, on tables and sofas scattered around the lounge area (and more in rooms like the speakers' lounge). The wired net was very fast and very reliable. I'm always surprised I don't see more wired connections at hotels and conferences, and it sure came in handy at OSCON.

The AV staff was great, very professional and helpful. I was speaking first thing Monday morning (ulp!) so I wanted to check the room Sunday night and make sure my laptop could talk to the projector and so forth. Everything worked fine.

Portland is a nice place to hold a convention -- the light rail is great, the convention center is very accessible, and street parking isn't bad either if you have a car there.

Dave went with me, so it made more sense for us to drive. The drive was interesting because the central valley was so thick with smoke from all the fires (including the terrible Paradise fire that burned for so long, plus a new one that had just started up near Yosemite) that we couldn't see Mt Shasta when driving right by it. It didn't get any better until just outside of Sacramento. It must have been tough for Sacramento valley residents, living in that for weeks! I hope they've gotten cleared out now.

I finally saw that Redding Sundial bridge I've been hearing so much about. We got there just before sunset, so we didn't get to check the sundial, but we did get an impressive deep red smoky sun vanishing into the gloom. Photos here.

End of my little blog-break, and time to get back to scrambling to get caught up on writing and prep for the GetSET Javascript class for high school girls. Every year we try to make it more relevant and less boring, with more thinking and playing and less rote typing. I think we're making progress, but we'll see how it goes next week.

Custom ringtones on a Motorola phone, from Linux

I sense your shock and amazement -- from people who know me, that I would do such a thing, and from everybody else at the concept that there's anybody in 2008 who didn't already have one.

I really don't think cellphones are evil, honest! (Except in the hands of someone driving a car -- wouldja please just put the phone down and pay attention to the friggin' road?) The truth is that I just don't much like talking on the phone, and generally manage fine with email. The land-line phone works fine for the scant time I spend on the phone, and I have to have the land line anyway (as part of the DSL package) so why pay another monthly bill for a second phone?

Prepaid plans looked like just the ticket, and that's what I got. With a cute little Motorola V195s. New toy! Rock! It can take custom MP3 ringtones and Java games ... but of course I don't want theirs, I want to make my own. So I wanted to talk to the phone from Linux.

The charger plug was a familiar shape -- looked a lot like a standard mini USB connector. Could the hardware be that easy? Sure enough, it's a standard mini USB. Kudos to Motorola for making that so easy! Now what about software?

My initial web searches led me down a false trail paved with programs like wammu and gnokii. I learned that I needed to enable ACM in my kernel (that's the modem protocol most cellphones use over USB), so as long as I was building a new kernel anyway, I grabbed the latest tarball from kernel.org (2.6.25.7). With that done, I was able to talk to the phone with gnokii, but the heavily Nokia-oriented program didn't show me much that looked useful.

Moto4lin is the answer

I set the project aside for a while. But half a week later while looking for something else, I stumbled across moto4lin, which turned out to be exactly what I needed. I had to run as root, or else when I try to connect, it prints on stderr:

sendControl Error:[error sending control message: Operation not permitted]

So run as root, click Connect, click File Manager if you're not already in that mode, then click Update List and it reads the files. Once they're there, you can click around in the folder list on the left looking for the audio files (on my phone, they're in a directory called audio somewhere under C, not A). Excellent!

Creating a ringtone leads to a kernel debugging digression

Okay, now I needed a ringtone. I wanted to use a bit of birdsong, so I loaded one of the tracks I use for tweet into Audacity and fiddled semi-randomly until I figured out how to cut and save a short clip. It would only save as WAV, but lame clip.wav clip.mp3 solved that just fine.

(Update: the easiest way is to select the clip you want, then do File->Export Selection...)

Except ... somewhere along the way, the clips stopped playing. I couldn't even play the original ogg track from tweet. It *looked* like it was playing ... it found the track, printed information about it, showed a running time-counter for the appropriate amount of time ... but made no sound.

It eventually turned out that the problem was that shiny new 2.6.25.7 kernel I'd downloaded. A bug introduced in 2.6.24 to the ymfpci sound card driver makes Yamaha sound cards unable to play anything with a bitrate of 44100 (which happens to be the typical CD bitrate). After a lot of debugging I eventually filed bug 10963 with a patch that reverts the old, working code from 2.6.23.17.

Ringtone success

Okay, a typical open source digression. But while I was still trying to track down the kernel bug, I meanwhile found this Razr page that tipped me off that I might need a different bitrate for ringtones anyway. So I converted it with:

lame -b 40 mock.wav mock.mp3

In the end, I was able to transfer the file easily to the motorola phone, and to use it as my nifty new ringtone. Success! Too bad nobody ever calls me and this phone is mostly for outgoing calls ...

Now to look for some fun Java apps.

Fixing scanner permissions on Hardy

Ha! Famous last words. For Hardy, I guess the Ubuntu folks decided that users had had it too easy for a while and it was time to throw us a challenge. Under Hardy, scanning works fine as root, but normal users can't access the scanner. sane-find-scanner sees the scanner, but xsane and the xsane-gimp plug-in can't talk to it (except as root).

It turns out the code for noticing you plugged in a scanner and setting appropriate permissions (like making it group "scanner") has been removed from udev, the obvious place for it ... and moved into hal. Except, you guessed it, whatever hal is supposed to be doing isn't working, so the device's group is never set to "scanner" to make it accessible to non-root users. Lots of people are hitting this and filing bugs (search for scanner permissions), in particular bug 121082 and bug 217571.

Fortunately, the fix is quite easy if you have a copy of your old gutsy install: just copy /etc/udev/rules.d/45-libsane.rules from gutsy to the same place on hardy. (If you don't have your gutsy udev rules handy, I attached the file to the latter of the two bugs I linked above.)

Then udev will handle your scanner just like it used to, and you don't have to wait for the hal developers to figure out what's wrong with the new hal rules.

How to set your time zone

Anyway, that means I have to check and reset the time periodically. So this morning I did a time check and found it many hours off. No, wait -- actually it was pretty close; it only looked like it was way off because the system had suddenly decided it was in UTC, not PDT. But how could I change that back?

I checked /etc/timezone -- sure enough, it was set to UTC. So I changed that, copying one from a debian machine -- "US/Pacific", but that didn't do it, even after a reboot.

I spent some time reading man hwclock -- there's a lot of good reading in that manual page, about the relation between the system (kernel) clock and the hardware clock. Did you know that you're not supposed to use the date command to set the system time while the system is running? Me neither -- I do that all the time. Hmm. Anyway, interesting reading, but nothing useful about the system time zone.

It has an extensive SEE ALSO list at the end, so I explored some of those documents. /usr/share/doc/util-linux/README.Debian.hwclock is full of lots of interesting information, well worth reading, but it didn't have the answer. man tzset sounded promising, but there was no such man page (or program) on my system. Just for the heckofit, I tried typing tz[tab] to see if I had any other timezone-related programs installed ... and found tzselect. And there was the answer, added almost as an afterthought at the end of the manual page:

Note that tzselect will not actually change the timezone for you. Use 'dpkg-reconfigure tzdata' to achieve this.

Two font mysteries solved

The first concerns the bitstream vera sans mono I've been using as a terminal font in apps like rxvt and xterm. I'd been specifying it in ~/.Xdefaults like this:
XTerm*font: -bitstream-bitstream vera sans mono-bold-r-normal-*-12-*-*-*-*-*-iso10646-1

The mystery is that I'd noticed that in xterm, the font looked slightly different -- slightly uglier -- than in rxvt (both apps use the same X class name of XTerm). It was hard to put my finger on what was different -- the shape of all the letters looked the same, but it just seemed a little more ragged, and a little less compact, in xterm. I figured it was just a minor difference in their drawing code, or something.

Well, I was fiddling with fonts (trying to get the new-to-me "Inconsolata" font working) and I noticed that iso10646 bit. I didn't know what 10646 was, but shouldn't it be 8859-1 or 8859-15, the codes for the Latin-1 alphabet? After finishing up my Inconsolata experiments, when I set the font back to Vera I changed the line to XTerm*font: -bitstream-bitstream vera sans mono-bold-r-normal-*-12-*-*-*-*-*-iso8859-15 and moved on to other things.

Until the next morning, when I booted up to a surprise: my main terminal window no longer fit on the screen. It seems it had reverted to the other (uglier) version of Vera Sans Mono, which is also very slightly taller, so instead of being a couple of lines shorter than the screen height, it was a couple of lines too tall to fit.

I checked .Xdefaults -- yes, it was still Vera. What was going on? I finally remembered the one thing I had changed: the language setting on the font, from 10646-1 to 8858-15. I changed it back: sure enough, now the font was pretty again and the terminal was short enough to fit.

I fired up xfontsel and did some experimenting. It turned out the difference between the two almost-identical Vera sans mono bold roman fonts is a field xfontsel calls "spc". It can be either 'c' or 'm'. The 'c' version is the pretty, compact font; the 'm' is the uglier, taller one. For some reason, specifying 10646-1 makes "spc" default to 'c', while 8859-15 makes it default to 'm'. But specifying 'c' in the font specifier gets the good version regardless of which language is specified.

So this would work: XTerm*font: -bitstream-bitstream vera sans mono-bold-r-normal-*-12-*-*-*-c-*-*-*

But then I read up on 10646-1 and it turns out to mean "the whole unicode character set". That sounds like a good idea, so I kept it in my font specifier after all: XTerm*font: -bitstream-bitstream vera sans mono-bold-r-normal-*-12-*-*-*-c-*-iso10646-1

(For the moment I still didn't know what spc, c or n meant; read on if you're curious.)

The second insight concerned a longstanding mystery of Dave's. He has been complaining for quite a while about the way Ubuntu's modern pango-based apps all refuse to see bitmapped fonts. (It bothered me too, but less so, because the terminal and editor apps I use can see X fonts.)

Dave has an Ubuntu install on one machine that he's been upgrading release after release, which does see his bitmapped fonts. But any fresh Ubuntu installation fails to see the fonts. What was the difference?

We knew about the trick of going into /etc/fonts/conf.d, removing the symbolic link 70-yes-bitmaps.conf and replacing it with a link to /etc/fonts/conf.avail/70-yes-bitmaps.conf ... But doing that doesn't actually change anything, and bitmap fonts still don't show up.

The secret turned out to be that you need to run fc-cache -fv after changing the font/conf.d links. This apparently never happens on its own -- not on a reboot, not on installing or uninstalling font packages. Somehow it had happened once on Dave's good install, and that's why it worked there but nowhere else.

I'm not sure how anyone is supposed to find out about fc-cache -- there's no man fontconfig, and the /etc/fonts/conf.avail/README offers no clue, just misleadingly says "Fontconfig scans this directory". man fc-cache mentions /usr/share/doc/fontconfig/fontconfig-user.html, which doesn't exist; it turns out on Ubuntu it's actually /usr/share/doc/fontconfig-config/fontconfig-user.html. But wait, that's just an html-ized manual page for fonts-conf, so actually you could just run man fonts-conf ... your guess is as good as mine why the fc-cache man page sends you on a hunt for html files instead.

man fonts-conf is good reading -- it even solves the mystery of that spc parameter. It stands for spacing and can be proportional, dual-width, monospace or charcell. Aha! And there's lots more useful-looking information in that manual page as well.

How often does updatedb really need to run?

/etc/cron.weekly/slocate:
slocate: fatal error: load_file: Could not open file: /etc/updatedb.conf: No such file or directory

That's the script that updates the database for locate, Linux's fast find system. I figured I must have screwed something up when I moved that slocate cron script from cron.daily to cron.weekly (because I hate having my machine slow to a crawl as soon as I boot it in the morning, and it doesn't bother me if the database doesn't necessarily have files added in the last day or two).

But after talking to some other folks and googling for Ubuntu bugs, I discovered I wasn't the only one getting that mail, and there was already a bug covering it. Comparing my setup with another Hardy user's, I found that the file slocate was failing to find, /etc/updatedb.conf, belongs to a different package, mlocate. If mlocate is installed, then slocate's cron script works; otherwise, it doesn't. Sounds like slocate should have a dependency that pulls in mlocate, no?

But wait, what do these two packages do? Let's try a little aptitude search locate:

p   dlocate                         - fast alternative to dpkg -L and dpkg -S   
p   kio-locate                      - kio-slave for the locate command          
i   locate                          - maintain and query an index of a directory
p   mlocate                         - quickly find files on the filesystem based
i   slocate                         - Secure replacement of findutil's locate   

Worse, if I install mlocate (so slocate will work) and then look in my cron directories, it turns out I now have, count 'em, five different cron scripts that run updatedb. They are:

In cron.daily:

locate: 72 lines! but a lot of that is comments and pruning, and a lot of fiddling to figure out what version of the kernel is running to see whether it can pass any advanced flags when it tries to renice the process. In the end it calls updatedb.findutils (note no full path, though it uses a full path when it checks for it earlier in the script).

slocate: A much simpler but unfortunately buggy 20 lines. It checks for /etc/updatedb.conf, runs it if it exists, fiddles with ionice, checks again for /etc/updatedb.conf, and based on whether it finds it, runs either /usr/bin/slocate -u or /usr/bin/slocate -u -f proc. The latter path is what was failing and sending root mail every time the script was run.

mlocate: an even slimmer 12 line script, which checks for /usr/bin/updatedb.mlocate and, if it exists, fiddles ionice then runs /usr/bin/updatedb.mlocate.

In cron.weekly:

Two virtually identical scripts called find.notslocate and find.notslocate.dpkg-new, which differ only in dpkg-new having more elaborate ionice options. They both run updatedb. And which updatedb would that be? Probably /usr/bin/updatedb, which links to /etc/alternatives/updatedb, which probably links to either updatedb.mlocate or updatedb.slocate, whichever you've installed most recently. But in either case, it's hard to see why you'd need this script running weekly if you're already running both flavors of updatedb from other scripts cron.daily. And having two copies of the script is just plain wrong (and there was already a bug filed on it). (As long as you're poking around in cron.daily and cron.weekly, check and see if you have any more of these extra dpkg-new or dpkg-old scripts -- they might be slowing down your machine for no reason.)

Further research reveals that mlocate is a new(ish) package intended to replace slocate. (There was a long discussion of that on ubuntu-devel, leading to the replacement of slocate with mlocate very late in the Hardy development cycle. There was also lots of discussion of "tracker", apparently a GUI fast find tool that can only search in the user's home directory.)

What is this mlocate? The m stands for "merge": the advantage of mlocate is that it can merge new results into its existing database instead of replacing the whole thing every time. Sounds good, right? However, the down side is that mlocate apparently can't to purge its database of old files that no longer exist, and these files will clutter up your locate results. Running locate -e will keep them from being printed -- but there seems to be no way to set this permanently, via an environment variable or .locaterc file, nor to tell updatedb.mlocate to clean up its database. So you'll need to alias locate to locate -e if you want sensible behavior. Or go back to slocate. Sigh.

Cleaning up

The important thing is to get rid of most of those spurious updatedb cron scripts. You might choose to run updatedb daily, weekly, or only when you choose to run it; but you probably don't want five different scripts running two different versions of updatedb at different times. The packages obviously aren't cleaning up after themselves, so let's do a little manual cleanup.

That find.slocate script looks suspicious. In fact, if you run dpkg -S find.notslocate, you find out that it doesn't belong to any package -- not only should the .dpkg-old version not be there, neither should the other one! So out they go.

As for slocate and mlocate, it's important to know that the two packages can coexist: installing mlocate doesn't remove slocate or vice versa. A clean Hardy install should have only mlocate; upgrades from Gutsy are more likely to have a broken slocate.

Having both packages probably isn't what you want. So pick one, and remove or disable the other. If mlocate is what you want, apt-get purge slocate and just make sure that /etc/cron.*/slocate disappears. If you decide you want slocate, it's a little trickier since the slocate package is broken; but you can fix it by creating an empty /etc/updatedb.conf so updatedb.slocate won't fail.

Autologin changes again for Hardy's upstart

The /usr/bin/loginscript still hasn't changed, and this still works:

#! /bin/sh
/bin/login -f yourusername

But the syntax has changed a little for the getty line in /etc/event.d/tty1: respawn is now on its own line (I don't know if that matters -- I still can't find any documentation on this file's syntax, though I found a new upstart page that links to some blog entries illustrating how upstart can be used to start system daemons like dbus). And the getty now needs an exec before it. Like this:

respawn
exec /sbin/getty -n -l /usr/bin/loginscript 38400 tty1

Upgrading from Ubuntu Gutsy to Hardy

1. cp -ax / /hardy
2. cp -ax /dev/.static/dev/* /hardy/dev/
3. Fix up files like /hardy/etc/fstab and /boot/grub/menu.lst
4. Reboot into the newly copied gutsy
5. do-release-upgrade -d

It took an hour or two to pull down all the files, followed by a long interval of occasionally typing Y or N, and then I was ready to start cleaning up some of the packages I'd noticed flying by that I didn't want. Oops! I couldn't remove or install anything with apt-get, because: dpkg --configure -a
But I couldn't dpkg --configure -a because several packages were broken.

The first broken package was plucker, which apparently had failed to install any files. Its postinstall script was failing because it had no files to operate on; and then I couldn't do anything further with it because apt-get wouldn't do anything until I did a dpkg --reconfigure -a

I finally got out of that by dpkg -P plucker; then after several more dpkg --reconfigure -a rounds I was eventually able to apt-get install plucker (which installed just fine the second time).

But apt still wasn't happy, because it wanted to run the trigger for initramfs-tools, which wouldn't run because it wanted kernel modules for some specific kernel version in /lib/modules. I didn't have any kernel modules because I'm not running Ubuntu's kernel (I'm stuck on 2.6.23 because bug 10118 makes all 2.6.24 variants unable to sync with USB Palm devices). But I couldn't remove initramfs-tools because udev (along with a bunch of other less important packages) depends on it. I finally found my way out of that by removing /var/lib/dpkg/triggers/initramfs-tools. I reported it as bug 220094.

Update: I forgot to mention one important thing I hit both on this machine and earlier, on the laptop: /usr/bin/play (provided by the "sox" package) no longer works because it now depends on a zillion separate libraries. apt-get install libsox-fmt-all to get all of them.

A minimal xorg.conf

Anyway, part of this has involved taking out parts of his /etc/X11/xorg.conf file to see which parts might be causing the problem, and he's found something interesting.

What do you suppose is the minimal useful xorg.conf file? You might suppose, oh, screen and monitor sections, an input section for the keyboard and another one for a generic mouse, and that might be all you need ... right?

Okay, try it. Let's start with a really minimal file -- nothing -- and gradually add sections. To try it, make a backup of your current xorg.conf, then zero out the file:

cd /etc/X11
mv xorg.conf xorg.conf.sav
cp /dev/null xorg.conf

Now exit X if you hadn't already, and start it up again (or let gdm do it for you). Be prepared to do repairs from the console in case X doesn't start up: e.g. sudo cp /etc/X11/xorg.conf.bak /etc/X11/xorg.conf

What happened?

In my case, on the laptop running Hardy beta, X starts right up and looks just the same as it did before.

xorg.conf -- who needs it?

A specious question, of course, which has a perfectly good answer: anyone who needs a resolution other than whatever xorg picks as the default; anyone with additional hardware, like a wacom tablet; anyone who wants customizations like XkbOptions = ctrl:nocaps. There are lots of reasons to have an xorg.conf. But it's fun to know that at least on some machines, it's possible to run without one.

Update: turns out this is part of Ubuntu's new BulletProof X feature. It doesn't work on other distros or older versions. Thanks to James D for the tip.

k3b breaks without hal

The burn went fine, and k3b ejected the CD, then sucked it back in again for the verification stage. At that point k3b started spewing lots of errors to the terminal, things like "/dev/hdd: READ 10 failed!" and "Failed to init HAL context!" repeated many times.

How annoying! k3b has added a new dependency on hal, and although it can burn a CD just fine, without hal it then forgets where the CD drive was so it can read the CD back in to verify it.

Fortunately dd /dev/cdrw | md5sum worked fine to verify that the burn was correct. I guess it's time to investigate other CD burning programs.

Ubuntu "Hardy Heron"

I was trying the beta primarily in the hope that it would fix a serious video out regression that appeared in Gutsy (the current Ubuntu) in January. My beloved old Vaio SR17 laptop can't switch video signals on the fly like some laptops can; I've always needed to boot it with an external monitor or projector connected. But as long as it saw a monitor at boot time, it would remember that state through many suspend cycles, so I could come out of suspend, plug in to a projector and be ready to go. But beginning some time in late January, somehow Gutsy started doing something that turned off the video signal when suspending. To talk to a projector, I could reboot with the projector connected (I hate making an audience watch that! and besides, it takes away the magic). I also discovered that switching to one of the alternate consoles, then back (ctl-alt-F2 ctl-alt-F7) got a signal going out on the video port -- but I found out the hard way, in front of an audience, that it was only a 640x480 signal, not the 1024x768 signal I expected. Not pretty! I could either go back to Feisty ... or try upgrading to Hardy.

I've already written about the handy debootstrap lightweight install process I used. (I did try the official Hardy "alternate installer" disk first, but after finishing package installation it got into a spin lock trying to configure kernel modules, so I had to pull the plug and try another approach.)

This left me with a system that was very minimal indeed, so I spent the next few hours installing packages, starting with tcsh, vim (Ubuntu's minimal install has something called vim, but it's not actually vim so you tend to get lots of errors about parsing your .vimrc until you install the real vim), acpi and acpi-support (for suspending), and the window system: xorg and friends. To get xorg, I started with:

apt-get install xserver-xorg-video-savage xbase-clients openbox xloadimage xterm

Then there was the usual exercise of aptitude search font and installing everything on that list that seemed relevant to European languages (I don't really need to scroll through dozens of Tamil, Thai, Devanagari and Bangla fonts every time I'm looking for a fancy cursive in GIMP).

But I hit a problem with that pretty early on: turns out most of the fonts I installed weren't actually showing up in xlsfonts, xfontsel, gtkfontsel, or, most important, the little xlib program I'm using for a talk I need to give in a couple weeks. I filed it as bug 212669, but kept working on it, and when a clever person on #ubuntu+1 ("redwhitewaldo") suggested I take a look at the x-ttcidfont-conf README, that gave me enough clue to get me the rest of the way. Turns out there's a Debian bug with the solution, and the workaround is easy until the Ubuntu folks pick up the update.

I hit a few other problems, like the PCMCIA/udev problem I've described elsewhere ... but mostly, my debootstrapped Hardy Heron is working quite well.

And in case you're wondering whether Hardy fixed the video signal problem, I'm happy to say it does. Video out is working just fine.

Debootstrap

It turns out Debian (and therefore also Ubuntu) has had this for years, and it's totally cool. It's called debootstrap.

Some folks on the #ubuntu+1 channel told me about it, and I found a nice clear howto article on how to use it for Debian. It works just the same for Ubuntu.

First, get the .deb package for the debootstrap you want to use. Here's debootstrap for Ubuntu Hardy Heron. Install it with dpkg -i. Then run it, giving it the name of the system you want to install and the directory (or mounted partition) where you want to install it. Like this: debootstrap hardy /mnt/hda3

That's all! It fetches the files it needs from the online repositories. It takes no time at all -- this really is a minimal system.

Then you need to do some fiddling to turn it into a bootable system. That includes (all paths relative to the newly installed filesystem unless otherwise stated):

• Set up etc/fstab to list the fileystems on the disk, and to mount / from the filesystem you just installed
• Define the hostname in etc/hostname
• Set up a grub boot stanza in /boot/grub/menu.lst (that's /boot on the current system, which should be the same as /boot in the new fstab you just created). Use whatever kernel you were using for your old system, for now.

Now you're read to reboot into the new system. Of course, since this is a very minimal system, you have a lot more work to do. Hardly anything is installed, and nothing has been configured for you. Some things may be challenging (for example, as I write this, X is installed but most of the fonts aren't showing up properly, which may be a bug in Hardy).

Anyway, you can get a good start by mounting your old system's root directory and copying some starter files from there, starting with these:

• Set up your important configuration files: /etc/network/interfaces, /etc/hosts, /etc/resolv.conf, /etc/passwd etc.
• edit /etc/apt/sources.list to include restricted universe multiverse
• Install a kernel package if you're using distro kernels
• Install vim if you're a vim user -- remember, ubuntu comes with something called vim that isn't really vim.
• Create users and homedirs and such
• Install all the other stuff you want -- X, gimp/gtk, development tools, editors, shells -- all that stuff that makes the system feel like home. You're on your own there, so have fun!

Handling a network card automatically: Calling ifup from udev

The card is a cardbus 3Com card that uses the 3c59x driver. When I plug it in, or when I boot or resume after a suspend, the card ends up in a state where it shows up in ifconfig eth0, but it isn't marked UP. ifup eth0 says it's already up; ifdown eth0 complains error: SIOCDELRT: No such process but afterward, I can run ifup eth0 and this time it works. I've made an alias, net, that does sudo ifdown eth0; sudo ifup eth0. That's silly -- I wanted to fix it so it happened automatically.

Unfortunately, there's nothing written anywhere on debugging udev. I fiddled a little with udevmonitor and udevtest /class/net/eth0 and it looked like udev was in fact running the ifup rule in /etc/udev/rules.d/85-ifupdown.rules, which calls: /sbin/start-stop-daemon --start --background --pid file /var/run/network/bogus --startas /sbin/ifup -- --allow auto $env{INTERFACE} So I tried running that by hand (with $env{INTERFACE} being eth0) and, indeed, it didn't bring the interface up.

But that suggested a fix: how about adding --force to that ifup line? I don't know why the card is already in a state where ifup doesn't want to handle it, but it is, and maybe --force would fix it. Sure enough: that worked fine, and it even works when resuming after a suspend.

I filed bug 211955 including a description of the fix. Maybe there's some reason for not wanting to use --force in 85-ifupdown (why wouldn't you always want to configure a network card when it's added and is specified as auto and allow-hotplug in /etc/network/interfaces?) but if so, maybe someone will suggest a better fix.

Synaptics and USB mouse simultaneously

I eventually got it working. I tried various options, but here's what I settled on:

Section "InputDevice"
        Identifier      "Trackpad"
        Driver          "synaptics"
        Option          "SHMConfig"             "true"
        Option          "SendCoreEvents"        "true"
        Option          "Device"                "/dev/psaux"
        Option          "Protocol"              "auto-dev"
        Option          "MinSpeed"              "0.5"
        Option          "MaxSpeed"              "0.75"
# AccelFactor defaults to .0015 -- synclient -l to check
        Option          "TouchpadOff"           "2"
        Option          "Emulate3Buttons"       "true"
EndSection

Section "InputDevice"
        Identifier      "Configured Mouse"
        Driver          "mouse"
        Option          "CorePointer"
        Option          "Device"                "/dev/input/mice"
        Option          "Protocol"              "ExplorerPS/2"
        Option          "ZAxisMapping"          "4 5"
        Option          "Emulate3Buttons"       "true"
EndSection

Life was groovy (I thought). Fast forward to LCA, a few days before my talk, when I decide to verify that I can run my USB mouse and the slide-advancing presentation gizmo through a hub off the single USB port. Quel surprise: the USB mouse doesn't work at all!

I didn't really need a mouse for that presentation (it was on GIMP scripting, not GIMP image editing) so I put it on the back burner, and came back to it when I got home. As I suspected, the USB mouse was working fine if I commented out the Synaptics entry from xorg.conf; it just couldn't run both at the same time.

A little googling led me to the answer, in a thread called Can't use Synaptics TouchPad and USB Mouse -- it wasn't the first google hit for synaptics "xorg.conf" usb mouse, so perhaps this entry will help its google-fu. The important part I was missing was in the "ServerLayout" section:

        InputDevice     "Trackpad"              "AlwaysCore"
        InputDevice     "Configured Mouse"      "CorePointer"

Adding "AlwaysCore" and "CorePointer" parts was what did the trick. Thanks to "finferflu" who posted the right answer in the thread.

Gutsy's persistent net rules don't persist

This week's wi-fi adventure involved Ubuntu's current "Gutsy Gibbon" release and my prism54 wireless card. I booted the machine, switched to the right (a href="http://shallowsky.com/linux/networkSchemes.html">network scheme, inserted the card, and ... no lights. ifconfig -a showed the card on eth1 rather than eth0.

After some fiddling, I ejected the card and re-inserted it; now ifconfig -a showed it on eth2. Each time I inserted it, the number incremented by one.

Ah, that's something I remembered from Debian Etch -- a problem with the udev "persistent net rules" file in /etc/udev.

Sure enough, /etc/udev/70-persistent-net.rules had two entries for the card, one on eth1 and the other on eth2. Ejecting and re-inserting added another one for eth3. Since my network scheme is set up to apply to eth0, this obviously wouldn't work.

A comment in that file says it's generated from 75-persistent-net-generator.rules. But unfortunately, the rules uesd by that file are undocumented and opaque -- I've never been able to figure out how to make a change in its behavior. I fiddled around for a bit, then gave up and chose the brute force solution:

• Remove /etc/udev/75-persistent-net-generator.rulesa
• Edit /etc/udev/70-persistent-net.rules to give the device the right name (eth1, eth0 or whatever).

And that worked fine. Without 75-persistent-net-generator.rules getting in the way, the name seen in 70-persistent-net.rules works fine and I'm able to use the network.

The weird thing about this is that I've been using Gutsy with my wired network card (a 3com) for at least a month now without this problem showing up. For some reason, the persistent net generator doesn't work for the Prism54 card though it works fine for the 3com. A scan of the Ubuntu bug repository reveals lots of other people hitting similar problems on an assortment of wireless cards; bug 153727 is a fairly typical report, but the older bug 31502 (marked as fixed) points to a likely reason this is apparently so common on wireless cards -- apparently some of them report the wrong MAC address before the firmware is loaded.

Smart Wrapping with Greedy and Non-Greedy Regular Expressions

I have no idea how regular attendees manage the information overload of deciding which talks to attend. I wasn't sure how I would, either, but I started by going through the schedule for the day I'd be there, picking out a (way too long) list of potentially interesting talks, and saving them as lines in a file.

Now I had a file full of lines like:

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights Into the Cratering Process From Geophysics and Geochemistry II

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics
                          and Geochemistry II

Only it wasn't line noise. Seems Mikael just happened to have been reading about some of the finer points of vim regular expressions earlier that day, and he knew exactly the trick I needed -- that .\{72\}, which matches lines that are at least 72 characters long. And amazingly, that expression did something very close to what I wanted.

Or at least the first step of it. It inserts the first line break, turning my line into

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics and Geochemistry II

But that was an easier problem -- just do essentially the same thing again, but limit it to only lines starting with a tab. After some tweaking, I arrived at exactly what I wanted:

%s/^\(.\{,65\}\) \(.*\)/\1^M^I^I^I\2/

%g/^^I^I^I.\{58\}/s/^\(.\{,55\}\) \(.*\)/\1^M^I^I^I\2/

Devdas helpfully translated the second one into English: "You have 3 tabs, followed by 58 characters, out of which you match the first 55 and put that bit in $1, and the capture the remaining in $2, and rewrite to $1 newline tab tab tab $2."

Here's a more detailed breakdown:

Line one:

%	Do this over the whole file
s/	Begin global substitute
^	Start at the beginning of the line
\(	Remember the result of the next match
.\{,65\}_	Look for up to 65 characters with a space at the end
\) \(	End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\)	Pattern #2 includes everything to the end of the line
/	End of matched pattern; begin replacement pattern
\1^M	Insert saved pattern #1 (the first 65 lines ending with a space) followed by a newline
^I^I^I\2	On the second line, insert three tabs then saved pattern #2
/	End replacement pattern

Line two:

%g/	Over the whole file, only operate on lines with this pattern
^^I^I^I	Lines starting with three tabs
.\{58\}/	After the tabs, only match lines that still have at least 58 characters (this guards against wrapping already wrapped lines when it's run repeatedly)
s/	Begin global substitute
^	Start at the beginning of the line
\(	Remember the result of the next match
.\{,55\}	Up to 55 characters
\) \(	End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\)	Pattern #2 includes everything to the end of the line
/	End of matched pattern; begin replacement pattern
\1^M	The first pattern (up to 55 chars) is one line
^I^I^I\2	Three tabs then the second pattern
/	End replacement pattern

Greedy and non-greedy brace matches

The real key is those curly-brace expressions, \{,65\} and \{58\} -- that's how you control how many characters vim will match and whether or not the match is "greedy". Here's how they work (thanks to Mikael for explaining).

The basic expression is {M,N} -- it means between M and N matches of whatever precedes it. (Vim requires that the first brace be escaped -- \{}. Escaping the second brace is optional.) So .{M,N} can match anything between M and N characters but "prefer" N, i.e. try to match as many as possible up to N. To make it "non-greedy" (match as few as possible, "preferring" M), use .{-M,N}

You can leave out M, N, or both; M defaults to 0 while N defaults to infinity. So {} is short for {0,∞} and is equivalent to *, while {-} means {-0,∞}, like a non-greedy version of *.

Given the string: one, two, three, four, five

,.\{},	matches , two, three, four,
,.\{-},	matches , two,
,.\{5,},	matches , two, three, four,
,.\{-5,},	matches , two, three,
,.\{,2},	matches nothing
,.\{,7},	matches , two,
,.\{5,7},	matches , three,

Of course, this syntax is purely for vim; regular expressions are unfortunately different in sed, perl and every other program. Here's a fun table of regexp terms in various programs.

XFCE dukes it out with Gnome; users lose

The mixer came up fine -- but after I exited, I discovered that my xchat had gone all wonky. None of my normal key bindings worked, my cursor was blinking, and the fonts used for tabs was about half its normal size. Over in my Firefox window, key bindings were also affected.

I've seen this sort of thing happen before with Gnome apps, and had found a way to solve it using gconf-editor. That app was not installed, so I installed it and discovered that it didn't help.

I tried killing the running gconfd-2, removing .gconf/ and .gconfd/ from my home directory, then removing the four gnome directories (.gnome/, .gnome2/, .gnome2_private/, and .gnome_private/). Nothing helped xchat (though Firefox did return to normal).

After much flailing and annoying people by restarting xchat repeatedly, it turned out the problem was that xfce-mixer had started a daemon called xfce-mcs-manager, which is like gconf, only different. Like gconf, it mucks with settings of all running gtk programs without asking first. It runs simultaneously with gconf, but overrides gconf, which in turn overrides the values set in ~/.gtkrc-2.0.

Killing xfce-mcs-manager caused my running xchat to revert to its normal settings.

... Well, *almost* revert. A few key bindings didn't get reset, as I discovered when I hit a ctrl-W to erase the last word and found myself disconnected from the channel. Another xchat restart, with xfce-mcs-manager not running, fixed that.

Aside from the ever-present issue of "Where do I look when some unfriendly program decides to change the settings in running applications?" (which begs the question, "What genius thought it would be a good idea to give any random app like a volume control the power to change settings in every other gtk application currently running on the system? And do they have their medications adjusted better now?") there's another reason this is interesting.

See, if an arbitrary app like xfce-mcs-manager can send a message to xchat to change key bindings like ctrl-W ... then maybe I could write a program that could send a similar message telling xchat to cancel those compiled-in bindings like ctrl-F and ctrl-L, ones that it doesn't allow the user to change. If I could get something like that working, I could use a standard xchat -- I'd no longer need to patch the source and build my own.

Bug fixes? Why would we bother to ship bug fixes?

I've been running on Ubuntu's latest, "Gutsy gibbon", for maybe a month now. Like any release, it has its problems that I've needed to work around. Like many distros, these problems won't be fixed before the next release. But unlike other distros, it's not just lack of developer time; it turns out Ubuntu's developers point to an official policy as a reason not to fix bugs.

Take the case of the aumix bug. Aumix just plain doesn't work in gutsy. It prints, "aumix: SOUND_MIXER_READ_DEVMASK" and exits.

This turns out to be some error in the way it was compiled. If you apt-get the official ubuntu sources, build the package and install it yourself, it works fine. So somehow they got a glitch during the process of building it, and produced a bad binary.

(Minor digression -- does that make this a GPL violation? Shipping sources that don't match the distributed binary? No telling what sources were used to produce the binary in Gutsy. Not that anyone would actually want the sources for the broken aumix, of course.)

It's an easy fix, right? Just rebuild the binary from the source in the repository, and push it to the servers.

Apparently not. A few days ago, Henrik Nilsen Omma wrote in the bug:

This bug was nominated for Gutsy but does currently not qualify for a 7.10 stable release update (SRU) and the nomination is therefore declined. According the the SRU policy, the fix should already be deployed and tested in the current development version before an update to the stable releases will be considered. [ ... ] See: https://wiki.ubuntu.com/StableReleaseUpdates.

Of course, I clicked on the link to receive enlightenment. Ubuntu's Stable Release page explains

Users of the official release, in contrast, expect a high degree of stability. They use their Ubuntu system for their day-to-day work, and problems they experience with it can be extremely disruptive. Many of them are less experienced with Ubuntu and with Linux, and expect a reliable system which does not require their intervention.

Stable release updates will, in general, only be issued in order to fix high-impact bugs. Examples of such bugs include:

• Bugs which may, under realistic circumstances, directly cause a security vulnerability
• Bugs which represent severe regressions from the previous release of Ubuntu
• Bugs which may, under realistic circumstances, directly cause a loss of user data

Clearly aumix isn't a security vulnerability or a loss of user data. But I could make a good argument that a package that doesn't work ... ever ... for anyone ... constitutes a severe regression from the previous version of that package.

Ubuntu apparently thinks that users get used to packages not working, and grow to like it. I guess that if you actually fixed packages that you broke, that would be disruptive to users of the stable release.

I'm trying to picture these Ubuntu target users, who embrace regressions and get upset when something that doesn't work at all gets fixed so that it works as it did in past releases. I can't say I've ever actually met a user like that myself. But evidently the Ubuntu Updates Team knows better.

Update: I just have to pass along Dave's comment: "When an organization gets to the point where it spends more energy on institutional processes for justifying not fixing something than on just fixing it -- it's over."

Update: Carla Schroder has also written about this.

More Tips on International Input

The biggest problem was the terminal. I was using urxvt, but it turns out that urxvt won't let me type any nonascii characters. It just ignores my multi-key sequences, or prints a space instead of the character I wanted. I have no idea why, but switching to plain ol' xterm solved that problem. Of course, I had to make sure that I was using a font that supported the characters I wanted (ISO 8859-1 or 8859-15 or something similar), which leaves out my favorite terminal font (Schumacher Clean bold), but Bitstream Vera Sans Mono bold is almost as readable.

Of course, it's important to have your locale variables set appropriately. There are several locale variables:

LC_CTYPE

Which encodings to use for typing and displaying characters.

LC_MESSAGES

Which translations to use, in programs that offer them.

LC_COLLATE

How to sort alphabetically (this one also affects whether ls groups capitalized filenames first).

LC_ALL

Overrides any of the others.

LANG

The default, in case none of the other variables is set.

Once I switched to xterm, I was able to set either LANG or LC_CTYPE to either en_US.UTF-8 or en_US.ISO-8859-1. I set LC_COLLATE and LANG or LC_MESSAGES to C, so that I get the default (usually US) translations for programs and so that ls groups all the capitalized files first.

Along the way, I learned about yet another way to type accented characters.

setxkbmap -model pc104 -layout us -variant intl

Even faster, the international layout also offers shortcuts to many common characters with the "AltGr" key, which I'd heard about for years but never knew how to enable. AltGr is the right alt key, and typing, say, AltGr followed by n gives an ñ. You can see a full map at Wikipedia (AltGr characters are blue, quote prefixes are red).

To get back to a US non-international layout:

setxkbmap -model pc104 -layout us

Of course, these aren't the only keyboard layouts to choose from -- there are lots, plus you can define your own. And I was going to write a little bit about that, except it turns out they've changed it all around again since I last did that two years ago (don't you love the digital world?). So that will have to wait for another time.

But the place to start exploring is /usr/share/X11/xkb. The file symbols/us contains the definitions for those US keyboards, and I believe it's included via the files in the rules directory, probably rules/base, base.xml and base.lst. From there you're on your own. But the standard layouts probably follow the ones in the Wikipedia article on keyboard layouts

Backing up a file system

Linux offers lots of ways to copy filesystems. I've used tar in the past, with a command like (starting in /gutsy): tar --one-file-system -cf - / | tar xvf - > /tmp/backup.out but cp seemed like an easier way, so I want to try it.

I mounted my freshly made backup partition as /gutsy and started a cp -ax /* /gutsy (-a does the right thing for permissions, owner and group, and file type; -x tells it to stay on the original filesystem). Count to ten, then check what's getting copied. Whoops! It clearly wasn't staying on the original filesystem.

It turned out my mistake was that /*. Pretty obvious in hindsight what cp was doing: for each entry in / it did a cp -ax, staying on the filesystem for that entry, not on the filesystem for /. So /home, /boot, /proc, etc. all got copied. The solution was to remove the *: cp -ax / /gutsy.

But it wasn't quite that simple. It looked like it was working -- a long wait, then cp finished and I had what looked like a nice filesystem backup. I adjusted /gutsy/etc/fstab so that it would point to the right root, set up a grub entry, and rebooted. Disaster! The boot hung right after Attached scsi generic sg0 type 0 with no indication of what was wrong.

Rebooting into the old partition told me that what's supposed to happen next is: * Setting preliminary keymap... But the crucial error message was actually several lines earlier: Warning: unable to open an initial console. It hadn't been able to open /dev/console.

Now, in the newly copied filesystem, there was no /dev/console: in fact, /dev was empty. Nothing had been copied because /dev is a virtual file system, created by udev.

But it turns out that the boot process needs some static devices in /dev, before udev has created anything. Of course, once udev's virtual filesystem has been mounted on /dev, you can no longer read whatever was in /dev on the root partition in order to copy it somewhere else. But udev nicely gives you access to it, in /dev/.static/dev. So what I needed to do to get my new partition booting was: cp -ax /dev/.static/dev/ /gutsy/dev/ With that done, I was able to boot into my new filesystem and upgrade to Gutsy.

Typing accented characters (for ignorant 'murricans)

For the online Spanish class I've been taking, so far I've been able to manage without having to type characters like ñ or á. Usually, if I need one I can find it in one of the class examples, copy it, and paste it wherever I need it. But obviously that would be tedious if I needed to type much.

I hacked up a quickie workaround: a python script that shows a set of buttons, one for each accented character I'm likely to need. Clicking a button copies that character to the clipboard, so I can now paste via mouse middleclick or ctrl-V. (I'm sure that sounds pathetic to those of you who type accented characters every day, but it's not something most US English speakers need to do. And besides, now I know how to access the X clipboard from Python-GTK -- hooray for learning new things from procrastination projects!)

Anyway, Mikael Magnusson took pity on me and explained in simple language how to use the X "Multi key" to type these characters the right way (well, a right way, anyway). Since all the online instructions I've seen have been rather complicated, here are the simple instructions for any of my fellow US monolingists who'd like to expand their horizons:

First, choose a key for the "Multi key" that you're not using for anything else. A lot of people use one of the Alt or Windows keys, but I use both of those already. What I don't use is the Menu key (that little key down by the right Ctrl key, at least on my keyboard) since not many Linux apps support it anyway.

Find the keycode for that key, by firing up xev and typing the key. For my Menu key, the keycode is 117.

Now type:

xmodmap -e "keycode 117 = Multi_key"

Now you're ready to type a sequence like: [Menu] ~ n to type an n-tilde, [Menu] ' a for an accented a, or [menu] ? ? for the upside-down question mark, in any app that supports those characters.

Of course, you don't want to type that xmodmap command every time you log in, so to make it permanent, put this in your .Xmodmap (you're on your own for figuring out whether your X environment reads .Xmodmap automatically or whether you need to tell it to run xmodmap .Xmodmap when X starts up):

keycode 117 = Multi_key

I have one final useful international input tidbit to offer: how to type Unicode characters by number. Hold ctrl+shift+U, then release U but keep holding the other two while you type a numeric sequence. (This may only work in gtk apps.) For instance, try this: hold down ctrl and shift, then type: u 2 6 6 c. Cool, huh? You can use the "gucharmap" program to find other neat sequences (hint: View->By Unicode Block otherwise you'll never find anything).

Now it's time to check the turkey. Have a good day, everyone!

Proposal: A Simpler Linux Installer

Which brings me to today's question: Why do Linux distros have installers at all?

That probably sounds like a silly question. Of course you need an installer to get the system onto your disk ... don't you?

Well, yes and no. You could make it a lot simpler than anyone currently does.

What if you distributed a Linux distro as a filesystem image? Make it tar, zip, CD iso or whatever format you like -- but provide the user with a tree of files that, when copied onto a hard drive, can boot as a running Linux.

Set up this minimal installation filesystem so that when you boot into it, you get a commandline (X hasn't been configured yet) and a set of scripts that make it easy to go the rest of the way. From your running minimal system, you can configure X, set up networking, install more programs from the distro repositories (or even from a CD image), and do all the rest of the machine-specific configuration that an installer does.

The key is that this is all happening from a running system, not from some cobbled-together installer kernel or live CD.

If you have a problem with any step, you still have a basic running system, and tools to fix the problem. You avoid the usual loop:

• Run installer
• Spend 20 minutes answering questions
• Spend 45 minutes waiting for installer
• Discover it failed
• Start over with slightly different parameters

If your X configuration fails, try running X configuration again -- no need to do another install from the beginning. If it doesn't see your network card -- ditto. Since this debugging all happens from a normal running Linux, you can use the normal Linux tools you're used to, not some busybox-based installer.

This model would be much more hardware agnostic than current installers:

• You can install on systems with weirdo graphics cards;
• You can install on systems that need special drivers for the network card or other hardware;
• You can install on systems with no CDROM or an external CDROM;
• You can install even if you don't have access to a CD burner.

Another advantage is that it makes it very easy to make a customized version of your distro: just take the basic system image, change the part that needs changing and tar it up again.

Some distros have gone a little way with this, with an installer that gives you a starter system, then scripts to download the rest -- but I've never seen one that made the minimal system available as a filesystem image, with easy instructions on going to the next step.

What about the people who aren't already running Linux or aren't comfortable writing a filesystem image to a partition?

No problem. They get a CD image with a very simple installer -- it handles the partitioning, copies the minimal install to the partition, and updates grub. It's as prone to hardware compatibility issues as any installer (though far less so than a live CD is) but it's still better than the current model, because it won't be trying to configure hardware until the user reboots into a working minimal system.

Of course, Live CDs are still cool -- on machines where they actually work -- for showing Linux to people not ready to commit to an install. But don't tie your installer to that. Give people a simpler way to install, one that's fast and straightforward and hardware agnostic and easy to understand or customize.

The tarball installer. An idea whose time has come. Now if I could just persuade the distros to offer it.

Update: a couple of people told me about Dynebolic, a distro that apparently does just that -- you install by copying a directory on the CD onto your partition, or rsyncing from their site. Nice!

Tried bash, went back to tcsh

Back in July, I solved the last blocker that had been keeping me off bash, so I put some effort into migrating all my .cshrc settings into a .bashrc so I could give bash a fair shot. It almost won; but after four months, I've switched back to tcsh, due mostly to a single niggling bash bug that I just can't seem to solve. After all these years, the crucial difference is still history. Specifically, history amnesia: bash has an annoying habit of forgetting history commands just when I most want them back.

Say I type some longish command. After it runs, I hit return a couple of times, wait a while, do a couple of other things, then decide I want to call that command back from history so I can run something similar, maybe with the filename changed or a different flag. I ctrl-P or up-arrow ... and the command isn't there!

If I type history at this point, I'll see most of my command history ... with an empty line in place of the line I was hoping to repeat. The command is gone. My only option is to remember what I typed, and type it all again.

Nobody seems to know why this happens, and it's sporadic, doesn't happen every time. Friends have been able to reproduce it, so it's not just me or my weird settings. It drives me batty. It wouldn't be so bad except it always seems to happen on the tricky commands that I really didn't want to retype.

It's too bad, because otherwise I had bash nicely whipped into shape, and it does have some advantages over tcsh. Some of the tradeoffs:

tcsh wins

• Totally reliable history: commands never disappear.
• History tab completion: typing !a<TAB> expands to the last command that started with a. In bash, I have to type !a:p to see the command, then !! to execute it.
• When I tab-complete a file and there are multiple matches, tcsh shows them, or at least beeps (depending on configuration). In bash I have to hit a second tab just in case there might be matches.
• When I tab-complete a directory, tcsh adds the / automatically. (Arguable. I find I want the / roughly 3/4 of the time.)
• tcsh doesn't drop remote connections if I suspend (with ~^Z). bash drops me within a minute or two, regardless of settings like $TMOUT. Bash users tell me I could solve this by using screen, but that seems like a heavyweight workaround when tcsh "just works".
• tcsh sees $REMOTEHOST and $DISPLAY automatically when I ssh. bash doesn't: ssh -X helps, but I still need some tricky code in .bash_profile.
• aliases can have arguments, e.g. alias llth 'ls -laFt \!* | head' In bash these have to be functions, which means they don't show up typing "which" or "alias".
• Prompt settings are more flexible -- there are options like %B for bold. In bash you have to get the terminal type and use the ansi color escape sequances, which don't include bold.
• Easier command editing setup -- behaviors like getting word-erase to stop at punctuation don't involve chasing through multiple semi-documented programs, and the answer doesn't vary with version.
• Documentation -- tcsh's is mostly in man tcsh, bash's is scattered all over man pages for various helper programs. And it's hard to google for bash help because "bash" as a keyword gets you lots of stuff not related to shells.

Of course, you bash users, set me straight if I missed out on some bash options that would have solved some of these problems. And especially if you have a clue about the evil disappearing history commands!

bash wins

• You don't have to rehash every time you add a program or change your path. That's a real annoyance of tcsh, and I could understand a person used to bash rejecting tcsh on this alone.
  Update: Holger Weiß has written a tcsh patch to fix this, and it has been accepted as of November 2009. Looking forward to the next version!
• History remembers entire multi-line commands, and shows them with semicolons when you arrow back through history. Very nice. tcsh only remembers the first line and you have to retype the rest.
• Functions: although I don't like having to use them instead of aliases, they're certainly powerful and nice to have.

Of course, bash and tcsh aren't the only shells around. From what I hear, zsh blends the good features of bash and tcsh. One of these days I'll try it and see.

Custom beep sounds

Today someone pointed me toward a Gentoo wiki page that led me to Fancy Beeper Daemon. This is a small kernel module that adds a device, /dev/beep, which outputs a byte every time there's a beep. You can write a very simple daemon (several samples in Python are included with the module) which reads /dev/beep and plays the sound of your choice.

I downloaded beep-2.6.15+.tar.gz, but it needed one tweak to build it under 2.6.23-rc3: I needed to add

#include <linux/sched.h>

/dev/beep is created with owner and group root, and readable/ writable only by owner and group, so to test it I had to chmod 666 /dev/beep to run the daemon. I fixed that by making a file in /etc/udev/rules.d called 41-beep.rules, containing:

KERNEL=="beep", GROUP="audio"

Finally, based on the nice samples that came with the module, I wrote my own very simple Python daemon, playbeepd, that uses aplay.

Lots of fun! I don't know how much I'll use it, but it's good to have the option of custom beep sounds.

Widescreen Monitor, Intel Graphics on Ubuntu

We got the lovely new monitor home, plugged it in, configured X and discovered that the screen showed severe vertical banding. It was beautiful at low resolutions, but whenever we went to the monitor's maximum resolution of 1680x1050, the bands appeared.

After lots of testing, we tentatively pinned the problem down to the motherboard. It turns out ancient machines with 1x AGP motherboards can't drive that many pixels properly, even if the video card is up to the job. Who knew?

Off we trooped to check out new computers. We'd been hinting for quite some time that it might be about time for a new machine, and Mom was ready to take the plunge (especially if it meant not having to return that beautiful monitor).

We were hoping to find something with a relatively efficient Intel Core 2 processor and Intel integrated graphics: I've been told the Intel graphics chip works well with Linux using open source drivers. (Mom, being a person of good taste, prefers Linux, and none of us wanted to wrestle with the proprietary nvidia drivers). We found a likely machine at PC Club. They were even willing to knock $60 off the price since she didn't want Windows.

But that raised a new problem. During our fiddling with her old machine, we'd tried burning a Xubuntu CD, to see if the banding problem was due to the old XFree86 she was running. Installing it hadn't worked: her CD burner claimed it burned correctly, but the resulting CD had errors and didn't pass verification. So we needed a CD burned. We asked PC Club when buying the computer whether we might burn the ISO to CD, but apparently that counts as a "data transfer" and their minimum data transfer charge is $80. A bit much.

No problem -- a friend was coming over for dinner that night, and he was kind enough to bring his Mac laptop ... and after a half hour of fiddling, we determined that his burner didn't work either (it gave a checksum error before starting the burn). He'd never tried burning a CD on that laptop.

What about Kinko's? They have lots of data services, right? Maybe they can burn an ISO. So we stopped at Kinko's after dinner. They, of course, had never heard of an ISO image and had no idea how to burn one on their Windows box. Fearing getting a disk with a filesystem containing one file named "xubuntu-7.04-alternate-i386.iso", we asked if they had a mac, since we knew how to burn an ISO there. They did, though they said sometimes the CD burner was flaky. We decided to take the risk.

Burning an ISO on a mac isn't straightforward -- you have to do things in exactly the right order. It took some fast talking to persuade them of the steps ("No, it really won't work if you insert the blank CD first. Yes, we're quite sure") and we had to wait a long time for Kinko's antivirus software to decide that Xubuntu wasn't malware, but 45 minutes and $10 later, we had a disc.

And it worked! We first set up the machine in the living room, away from the network, so we had to kill aptitude update when the install hung installing "xubuntu-desktop" at 85% (thank goodness for alternate consoles on ctl-alt-F2) but otherwise the install went just fine. We rebooted, and Xubuntu came up ... at 1280x1024, totally wrong. Fiddling with the resolution in xorg.conf didn't help; trying to autodetect the monitor with dpkg-reconfigure xorg crashed the machine and we had to power cycle.

Back to the web ... turns out that Ubuntu "Feisty" ships with a bad Intel driver. Lots of people have hit the problem, and we found a few elaborate workarounds involving installing X drivers from various places, but nothing simple. Well, we hadn't come this far to take all the hardware back now.

First we moved the machine into the computer room, hooked up networking and reinstalled xubuntu with a full network, just in case. The resolution was still wrong. Then, with Dave in the living room calling out steps off a web page he'd found, we began the long workaround process.

"First," Dave suggested, reading, "check the version of xserver-xorg-video-intel. Let's make sure we're starting with the same version this guy is."

dpkg -l xserver-xorg-video-intel ... "Uh, it isn't installed," I reported. I tried installing it. "It wants to remove xserver-xorg-video-i810." Hmm! We decided we'd better do it, since the rest of the instructions depended on having the intel, not i810, driver.

And that was all it needed! The intel driver autodetected the monitor and worked fine at 1680x1050.

So forget the elaborate instructions for trying X drivers from various sources. The problem was that xubuntu installed the wrong driver: the i810 driver instead of the more generic intel driver. (Apparently that bug is fixed for the next Ubuntu release.)

With that fix, it was only a few more minutes before Mom was happily using her new system, widescreen monitor and all.

The Importance of Being ESSID (simple Linux wi-fi troubleshooting)

I've seen this both on networks which require a WEP key and those that don't, and on nets where my older Prism2/Orinoco based card will connect fine.

Apparently, the root of the problem is that the Prism54 is more sensitive than the Prism2: it can see more nearby networks. The Prism2 (with the orinoco_cs driver) only sees the strongest network, and gloms onto it. But the Prism54 chooses an access point according to arcane wisdom only known to the driver developers. So even if you're sitting right next to your access point and the next one is half a block away and almost out of range, you need to specify which one you want. How do you do that? Use the ESSID.

Every wireless network has a short identifier called the ESSID to distinguish it from other nearby networks. You can list all the access points the card sees with:

iwlist eth0 scan

You'll probably see a lot of ESSIDs like "linksys" or "default" or "OEM" -- the default values on typical low-cost consumer access points. Of course, you can set your own access point's ESSID to anything you want.

So what if you think your wireless card should be working, but it can't connect anywhere? Check the ESSID first. Start with iwconfig:

iwconfig eth0

First, change it temporarily to make sure you're choosing the right ESSID:

iwconfig eth0 essid MyESSID

If your accesspoint requires a key, add key nnnnnnnnnn to the end of that line. Then see if your network is working.

If that works, you can make it permanent. On Debian-derived distros, just add lines to the entry in /etc/network/interfaces:

wireless-essid MyESSID
wireless-key nnnnnnnnnn

Some older howtos may suggest an interfaces line that looks like this:
up iwconfig eth0 essid MyESSID
Don't get sucked in. This "up" syntax used to work (along with pre-up and post-up), but although man interfaces still mentions it, it doesn't work reliably in modern releases. Use wireless-essid instead.

Of course, you can also use a gooey tool like gnome-network-manager to set the essid and key. Not being a gnome user, some time ago I hacked up the beginnings of a standalone Python GTK tool to configure networks. During this week's wi-fi fiddlings, I dug it out and blew some of the dust off: wifi-picker.

You can choose from a list of known networks (including both essid and key) set up in your own configuration file, or from a list of essids currently visible to the card, and (assuming you run it as root) it can then set the essid and key to whatever you choose. For networks I use often, I prefer to set up a long-term network scheme, but it's fun to have something I can run once to show me the visible networks then let me set essid and key.

Powertop BOF at Linuxworld 2007

The best thing at Linuxworld was the Powertop BOF, despite the fact that it ended up stuck in a room with no projector. The presenter, Arjan van de Ven, coped well with the setback and managed just fine.

The main goal of Powertop is to find applications that are polling or otherwise waking the CPU unnecessarily, draining power when they don't need to. Most of the BOF focused on "stupid stuff": programs that wake up too often for no reason. Some examples he gave (many of these will be fixed in upcoming versions of the software):

• gnome-screensaver checked every 2 sec to see if the mouse moved (rather than using the X notification for mouse move);
• gnome volume checked 10 times a second whether the volume has changed;
• gnome-clock woke up once a second to see if the minute had rolled over, rather than checking once a minute;
• firefox in an ssl layer polled 10 times a second in case there was a notification;
• the gnome file monitor woke up 40 times a second to check a queue even if there was nothing in the queue;
• evolution woke up 10 times a second;
• the fedora desktop checked 10 times a second for a smartcard;
• gksu used a 10000x/sec loop (he figures someone mistook milliseconds/microseconds: this alone used up 45 min on one battery test run)
• Adobe's closed-source flash browser plugin woke up 2.5 times a second, and acroread had similar problems (this has been reported to Adobe but it's not clear if a fix is coming any time soon).

And that's all just the desktop stuff, without getting into other polling culprits like hal and the kernel's USB system. The kernel itself is often a significant culprit: until recently, kernels woke up once a millisecond whether they needed to or not. With the recent "tickless" option that appeared in the most recent kernel, 2.6.22, the CPU won't wake up unless it needs to.

A KDE user asked if the KDE desktop was similarly bad. The answer was yes, with a caveat: Arjan said he gave a presentation a while back to a group of KDE developers, and halfway through, one of the developers interrupted him when he pointed out a problem to say "That's not true any more -- I just checked in a fix while you were talking." It's nice to hear that at least some developers care about this stuff! Arjan said most developers responded very well to patches he'd contributed to fix the polling issues.

(Of course, those of us who use lightweight window managers like openbox or fvwm have already cut out most of these gnome and kde power-suckers. The browser issues were the only ones that applied to me, and I certainly do notice firefox' polling: when the laptop gets slow, firefox is almost always the culprit, and killing it usually brings performance back.)

As for hardware, he mentioned that some linux LCD drivers don't really dim the backlight when you reduce brightness -- they just make all the pixels darker. (I've been making a point of dimming my screen when running off batteries; time to use that Kill-A-Watt and find out if it actually matters!) Wireless cards like the ipw100 use a lot of power even when not transmitting -- sometimes even more than when they're transmitting -- so turning them off can be a big help. Using a USB mouse can cut as much as half an hour off a battery. The 2.6.23 kernel has lots of new USB power saving code, which should help. Many devices have activity every millisecond, so there's lots of room to improve.

Another issue is that even if you get rid of the 10x/sec misbehavers, some applications really do need to wake up every second or so. That's not so bad by itself, but if you have lots of daemons all waking up at different times, you end up with a CPU that never gets to sleep.

The solution is to synchronize them by rounding the wakeup times to the nearest second, so that they all wake up at about the same time, and the CPU can deal with them all then go back to sleep. But there's a trick: each machine has to round to a different value. You don't want every networking application on every machine across the internet all waking up at once -- that's a good way to flood your network servers. Arjan's phrase: "You don't want to round the entire internet" [to the same value].

The solution is a new routine in glib: timeout_add_seconds. It takes a hash of the hostname (and maybe other values) and uses that to decide where to round timeouts for the current machine. If you write programs that wake up on a regular basis, check it out. In the kernel, round_jiffies does something similar.

After all the theory, we were treated to a demo of powertop in action. Not surprisingly, it looks a bit like top. High on the screen is summary information telling you how much time your CPU is spending in the various sleep states. Getting into the deeper sleep states is generally best, but it's not quite that simple: if you're only getting there for short periods, it takes longer and uses more power to get back to a running state than it would from higher sleep states.

Below that is the list of culprits: who is waking your CPU up most often? This updates every few seconds, much like the top program. Some of it's clear (names of programs or library routines); other lines are more obscure if you're not a kernel hacker, but I'm sure they can all be tracked down.

At the bottom of the screen is a geat feature: a short hint telling you how you could eliminate the current top offender (e.g. kill the process that's polling). Not only that, but in many cases powertop will do it for you at the touch of a key. Very nice! You can try disabling things and see right away whether it helped.

Arjan stepped through killing several processes and showing the power saving benefits of each one. (I couldn't help but notice, when he was done, that the remaining top offender, right above nautilus, was gnome-power-manager. Oh, the irony!)

It's all very nifty and I'm looking forward to trying it myself. Unfortunately, I can't do that on the laptop where I really care about battery life. Powertop requires a kernel API that went in with the "tickless" option, meaning it's in 2.6.22 (and I believe it's available as a patch for 2.6.21). My laptop is stuck back on 2.6.18 because of an IRQ handling bug (bug 7264). Powertop also requires ACPI, which I have to disable because of an infinite loop in kacpid (bug 8274, Ubuntu bug 75174). It's frustrating to have great performance tools like powertop available, yet not be able to use them because of kernel regressions. But at least I can experiment with it on my desktop machine.

Linuxworld 2007

There wasn't much of interest on the exhibit floor. Lots of small companies doing virtualization or sysadmin tools. The usual assortment of publishers. A few big companies, but fewer than in past years. Not much swag. Dave commented that there was a much higher "bunny quotient" this year than last (lots of perky booth bunnies, very few knowledgeable people working the floor). The ratio of Linux to Windows in the big-company booths was much lower than last year, especially at AMD and HP, who both had far more Windows machines visible than Linux ones.

The most interesting new hardware was the Palm Foleo. It looks like a very thin 10-inch screen laptop, much like my own Vaio only much thinner and lighter, with a full QWERTY keyboard with a good feel to it. The booth staff weren't very technical, but apparently it sports a 300MHz Intel processor, built-in wi-fi and bluetooth, a resolution a hair under 1024x768 (I didn't write down the exact numbers and their literature doesn't say), a claimed battery life of 5 hours, and runs a Linux from Wind River. The booth rep I talked to said it would run regular Linux apps once they were recompiled for the processor, but he didn't seem very technical and I doubt it runs X, so I'm not sure I believe that. For a claimed price of around $400 it looks potentially quite interesting.

Their glossy handout says it has VGA out and can display PowerPoint presentations, which was interesting since the only powerpoint reader I know of on Linux is OpenOffice and I don't see that running on 300MHz (considering how slow it is on my P3 700). Apparently they're using Documents To Go from DataVis, a PalmOS app.

Aside from that there wasn't much of interest going on. They split up the "Dot Org Pavilion" this year so not all the community groups were in the same place, which was a bummer -- usually that's where all the interesting booths are (local LUGs, FSF, EFF, Debian, Ubuntu and groups like that: no Mozilla booth this time around). But this year the dotorgs were too spread out to offer a good hangout spot. It didn't look like there was much of interest at the conference either: this year they gave us Exhibit Hall pass attendees a free ticket to attend one of the paid talks, and I couldn't find one on the day we attended that looked interesting enough to bother.

However, that changed at the end of the day with the BOF sessions. The Intel Powertop BOF was an easy choice -- I've been curious about Powertop ever since it was announced, and was eager to hear more about it from one of the developers. The BOF didn't disappoint, though the room did: they didn't even provide a projector (!), so we all had to cluster around the presenter's laptop when he wanted to show something. Too bad! but it didn't keep the BOF from being full of interesting information. I'll split that off into a separate article.

Turn off gtk cursor blink

Xchat is a gtk application, of course. There are lots and lots of pages on the web telling developers about gtk's gtk-cursor-blink property (and related properties like gtk-cursor-blink-timeout) and what they do in at a library level, so it's clearly possible. But I found nothing about where a user should set these properties to make gtk find them.

Here's the answer. Add to $HOME/.gtkrc-2.0 (or any other file loaded by $HOME/.gtkrc-2.0):

gtk-cursor-blink = 0

I had to restart X (maybe shutting down all gtk apps would have been enough) before I saw any effect.

While I was searching, I did find a nice page on How to disable blinking cursors in lots of different apps. Its gtk section didn't seem to do anything here (maybe it only works if a gnome desktop is running), but it's a good page nontheless, full of useful advice for turning off cursor blink in other programs.

Adventures with bash's word erase

But I always come up against one problem right away: word erase (Control-W). For those who don't use ^W, suppose I type something like:

% ls /backups/images/trips/arizona/2007

Then the other day I was chatting about this issue with a friend who uses zsh for that reason (zsh is much more flexible at defining key bindings) and someone asked, "Is that like Meta-Delete?"

It turned out that Alt-Backspace (like many Linux applications, bash calls the Alt key "Meta", and Linux often confuses Delete and Backspace) did exactly what I wanted. Very promising!

But Alt-Backspace is not easy to type, since it's not reachable from the "home" typing position. What I needed, now that I knew bash and readline had the function, was a way to bind it to ^W.

Bash's binding syntax is documented, though the functions available don't seem to be. But bind -p | grep word gave me some useful information. It seems that \C-w was bound to "unix-word-rubout" (that was the one I didn't want) whereas "\e\C-?" was bound to "backward-kill-word". ("\e\C-?" is an obscure way of saying Meta-DEL: \e is escape, and apparently bash, like emacs, treats ESC followed by a key as the same as pressing Alt and the key simultaneously. And Control-question-mark is the Delete character in ASCII.)

So my task was to bind \C-w to backward-kill-word. It looked like this ought to work:

bind '\C-w:backward-kill-word'

... Except it didn't. bind -p | grep w showed that C-W was still bound to "unix-word-rubout".

It turned out that it was the terminal (stty) settings causing the problem: when the terminal's werase (word erase) character is set, readline hardwires that character to do unix-word-rubout and ignores any attempts to change it.

I found the answer in a bash bug report. The stty business was introduced in readline 5.0, but due to complaints, 5.1 was slated to add a way to override the stty settings. And happily, I had 5.2! So what was this new way override method? The posting gave no hint, but eventually I found it.

Put in your .inputrc:

set bind-tty-special-chars Off

And finally my word erase worked properly and I could use bash!

A Quartet of Workarounds

But Feisty is cutting edge in other ways, so it's been a week of workarounds, in two areas: Firefox and the kernel. I'll start with Firefox.

Firefox crashes playing flash

First, the way Ubuntu's Firefox crashes when running Flash. I run flashblock, fortunately, so I've been able to browse the web just fine as long as I don't click on a flashblock button. But I like being able to view the occasional youtube video, and flash 7 has worked fine for me on every distro except Ubuntu. I first saw the problem on Edgy, and upgrading to Feisty didn't cure the problem.

But it wasn't their Firefox build; my own "kitfox" firefox build crashed as well. And it wasn't their flash installation; I've never had any luck with either their adobe flash installer nor their opensource libswfdec, so I'm running the same old flash 7 plug-in that I've used all along for other distros.

To find out what was really happening, I ran Firefox from the commandline, then went to a flash page. It turned out it was triggering an X error:

The error was: 'BadMatch (invalid parameter attributes)'.
(Details: serial 104 error_code 8 request_code 145 minor_code 3)

That gave me something to search for. It turns out there's a longstanding Ubuntu bug, 14911 filed on this issue, with several workarounds. Setting the environment variable XLIB_SKIP_ARGB_VISUALS to 1 fixed the problem, but, reading farther in the bug, I saw that the real problem was that Ubuntu's installer had, for some strange reason, configured my X to use 16 bit color instead of 24. Apparently this is pretty common, and due to some bug involving X's and Mozilla's or Flash's handling of transparency, this causes flash to crash Mozilla.

So the solution is very simple. Edit /etc/X11/xorg.conf, look for the DefaultDepth line, and if it's 16, that's your problem. Change it to 24, restart X and see if flash works. It worked for me!

Eliminating Firefox's saved session pester dialog

While I was fiddling with Firefox, Dave started swearing. "Why does Firefox always make me go through this dialog about restoring the last session? Is there a way to turn that off?"

Sure enough, there's no exposed preference for this, so I poked around about:config, searched for browser and found browser.sessionstore.resume_from_crash. Doubleclick that line to change it to false and you'll have no more pesky dialog.

For more options related to session store, check the Mozillazine Session Restore page.

Kernel: runaway kacpid

Alas, having upgraded to Feisty expressly so that I could build cutting-edge programs like GIMP, I discovered that I couldn't build anything at all. Anything that uses heavy CPU for more than a minute or two triggers a kernel daemon, kacpid, to grab most of the CPU for itself. Being part of the kernel (even though it has a process ID), kacpi is unkillable, and prevents the machine from shutting down, so once this happens the only solution is to pull the power plug.

This has actually been a longstanding Ubuntu problem (bug 75174) but it used to be that disabling acpid would stop kacpid from running away, and with feisty, that no longer helps. The bug is also kernel.org bug 8274.

The Ubuntu bug suggested that disabling cpufreq solved it for one person. Apparently the only way to do that is to build a new kernel. There followed a long session of attempted kernel building. It was tricky because of course I couldn't build on the target machine (inability to build being the problem I was trying to solve), and even if I built on my desktop machine, a large rsync of the modules directory would trigger a runaway kacpi. In the end, building a standalone kernel with no modules was the only option.

But turning off cpufreq didn't help, nor did any of the other obvious acpi options. The only option which stops kacpid is to disable acpi altogether, and use apm. I'm sorry to lose hibernate, and temperature monitoring, but that appears to be my only option with modern kernels. Sigh.

Kernel: Hangs for 2 minutes at boot time initializing sound card

While Dave and I were madly trying to find a set of config options to build a 2.6.21 that would boot on a Vaio (he was helping out with his SR33 laptop, starting from a different set of config options) we both hit, at about the same time, an odd bug: partway through boot, the kernel would initialize the USB memory stick reader:

sd 0:0:0:0: Attached scsi removable disk sda
sd 0:0:0:0: Attached scsi generic sg0 type 0

We eventually narrowed the blame to the sound card (which is a Yamaha, using the ymfpci driver). And that was enough information for google to find this Linux Kernel Mailing List thread. Apparently the sound maintainer decided, for some reason, to make the ymfpci driver depend on an external firmware file ... and then didn't include the firmware file, nor is it included in the alsa-firmware package he references in that message. Lovely. I'm still a little puzzled about the timeout: the post does not explain why, if a firmware file isn't found on the disk, waiting for two minutes is likely to make one magically appear.

Apparently it will be fixed in 2.6.22, which isn't much help for anyone who's trying to run a kernel on any of the 2.6.21.* series in the meantime. (Isn't it a serious enough regression to fix in 2.6.21.*?) And he didn't suggest a workaround, except that alsa-firmware package which doesn't actually contain the firmware for that card. Looks like it's left to the user to make things work.

So here's what to do: it turns out that if you take a 2.6.21 kernel, and substitute the whole sound/pci/ymfpci directory from a 2.6.20 kernel source tree, it builds and boots just fine. And I'm off and running with a standalone apm kernel with no acpi; sound works, and I can finally build GIMP again.

So it's been quite a week of workarounds. You know, I used to argue with all those annoying "Linux is not ready for the desktop" people. But sometimes I feel like Linux usability is moving in the wrong direction. I try to imagine explaining to my mac-using friends why they should have to edit /etc/X11/xorg.conf because their distro set up a configuration that makes Firefox crash, or why they need to build a new kernel because the distributed ones all crash or hang ... I love Linux and don't regret using it, but I seem to need workarounds like this more often now than I did a few years ago. Sometimes it really does seem like the open source world is moving backward, not forward.

Openbox 3.4

Openbox impressed me right away. I'd tried it once before, a couple of years ago, when I found it a little inconsistent and immature. It's grown up a lot since then! It's still very fast and lightweight, but it has good focus handling, excellent window positioning, a good configuration window (obconf), and a wide variety of themes which are pretty but still don't take up too much of my limited screen space.

But more important, what it has is a very active and friendly community. I hit a couple of snags, mostly having to do with focus handling while switching desktops (the problem that drove me off icewm to fvwm), so I hopped onto the IRC channel and found myself chatting with the active developers, who told me that most of my problems had already been fixed in 3.4, and there were .deb files on the website for both of the distros I'm currently using. Indeed, that cured the problem; and when I later hit a more esoteric focus bug, the developers (particularly Dana Jansens) were all over it and fixed it that same day. Wow!

Since then I've been putting it through its paces. I have yet to see a window positioned badly in normal usage, and it handles several other problems I'd been seeing with fvwm, like focus handling when popping up dialogs (all those secondary GIMP Save-as dialogs that don't get focused when they appear). It's just as flexible as fvwm was when it comes to keyboard and mouse configuration, maybe even more so (plus it has lots of useful default bindings I might not have thought of, like mousewheel bindings to change desktops or "shade" a window).

I was going to stay out of theme configuration, because there were several pretty good installed themes already. But then in response to a half-joking question on my part of whether a particular theme came in blue, someone on the IRC channel made me a custom theme file -- and I couldn't resist tweaking it from there, and discovered that tweaking openbox themes is just as easy as fiddling with its other defaults.

I don't use transparency (I find it distracting), but my husband is addicted to transparent windows, so when I noticed on the web site that openbox handles transparency I pointed him there. (He's been using an old Afterstep, from back when it was still small and light, but it's been a constant battle getting it to build under newer gccs.) He reports that openbox handles transparency as well as afterstep did, so he's switched too.

I haven't looked at the openbox code yet, but based on how fast the developers add features and fix bugs, I bet it's clean, and I hope I can contribute at some point.

Anyway, great focus handling, great window positioning, fast, lightweight, super configurable, and best of all a friendly and helpful developer and user community. What more could you ask for in a window manager? I'm an openbox convert. Thanks, Dana, Mikachu and all the rest.

Etch: Blacklisting Modules, Udev, and Firewire Networking

Poking around dmesg revealed that it was related to the IEEE 1394 and the eth1394 module. It was firewire networking.

This laptop, being a Vaio, does have a built-in firewire interface (Sony calls it i.Link). The Etch installer, when it detected no network present, had noted that it was "possible, though unlikely" that I might want to use firewire instead, and asked whether to enable it. I declined.

Yet the installed system ended up with firewire networking not only installed, but taking the first network slot, ahead of any network cards. It didn't get in the way of functionality, but it was annoying and clutters the output whenever I type ifconfig -a. Probably took up a little extra boot time and system resources, too. I wanted it gone.

Easier said than done, as it turns out.

I could see two possible approaches.

1. Figure out who was setting it to eth1, and tell it to ignore the device instead.
2. Blacklist the kernel module, so it couldn't load at all.

I begain with approach 1. The obvious culprit, of course, was udev. (I had already ruled out hal, by removing it, rebooting and observing that the bogus eth0 was still there.) Poking around /etc/udev/rules.d revealed the file where the naming was happening: z25_persistent-net.rules.

It looks like all you have to do is comment out the two lines for the firewire device in that file. Don't believe it. Upon reboot, udev sees the firewire devices and says "Oops! persistent-net.rules doesn't have a rule for this device. I'd better add one!" and you end up with both your commented-out line, plus a brand new uncommented line. No help.

Where is that controlled? From another file, z45_persistent-net-generator.rules. So all you have to do is edit that file and comment out the lines, right?

Well, no. The firewire lines in that file merely tell udev how to add a comment when it updates z25_persistent-net.rules. It still updates the file, it just doesn't comment it as clearly.

There are some lines in z45_persistent-net-generator.rules whose comments say they're disabling particular devices, by adding a rule GOTO="persistent_net_generator_end". But adding that in the firewire device lines caused the boot process to hang. There may be a way to ignore a device from this file, but I haven't found it, nor any documentation on how this system works.

Defeated, I switched to approach 2: prevent the module from loading at all. I never expect to use firewire networking, so it's no loss. And indeed, there are lots of other modules loaded I'd like to blacklist, since they represent hardware this machine doesn't have. So it would be nice to learn how.

I had a vague memory of there having been a file with a name like /etc/modules.blacklist some time back in the Pliocene. But apparently no such file exists any more. I did find /etc/modprobe.d/blacklist, which looked promising; but the comment at the beginning of that file says

# This file lists modules which will not be loaded as the result of
# alias expsnsion, with the purpose of preventing the hotplug subsystem
# to load them. It does not affect autoloading of modules by the kernel.

I fiddled with several other approaches before Debian diva Erinn Clark found this helpful page. I created a file called /etc/modprobe.d/00local and added this line to it:

install eth1394 /bin/true

This /etc/modprobe.d/00local technique probably doesn't bear examining too closely. It has "hack" written all over it. But if that's the only way to blacklist problematic modules, I guess it's better than nothing.

Debian "Etch": a Sketch

The installer was the same as always, and the same as the Ubuntu installer. No surprises, although I do like the way Debian gives me a choice of system types to install (Basic desktop, Web server, etc. ... though why isn't "Development" an option?) compared to Ubuntu's "take the packages we give you and deal with it later" approach.

Otherwise, the install went very much like a typical Ubuntu install. I followed the usual procedures and workarounds so as not to overwrite the existing grub, to get around the Vaio hardware issues, etc. No big deal, and the install went smoothly.

The good

But the real surprise came on booting into the new system. Background: my Vaio SR-17 has a quirk (which regular readers will have heard about already): it has one PCMCIA slot, which is needed for either the external CDROM drive or a network card. This means that at any one time, you can have a network, or a CDROM, but not both. This tends to throw Debian-based installers into a tizzy -- you have to go through five or more screens (including timing out on DHCP even after you've told it that you have no network card) to persuade the installer that yes, you really don't have a network and it's okay to continue anyway.

That means that the first step after rebooting into the new system is always configuring the network card. In Ubuntu installs, this typically means either fiddling endlessly with entries in the System or Admin menus, or editing /etc/network/interfaces.

Anticipating a vi session, I booted into my new Etch and inserted the network card (a 3COM 3c59x which often confuses Ubuntu). Immediately, something began spinning in the upper taskbar. Curious, I waited, and in ten seconds or so a popup appeared informing me "You are now connected to the wired net."

And indeed I was! The network worked fine. Kudos to debian -- Etch is the first distro which has ever handled this automatically. (I still need to edit /etc/network/interfaces to set my static IP address -- network manager

Of course, since this was my laptop, the next most important feature is power management. Happily, both sleep and hibernate worked correctly, once I installed the hibernate package. That had been my biggest worry: Ubuntu was an early pioneer in getting ACPI and power management code working properly, but it looks like Debian has caught up.

The bad

I did see a couple of minor glitches.

First, I got a lot of system hangs in X. These turned out to be the usual dri problem on S3 video cards. It's a well known bug, and I wish distros would fix it!

I've also gotten at least one kernel OOPS, but I have a theory about what might be causing that. Time will tell whether it's a real problem.

It took a little googling to figure out the line I needed to add to /etc/apt/sources.list in order to install programs that weren't included on the CD. (Etch automatically adds lines for security updates, but not for getting new software). But fortunately, lots of other people have already asked this in a variety of forums. The answer is:

deb http://http.us.debian.org/debian etch main contrib non-free

My husband had suggested that Etch might be lighter weight than Ubuntu and less dependent on hal (which I always remove from my laptop, because its constant hardware polling makes noise and sucks power). But no: Etch installed hal, and any attempt to uninstall it takes with it the whole gnome desktop environment, plus network-manager (that's apparently that nice app that noticed my network card earlier) and rhythmbox. I don't actually use the gnome desktop or these other programs, but it would be nice to have the option of trying them when I want to check something out. So for now I've resorted to the temporary solution: mv /usr/sbin/hald /usr/sbin/hald-not

The ugly

Etch looks fairly nice, and I'm looking forward to exploring it. I'm mostly kidding about the "ugly". I did hit one minor bit of ugliness involving network devices which led me on a two-hour chase ... but I'll save that for its own article.

Feisty Fawn Versus Apache

In the last installment, I got the Visor driver working. My sitescooper process also requires that I have a local web server (long story), so I needed Apache. It was already there and running (curiously, Apache 1.3.34, not Apache 2), and it was no problem to point the DocumentRoot to the right place.

But when I tested my local site, I discovered that although I could see the text on my website, I couldn't see any of the images. Furthermore, if I right-clicked on any of those images and tried "View image", the link was pointing to the right place (http://localhost/images/foo.jpg). The file (/path/to/mysite/images/foo.jpg) existed with all the right permissions. What was going on?

/var/log/apache/error.log gave me the clue. When I was trying to view http://localhost/images/foo.jpg, apache was throwing this error:

 [error] [client 127.0.0.1] File does not exist: /usr/share/images/foo.jpg

Searching for usr/share/images in /etc/apache/httpd.conf gave the answer. It turns out that Ubuntu, in their infinite wisdom, has decided that no one would ever want a directory called images in their webspace. Instead, they set up an alias so that any reference to /images gets redirected to /usr/share/images.

WTF?

Anyway, the solution is to comment out that stanza of httpd.conf:

<IfModule mod_alias.c>
#    Alias /icons/ /usr/share/apache/icons/
#
#    <Directory /usr/share/apache/icons>
#         Options Indexes MultiViews
#         AllowOverride None
#         Order allow,deny
#         Allow from all
#    </Directory>
#
#    Alias /images/ /usr/share/images/
#
#    <Directory /usr/share/images>
#         Options MultiViews
#         AllowOverride None
#         Order allow,deny
#         Allow from all
#    </Directory>
</IfModule>

I suppose it's nice that they provided an example for how to use mod_alias. But at the cost of breaking any site that has directories named /images or /icons? Is it just me, or is that a bit crazy?

Feisty Fawn: The Adventure Continues, with the Visor Driver

When we left off, I had just found a workaround for my Feisty Fawn installer problems and had gotten the system up and running.

By now, it was late in the day, time for my daily Sitescooper run to grab some news to read on my Treo PDA. The process starts with making a backup (pilot-xfer -s). But pilot-xfer failed because it couldn't find the device, /dev/ttyUSB1. The system was seeing the device connection -- dmesg said

[ 1424.598770] usb 5-2.3: new full speed USB device using ehci_hcd and address 4
[ 1424.690951] usb 5-2.3: configuration #1 chosen from 1 choice

# Create /dev/pilot symlink for Palm Pilots
KERNEL=="ttyUSB*", ATTRS{product}=="Palm Handheld*|Handspring *|palmOne Handheld", \
             SYMLINK+="pilot"

But this time googling led me right to the bug, bug 108512. Turns out that for some reason (which no one has investigated yet), feisty doesn't autoload the visor module when you plug in a USB palm device the way other distros always have. The temporary workaround is sudo modprobe visor; the long-term workaround is to add visor to /etc/modules.

On the subject of Feisty's USB support, though, I do have some good news to report.

My biggest motivation for upgrading from edgy was because USB2 had stopped working a few months ago -- bug 54419. I hoped that the newer kernel in Feisty might fix the problem.

So once I had the system up and running, I plugged my trusty hated-by-edgy MP3 player into the USB2 hub, and checked dmesg. It wasn't working -- but the error message was actually useful. Rather than obscure complaints like end_request: I/O error, dev sde, sector 2033440 or device descriptor read/64, error -110 or 3:0:0:0: rejecting I/O to dead device it had a message (which I've since lost) about "insufficient power". Now that's something I might be able to do something about!

So I dug into my bag o' cables and found a PS/2 power adaptor that fit my USB2 hub, plugged it in, plugged the MP3 player into the hub, and voila! it was talking on USB2 again.

Installing "Feisty Fawn"

I finally found some time to try the latest Ubuntu, "Feisty Fawn", on my desktop machine.

I used a xubuntu alternate installer disk, since I don't need the gnome desktop, and haven't had much luck booting from the Ubuntu live CDs lately. (I should try the latest, but my husband had already downloaded and burned the alternate disk and I couldn't work up the incentive to download another disk image.

The early portions of the install were typical ubuntu installer: choose a few language options, choose manual disk partitioning, spend forever up- and down-arrowing through the partitioner trying to persuade it not to automount every partition on the disk (after about the sixth time through I gave up and just let it mount the partitions; I'll edit /etc/fstab later) then begin the install.

Cannot find /lib/modules/2.6.20-15-generic
update-initramfs: failed for /boot/initrd.img-2.6.0-15-generic

Couldn't install grub, and warning direly, "This is a fatal error".

But then popcorn on #linuxchix found Ubuntu bug 37527. Turns out the problem is due to using an existing /boot partition, which has other kernels installed. Basically, Ubuntu's new installer can't handle this properly. The workaround is to go through the installer without a separate /boot partition, let it install its kernels to /boot on the root partition (but don't let it install grub, even though it's fairly insistent about it), then reboot into an old distro and copy the files from the newly-installed feisty partition to the real /boot. That worked fine.

The rest of the installation was smooth, and soon I was up and running. I made some of my usual tweaks (uninstall gdm, install tcsh, add myself to /etc/password with my preferred UID, install fvwm and xloadimage, install build-essentials and the zillion other packages needed to compile anything useful) and I had a desktop.

Of course, the adventure wasn't over. There was more fun yet to come! But I'll post about that separately.

Gnome Knows Best

"So?" I shrugged.

He pointed to his panel. "But it's really using that icon." A little yellow happy-face-with-blob thing.

He right-clicked on the panel icon and brought up a dialog. "See, it should be using /usr/share/pixmaps/xchat.png. Now, I run pho /usr/share/pixmaps/xchat.png ..." And sure enough, the image it said it was using wasn't the image it was actually putting in the panel.

That jogged a memory. "That happened to me once back when I used Gnome. Try a locate xchat | grep png. I think it was using an icon from somewhere else -- that might find it for you."

Sure enough, there were several xchat png images on his system. I suggested going one step further, and actually viewing all of them:

pho `locate xchat | grep png`

We stepped through the images, and sure enough, we found the icon he was seeing. It was at /usr/share/icons/gnome/32x32/apps/xchat.png (with a larger sibling at /usr/share/icons/gnome/48x48/apps/xchat.png).

Good of Gnome to pretend let the user customize the icon location, even though it actually doesn't bother to use the icon specified there! At least you get a nice feeling of empowerment from pretending to choose the icon.

Later in the day, continuing to fiddle with the desktop settings, Dave burst out laughing. "You've got to see this. It's so Gnome." When I saw it, I had to laugh too. You may think you know what you want, but Gnome knows better! If you've ever tried to customize Gnome, you'll laugh, too, when you see the short video we took of it: Gnome knows best (764K).

Desktop Suspend!

Dave was annoyed by it too: his Mac G4 sleeps just fine, but none of his Linux desktops could. And finally he got annoyed enough to spend half a day playing with different options. With what he learned, both he and I now have desktops that can suspend to RAM (his under Debian Sarge, mine under Ubuntu Edgy).

One step was to install hibernate (available as a deb package in both Sarge and Edgy, but distros which don't offer it can probably get it from somewhere on suspend2.net). The hibernate program suspends to disk by default (which is what its parent project, suspend2, is all about) but it can also suspend to RAM, with the following set of arcane arguments:

hibernate -v 4 -F /etc/hibernate/ram.conf

Though actually, in retrospect I suspect I didn't need to install hibernate at all, and Ubuntu's /etc/acpi/sleep.sh script would have done just as well, once I'd finished the other step:

Fiddle with BIOS options. Most BIOSes have a submenu named something like "Power Management Options", and they're almost always set wrong by default (if you want suspend to work). Which ones are wrong depends on your BIOS, of course. On Dave's old PIII system, the key was to change "Sleep States" to include S3 (S3 is the ACPI suspend-to-RAM state). He also enabled APM sleep, which was disabled by default but which works better with the older Linux kernels he uses under Sarge.

On my much newer AMD64 system, the key was an option to "Run VGABIOS if S3 Resume", which was turned off by default. So I guess it wasn't re-enabling the video when I resumed. (You might think this would mean the machine comes up but doesn't have video, but it's never as simple as that -- the machine came up with its disk light solid red and no network access, so it wasn't just the screen that was futzed.)

Such a simple fix! I should have fiddled with BIOS settings long ago. It's lovely to be able to suspend my machine when I go away for a while. Power consumption as measured on the Kill-a-Watt goes down to 5 watts, versus 3 when the machine is "off" (desktop machines never actually power off, they're always sitting there on standby waiting for you to press the power button) and about 75 watts when the machine is up and running.

Now I just have to tweak the suspend scripts so that it gives me a new desktop background when I resume, since I've been having so much fun with my random wallpaper script.

Later update: Alas, I was too optimistic. Turns out it actually only works about one time out of three. The other two times, it hangs after X comes up, or else it initially reboots instead of resuming. Bummer!

The Various Debian Upgrade Methods

But Erinn Clark, our Debian Diva, checked with some of her fellow Debian experts and got us most of the answers, which I will attempt to summarize with a little extra help from web references and man pages.

First, apt-get vs. aptitude: we were told that the primary difference between them is that "aptitude is less likely to remove packages." I confess I'm still not entirely clear on what that means, but aptitude is seen as safer and smarter and I'll go on using it.

aptitude upgrade gets updates (security, bug fixes or whatever) to all currently installed packages. No packages will be removed, and no new packages will be installed. If a currently installed package changes to require a new package that isn't installed, upgrade will refuse to update those packages (they will be "kept back"). To install the "kept back" packages with their dependencies, you can use:

aptitude dist-upgrade gets updates to the currently installed packages, including any new packages which are now required. But sometimes you'll encounter problems in the dependencies, in which case it will suggest that you:

aptitude -f dist-upgrade tries to "fix broken packages", packages with broken dependencies. What sort of broken dependencies? Well, for example, if one of the new packages conflicts with another installed package, it will offer to remove the conflicting package. Without -f, all you get is that a package will be "held back" for unspecified reasons, and you have to go probing with commands like aptitude -u install pkgname or apt-get -o Debug::pkgProblemResolver=yes dist-upgrade to find out the reason.

The upshot is that if you want everything to just happen in one step without pestering you, use aptitude -f dist-upgrade; if you want to be cautious and think things through at each step, use aptitude upgrade and be willing to type the stronger commands when it runs into trouble.

Sections 6.2 and 6.3 of the Debian Reference cover these commands a little, but not in much detail. The APT Howto is better, and runs through some useful examples (which I used to try to understand what -f does).

Thanks go to Erinn, Ari Pollak, and Martin Krafft (whose highly rated book, The Debian System: Concepts and Techniques, apparently would have answered these questions, and I'll be checking it out).

Emacs with Long Lines

So when I need to write a paper (or a book), I prefer to compose in a text editor like vim or emacs, something that won't get in the way of my train of thought. When it's mostly written and ready to format, then I start up the big heavyweight word processor and import or paste the text into it.

(For those of you who think I'm insane and should just live in Open Office all day, the same problem comes up for people who do a lot of composing for web applications, such as an online blog, gmail, a web forum, or a wiki, and for people who want a choice of editor for their GUI mail app.)

Fine, but that introduces a problem. See, text editors have a fixed line width (typically 80 characters, though of course you can adjust this) and paragraphs are usually separated by blank lines (two newline characters together). Word processors expect each paragraph to be one long line for the whole paragraph, and line breaks are used as paragraph breaks (but you only want one of them, not two). How do you reconcile these two models in order to paste plaintext from an editor into a word processor?

Several years ago when I first encountered this problem, I investigated solutions in both vim and emacs (oddly enough, I'm an editor agnostic and equally happy in either one).

For vim, I never did find a solution to the problem, so that settled the editor choice for me. Perhaps some vim expert can let me know what I missed.

For emacs, I found longlines-mode, a hack which lets long lines appear to be wrapped while you're editing them even though they're really not. Apparently Wikipedia has this issue and some Wikipedia contributors use longlines-mode too. (That page also has brief notes on alternate solutions.)

I used longlines-mode for a long time, and it's more or less functional, but I was never really happy with it. It turns out to have some pretty annoying bugs which I was forever needing to work around, and it doesn't solve the blank-lines problem -- you still need to delete blank lines before or after pasting.

Yesterday I was working on an essay for a class I'm taking and decided I'd had enough of longlines-mode and wanted a better solution. I poked around and chatted with the nice folks on #emacs (hoping that someone had come up with a better solution, but no one knew of one) and based on some ideas they had, I came up with one of my own.

My new method is to edit the text file normally: line breaks where they look good, blank lines to separate paragraphs. When I'm finished writing and ready to paste, I run M-x wp-munge, which calls up a very simple function I wrote and added to my .emacs:

;; For composing in emacs then pasting into a word processor,
;; this un-fills all the paragraphs (i.e. turns each paragraph
;; into one very long line) and removes any blank lines that
;; previously separated paragraphs.
;;
(defun wp-munge () "un-fill paragraphs and remove blank lines" (interactive)
  (let ((save-fill-column fill-column))
    (set-fill-column 1000000)
    (mark-whole-buffer)
    (fill-individual-paragraphs (point-min) (point-max))
    (delete-matching-lines "^$")
    (set-fill-column save-fill-column) ))

So simple! Why didn't I think of doing it that way before?

Linus vs. GNOME, part deux: Linus Sends Patches

Here's the latest installment.

Linus, challenged to "start a constructive dialog" by using GNOME for a while then giving a talk on his experiences, went them one better: he sent in patches to fix the usability problems he experienced.

An excerpt from his posting to the Desktop_architects list:

I've sent out patches. The code is actually _cleaner_ after my patches, and the end result is more capable. We'll see what happens.

THAT is constructive.

What I find unconstructive is how the GNOME people always make *excuses*. It took me a few hours to actually do the patches. It wasn't that hard. So why didn't I do it years ago?

I'll tell you why: because gnome apologists don't say "please send us patches". No. They basically make it clear that they aren't even *interested* in fixing things, because their dear old Mum isn't interested in the feature.

Do you think that's "constructive"?

Instead, I _still_ (now after I sent out the patch) hear more of your kvetching about how you actually do everything right, and it's somehow *my* fault that I find things limiting.

Here's a damn big clue: the reason I find gnome limiting is BECAUSE IT IS.

Linus is back, and he's pissed. Go Linus!

Read more details in the linux.com story, or in Linus' actual email in the Desktop_architects list, where you can also follow the rest of the thread.

Autologin under Upstart

There's a little bit of documentation available, but in the end it just came down to fiddling. Here's how it works:

First, use the same /usr/bin/loginscript you used for the old setup, which contains something like this:

#! /bin/sh
/bin/login -f yourusername

Then edit /etc/event.d/tty1 and find the getty line: probably the last line of the file, looking like

respawn /sbin/getty 38400 tty1

respawn /sbin/getty -n -l /usr/bin/loginscript 38400 tty1

That's it! If you want to run X (or anything else) automatically, that works the same way as always.

Update: This changed again in Hardy. Here are the details.

Logitech Cordless Presenter

I got a nifty new toy: a Logitech Cordless Presenter.

I've been watching some of the videos from LCA2007, and I like how some of the speakers made their presentations smoother, and avoided being tied to standing next to their computer, by using remote slide-changing gizmos. It wouldn't help for my GIMP presentations, since those are usually live demos, but I do give other types of talks.

I didn't find much on the web about remote presenters and Linux, and wasn't at all sure whether or how they worked. As usual with hardware, the only way to find out is to buy one and try it.

As it turns out, it works just fine, so I wrote up a review with details.

Udev Tidbits Picked Up at LCA

I'd been having trouble getting my laptop to read its built-in memory stick since upgrading to Ubuntu Edgy. It's basically the same problem I described in an earlier article: the machine boots, sees the built-in reader with no card there, and udev creates /dev/sda but not /dev/sda1. Later, I insert a memory stick, but the reader (like so many other USB-based flash card readers) does not generate an event, so no new device is created.

In that earlier article, the solution was to change the udev rule that creates the device and add something like NAME{all_partitions}="stick". The all_partitions tells it to create /dev/stick1, /dev/stick2 etc. up through the possible maximum number of partitions. (It would be nice to limit it just to /dev/stick1, but there doesn't seem to be any way to do that.)

Unfortunately, in edgy, the udev rules have been rewritten to be a lot more general, and adding {all_partitions} wasn't working. But LinuxConf gave me two solutions to this problem:

First, I was able to pester one of the hal developers about hal's annoying mandatory polling. (This is the official Ubuntu solution to the problem, by the way: if you let hald wake up twice a second to poll every device on the USB bus to see whether anything new has been added, then you'll get that /dev/sda1 device appearing. I wasn't the only one at the conference, I was happy to find, who was unhappy about this hald misbehavior. It got mentioned in at least two other talks as an example of inefficient behavior that can eat batteries and CPU, and a questioner during the hal talk echoed my opinion that the polling should be made optional for those of us who don't want it.)

Anyway, I asked him what hald does to create the /dev/sda1 device once it sees a card. It turns out that touch /dev/sda causes udev to wake up and re-check the device, and create any new device nodes which might have appeared. Hurrah! That's a much cleaner workaround than sudo mknod.

But at breakfast a few days later, I found myself sitting next to a udev expert. He took a look at the file I'd created, /etc/udev/rules.d/memstick.rules, and after a few minutes of fiddling discovered what it was missing: a crucial ACTION=="add" directive which hadn't been required under the old system. The working line now looks like this:

KERNEL=="sda", ACTION=="add", OPTIONS=="all_partitions", NAME{all_partitions}="stick"

linux.conf.au 2007

I had lots of plans to write about it as it was happening. Jot down events of the day, impressions of the talks, etc. In retrospect I have no idea how anyone manages to do that. There's just so much stuff going on at LCA that I was busy the whole time. Blogging or sleep ... that might be a hard choice for some people, but I like sleep. Sleep is good. Sleep lets me have a lot more fun at the talks and the social events afterward.

First, about technical conferences. With the emphasis on technical. In California we have a bunch of conferences like Linux World Expo and the O'Reilly Emerging Tech conference, where a few geeks-turned-PR-people make whizzy presentations to marketing and CIO sorts. Ick. Sometimes there are a few presentations that are actually technical, but not many. And oh, did I mention the multi-kilobuck reg fees?

Linux.conf.au isn't like that at all. It's all geeks, all the time. Not everyone is a programmer (though the majority are), but of the hundreds of people I talked to during the week of the conference I didn't meet a single person who wasn't deeply and passionately involved with Linux in some way. You could pick any person at random, start a conversation and immediately be deep in conversation about interesting details of some aspect of Linux you hadn't thought much about before.

Picking people at random and talking to them? What sort of a geek would do that? Well, the cool thing is that in an environment like LCA, the shyest geek can still network pretty well. If you can't make small talk or force a fake smile, you can jump to the meaty stuff right away and start trading notes on filesystems or network configuration or IRQs or python GUI toolkits. I was almost late to the post-conference LinuxChix meetup because it turned out the person sitting next to me at breakfast was a udev expert who knew how to get my memory stick reader recognized (more on that in a separate article).

Not that you'd really need to talk to random people if you didn't want to. One of the many highlights of the conference was the chance to meet people from all over the world whom I'd only met before on IRC or mailing lists. I already "knew" lots of people there, even if I'd never seen their faces before.

There were tons of talks, with four or five tracks going at all times, and all on good topics. It was quite common to want to go to two or three or even more simultaneous presentations. Fortunately nearly everything was video taped, so with any luck we'll be able to catch up on sessions that we missed (and folks who couldn't afford the trip can benefit from all the great talks). The videos are still being uploaded and aren't all there yet, but they've done an amazing job getting as many transcoded and uploaded as they have so far, and I'm sure the rest won't be too far behind. (Some of them are on the mirror but not yet linked from the Schedule page.)

How do they get all those great talks? I must say, LCA treats its speakers well. In addition to the super-secret "Speakers Adventure", which we were assured was worth getting up at 6am for (it was), they gave us a dinner cruise on scenic Sydney harbor, which included an after-dinner talk on how to give better talks (focused on flash rather than content). I didn't agree with all his points, but that's okay, the point is to get people thinking. I bet every one of us (certainly everyone I talked to) went back and revised our talks at least a little bit based on the presentation.

I hope my GIMP tutorial and my miniconf bugfixing talk lived up to the organizing committee's expectations -- it's intimidating sharing a schedule with so many smart people who are also good speakers!

The first two days of the conference were taken up by "miniconfs". I originally had my eyes on several of the miniconfs, on topics such as Kernel, Education and Research, though I knew I'd start the day at the LinuxChix miniconf. As it turned out, that miniconf was so excellent that I spent the whole day there. It included a mixture of technical and social issues: talks on women in FOSS (Sulamita), my talk on "Bug Fixing for Non Programmers", "Demystifying PCI" (Kristin), a set of terrific "Lightning Talks" under five minutes, and eventually concluded with talks on networking in the social sense (Jacinta) and negotiating wages (Val). After Jacinta's and Val's talks we broke up into small groups and headed for the lawn outside for some very productive discussions of networking and negotiation, which were so interesting we kept the discussions going all afternoon.

The LinuxChix miniconf was Standing Room Only all day, with plenty of men listening in. It was quite a rush to see so many technical women all together, giving talks and discussing details of Linux and FOSS.

Another miniconf-like activity was Open Day, on Thursday afternoon, when the conference invited people from the area (particularly teachers and students) to wander through displays on all sorts of FOSS topics. There were booths from most of the major distros handing out CDs or inviting people to do network installs, a booth showing the One Laptop Per Child project, booths showing games and interesting projects such as amateur rocket and satellite projects or the open source Segway clone, a Linuxchix booth, and booths from a few companies such as Google. Open Day was jam packed, people seemed to be having fun and they gave away a few amazing prizes, like Vaio laptops (donated by IBM) which came in an amazingly small box. I was itching to see what was in those little boxes (we never get the cool small laptops in the US, where the national philosophy is "Bigger is Better") but alas, I wasn't one of the lucky winners.

A couple of other notable talks I went to: Making Things Move: Finding Inappropriate Uses for Scripting Languages by Jonathan Oxer, which included live demos of hooking up radio switches and controlling them from the commandline (with a little simple C glue); and "burning cpu and battery on the gnome desktop" by Ryan Lortie, who not only gave an excellent and entertaining list of programs and services which use up system resources inefficiently by polling, opening too many files or other evils (several other speakers offered similar lists), but also gave concrete advice for finding such programs and fixing them. I'm looking forward to seeing his slides uploaded (I'll link them here when I find them).

The Joys of Shell Pipelines ...

This is Unix, so of course it's possible somehow! The obvious place to start is with the find command, and man find showed how to find all the 1G+ files:

find / -size +1G

(Turns out that's a GNU find syntax, and BSD find, on OS X, doesn't support it. I left it to my friend to check man find for the OS X equivalent of -size _1G.)

But for a problem like this, it's pretty clear we'd need to get find to execute a program that prints both the filename and the size. Initially I used ls -ls, but Saz (who was helping on IRC) pointed out that du on a file also does that, and looks a bit cleaner. With find's unfortunate syntax, that becomes:

find / -size +1G -exec du "{}" \;

But now we needed awk, to collect and add up all the sizes while printing just the filenames. A little googling (since I don't use awk very often) and experimenting led to the final solution:

find / -size +1G -exec du "{}" \; | awk '{print $2; total += $1} END { print "Total is", total}'

Ah, the joys of Unix shell pipelines!

Update: Ed Davies suggested an easier way to do the same thing. turns out du will handle it all by itself: du -hc `find . -size +1G` Thanks, Ed!