Shallow Thoughts : : linux

Open the X selection in a browser window, from any desktop

Like most Linux users, I use virtual desktops. Normally my browser window is on a desktop of its own.

Naturally, it often happens that I encounter a link I'd like to visit while I'm on a desktop where the browser isn't visible. From some apps, I can click on the link and have it show up. But sometimes, the link is just text, and I have to select it, change to the browser desktop, paste the link into firefox, then change desktops again to do something else while the link loads.

So I set up a way to load whatever's in the X selection in firefox no matter what desktop I'm on.

In most browsers, including firefox, you can tell your existing browser window to open a new link from the command line: firefox http://example.com/ opens that link in your existing browser window if you already have one up, rather than starting another browser. So the trick is to get the text you've selected.

At first, I used a program called xclip. You can run this command: firefox `xclip -o` to open the selection. That worked okay at first -- until I hit my first URL in weechat that was so long that it was wrapped to the next line. It turns out xclip does odd things with multi-line output; depending on whether it thinks the output is a terminal or not, it may replace the newline with a space, or delete whatever follows the newline. In any case, I couldn't find a way to make it work reliably when pasted into firefox.

After futzing with xclip for a little too long, trying to reverse-engineer its undocumented newline behavior, I decided it would be easier just to write my own X clipboard app in Python. I already knew how to do that, and it's super easy once you know the trick:

mport gtk
primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)
if primary.wait_is_text_available() :
    print primary.wait_for_text()

That just prints it directly, including any newlines or spaces. But as long as I was writing my own app, why not handle that too?

It's not entirely necessary on Firefox: on Linux, Firefox has some special code to deal with pasting multi-line URLs, so you can copy a URL that spans multiple lines, middleclick in the content area and things will work. On other platforms, that's disabled, and some Linux distros disable it as well; you can enable it by going to about:config and searching for single, then setting the preference editor.singlelinepaste.pasteNewlines to 2.

However, it was easy enough to make my Python clipboard app do the right thing so it would work in any browser. I used Python's re (regular expressions) module:

#!/usr/bin/env python

import gtk
import re

primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)

if not primary.wait_is_text_available() :
    sys.exit(0)
s = primary.wait_for_text()

# eliminate newlines, and any spaces immediately following a newline:
print re.sub(r'[\r\n]+ *', '', s)

That seemed to work fine, even on long URLs pasted from weechat with newlines and spaces, like that looked like

http://example.com/long-
    url.html

All that was left was binding it so I could access it from anywhere. Of course, that varies depending on your desktop/window manager. In Openbox, I added two items to my desktop menu in menu.xml:

  <item label="open selection in Firefox">
    <action name="Execute"><execute>sh -c 'firefox `xclip -o`'</execute></action>
  </item>
  <item label="open selection in new tab">
    <action name="Execute"><execute>sh -c 'firefox -new-tab `xclip -o`'</execute></action>
  </item>

I also added some code in rc.xml inside <context name="Desktop">, so I can middle-click or control-middle-click on the desktop to open a link in the browser:

      <mousebind button="Middle" action="Press">
        <action name="Execute">
          <execute>sh -c 'firefox `pyclip`'</execute>
        </action>
      </mousebind>
      <mousebind button="C-Middle" action="Press">
        <action name="Execute">
          <execute>sh -c -new-tab 'firefox `pyclip`'</execute>
        </action>
      </mousebind>

I set this up maybe two hours ago and I've probably used it ten or fifteen times already. This is something I should have done long ago!

Tags: tech, firefox, linux, cmdline
[ 21:37 Jan 03, 2012    More linux | permalink to this entry | comments ]

Convert patterns in only some lines to title case

A friend had a fun problem: she had some XML files she needed to import into GNUcash, but the program that produced them left names in all-caps and she wanted them more readable. So she'd have a file like this:

<STMTTRN>
   <TRNTYPE>DEBIT
   <DTPOSTED>20111125000000[-5:EST]
   <TRNAMT>-22.71
   <FITID>****

   <NAME>SOME    COMPANY
   <MEMO>SOME COMPANY    ANY TOWN   CA 11-25-11 330346
</STMTTRN>

and wanted to change the NAME and MEMO lines to read Some Company and Any Town. However, the tags, like <NAME>, all had to remain upper case, and presumably so did strings like DEBIT. How do you change just the NAME and MEMO lines from upper case to title case?

The obvious candidate to do string substitutes is sed. But there are several components to the problem.

Addresses

First, how do you ensure the replacement only happens on lines with NAME and MEMO?

sed lets you specify address ranges for just that purpose. If you say sed 's/xxx/yyy/' sed will change all xxx's to yyy; but if you say sed '/NAME/s/xxx/yyy/' then sed will only do that substitution on lines containing NAME.

But we need this to happen on lines that contain either NAME or MEMO. How do you do that? With \|, like this: sed '/\(NAME\|MEMO\)/s/xxx/yyy/'

Converting to title case

Next, how do you convert upper case to lower case? There's a sed command for that: \L. Run sed 's/.*/\L&/' and type some upper and lower case characters, and they'll all be converted to lower-case.

But here we want title case -- we want most of each word converted to lowercase, but the first letter should stay uppercase. That means we need to detect a word and figure out which is the first letter.

In the strings we're considering, a word is a set of letters A through Z with one of the following characteristics:

1. It's preceded by a space
2. It's preceded by a close-angle-bracket, >

So the pattern /[ >][A-Z]*/ will match anything we consider a word that might need conversion.

But we need to separate the first letter and the rest of the word, so we can treat them separately. sed's \( \) operators will let us do that. The pattern \([ >][A-Z]\) finds the first letter of a word (including the space or > preceding it), and saves that as its first matched pattern, \1. Then \([A-Z]*\) right after it will save the rest of the word as \2.

So, taking our \L case converter, we can convert to title case like this: sed 's/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g

Starting to look long and scary, right? But it's not so bad if you build it up gradually from components. I added a g on the end to tell sed this is a global replace: do the operation on every word it finds in the line, otherwise it will only make the substitution once, on the first word it sees, then quit.

Putting it together

So we know how to seek out specific lines, and how to convert to title case. Put the two together, and you get the final command:

sed '/\(NAME\|MEMO\)/s/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g'

I ran it on the test input, and it worked just fine.

For more information on sed, a good place to start is the sed regular expressions manual.

Tags: regexp, cmdline, sed
[ 13:13 Dec 18, 2011    More linux/cmdline | permalink to this entry | comments ]

Taming the loud system beep

One of the distros I'm trying on my Dell Latitude 2120 laptop is Arch Linux. I like a lot of things about Arch; I had to stop using it on my previous laptop because something broke in the wi-fi drivers, but I always regretted giving it up. And it seems to work quite well on the Dell, with one exception: the system beep (which other distros don't support at all) was ear-shatteringly loud, far too loud to consider being able to use this laptop in a public space. Clearly that needed to be fixed.

The usual approach to system beeps, unloading or blacklisting the pcspkr module, had no effect. xset b off turned the beep off in X; I could also set its pitch and duration to change it to a nice quiet click, though I wasn't able to change the volume that way. I actually do like having a system beep, as long as it's fairly quiet and won't disturb people nearby. Unfortunately, xset b only affects the bell while in X; it didn't have any effect on the deafening sound Arch gave upon shutdown or reboot.

It turns out that on some laptops, including this Dell, the system beep goes not through the old-style pcspkr driver, but through the normal sound card. And the sound card has a separate channel for the system beep, so even if you have your volume turned down, the beep may still be at 100%. All I needed to do was run alsamixer and find out what the channel was called: "Beep".

Given that, I could use the amixer program to ensure the beep volume will be sane when I log in. I added the following to .zlogin (for zsh; obviously, adjust for your own shell):

amixer -q -c 0 set Beep 5

That gave me a nice quiet beep. If I need to turn it off completely, amixer can do that too: amixer -q -c 0 set Beep mute (curiously, amixer -q -c 0 set Beep 0 doesn't actually set the volume to zero, just sets it very low).

That volume setting applies to the shutdown beep, too, fortunately. Though what I'd really like is to have quiet beeps while I'm running, but no shutdown beep at all. I don't understand the purpose of the shutdown beep; obviously I know when I've told my machine to shut down or reboot, so why do I need an audible reminder? But I've been unable to find anything explaining what's causing this beep. I tried adding a amixer -q -c 0 set Beep mute to /etc/rc.local.shutdown, but it didn't help; apparently the shutdown beep is called before that file is run. Which strongly suggests it is being run by Arch Linux, not by something in the BIOS. But nobody I've asked had any suggestions as to its source, or how to change it. Another enduring Linux mystery ...

Update: I mentioned xset b as a way to adjust beeps inside X -- in addition to turning beeps totally off, you can also set pitch, duration, and sometimes volume though the volume part didn't work for me. But outside X, you can make similar adjustments with setterm, e.g. setterm -bfreq 400 -blength 50. Thanks to Mikachu for the tip!

Tags: linux, archlinux, alsa
[ 11:05 Dec 13, 2011    More linux | permalink to this entry | comments ]

Set Ubuntu's system clock to use localtime, not UTC

Need your Ubuntu clock to stay in sync with a dual-boot Windows install? It seems to have changed over the years, and google wasn't finding any pages for me offering suggestions that still worked.

Turns out, in Ubuntu Oneiric Ocelot, it's controlled by the file /etc/default/rcS: set UTC=no.

Apparently it's also possible to get Windows to understand a UTC system clock using a registry tweak.

Ironically, that page, which I found by searching for windows system clock utc, also has the answer for setting Ubuntu to local time. So if I'd searched for Windows in the first place, I wouldn't have had to puzzle out the Ubuntu solution myself. Go figure!

[ 12:56 Dec 11, 2011    More linux | permalink to this entry | comments ]

Configuring extlinux's auto-update on Debian

A few days ago, I wrote about how to set up and configure extlinux (syslinux) as a bootloader. But on Debian or Ubuntu, if you make changes to files like /boot/extlinux/extlinux.conf directly, they'll be overwritten.

The configuration files are regenerated by a program called extlinux-update, which runs automatically every time you update your kernel. (Specifically, it runs from the postinst script of the linux-base package: you can see it in /var/lib/dpkg/info/linux-base.postinst.)

So what's a Debian user to do if she wants to customize the menus, add a splash image or boot other operating systems?

First, if you decide you really don't want Debian overwriting your configuration files, you can change disable updates by editing /etc/default/extlinux. Just be aware you won't get your boot menu updated when you install new kernels -- you'll have to remember to update them by hand.

It might be worth it: the automatic update is nearly as annoying as the grub2 updater: it creates two automatic entries for every kernel you have installed. So if you have several distros installed, each with a kernel or two in your shared /boot, you'll get an entry to boot Debian Squeeze with the Ubuntu Oneiric kernel, one for Squeeze with the Natty kernel, one for Squeeze with the Fedora 16 kernel ... as well as entries for every kernel you have that's actually owned by Debian. And then for each of these, you'll also get a second entry, to boot in recovery mode. If you have several distros installed, it makes for a very long and confusing boot menu!

It's a shame that the auto-updater doesn't restrict itself to kernels managed by the packaging system, which would be easy enough to do. (Wonder if they would accept a patch?) You might be able to fudge something that works right by setting up symlinks so that the only readable kernels actually live on the root partition, so Debian can't read the kernels from the other distros. Sounds a bit complicated and I haven't tried it. For now, I've turned off automatic updating on my system.

But if your setup is simpler -- perhaps just one Debian or one Ubuntu partition plus some non-Linux entries such as BSD or Windows -- here's how to set up Debian-style automatic updating and still keep all your non-Linux boot entries and your nice menu customizations.

Debian automatic updates and themes

First, take a quick look at /etc/default/extlinux and customize anything there you might need, like the names of the kernels, kernel boot parameters or timeout. See man extlinux-update for details.

For configuring menu colors, image backgrounds and such, you'll need to make a theme. You can see a sample theme by installing the package syslinux-themes-debian -- but watch out. If you haven't configured apt not to pull in suggested packages, that may bring back grub or grub-legacy, which you probably don't want.

You can make a theme without needing that package, though. Create a directory /usr/share/syslinux/themes/mythemename (the extlinux-update man page claims you can put a theme anywhere and specify it by its full path, but it lies). Create a directory called extlinux inside it, and make a file with everything you want from extlinux.conf. For example:

default 0
prompt 1
timeout 50

ui vesamenu.c32
menu title Welcome to my Linux machine!
menu background mysplash.png
menu color title 1;36 #ffff8888 #00000000 std
menu color unsel 0    #ffffffff #00000000 none
menu color sel   7    #ff000000 #ffffff00 none

include linux.cfg
menu separator
include themes/mythemename/other.cfg

Note that last line: you can include other files from your theme. For instance, you can create a file called other.cfg with entries for other partitions you want to boot:

label oneiric
menu label Ubuntu Oneiric Ocelot
kernel /vmlinuz-3.0.0-12-generic
append initrd=/initrd.img-3.0.0-12-generic root=UUID=c332b3e2-5c38-4c50-982a-680af82c00ab ro quiet

label fedora
menu label Fedora 16
kernel /vmlinuz-3.1.0-7.fc16.i686
append initrd=/initramfs-3.1.0-7.fc16.i686.img root=UUID=47f6b1fa-eb5d-4254-9fe0-79c8b106f0d9 ro quiet

menu separator

LABEL Windows
KERNEL chain.c32
APPEND hd0 1

Of course, you could have a debian.cfg, an ubuntu.cfg, a fedora.cfg etc. if you wanted to have multiple distros all keeping their kernels up-to-date. Or you can keep the whole thing in one file, theme.cfg. You can make a theme as complex or as simple as you like.

Tags: linux, boot, extlinux, syslinux, debian, ubuntu
[ 11:26 Nov 24, 2011    More linux/install | permalink to this entry | comments ]

How to install extlinux (syslinux) as a bootloader

When my new netbook arrived, I chose Debian Squeeze as the first Linux distro to install, because I was under the impression it still used grub1, and I wanted to avoid grub2. I was wrong -- Squeeze uses grub2. Uninstalling grub2, installing grub-legacy and running grub-install and update-grub didn't help; it turns out even in Debian's grub-legacy package, those programs come from grub2's grub-common package.

What a hassle! But maybe it was a blessing in disguise -- I'd been looking for an excuse to explore extlinux as a bootloader as a way out of the grub mess.

Extlinux is one of the many spinoffs of syslinux -- the bootloader used for live CDs and many other applications. It's not as commonly used as a bootloader for desktops and laptops, but it's perfectly capable of that. It's simple, well tested and has been around for years. And it supports the few things I want out of a bootloader: it has a simple configuration file that lives on the /boot partition; it can chain-load Windows, on machines with a Windows partition; it even offers pretty graphical menus with image backgrounds.

Since there isn't much written about how to use extlinux, I wrote up my experiences along with some tips for configuring it. It came out too long for a blog article, so instead I've made it its own page: How to install extlinux (syslinux) as a bootloader.

Tags: linux, boot, extlinux, syslinux
[ 15:19 Nov 20, 2011    More linux/install | permalink to this entry | comments ]

Synaptics horizontal scrolling (and other touchpad configurations)

My new netbook (about which, more later) has a trackpad with areas set aside for both vertical and horizontal scrolling. Vertical scrolling worked out of the box on Squeeze without needing any extra fiddling. My old Vaio had that too, and I loved it.

Horizontal scrolling took some extra research. I was able to turn it on with a command:

synclient HorizEdgeScroll=1

(thank you, AbsolutelyTech). Then it worked fine, for the duration of that session.

But it took a lot more searching to find out the right place to set it permanently. Nearly every page you find on trackpad settings tells you to edit /etc/X11/xorg.conf. Distros haven't normally used xorg.conf in over three years! Sure, you can generate one, then edit it -- but surely there's a better way.

And there is. At least in Debian Squeeze and Ubuntu Natty, you can edit /usr/share/X11/xorg.conf.d/50-synaptics.conf and add this options line inside the "InputClass" section:

        options "HorizEdgeScroll" "1"

Don't forget the quotes, or X won't even start.

In theory, you can use this for any of the Synaptics driver options -- tap rate and sensitivity, even multitouch. Your mileage may vary -- horizontal scroll is the only one I've tested so far. But at least it's easier than generating and maintaining an xorg.conf file!

Tags: linux, X11, laptops
[ 15:20 Oct 31, 2011    More linux/laptop | permalink to this entry | comments ]

Making an Ubuntu live USB stick persistent

I wrote a few days ago about my multi-distro Linux live USB stick. Very handy!

But one thing that bugs me about live distros: they're set up with default settings and don't have a lot of the programs I want to use. Even getting a terminal takes quite a lot of clicks on most distros. If only they would save their settings!

It's possible to make a live USB stick "persistent", but not much is written about it. Most of what's written tells you to create the USB stick with usb-creator -- a GUI app that I've tried periodically for the past two years without ever once succeeding in creating a bootable USB stick.

Even if usb-creator did work, it wouldn't work with a multi-boot stick like this one, because it would want to overwrite the whole drive. So how does persistence really work? What is usb-creator doing, anyway?

How persistence works: Casper

The best howto I've found on Ubuntu persistence is LiveCD Persistence. But it's long and you have to wade through a lot of fdisk commands and similar arcana. So here's how to take your multi-distro stick and make at least one of the installs persistent.

Ubuntu persistence uses a package called casper which overlays the live filesystem with the contents of another filesystem. Figuring out where it looks for that filesystem is the key.

Casper looks for its persistent storage in two possible places: a partition with the label "casper-rw", and a file named "casper-rw" at the root of its mounted partitions.

So you could make a separate partition labeled "casper-rw", using your favorite partitioning tool, such as gparted or fdisk. But if you already have your multi-distro stick set up as one big partition, it's just as easy to create a file. You'll have to decide how big to make the file, based on the size of your USB stick.

I'm using a 4G stick, and I chose 512M for my persistent partition:

$ dd if=/dev/zero of=/path/to/casper-rw bs=1M count=512

Be patient: this step takes a while.

Next, create a filesystem inside that file. I'm not sure what the tradeoffs are among various filesystem types -- no filesystem is optimized for being run as a loopback file read from a vfat USB stick that was also the boot device. So I flipped a coin and used ext3:

$ mkfs.ext3 /path/to/casper-rw
/path/to/casper-rw is not a block special device.
Proceed anyway? (y,n) y

One more step: you need to add the persistent flag to your boot options. If you're following the multi-distro USB stick tutorial I linked to earlier, that means you should edit boot/grub/grub.cfg on the USB stick, find the boot stanza you're using for Ubuntu, and make the line starting with linux look something like this:

    linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile quiet splash noprompt persistent --

Now write the stick, unmount it, and try booting your live install.

Testing: did it work?

The LiveCD/Persistence page says persistent settings aren't necessarily saved for the default "ubuntu" user, so it's a good idea to make a new user. I did so.

Oops -- about that Ubuntu new user thing

But at least in Ubuntu Oneiric: there's a problem with that. If you create a user, even as class Administrator (and of course you do want to be an Administrator), it doesn't ask you for a password. If you now log out or reboot, your new user should be saved -- but you won't be able to do anything with the system, because anything that requires sudo will prompt you for your nonexistent password. Even attempting to set a password will prompt you for the nonexistent password.

Apparently you can "unlock" the user at the time you create it, and then maybe it'll let you set a password. I didn't know this beforehand, so here's how to set a password on a locked user from a terminal:

$ sudo passwd username

For some reason, sudo will let you do this without prompting for a password, even though you can't do anything administrative through the GUI.

Testing redux

Once you're logged in as your new user, try making some changes. Add and remove some items from the unity taskbar. Install a couple of packages. Change the background.

Now try rebooting. If your casper-rw file worked, it should remember your changes.

When you're not booted from your live USB stick, you can poke around in the filesystem it uses by mounting it in "loopback" mode. Plug the stick into a running Linux machine, mount it the usb stick, then mount it with

$ sudo mount -o loop /path/to/casper-rw /mnt

/path/to is wherever you mounted your usb stick -- e.g. /media/whatever. With the file mounted in loopback mode, you should be able to adjust settings or add new files without needing to boot the live install -- and they should show up the next time you use the live install.

My live Ubuntu Oneiric install is so much more fun to use now!

Tags: ubuntu, linux, install
[ 14:41 Oct 28, 2011    More linux/install | permalink to this entry | comments ]