Shallow Thoughts : tags : editors

Using default-frame-alist to specify Emacs window size, position and font

Seems like every few years I need to change the way I specify my preferred emacs fonts and window sizes.

Historically this all used to happen from one file, ~/.Xdefaults, where you set up your defaults for all X programs. In a way that was nice, since you could set up defaults and see the same font everywhere. On the other hand, it made for a huge, somewhat hard to read file, and it's increasingly out of favor on modern desktops, with modern toolkits like GTK just ignoring it.

Emacs still reads Xdefaults -- but only sort of. A lot of the values I used to set there no longer work properly. Some time ago I commented out my various attempts at setting emacs font, like

Emacs*font: -*-clean-bold-*-*-*-13-*-*-*-c-*-*-*
Emacs*font: DejaVu Sans Mono-10:bold
Emacs*font: clean-13:bold
Wmacs*font: Liberation Mono-10:bold
Emacs.font: 7x13bold
Emacs.faceName: Dejavu-10:style=bold

since none of them worked, and worked out a way of setting fonts inside my .emacs file:

(set-face-attribute 'default nil :font "Terminus-12:bold")

That worked to set the font, but it had another annoying attribute: it doesn't happen at startup, so it messed up my window size. See, emacs would start up, see the size I specified in .Xdefaults:

Emacs*geometry: 80x45

and try to set that. But it hadn't read .emacs yet, so it was still using whatever its default font and size is, and that's huge -- so 45 lines made a window too tall to fit on my laptop screen. Emacs would then shrink its window to fit the screen (41 lines). Only then would it open .emacs, whereupon it would see the set-face-attribute, change the font, and resize the window again, much, smaller, still 41 lines.

What a pain!

The emacs manual, in addition to talking about these various Xdefaults properties and command-line options, does mention a couple of variables, set-screen-height and set-screen-width, that looked promising. I tried putting (set-screen-height 45) in my .emacs right after I set the font -- no dice. Apparently that doesn't work because by the time those are read, emacs has already decided that 41 lines is as big as the window can possibly be.

Here's the answer: another variable that goes inside .emacs, default-frame-alist, but this one can override that maximum-height decision that emacs has already made. Here's an example of it in some useful defaults for emacs, and based on that, I was able to come up with this:

(setq default-frame-alist
      '((top . 10) (left . 2)
        (width . 80) (height . 53)
        (font . "terminus-iso8859-1-bold-14")
        ))

Curiously, that height setting, 53, needs to be 3 more than what I actually want according to the size emacs reports to the window manager. So don't take the number too seriously; just try numbers a little bigger than what you actually want until you get the size you're after. The font setting is the X font specifier: I ran xlsfonts | grep -i terminus | grep 14 then picked one of the simpler of the lines it printed out, but you can use a full specifier like -xos4-terminus-bold-r-normal--14-140-72-72-c-80-iso8859-1 like you get from xfontsel, if you prefer.

Startup still isn't pretty -- emacs still shows a big window at one place on the screen, resizes it several times then jumps it over to the top/left coordinates I specified. Of course, I could tell my window manager to start it in the right place so the jumping-around would be minimized; but that wouldn't help the visible resizing. Just a minor irritation.

I'm sure there's lots more useful stuff buried in that sample emacs config file (it was suggested to me when I asked about this on the #emacs IRC channel), so I'll be reading it to see what else I can glean.

Tags: emacs, editors, X11
[ 11:24 Jul 14, 2011    More linux/editors | permalink to this entry | comments ]

Vim/Emacs tip: use modelines for files that need special behavior

Most of the time when I edit a text file in vim, I want lines to wrap automatically as I type, at somewhere around 70 columns wide. So I set textwidth=70 in .vimrc.

But sometimes that isn't appropriate. For instance, I have a procmail rules file where I put common spam patterns that programs like spamassassin don't seem to be able to catch. So I might have lines like:

*^Subject:.*(Ink Cartridges|Hummingbird Vine|who's who of executives|Avandia|Botox|Your Email ID|Zoosk|Best airfares on the internet|UGG Boots|police training)

... and so on -- you get the idea. I can't have lines breaking in the middle, because then the procmail rule wouldn't work. So every time I add a new phrase, I have to :set tw=0 (or one of the other umpteen ways one can tell vim not to wrap lines) first.

But you can set special behavior for one specific file by adding a special comment called a "modeline" as the first line of the file.

Procmail treats any line starting with a hash, #, as a comment, and vim recognizes # as a comment. So I can add this as the first line of the procmail file:

# vim: set tw=0:

then vim will see that and un-set that default text width I specify in .vimrc.

Vim understands most common comment styles, so it should understand lines like /* vim: set tw=0: */ and // vim: set tw=0: and ; vim: set tw=0: as well.

But to make this work I had to do one more thing: in .vimrc, I had to add

set modeline

Apparently on some versions of vim this is on by default; in others it's turned off for security reasons (someone could put an evil modeline into a file which would make your vim do something when you edited it). Definitely something to be aware of, but if you mostly edit files you created yourself on your local machine, and no one else uses your machine, it's your choice whether to worry about it.

Emacs has modelines too

Emacs has mode lines too. For instance, C++ files in Mozilla's source tend to start with:

/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */

It's awfully handy to be able to define specific indentation style for the files within a project, making it easy for emacs users, at least, to follow your preferred coding style. If only all editors understood them!

Tags: editors, vim, emacs, tip
[ 20:26 May 25, 2011    More linux/editors | permalink to this entry | comments ]

Quick tip: Disabling version control in Emacs

For some time I've been mildly annoyed that whenever I start emacs and open a file that's under any sort of version control -- cvs, svn, git or whatever -- I can't start editing right away, because emacs has to pause for a while and load a bunch of version-control cruft I never use. Sometimes it also causes problems later, when I try to write to the file or if I update the directory.

It wasn't obvious what keywords to search for, but I finally found a combination, emacs prevent OR disable autoload vc (the vc was the important part), which led me to the solution (found on this page):

;; Disable all version control
(setq vc-handled-backends nil)

Files load much faster now!

Tags: editors, emacs, git, tip
[ 12:11 Feb 04, 2011    More linux/editors | permalink to this entry | comments ]

Escaping HTML characters in Emacs (and how to do replaces in elisp)

Last week I found myself writing another article that includes code snippets in HTML.

So what, you ask? The problem is, when you're writing articles in HTML, every time you include a code snippet inside a <pre> tag you invariably forget that special characters like < > & have special meanings in HTML, and must be escaped. Every < has to change to &lt;, and so forth, after you paste the code.

In vi/vim, replacing characters is straightforward. But I usually write longer articles in emacs, for various unimportant reasons, and although emacs has global replace, it only works from wherever you are now (called "point" in emacs lingo) to the end of the file. So if you're trying to fix something you pasted in the middle of the article, you can't do it with normal emacs replace.

Surely this is a wheel that has already been re-invented a thousand times, I thought! But googling and asking emacs experts turned up nothing. Looks like I'd have to write it.

And that turned out to be more difficult than I expected, for the same reason: emacs replace-string works the same way from a program as it does interactively, and replaces from point to the end of the file, and there's no way to restrict it to a more limited range.

Several helpful people on #emacs chimed in with ideas, but most of them didn't pan out. But ggole knew a way to do it that was both clean and reliable (thanks!).

Here's the elisp function I ended up with. It uses save-excursion to put the cursor back where it started before you ran the function, narrow-to-region to make replace-string work only on the region, and save-restriction get rid of that narrow-to-region after we're done. Nice!

(defun unhtml (start end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region start end)
      (goto-char (point-min))
      (replace-string "&" "&")
      (goto-char (point-min))
      (replace-string "<" "&lt;")
      (goto-char (point-min))
      (replace-string ">" "&gt;")
      )))

And yes, I used it just now on that elisp snippet.

Tags: emacs, editors, programming
[ 19:08 Dec 01, 2010    More linux/editors | permalink to this entry | comments ]

Non-greedy regular expressions to clean up crappy autogenerated HTML

I maintain the websites for several clubs. No surprise there -- anyone with even a moderate knowledge of HTML, or just a lack of fear of it, invariably gets drafted into that job in any non-computer club.

In one club, the person in charge of scheduling sends out an elaborate document every three months in various formats -- Word, RTF, Excel, it's different each time. The only regularity is that it's always full of crap that makes it hard to turn it into a nice simple HTML table.

This quarter, the formats were Word and RTF. I used unrtf to turn the RTF version into HTML -- and found a horrifying page full of lines like this:

<body><font size=3></font><font size=3><br>
</font><font size=3></font><b><font size=4></font></b><b><font size=4><table border=2>
</font></b><b><font size=4><tr><td><b><font size=4><font face="Arial">Club Schedule</font></font></b><b><font size=4></font></b><b><font size=4></font></b></td>
<font size=3></font><font size=3><td><font size=3><b><font face="Arial">April 13</font></b></font><font size=3></font><font size=3><br>
</font><font size=3></font><font size=3><b></b></font></td>

I've put the actual page content in bold; the rest is just junk, mostly doing nothing, mostly not even legal HTML, that needs to be eliminated if I want the page to load and display reasonably.

I didn't want to clean up that mess by hand! So I needed some regular expressions to clean it up in an editor. I tried emacs first, but emacs makes it hard to try an expression then modify it a little when the first try doesn't work, so I switched to vim.

The key to this sort of cleanup is non-greedy regular expressions. When you have a bad tag sandwiched in the middle of a line containing other tags, you want to remove everything from the <font through the next > -- but no farther, or else you'll delete real content. If you have a line like

<td><font size=3>Hello<font> world</td>

you only want to delete through the <font>, not through the </td>.

In general, you make a regular expression non-greedy by adding a ? after the wildcard -- e.g. <font.*?>. But that doesn't work in vim. In vim, you have to use \{M,N} which matches from M to N repetitions of whatever immediately precedes it. You can also use the shortcut \{-} to mean the same thing as *? (0 or more matches) in other programs.

Using that, I built up a series of regexp substitutes to clean up that unrtf mess in vim:

:%s/<\/\{0,1}font.\{-}>//g
:%s/<b><\/b>//g
:%s/<\/b><b>//g
:%s/<\/i><i>//g
:%s/<td><\/td>/<td><br><\/td>/g
:%s/<\/\{0,1}span.\{-}>//g
:%s/<\/\{0,1}center>//g

That took care of 90% of the work, leaving me with hardly any cleanup I needed to do by hand. I'll definitely keep that list around for the next time I need to do this.

Tags: regexp, html, editors, vim
[ 22:02 Mar 29, 2010    More linux/editors | permalink to this entry | comments ]

Make scroll lock stop beeping in Emacs 23

I haven't been using the spare machine much lately. So I hadn't noticed until last week that since upgrading to the emacs 23.1.1 on Ubuntu Karmic koala, every time I press the Scroll Lock key -- the key my KVM uses to switch to the other computer -- with focus in an emacs window, emacs beeps and complains that the key is unbound.

That was a problem I thought I'd solved long ago, an easy fix in .emacs:

(global-set-key [scroll-lock] 'ignore)

But in emacs 23, it wasn't working any more. Emacs listed the key as "<Scroll_Lock>", but using that directly in global-set-key doesn't work.

The friendly and helpful (really!) crew at #emacs found me a solution, after some fiddling around.

(global-set-key (kbd "<Scroll_Lock>") 'ignore)

Tags: emacs, editors, kvm, tips
[ 22:47 Feb 09, 2010    More linux/editors | permalink to this entry | comments ]

Taming Emacs' text mode wrapping and indenting

To wrap long lines, or not to wrap? It's always a dilemma. Automatic wrapping is great when you're hammering away typing lots of text. But it's infuriating when you're trying to format something yourself and the editor decides it wants to line-wrap a little too early.

Although of course you can set the wrapping width, Emacs has a tendency to wrap early -- especially when you hit return. All too often, I'll be typing away at a long line, get to the end of the sentence and paragraph with the last word on the same line with the rest -- then realize that as soon as I hit return, Emacs is going to move that last word to a line by itself. Drives me nuts!

And the solution turns out to be so simple. The Return key, "\C-m". was bound to the (newline) function (you can find out by typing M-x, then describe-key, then hitting Return). Apparently (newline) re-wraps the current line before inserting a line break. But I just wanted it to insert a line break.

No problem -- just bind "C-m" to (insert "\n").

But there's a second way, too, if you don't want to rebind: there's a magic internal emacs table you can change.

(set-char-table-range auto-fill-chars 10 nil)

But wait -- there's one other thing I want to fix in text mode.

Automatic indent is another one of those features that's very convenient ... except when it's not.

If I have some text like:

First point:
  - subpoint a
  - subpoint b

then it's handy if, when I hit Return after subpoint a, emacs indents to the right level for subpoint b. But what happens when I get to the end of that list?

First point:
  - subpoint a
  - subpoint b

Second point:
  - subpoint c

When I hit Return after subpoint b, Emacs quite reasonably indents two spaces. If I immediately type another Return, Emacs sensibly deletes the two spaces it just inserted, opens a new line -- but then it indents that new line another two spaces.

After a blank line, I always want to start at the beginning, not indented at all.

Here's how to fix that. Define a function that will be called whenever you hit return in text mode. That function tests whether the caret comes immediately after a blank line, or at the beginning of the file. It indents except in those two cases; and in neither case does it re-wrap the current line.

;; In text mode, I don't want it auto-indenting for the first
;; line in the file, or lines following blank lines.
;; Everywhere else is okay.
(defun newline-and-text-indent ()
  "Insert a newline, then indent the next line sensibly for text"
  (interactive)
  (cond
   ;; Beginning of buffer, or beginning of an existing line, don't indent:
   ((or (bobp) (bolp)) (newline))

   ;; If we're on a whitespace-only line,
   ((and (eolp)
         (save-excursion (re-search-backward "^\\(\\s \\)*$"
                                             (line-beginning-position) t)))
    ;; ... delete the whitespace, then add another newline:
    (kill-line 0)
    (newline))

   ;; Else (not on whitespace-only) insert a newline,
   ;; then add the appropriate indent:
   (t (insert "\n")
      (indent-according-to-mode))
   ))

Then tell emacs to call that function when it sees the Return key in text mode:

(defun text-indent-hook ()
  (local-set-key "\C-m" 'newline-and-text-indent)
  )
(setq text-mode-hook 'text-indent-hook)

Finally, this is great for HTML mode too, if you get irritated at not being able to put an <a href="longurl"> all on one line:

(defun html-hook ()
  (local-set-key "\C-m" (lambda () (interactive) (insert "\n")))
  )
(setq sgml-mode-hook 'html-hook)

Tags: emacs, editors, tips
[ 10:29 Jan 13, 2010    More linux/editors | permalink to this entry | comments ]

Emacs: Insert tags in HTML Mode

Wouldn't it be nice if Emacs HTML mode had a way to insert HTML tags, so you didn't have to type <b></b> all the time? Sort of like what's described in this page -- except that page describes an HTML mode that clearly isn't the one that's installed on Ubuntu, since none of those bindings actually work?

I've been meaning to figure out a way to do that for ages, and finally got around to it. Turns out Emacs SGML mode (which is really what Ubuntu installs and uses for HTML files) doesn't have functions for specific HTML tags like <b>, but it does have a general tag-inserting function.

Type C-c C-t -- emacs prompts you for the tag, so type b or whatever, and hit return -- and you get the tag, with the cursor correctly positioned for you to type your new bold text.

But that's four keystrokes. What if you want shorter bindings for particular tags, like C-b C-b to insert a bold?

For that, you need to use a lambda and a mode hook. In your .emacs it looks like this:

;; Define keys for inserting tags in HTML mode:
(defun html-hook ()
  (local-set-key "\C-c\C-b" (lambda () (interactive) (sgml-tag "b")))
  )
(setq sgml-mode-hook 'html-hook)

There's apparently also supposed to be a command bound to C-c / that closes the current tag, but my version of sgml-mode doesn't bind anything to that key, and the only likely-looking function name, sgml-maybe-end-tag, doesn't end the current tag. Such is life!

But one more don't-miss feature that I'd missed all along is C-c C-n: type it before a special character like < or & and emacs will insert the appropriate &lt; or &amp; for you. Nice!

(Thanks to bojohan on #emacs for the tips!)

Tags: editors, emacs, html-mode
[ 20:31 Jul 29, 2009    More linux/editors | permalink to this entry | comments ]

Emacs bookmarks -- a huge time-saver

Oh, wow. I can't believe I've used Emacs all these years without knowing about bookmarks.

I wanted something in Emacs akin to the "Open Recent" menu that a lot of GUI apps have. Except, well, I didn't want it to need a menu (I don't normally show a menubar in Emacs) and I didn't want it limited only to recently accessed files. So ... just like Open Recent, only completely different.

What I really wanted was a way to nickname files I access regularly, so I don't have to type ~/foo/bar/blaz/route-66/dufus/velociraptor/archaeopteryx/filename every time. Even with tab completion, remembering long paths gets old. Of course emacs must have a way to do that; it has everything. The trick was guessing what it might be called in order to search for it.

The answer is emacs bookmarks and they're super easy to use.

C-x r m sets a bookmark for the current location in the current file. It prompts for a bookmark name; give it a nickname, or hit return to default it to the current filename.

C-x r b bookmark-name jumps back to a bookmark, opening the file if it isn't already. Of course, tab completion works for the bookmark name.

Bookmarks are saved in ~/.emacs.bmk so they're persistent.

It's perfect. I just wish I'd thought to look for it years ago.

(Of course, Emacs can do recent files too.)

Tags: editors, emacs, bookmarks
[ 09:21 Mar 27, 2009    More linux/editors | permalink to this entry | comments ]

Vim tip: fixing the light-background color schemes

I use a light background for my X terminals (xterm and rxvt): not white, but a light grey that I find easy on the eyes. Long ago, I spent the time to set up a custom vim color scheme that works with the light background.

But sometimes I need to run vim somewhere where I don't have access to my custom scheme. It always starts up with a lot of the words displayed in yellow, completely unreadable against a light background. :set background=light doesn't help -- the default colorscheme is already intended for a light background, yet it still uses yellow characters.

I tried all the colorschemes installed with ubuntu's vim (you can get a list of them with ls /usr/share/vim/vim71/colors). The only light-background vim schemes that don't use yellow all have their primary text color as red. Do a lot of people really want to edit red text? Maybe the same people who think that yellow is a dark color?

Curiously, it turns out that if you use one of these light color schemes on a Linux console (with its black background), the yellow text isn't yellow (which would show up fine against black), but orange (which would be better on a light background).

Mikael knew the answer:

:set t_Co=88

This tells vim to use 88-color mode instead of its default of 8, and the yellow text turns light blue. Not terrifically readable but much better than yellow. Or, instead, try

:set t_Co=256

and the yellow/light blue text turns an ugly, but readable, orange (probably the same orange as the console used).

So, vim users with dark-on-light terminal schemes: add set t_Co=256 in your .vimrc (no colon) and you'll be much happier.

Update: Pádraig Brady has a great page explaining more about terminal colour highlights, including a TERM=xterm-256color setting to get vim to use 256 colors automatically. There's also a lot of good advice there on enabling colors in other console apps.

The only catch: on Ubuntu you do have to install the ncurses-term package, which will get you xterm-256color as well as 256color variants for lots of other terminal types. Here's useful page on 256-Color XTerms in Ubuntu.

Tags: editors, vim, color, tips
[ 21:29 Mar 22, 2009    More linux/editors | permalink to this entry | comments ]

More fun with regexps: Adding "[no output]" in shell logs

Someone on LinuxChix' techtalk list asked whether she could get tcsh to print "[no output]" after any command that doesn't produce output, so that when she makes logs to help her co-workers, they will seem clearer.

I don't know of a way to do that in any shell (the shell would have to capture the output of every command; emacs' shell-mode does that but I don't think any real shells do) but it seemed like it ought to be straightforward enough to do as a regular expression substitute in vi. You're looking for lines where a line beginning with a prompt is followed immediately by another line beginning with a prompt; the goal is to insert a new line consisting only of "[no output]" between the two lines.

It turned out to be pretty easy in vim. Here it is:

:%s/\(^% .*$\n\)\(% \)/\1[no results]\r\2/

Explanation:

:

starts a command

%

do the following command on every line of this file

s/

start a global substitute command

\(

start a "capture group" -- you'll see what it does soon

^

match only patterns starting at the beginning of a line

%

look for a % followed by a space (your prompt)

.*

after the prompt, match any other characters until...

$

the end of the line, after which...

\n

there should be a newline character

\)

end the capture group after the newline character

\(

start a second capture group

%

look for another prompt. In other words, this whole

expression will only match when a line starting with a prompt

is followed immediately by another line starting with a prompt.

\)

end the second capture group

/

We're finally done with the mattern to match!

Now we'll start the replacement pattern.

\1

Insert the full content of the first capture group

(this is also called a "backreference" if you want

to google for a more detailed explanation).

So insert the whole first command up to the newline

after it.

[no results]

After the newline, insert your desired string.

\r

insert a carriage return here (I thought this should be

\n for a newline, but that made vim insert a null instead)

\2

insert the second capture group (that's just the second prompt)

/

end of the substitute pattern

Of course, if you have a different prompt, substitute it for "% ". If you have a complicated prompt that includes time of day or something, you'll have to use a slightly more complicated match pattern to match it.

Tags: regexp, shell, CLI, linux, editors
[ 13:34 Oct 12, 2008    More linux/editors | permalink to this entry | comments ]

Smart Wrapping with Greedy and Non-Greedy Regular Expressions

I had a chance to spend a day at the AGU conference last week. The American Geophysical Union is a fabulous conference -- something like 14,000 different talks over the course of the week, on anything related to earth or planetary sciences -- geology, solar system astronomy, atmospheric science, geophysics, geochemistry, you name it.

I have no idea how regular attendees manage the information overload of deciding which talks to attend. I wasn't sure how I would, either, but I started by going through the schedule for the day I'd be there, picking out a (way too long) list of potentially interesting talks, and saving them as lines in a file.

Now I had a file full of lines like:

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights Into the Cratering Process From Geophysics and Geochemistry II

Fine, except that I couldn't print out something like that -- printers stop at 80 columns. I could pass it through a program like "fold" to wrap the long lines, but then it would be hard to scan through quickly to find the talk titles and room numbers. What I really wanted was to wrap it so that the above line turned into something like:

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics
                          and Geochemistry II

But how to do that? I stared at it for a while, trying to figure out whether there was a clever vim substitute that could handle it. I asked on a couple of IRC channels, just in case there was some amazing Linux smart-wrap utility I'd never heard of. I was on the verge of concluding that the answer was no, and that I'd have to write a python script to do the wrapping I wanted, when Mikael emitted a burst of line noise:
%s/\(.\{72\}\)\(.*\)/\1^M^I^I^I\2/

Only it wasn't line noise. Seems Mikael just happened to have been reading about some of the finer points of vim regular expressions earlier that day, and he knew exactly the trick I needed -- that .\{72\}, which matches lines that are at least 72 characters long. And amazingly, that expression did something very close to what I wanted.

Or at least the first step of it. It inserts the first line break, turning my line into

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics and Geochemistry II

but I still needed to wrap the second and subsequent lines.

But that was an easier problem -- just do essentially the same thing again, but limit it to only lines starting with a tab. After some tweaking, I arrived at exactly what I wanted:

%s/^\(.\{,65\}\) \(.*\)/\1^M^I^I^I\2/

%g/^^I^I^I.\{58\}/s/^\(.\{,55\}\) \(.*\)/\1^M^I^I^I\2/

I had to run the second line two or three times to wrap the very long lines.

Devdas helpfully translated the second one into English: "You have 3 tabs, followed by 58 characters, out of which you match the first 55 and put that bit in $1, and the capture the remaining in $2, and rewrite to $1 newline tab tab tab $2."

Here's a more detailed breakdown:

Line one:

%	Do this over the whole file
s/	Begin global substitute
^	Start at the beginning of the line
\(	Remember the result of the next match
.\{,65\}_	Look for up to 65 characters with a space at the end
\) \(	End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\)	Pattern #2 includes everything to the end of the line
/	End of matched pattern; begin replacement pattern
\1^M	Insert saved pattern #1 (the first 65 lines ending with a space) followed by a newline
^I^I^I\2	On the second line, insert three tabs then saved pattern #2
/	End replacement pattern

Line two:

%g/	Over the whole file, only operate on lines with this pattern
^^I^I^I	Lines starting with three tabs
.\{58\}/	After the tabs, only match lines that still have at least 58 characters (this guards against wrapping already wrapped lines when it's run repeatedly)
s/	Begin global substitute
^	Start at the beginning of the line
\(	Remember the result of the next match
.\{,55\}	Up to 55 characters
\) \(	End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\)	Pattern #2 includes everything to the end of the line
/	End of matched pattern; begin replacement pattern
\1^M	The first pattern (up to 55 chars) is one line
^I^I^I\2	Three tabs then the second pattern
/	End replacement pattern

Greedy and non-greedy brace matches

The real key is those curly-brace expressions, \{,65\} and \{58\} -- that's how you control how many characters vim will match and whether or not the match is "greedy". Here's how they work (thanks to Mikael for explaining).

The basic expression is {M,N} -- it means between M and N matches of whatever precedes it. (Vim requires that the first brace be escaped -- \{}. Escaping the second brace is optional.) So .{M,N} can match anything between M and N characters but "prefer" N, i.e. try to match as many as possible up to N. To make it "non-greedy" (match as few as possible, "preferring" M), use .{-M,N}

You can leave out M, N, or both; M defaults to 0 while N defaults to infinity. So {} is short for {0,∞} and is equivalent to *, while {-} means {-0,∞}, like a non-greedy version of *.

Given the string: one, two, three, four, five

,.\{},	matches , two, three, four,
,.\{-},	matches , two,
,.\{5,},	matches , two, three, four,
,.\{-5,},	matches , two, three,
,.\{,2},	matches nothing
,.\{,7},	matches , two,
,.\{5,7},	matches , three,

Of course, this syntax is purely for vim; regular expressions are unfortunately different in sed, perl and every other program. Here's a fun table of regexp terms in various programs.

Tags: linux, editors, regexp
[ 11:44 Dec 20, 2007    More linux/editors | permalink to this entry | comments ]

Emacs with Long Lines

I don't like composing text documents in word processors like Open Office. Call it a quirk if you like, but I find them intrusive: they take up a lot of CPU and memory, they take up a lot of window space for stuff I don't need while I'm writing (all those margins and rulers and toolbars and such) making it hard to compare two documents at once, and they tend to have intrusive focus behavior (like popping windows to the front when I didn't ask for it).

So when I need to write a paper (or a book), I prefer to compose in a text editor like vim or emacs, something that won't get in the way of my train of thought. When it's mostly written and ready to format, then I start up the big heavyweight word processor and import or paste the text into it.

(For those of you who think I'm insane and should just live in Open Office all day, the same problem comes up for people who do a lot of composing for web applications, such as an online blog, gmail, a web forum, or a wiki, and for people who want a choice of editor for their GUI mail app.)

Fine, but that introduces a problem. See, text editors have a fixed line width (typically 80 characters, though of course you can adjust this) and paragraphs are usually separated by blank lines (two newline characters together). Word processors expect each paragraph to be one long line for the whole paragraph, and line breaks are used as paragraph breaks (but you only want one of them, not two). How do you reconcile these two models in order to paste plaintext from an editor into a word processor?

Several years ago when I first encountered this problem, I investigated solutions in both vim and emacs (oddly enough, I'm an editor agnostic and equally happy in either one).

For vim, I never did find a solution to the problem, so that settled the editor choice for me. Perhaps some vim expert can let me know what I missed.

For emacs, I found longlines-mode, a hack which lets long lines appear to be wrapped while you're editing them even though they're really not. Apparently Wikipedia has this issue and some Wikipedia contributors use longlines-mode too. (That page also has brief notes on alternate solutions.)

I used longlines-mode for a long time, and it's more or less functional, but I was never really happy with it. It turns out to have some pretty annoying bugs which I was forever needing to work around, and it doesn't solve the blank-lines problem -- you still need to delete blank lines before or after pasting.

Yesterday I was working on an essay for a class I'm taking and decided I'd had enough of longlines-mode and wanted a better solution. I poked around and chatted with the nice folks on #emacs (hoping that someone had come up with a better solution, but no one knew of one) and based on some ideas they had, I came up with one of my own.

My new method is to edit the text file normally: line breaks where they look good, blank lines to separate paragraphs. When I'm finished writing and ready to paste, I run M-x wp-munge, which calls up a very simple function I wrote and added to my .emacs:

;; For composing in emacs then pasting into a word processor,
;; this un-fills all the paragraphs (i.e. turns each paragraph
;; into one very long line) and removes any blank lines that
;; previously separated paragraphs.
;;
(defun wp-munge () "un-fill paragraphs and remove blank lines" (interactive)
  (let ((save-fill-column fill-column))
    (set-fill-column 1000000)
    (mark-whole-buffer)
    (fill-individual-paragraphs (point-min) (point-max))
    (delete-matching-lines "^$")
    (set-fill-column save-fill-column) ))

So simple! Why didn't I think of doing it that way before?

Tags: linux, editors
[ 20:10 Feb 19, 2007    More linux/editors | permalink to this entry | comments ]

Linkifying with Regular Expressions

I had a page of plaintext which included some URLs in it, like this:

Tour of the Hayward Fault
http://www.mcs.csuhayward.edu/~shirschf/tour-1.html

Technical Reports on Hayward Fault
http://quake.usgs.gov/research/geology/docs/lienkaemper_docs06.htm

I wanted to add links around each of the urls, so that I could make it part of a web page, more like this:

Tour of the Hayward Fault
http://www.mcs.csu hayward.edu/~shirschf/tour-1.html

Technical Reports on Hayward Fault
htt p://quake.usgs.gov/research/geology/docs/lienkaemper_docs06.htm

Surely there must be a program to do this, I thought. But I couldn't find one that was part of a standard Linux distribution.

But you can do a fair job of linkifying just using a regular expression in an editor like vim or emacs, or by using sed or perl from the commandline. You just need to specify the input pattern you want to change, then how you want to change it.

Here's a recipe for linkifying with regular expressions.

Within vim:

:%s_\(https\=\|ftp\)://\S\+_<a href="&">&</a>_

If you're new to regular expressions, it might be helpful to see a detailed breakdown of why this works:

:

Tell vim you're about to type a command.

%

The following command should be applied everywhere in the file.

s_

Do a global substitute, and everything up to the next underscore will represent the pattern to match.

\(

This will be a list of several alternate patterns.

http

If you see an "http", that counts as a match.

s\=

Zero or one esses after the http will match: so http and https are okay, but httpsssss isn't.

\|

Here comes another alternate pattern that you might see instead of http or https.

ftp

URLs starting with ftp are okay too.

\)

We're done with the list of alternate patterns.

://

After the http, https or ftp there should always be a colon-slash-slash.

\S

After the ://, there must be a character which is not whitespace.

\+

There can be any number of these non-whitespace characters as long as there's at least one. Keep matching until you see a space.

_

Finally, the underscore that says this is the end of the pattern to match. Next (until the final underscore) will be the expression which will replace the pattern.

<a href="&">

An ampersand, &, in a substitute expression means "insert everything that was in the original pattern". So the whole url will be inserted between the quotation marks.

&</a>

Now, outside the <a href="..."> tag, insert the matched url again, and follow it with a </a> to close the tag.

_

The final underscore which says "this is the end of the replacement pattern". We're done!

Linkifying from the commandline using sed

Sed is a bit trickier: it doesn't understand \S for non-whitespace, nor = for "zero or one occurrence". But this expression does the trick:
sed -e 's_\(http\|https\|ftp\)://[^ \t]\+_<a href="&">&</a>_' <infile.txt >outfile.html

Addendum: George Riley tells me about VST for Vim 7, which looks like a nice package to linkify, htmlify, and various other useful things such as creating HTML presentations. I don't have Vim 7 yet, but once I do I'll definitely check out VST.

Tags: linux, editors, pipelines, regexp, shell, CLI
[ 12:40 May 14, 2006    More linux/editors | permalink to this entry | comments ]

Emacs: Typing dashes in html mode

What to do with a few extra hours in a boring motel with no net access? How about digging into fixing one of Emacs' more annoying misfeatures?

Whenever I edit an html file using emacs, I find I have to stay away from double dashes -- I can't add a phrase such as this one. If I forget and type a phrase with a double dash, then as soon as I get to the end of that line and emacs decides it's time to wrap to the next line, it "helpfully" treats the double dashes as a comment, and indents the next line to the level where the dashes were, adding another set of dashes. I've googled, I've asked on emacs IRC help channels, but there doesn't seem to be any way out. (I guess no one else ever uses double dashes in html files?)

It's frustrating: I like using double dashes now and then. And aside from the occasional boneheaded misfeature like this one, I like using emacs. But the dash problem been driving me nuts for a long time now. So I finally dug into the code to cure it.

First, the file is sgml-mode.el, so don't bother searching anything with html in the name. On my system it's /usr/share/emacs/21.4/lisp/textmodes/sgml-model.el. Edit that file and search for "--" and the first thing you'll find (well, after the file's preamble comments) is a comment in the definition of "sgml-specials" saying that if you include ?- in the list of specials, it will hork the typing of double dashes, so that's normally left out.

A clue! Perhaps some Debian or Ubuntu site file has changed sgml-specials for me, and all I need to do is change it back! So I typed

M-x describe-variable sgml-specials

to see the current setting.

Um ... it's set to "34". That's not very helpful. I haven't a clue how that translates to the list of characters I see in sgml-mode.el. Forget about that approach for now.

Searching through the file for the string "comment" got me a few more hits, and I tried commenting out various comment handling lines until the evil behavior went away. (I had to remove sgml-mode.elc first, otherwise emacs wouldn't see any changes I made to sgml-mode.el. If you haven't done much elisp hacking, the .el is the lisp source, while the .elc is a byte-compiled version which loads quicker but isn't intended to be edited by humans. For Java programmers, the .elc is sort of like a .class file.)

Commenting out these four lines did the trick:

  (set (make-local-variable 'font-lock-syntactic-keywords)
       '(("\\(<\\)! *--.*-- *\\(>\\)" (1 "!") (2 "!"))))
  ;; This will allow existing comments within declarations to be
  ;; recognized.
  (set (make-local-variable 'comment-start-skip) "\\(?:\\)?")

To regenerate the .elc file so sgml-mode will load faster, I ran emacs as root from the directory sgml-mode.el was in, and typed:

M-x byte-compile-file sgml-mode.el

All better! And now I know where to find documentation for all those useful-looking, but seemingly undocumented, keyboard shortcuts that go along with emacs' html mode. Just search in the file for html-mode-map, and you'll find all sorts of useful stuff.

For instance, that typing Ctrl-C Ctrl-C followed by various letters: u gets you an unordered list, h gets you an href tag, i an image tag, and so on, with the cursor positioned where you want to type next.

It doesn't seem to offer any basic inline formatting (like <i> or <em>), alas; but of course that's easy to add by editing the file (or maybe even in .emacs). To add an <em> tag, add this line to html-mode-map:

    (define-key map "\C-c\C-ce" 'html-em)

then add this function somewhere near where html-headline-1 and friends are defined:

(define-skeleton html-em
  "HTML emphasis tags."
  nil
  "" _ "")

Of course, you can define any set of tags you use often, not just <em>.

HTML mode in emacs should be much more fun and less painful now!

Update: If you don't want to modify the files as root, it also works fine to copy sgml-mode.el to wherever you keep personal elisp files. For instance, put them in a directory called ~/.emacs-lisp then add this to your .emacs:
(setq load-path (cons "~/.emacs-lisp/" load-path))

Tags: linux, editors
[ 21:48 Mar 29, 2006    More linux/editors | permalink to this entry | comments ]

Helpful Vim Tip: Finding Syntax for Colors

An upgrade from woody to sarge introduced a new problem with editing mail messages in vim: Subject lines appeared in yellow, against my light grey background, so they weren't readable any more.

Vim color files have always been a mystery to me. I have one which I adapted from one of the standard color schemes, but I've never been clear what the legal identifiers are or how to find out. But I changed both places where it said "ctermfg=Yellow" to another color, and nothing changed, so this time I had to find out.

Fortunately a nice person on #vim suggested :he synID (he is short for "help", of course) which told me all I needed to know. Put the cursor on the errant line and type: :echo synIDattr(synID(line("."), col("."), 1), "name")

That told me that the Subject line was syntax class "mailSubject". So I tried (copying other lines in my color file) adding this line:

hi mailSubject term=underline ctermfg=Red guifg=Red

and now all is happy again in vim land. I wish I'd learned that synID trick a long time ago!

Tags: vim, color, editors, tips
[ 09:59 Jun 22, 2005    More linux/editors | permalink to this entry | comments ]

Tweaking Emacs' Text Indent: Don't Indent So Aggressively

Encouraged by my success a few days ago at finally learning how to disable vim's ctrl-spacebar behavior, the next day I went back to an emacs problem that's been bugging me for a while: in text mode, newline-and-indent always wants to indent the first line of a text file (something I almost never want), and skips blank lines when calculating indent (so starting a new paragraph doesn't reset the indent back to zero).

I had already googled to no avail, and had concluded that the only way was to write a new text-indent function which could be bound to the return key in the text mode hook.

This went fairly smoothly: I got a little help in #emacs with checking the pattern immediately before the cursor (though I turned out not to need that after all) and for the function called "bobp" (beginning of buffer predicate). Here's what I ended up with:

(defun newline-and-text-indent ()
  "Insert a newline, then indent the next line sensibly for text"
  (interactive)
  (if (or (bobp)
          (looking-at "^$"))
      (newline)
      (newline-and-indent)
  ))
(defun text-indent-hook ()
  (local-set-key "\C-m" 'newline-and-text-indent)
  )
(setq text-mode-hook 'text-indent-hook)

It seems to work fine. For the curious, here's my current .emacs

Tags: linux, editors
[ 13:03 Feb 19, 2005    More linux/editors | permalink to this entry | comments ]

Turning off Ctrl-Space in Vim

One of those niggling problems that has plagued me for a long time: in the editor vim, if I'm typing along in insert mode and instead of a space I accidentally hit control-space, vim inserts a bunch of text I didn't want, then exits insert mode. Meanwhile I'm still merrily typing away, typing what are now vim comments which invariably end up deleting the last two paragraphs I typed then doing several more operations which end up erasing the undo buffer so I can't get those paragraphs back.

Ctrl-space inserts a null character (you can verify this by putting it in a file and running od -xc on it). I've done lots of googling in the past, but it's hard to google on strings like " " or even "space" or "null", and nobody I asked had a clue what this function was called (it turns out it re-inserts whatever the last inserted characters were) so I couldn't google on the function name.

Vim's help suggests that <Nul>, <Char-0>, or <C-^V > should do it. I tried them with map, vmap, cmap, and nmap, to no avail. I also tried <C-@> since that's a historical way of referring to the null character, googling found some references to that in vim, and that's how it displays if I type it in vim.

I finally found #vim on freenode, and asked there. Last night nobody knew, but this morning, p0g found the problem: I needed to use imap, not the map/vmap/cmap/nmap I'd been using.

So here, perserved for google posterity in case other people are plagued by this problem, is the answer:

imap <Nul> <Space>

For good measure, I also mapped the character to no-op in all the other modes as well:

map  <Nul> <Nop>
vmap <Nul> <Nop>
cmap <Nul> <Nop>
nmap <Nul> <Nop>

My current .vimrc.

Tags: vim, editors, tips
[ 10:24 Feb 17, 2005    More linux/editors | permalink to this entry | comments ]

Emacs Color Themes

A nifty emacs trick I learned about today: ColorThemes.

Instead of the old hacked-together color collection I've been using in emacs, I can load color-theme.el and choose from lots of different color schemes.

I added these lines to .emacs:

(require 'font-lock)
(if (fboundp 'global-font-lock-mode) (global-font-lock-mode 1))
(load "~/.emacs-lisp/color-theme.el")
(color-theme-ramangalahy)  ;; pick a favorite theme

The disadvantage is that color-theme.el is fifteen thousand lines long! So I'll probably make a local version that strips out all but the theme I actually use (then I can customize that).

The (global-font-lock-mode 1) tells emacs to use syntax highlighting on every file, not just certain types. So now I get at least some highlighting even in html files, though it still doesn't seem to be able to highlight like vim does (e.g. different colors for text inside <b> or <b> tags).

Tags: linux, editors
[ 17:57 Feb 03, 2005    More linux/editors | permalink to this entry | comments ]

Nifty emacs hack: how to wrap only certain files

For a long time I've wanted some, but not all, text and html files to line-wrap automatically in emacs. For instance, it drives me nuts when I edit a system configuration file and it wraps each long line, or when I edit an html file containing lots of long links and it keeps wrapping between the <a and the href=. But for files which are mostly text (such as these blog entries), I want line wrapping.

I'd been trying to do this with html-mode-hook and text-mode-hook, then checking the filename and calling (auto-fill-mode) if appropriate, but it wasn't working, because buffer-file-name isn't always defined at the time the mode hook is called. (No one seems to know why.) The buffer name seems to be defined at that point, but it doesn't contain path information so I can't say "Use wrapping for anything under ~/Docs" or "Don't wrap anything in /etc".

But with some help from sachac and the nice folks on #emacs I came up with a much better solution, and it's way simpler than the mode-hook approach: derived modes.

I set up two new modes, called html-wrap-mode and text-wrap-mode, which are the same as html-mode and text-mode except that they turn on auto-fill. Then I use the easy auto-mode-alist mechanism, which already does string matching on the filename, to call these modes, instead of the regular text and html modes, based on the extension or some other aspect of the file's pathname. Here's what I added to .emacs:

;; Want auto-fill-mode for some text and html files, but not all.
;; So define two derived modes for that, and we'll use auto-mode-alist
;; to choose them based on filename.
(define-derived-mode html-wrap-mode html-mode "HTML wrap mode"
  (auto-fill-mode))
(define-derived-mode text-wrap-mode text-mode "Text wrap mode"
  (auto-fill-mode))

(setq auto-mode-alist
      (cons '("\\.blx$" . html-wrap-mode)
      (cons '("Docs/.*.html$" . html-wrap-mode)
      (cons '("Docs/" . text-wrap-mode)
            auto-mode-alist) ) ) )

Here's my current .emacs.

I wonder if vim has a way to do this?

Tags: linux, editors
[ 22:30 Jan 13, 2005    More linux/editors | permalink to this entry | comments ]