trembits

Extracting audio from CD images

2012-01-19T06:35:00.005Z

I always wondered what the point was in cue/bin CD images compared to the ISO format, but never bothered to look it up.

A couple of days ago I was looking for music from an old Playstation game (which I own but couldn't be bothered to track down). I knew the music was on the disc as CD audio since I'd copied it to tape a dozen years or more ago, so I found and downloaded an image of the game from the web.

The thought then occurred -- how do I rip the music from this image? I usually use cdparanoia to get music off CDs, but could I point cdparanoia to a mounted CD image? The answer is no -- I would only be able to mount the data partition. You don't mount an audio CD before ripping music from it, cdparanoia just accesses the drive directly. Would I have to burn the image to a CD, then rip it back onto the machine? That's ridiculous.

Meanwhile the download finished and I saw that it was bin/cue rather than ISO. This is when I looked it up. It seems obvious now, but an ISO is just the ISO-9660 filesystem -- the data part. As such, an ISO file can't include CD audio. The bin/cue format is needed in order to include the data on the separate tracks of the CD. Converting the bin/cue to ISO would have lost me all the music.

So for future reference, extracting the audio (and the ISO at the same time) from bin/cue is as easy as this:

bchunk -w wild9.bin wild9.cue wild9

The -w there makes it write any audio as wave files so I can then convert to Flac. I assume it'd write them as raw PCM otherwise, but I've no desire to check.

Archiving mailing list messages but not replies to my own posts

2012-01-11T00:05:00.004Z

Sometimes I sign up to a mailinglist and want to see replies to my own threads but not have my inbox overtaken by everything else.

I can set up a filter in Gmail to label the mailing list posts and archive them, obviously, but then replies to my own threads (or threads I've participated in) are not obvious.

So let's make a more sophisticated filter, to match all messages with the relevant List-ID header except those which mention my messages in their References header.

After a few test queries it seems that search queries of the form References:*@segnus matches messages replying to things I sent from my laptop. The negative version -References:*@segnus appears to match the opposite. So it's not hard to add more negatives to avoid matching messages from my other machines and to build up a final query:

List:django-users.googlegroups.com -References:*@segnus -References:*@t900 -References:*@perihelion

This then goes in the "includes the words" box of the filter, and is set to apply a label and skip the inbox.

It's possible that other people send messages with Message-ID headers like mine and so I'd get some extra messages in my inbox, but I can live with that.

This doesn't seem to work quite correctly on existing messages. I'm guessing this is because Gmail sees the messages in the thread before any of them referenced me -- my original message or the existing conversation before I turned up -- and those messages match the filter, leading Gmail to lump the rest of the conversation in with that positive match and archive the lot. Fingers crossed it'll work with fresh incoming messages, though -- I'll update this post when I confirm.

Viewing HTML in mutt

2011-12-27T21:21:00.006Z

I use mutt to read my email, and every now and then I get sent a message in HTML format with no plain text alternative. I don't like to load these files in a browser, since it'd go ahead and fetch any messages, run scripts and so on with potential privacy risks. In other words, a message from a dubious source might phone home and confirm my email address or track me or whatever, just from opening their HTML in my browser.

So generally I just mind-parse the HTML. In more obfuscated cases (like the garbage output as newsletters by various websites) I manually pipe the message through lynx or similar.

Then when I reply to the message I have to pipe it through again if I want to quote something other than the HTML code.

I got fed up of this and looked for a solution. It consists of two parts -- changing up the entries in my mailcap file so that filtering the HTML to plain text is preferred to opening up a browser; and telling mutt to automatically filter text/html files using the rules it finds in the mailcap file. I've added some redundancy in to the mailcap entries so that it works both on my main machines (where I prefer pandoc since Markdown is nice to read, then I prefer lynx to either w3m or html2text, since lynx displays the links as references at the bottom) and on my phone (where only lynx is available).

In ~/.mailcap:

text/html; pandoc -f html -t markdown; copiousoutput; description=HTML Text; test=type pandoc >/dev/null
text/html; lynx -stdin -dump -force_html -width 70; copiousoutput; description=HTML Text; test=type lynx >/dev/null
text/html; w3m -dump -T text/html -cols 70; copiousoutput; description=HTML Text; test=type w3m >/dev/null
text/html; html2text -width 70; copiousoutput; description=HTML Text; test=type html2text >/dev/null

In ~/.mutt/muttrc:

auto_view text/html

Now HTML is automatically piped through one of those programs to turn it into plain text, and when I reply the quoted text is the plain text version rather than the raw HTML.

Different keys to push and pull to git repository

2011-12-07T09:01:00.002Z

I was getting fed up of typing in the password for my home server when pulling changes from a git repository up to the live server. But I'm not comfortable with putting my private key on the remote since other people have root. What to do?

I made two new SSH key pairs. One with a passphrase and one without. I told gitosis, which handles permissions for the git repositories on my home server, to accept my main passphraseless key with read/write access, the passphrased one also with read/write access, and the new passphraseless one with read-only access. I then uploaded the private keys for the passphrased key and the new passphraseless key to the remote host.

So even though they have one of my private keys, it's only good for read access to the repositories -- data which they already have anyway.

To tell SSH it has multiple keys you edit the config file and add an IdentityFile line for each key. But when connecting to the remote SSH server only the first acceptable key is tried. So if the passphraseless key is first everything will be fine when doing a read operation but gitosis will give a no permission error message when doing a write operation, and the other key won't be tried. If the key with the passphrase is first, the passphrase is asked for no matter whether it's a read or write operation.

So here's the solution: pretend to git using the pushurl option that we're pulling from and pushing to different hosts, then set up SSH to use different keys for these different hosts, but in fact then point them to the same host. Here's the configuration to illustrate.

Configuration in repository/.git/config:

[remote "origin"]
    fetch = +refs/heads/*:refs/remotes/origin/*
    url = gitosis@example.com:repository.git
    pushurl = gitosis@rw.example.com:repository.git

Configuration in ~/.ssh/config:

Host example.com
IdentityFile ~/.ssh/id_rsa.ro

Host rw.example.com
IdentityFile ~/.ssh/id_rsa.rw
HostName example.com

So the dummy hostname rw.example.com triggers SSH to use the passphrased private key at the correct hostname. A passphrase prompt appears when pushing but not when pulling.

Automatically switch main group when logging in to a host

2011-04-06T18:05:00.006+01:00

I have accounts on couple of hosts where I don't have root powers, and on some of them I am working alongside others. We're in the same group, but our primary groups are named after ourselves. Sometimes we need to edit files the other person has made and it can be a pain if one of us forgets to set the permissions such that the other user can edit the file.

Instead of having to remember to run chgrp semsorgrid newfile and then chmod g+w newfile every time we make a new file, we can change our primary group after logging in with newgrp semsorgrid and set the file creation mask to give the group maximum privileges with umask 002. But we still have to remember to do that when logging in.

I tried putting those commands in our shell configuration files (my .zshrc and his .bashrc) but then when logging in new shells are spawned recursively. The solution is to check which group we're in and conditionally run newgrp like this:

umask 007
if [ $(groups | awk '{print $1}') != "semsorgrid" ]; then
    exec newgrp semsorgrid
fi

This replaces the shell which was originally spawned with a new one with the primary group properly set, then when that shell initializes it skips the newgrp command since the primary group is already "semsorgrid".

Open online PHP reference from vim in a new split vim window

2011-03-25T14:41:00.008Z

Following the last post I went a bit further. I haven't decided which solution out of that and the following I like best yet.

This time I wanted the documentation to open in a new window within vim. That means grabbing the documentation HTML, cutting out what I don't want to see, rendering to plain text and putting the result in a new window.

Since the documentation is given as XHTML, as long as it's valid the safest way to chop out the bits I don't want is by parsing it as XML. So since PHP is my quick hacking language of choice I cooked up the following script and saved it as ~/bin/phpman-text.

#!/usr/bin/env php
<?php

if (!isset($_SERVER["argv"][1])) {
 fwrite(STDERR, "No keyword given\n");
 exit(1);
}

$xmlstring = @file_get_contents("http://php.net/" . urlencode($_SERVER["argv"][1]));
if ($xmlstring === false) {
 fwrite(STDERR, "Failed to fetch doc page\n");
 exit(2);
}

// remove default namespace
$xmlstring = preg_replace('%\bxmlns="[^"]*"%', "", $xmlstring);

$xml = @simplexml_load_string($xmlstring);
if ($xml === false) {
 fwrite(STDERR, "Failed to parse doc page\n");
 exit(3);
}

// get content div
$content = array_pop($xml->xpath("//div[@id='content']"));

if (is_null($content)) {
 fwrite(STDERR, "Couldn't find div with ID 'content'\n");
 exit(4);
}

// remove nav bars
foreach ($content->xpath("./div") as $nav)
 if (in_array("manualnavbar", explode(" ", $nav["class"])))
  simplexml_remove_node($nav);

// get new XML
$newxml = $content->asXML();

// run lynx
$lynx = proc_open("lynx -dump -stdin", array(array("pipe", "r"), array("pipe", "w"), array("pipe", "w")), $pipes);
if (!is_resource($lynx)) {
 fwrite(STDERR, "Couldn't run lynx\n");
 exit(5);
}

// poke new XML to lynx's stdin
fwrite($pipes[0], $newxml);
fclose($pipes[0]);

// get lynx's stdout
echo stream_get_contents($pipes[1]);
fclose($pipes[1]);
$erroroutput = stream_get_contents($pipes[2]);
fclose($pipes[2]);

// close lynx
$returnval = proc_close($lynx);

// check return value
if ($returnval != 0) {
 fwrite(STDERR, "lynx exited with code $returnval -- error output follows.\n$erroroutput");
 exit(6);
}

exit(0);

function simplexml_remove_node($node) {
 $domnode = dom_import_simplexml($node);
 $domnode->parentNode->removeChild($domnode);
}

?>

Then, with help from the vim wiki, I came up with this, to be added to my .vimrc:

function! OpenPhpFunction (keyword)
 exe "12new"
 exe "silent read !phpman-text ".substitute(a:keyword, "_", "-", "g")
 exe "set buftype=nofile bufhidden=delete filetype=php readonly"
 exe "1"
endfunction
autocmd FileType php map <buffer> K :call OpenPhpFunction('<C-r><C-w>')<CR>

This mostly works. It loads the PHP documentation into a temporary new buffer and sets the filetype to PHP so that bits of text in <?php ?> tags are highlighted as PHP code. There are a few bad things -- some of the blocks of PHP code in the manual don't have the end tag and so highlighting continues beyond the end of the code, some of the manual pages (probably due to dodgy comments) aren't valid XML and so won't go through the script, and lynx's output isn't completely ideal.

Open online PHP reference from vim

2011-03-09T17:18:00.005Z

Pressing K when over a keyword in vim usually opens the keyword's manpage. In Python it invokes pydoc instead. I wanted it to open PHP's online documentation when I press it over a function name in PHP. Easy.

I wrote a tiny shell script in my ~/bin directory first, phpman:

#!/bin/sh sensible-browser http://php.net/"$*"

I then added to my .vimrc the line

autocmd FileType php set keywordprg=phpman

Done. And using sensible-browser has the added bonus that it'll run whatever my preferred graphical browser is when I am in X or a textmode browser when I'm not.

Merging files with unison and vimdiff -- update

2010-10-10T01:31:00.004+01:00

I've just updated my post from February about merging files with unison and vimdiff -- I figured out how to do so without launching a new terminal emulator. The solution uses screen. See the updated post.

Committing part of a file

2010-08-30T15:28:00.003+01:00

A fantastic feature of git I only found a few days ago is committing just part of a file. Usually I have the foresight to use git stash but sometimes that doesn't occur or it's not practical for whatever reason, such as only realizing after the edits that the changes in a particular file should be two commits rather than one.

When this happens you can interactively choose which parts of the changes to stage for committing. To do that:

git add -p file.php

Git then shows each part of the patch in sequence, asking what you want to do with them. You can stage the change, not stage it, split it into smaller chunks, decide later or even edit the patch manually.

Options for lpr

2010-08-24T13:46:00.005+01:00

I keep on forgetting these so it's about time I wrote down the ones I use most often.

You can list the current options (it definitely includes the options I've set to default and I'm not sure if it also shows all other default options or just those which the Cups Gui on my machine has set) with lpoptions. I've set it to print full duplex by default in the past with lpoptions -o sides=two-sided-long-edge so that shows up when I run loptions and everything's printed double sided by default.

Sometimes I don't want to print double sided and so I'll do lpr flyer.pdf -o copies=8 -o sides=one-sided to print off a bunch, one per sheet. (Could also do -#8 as on option to lpr for eight copies but that requires escaping.)

I can also choose a printer other than the default one such as the colour laser with lpr poster.pdf -P renoir


To do a quick printout of a PDF 2-up and reordered so it can just be folded up into a pamphlet fresh out of the printer I can do this:
pdftops somebook.pdf - | psbook | lpr -o number-up=2 -o sides=two-sided-short-edge
To print an A4 page on an A3 printer and fill the page I should (not tested) be able to do this:
lpr a4poster.pdf -o media=A3 -o fit-to-page
A landscape option might be necessary, I guess depending on the input. I'll come back and confirm what works at some point.



copies=num
Number of copies. Also possible to use -#num but that might need escaping.

landscape
Rotates 90 degrees but doesn't resize the page. Printing an A4 portrait PDF will not fit on A4 media but it doesn't complain, it just cuts off what doesn't fit.

number-up=num
The number of pages to print on each side.

number-up-layout=lrtb|lrbt|rltb|rlbt|tblr|tbrl|btlr|btrl
Choose how the pages are laid out when printing multiple pages on a side.

sides=one-sided|two-sided-long-edge|two-sided-short-edge
Choose how duplex works -- how the paper is turned over before the next page is printed. two-sided-long-edge is desirable for 1-up printing, two-sided-short-edge for 2-up.

media=A4|whatever
Choose the media size. If the printer only does as large as A4 and you tell it to use A3 and fit-to-page it'll print on A4, sideways and too big. Doesn't sound useful but it potentially is.

page-set=odd|even
Print only odd or even pages.

outputorder=normal|reverse
This takes effect after number-up so you still get 1234 on a page, not 4321

cpi=charsperinch
The number of characters per inch when printing plain text (default 10).

lpi=linesperinch
The number of lines per inch when printing plain text (default 6).

columns=num
The number of columns to format when printing plain text.

page-(left|right|top|bottom)=margin
The margin sizes in points when printing plain text.

Collate=true|false
Whether or not to collate the output pages -- that is, print all pages of the first copy followed by all pages of the next. By default it won't -- it'll print all copies of page 1, then all copies of page 2 and so on.

page-ranges=page,page,start-end
Choose which pages to print. This takes effect after number-up so the second page is 5678 when printing 4-up. They're always printed in ascending order, regardless of the order the ranges are given in.



Nice bit of piping
2010-07-07T23:40:00.005+01:00
I just made a nice little pipeline. I love the Linux shell.
cat file | tr A-Z a-z | tr -c a-z "\n" | sort | uniq -c | sort -nr
That gives a nice ordered count of the most used words in the file. It's pretty dumb -- it breaks words at anything which isn't an alphabetical character -- but it does the trick.
The first tr changes uppercase characters to lowercase, the second changes anything which isn't alphabetic to a newline. Then the words are sorted alphabetically, then uniq counts successive identical lines and outputs the count with the word, then that list is sorted numerically in descending order. Lovely.



Make a single-line XML stream pretty in vim
2010-06-23T16:42:00.002+01:00
Usually I view XML in Firefox as that's the quickest way to get a nice tree I can explore. But when I might want to edit it until today I've been loading it into vim and then if necessary adding newlines manually or using a fairly dumb regex to do it for me. Today I made a slightly more sophisticated one.
:%s/>\s*</>\r</g adds the newlines in decent places and then gg=G indents it all.



Moving the pointer to the top left corner of the screen
2010-06-23T11:45:00.007+01:00
I wanted a keystroke to send the pointer to the top left of the screen. Usually I do this by flipping to the other screen and back but on my laptop (with only one screen) this doesn't work.
The solution was to use xmousepos to get the mouse position (want to know which screen we're on) and xte to set it. These are part of the xautomation package in Ubuntu/Debian.
But how big are the screens? I'm using xrandr and assuming that the resolutions listed are in order left to right and no screens are up or down or rotated or anything crazy like that.
Here's my script.
#!/bin/sh

# get current horizontal mouse position
mousex=$(xmousepos | awk '{print $1}')

# walk through widths of screens (assuming they're in order left to right) until 
# the mouse is on the current screen, building offset from left
offset=0
for width in $(xrandr | grep '*' | awk '{print $1}' | sed -r 's/x.*//'); do
 [ $mousex -lt $(expr $offset + $width) ] && break
 offset=$(expr $offset + $width)
done

# move the pointer to the top left of the screen
xte "mousemove $offset 0"


xte can send other fake input to X too such as clicking, pressing and releasing keys and so on. Potentially very useful.



Giving standard input to a program run in a shell script
2010-06-09T17:01:00.005+01:00
I modified my "lock screen" bash script just now to also work when X isn't running -- that is, I'm on a virtual console or shelled into my machine.
The script as it was did this: use purple-remote to set my status to "away" (if it was "available" -- otherwise don't touch it), then run i3lock to lock the X screen, then switch the screen off to save power, then when i3lock exits restore my IM status if it was changed.
I added an if [ $DISPLAY ] test to branch for X and non-X, putting vlock in the place of the i3lock and xset commands for the non-X branch. It didn't like this, though -- vlock complained that it wasn't a virtual console.
In order to get it working all I had to do was redirect the script's standard input to vlock like this: vlock -a </dev/stdin.



Printing from the command line
2010-05-25T16:56:00.002+01:00
I just selected some instructions from a Pidgin window and then typed xsel | lpr in a terminal. Fantastic.



Debugging an already-running process
2010-05-25T14:38:00.006+01:00
I was getting a segfault in pms, an ncurses application I contribute to, so I restarted the program within gdb. But since the bug seems to be something to do with resizing the console window the segfault never appeared again -- gdb doesn't pass the resize signal (or perhaps the new dimensions) on to the program running within it.
I Googled around for a way to run gdb in a separate console window from the ncurses application. It's rather simple: change to the directory containing the binary so it can find the right one to debug against and run gdb -p $(pidof pms).
That'll attach gdb to the already running process and pause it. Type continue in the gdb window and await the crash.



Ruby dependencies
2010-04-30T14:02:00.004+01:00
After getting a message that a module which was definitely installed was not found, a colleague eventually had the idea of checking the require path by running puts $: in irb. Sure enough there was a different module with the same name earlier in the path than the one gem had installed. I uninstalled Ubuntu's own libxml-ruby and libxml-ruby1.8 and all was well. Interestingly, though, one of these (I forget which) was required to install the gem to begin with -- I guess it needed some header it provided. There's no separate -dev package (at least in this old release of Ubuntu).



Which package did a particular file come from
2010-02-09T18:12:00.005Z
I sometimes find myself trying to figure out which package a particular file came from. It's usually a binary I want available on a different Debian-based machine and an aptitude search binaryname isn't returning anything.
I think I've done this a few times before with dpkg-query commands of some kind but it was never simple enough to remember. I Googled the problem again today and came up with an article introducing dlocate. It just has to be installed (its package is helpfully called "dlocate") and then to find out where pdfimages came from I just have to type dlocate pdfimages.
But that gave me three lines of output -- I have a python script called pdfimages.py which came from python-reportlab, a gzipped manpage which came with poppler-utils and the binary I was actually looking for, also from poppler-utils. So I could have made the query more specific by giving a full path to the file -- quickest way with a binary is of course using which: dlocate $(which pdfimages).



Merging a unison conflict with vim
2010-02-04T13:30:00.008Z
I use unison to synchronize various files. If I've edited the same file on both ends I can pick a version to keep or, until now, I aborted, manually merged them with vimdiff or some other tool and then started synchronizing again.
But unison is able to call an external tool to merge for you. I figured out how to get it to use vimdiff -- add the option -merge "Name * -> urxvt -e vimdiff CURRENT1 CURRENT2".
I've used urxvt rather than my usual urxvtcd because urxvtcd immediately forks and unison continues as soon as the process it runs exits. So why did I run a terminal emulator at all rather than just running vimdiff in the existing terminal? It won't display. Not sure why -- probably to do with it not realizing it has a virtual terminal to output on or something.
Update on 2010-10-10: I had another shot at figuring it out. I tried using screen -- "Must be connected to a terminal". Then I tried using ssh -t and screen -- same issue (but I think the error was from SSH this time rather than screen). Finally I found an option in screen's manual page which can start a screen session detached but yet not fork. That'll do. So now in default.prf (which my other unison config files include) I have these two lines:
merge = Name * -> screen -DmS unisonmerge vimdiff CURRENT1 CURRENT2
merge = Name .* -> screen -DmS unisonmerge vimdiff CURRENT1 CURRENT2

The second line is there because I found just now that dotfiles weren't included when only the first line was present.
So here's what happens when a merge is now requested. A new screen session is started with the session name "unisonmerge". That's started detached but the process doesn't fork as it usually would and so unison stops, waiting. We get the shell to stop the process and give us a prompt by pressing ^Z, then attach the new screen session with screen -RS unisonmerge. The vimdiff instance is running in that -- we merge the files (modifying only the one we want to keep, which will end up on both machines) and then exit. The screen session ends and we're back to the prompt. Then we bring unison back to the foreground with fg. Unison continues.
So now I can run unison on a headless server over SSH, merging when necessary.



Postpone current command and run another first in zsh
2010-01-14T14:10:00.003Z
Sometimes I am halfway through writing a command and realize I need to run something else first. Just now I was writing a git commit command and realized I should edit the TODO file first. So usually I either ^U to clear the command and type vim TODO before writing the commit command again or maybe open a new terminal or maybe even ^A to go to the start of the line and add vim TODO; to the beginning.
But zsh has a buffer stack which comes in useful here. The default key binding for push-line is ESC q. This pushes the current buffer (command in progress) onto the stack and clears the buffer, then pops from the stack next time the prompt is reached. So halfway through typing a commit command I do ESC q, type vim TODO, make whatever changes and exit and I'm back at a commandline with my half-typed commit command restored.



Transcription of Stream of Consciousness by Textures
2010-01-12T17:09:00.001Z
Forgot to post that I've transcribed Stream of Consciousness. It has some crazy fast bits.



git stash
2010-01-12T17:01:00.002Z
I'm currently working on a project using git as my version control system.
Often I'm halfway through making some change or other and an unrelated bug becomes apparent. It's an easy fix and so I go ahead and fix it but then I can't make a clean commit. I found out today how to handle this.
Save the work in progress before the unrelated bug is fixed and run git stash
The source tree is now clean -- the unfinished changes have disappeared. Reload the file in vim and fix the small bug.
Commit the patch
Restore (and merge if necessary) the stashed tree with git stash pop



Transcription of Drive by Textures
2009-12-22T01:50:00.003Z
I learnt Denying Gravity the other day (but didn't transcribe it). Today I thought since I know that and Regenesis maybe I'd learn the whole album. Transcribed Drive just now -- it's up on my tabs page



Vim can read zip files
2009-12-16T15:34:00.005Z
I accidentally opened a zip file with vim just now. It gives a directory listing just like when opening a directory. Fantastic.



Transcription of Old Days Born Anew by Textures
2009-12-04T11:04:00.005Z
I finished transcribing Old Days Born Anew by Textures to guitar tablature. It's up with my other plaintext tabs at my tabs page.



Ranges in vim
2009-11-25T14:02:00.003Z
I hadn't used ranges in vim (other than % -- the whole file) until now. Thought it was about time I found out how to use them.
. is the current line, $ is the last line. Numbers mean that line so 1 is the first line in the file. That's the most important stuff -- the full list is at :he range.
They can be followed with + or - which adds or takes one from the line number, or the plus or minus can be followed with a number to add or subtract that many.
Ranges can be just one line or from,to. So the range I need to use right now, current line until the end of the file, is .,$ and so to do my replacement I type :.,$s/"#ffbaba"/errorcolour/g.



Get everything from an FTP server
2009-11-18T11:24:00.006Z
When there's no shell access (and so no SCP) FTP is unavoidable. I'd usually do this with a Gui FTP program like Filezilla but I thought it was about time I figured out how to do it on the command line.
wget does the trick but I noticed it got stuck once or twice and then didn't move. When that happened I just hit ^C to stop it and then ran again with the c switch (continue). That switch doesn't hurt the first time either so I'll include it below:
wget -cr --user username --password password ftp://server



imgur
2009-10-29T17:54:00.004Z
It's a bit like sprunge -- a pastebin for images. I wrote a bash script similar to my sprunge one to upload an image (given by filename) and return (and put on the clipboard) its new URL. The URL to delete it is given on stderr.
#!/bin/bash
apikey="<API KEY>"
if [ $# -lt 1 ]; then
 echo "no file specified" >&2
 exit 127
fi
response=$(curl -F "key=$apikey" -F "image=@$1" http://imgur.com/api/upload.xml 2>/dev/null)
if [ $? -ne 0 ]; then
 echo "request failed" >&2
 exit 1
fi
url=$(echo $response | sed -r 's/.*<original_image>(.*)<\/original_image>.*/\1/')
deleteurl=$(echo $response | sed -r 's/.*<delete_page>(.*)<\/delete_page>.*/\1/')
echo $url
echo "Delete page: $deleteurl" >&2
if [ $DISPLAY ]; then
 { which xsel >/dev/null 2>/dev/null && echo -n $url | xsel; } \
  || { which xclip >/dev/null 2>/dev/null && echo -n $url | xclip; } \
  || echo "haven't copied to the clipboard: no xsel or xclip" >&2
else
 echo "haven't copied to the clipboard: no \$DISPLAY" >&2
fi


Update: a later version of this script is now featured on the imgur site. At the moment it's at the bottom of the tools page.



Package a source tree ignoring .svn directories
2009-10-29T14:47:00.002Z
Simple: tar cvzf py-translate-svnr8.tar.gz --exclude=.svn py-translate



Swim/swam/swum
2009-10-26T10:17:00.005Z
I couldn't figure out just now whether to use "swam" or "swum".
Googled around a bit for something authoritative and came up with a page from A semantic approach to English grammar by Robert M W Dixon.
This morning I swam. I have swum today. I swam already. I've already swum. Should have known it'd be that simple, really -- it seems if you use the verb "have" as an auxiliary you use the "swum" form. The technical term for when you use "swum" seems to be the "previous perfective aspect", both in the present ("I have swum") and past ("I had swum") -- otherwise use "swam" (or "swim", but it's obvious when to use that).



Print a particular line of a file
2009-10-16T09:46:00.003+01:00
Saw this on commandlinefu today.
I'd been doing command | head -5 | tail -1 to print the 5th line but it makes more sense to use sed:
command | sed -n 5p



Yanking URLs from the console
2009-10-15T16:01:00.002+01:00
It's a pain to select URLs manually from the console with the mouse for pasting into Vimperator or whatever other program. On the PMS IRC channel today smeea mentioned a perl extension for urxvt which scans for URLs and allows them to be yanked or put straight into a browser. It's on another fella called Bart's blog.
Very handy. Now I can ignore the mouse even more.



find and xargs
2009-10-06T15:47:00.005+01:00
How hard can it be to send a bunch of filenames from find as arguments to a shell command? It turns out that it's much more difficult than it should be if you want to add another argument after the list of filenames.
The problem today was that my colleague and I wanted to find all images with a particular name and send them to Imagemagick to put them all together. The Imagemagick command for that looks like

montage -size 1 -geometry 66x50 image*.png montage.png
The issue is that Imagemagick isn't flexible. The output filename has to be the last argument and the input files can't be passed by standard input. The contenders for solving the problem are find's -exec option and xargs.
So the first thing we tried was this:

find . -name "image*.png" -exec montage -size 1 -geometry 66x50 {} montage.png +

but this gives a "missing argument to `exec'" error -- find doesn't allow extra arguments after the placeholder.
Then we tried using xargs:

find . -print0 -name "image*.png" | xargs -0 -I {} montage -size 1 -geometry 66x50 {} montage.png

but this runs the command once for each input unless we increase xargs's -L (max input arguments per command) option back up from 1 (which is set by using the -I option -- no idea why) to some arbitrary large number. This seems like a bad way to do things.
The solution we came up with was to add an extra argument before the stream gets to xargs:

find . -print0 -name "image*.png" | cat - <(echo -ne "montage.png\0") | xargs -0 montage -size 1 -geometry 66x50
This is using process substitution to put montage and a null character as a file descriptor input to cat, which adds that to the end of the first argument, its standard input.
It's still a mess -- any better solutions?



Herestrings in the shell
2009-09-29T16:35:00.003+01:00
Like heredoc syntax there is also herestring.
For a heredoc you'd do something like
cat >file <<EOF
This is a file
all this text will get written to the file
This dir contains
$(ls)
EOF

or, if expansion (command, parameter and arithmetic) isn't wanted, quote the word at the start: cat >file <<"EOF".
Herestrings are a quicker way than using echo and piping to a command -- for instance to get torrent info I can do ssh server deluge --ui=console <<<info rather than echo info | ssh server deluge --ui=console.



Put SSH in the background
2009-09-09T02:45:00.004+01:00
While looking for something unrelated in SSH's manpage I came across something I didn't know before: when logged into a remote machine various escape sequences are supported to talk to the local end of the connection.
It's possible to do fancy things like add port forwardings or remove existing ones but I think the only one I'll now use commonly is ~^Z, which is like sending ^Z to your local shell and backgrounds the SSH process. You get your local shell back, can run whatever commands you need and then foreground SSH again with fg.
The escape character (~ is the default) has to come after a newline character to be recognized as special.
I've entirely forgotten what I was actually looking for in the manual.



Brackets in bash
2009-09-08T14:18:00.006+01:00
Was just having a problem with a command which worked in the Z shell not working in bash and had to brush up on my knowledge of the difference between different types of brackets. Found a very useful article which explains it all.
I was trying to use braces to group commands but have them execute in the current shell. Z shell liked it but bash didn't. The solution to my problem was to put whitespace around the braces and to terminate the last command with a semicolon before the closing brace. Apparently this stuff is required in bash for "historical reasons".



Sprunge
2009-09-01T16:18:00.012+01:00
Sprunge is lovely. It's a pastebin but without all the crap. Here's my sprunge script:
#!/bin/bash
url=$(curl -F 'sprunge=<-' -H "Expect: " http://sprunge.us 2>/dev/null)
if [ $? -ne 0 ]; then
 echo "request failed" >&2
 exit 1
fi
if [ $# -gt 0 ]; then
 url="$url?$1"
fi
echo $url
if [ $DISPLAY ]; then
 { which xsel >/dev/null 2>/dev/null && echo -n $url | xsel; } \
  || { which xclip >/dev/null 2>/dev/null && echo -n $url | xclip; } \
  || echo "haven't copied to the clipboard: no xsel or xclip" >&2
else
 echo "haven't copied to the clipboard: no \$DISPLAY" >&2
fi

The empty Expect header is to get around a bug which rears its head if the request goes through a proxy like Squid.

This will pastebin the standard input and output the URL at which you can retrieve it. Additionally, if it can it'll put it on the X selection so you can paste it into IRC or whatever.
Give a filetype as an argument if you want the code highlighted. All that does is add a parameter to the URL which is printed and put on the clipboard.
So I might run some command | sprunge to show the command's output to someone or sprunge python <script.py to show a buggy Python file to someone.



Clipboard from the command line -- xclip versus xsel
2009-09-01T15:45:00.009+01:00
Until last week I'd been using xclip to set and retrieve the X selection and X clipboard. But the syntax is a bit nasty -- to output the clipboard it's xclip -selection clipboard -o. (I have since found out, thanks to the comments on this post, that it's not quite so bad, the options can be abbreviated as long as it's non-ambiguous.) I came across xsel and decided to give it a go. They both do roughly the same thing, but which is better?

Put simply, xsel is a little more advanced and has a nicer syntax.
They can both be installed easily in Ubuntu with sudo aptitude install xclip and sudo aptitude install xsel.
First of all, it's important to realize there are three selections X keeps track of: the primary X selection, the secondary X selection (but I've never actually seen this used) and the X clipboard.

The (primary) X selection is what is set when some text is selected and is usually retrieved when the middle mouse button is clicked.
The X clipboard is what most programs set when the copy command is used, and what they retrieve from the owner when the paste command is used.
I used the word "owner" there – in the X environment the program in which text is copied/selected takes "ownership" of the X clipboard or X selection. When another program asks for the contents of the clipboard or selection, X tells it which program is the owner and then the querying program asks the owning program for the contents directly. This means when the owning program exits the contents of the clipboard and selection are lost, unless something else takes over selection ownership. There exist system-tray type programs for this purpose.
Both xsel and xclip use the primary X selection by default. To set it:

some command | xsel

or

some command | xclip
The same so far. To retrieve it:

xsel

or

xclip -o

No output flag is required for xsel since it detects what stdin and stdout are and picks input or output mode automatically. See its manpage.
To set the X clipboard, xsel's command is much shorter:

some command | xsel -b

as compared to xclip's equivalent

some command | xclip -selection clipboard
And to retrieve the X clipboard

xsel -b

versus

xclip -selection clipboard -o
I often want to put the contents of a file into the selection. I do that with xsel <file.
Other features
I've never found any of xclip's other features useful – it can be set to hold the selection until it has been retrieved n times and then delete it, for instance. On the other hand, some of xsel's options look very useful. Some examples: some command | xsel -a appends text to the selection, xsel -d deletes the selection and xsel -k keeps (takes over) the selection so that it won't be lost when the program which set it exits.
Getting at the selection and clipboard from vim
In vim you can read and write to both of these by using the "+ (X clipboard) and "* (X selection) registers. For example, to copy the current line to the X clipboard type "+yy or to paste from the X selection type "*p.



Datetimes in PHP and MySQL
2008-09-25T19:45:00.001+01:00
Until a few days ago, I’d never properly figured out exactly how to handle datetimes in PHP – whether to use date() or gmdate() in various situations, the best thing to do when dates are going to or coming from a database and how to handle timezones.
I just sort of winged it, mostly using date() and hoping timezones would just behave. But here’s what I’ve now found out.

Storing to and retrieving from MySQL
Datetimes are stored in MySQL very simply – the timezone doesn’t affect how they are stored or retrieved at all. And just so we’re clear, the timezone isn’t stored alongside a date either – only the familiar YYYY-MM-DD HH:MM:SS format. For this reason, it’s clearly essential to always store datetimes in a single timezone, and the best contender is UTC since this gets around daylight saving time issues and is… well, universal. (In fact, Unix timestamps aren’t true representations of UTC, but they’re damn close.)
Whether to store these UTC datetimes as Unix timestamps (number of seconds since the start of 1970, UTC) or in columns of the MySQL datetime datatype is up to you. I personally opt to use datetime fields since I often dig around with a commandline MySQL client – to understand the timestamps I’d have to put them through MySQL’s FROM_UNIXTIME() every time.

So given a Unix timestamp, perhaps generated by PHP’s time() or strtotime(), you need to give it to MySQL in the format it wants, Y-m-d H:i:s. You don’t want MySQL to store the the time in your local timezone, so use gmdate().
To get it back you’ll almost certainly want to put it through strtotime() since a timestamp is the easiest form in which to manipulate a datetime in PHP. But just sticking the date string from the database through that function will make PHP interpret it as being in the server’s current timezone. This is no good if the server is not always in UTC, so specify the timezone: strtotime($datestring . " UTC").

Timezones in MySQL
As noted above, MySQL doesn’t care about timezones when storing or retrieving datetimes. The only time MySQL cares about timezones is when certain built-in date functions are used – NOW(), for instance, returns the current datetime in the currently set timezone.
I don’t actually use MySQL’s date functions very often, preferring to let PHP do the work. Check the MySQL manual to find out about the UTC functions if you need them, but the most important are UTC_TIMESTAMP(), which returns the current datetime as a string, and UNIX_TIMESTAMP(), which returns a Unix timestamp, funnily enough.
What if you need to use NOW() to get the current local datetime, but the server isn’t using the timezone you want? You can override the timezone which will be used for that session with a query such as SET time_zone = '+01:00';. Set the timezone as soon as you connect to the database to whatever timezone PHP is in (info on setting that below) with mysql_query("SET time_zone = '" . date("P") . "';"); soon after your mysql_connect() call.

Timezones and daylight saving time in PHP
In order to show dates in suitable local time, it’s important to set the timezone. You can set the timezone which suits the site best (for instance a UK business would use UK time) or allow the user to choose their own. You could even try to detect the user’s location and apply a timezone based on that, but in my experience this is dodgy at best.
I have some UK-centric sites hosted on a server in the USA and so before I generate any output I set the timezone with date_default_timezone_set("Europe/London"); – the supported timezones are shown at php.net/timezones. Daylight saving time is worked into it – use the timezone of a city which observes the daylight saving time rules you want your site to observe and everything will be handled for you. If you want to force a particular timezone with no daylight saving, there are suitable fixed timezones under the “Others” header such as UTC and Etc/GMT+5.
One thing which I found very confusing at first is how PHP handles daylight saving time when you give it a datetime in the past or future. It turns out I needn’t have been confused – maybe it’s blindingly obvious, but it just didn’t click with me – PHP simply uses the state of daylight saving time at whatever date it is considering. For example, given a London timezone and a couple of timestamps $xmas2008 = strtotime("2008-12-25 08:00"); $birthday = strtotime("1986-07-25 noon");, if you put them through date() you get a GMT datetime for $xmas2008 and a BST datetime for $birthday. You can verify this by outputting a full-format date including an indication of timezone.

What this means is that given a correctly set timezone, the output of date() given a correct Unix timestamp will always make sense, no matter when the timestamp was stored, what date it represents nor what the current date is. As an illustration, say you publish a blog post just after midnight once a week all year. Even though the state of daylight saving time changed twice (if you’re in an area which observes DST, at least), the blog posts will always show as just after midnight. If you then change the timezone to a region at roughly the same longitude but which doesn’t observe DST (Iceland is an example of an always-GMT region), you’ll see that some of your blog posts show as just after 11pm – these are the ones you made during DST – and the rest still show as just after midnight. Use date() with a format including the timezone to see for yourself.

User-entered datetimes
The only thing left is taking datetimes from user input. The best option here is to feed the user input through strtotime() without a timezone suffix, and hence have PHP interpret the time in whatever timezone applied (or will apply) on the given date. If you allow your users to set a custom timezone, this means the datetimes will (should) be interpreted exactly as the user intends. The other option would be to demand UTC input, but not everyone would know, if the date is a few months away, whether to shift the date by an hour or not and, if so, in which direction to shift it.

To sum up

 Set suitable timezones in both PHP and MySQL with date_default_timezone_set("Europe/London"); mysql_query("SET time_zone = '" . date("P") . "';");
 Store and manipulate datetimes in PHP as Unix timestamps, converting them for display with date() only when necessary. I like to give HTML datetimes a title attribute with a full-format date such as Y-m-d H:i:s P (you can use T instead of P to show the timezone’s abbreviation rather than its offset from UTC) so things are totally clear.

 To store a datetime in MySQL, store it as a timestamp (integer) directly or store the output of gmdate("Y-m-d H:i:s", $timestamp) to a column with the datetime datatype – gmdate() rather than date() to ensure a UTC datetime is stored
 To retrieve a datetime stored in a datetime column, use strtotime($datestring . " UTC") to ensure it is interpreted as UTC

 Put datetime input from users directly into strtotime() to get a timestamp, then store or display it as above

I hope this helps someone who is as confused as I was. If you have any other datetime-related tips, go ahead and leave a comment.