Extracting audio from CD images

I always wondered what the point was in cue/bin CD images compared to the ISO format, but never bothered to look it up.

A couple of days ago I was looking for music from an old Playstation game (which I own but couldn't be bothered to track down). I knew the music was on the disc as CD audio since I'd copied it to tape a dozen years or more ago, so I found and downloaded an image of the game from the web.

The thought then occurred -- how do I rip the music from this image? I usually use cdparanoia to get music off CDs, but could I point cdparanoia to a mounted CD image? The answer is no -- I would only be able to mount the data partition. You don't mount an audio CD before ripping music from it, cdparanoia just accesses the drive directly. Would I have to burn the image to a CD, then rip it back onto the machine? That's ridiculous.

Meanwhile the download finished and I saw that it was bin/cue rather than ISO. This is when I looked it up. It seems obvious now, but an ISO is just the ISO-9660 filesystem -- the data part. As such, an ISO file can't include CD audio. The bin/cue format is needed in order to include the data on the separate tracks of the CD. Converting the bin/cue to ISO would have lost me all the music.

So for future reference, extracting the audio (and the ISO at the same time) from bin/cue is as easy as this:

bchunk -w wild9.bin wild9.cue wild9

The -w there makes it write any audio as wave files so I can then convert to Flac. I assume it'd write them as raw PCM otherwise, but I've no desire to check.

Viewing HTML in mutt

I use mutt to read my email, and every now and then I get sent a message in HTML format with no plain text alternative. I don't like to load these files in a browser, since it'd go ahead and fetch any images, run scripts and so on with potential privacy risks. In other words, a message from a dubious source might phone home and confirm my email address or track me or whatever, just from opening their HTML in my browser.

So generally I just mind-parse the HTML. In more obfuscated cases (like the garbage output as newsletters by various websites) I manually pipe the message through lynx or similar.

Then when I reply to the message I have to pipe it through again if I want to quote something other than the HTML code.

I got fed up of this and looked for a solution. It consists of two parts -- changing up the entries in my mailcap file so that filtering the HTML to plain text is preferred to opening up a browser; and telling mutt to automatically filter text/html files using the rules it finds in the mailcap file. I've added some redundancy in to the mailcap entries so that it works both on my main machines (where I prefer pandoc since Markdown is nice to read, then I prefer lynx to either w3m or html2text, since lynx displays the links as references at the bottom) and on my phone (where only lynx is available).

In ~/.mailcap:

text/html; pandoc -f html -t markdown; copiousoutput; description=HTML Text; test=type pandoc >/dev/null
text/html; lynx -stdin -dump -force_html -width 70; copiousoutput; description=HTML Text; test=type lynx >/dev/null
text/html; w3m -dump -T text/html -cols 70; copiousoutput; description=HTML Text; test=type w3m >/dev/null
text/html; html2text -width 70; copiousoutput; description=HTML Text; test=type html2text >/dev/null

In ~/.mutt/muttrc:

auto_view text/html

Now HTML is automatically piped through one of those programs to turn it into plain text, and when I reply the quoted text is the plain text version rather than the raw HTML.

Merging files with unison and vimdiff -- update

I've just updated my post from February about merging files with unison and vimdiff -- I figured out how to do so without launching a new terminal emulator. The solution uses screen. See the updated post.

Options for lpr

I keep on forgetting these so it's about time I wrote down the ones I use most often.

You can list the current options (it definitely includes the options I've set to default and I'm not sure if it also shows all other default options or just those which the Cups Gui on my machine has set) with lpoptions. I've set it to print full duplex by default in the past with lpoptions -o sides=two-sided-long-edge so that shows up when I run loptions and everything's printed double sided by default.

Sometimes I don't want to print double sided and so I'll do lpr flyer.pdf -o copies=8 -o sides=one-sided to print off a bunch, one per sheet. (Could also do -#8 as on option to lpr for eight copies but that requires escaping.)

I can also choose a printer other than the default one such as the colour laser with lpr poster.pdf -P renoir

To do a quick printout of a PDF 2-up and reordered so it can just be folded up into a pamphlet fresh out of the printer I can do this:

pdftops somebook.pdf - | psbook | lpr -o number-up=2 -o sides=two-sided-short-edge

To print an A4 page on an A3 printer and fill the page I should (not tested) be able to do this:

lpr a4poster.pdf -o media=A3 -o fit-to-page

A landscape option might be necessary, I guess depending on the input. I'll come back and confirm what works at some point.

copies=num

Number of copies. Also possible to use -#num but that might need escaping.

landscape

Rotates 90 degrees but doesn't resize the page. Printing an A4 portrait PDF will not fit on A4 media but it doesn't complain, it just cuts off what doesn't fit.

number-up=num

The number of pages to print on each side.

number-up-layout=lrtb|lrbt|rltb|rlbt|tblr|tbrl|btlr|btrl

Choose how the pages are laid out when printing multiple pages on a side.

sides=one-sided|two-sided-long-edge|two-sided-short-edge

Choose how duplex works -- how the paper is turned over before the next page is printed. two-sided-long-edge is desirable for 1-up printing, two-sided-short-edge for 2-up.

media=A4|whatever

Choose the media size. If the printer only does as large as A4 and you tell it to use A3 and fit-to-page it'll print on A4, sideways and too big. Doesn't sound useful but it potentially is.

page-set=odd|even

Print only odd or even pages.

outputorder=normal|reverse

This takes effect after number-up so you still get 1234 on a page, not 4321

cpi=charsperinch

The number of characters per inch when printing plain text (default 10).

lpi=linesperinch

The number of lines per inch when printing plain text (default 6).

columns=num

The number of columns to format when printing plain text.

page-(left|right|top|bottom)=margin

The margin sizes in points when printing plain text.

Collate=true|false

Whether or not to collate the output pages -- that is, print all pages of the first copy followed by all pages of the next. By default it won't -- it'll print all copies of page 1, then all copies of page 2 and so on.

page-ranges=page,page,start-end

Choose which pages to print. This takes effect after number-up so the second page is 5678 when printing 4-up. They're always printed in ascending order, regardless of the order the ranges are given in.

Nice bit of piping

I just made a nice little pipeline. I love the Linux shell.

cat file | tr A-Z a-z | tr -c a-z "\n" | sort | uniq -c | sort -nr

That gives a nice ordered count of the most used words in the file. It's pretty dumb -- it breaks words at anything which isn't an alphabetical character -- but it does the trick.

The first tr changes uppercase characters to lowercase, the second changes anything which isn't alphabetic to a newline. Then the words are sorted alphabetically, then uniq counts successive identical lines and outputs the count with the word, then that list is sorted numerically in descending order. Lovely.

Moving the pointer to the top left corner of the screen

I wanted a keystroke to send the pointer to the top left of the screen. Usually I do this by flipping to the other screen and back but on my laptop (with only one screen) this doesn't work.

The solution was to use xmousepos to get the mouse position (want to know which screen we're on) and xte to set it. These are part of the xautomation package in Ubuntu/Debian.

But how big are the screens? I'm using xrandr and assuming that the resolutions listed are in order left to right and no screens are up or down or rotated or anything crazy like that.

Here's my script.

#!/bin/sh

# get current horizontal mouse position
mousex=$(xmousepos | awk '{print $1}')

# walk through widths of screens (assuming they're in order left to right) until 
# the mouse is on the current screen, building offset from left
offset=0
for width in $(xrandr | grep '*' | awk '{print $1}' | sed -r 's/x.*//'); do
 [ $mousex -lt $(expr $offset + $width) ] && break
 offset=$(expr $offset + $width)
done

# move the pointer to the top left of the screen
xte "mousemove $offset 0"

xte can send other fake input to X too such as clicking, pressing and releasing keys and so on. Potentially very useful.

Giving standard input to a program run in a shell script

I modified my "lock screen" bash script just now to also work when X isn't running -- that is, I'm on a virtual console or shelled into my machine.

The script as it was did this: use purple-remote to set my status to "away" (if it was "available" -- otherwise don't touch it), then run i3lock to lock the X screen, then switch the screen off to save power, then when i3lock exits restore my IM status if it was changed.

I added an if [ $DISPLAY ] test to branch for X and non-X, putting vlock in the place of the i3lock and xset commands for the non-X branch. It didn't like this, though -- vlock complained that it wasn't a virtual console.

In order to get it working all I had to do was redirect the script's standard input to vlock like this: vlock -a </dev/stdin.

Printing from the command line

I just selected some instructions from a Pidgin window and then typed xsel | lpr in a terminal. Fantastic.

Debugging an already-running process

I was getting a segfault in pms, an ncurses application I contribute to, so I restarted the program within gdb. But since the bug seems to be something to do with resizing the console window the segfault never appeared again -- gdb doesn't pass the resize signal (or perhaps the new dimensions) on to the program running within it.

I Googled around for a way to run gdb in a separate console window from the ncurses application. It's rather simple: change to the directory containing the binary so it can find the right one to debug against and run gdb -p $(pidof pms).

That'll attach gdb to the already running process and pause it. Type continue in the gdb window and await the crash.

Which package did a particular file come from

I sometimes find myself trying to figure out which package a particular file came from. It's usually a binary I want available on a different Debian-based machine and an aptitude search binaryname isn't returning anything.

I think I've done this a few times before with dpkg-query commands of some kind but it was never simple enough to remember. I Googled the problem again today and came up with an article introducing dlocate. It just has to be installed (its package is helpfully called "dlocate") and then to find out where pdfimages came from I just have to type dlocate pdfimages.

But that gave me three lines of output -- I have a python script called pdfimages.py which came from python-reportlab, a gzipped manpage which came with poppler-utils and the binary I was actually looking for, also from poppler-utils. So I could have made the query more specific by giving a full path to the file -- quickest way with a binary is of course using which: dlocate $(which pdfimages).

Merging a unison conflict with vim

I use unison to synchronize various files. If I've edited the same file on both ends I can pick a version to keep or, until now, I aborted, manually merged them with vimdiff or some other tool and then started synchronizing again.

But unison is able to call an external tool to merge for you. I figured out how to get it to use vimdiff -- add the option -merge "Name * -> urxvt -e vimdiff CURRENT1 CURRENT2".

I've used urxvt rather than my usual urxvtcd because urxvtcd immediately forks and unison continues as soon as the process it runs exits. So why did I run a terminal emulator at all rather than just running vimdiff in the existing terminal? It won't display. Not sure why -- probably to do with it not realizing it has a virtual terminal to output on or something.

Update on 2010-10-10: I had another shot at figuring it out. I tried using screen -- "Must be connected to a terminal". Then I tried using ssh -t and screen -- same issue (but I think the error was from SSH this time rather than screen). Finally I found an option in screen's manual page which can start a screen session detached but yet not fork. That'll do. So now in default.prf (which my other unison config files include) I have these two lines:

merge = Name * -> screen -DmS unisonmerge vimdiff CURRENT1 CURRENT2
merge = Name .* -> screen -DmS unisonmerge vimdiff CURRENT1 CURRENT2

The second line is there because I found just now that dotfiles weren't included when only the first line was present.

So here's what happens when a merge is now requested. A new screen session is started with the session name "unisonmerge". That's started detached but the process doesn't fork as it usually would and so unison stops, waiting. We get the shell to stop the process and give us a prompt by pressing ^Z, then attach the new screen session with screen -RS unisonmerge. The vimdiff instance is running in that -- we merge the files (modifying only the one we want to keep, which will end up on both machines) and then exit. The screen session ends and we're back to the prompt. Then we bring unison back to the foreground with fg. Unison continues.

So now I can run unison on a headless server over SSH, merging when necessary.

Postpone current command and run another first in zsh

Sometimes I am halfway through writing a command and realize I need to run something else first. Just now I was writing a git commit command and realized I should edit the TODO file first. So usually I either ^U to clear the command and type vim TODO before writing the commit command again or maybe open a new terminal or maybe even ^A to go to the start of the line and add vim TODO; to the beginning.

But zsh has a buffer stack which comes in useful here. The default key binding for push-line is ESC q. This pushes the current buffer (command in progress) onto the stack and clears the buffer, then pops from the stack next time the prompt is reached. So halfway through typing a commit command I do ESC q, type vim TODO, make whatever changes and exit and I'm back at a commandline with my half-typed commit command restored.

Get everything from an FTP server

When there's no shell access (and so no SCP) FTP is unavoidable. I'd usually do this with a Gui FTP program like Filezilla but I thought it was about time I figured out how to do it on the command line.

wget does the trick but I noticed it got stuck once or twice and then didn't move. When that happened I just hit ^C to stop it and then ran again with the c switch (continue). That switch doesn't hurt the first time either so I'll include it below:

wget -cr --user username --password password ftp://server

imgur

It's a bit like sprunge -- a pastebin for images. I wrote a bash script similar to my sprunge one to upload an image (given by filename) and return (and put on the clipboard) its new URL. The URL to delete it is given on stderr.

#!/bin/bash
apikey="<API KEY>"
if [ $# -lt 1 ]; then
 echo "no file specified" >&2
 exit 127
fi
response=$(curl -F "key=$apikey" -F "image=@$1" http://imgur.com/api/upload.xml 2>/dev/null)
if [ $? -ne 0 ]; then
 echo "request failed" >&2
 exit 1
fi
url=$(echo $response | sed -r 's/.*<original_image>(.*)<\/original_image>.*/\1/')
deleteurl=$(echo $response | sed -r 's/.*<delete_page>(.*)<\/delete_page>.*/\1/')
echo $url
echo "Delete page: $deleteurl" >&2
if [ $DISPLAY ]; then
 { which xsel >/dev/null 2>/dev/null && echo -n $url | xsel; } \
  || { which xclip >/dev/null 2>/dev/null && echo -n $url | xclip; } \
  || echo "haven't copied to the clipboard: no xsel or xclip" >&2
else
 echo "haven't copied to the clipboard: no \$DISPLAY" >&2
fi

Update: a later version of this script is now featured on the imgur site. At the moment it's at the bottom of the tools page.

Package a source tree ignoring .svn directories

Simple: tar cvzf py-translate-svnr8.tar.gz --exclude=.svn py-translate

Print a particular line of a file

Saw this on commandlinefu today.

I'd been doing command | head -5 | tail -1 to print the 5th line but it makes more sense to use sed:
command | sed -n 5p

Yanking URLs from the console

It's a pain to select URLs manually from the console with the mouse for pasting into Vimperator or whatever other program. On the PMS IRC channel today smeea mentioned a perl extension for urxvt which scans for URLs and allows them to be yanked or put straight into a browser. It's on another fella called Bart's blog.

Very handy. Now I can ignore the mouse even more.

find and xargs

How hard can it be to send a bunch of filenames from find as arguments to a shell command? It turns out that it's much more difficult than it should be if you want to add another argument after the list of filenames.

The problem today was that my colleague and I wanted to find all images with a particular name and send them to Imagemagick to put them all together. The Imagemagick command for that looks like
montage -size 1 -geometry 66x50 image*.png montage.png

The issue is that Imagemagick isn't flexible. The output filename has to be the last argument and the input files can't be passed by standard input. The contenders for solving the problem are find's -exec option and xargs.

So the first thing we tried was this:
find . -name "image*.png" -exec montage -size 1 -geometry 66x50 {} montage.png +
but this gives a "missing argument to `exec'" error -- find doesn't allow extra arguments after the placeholder.

Then we tried using xargs:
find . -print0 -name "image*.png" | xargs -0 -I {} montage -size 1 -geometry 66x50 {} montage.png
but this runs the command once for each input unless we increase xargs's -L (max input arguments per command) option back up from 1 (which is set by using the -I option -- no idea why) to some arbitrary large number. This seems like a bad way to do things.

The solution we came up with was to add an extra argument before the stream gets to xargs:
find . -print0 -name "image*.png" | cat - <(echo -ne "montage.png\0") | xargs -0 montage -size 1 -geometry 66x50

This is using process substitution to put montage and a null character as a file descriptor input to cat, which adds that to the end of the first argument, its standard input.

It's still a mess -- any better solutions?

Herestrings in the shell

Like heredoc syntax there is also herestring.

For a heredoc you'd do something like

cat >file <<EOF
This is a file
all this text will get written to the file
This dir contains
$(ls)
EOF

or, if expansion (command, parameter and arithmetic) isn't wanted, quote the word at the start: cat >file <<"EOF".

Herestrings are a quicker way than using echo and piping to a command -- for instance to get torrent info I can do ssh server deluge --ui=console <<<info rather than echo info | ssh server deluge --ui=console.

Put SSH in the background

While looking for something unrelated in SSH's manpage I came across something I didn't know before: when logged into a remote machine various escape sequences are supported to talk to the local end of the connection.

It's possible to do fancy things like add port forwardings or remove existing ones but I think the only one I'll now use commonly is ~^Z, which is like sending ^Z to your local shell and backgrounds the SSH process. You get your local shell back, can run whatever commands you need and then foreground SSH again with fg.

The escape character (~ is the default) has to come after a newline character to be recognized as special.

I've entirely forgotten what I was actually looking for in the manual.