git branch in ZSH prompt

Screenshot of the terminal showing the current git branch

Today, I came across a little trick on how to output the current git branch on your bash prompt. This is very useful, but not as much for me as I’m using ZSH. Of course, I wanted to adapt the method (and to use fewer backslashes :-) ).

Also, in my setup, I’m making use of ZSH’s prompt themes feature of which I’ve chosen the theme “adam1”. So let’s use that as a starting point.

  1. First, create a copy of the prompt theme into a directory of your control where you intend to store private ZSH functions (~/zshfuncs in my case).
    cp /usr/share/zsh/4.3.4/functions/prompt_adam1_setup ~/zshfuncs/prompt_pilif_setup
  2. Tweak the file. I’ve adapted the prompt from the original article, but I’ve managed to get rid of all the backslashes (to actually make the regex readable) and to place it nicely in the adam1 prompt framework.
  3. Advise ZSH about the new ZSH function directory (if you haven’t already done so).
    fpath=(~/zshfunc $fpath)
  4. Load your new prompt theme.
    prompt pilif

And here’s the adapted adam1 prompt theme:

# pilif prompt theme

prompt_pilif_help () {
  cat <<'EOF'
This prompt is color-scheme-able.  You can invoke it thus:

  prompt pilif [<color1> [<color2> [<color3>]]]

This is heavily based on adam1 which is distributed with ZSH. In fact,
the only change from adam1 is support for displaying the current branch
of your git repository (if you are in one)
EOF
}

prompt_pilif_setup () {
  prompt_adam1_color1=${1:-'blue'}
  prompt_adam1_color2=${2:-'cyan'}
  prompt_adam1_color3=${3:-'green'}

  base_prompt="%{$bg_no_bold[$prompt_adam1_color1]%}%n@%m%{$reset_color%} "
  post_prompt="%{$reset_color%}"

  base_prompt_no_color=$(echo "$base_prompt" | perl -pe "s/%{.*?%}//g")
  post_prompt_no_color=$(echo "$post_prompt" | perl -pe "s/%{.*?%}//g")

  precmd  () { prompt_pilif_precmd }
  preexec () { }
}

prompt_pilif_precmd () {
  setopt noxtrace localoptions
  local base_prompt_expanded_no_color base_prompt_etc
  local prompt_length space_left
  local git_branch

  git_branch=`git branch 2>/dev/null | grep -e '^*' | sed -E 's/^* (.+)$/(1) /'`
  base_prompt_expanded_no_color=$(print -P "$base_prompt_no_color")
  base_prompt_etc=$(print -P "$base_prompt%(4~|...|)%3~")
  prompt_length=${#base_prompt_etc}
  if [[ $prompt_length -lt 40 ]]; then
    path_prompt="%{$fg_bold[$prompt_adam1_color2]%}%(4~|...|)%3~%{$fg_bold[white]%}$git_branch"
  else
    space_left=$(( $COLUMNS - $#base_prompt_expanded_no_color - 2 ))
    path_prompt="%{$fg_bold[$prompt_adam1_color3]%}%${space_left}<...<%~ %{$reset_color%}$git_branch%{$fg_bold[$prompt_adam1_color3]%} $prompt_newline%{$fg_bold_white%}"
  fi

  PS1="$base_prompt$path_prompt %# $post_prompt"
  PS2="$base_prompt$path_prompt %_&gt; $post_prompt"
  PS3="$base_prompt$path_prompt ?# $post_prompt"
}

prompt_pilif_setup "$@"

The theme file can be downloaded here

Converting Java keytool-certificates

To be able to read barcodes from connected barcode-scanners into the webbased version of PopScan, we have to use a signed applet – there is no other way for getting the needed level of hardware access without signing your applet.

The signature, by the way, doesn’t at all prevent any developer from doing bad stuff – it just puts their signature below it (literally), so it kind of raises the bar to distribute malware that way – after all, the checks when applying for a certificate usually are very rigid, so there is no way anybody could forge their application, so the origin of any piece of code is very tracable.

But there is no validation done of the actual code to be signed and I doubt that the certificate authorities out there actually revoke certificates used to certify malware, thought that remains to be seen.

Anyways. Back to the topic.

In addition to the Java Applet, we also develop the windows client frontend to the PopScan server. And we have a small frontend to run on Windows CE (or Windows Mobile) based barcode capable devices. Traditionally, both of these were never signed.

But lately with Vista and Windows Mobile 6, signing becomes more and more important: Both systems complain in variable loudness about unsigned code, so I naturally prefer the code to be signed – we DO have a code signing certificate after all – for our Applet.

Now the thing is that keytool, Java’s way of handling code signing keys doesn’t allow a private key to be exported. This means that there was no obvious way for me to ever use the certificate we got for our applet to sign Windows EXEs.

Going back to the CA and ask them to send over an additional certificate was no option for me: Aside of the fact that it would certainly have cost another two years fee, this would have ment to prove our identity all over again – one year too early as our current certificate is valid till 2009.

But then, I found a solution. Here’s how you convert a java keystore certificate to something you can use with Microsoft’s Authenticode:

  1. Start KeyTool GUI
  2. In the Treeview, click “Export”, “Private Key”
  3. Select your java keystore-file
  4. Enter two trarget file names for your key and the certificate chain (and select PEM format)
  5. Click OK

Now you will have two more files. One is your private key (I’ve named it key.pem), the other is the certificate chain (named cert.pem in my case). Now, use OpenSSL to covert this into something Microsoft likes to see:

% openssl pkcs12 -inkey key.pem -in cert.pem -out keypair.pfx -export

openssl will ask for a password to encrypt the pfx file with and you’ll be done. Now you can use the pfx-file like any other pfx file you recived from your certificate authority (double click it to install it or use it with signcode.exe to directly sign your code).

Remember to delete key.pem as it’s the unencrypted private key!

PostgreSQL on Ubuntu

Today, it was time to provision another virtual machine. While I’m a large fan of Gentoo, there were some reasons that made me decide to gradually start switching over to Ubuntu Linux for our servers:

  • One of the large advantages of Gentoo is that it’s possible to get bleeding edge packages. Or at least you are supposed to. Lately, it’s taking longer and longer for an ebuild of an updated version to finally become available. Take PostgreSQL for example: It took about 8 months for 8.2 to become available and it looks like history is repeating itself for 8.3
  • It seems like there are more flamewars than real development going on in Gentoo-Land lately (which in the end leads to above problems)
  • Sometimes, init-scripts and stuff changes over time and there is not always a clear upgrade-path. emerge -u world once, then forget to etc-update and on next reboot, hell will break loose.
  • Installing a new system takes ages due to the manual installation process. I’m not saying it’s hard. It’s just time-intensive

Earlier, the advantage of having current packages greatly outweighted the issues coming with Gentoo, but lately, due to the current state of the project, it’s taking longer and longer for packages to become available. So that advantage fades away, leaving me with only the disadvantages.

So at least for now, I’m sorry to say, Gentoo has outlived it’s usefulness on my productive servers and has been replaced by Ubuntu, which albeit not being bleeding-edge with packages, at least provides a very clean update-path and is installed quickly.

But back to the topic which is the installation of PostgreSQL on Ubuntu.

(it’s ironic, btw, that Postgres 8.3 actually is in the current hardy beta, together with a framework to concurrently use multiple versions whereas it’s still nowhere to be seen for Gentoo. Granted: An experimental overlay exists, but that’s mainly untested and I had some headaches installing it on a dev machine)

After installing the packages, you may wonder how to get it running. At least I wondered.

/etc/init.d/postgresql-8.3 start

did nothing (not very nice a thing to do, btw). initdb wasn’t in the path. This was a real WTF moment for me and I assumed some problem in the package installation.

But in the end, it turned out to be an (underdocumented) feature: Ubuntu comes with a really nice framework to keep multiple versions of PostgreSQL running at the same time. And it comes with scripts helping to set up that configuration.

So what I had to do was to create a cluster with

pg_createcluster --lc-collate=de_CH --lc-ctype=de_CH -e utf-8 8.3 main

(your settings my vary – especially the locale settings)

Then it worked flawlessly.

I do have some issues with this process though:

  • it’s underdocumented. Good thing I speak perl and bash, so I could use the source to figure this out.
  • in contrast to about every other package in Ubuntu, the default installation does not come with a working installation. You have to manually create the cluster after installing the packages
  • pg_createcluster –help bails out with an error
  • I had /var/lib/postgresql on its own partition and forgot to remount it after a reboot which caused the init-script to fail with a couple of uninitialized value errors in perl itself. This should be handeled cleaner.

Still. It’s a nice configuration scheme and a real progress from gentoo. The only thing left for me now is to report these issues to the bugtracker and hope to see this fixed eventually. And it it isn’t, there is this post here to remind me and my visitors.

PHP 5.2.4

Today, the bugfix-release 5.2.4 of PHP has been released.

This is an interesting release, because it includes my fix for bug 42117 which I discovered and fixed a couple of weeks ago.

This means that with PHP 5.2.4 I will finally be able to bzip2-encode data as it is generated on the server and stream it out to the client, greatly speeding up our windows client.

Now I only need to wait for the updated gentoo package to update our servers.

More iPod fun

Last time I explained how to get .OGG-feeds to your iPod.

Today I’ll show you one possible direction one could go to greatly increase the usability of non-official (read: not bought at audible.com) audiobooks you may have lying around in .MP3 format.

You see, your iPod threats every MP3-File of your library as music, regardless of length and content. This can be annoying as the iPod (rightly so) forgets the position in the file when you stop playback. So if you return to the file, you’ll have to start from the beginning and seek through the file.

This is a real pain in case of longer audiobooks and / or radio plays of which I have a ton

One way is to convert your audiobooks to AAC and rename the file to .m4b which will convince iTunes to internally tag the files as audiobooks and then enable the additional features (storing the position and providing UI to change play speed).

Of course this would have meant converting a considerable part of my MP3 library to the AAC-format which is not yet as widely deployed (not to speak of the quality-loss I’d have to endure when converting a lossy format into another lossy format).

It dawned me that there’s another way to make the iPod store the position – even with MP3-files: Podcasts.

So the idea was to create a script that reads my MP3-Library and outputs RSS to make iTunes think it’s working with a Podcast.

And thus, audiobook2cast.php was born.

The script is very much tailored to my directory structure and probably won’t work at your end, but I hope it’ll provide you with something to work with.

In the script, I can only point out two interesting points:

  • When checking a podcast, iTunes ignores the type-attribute of the enclosure when determining whether a file can be played or not. So I had to add the fake .mp3-extension.
  • I’m outputting a totally fake pubDate-Element in the <item>-Tag to force iTunes to sort the audiobooks in ascending order.

As I said: This is probably not useful to you out-of-the-box, but it’s certainly an interesting solution to an interesting problem.

Updating or replacing datasets

This is maybe the most obvious trick in the world but I see people not doing it all over the place, so I guess it’s time to write about it.

Let’s say you have a certain set of data you need to enter into your RDBMS. Let’s further assume that you don’t know whether the data is already there or not, so you don’t know whether to use INSERT or UPDATE

Some databases provide us with something like REPLACE or “INSERT OR REPLACE”, but others do not. Now the question is, how to do this efficiently?

What I always see is something like this (pseudo-code):

  1. select count(*) from xxx where primary_key = xxx
  2. if (count > 0) update; else insert;

This means that for every dataset you will have to do two queries. This can be reduced to only one query in some cases by using this little trick:

  1. update xxx set yyy where primary_key = xxx
  2. if (affected_rows(query) == 0) insert;

This method just goes ahead with the update, assuming that data is already there (which usually is the right assumption anyways). Then it checks if an update has been made. If not, it goes ahead and inserts the data set.

This means that in cases where the data is already there in the database, you can reduce the work on the database to one single query.

Additionally, doing a SELECT and then an UPDATE essentially does the select twice as the update will cause the database to select the rows to update anyways. Depending on your optimizer and/or query cache, this can be optimized away of course, but there are no guarantees.

Careful when clean-installing TabletPCs

At work, I got my hands on a LS-800 TabletPC by motion computing and after spending a lot of time with it and as I’m very interested in TabletPCs anyways, I finally got myself its bigger brother, the LE-1700

The device is a joy to work with: Relatively small and light, one big display and generally nice to handle.

The tablet came with Windows XP preinstalled and naturally, I wanted to have a look at the new Tablet-centric features in Vista, so I went ahead and upgraded.

Or better: Clean-installed.

The initial XP installation was german and I was installing an english copy of Vista which makes the clean installation mandatory.

The LE-1700 is one of the few devices without official Vista-support, but I guess that’s because of the missing software for the integrated UMTS modem – for all other devices, drivers either come prebundled with Vista, are available on Windows update or you can use the XP drivers provided at the Motion computing support site.

After the clean installation, I noticed that the calibration of the pen was a bit off – depending on the position on the screen, the tablet noticed the pen up to 5mm left or above the actual position of the pen. Unfortunately, using the calibration utility in the control panel didn’t seem to help much.

After some googling, I found out what’s going on:

The end-user accessible calibration tool only calibrates the screen for the tilt of the pen relative to the current position. The calibration of the pens position is done by the device manufacturer and there is no tool available for end-users to do that.

Which, by the way, is understandable considering how the miscalibration showed itself: To the middle of the screen it was perfect and near the sides it got worse and worse. This means that a tool would have to present quite a lot of points for you to hit to actually get a accurately working calibration.

Of course, this was a problem for me – especially when I tried out journal and had to notice that the error was bad enough to take all the fun out of hand-writing (imagine writing on a paper and the text appearing .5cm left of where you put the pen).

I needed to get the calibration data and I needed to put it back after the clean installation.

It turns out that the linear calibration data is stored in the registry under HKLMSYSTEMCurrentControlSetControlTabletPCLinearityData in the form of a (large) binary blob.

Unfortunately, Motion does not provide a tool or even reg-file to quickly re-add the data should you clean-install your device, so I had to do the unthinkable (I probably could have called support, but my method had the side effect of not making me wait forever for a fix):

I restored the device to the factory state (by using the preinstalled Acronis True Image residing on a hidden partition), exported the registry settings, reinstalled Vista (at which time the calibration error resurfaced), imported the .reg-File and rebooted.

This solved the problem – the calibration was as smooth as ever.

Now, I’m not sure if the calibration data is valid for the whole series or even defined per device, but here is my calibration data in case you have the same problem as I had.

If the settings are per device or you have a non-LE-1700, I strongly advise you to export that registry key before clean-installing

Obviously I would have loved to know this beforehand, but… oh well.

PHP, stream filters, bzip2.compress

Maybe you remember that, more than a year ago, I had an interesting problem with stream filters.

The general idea is that I want to output bz2-compressed data to the client as the output is being assembled – or, more to the point: The PopScan Windows-Client supports the transmission of bzip2 encoded data which gets really interesting as the amount of data to be transferred increases.

Even more so: The transmitted data is in XML format which is very easily compressed – especially with bzip2.

Once you begin to transmit multiple megabytes of uncompressed XML-data, you begin to see the sense in jumping through a hoop or two to decrease the time needed to transmit the data.

On the receiving end, I have an elaborate construct capable of downloading, decompressing, parsing and storing data as it arrives over the network.

On the sending end though, I have been less lucky: Because of that problem I had, I was unable to stream out bzip2 compressed data as it was generated – the end of the file was sometimes missing. This is why I’m using ob_start() to gather all the output and then compress it with bzcompress() to send it out.

Of course this means that all the data must be assembled before it can be compressed and the sent to the client.

As we have more and more data to transmit, the client must wait longer and longer before the data begins to reach it.

And then comes the moment when the client times out.

So I finally really had to fix the problem. I could not believe that I was unable to compress and stream out data on the fly.

It turns out that I finally found the smallest possible amount of code to illustrate the problem in a non-hacky way:

So: This fails under PHP up until 5.2.3:

<?
$str = "BEGIN (%d)n
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip
ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur
sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum.
nEND (%d)n";

$h = fopen($_SERVER['argv'][1], 'w');
$f = stream_filter_append($h, "bzip2.compress", STREAM_FILTER_WRITE);
for($x=0; $x < 10000; $x++){
   fprintf($h, $str, $x, $x);

}
fclose($h);
echo "Writtenn";
?>

Even worse though: It doesn’t fail with a message, but it writes out a corrupt bzip-File.

And it gets worse: With a little amount of data it works, but as the amount of data increases, it begins to fail – at different places depending on how you shuffle the data around.

Above script will write a bzip file which – when uncompressed – will end around iteration 9600.

So now that I had a small reproducible testcase, I could report a bug in PHP: Bug 47117.

After spending so many hours on a problem which in the end boiled down to a bug in PHP (I’ve looked anywhere, believe me. I also tried workarounds, but all to no avail), I just could not let the story end there.

Some investigation quickly turned up a wrong check for a return value in bz2_filter.c which I was able to patch up very, very quickly, so if you visit that bug above, you will find a patch correcting the problem.

Then, when I finished patching PHP itself, hacking up the needed PHP-code to let the thing stream out the compressed data as it arrived was easy. If you want, you can have a look at bzcomp.phps which demonstrates how to plug the compression into either the output buffer handling or something quick, dirty and easier else.

Oh, and if you are tempted to do this:

function ob($buf){
        return bzcompress($buf);
}

ob_start('ob');

… it won’t do any good because you will still gobble up all the data before compressing. And this:

function ob($buf){
        return bzcompress($buf);
}

ob_start('ob', 32768);

will encode in chunks (good), but it will write a bzip2-end-of-stream marker after every chunk (bad), so neither will work.

Nothing more satisfying than to fix a bug in someone else’s code. Now let’s hope this gets applied to PHP itself so I don’t have to manually patch my installations.

Altering the terminal title bar in Mac OS X

After one year of owning a MacBook Pro, I finally got around to fix my precmd() ZSH-hack to really make the current directory and stuff appear in the title bar of Terminal.app and iTerm.app.

This is the code to add to your .zshrc:

case $TERM in
    *xterm*|ansi)
		function settab { print -Pn "e]1;%n@%m: %~a" }
		function settitle { print -Pn "e]2;%n@%m: %~a" }
		function chpwd { settab;settitle }
		settab;settitle
        ;;
esac

settab sets the tab contents in iTerm and settitle does the same thing for the title bar both in Terminal.app and iTerm.

The sample also shows the variables ZSH replaces in the strings (the parameter -P to print lets ZSH do prompt expansion. See zshmisc(1) for a list of all variables): %n is the currently logged on user, %m the hostname up until the first dot and %~ is displaying the current directory or ~ if you are in $HOME. You can certainly add any other environment variable of your choice if you need more options, but this more or less does it for me.

Usually, the guides in the internet make you use precmd to set the title bar, but somehow, Terminal wasn’t pleased with that method and constantly kept overwriting the title with the default string.

And this is how it looks in both iTerm (above) and Terminal (below):

Vista, AC3, S/PDIF

Since around December 31th last year, my home cinema is up and running. That day was the day when I finally had all the equipment needed to mount the screen that has arrived on December 29th.

It was a lot of work, but the installation just rocks. And I’ve already blogged about the main components of the thing: The Gefen HDMI extender and the Denon AVR-4306.

The heart of the system consists of shion serving content (thankfully, the TB harddrive was announced last week – it’s about time) and a brand new 1.8Ghz Mac Mini running Windows Vista (RC2) in BootCamp which is actually displaying the content.

I’ve connected a windows media center remote receiver which Microsoft sells to OEMs to use the old IR remote of my Shuttle MCE machine.

The mac mini is connected to the Denon receiver via a DVI to HDMI adaptor and optical digital cable for the audio.

And that last thing is what I’m talking about now.

The problem is that Microsoft changed a lot about how audio works in Vista and I had to learn it the hard way.

At first, I couldn’t not hear any sound at all. That’s because Vista treats all outputs of your sound card as separate entities and you can configure over which connector you want to hear which sounds.

The fix there was to set the S/PDIF connector as system default (in the sound applet of control panel) which fixed the standard windows sounds and stereo sound for me.

Actually, the properties screen of the S/PDIF connector already contains options for AC3 and DTS, complete with a nice testing feature allowing you to check your receiver’s support for the different formats by actually playing some multichannel test sound.

The problem is that this new framework is mostly unsupported by the various video codecs out there.

This means that even if you get that control panel applet to play the test sound (which is easy enough), you won’t get AC3 sound when you are playing a movie file. You still need to get a codec for that.

But most codecs don’t work right any more in vista as the S/PDIF connector now is a separate entity and seems to be accessed differently than in XP.

Usually, the only thing I install on a new windows machine I need to play video with is ffmpeg which actually has some limited support for Vista’s way of handing S/PDIF: In the audio settings dialog, you can select “Output” and then in the formats list for S/PDIF, you can check AC/3. Unfortunately, this unchecks the PCM formats.

This means that you will get sound in movies with an AC3 track, but no sound at all in every other movie – ffmpeg seems (emphasis on seems – I may just not have found a way yet) unable to either encode stereo to AC3 or output both PCM and AC3 without changing settings (not at the same time of course).

AC3filter works better in that regard.

Depending on hour of the day (…), it’s even able to work with the S/PDIF output without forcing it to encode stereo to AC3 (which AC3filter is capable to do).

So for me the solution to the whole mess was this:

  1. Install the latest build of ffmpeg, but don’t let it handle audio
  2. Install AC3filter
  3. Open the configuration tool and on the first page enable S/PDIF.
  4. On the system tab, enable passthrough for AC3 and DTS.

This did the trick for me.

As time progresses, I’m certain that the various projects will work better and better with the new functionality in Vista which will make hassles like this go away.

Until then, I’m glad I found a workable solution.