Free Software- gnegg

OpenStreetMap

The last episode of FLOSS Weekly consisted of an interview with Steve Coast from OpenStreetMap. I knew about the project, but I was of the impression that it was in its infancy both content-wise and from a technical perspective.

During the interview I learned that it’s surprisingly complete (unless, of course, you need a map of Canada it seems) and highly advanced from a technical point of view.

But what’s really interesting is the fact how terribly easy it is to contribute. For smaller edits, you just click the edit-Link and use the Flash editor to paint a road or give it a name. If you need or want to do more, then there’s a really easy to use Java based editor:

First you drag a rectangle onto a pre-rendered version of the map which will cause the server to send you the vector information consisting of that part and then you can edit whatever you want.

If you have them, you can import traces of a GPS logger to help you add roads and paths and when you are finished, you press a button and the changes get uploaded and will be visible to the public a few minutes later (though one modification I made took about an hour to arrive on the web).

When the same nodes where updated in the meantime, a really nice conflict resolution assistant will help you to resolve the conflicts.

For me personally, this has the potential to become my new after-work time sink as it combines quite many passions of mine:

The GPS tracking, importing and painting of maps is pure technology fun.
Actually being outside to generate the traces is healthy and also a lot of fun
Maps also are a passion of mine. I love to look at maps and I love to compare them to my mental image of the places they are showing.

And besides all that, Open Street Map is complete enough to be of real use. For biking or hiking it even trumps Google Maps by much.

Still, at least near where I live, there are many small issues that can easily be fixed.

As the different editors are really easy to use, fixing these issues is a lot of fun and I’m totally seeing myself cleaning out all small mistakes I come across or even adding stuff that’s missing. After all, this also provides me with a very good reason to visit the places where I grew up to complete some parts.

The whole concept behind being able to update a map by just a couple of mouse clicks is very compelling too as it finally gives us the potential to have really accurate maps in a very timely fashion. For example: Last October, one of the roads near my house closed and just recently the tracks of the Forchbahn were moved a bit.

Just today I added these changes to OpenStreetMap and now OSM is the only publically available map that correctly shows the traffic situation. And all that with 15 minutes of easy but interesting work.

For those interested, my Open Street Map user profile is, of course, pilif.

PostgreSQL 8.4

Like a clockwork, about one year after the release of PostgreSQL 8.3, the team behind the best database on this world did it again and released PostgreSQL 8.4, the latest and greatest in a long series of awesomeness.

Congratulations to everyone involved and might you have the strength to continue to improve your awesome piece of work.

For me, the hightlights of this new release are

parallel restore: I just tried this out and restoring a dump that usually took around 40 minutes (in standard sql/text format) now takes 5 minutes.
The improvements to psql usability just make it even clearer that psql isn’t just a command line database tool, but that it’s one of the best interfaces to access the data and administer the server. psql hands-down beats whatever database GUI tool I have seen so far.
truncate table reset identity is very useful during development
no more max_fsm_pages makes maintaining the database even easier and removes one variable to keep track of.

Thanks again for yet another awesome release.

New MacMini (early 09) and Linux

The new MacMinis that were announced this week come with a Firewire 800 port which was reason enough for me to update shion yet again (keeping the host name of course).

All my media she’s serving to my various systems is stored on a second generation Drobo which is currently connected via USB2, but has a lingering FW800 port.

Of course the upgrade to FW800 will not double the transfer rate to and from the drobo, but it should increase it significantly, so I went ahead and got one of the new Minis.

As usual, I entered the Ubuntu (Intrepid) CD, hold c while turning the device on and completed the installation.

This left the Mini in an unbootable state.

It seems that this newest generation of Mac Hardware isn’t capable of booting from an MBR partitioned harddrive. Earlier Macs complained a bit if the harddrive wasn’t correctly partitioned, but then went ahead and booted the other OS anyways.

Not so much with the new boxes it seems.

To finally achieve what I wanted I had to do the following complicated procedure:

Install rEFIt (just download the package and install the .mpkg file)
Use the Bootcamp assistant to repartition the drive.
Reboot with the Ubuntu Desktop CD and run parted (the partitioning could probably be accomplished using the console installer, but I didn’t manage to do it correctly).
Resize the FAT32-partition which was created by the Bootcamp partitioner to make room at the end for the swap partition.
Create the swap partition.
Format the FAT32-partition with something useful (ext3)
Restart and enter the rEFIt partitioner tool (it’s in the boot menu)
Allow it to resync the MBR
Insert the Ubuntu Server CD, reboot holding the C key
Install Ubuntu normally, but don’t change the partition layout – just use the existing partitions.
Reboot and repeat steps 7 and 8
Start Linux.

Additionally, you will have to keep using rEFIt as the boot device control panel item does not recognize the linux partitions any more, so can’t boot from them.

Now to find out whether that stupid resistor is still needed to make the new mini boot headless.

All-time favourite tools – update

It has been more than four years since I’ve last talked about my all-time favourite tools. I guess it’s time for an update.

Surprisingly, I still stand behind the tools listed there: My love for Exim is still un-changed (it just got bigger lately – but that’s for another post). PostgreSQL is cooler than ever and powers PopScan day-in, day-out without flaws.

Finally, I’m still using InnoSetup for my Windows Setup programs, though that has lost a bit of importance in my daily work as we’re shifting more and more to the web.

Still. There are two more tools I must add to the list:

jQuery is a JavaScript helper libary that allows you to interact with the DOM of any webpage, hiding away browser incompatibilities. There are a couple of libraries out there which do the same thing, but only jQuery is such a pleasure to work with: It works flawlessly, provides one of the most beautiful APIs I’ve ever seen in any library and there are tons and tons of self-contained plug-ins out there that help you do whatever you would want to on a web page.
jQuery is an integral part of making web applications equivalent to their desktop counterparts in matters of user interface fluidity and interactivity.
All while being such a nice API that I’m actually looking forward to do the UI work – as opposed to the earlier days which can most accurately be described as UI sucks.
git is my version control system of choice. There are many of them out there in the world and I’ve tried the majority of them for one thing or another. But only git combines the awesome backwards-compatibility to what I’ve used before and what’s still in use by my coworkers (SVN) with abilities to beautify commits, have feature branches, very high speed of execution and very easy sharing of patches.
No single day passes without me using git and running into a situation where I’m reminded of the incredible beauty that is git.

In four years, I’ve not seen one more other tool I’ve as consistenly used with as much joy as git and jQuery, so those two certainly have earned their spot in my heart.

My new friend: git rebase -i

Last summer, I was into making git commits look nice with the intent of pushing a really nice and consistent set of patches to the remote repository.

The idea is that a clean remote history is a convenience for my fellow developers and for myself. A clean history means very well-defined patches – should a merge of a branch be neccesary in the future. It also means much easier hunting for regressions and generally more fun doing some archeology in the code.

My last post was about using git add -i to refine the commits going into the repository. But what if you screw up the commit anyways? What if you forget to add a new file and notice it only some commits later?

This is where git rebase -i comes into play as this allows you to reorder your local commits and to selectively squash multiple commits into one.

Let’s see how we would add a forgotten file to a commit a couple of commits ago.

You add the forgotten file and commit it. The commit message doesn’t really matter here.
You use git log or gitk to find the commit id before the one you want to amend this new file to. Let’s say it’s 6bd80e12707c9b51c5f552cdba042b7d78ea2824
Pick the first few characters (or the whole ID) and pass them to git rebase -i.

 % git rebase -i 6bd80e12

git will now open your favorite editor displaying your list of commits since the revision you have given. This could look like this.

pick 6bd80e1 some commit message. This is where I have forgotten the file
pick 4c1d210 one more commit message
pick 5d2f4ed this is my forgotten file

# Rebase fc9a0c6..5d2f4ed onto fc9a0c6
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

The comment in the file says it all – just reorder the first three (or how many there are in your case) to look like this:

pick 6bd80e1 some commit message. This is where I have forgotten the file
squash 5d2f4ed this is my forgotten file
pick 4c1d210 one more commit message

Save the file. Git will now do some magic and open the text editor again where you can amend the commit message for the commit you squashed your file into. If it’s really just a forgotten file, you’ll probably keep the message the same.

One word of caution though: Do not do this on branches you have already pushed to a remote machine or otherwise shared with somebody else. git gets badly confused if it has to pull altered history.

Isn’t it nice that after moths you still find new awesomeness in your tool of choice?

I guess I’ll have to update my all-time favorite tools list. It’s from 2004, so it’s probably ripe for that update.

Git rules.

Tunnel munin nodes over HTTP

Last time I’ve talked about Munin, the one system monitoring tool I feel working well enough for me to actually bother to work with. Harsh words, I know, but the key to every solution is simplicity. And simple Munin is. Simple, but still powerful enough to do everything I would want it to do.

The one problem I had with it is that the querying of remote nodes works over a custom TCP port (4949) which doesn’t work behind firewalls.

There are some SSH tunneling solutions around, but what do you do if even SSH is no option because the remote access method provided to you relies on some kind of VPN technology or access token.

Even if you could keep a long-running VPN connection, it’s a very performance intensive solution as it requires resources on the VPN gateway. But this point is moot anyways because nearly all VPNs terminate long running connections. If re-establishing the connection requires physical interaction, then you are basically done here.

This is why I have created a neat little solution which tunnels the munin traffic over HTTP. It works with a local proxy server your munin monitoring process will connect to and a little CGI-script on the remote end.

This will cause multiple HTTP connections per query interval (the proxy uses Keep-Alive though so it’s not TCP connections we are talking about – it’s just hits in the access.log you’ll have to filter out somehow) because it’s impossible for a CGI script to keep the connection open and send data both ways – at least not if your server-side is running plain PHP which is the case in the setup I was designing this for.

Aynways – the solution works flawlessly and helps me to monitor a server behind one hell of a firewall and behind a reverse proxy.

You’ll find the code here (on GitHub as usual) and some explanation on how to use it is here.

Licensed under the MIT license as usual.

Mozilla Weave 0.2

I have quite many computers I use regularely, on all of which runs Firefox. Of course I’ve accumulated quite a lot of bookmarks, passwords and “keep me logged in”-cookies.

During my use of FF2, I’ve come across Google Browser Sync which was incredibly useful, albeit a bit unstable here and then, so last Christmas, I was very happy to see the prototype of Mozilla Weave to be released. It promised the same feature set as Google Browser Sync, but build from the makers of the browser on an open architecture.

I have been a user of Weave ever since and it was even more inconsistent in availability than what Google Browser Sync ever provided, but at least it was just the server not working, not affecting the client which GBS did here and then, which made me lose parts or all of my bookmarks.

Over time though, Weave got better and better and with todays 0.2 release, the installation and setup process actually got streamlined enough so that I can recommend the tool to anybody using more than one PC at any time.

Especially with the improved bookmarking functionality we got in Firefox 3, synchroniuzing bookmarks has become really important. I’m very happy to see a solution for this problem and I’m overjoyed that the solution is as open as weave is.

Congratulations, Mozilla Team!

Converting ogg streams into mp3 ones

This is just an announcement for my newest quick-hack which can be used to on-the-fly convert streams from webradios which use the ogg/vorbis format into the mp3 format which is more widely supported by the various devices out there.

I have created an own dedicated page for the project for those who are interested.

Also, I really got to like github.com, not as the commercial service they intend to be (I’ve already written about the stupidity of hosting your company trade secrets at a company in a foreign country with foreign legislation), but as a place to quickly and easily dump some code you want to be publically available without going through all the hassle otherwise associated with public project hosting.

This is why this little script is hosted there and not here. As I’m using git, even if github goes away, I still have the full repository around to either self-host or let someone else host for me, which is a crucial requirement for me to outsource anything.

Broken by design

The concept sounds nice: To control all the various remote controllable devices you accumulate in your home cinema, why not just use one programmable remote? With enough intelligence, I would even be able to do much more than provide some way of switching personality.

I mean: Press one button and you have a remote for your receiver, press another and it’ll be for your media center, but losing its receiver functionality.

Why not put it in “Media Mode” where it controls the volume by sending commands the receiver understands while still providing full navigation support for your media center.

Logitech’s Harmony family promises to provide that functionality.

Unfortunately, it’s broken by design as

it tries to be intelligent while it is completely stupid. For example, I can add a “Music Player”-Functionality, with the intention of it sending commands to a Squeezebox, but as soon as you add a media center, it insists to use that to play music without a way to change that.
The web based programming interface is awful. It forces you through multi step assistants, each time reloading the (ugly) pages, asking questions which could easily be placed on one screen.
It only works on Mac and Windows (no Linux support)

Especially the first point rendered this interesting concept completely unusable for me.

Now, Engadget just had an article about project Concordance, a free software project allowing to access the functionality (the whole functionality) from any UNIX machine using a command line tool, while also providing a library (with Perl and Python bindings) for us to write a useful GUI for.

I can’t wait to try this out as this easily circumvents the awful UI and may actually provide me with means to make Harmony work for my setup.

Also, it’s a real shame to see a very interesting project be made completely unusable by bad UI design.

Dependent on working infrastructure

If you create and later deploy and run a web application, then you are dependent on a working infrastructure: You need a working web server, you need a working application server and in most cases, you’ll need a working database server.

Also, you’d want a solution that always and consistently works.

We’ve been using lighttpd/FastCGI/PHP for our deployment needs lately. I’ve preferred this to apache due to the easier configuration possible with lighty (out of the box automated virtual hosting for example), the potentially higher performance (due to long-running FastCGI processes) and the smaller amount of memory consumed by lighttpd.

But last week, I had to learn the price of walking off the beaten path (Apache, mod_php).

In one particular constellation, the lighty, fastcgi, php combination, running on a Gentoo box sometimes (read: 50% of the time) a certain script didn’t output all the data it should have. Instead, lighty randomly sent out RST packets. This without any indication of what could be wrong in any of the involved log files.

Naturally, I looked everywhere.

I read the source code of PHP. I’ve created reduced test cases. I’ve tried workarounds.

The problem didn’t go away until I tested the same script with Apache.

This is where I’m getting pragmatic: I depend on a working infrastructure. I need it to work. Our customers need it to work. I don’t care who is to blame. Is it PHP? Is it lighty? Is it Gentoo? Is it the ISP (though it would have to be on the senders end as I’ve seen the described failure with different ISPs)?

I don’t care.

My interest is in developing a web application. Not in deploying one. Not really, anyways.

I’m willing (and able) to fix bugs in my development environment. I may even be able to fix bugs in my deployment platform. But I’m certainly not willing to. Not if there is a competing platform that works.

So after quite some time with lighty and fastcgi, it’s back to Apache. The prospect of having a consistently working backed largely outweighs the theoretical benefits of memory savings, I’m afraid.