How we use git

the following article was a comment I made on Hacker News, but as it’s quite big and as I want to keep my stuff at a central place, I’m hereby reposting it and adding a bit of formating and shameless self-promotion (i.e. links):

My company is working on a – by now – quite large web application. Initially (2004), I began with CVS and then moved to SVN and in the second half of last year, to git (after a one-year period of personal use of git-svn).

We deploy the application for our customers – sometimes to our own servers (both self-hosted and in the cloud) and sometimes to their machines.

Until middle year, as a consequence of SVN’s really crappy handling of branches (it can branch, but it fails at merging), we did very incremental development, adding features on customer requests and bugfixes as needed, often times uploading specific fixes to different sites, committing them to trunk, but rarely ever updating existing applications to trunk to keep them stable.

Huge mess.

With the switch to git, we also initiated a real release management, doing one feature release every six months and keeping the released versions on strict maintenance (for all intents and purposes – the web application is highly customizable and we do make exceptions in the customized parts as to react to immediate feature-wishes of clients).

What we are doing git-wise is the reverse of what the article shows: Bug-fixes are (usually) done on the release-branches, while all feature development (except of these customizations) is done on the main branch (we just use the git default name “master”).

We branch off of master when another release date nears and then tag a specific revision of that branch as the “official” release.

There is a central gitosis repository which contains what is the “official” repository, but every one of us (4 people working on this – so we’re small compared to other projects I guess) has their own gitorious clone which we heavily use for code-sharing and code review (“hey – look at this feature I’ve done here: Pull branch foobar from my gitorious repo to see…”).

With this strict policy of (for all intents and purposes) “fixes only” and especially “no schema changes”, we can even auto-update customer installations to the head of their respective release-branches which keeps their installations bug-free. This is a huge advantage over the mess we had before.

Now. As master develops and bug-fixes usually happen on the branch(es), how do we integrate them back into the mainline?

This is where the concept of the “Friday merge” comes in.

On Friday, my coworker or I usually merge all changes in the release-branches upwards until they reach master. Because it’s only a week worth of code, conflicts rarely happen and if they do, we remember what the issue was.

If we do a commit on a branch that doesn’t make sense on master because master has sufficiently changed or a better fix for the problem is in master, then we mark these with [DONTMERGE] in the commit message and revert them as part of the merge commit.

On the other hand, in case we come across a bug during development on master and we see how it would affect production systems badly (like a security flaw – not that they happen often) and if we have already devised a simple fix that is save to apply to the branch(es), we fix those on master and then cherry-pick them on the branches.

This concept of course heavily depends upon clean patches, which is another feature git excels at: Using features like interactive rebase and interactive add, we can actually create commits that

  • Either do whitespace or functional changes. Never both.
  • Only touch the lines absolutely necessary for any specific feature or bug
  • Do one thing and only one.
  • Contain a very detailed commit message explaining exactly what the change encompasses.

This on the other hand, allows me to create extremely clean (and exhaustive) change logs and NEWS file entries.

Now some of these policies about commits were a bit painful to actually make everyone adhere to, but over time, I was able to convince everybody of the huge advantage clean commits provide even though it may take some time to get them into shape (also, you gain that time back once you have to do some blame-ing or other history digging).

Using branches with only bug-fixes and auto-deploying them, we can increase the quality of customer installations and using the concept of a “Friday merge”, we make sure all bug-fixes end up in the development tree without each developer having to spend an awful long time to manually merge or without ending up in merge-hell where branches and master have diverged too much.

The addition of gitorious for easy exchange of half-baked features to make it easier to talk about code before it gets “official” helped to increase the code quality further.

git was a tremendous help with this and I would never in my life want to go back to the dark days.

I hope this additional insight might be helpful for somebody still thinking that SVN is probably enough.

JSONP. Compromised in 3…2…1…

To embed a vimeo video on some page, I had a look at their different methods for embedding and the easiest one seemed to be what is basically JSONP – a workaround for the usual restriction of disallowing AJAX over domain boundaries.

But did you know, that JSONP not only works around the subdomain restriction, it basically is one huge cross site scripting exploit and there’s nothing you can do about it?

You might have heard this and you might have found articles like this one thinking that using libraries like that would make you save. But that’s an incorrect assumption. The solution provided in the article has it backwards and only helps to protect the originating site against itself, but it does not help at all to protect the calling site from the remote site.

You see, the idea behind JSONP is that you source the remote script using <script src=”http://remote-service.example.com/script.js”> and the remote script then (after being loaded into your page and thus being part of your page) is supposed to call some callback of the original site (from a browsers standpoint it is part the original site).

The problem is that you do not get control over the loading let alone content of that remote script. Because the cross-domain restrictions prevent you from making an AJAX request to a remote server, you are using the native HTML methods for cross domain requests (which should not have been allowed in the first place) and at that moment you relinquish all control over your site as that remotely loaded script runs in the context of your page, which is how you get around the cross domain restrictions – by loading the remote script into your page and executing it in the context of your page.

Because you never see that script until it is loaded, you cannot control what it can do.

Using JSONP is basically subjecting yourself to an XSS attack by giving the remote end complete control over your page.

And I’m not just talking about malicious remote sites… what if they themselves are vulnerable to some kind of attack? What if they were the target of a successful attack? You can’t know and once you do know it’s too late.

This is why I would recommend you never to rely on JSONP and find other solutions for remote scripting: Use a local proxy that does sanitization (i.e. strict JSON parsing which will save you), rely on cross-domain messaging that was added in later revisions of the upcoming HTML5 standard.

Programming languages names

Today in the office, a discussion about the merits of Ruby compared to Python and the other way around (isn’t it fun to have people around actually willing to discuss such issues?) lead into us making fun of different programming languages by interjecting some sore points about them into their names.

The Skype conversation went roughly as follows (I removed some stuff for brevity but all the language names are intact):

thepilif: ja-long variable names and no function pointers-va really sucks
thepilif: though there’s always C(*^~**<<)++
thepilif: and then there’s alyways Del-Access violation at address 02E41C10. Read of address 02E41C10-phi
thepilif: or P-false==true-HP
Coworker: ok so for the sake of it i should add py thon
thepilif: or java-everything is global-script
thepilif: too bad it doesn’t work for C
thepilif: C-sigsegv
thepilif: they know why they just chose one letter
Coworker: exactly, k&r are smart
Coworker: has-how the fuck do i do a print-skell?
Coworker: pe/(^$^)/rl
thepilif: or pe-module? object? hash? what’s the difference-rl
Coworker: so we could say pe/$^/rl
thepilif: and ru-lets rewrite our syntax on the fly-by
Coworker: l(i(s(p)))
thepilif: can’t you wrap this into another pair of ()?
thepilif: (l(i(s(p))))))
Coworker: yes even better
thepilif: and add the syntax error
thepilif: one too many )
Coworker: it’s impossible to match them just by looking
thepilif: totally impossible. yes
Coworker: the human brain is no fucking pushdown automata
Coworker: but maybe the lisp people are
Coworker: vb! vb needs one
thepilif: visual-on error resume next-basic
thepilif: and of course brain-<<<<<******<<<>>>>-fuck
thepilif: c-tries to be dynamic, but var just doesn’t cut it-#
thepilif: c-not quite java nor c(++)?-#
thepilif: though the first one feels better
thepilif: oh.. and of course HT-unknown error-ML
thepilif: as a tribute to IE6
thepilif: and of course la-no bugs but still not usable-tex
thepilif: sorry, Knuth
thepilif: and send-$*$_**^$$$-mail

So the question is: Do you have anything to add? Do you feel that we were overly unfair?

Introducing sacy, the Smarty Asset Compiler

We all know how beneficial to the performance of a web application it can be to serve assets like CSS files and JavaScript files in larger chunks as opposed to smaller ones.

The main reason behind this is the latency incurring from requesting a resource from the server plus the additional bandwidth of the request metadata which can grow quite large when you take cookies into account.

But knowing this, we also want to keep files separate during development to help us with the debugging and development process. We also want the deployment to not increase too much in difficulty, so we naturally dislike solutions that require additional scripts to run at deployment time.

And we certainly don’t want to mess with the client-side caching that HTTP provides.

And maybe we’re using Smarty and PHP.

So this is where sacy, the Smarty Asset Compiler plugin comes in.

The only thing (besides a one-time configuration of the plugin) you have to do during development is to wrap all your <link>-Tags with {asset_compile}….{/asset_compile} and the plugin will do everything else for you, where everything includes:

  • automatic detection of actually linked files
  • automatic detection of changed files
  • automatic minimizing of linked files
  • compilation of all linked files into one big file
  • linking that big file for your clients to consume. Because the file is still served by your webserver, there’s no need for complicated handling of client-side caching methods (ETag, If-Modified-Since and friends): Your webserver does all that for you.
  • Because the cached file gets a new URL every time any of the corresponding source files change, you can be sure that requesting clients will retrieve the correct, up-to-date version of your assets.
  • sacy handles concurrency, without even blocking while one process is writing the compiled file (and of course without corrputing said file).

sacy is released under the MIT license and ready to be used (though it currently only handles CSS files and ignores the media-attribute – stuff I’m going to change over the next few days).

Interested? Visit the project’s page on GitHub or even better, fork it and help improving it!

Twisted Tornado

Lately, the net is all busy talking about the new web server released by FriendFeed last week and how their server basically does the same thing as the Twisted framework that was around so much longer. One blog entry ends with

Why Facebook/Friendfeed decided to create a new web server is completely beyond us.

Well. Let me add my two cents. Not from a Python perspective (I’m quite the Python newbie, only having completed one bigger project so far), but from a software development perspective. I feel qualified to add the cents because I’ve been there and done that.

When you start any project, you will be on the lookout for a framework or solution to base your work on. Often times, you already have some kind of idea of how you want to proceed and what the different requirements of your solution will be.

Of course, you’ll be comparing existing requirements against the solutions around, but chances are that none of the existing solutions will match your requirements exactly, so you will be faced with changing them to match.

This involves not only the changes themselves but also other considerations:

  • is it even possible to change an existing solution to match your needs?
  • if the existing solution is an open source project, is there a chance of your changes being accepted upstream (this is not a given, by the way).
  • if not, are you willing to back- and forward-port your changes as new upstream versions get released? Or are you willing to stick with the version for eternity, manually back-porting security-issues?

and most importantly

  • what takes more time: Writing a tailor-made solution from scratch or learning how the most-matching solutions ticks to make it do what you want?

There is a very strong perception around, that too many features mean bloat and that a simpler solution always trumps the complex one.

Have a look at articles like «Clojure 1, PHP 0» which compares a home-grown, tailor-made solution in one language to a complete framework in another and it seems to favor the tailor-made solution because it was more performant and felt much easier to maintain.

The truth is, you can’t have it both ways:

Either you are willing to live with «bloat» and customize an existing solution, adding some features and not using others, or you are unwilling to accept any bloat and you will do a tailor-made solution that may be lacking in features, may reimplement other features of existing solutions, but will contain exactly the features you want. Thus it will not be «bloated».

FriendFeed decided to go the tailor-made route but instead of many other projects each day who go the tailor made route (take Django’s reimplementations of many existing Python technologies like templating and ORM as another example) and keep using that internally, they actually went public.

Not with the intention to bad-mouth Twisted (though it kinda sounded that way due to bad choice of words), but with the intention of telling us: «Hey – here’s the tailor-made implementation which we used to solve our problem – maybe it is or parts of it are useful to you, so go ahead and have a look».

Instead of complaining that reimplementation and a bit of NIH was going on, the community could embrace the offering and try to pick the interesting parts they see fitting for their implementation(s).

This kind of reinventing the wheel is a standard process that is going on all the time, both in the Free Software world as in the commercial software world. There’s no reason to be concerned or alarmed. Instead we should be thankful for the groups that actually manage to put their code out for us to see – in so many cases, we never get a chance to see it and thus lose a chance at making our solutions better.

(Unicode-)String handling done right

Today, found myself reading the chapter about strings on diveintopython3.org.

Now, I’m no Python programmer by any means. Sure. I know my share of Python and I really like many of the concepts behind the language. I have even written some smaller scripts in Python, but it’s not my day-to-day language.

That chapter about string handling really really impressed me though.

In my opinion, handling Unicode strings they way python 3 is doing is exactly how it should be done in every development environment: Keep strings and collections of bytes completely separate and provide explicit conversion functions to convert from one to the other.

And hide the actual implementation from the user of the language! A string is a collection of characters. I don’t have to care how these characters are stored in memory and how they are accessed. When I need that information, I will have to convert that string to a collection of bytes, giving an explicit encoding how I want that to be done.

This is exactly how it should work, but implementation details leaking into the language are mushing this up in every other environment I know of making it a real pain to deal with multibyte character sets.

Features like this is what convinces me to look into new stuff. Maybe it IS time to do more python after all.

JavaScript and Applet interaction

As I said earlier this month: While Java applets are dead for games and animations and whatever else they were used back in the nineties, they still have their use when you have to access the local machine from your web application in some way.

There are other possibilities of course, but they all are limited:

  • Flash loads quickly and is available in most browsers, but you can only access the  hardware Adobe has created an API for. That’s upload of files the user has to manually select, webcams and microphones.
  • ActiveX doesn’t work in browsers, but only in IE.
  • .NET dito.
  • Silverlight is neither commonly installed on your users machines, nor does it provide the native hardware access.

So if you need to, say, access a bar code scanner. Or access a specific file on the users computer – maybe stored in a place that is inconvenient for the user to get to (%Localappdata% for example is hidden in explorer). In this case, a signed Java applet is the only way to go.

You might tell me that a website has no business accessing that kind of data and generally, I would agree, but what if your requirements are to read data from a bar code scanner without altering the target machine at all and without requiring the user to perform any steps but to plug the scanner and click a button.

But Java applets have that certain 1996 look to them, so even if you access the data somehow, the applet still feels foreign to your cool Web 2.0 application: It doesn’t quite fit the tight coupling between browser and server that AJAX gets us and even if you use Swing, the GUI will never look as good (and customized) as something you could do in HTML and CSS.

But did you know that Java Applets are fully scriptable?

Per default, any JavaScript function on a page can call any public method of any applet on the site. So let’s say your applet implements

public String sayHello(String name){
    return "Hello "+name;
}

Then you can use JavaScript to call that method (using jQuery here):

$('#some-div').html(
    $('#id_of_the_applet').get(0).sayHello(
        $('#some-form-field').val())
);

If you do that, you have to remember though that any applet method called this way will run inside the sandbox regardless if the applet is signed or not.

So how do you access the hardware then?

Simple: Tell the JRE that you are sure (you are. aren’t you?) that it’s ok for a script to call a certain method. To do that, you use AccessController.doPrivileged(). So if for example, you want to check if some specific file is on the users machine. Let’s further assume that you have a singleton RuntimeSettings that provides a method to check the existence of the file and then return its name, you could do something like this:

   public String getInterfaceDirectory(){
        return (String) AccessController.doPrivileged(
                new PrivilegedAction() {
                    public Object run() {
                        return RuntimeSettings.getInstance().getInterfaceDirectory();
                    }
                }
            );
    }

Now it’s safe to call this method from JavaScript despite the fact that RuntimeSettings.getInterfaceDirectory() directly accesses the underlying system. Whatever is in PrivilegedAction.run() will have full hardware access (provided the applet in question is signed and the user has given permission).

Just keep one thing in mind: Your applet is fully scriptable and if you are not very careful where that Script comes from, your applet may be abused and thus the security of the client browser might be at risk.

Keeping this in mind, try to:

  • Make these elevated methods do one and only one thing.
  • Keep the interface between the page and the applet as simple as possible.
  • In elevated methods, do not call into javascript (see below) and certainly do not eval() any code coming from the outside.
  • Make sure that your pages are sufficiently secured against XSS: Don’t allow any user generated content to reach the page unescaped.

The explicit and cumbersome declaration of elevated actions was put in place to make sure that the developer keeps the associated security risk in mind. So be a good developer and do so.

Using this technology, you can even pass around Java objects from the Applet to the page.

Also, if you need your applet to call into the page, you can do that too, of course, but you’ll need a bit of additional work.

  1. You need to import JSObject from netscape.javascript (yes – that’s how it’s called. It works in all browsers though), so to compile the applet, you’ll have to add plugin.jar (or netscape.jar – depending on the version of the JRE) from somewhere below your JRE/JDK installation to the build classpath. On a Mac, you’ll find it below /System/Library/Frameworks/JavaVM.framework/Versions/<your version>/Home/lib.
  2. You need to tell the Java plugin that you want the applet to be able to call into the page. Use the mayscript attribute of the java applet for that (interestingly, it’s just mayscript – without value, thus making your nice XHTML page invalid the moment you add it – mayscript=”true” or the correct mayscript=”mayscript” don’t work consistently on all browsers).
  3. In your applet, call the static JSObject.getWindow() and pass it a reference to your applet to acquire a reference to the current pages window-object.
  4. On that reference you can call eval() or getMember() or just call() to call into the JavaScript on the page.

This tool set allows you to add the applet to the page with 1 pixel size in diameter placed somewhere way out of the viewport distance and with visibility: hidden, while writing the actual GUI code in HTML and CSS, using normal JS/AJAX calls to communicate with the server.

If you need access to specific system components, this (together with JNA and applet-launcher) is the way to go, IMHO as it solves the anachronism that is Java GUIs in applets.

There is still the long launch time of the JRE, but that’s getting better and better with every JRE release.

I was having so much fun last week discovering all that stuff.

Do not change base library behavior

Modern languages like JavaScript or Ruby provide the programmer with an option to “reopen” any class to add additional behavior to them. In the case of Ruby and JavaScript, this is not constrained in any way: You are able to reopen any  class – even the ones that come with your language itself and there are no restrictions on the functionality of your extension methods.

Ruby at least knows of the concept of private methods and fields which you can’t call from your additional methods, but that’s just Ruby. JS knows of no such thing.

This provides awesome freedom to the users of these languages. Agreed. Miss a method on a class? Easy. Just implement that and call it from wherever you want.

This also helps to free you from things like

BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(of)));

which is lots of small (but terribly inconventiently named) classes wrapped into each other to provide the needed functionality. In this example, what the author wanted is to read a file line-by-line. Why exactly do I need three objects for this? Separation of concern is nice, but stuff like this make learning a language needlessly complicated.

In the world of Ruby or JS, you would just extend FileInputStream with whatever functionality you need and then call that, creating code that is much easier to read.

FileInputStream.prototype.readLine = function(){...}
//...
of.readLine();
//...

And yet, if you are a library (as opposed to consumer code), this is a terrible, terrible thing to do!

We have seen previous instances of the kind of problems you will cause: Libraries adding functionality to existing classes create real problems when multiple libraries are doing the same thing and the consuming application is using both libraries.

Let’s say for example, that your library A added that method sum() to the generic Array class. Let’s also say that your consumer also uses library B which does the same thing.

What’s the deal about this, you might ask? It’s pretty clear, what sum does after all?

Is it? It probably is when that array contains something that is summable. But what if there is, say, a string in the array you want to sum up? In your library, the functionality of sum() could be defined as “summing up all the numeric values in the array, assuming 0 for non-numeric values”. In the other library, sum() could be defined as “summing up all the numeric values in the array, throwing an exception if sum() encounters invalid value”.

If your consumer loads your library A first and later on that other library B, you will be calling B’s Array#sum().

Now due to your definition of sum(), you assume that it’s pretty safe to call sum() with an array that contains mixed values. But because you are now calling B’s sum(), you’ll get an exception you certainly did not expect in the first place!

Loading B after A in the consumer caused A to break because both created the same method conforming to different specs.

Loading A after B would fix the problem in this case, but what, say, if both you and B implement Array#avg, but with reversed semantics this time around?

You see, there is no escape.

Altering classes in the global name space breaks any name spacing facility that may have been available in your language. Even if all your “usual” code lives in your own, unique name space, the moment you alter the global space, you break out of your small island and begin to compete with the rest of the world.

If you are a library, you cannot be sure that you are alone in that competition.

And even if you are a top level application you have to be careful not to break implementations of functions provided by libraries you either use directly or, even worse, indirectly.

If you need a real-life example, the following code in an (outdated) version of scriptaculous’ effects.js broke jQuery, despite the latter being very, very careful to check if it can rely on the base functionality provided:

Array.prototype.call = function() {
 var args = arguments;
 this.each(function(f){ f.apply(this, args) });
}

Interestingly enough, Array#call wasn’t used in the affected version of the library. This was a code artifact that actually did nothing but break a completely independent library (I did not have time to determine the exact nature of the breakage).

Not convinced? After all I was using an outdated version of scriptaculous and I should have updated (which is not an option if you have even more libraries dependent on bugs in exactly that version – unless you update all other components as well and then fix all the then broken unit tests).

Firefox 3.0 was the first browser to add document.getElementByClassName, a method also implemented by Prototype. Of course the functionality in Firefox was slightly different from the implementation in Prototype, which now called the built-in version instead its own version which caused a lot of breakage all over the place.

So, dear library developers, stay in your own namespace, please. You’ll make us consumers (and your own) lives so much more easier.

This month’s find: jna and applet-launcher

Way way back, I was talking about java applets and native libraries and the things you need to consider when writing applets that need access to native libraries (mostly for hardware access). And let’s be honest – considering how far HTML and JavaScript have come, native hardware access is probably the only thing you still needs applets for.

Java is slow and bloated and users generally don’t seem to like it very much, but the moment you need access to specific hardware – or even just to specific files on the users filesystem, Java becomes an interesting option as it is the only technology readily available on multiple platforms and browsers.

Flash only works for hardware Adobe has put an API in (cameras and microphones) and doesn’t allow access to arbitrary files. .NET doesn’t work on browsers (it works on IE, but the solution at hand should work on browsers too) and ActiveX is generally horrible, doesn’t work in browsers and additionally only works under windows (.NET works in theory on Unixes and Macs as well).

Which leaves us with Java.

Because applets are scriptable, you get away with hiding the awful user interface that is Swing (or, god forbid, AWT) and writing a nice integrated GUI using web technologies.

But there’s still the issue with native libraries.

First, your applet needs to be signed – no way around that. Then, you need to manually transfer all the native libraries and extension libraries. Also, you’ll need to put them in certain predefined places – some of which require administration privileges to be written into.

And don’t get me started about JNI. Contrary to .NET, you can’t just call into native libraries. You’ll have to write your own glue layer between the native OS and the JRE. That glue layer is platform specific of course, so you better have your C compiler ready – and the plattforms you intend to run on, of course.

So even if Java is the only way, it still sucks.

Complex deployment, administrative privileges and antiquated glue layers. Is this what you would want to work with?

Fortunately, I’ve just discovered two real pearls completely solving the two problems leaving me with the hassle that is Java itself, but it’s always nice to keep some practice in multiple programming languages, as long as it doesn’t involve C shudder.

The first component I’m going to talk about is JNA (Java Native Access) which is for Java what P/Invoke is for .NET: A way for directly calling into the native API from your Java code. No JNI and thus no custom glue code and C compiler needed. Translating the native calls and structures into what JNA wants still isn’t as convenient as P/Invoke, but it sure as hell beats JNI.

In my case, I needed to get find the directory corresponding to CSIDL_LOCAL_APPDATA when running under Windows. While I could have hacked something together, the only really reliable way of getting the correct path is to query the Windows API, for which JNA proved to be the perfect fit.

JNA of course comes with its own glue layer (available in precompiled form for more plattforms than I would ever want to support in the first place), so this leads us directly to the second issue: Native libraries and applets don’t go very well together.

This is where applet-launcher comes into play. Actually, applet-launcher’s functionality is even built into the JRE itself – provided you target JRE 1.6 Update 10 and later, which isn’t realistic in most cases (just today I was handling a case where an applet had to work with JRE 1.3 which was superseded in 2002), so for now, applet-launcher which works with JRE 1.4.2 and later is probably the way to go.

The idea is that you embed the applet-launcher applet instead of the applet you want to embed in the first place. The launcher will download a JNLP file from the server, download and extract external JNI glue libraries and finally load your applet.

When compared with the native 1.6 method, this has the problem that the library which uses the JNI glue has to have some special hooks in place, but it works like a charm and fixes all the issues I’ve previously had with native libraries in applets.

These two components renewed my interest in Java as a glue layer between the webbrowser where your application logic resides and the hardware the user is depending upon. While earlier methods kind of worked but were either hacky or a real pain to implement, this is as clean as it gets and works like a charm.

And next time we’ll learn about scripting Java applets.

My new friend: git rebase -i

Last summer, I was into making git commits look nice with the intent of pushing a really nice and consistent set of patches to the remote repository.

The idea is that a clean remote history is a convenience for my fellow developers and for myself. A clean history means very well-defined patches – should a merge of a branch be neccesary in the future. It also means much easier hunting for regressions and generally more fun doing some archeology in the code.

My last post was about using git add -i to refine the commits going into the repository. But what if you screw up the commit anyways? What if you forget to add a new file and notice it only some commits later?

This is where git rebase -i comes into play as this allows you to reorder your local commits and to selectively squash multiple commits into one.

Let’s see how we would add a forgotten file to a commit a couple of commits ago.

  1. You add the forgotten file and commit it. The commit message doesn’t really matter here.
  2. You use git log or gitk to find the commit id before the one you want to amend this new file to. Let’s say it’s 6bd80e12707c9b51c5f552cdba042b7d78ea2824
  3. Pick the first few characters (or the whole ID) and pass them to git rebase -i.
 % git rebase -i 6bd80e12

git will now open your favorite editor displaying your list of commits since the revision you have given. This could look like this.

pick 6bd80e1 some commit message. This is where I have forgotten the file
pick 4c1d210 one more commit message
pick 5d2f4ed this is my forgotten file

# Rebase fc9a0c6..5d2f4ed onto fc9a0c6
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

The comment in the file says it all – just reorder the first three (or how many there are in your case) to look like this:

pick 6bd80e1 some commit message. This is where I have forgotten the file
squash 5d2f4ed this is my forgotten file
pick 4c1d210 one more commit message

Save the file. Git will now do some magic and open the text editor again where you can amend the commit message for the commit you squashed your file into. If it’s really just a forgotten file, you’ll probably keep the message the same.

One word of caution though: Do not do this on branches you have already pushed to a remote machine or otherwise shared with somebody else. git gets badly confused if it has to pull altered history.

Isn’t it nice that after moths you still find new awesomeness in your tool of choice?

I guess I’ll have to update my all-time favorite tools list. It’s from 2004, so it’s probably ripe for that update.

Git rules.