Tag: popscan

background events

Today is the day that one of the coolest things I had the pleasure to
develop so far in my life has gone live to production use.

One installation of PopScan is connected to
a SAP system that had at times really bad performance and yet it
needed to be connected even just to query for price information.

This is a problem because of features like our persistent shopping
basket or the users templates which cause a lot of products to be
displayed at once.

Up until now, PopScan synchronously queried for the prices and would
not render any products until all the product data has been assembled.

When you combine this with the sometimes bad performance of that SAP
system, you’ll quickly see unhappy users waiting for the pages to
finally load.

We decided to fix this problem for the users.

Aside of the price, all product data is in PopScan’s database anyways, so
while we need to wait for prices, everything else, we could display
immediately.

So that’s what we do now: Whenever we load products and we don’t have a price
yet, we’ll launch a background job which asynchronously retrieves the prices.
The frontend will immediately get the rendered products minus the prices.

But of course, we still need to show the user the fully loaded products once
they become available and this is where the cool server based event framework
comes into play:

The JS client in PopScan now gets notified on arbitrary events that can happen
on the server (like “product data loaded”, but also “GPRS scaner data
received”). The cool thing about this is that events are seemingly pushed
through instantly as they happen on the server giving the user the immediate
response they would want and lessening the load on the server as there’s no
(well. only long-) polling going on.

$(ServerEvents).bind('product-data', function(data){
    // product data has changed!
}

is all that we need on the client. The rest happens automatically.

Also remember though that PopScan is often used in technology-hostile
enterprise environments. Thus, features like web-sockets are out and in
general, we had to support ancient software all over the place.

We still managed to make it work and today this framework went to production
use for that one customer with the badly performing SAP system.

Over the course of the next few weeks, I might write in detail about how this
stuff works given the constratins (ancient client-software behind hostile
firewalls) and what software components we used.

Seeing this work go life fills me with joy: I’ve spend to many hours designing
this framework in a fool-proof way in order to not lose events and in order to
gracefully continue working as components in the big picture die.

Now it’s finally live and already contributing to lower waiting times for all
users.

How we use git

the following article was a comment I made on Hacker News, but as it’s quite big and as I want to keep my stuff at a central place, I’m hereby reposting it and adding a bit of formating and shameless self-promotion (i.e. links):

My company is working on a – by now – quite large web application. Initially (2004), I began with CVS and then moved to SVN and in the second half of last year, to git (after a one-year period of personal use of git-svn).

We deploy the application for our customers – sometimes to our own servers (both self-hosted and in the cloud) and sometimes to their machines.

Until middle year, as a consequence of SVN’s really crappy handling of branches (it can branch, but it fails at merging), we did very incremental development, adding features on customer requests and bugfixes as needed, often times uploading specific fixes to different sites, committing them to trunk, but rarely ever updating existing applications to trunk to keep them stable.

Huge mess.

With the switch to git, we also initiated a real release management, doing one feature release every six months and keeping the released versions on strict maintenance (for all intents and purposes – the web application is highly customizable and we do make exceptions in the customized parts as to react to immediate feature-wishes of clients).

What we are doing git-wise is the reverse of what the article shows: Bug-fixes are (usually) done on the release-branches, while all feature development (except of these customizations) is done on the main branch (we just use the git default name “master”).

We branch off of master when another release date nears and then tag a specific revision of that branch as the “official” release.

There is a central gitosis repository which contains what is the “official” repository, but every one of us (4 people working on this – so we’re small compared to other projects I guess) has their own gitorious clone which we heavily use for code-sharing and code review (“hey – look at this feature I’ve done here: Pull branch foobar from my gitorious repo to see…”).

With this strict policy of (for all intents and purposes) “fixes only” and especially “no schema changes”, we can even auto-update customer installations to the head of their respective release-branches which keeps their installations bug-free. This is a huge advantage over the mess we had before.

Now. As master develops and bug-fixes usually happen on the branch(es), how do we integrate them back into the mainline?

This is where the concept of the “Friday merge” comes in.

On Friday, my coworker or I usually merge all changes in the release-branches upwards until they reach master. Because it’s only a week worth of code, conflicts rarely happen and if they do, we remember what the issue was.

If we do a commit on a branch that doesn’t make sense on master because master has sufficiently changed or a better fix for the problem is in master, then we mark these with [DONTMERGE] in the commit message and revert them as part of the merge commit.

On the other hand, in case we come across a bug during development on master and we see how it would affect production systems badly (like a security flaw – not that they happen often) and if we have already devised a simple fix that is save to apply to the branch(es), we fix those on master and then cherry-pick them on the branches.

This concept of course heavily depends upon clean patches, which is another feature git excels at: Using features like interactive rebase and interactive add, we can actually create commits that

  • Either do whitespace or functional changes. Never both.
  • Only touch the lines absolutely necessary for any specific feature or bug
  • Do one thing and only one.
  • Contain a very detailed commit message explaining exactly what the change encompasses.

This on the other hand, allows me to create extremely clean (and exhaustive) change logs and NEWS file entries.

Now some of these policies about commits were a bit painful to actually make everyone adhere to, but over time, I was able to convince everybody of the huge advantage clean commits provide even though it may take some time to get them into shape (also, you gain that time back once you have to do some blame-ing or other history digging).

Using branches with only bug-fixes and auto-deploying them, we can increase the quality of customer installations and using the concept of a “Friday merge”, we make sure all bug-fixes end up in the development tree without each developer having to spend an awful long time to manually merge or without ending up in merge-hell where branches and master have diverged too much.

The addition of gitorious for easy exchange of half-baked features to make it easier to talk about code before it gets “official” helped to increase the code quality further.

git was a tremendous help with this and I would never in my life want to go back to the dark days.

I hope this additional insight might be helpful for somebody still thinking that SVN is probably enough.