VMWare Server, chrony and slow clocks

We have quite many virtual machines running under VMWare server. Some for testing purposes, some for quite real systems serving real webpages.

It’s wonderful. Need a new server? Just cp -r the template I created. Need more RAM in your server? No problem. Just add it via the virtual machine configuration file. Move to another machine? No problem at all. Power down the virtual machine and move the file where you want it to be.

Today I noticed something strange: The clocks on the virtual machines were way slow.

One virtual second was about ten real seconds.

This was so slow that chrony which I used on the virtual machines thought that the data sent from the time servers was incorrect, so chrony was of no use.

After a bit of digging around, I learned that VMware server needs access to /dev/rtc to provide the virtual machines with an usable time signal (usable as in “not too slow”).

The host’s /var/log/messages was full of lines like this (you’ll notice that I found yet another girl from a console RPG to name that host):

Dec 15 16:12:58 rikku /dev/vmmon[6307]: /dev/rtc open failed: -16
Dec 15 16:13:08 rikku /dev/vmmon[6307]: host clock rate change request 501 -> 500

-16 means “device busy”

The fix was to stop chrony from running on the host machine so VMWare could open /dev/rtc. This made the error messages vanish and additionally it allowed the clocks of the virtual machines to work correctly.

Problem solved. Maybe it’s useful for you too.

ServeRAID – Fun with GUI-Tools

We’ve recently bought three more drives for our in-house file server. Up until now, we had a RAID 5 array (using a IBM ServeRAID controller) spawning three 33GB drives. That array recently got very, very close to being full.

So today, I wanted to create a second array using the three new 140GB drives.

When you download the ServeRAID support CD image, you get access to a nice GUI-tool which is written in Java and can be used to create Arrays on these ServeRAID controllers.

Unfortunately, I wasn’t able to run the GUI at first because somehow, the Apple X11 server wasn’t willing/able to correctly display the GUI. I always got empty screens when I tried (the server is headless, so I had to use X11 forwarding via ssh).

Using a Windows machine with Xming (which is very fast, working perfectly and totally free as in speech) worked though and I got the GUI running.

All three drives where recognized, but one was listed as “Standby” and could not be used for anything. Additionally, I wasn’t able to find any way in the GUI to actually move the device from Standby to Ready.

Even removing and shuffling the drives around didn’t help. That last drive was always recognized as “Standby”, independant of the bay I plugged it into.

Checking the feature list of that controller showed nothing special – at first I feared that the controller just didn’t support more than 5 drives. That fear was without reason though: The controller supports up to 32 devices – more than enough for the server’s 6 drive bays.

Then, looking around on the internet, I didn’t find a solution for my specific problem, but I found out about a tool called “ipssend” and there was documentation how to use it in an old manual by IBM.

Unfortunately, newer CD images don’t contain ipssend any more, Forcing you to use the GUI which in this case didn’t work for me. It may be that there’s a knob to turn somewhere, but I just failed to see it.

In the end, I found a very, very old archive at the IBM website which was called dumplog and contained that ipssend command in a handy little .tgz archive. Very useful.

Using that utility solved the problem for me:

# ./ipssend setstate 1 1 5 RDY

No further questions asked.

Then I used the Java-GUI to actually create the second array.

Now I’m asking myself a few questions:

  • Why is the state “Standby” not documented anywhere (this is different from a drive in Ready state configured as Standby drive)?
  • Why is there no obvious way to de-standby a drive with the GUI?
  • Why isn’t that cool little ipssend utility not officially available any more?
  • Why is everyone complaining that command line is more complicated to use and that GUIs are so much better when obviously, the opposite is true?

podcast recommendation

I haven’t been much into podcasts till now: The ones I heard were boring, unprofessional or way too professional. Additionally, I didn’t have a nice framework set up to get them and to listen to them.

That’s because I don’t often sync my ipod. Most of the time, it’s not connected to a computer: About once every two months, I connect it to upload a new batch of audiobooks (I can’t fit my whole connection on the nano). So podcasting was – even if I had found one that I could interest myself in, an experience to have while behind the computer monitor.

Now two things have changed:

  1. I found the Linux Action Show. They guy doing that podcast are incredibly talented people. The entries sound very professionally made, while still not being on the obviously commercial side of things. They cover very, very interesting topics and they are everything but boring. Funny, entertaining and competent. Very good stuff.
  2. At least since the release of SlimServer 6.5, my Squeezebox is able to tune into RSS feeds with enclosures (or podcast for the less technical savy people – not that those would read this blog). Even better: The current server release brought a firmware which finally gives the Squeezebox the capability of natively playing ogg streams.

    Up until now, it could only play FLAC, PCM and MP3, requiring tools like sox to convert ogg streams on the fly. Unfortunately, that didn’t work as stable as I would have liked, but native OGG support helped a lot

So now, whenever a new episode of the podcast is released (once per week – and each episode is nearly two hours in length), I can use my Squeezebox to hear it via my home stereo.

Wow… I’m so looking forward to do that in front of a cozy fire in my fireplace once I can finally move into my new flat.

ripping DVDs

I have plenty of DVDs in my possession: Some movies of dubious quality which I bought when I was still going to school (like “Deep Rising” – eeew) and many, many episodes of various series (Columbo, the complete Babylon 5 series, A-Team and other pearls).

As you may know, I’m soon to move into a new flat which I thought would be a nice opportunity to reorganize my library.

shion has around 1.5TB of storage space and I can easily upgrade her capacity (shion is the only computer I own I’m using a female pronoun for – the machine is something really special to me – like the warships of old times) by plugging in yet another USB hub and USB hard drives.

It makes totally sense to use that unlimited amount of storage capacity to store all my movies – not only the ones I’ve downloaded (like video game speed runs). Spoiled by the ease of use of ripping CDs, I thought, that this would be just another little thing to do before moving.

You know: Enter the DVD, use the ripper, use the encoder, done.

Unfortunately, this is proving to be harder than it looked like in the first place:

  • Under Mac OS X, you can try to use the Unix tools with fink or some home-grown native tools. Whatever you do, you either get outdated software (fink) or not really working freeware tools documented in outdated tutorials. Nah.
  • Under Windows, there are two kinds of utilities: On one hand, you have the single-click ones (like AutoGK) which really do what I initially wanted. Unfortunately, they are limited in their use: They provide only a limited amount of output formats (like no x264) and they hard-code the subtitles into the movie stream. But they are easy to use. On the other hand, you have the hardcore tools like Gordian Knot or MeGUI or even StaxRip. These tools are frontends for other tools that work like Unix tools: Each does one thing, but tries to excel at that one thing.

    This could be a good thing, but unfortunately, it fails at things like awful documentation, hard-coded paths to files everywhere and outdated tools.

    I could not get any of the tools listed above to actually create a x264 AVI or MKV-File without either throwing a completely unusable error message (“Unknown exception ocurred”) or just not working at all or missing things like subtitles.

  • Linux has dvd::rip which is a really nice solution, but unfortunately, no solution for me as I don’t have the right platform to run it on: My MCE machine is – well – running Windows MCE, my laptop is running Ubuntu (no luck with the debian packages and no ubuntu-packages). shion is running Gentoo, but she’s headless, so I have to use a remote X-connection which is awfully slow and non-scriptable.

The solution I want works on the Linux (or MacOS X) console, is scriptable and – well – works.

I guess I’m going the hard-core way and use transcode which is what dvd::rip is using – provided I find good documentation (I’m more than willing to read and learn – if the documentation is current enough and actually documents the software that I’m running and not the software at the state of two years ago).

I’ll keep you posted on how I’m progressing.

Intel Mac Mini, Linux, Ethernet

If you have one of these new Intel Macs, you will sooner or later find yourself in the situation of having to run Linux on one of them. (Ok. Granted: The situation may be coming sooner for some than for others).

Last weekend, I was in that situation: I had to install Linux on an Intel Mac Mini.

The whole thing is quite easy to do and if you don’t need Mac OS X, you can just go ahead and install Linux like you would on any other x86 machine (provided the hardware is sufficiently new to have the BIOS emulation layer already installed – otherwise you have to install the Firmware Update first – you’ll notice by the mac not booting from the CD despite holding c during the initial boot sequence).

You can partition the disk to your liking – the Mac bootloader will notice that there’s something fishy with the parition layout (the question-mark-on-a-folder icon will blink one or two times) before passing control to the BIOS emulation which will be able to boot Linux from the partitions you created during installation.

Don’t use grub as bootloader though.

I don’t know if it’s something grub does to the BIOS or if it’s something about the partition table, but grub can’t launch stage 1.5 and thus is unable to boot your installation.

lilo works fine though (use plain lilo when using the BIOS emulation for the boot process, not elilo)

When you are done with the installation process, something bad will happen sooner or later though: Ethernet will stop working.

This is what syslog has to say about it:

NETDEV WATCHDOG: eth0: transmit timed out
sky2 eth0: tx timeout
sky2 eth0: transmit ring 60 .. 37 report=60 done=60
sky2 hardware hung? flushing

When I pulled the cable and plugged it in again, the kernel even oops’ed.

The macs have a Marvel Yukon ethernet chipset. This is what lspci has to tell us: 01:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller (rev 22). The driver to use in the kernel config is “SysKonnect Yukon2 support (EXPERIMENTAL)” (CONFIG_SKY2)

I guess the EXPERIMENTAL tag is warranted for once.

The good news is, that this problem is fixable. The bad news is: It’s tricky to do.

Basically, you have to update the driver with the version that is in the repository of what’s going to be kernel 2.6.19

Getting a current version of sky.c and sky.h is not that difficult. Unfortunately though, the new driver won’t compile with the current 2.6.18 kernel (and upgrading to a pre-rc is out of the question – even more so considering the ton of stuff going into 2.6.19).

So first, we have to patch in this changeset to make the current release of sky compile.

Put the patch to /usr/src/linux and patch with patch -p1

Then fetch the current revision of sky2.c and sky2.h and overwrite the existing files. I used the web interface to git for that as I have no idea how the command line tools work.

Recompile the thing and reboot.

For me, this fixed the problem with the sky2 driver: The machine in question is now running for a whole week without any networking lockups – despite heavy network load at times.

While happy to see this fixed, my statement about not buying too new hardware (posting number 6 here on gnegg.ch – ages ago) if you intend to use Linux on it seems to continue to apply.