GNU/Linux

In short: Debian lenny’s current libc6 version 2.7 doesn’t work on kernel 2.6.9 with SMP enabled. Downgrade to libc6 2.6 if you can’t change the kernel. You might have to do this on a non-2.6.9 (or >2.6.9) system.

Here’s the story: I use to maintain a duplicate system (on real hardware) with the same package structure as my virtual private server to be able to run through the upgrade procedures in advance. Yesterday, the regular upgrade included a new libc6 package going from 2.6.1 to 2.7. As everything worked well on the duplicate system, I started the apt-get upgrade on the virtual machine. But then dpkg crashed with an “Unknown error 530”. I could still navigate through the file system, but directory listings returned e.g.

and the system slowly began to “fade”. I tried a reboot, but the system didn’t come up again, showing a problem with the quota in the log (“Running vzquota off failed... vzquota: (error) Quota off syscall for id 12345: Device or resource busy”).

I opened a ticket at my provider, and the guy was telling me that the system were completely broken and had to be reinstalled! I was shocked and didn’t believe him. The virtualization engine provides a repair mode, but as I chroot’ed into the mounted root directory of my system, those errors came up again.

I asked the support guy to have a look at that article, but he didn’t seem to find it useful. As it became late afternoon and their support went off duty, I synced the mounted repair directory to my duplicate system overnight (about 5.6GB; I had to install rsync into the temporary system). This should enable me to chroot there and make experiments.

As I finally chroot’ed into that directory this morning, the system was working perfectly! So it couldn’t be completely broken. Another inquiry on Google unveiled this new posting. So the problems are really due to the new libc6 version! It worked on the duplicate system as it runs on kernel 2.6.22, and the virtual server depends on a custom 2.6.9! And that’s also why I couldn’t repair it with their repair system, as it uses the same kernel! On the duplicate system I looked into /var/log/dpkg.log what the previous version of libc6 was, and downloaded the libc6_2.6.1-1+b1_i386.deb from a Debian mirror. I did the downgrade and synced the directory back to the virtual server’s repair directory. I went out of repair mode and—lo and behold!—the system came up again! I also had to downgrade the locale package to match with the old libc6 version.

The support guy was glad that I provided them with a solution for a problem that could emerge for other customers as well. I’ll better stick to Debian ‘stable’ as soon as it is out.

Addendum 12/08: The correct way to fetch previous package versions is via the Debian snapshot archive, as they’re removed from the official mirrors within a short time.