Information Technology - 5

In short: Debian lenny’s current libc6 version 2.7 doesn’t work on kernel 2.6.9 with SMP enabled. Downgrade to libc6 2.6 if you can’t change the kernel. You might have to do this on a non-2.6.9 (or >2.6.9) system.

Here’s the story: I use to maintain a duplicate system (on real hardware) with the same package structure as my virtual private server to be able to run through the upgrade procedures in advance. Yesterday, the regular upgrade included a new libc6 package going from 2.6.1 to 2.7. As everything worked well on the duplicate system, I started the apt-get upgrade on the virtual machine. But then dpkg crashed with an “Unknown error 530”. I could still navigate through the file system, but directory listings returned e.g.

and the system slowly began to “fade”. I tried a reboot, but the system didn’t come up again, showing a problem with the quota in the log (“Running vzquota off failed... vzquota: (error) Quota off syscall for id 12345: Device or resource busy”).

I opened a ticket at my provider, and the guy was telling me that the system were completely broken and had to be reinstalled! I was shocked and didn’t believe him. The virtualization engine provides a repair mode, but as I chroot’ed into the mounted root directory of my system, those errors came up again.

I asked the support guy to have a look at that article, but he didn’t seem to find it useful. As it became late afternoon and their support went off duty, I synced the mounted repair directory to my duplicate system overnight (about 5.6GB; I had to install rsync into the temporary system). This should enable me to chroot there and make experiments.

As I finally chroot’ed into that directory this morning, the system was working perfectly! So it couldn’t be completely broken. Another inquiry on Google unveiled this new posting. So the problems are really due to the new libc6 version! It worked on the duplicate system as it runs on kernel 2.6.22, and the virtual server depends on a custom 2.6.9! And that’s also why I couldn’t repair it with their repair system, as it uses the same kernel! On the duplicate system I looked into /var/log/dpkg.log what the previous version of libc6 was, and downloaded the libc6_2.6.1-1+b1_i386.deb from a Debian mirror. I did the downgrade and synced the directory back to the virtual server’s repair directory. I went out of repair mode and—lo and behold!—the system came up again! I also had to downgrade the locale package to match with the old libc6 version.

The support guy was glad that I provided them with a solution for a problem that could emerge for other customers as well. I’ll better stick to Debian ‘stable’ as soon as it is out.

Addendum 12/08: The correct way to fetch previous package versions is via the Debian snapshot archive, as they’re removed from the official mirrors within a short time.

I finally had to buy a 160GB notebook disk and already transferred all data from the old 60GB disk to the new one by using an IDE-to-USB converter. After having transferred the remaining data from my aging backup PC, I’ll have 52GB free space.

As that backup host dates back to 2001, has various quirks (Gentoo Linux, /usr on LVM on RAID-5 ) and isn’t really reliable anymore for data backup (the array sometimes dissolves), I’ll have to change some things, but won’t buy new hardware. I’ll replace Gentoo with Debian and arrange the three 80GB disks as a linear NRAID array. The system will take 5GB on RAID-1 over three disks, and the remaining three 75GB partitions become a linear 225GB device. This means that if one disk fails, only the data on that disk is lost. But that doesn’t matter, as I still have the things on my notebook anyway. Various personal and multimedia files will be backed up on DVD unregularly, and my photos are already getting backed up to CDs and stored out of house.

I elaborated the following strategy, mainly leering towards collecting a huge number of photos with my future DSLR camera:

• Short-term storage: Notebook. Here I’ll keep the most recent pictures only, for digital manipulation or simply for quick access.
• Mid-term storage: Backup host (without RAID), DVDs or plain external USB harddisks. Everything that’s on my notebook is also on that disks. But the disks will contain all pictures ever made. If a disk breaks, I’ll either have the data on the notebook or on the DVDs anyway. Only photos undergo the long-term storage.
• Long-term storage of photos: CDs. Only CDs are designed for long-term storage, DVDs aren’t. And to be safe from environmental influences, they are transferred to a house 160km away from here. Sure, CDs only take 700MB, but they’re cheap; for duplicating a possible 500GB disk full of photos, I’ll need 732 CDs. Gee! Maybe I should nevertheless take DVD±Rs for that, where I’ll only need 125 pieces. But they might not be readable anymore in a few years.

The reason for considering an external 500GB disk is that portable data tanks for photographers nowadays have a capacity of 40-80GB. I’ll reuse my 60GB notebook disk by placing it into a corresponding device. If I really manage to fill it on a holiday (>3000 RAWs?? Well, 210 RAWs per day on an exciting 2-week holiday possible, why not?), this already takes 88 CDs for long-term storage. That takes a lot of time to burn, and a lot of room to store. We’ll see how my photo rate of yield really changes.

Addendum: It seems that DVD-RAM is the appropriate media, as it lasts for 30 years according to Wikipedia, whereas CDs only last for a few years. I’d need 15 DVD-RAMs for a filled 60GB data tank, what is a more reasonable number.

I wanted to do a regular upgrade of xserver-xorg-core in Debian ‘testing’. Unfortunately, dpkg complained about a file ./usr/lib/xorg/modules/extensions/libglx.so not being present, although Nvidia’s installer had placed it right there and it was still present. Because of that broken upgrade my X server couldn’t start anymore. Removal of Nvidia or xorg didn’t change anything on this situation. With Google I saw that someone found a solution by installing nvidia-glx-legacy temporarily. Here’s what to do:

1. Remove Nvidia by calling sh /path/to/NVIDIA-$SOMETHING --uninstall.
2. Install nvidia-glx-legacy. This package depends on a linux-image-2.6.18-n-486, what will therefore be installed and will replace your current symlinks /vmlinuz and /initrd.img.
3. Install/upgrade xserver-xorg-core. (The top-dependency would be xorg).
4. Remove nvidia-glx-legacy and the above mentioned linux-image-$SOMETHING again and purge their configuration files (Key “_” in aptitude).
5. Restore your original symlinks /vmlinuz[.old] and /initrd.img[.old]. You’ll have to run lilo. Run lilo. Did I mention to run lilo?
6. Reinstall Nvidia and restart gdm.

Recently, I had another issue with the Nvidia installer: It didn’t work anymore due to the activated paravirtualization feature in Debian’s default kernel what conflicts with the GPL-incompatible module nvidia.ko. Here’s a solution I found in the web:

1. Install linux-source-2.6.xx-n-686.
2. Uncompress /usr/src/linux-source-2.6.xx-n-686.tar.bz2.
3. Delete the symlink /lib/modules/2.6.xx-n-686/build and make it new by ln -s /usr/src/linux-source-2.6.xx /lib/modules/2.6.xx-n-686/build.
4. Copy .config from headers to sources by cp /usr/src/linux-headers-2.6.xx-n-686/.config /usr/src/linux-source-2.6.xx.
5. In the sources dir make menuconfig and disable paravirtualization in section ‘processor features’.
6. make prepare.
7. make scripts.
8. Now compile and install Nvidia with its own installer.

For the sake of documentation, I list the enhancements I configured for SpamAssassin since February:

• I use SA’s internal sa-update script nightly to update the standard rules that are changed between the releases of new versions. The standard channel is updates.spamassassin.org. The files automatically go into /var/lib/spamassassin/$VERSION and Debian automatically finds it, as can be seen from a call of spamassassin -D --lint.
• In addition to the previous, I call sa-compile to compile the body rulesets into binary form. This makes checking of body rules more efficient. To enable SA using these binary rules in /var/lib/spamassassin/compiled/$VERSION, I have to activate the Rule2XSBody plugin in /etc/spamassassin/v320.pre.
• I don’t update the SARE rules against stock spam manually anymore, but also use the provided sa-update channel for it.

Recently, PDF spam has become “popular”. Therefore I enabled some more things to accomplish this:

• The ClamAV virus scanner provides inofficial databases by Sanesecurity to catch various sorts of spam. So there’s actually some kind of redundancy, as the virus scanner and the anti-spam filter are now sharing some responsibility. As not a single virus occured at my site in the recent months, this has now changed in some way.
• I installed the PDFInfo plugin from SARE and enabled it in /etc/spamassassin/init.pre.

Sure, as soon as we catch enough of that new PDF spam, spammers might change to some other document file format, such as DOC or RTF or even ODF, and we are forced to scan those attached documents for spam text or even for contained images that contain spam text, what we are already considering with FuzzyOCR. There must be some better way, actually.

However, I had to reduce the score of the Botnet plugin, as the default value of 5 points is way too high. Maybe I should add the plugin that checks the operating system such that only botnet clients using that Windows crap get high scores.

The fight continues.

The following video I found on YouTube shows a bot gameplay of DEFCON, a computer game inspired by the film WarGames. This game is even available for GNU/Linux. It is a simulation of a global thermonuclear war and therefore of my most fearful imagination. The background music of this game is horrible, but in a certain way: Its composer perfectly creates a terrifying atmosphere. The game in this video runs at full speed, whereas it is even more spooky when you try that game in realtime, watching how the ICBM’s trails slowly rise and pave their way through the sky for half an hour until they reach their targets. The impact is accompanied by a creepy close rumbling. Unfortunately, the video is too short for the sounds of a crying woman to come up. Turn up your speakers with lots of bass and consider this:

In my notebook, I have a 60GB disk, 27+14=41GB considered for the whole Linux system and 11.5GB left for various personal files. In my backup PC, I have a 112GB /home partition (LVM on RAID-5) and 12GB left, another 12GB are available on the /usr partition and 8.6GB in /opt, and as everything is on LVM, it might be resizable. So there’s currently no real need for additional disk space. The only drawback is that I can’t hold all music files on my notebook and that the available space is splitted into two partitions. But I can live with that, I just have to consider that some things will be in a mounted subdirectory.

When space will get small, I’ll buy a 160GB notebook-disk. And my backup PC will be a new one with two 500GB SATA-disks on RAID-1. I won’t take a kind of commercial network storage array, as these aren’t capable of rsync or the like.

I thought about taking RAID-5 again instead of RAID-1. Three 320GB disks are cheaper than two with 500GB, but data redundancy decreases from 2 to 1.5. Only one disk may fail in both cases, no matter if you’ve got two disks at RAID-1 or three disks at RAID-5. In addition, the probability that two disks fail is three(!) times higher when there are three disks as if there were only two. For me, RAID-5 is therefore just a strategy for expansion, not for starting freshly.

I posted my question to comp.os.linux.security and continued the discussion on the Serendipity mailing list. For the sake of documentation and to provide another spot in the net with a solution, I repeat the posting here:

I wondered about strange HTTP connections from 127.0.0.1 appearing in my access.log at irregular times:

What irritated me was that those requests originate locally, are invalid (400 = Bad Request) and have no User-Agent identification string. [...]

I finally found out that this ought to be Apache-2.2’s internal dummy connections. They had the above form as long as my Apache-SSL config looked like

Now, I use the IP instead of the ‘*’ and—lo and behold—the requests transform into

I didn’t want to spend much time trying to understand what that dummy connections are good for. It seems like Apache2 kills some of its children such that the number of MaxSpareServers isn’t exceeded. And I wasn’t aware that the Apache syntax ‘*:443’ is somehow deprecated.

I want to keep track of the spam scores that show up in my /var/log/mail.log on my Debian Etch machine. My idea was to run a shell script grep’ing the wanted info out of the mail.log right before that file is to be rotated. logrotate provides configuration options for how and when log files should be rotated. This includes a possibility to run a script right before and after the rotation is done. So, I created a configuration /etc/logrotate.d/mail like

sysklogd is Debian’s package containing syslogd and klogd. The syslog itself already cares for rotating the basic log files, and this includes /var/log/mail.{log,info,warn,err}. sysklogd contains the cron-jobs /etc/cron.daily/sysklogd and /etc/cron.weekly/sysklogd, both of which contain logic for rotating the system’s log files. To obtain the list of the log files, the command syslogd-listfiles is issued in both of the scripts—the weekly script uses the option --weekly. When I issue syslogd-listfiles, I only get /var/log/syslog as answer, whereas with --weekly I get:

So I have to use the -s switch to exclude mail.log and mail.info such that logrotate can take care of them. As I also have a separate config for /var/log/messages in /etc/logrotate.d/messages, I modified the call of syslogd-listfiles --weekly in /etc/cron.weekly/sysklogd to