• View
• Edit
• History
• Print

HumbabaDebian7Upgrade

Upgrading OS from v6 "Squeeze" to v7 "Wheezy"

2017-01-11

While preparing/experimenting for an upgrade from Debian 6 "Squeeze" to Debian 8 "Jessie" or Debian 9 "Stretch" I realized that I could have a perfect backup of the system, including boot facilities and everything, if I did my upgrades with only one of the mirrored disks enabled at the time, then after a successful upgrade I could re-enable the 'backup' drive and resync. Or, if things don't go well, shut down, reboot with the 'backup' drive and then resync back to the original state.

I also realized that having more backups is more safety and I added another device (at least temporarily during the upgrade) to the RAID array giving me that much more security. Here's what I did:

1. physically attached another USB drive, which became /dev/sdd
2. cloned the disk partitioning with:
   • sfdisk -d /dev/sda | sfdisk -f /dev/sdd
3. added the new device to the two arrays, becoming a 'spare':
   • mdadm --manage /dev/md0 -add /dev/sdd1
   • mdadm --manage /dev/md5 -add /dev/sdd5
4. enable the 'spares', causing auto-sync
   • mdadm --grow /dev/md0 --level=1 --raid-devices=3
   • mdadm --grow /dev/md1 --level=1 --raid-devices=3
5. install grub bootloader to the new device
   • grub-install /dev/sdd

2017-01-12

Now, I have a complete and perfect copy of the the system. The third drive in the RAID array synced overnight. I removed it from the arrays:

• cat /proc/mdstat to confirm all okay
• mdadm --manage /dev/md0 --fail /dev/sdd1
• mdadm --manage /dev/md0 --remove /dev/sdd1
• mdadm --manage /dev/md1 --fail /dev/sdd5
• mdadm --manage /dev/md1 --remove /dev/sdd5

Then I physically removed the drive from the system.

The Debian folks DO NOT recommend upgrading over mulitple releases at once, rather an upgrade to each in turn, one at a time. In my case, I'll perform the upgrade this way:

• from Debian 6 "Squeeze" to Debian 7 "Wheezy"
• from Debian 7 "Wheezy" to Debian 8 "Jessie"
• from Debian 8 "Jessie" to Debian 9 "Stretch"

repeating the process for each release upgrade as directed in the relevant Release Notes for the target release.

Using an identical HP t5470 ("pancake") as the testing platform, I connected the third RAID drive to it alone and booted. Indeed, it booted just fine, only complaining that it could find only one drive for the RAID arrays instead of three. This was expected and is perfectly fine. Since this is a complete clone of my still-running main system, I had to change it's ethernet address and hostname so that it would not conflict with the original system:

• ifconfig eth1 192.168.1.217
• hostname humtest
• update the network configuration /etc/network/interfaces for future reboots with new ethernet address

Now, using directions from the Debian 7 "Wheezy" Release Notes as guidance, in particular, Chapter 4, I started the upgrade process.

First, I confirmed that the system was fully up to date, as far as Debian 6 "Squeeze" goes. This system is well past its support date, but I wanted to make sure that it was current.

• apt-get update

It was, so there was nothing further to do.

Next, was to determine if the system was 'clean' using Chapter 4 guidance. I checked for failed packages with:

• dpkg --audit

which came back clean.

Next I fired up aptitude and entered 'g' to see if any packages were marked for install, upgrade, or removal. There were two orphan packages that were no longer used, and I allowed aptitude to remove them. After that no other actions were pending, which is the desired situation. I should note that I have two 'obsolete' packages installed. These I want to keep for continued use in the upgraded system:

• teapop 0.3.7-4.2 which is from an earlier Debian release
• hd-idel 1.04 which was locally built and installed

Now, on with the process.

Change /etc/apt/sources.list to point to Debian 7 "Wheezy". I commented out the "Squeeze" lines and added the "Wheezy" lines:

• deb http://mirrors.xmission.com/debian/ wheezy main non-free contrib
• deb-src http://mirrors.xmission.com/debian/ wheezy main non-free contrib

Next, start the script recording, update the package list, select a kernal meta package, and perform the 'minimal package upgrade':

• script -t 2>~/upgrade-wheezy1.time -a ~/upgrade-wheezy1.script
• apt-get update which warns of no public key for one ID. This is okay.
• apt-get upgrade which shows 152 packages to upgrade, 322 not upgraded.
• I answered 'Y' to the prompt

After downloading all of the packages, a list of warnings was presented, indicating behavior changes in some software, including:

• apache2
• cron
• linux-base
• logrotate
• lsb
• pam
• patch
• pcmciautils
• sysvinit-utils
• vim

exited with 'q' and moved on. Package processing began. At one point, a slowly rendered and lengthy list of /etc/bash-completion.d/ files was given. Patience was required! Things continued.

I was queried about a changed /etc/init.d/hdparm script. The main difference is that I had spindown_time=6. I no longer use hdparm with this thin-client platform, so I selected 'Y' to install the package version.

Next:

• apt-get dist-upgrade

Warning notices for packages upgraded in this phase include:

• exim4
• mysql 5.5
• wget
• eglibc 2.13-25
• eglibec 2.13-7
• php5 (x8)
• apt
• ca-certificates (x6)
• cups
• cyrus-sasl2
• ifupdown
• imagemagick
• mdadm
• mutt
• netatalk
• procps
• rsyslog
• samba (x2)
• screen (x2)
• sgml-base
• smbnetfs

Most changes appear benign to my installation.

libc6: As packages were unpacked and installed, I was asked if I would allow automatic restart of services depending on libc6 to which I answered 'Yes'.

MySQL: Changes to /etc/mysql/my.cnf caused a query. I answered 'Y'. I was asked for a new password for MySQL root user, but I left it as is.

Samba: Of course I have made significant changes to /etc/samba/smb.conf and I was asked what to do. Through a telnet session I made a copy of the old file and allowed the maintainer's version to replace the old config file.

PHP: Changes to /etc/php5/cli/php.ini caused a query. I made a copy of the old file and allowed the maintainer's version to replace the old config file.

grup-pc: Changes to /etc/default/grub caused a query. I made a copy of the old file and allowed the maintainer's version to replace the old config file. Also a query came up about not having the same disk configuration as earlier (may be due to the reduced RAID on this test rig) and asked which is/are the GRUB install devices. I was offered:

• /dev/sda (external USB disk)
• /dev/sdb (internal FLASH disk)
• /dev/md0 (RAID virtual device)

I selected /dev/sda. It failed. I selected /dev/md0. It failed. Hmm. I had no choice and allowed the system to continue.

MySQL: Again I was asked to set a new passowrd for 'root' user. I kept the old one.

Apache2: Changes to /etc/apache2/mods-available/mime/conf caused a query. I made a copy of the old file and allowed the maintainer's version to be installed. Changes to /etc/apache2/apache2.conf caused a query. Again, I made a copy and allowed the new file to be installed. Changes to /etc/apache2/mods-available/php5.conf caused a query. Again, I made a copy and allowed the new file to be installed.

PHP5: Changes to /etc/php5/apache2/php.ini caused a query. I made a copy of the old file and allowed the new file to be installed.

netatalk: Changes to /etc/netatalk/afpd.conf caused a query. I made a copy of the old file and allowed the new file to be installed. Changes to /etc/default/netatalk caused a query. I made a copy of the old file and allowed the new file to be installed.

FINAL RESULT: apt-get dist-upgrade finished indicating errors with the following:

• linux-image-3.2.0-4-686-pae
• linux-image-686-pae
• linux-image-2.6-686

probably due to the failure to install the grub bootloader to the 'correct' boot device. Also, rather frequent warnings were:

• insserve: warning: script 'K01teapop' missing LSB tags and overrides
• insserve: warning: script 'teapop' missing LSB tags and overrides

Just for kicks, I ran it again:

• apt-get dist-upgrade

No package downloads were needed, but 3 packages were shown as not fully installed or removed. It then went to work on the linux-image packages again. Complains about not determining the canonical root device appeared, as before, but this time apt-get finished without indicated errors. Am I done?

Observations:

• Apache was off-line more-or-less throughout the dist-upgrade process, coming back on line toward the end. However, perl scripts are no longer executed, rather are offered for download.
• MySQL is functional via testing with phpmyadmin which also confirms that PHP5 is working.
• PHP is working, but many of my self-coded sites fail to operate, probably due to my use of global query string variables, which are no longer considered good practice. Need to check php.ini.
• Samba required a fixup to /etc/samba/smb.conf to allow home directories to be used by guest

In the long run, it's all for naught. Upon reboot, GRUB fails, and I can't boot, even with the rescue environment. Serious problem.

So, putting things back to normal, I restored the arrays on the real humbaba to use just the original two drives:

• mdadm --grow /dev/md0 --level=1 --raid-devices=2
• mdadm --grow /dev/md1 --level=1 --raid-devices=2

2017-01-14

Spent more time today working with the test server, humtest. It simply wouldn't boot and dropped out into the GRUB recovery shell, which I found quite useless. Some (a lot really) of Google searching on the Net, finally turned up some useful information. Not exactly the same thing, but someone else was unable to boot after a full upgrade from Squeeze to Wheezy. Their solution?

• apt-get install --reinstall grub-pc
• dpkg-reconfigure grub-pc

Since I was not able to boot to a working system, I couldn't do the first command (I thought at the time), but instead, I booted to a recovery shell from a handy Debian 6 "Squeeze" USB thumb drive. Fortunately, when I was asked which device to use for root, I selected "Auto-build RAID array" and it worked! I then selected it as my root (/) device and dropped into the shell. Interestingly, the RAID arrays (root (/), and swap) were not 'md0' and 'md1' as usual, but instead were 'md126' and 'md127'. Huh? Well, anyway, I was mounted and I entered:

• dpkg-reconfigure grub-pc

which went to work. It detected the installed kernels and finished with no errors reported. Hmmm. That was easy. I rebooted.

Well, we got a little further. At least the GRUB menu appeared, but it failed to boot, claiming that the root device named in the conf file could not be found, eventually (20 seconds, maybe) dropping to the (initramfs) shell prompt. Well, that was interesting. I rebooted.

I interrupted the GRUB sequence and edited the menu item, which was set up to use UUIDs to use (md0) instead, on both the 'set root=' line and the linux load line. It worked! I booted! This is good news, but it doesn't solve the real problem. The GRUB setup on the MBR is wrong.

I tried dpkg-reconfigure grub-pc again, now that I was really booted from the /dev/md0 device, but it didn't help. I went back to what I found on the web posing and tried both commands: reinstalling grub-pc and then another dpkg-reconfiguration. Nope, won't work. The grub config file (/boot/grub/grub.cfg) is still getting setup to use UUIDs which, as it turns out, don't really match up with what I see in blkid, so, what to do? I'll probably have to manually reconfigure /boot/grub/grub.cfg and install it manually. Let's try it. I changed the following lines in the first and second GRUB menuentry sections:

• set root=(md0)
• linux /boot/vmlinuz-3.2.0-4-686-pae rootdelay=10 root=/dev/md0 ro quiet

then, re-install grub:

• # grub-install /dev/sda

Now the system automatically boots! This is something to remember as we upgrade to Jessie and perhaps Stretch. I'm sure I'll have to redo this by hand with every kernel upgrade.

Random Cleanup Items

• PHP5 Suhosin warnings: I am getting an every-30-minutes log alert and email indicating that php5 can't find the 'suhosin' extension. Turns out this extension has been removed from Wheezy, but it leaves behind a config file that keeps trying to process. Get rid of it with:
• apt-get --purge php5-suhosin

2018-01-27

Today I upgraded humbaba for real. Using the experience of upgrading a test rig, as described above, I was prepared. As I did with the test rig, I added a third drive to my RAID array (now consisting of /dev/sda, /dev/sdb, /dev/sdd), which gave me triple redundancy if anything went awry. In a nutshell (referencing the above notes), here is what I did:

• once the third RAID drive was synchronized I removed it and the second drive from the RAID array:
  • # cat /proc/mdstat
  • # mdadm --manage /dev/md0 --fail /dev/sdd1
  • # mdadm --manage /dev/md0 --remove /dev/sdd1
  • # mdadm --manage /dev/md1 --fail /dev/sdd5
  • # mdadm --manage /dev/md1 --remove /dev/sdd5
  • # mdadm --manage /dev/md0 --fail /dev/sdb1
  • # mdadm --manage /dev/md0 --remove /dev/sdb1
  • # mdadm --manage /dev/md1 --fail /dev/sdb5
  • # mdadm --manage /dev/md1 --remove /dev/sdb5
• edited /etc/apt/sources.list and update package database:
  • commented out the squeeze lines
  • added wheezy lines
  • # apt-get update
• performed partial upgrade:
  • # apt-get upgrade
• performed distribution upgrade:
  • # apt-get dist-upgrade
  • Note: Interestingly, I had no troubles with the kernel packages this time, unlike the trial run described above
• before reboot, I made sure grub was setup correctly (referencing the diff of /etc/default/grub before and after the upgrade:
  • edited /etc/default/grub:
    • uncommented GRUB_DISABLE_LINUX_UUID=true
  • # apt-get install --reinstall grub-pc
  • # dpkg --reconfigure grub-pc
  • Note: I experienced no difficulties with the last two commands
• rebooted the system:
  • # sync
  • # reboot
  • Note: the system came right up, grub was configured properly. Wahoo!
• Lastly, testing the setup proved that all was working

All of the PHP code that I had written over the years had been fixed in the previous week to not use the deprecated global variables, or to depend on auto-variables. With these fixes already in place, all PHP-based web pages work just file.

SAMBA required a fixup to /etc/samba/smb.conf to enable home directories as I wanted:

• comment out:
  • read only = yes
• add:
  • writeable = yes
  • force user = %S
  • guest ok = yes

The only remaining issue was that a few web-sites/-pages that were Perl scripts still failed. I had overlooked this, and had not fixed this issue yet. Turned out to be a configuration issue. For DropZone I had to add the following line to /etc/samba2/sites-available/jaredblaser.com inside the <Directory /> section:

• AddHandler cgi-script .cgi .pl

With that, the upgrade is complete, and all services are working.

On last item before the ribbon was tied on the upgrade was to remove all obsolete and unused packages with aptitude, keeping teapop and hd-idle which both appear in the obsolete packages section. Unfortunately, this action removed wu-ftpd, leaving me with no installed FTP server. wu-ftpd is no longer offered in the package catalog for "Wheezy". I'll have to find a replacement, probably proftpd or pure-ftpd, though each appear to be more than I need. Actually, I ended up using vsftpd. As installed the package is setup to run as a standalone daemon. In order to have it auto-started by inetd I had to do the following:

• add the following to /etc/inetd.conf:
  • ftp stream tcp nowait root /usr/sbin/tcpd /usr/sbin/vsftpd
• edit /etc/vsftpd.conf:
  • Listen=YES -> Listen=NO
  • and other options as desired: allow read/write from local users
• # services vsftpd stop
• # services openbsd-inetd restart

The final step was to restore the removed drives to the RAID array:

• # mdadm --manage /dev/md0 --add /dev/sdb1
• # mdadm --manage /dev/md1 --add /dev/sdb5
• # mdadm --manage /dev/md0 --add /dev/sdd1
• # mdadm --manage /dev/md1 --add /dev/sdd5