Fun abusing grub...

[Christopher Caldwell]

Greetings,

I didn't try to break any records, but I did try to make a good
reference machine for each of these operating systems and
get real experience about handling real devices.

Enclosed, read the touching tale of my attempts at installing
9 different operating systems under Grub.

--Chris

Too Many Operating Systems

Christopher Caldwell, 6/27/05

A few months ago, I began a fool's errand.  Today, I can say that I have
completed a dubious achievement:  I can multi-boot 9 different operating
systems on my laptop.

I did this, not as an exercise in self abuse, but rather to learn how to install each of the operating systems at a deeper level than is required with a simple install on a naked system. If all I had wanted was 9 different operating systems running on one machine, the easiest, most efficient way would be to install 8 of them as virtual machines on a VMWare platform running on the 9th. Indeed, VMWare would have had many advantages:

• I could have run more than one system at once (even communicating with each other over private IP addresses).
• The hardware emulated by VMWare is sufficiently standard that it has been tested in all of the operating systems that I tried.
• The size of one installation doesn't effect the installation of any other as the "Virtual disks" are just files in a random-access file system.

Instead:

• I had to deal with the fact that several of the operating systems didn't have a driver for my ethernet card or my Synoptics touch pad.
• Some operating systems are not able to calculate the final amount of disk space required for their installation. Consequently, they may start an installation that is doomed to failure due to insufficient disk space, which can waste a lot of time.
• On more than one occasion, I had to save multiple operating systems away, repartition the disk, re-make the file systems and restore them to make room for additional software on an operating system earlier on the disk.

What I installed

Operating system	Revision	Where to get it
FreeBSD	5.4-RC3-i386	http://www.freebsd.org/where.html
OpenDarwin	7.2.1	http://www.opendarwin.org/en/downloads
Solaris*	10	http://www.sun.com/software/solaris/get.jsp
Windows	XP Professional	(available at a Walmart near you)
Fedora Core	4	http://fedora.redhat.com/download
SuSE	9	http://download.novell.com/index.jsp?search=Search&platforms=2729&x=25&y=8
Ubuntu	5.04	http://www.ubuntulinux.org/download
Debian	30r5-i386	http://www.debian.org/distrib
Slackware	10.1	http://www.slackware.org/getslack
Gentoo	2005.0	http://www.Gentoo.org/main/en/where.xml

* Note that I did not end up installing Solaris in the final machine configuration due to conflicts with partitioning and lack of ethernet support.

In all cases, I performed the installations from CDs. Most of the Linuxes provide procedures where you create a small amount of read-only media (i.e. floppies, a single CDROM or a single DVD) and then "bootstrap" up enough to get the rest of the distribution over the Internet. I chose not to do that because I kept making mistakes and it was faster to make mistakes, correct them and start over using CDs then to try to re-download all of the information from the Internet. If you know what you are doing, by all means, install directly from the Internet. However, the second time you find yourself doing the same installation, remember... "I told you so."

Defining "Install"

"Install" covers a lot of territory.  I could have considered an operating
system as installed if I could boot the operating system's kernel off
my laptop disk.  Alternately, I could have defined it as installed if I had 100
users doing software development using graphical tools and interacting
with web technology over VPNs.

I chose to consider a system completely installed if I could:

• Login as both root and a user I created
• Use some sort of windowing technology to perform the remaining testing
• Be able to print graphical data to a remote printer
• Be able to browse the Internet
• Use some sort of text processor
• Compile and run my favorite utility, hello.c

It would be easy to add to this list - but recognize that most tasks
must be implemented 9 times, and not always in the same way.  For instance,
making Linux act as an e-mail server is an entirely different problem
than making Windows do the same task.

Hardware

So, here are the specifications for my poor beast:

Product:	Dell Latitude CPi
Processor:	Mobile Pentium II
Frame buffer:	Neomagic Corporation NM2360
Mouse device:	Synoptics touchpad
Clock rate:	400MHz
RAM:	256MB
Disk space:	40GB
CardBus:	Xircom CardBus Ethernet 10/100 + Modem 56 SMC 2632W V2 Wireless adapter

As I purchased it, down in the lower right hand corner of the keyboard, there
were two stickers: One says "intel inside, pentium II" (which is true).
The other says "Designed for (Windows Logo) Microsoft Windows NT© and
Windows 98©" (which is probably also true.  What it was designed
for and what it actually runs are two different animals).

Software

All of the operating systems I installed expect to have their utilities
in their own partition.  Without VMWare, this required laying out the disk
table prior to installing the first byte.

Well, not actually.

Since I made errors sizing the software, I ended up installing, re-installing, saving, restoring etc. multiple times. It is possible to recover from partitioning mistakes without backing up and restoring everything though absolutely no vendor would recommend it.

In determining the layout of the disk, I ran into a nomenclature problem with vendor documentation. The term "partition" and "slice" are used, basically, interchangeably, and sometimes a partition within a partition is called a "sub-partition" and sometimes it is just referred to as a "partition".

All of the operating systems try to prevent the administrator from having to care about what is a partition or a subpartition, so their documentation simply refers to the chunk of disk where a file system goes as a "partition" whether it is a partition, a sub-partition or a slice.

The Linuxes all refer to partitions on the laptop's primary disk as "hda" followed by a number. It numbers the partitions as it finds them and it looks for them in the following order:

• traditional Windows primary partitions
• traditional Windows extended partitions
• other labels (including BSD)

In my case, even though I had a sub-partition within partition 1, that
partition doesn't get a low "hda" number because it is within a BSD label.
Therefore the kernel doesn't "number" it until it has accounted for
all of the partitions in the Microsoft Windows extended partition.
"hda5" is actually a sub-partition of partition 4 (extended Windows partition).
Only after all of the partitions were found on the Windows extended partition
does Linux search the BSD label to find FreeBSD's UFS partition on /dev/hda13
(and its swap on /dev/hda14).  This accounts for the non consecutive "hda"
numbering in the table below.

On the other hand, FreeBSD attempts to denote the partitions and the subpartitions. For instance, the first partition in the BSD system is "ad0s1a". FreeBSD can see the first sub-partition on the extended partition (aka hda5) as "ad0s5".

OpenDarwin understands Windows slicing about as well as Windows does: It needs to boot off a top level partition but can access extended partitions.

Solaris is very like FreeBSD in that it has its own label-within-a-label. Unfortunately, the Solaris and FreeBSD labels are not identical, so it is not possible to have both operating systems reside within a single label as it is possible to have all of the Linuxes within the Microsoft Windows extended partitions.

The label (and sub-labels)

Partition		Device				Operating System	Size	Installer
		BSD	Darwin	MS	Linux
01		ad0s1	disk0s1		hda1	FreeBSD 5.4-RC3-i386 label	4000 MB	sysinstall
BSD label	1	ad0s1a			hda13	FreeBSD 5.4-RC3-i386 /	1300 MB
	2	ad0s1b			hda14	FreeBSD 5.4-RC3-i386 swap	2700 MB
02		ad0s2			hda2	Solaris 10**		Suninstall, pkgadd
02		ad0s2	disk0s2		hda2	OpenDarwin 7.2.1	916 MB	rpm
03		ad0s3	disk0s3	C:	hda3	Windows XP Professional, 2002 w/sp 1	1700 MB	setup
04		ad0s4	disk0s4		hda4	Extended partition	17391 MB
MS extended label	1	ad0s5	disk0s5		hda5	Fedora Core 4	6800 MB	Anaconda, yum
	2	ad0s6	disk0s6		hda6	SuSE 9	2000 MB	YaST2
	3	ad0s7	disk0s7		hda7	Ubuntu 5.04	1200 MB	apt-get
	4	ad0s8	disk0s8		hda8	Debian 30r5-i386	3000 MB	apt-get, dselect, dpkg
	5	ad0s9	disk0s9		hda9	Slackware 10.1	1400 MB	setup, pkgtool, installpkg
	6	ad0s10	disk0s10		hda10	Gentoo 2005.0	1600 MB	emerge
	7	ad0s11	disk0s11		hda11	/u0	879 MB
	8	ad0s12	disk0s12		hda12	Swap	512 MB

Generally, the operating systems installations did not interfere with
each other, with the exception of the disk label (including the master
boot record).  As near as I can tell, the only thing that prevented
me from adding as many versions of Linux to the extended partition (4),
was simply disk space.  The other operating systems wanted to at least
have one top level partition.  OpenDarwin and FreeBSD both sub-partitioned
their partition as well (so that they could have root and swap file systems).

**I also had installed Solaris version 10 on the second partition during this experiment. However, I discovered that the Solaris, OpenDarwin, FreeBSD and Windows labeling schemes were only compatible at the top level. That is, Solaris, OpenDarwin, FreeBSD and Windows were all going to end up occupying a top level partition. Since you only get four top level partitions and Solaris had no support for any of the CardBus ethernet cards I had, it became the odd-man-out. If Windows had fully supported its own extended partitioning system, then it could have ended up being installed on one of the logical partitions in the extend partition (thereby freeing up a top level partition for Solaris). However, the Windows booter must be in a top level partition.

Trying to be wise about disk space

When I could share, I did.  Unfortunately, this only involved two partitions:

• The Linux swap file system could be used by all of the Linuxes but nothing else. Actually, if I will willing to add scripts to the startup procedure for each of the operating systems, I could probably have used that disk space for paging for all of them. In the Linux cases, early on in the boot process, a script would have had to do something like "mkswap /dev/hda12" prior to the "swapon -a" command.
• I created a /u0 which was mounted by most of the operating systems for my home directory. The only file system type that each operating system could access was FAT (e.g. Windows). I found the naming constraints and the lack of file modes too constricting so I chose to use a ext2 file system for everything but OpenDarwin and Windows. Although I believe there are ext2 & ext3 ports for OpenDarwin, until I am able to get an ethernet driver running for it, I will leave it as is. Windows XP does not natively support ext2 or ext3 file systems, but I installed a software module called "ext2ifs" from http://uranus.it.swin.edu.au/~jn/linux/ext2ifs.htm and was able to get read-only access to my home directory. Although it seems to work well, I see no evidence that there is a lot of run-time on this software, it is read-only and its version is only 0.3. Microsoft may back-port WinFS for Windows-XP, but it wasn't available for this writing either. Paragon also offers a file system product (which offers write access) for ext2 and ext3 at http://www.softcities.com/Paragon-Ext2FS-Anywhere/transfer/2685.htm but I have no experience with it.

Knowing how big something is helps a lot

Obviously, each operating system partition has to be big enough to hold the
operating system (and a little space for making changes in configurations, as
well as space for any packages you are going to add).  Unfortunately, even if
you know the number of blocks (including directories), you still don't have
quite enough information to determine partition sizes.  Filesystems
have overhead and the installation process sometimes requires more disk space
than the actual distribution (Gentoo, for instance, ends up compiling a lot).
Ultimately, I chose to use 125% of the distribution size for my partition size.
I had enough disk space for compilation for Gentoo because of my trick below
for installing Linuxes.

Installing the Linuxes

I discovered that the easiest way to install a new Linux variant was:

• Install it on the last logical partition on the disk
• When installing, do not have it overwrite the master boot record
• Note how big it actually is (vs. what the vendor says)
• dump it (back it up) to a great big compressed archive elsewhere
• Repartition the last partition into two partitions, one big enough to hold the distribution (with some space left over), and the other to contain all of the remaining disk space.
• mkfs.ext3 the now smaller partition
• mount and restore the dump archive into it
• Repeat with next Linux

This may seem unnecessarily slow because you pass over the data 3 times
(installation, backup and restore), but in practice the backup and restore
phases took far less time then the installation and I was left with a handy
backup of that partition.  Additionally, if I got the sizing wrong, I only
had to re-partition, re-mkfs and re-restore.

Booting

Since each operating system is meant to operate without the aid of any other,
they each have their own booting scheme.  By now, the vendors creating them
realize that they may be working on multi-boot systems, so now each provides
a method to select which system to boot from.

Generally, each operating system has a boot block which, if you execute it, will understand enough of that particular operating system's filesystem structure to at least track down the next phase of the booting process on the disk. Most of the operating systems will give you the choice of installing that boot block in the master boot record or as the boot record for the partition where that particular operating system is located. I chose to do the latter.

I chose to use the GRand Unified Booter (GRUB) booter for the primary boot and had it get its grub.conf configuration file from my Fedora Core 4 /boot directory. It could have been any one of the Linuxes and perhaps any of the others, but that configuration file allows me to specify where the other boot blocks are located.

Grub not only understands grabbing the boot block from a specified partition, but it also understands several file system types including ext3 and reiserfs (used by default by SuSE), so generally I am able to specify which kernel and which RAM disk to boot from directly in that file for any of the Linuxes.

In the case of the other operating system's, I simply load the first block from the partition associated with that operating system and let it take control.

Since Grub has been around awhile, I'll bet it is able to read FreeBSD style file systems and directly load in the FreeBSD kernel, but I simply have it read the first block in the FreeBSD partition and let it handle the next phase.

How far I got

Task	FreeBSD	OpenDarwin	Windows-XP	Fedora	SuSE	Ubuntu	Debian	Slackware	Gentoo
Administrative user:	root	root	administrator	root	root	root	root	root	root
Create & login as user:	yes	yes	yes	yes	yes	yes	yes	yes	yes
Run graphical interface:	Gnome	X11 & xdm	Windows	Gnome & KDE	Gnome	Gnome	Gnome	Gnome	Gnome
Ethernet:	yes	no***	yes	yes	yes	yes	yes	yes	yes
Wireless:	no****	no	yes	yes	yes	no	yes	no	no
Internet:	yes	no	yes	yes	yes	yes	yes	yes	yes
Browse Internet:	Mozilla	no	Internet Explorer	Mozilla	Mozilla	Mozilla	Mozilla	Mozilla	Mozilla
Print graphics:	yes	no	yes	yes	yes	yes	yes	yes	yes
Compile & run "hello.c":	gcc	gcc	cygwin*****	gcc	gcc	gcc	gcc	gcc	gcc
Edit a document:	kword	vi	OpenOffice******	OpenOffice	OpenOffice	OpenOffice	OpenOffice	AbiWord/a>	vi*******

***Clearly, my lack of ability to get packets to the machine while it was running OpenDarwin caused me to fail many of my goals while running that operating system.

****Wireless was not a requirement, but I like to work on my porch and tried it out when it was easy. Some of the operating systems have nice wireless configuration support, and others leave you on your own with kernel module loading, etc. The fact that I don't list an operating system with wireless doesn't mean it doesn't have it, it just means I didn't try it or it was hard or the mosquitos were getting to me.

*****Yes, you can get a full compiler suite and almost every other standard Unix/Linux utility from http://sources.redhat.com/cygwin.

******Yes, you can run OpenOffice on Windows products. Get it from http://www.openoffice.org.

*******Also, lack of disk space prevented me from installing all kinds of software that would have allowed me to do more graphical text processing than "vi" with Gentoo and Slackware.

What would have made it easier

• If everybody used the same kind of label and you could have lots of partitions in that label, then we would wouldn't be talking about partitions and sub-partitions, etc.
• FreeBSD's install process determines the order in which things get sucked off the CDs by dependencies. Unfortunately, this means that the install process may ask you to switch back and forth between the two disks multiple times. FreeBSD should have had enough information on the first disk to determine exactly what software would be installed off of which disk before installing anything. Then it should go through each disk required, only once.
• FreeBSD also looked at the geometry of the disk, decided it wasn't sane, and "fixed it." None of the other operating systems had the hutzpa to do that. Indeed, the reason that I ended up installing it first was so that none of the other operating systems would have their idea of the geometry changed out from underneath them. FreeBSD should not adjust the geometry unless it has a "can't work with it this way" rational, and it should certainly allow the user to abort the installation if he thinks the adjustment is not a good plan.