Thursday, May 12, 2011

Recursive Virtualization Xen in KVM

As I wrap up my contributions to UF's, or more importantly Renato Figueiredo's, Virtual Computing course, I am putting together a Debian Xen image, which should work in Simics and a host of other VMMs. Because VMware is the least compatible, I always build my initial image in VMware and then quickly continue working on it in a chroot environment of course after converting it into a raw disk format. Anyway, in my most recent attempt, I was trying to figure out why Xen would not boot inside KVM, when it worked fine in VMware and Qemu.

First a little background, Xen was compiled for 32-bit and running a 32-bit userspace, while my KVM install was 64-bit. As you may or may not be aware, x86-64 does not support privileged rings any more and instead relies on the paging modes (U/S) for privilege. Xen 32-bit relies on the rings for performance purposes. So you would think that KVM would support the rings as well. As it turns out, KVM 64-bit does not appear to virtualize the rings and thus Xen 32-bit does not work.

While this is still somewhat of a hunch, I tested a Xen 64-bit, which does not use the privilege rings. In KVM 64-bit, it worked flawlessly. Leaving me two more things to check: determine if Xen 32-bit boots in KVM 32-bit and if a simple ring test boot code works in 32 or 64 bit KVM. The former would be useful for other's, while the latter would be a conclusive simple test to verify the functionality.

Anyway, the point is, you can run Xen in KVM, if they are compiled for the same platform. I'd be curious to see if you can run a 64-bit VM in 32-bit KVM and if so, if the Xen 64-bit would work.

Wednesday, May 11, 2011

Dangers of IPTables NAT with DHCP

For Archer, I have setup a small private LAN for all the core resources to share with a common out into the main network, which connects to the Internet. I use Linux's IPTables via masquerade for NAT or routing the packets from the private network into the Internet and DHCP to automatically assign IP addresses to the resources. Recently, I extended the router to support other hosts to use it to bridge into the local network, which may or may not be responsiblef or a whole host of other issues. The two primary issues were routing of bridged packets through IPTables and DHCP messages traversing the bridge, both request and replies from both sides.

The routing of bridged packets was particularly difficult to figure out. I had recently setup a web server, which had a public IP address and was bridged into the LAN. It was pingable and could ping other IP addresses within the LAN, but it couldn't ping the gateway. Furthermore, it was accessible from the Internet but could not initiate communication with the Internet. It was very strange behavior. I whipped out tcpdump and quickly noticed that packets it was sending out via the bridge were being picked up by IPtables masquerade and having their source address change. So that when the packet came back, it had no where to go. The resolution to this problem was pretty easy: add "-s ip_range/mask" to the masquerade iptables line. So this:
iptables -t nat -A POSTROUTING -o bridged -j MASQUERADE became
iptables -t nat -A POSTROUTING -s -o bridged -j MASQUERADE
where bridged is the network bridge into the LAN, i.e., a Linux bridge connecting eth0 to other devices wanting access to the LAN and is the private address space. Now masquerade will only forward packets with the source address

Perhaps, doing this opened up the other problem, which was that resources on the LAN were somehow getting DHCP requests to the private network. This never makes people happy and worst of all, those individuals were losing their Internet connectivity. This could be related to forwarding of broadcast Ethernet packets as well. The resolution for this was a bit easier, simply drop all DHCP packets that come across the NAT:
iptables -A FORWARD -p udp --dport 67 -i bridged -j DROP
iptables -A FORWARD -p udp --dport 67 -o nat -j DROP
In this case, packets coming from the bridged device (-i bridged) are those on the LAN looking for IPs. The packets coming from the nat device (-o nat) are those on the private network looking for IPs. These rules effectively prevent the retransmission of dhcp packets. The reality is, I probably should prevent the transmission of multicast / broadcast messages from crossing the NAT, since there exists no gateway for packets to get into the private network.

Anyway, this was a rather unfun discovery as I found NOWHERE online that discusses these two issues when configuring NATs. I hope this turns out to be useful for someone else out there.

Tuesday, May 10, 2011

Ubuntu Xen -> Debian Xen

After using the Ubuntu Xen setup for a class, I must admit I am very disappointed with it. Ubuntu is just really not that great as an OS for learning. While upstart may have some long term potential, nothing beats a sys-V or BSD mechanism for startup. The simplicity allows for use of chroot, which is critical when the resulting image or kernel will be used inside a virtual machine. This was the case for the Ubuntu-Xen, which was used inside Simics, VMware, or Qemu depending on the students project. Some other hiccups included:
  • Non-booting kernels when compiling additionals functionality into the kernel
  • Inability to use grub2 and having to install grub1
  • Default kernel didn't have devtmpfs included and Ubuntu relies on it
I still think that apt is superior to yum. One thing to note is that even though Debian has Xen built-in, it is useless since this is for academics, where students will certainly be expected to modify the Xen and potentially the dom0 kernels.

Tuesday, May 3, 2011

The conclusion

... is only a new beginning... or rather I am not quite there yet.

Today was my dissertation defense. After approximately 23 years in academics, I am finished with the "student" title in my life. I have completed pre-school, elementary school, middle school, high school, undergraduate school, graduate school, and, finally now, graduate school. I am a PhD as of some 10 hours ago! I put the title "student" in quotes, because as we all know, we are always students in that we continue to grow and evolve as we become older and wiser. While the years have produced both good and bad memories, I am most blessed by the individuals that I have met along the way. I could go on and on, but for now, I just want to express the elation that after 6 long years as a graduate student, I can say that I'm done. While the path was certainly not friendly, I am just happy that I conquered the PhD and was not conquered by it!

Friday, January 14, 2011

Ubuntu 10.04 + Xen + Simics

The goal of my previous posts were to create a Xen 4.0.1 + Ubuntu 10.04 and run it on top of Simics 3. Well everything works great until I got to Simics and then I would get a dreaded triple fault due to MSR 0x8000020 or MSR_EFER, which appears to be a MSR for 64-bit CPUs. It does not exist on Pentium 4 machines, Simics x86 cpu, and it is unclear why that instruction was being executed. After figuring out the stack trace in Simics, I disassembled Linux and Xen and could not find a matching set of instructions. That meant the only other possibility was that grub was at fault. It seems that during the booting of a multiboot the instruction was being called. After installing grub1 onto the machine, the machine boots in Simics.

Tuesday, January 11, 2011

Creating a Xen DomU in Ubuntu

As a follow up to my previous post, where I installed Xen onto Ubuntu 10.04, in this post, I will use debootstrap to create a DomU Ubuntu 10.04 sparse image.

Begin by installing debootstrap: apt-get install debootstrap

Create a sparse image (1 GB) -- sparse is attractive as it doesn't use real disk space until the image has content: dd if=/dev/null of=ubuntu.img bs=1M seek=1024

Create an ext3 file system in the image: mkfs.ext3 ubuntu.img

Prepare for debootstrap:
mkdir ubuntu
mount -oloop ubuntu.img ubuntu

Run debootstrap: debootstrap lucid ubuntu
Add the Xen console device to init: sed 's/tty1/hvc0/' ubuntu/etc/init/tty1.conf > ubuntu/etc/init/hvc0.conf

Add network initialization to your image by adding the following or something similar to ubuntu/etc/network/interfaces:
auto eth0
iface eth0 inet dhcp

Set a root password:
chroot ubuntu
(set password)

Clean up:
umount ubuntu
rmdir ubuntu

Create a xen configuration file (ubuntu.cfg):

kernel = "/boot/vmlinuz-"
ramdisk = "/boot/initrd.img-"
memory = 64
name = "ubuntu"
root = "/dev/xvda ro"
disk = [ "file:/root/ubuntu.img,xvda,w" ]
vif = ['bridge=eth0']

The above assumes your kernel is Xen kernel, the current working directory is "/root", and your default network device is "eth0". Adjust as necessary. Also the memory is configurable, the default listed above is 64M.

Check that everything works -- assuming xen is on (verify xm list): xm create -c ubuntu.cfg

It should boot into a domain that you can log into as root.