Installing Xen 4.0.1 and testing on Ubuntu 10.04

Several of my colleagues have recently had a great time trying to install Xen on Ubuntu. I've seen various locations where the task has been partially done, but the authors always leave something missing. I guess in their cases, the goal is to get their job done and not make a very generic tutorial. The goal of this tutorial is to be generic, so that the process can be repeated easily without going having to return to google to figure out what the contents of this tutorial are expressing.

So here's a rough outline:

Install or having a working Ubuntu 10.04. If you are doing a fresh install, I recommend using ext3 on your hard disks otherwise you'll have to reconfigure the Xen-Linux kernel.

Install the necessary utilities to download, compile, and install Xen:apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial build-essential make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg62-dev iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml libx11-dev

If you are using 32-bit install libc6-xen (This will get rid of the annoying TLS segmentation message): apt-get install libc6-xen

If you are using 64-bit install 32-bit utils: apt-get install gcc-multilib

Download Xen, either get the Xen 4.0.1 source or get testing. If you want the 4.0.1 source, its faster to download the tar.gz than the entire mecurial repository!

4.0.1 source: wget http://bits.xensource.com/oss-xen/release/4.0.1/xen-4.0.1.tar.gz

Testing source: hg clone http://xenbits.xen.org/xen-4.0-testing.hg

Build Xen, this includes building all tools but not the Xen-Linux Kernel:
cd into the Xen dir
make xen
make tools
make stubdom
make install-xen
make install-tools PYTHON_PREFIX_ARG=
make install-stubdom

Build the Xen-Linux Kernel (if you need to configure the kernel see the bottom of this post):

make prep-kernels
Optionally configure the kernel especially if you are using ext4 (see below)
make kernels
make install-kernels

Figure out kernel version by looking at the .config file in the kernel build path (typically starts with: build-linux-2.6-pvops_), so typing "head build-linux-2.6-pvops_x86_32/.config" in 32-bit case will print out the "Linux kernel version" near the top (the current version is 2.6.32.26)

update-initramfs -c -k [kernel version]

Making Xend auto-start:

update-rc.d xend defaults 20 21

Update Grub

Place this in /etc/grub/40_custom:

menuentry "Xen-Testing - Linux 2.6.32.26" {
        insmod ext2
        set root="(hd0,1)"
        multiboot /boot/xen.gz dom0_mem=256M
        module /boot/vmlinuz-2.6.32.26 ro nomodeset root=UUID=24aece87-9b12-4966-8c04-4bd59d5808ae
        module /boot/initrd.img-2.6.32.26
}

Note: you can obtain your disks partition (hd0,1) by running blkid on the device, the default device will usually be /dev/sda1 (e.g., blkid /dev/sda1). Also feel free to adjust dom0_mem

Finally update grub:

update-grub

Reboot into Xen. While testing you should leave the default boot into your regular kernel. Hold shift during reboot to access a grub menu and select Xen. Determine which grub entry Xen, hint, they are incremented in order from 0 to N. In my case, it was 6, so edit the /etc/default/grub and replace 0 with 6 on the line "GRUB_DEFAULT". The next step is to create a domU!

The Difference Between Polling and Events

(Modified from a presentation I gave a few years back...)

The scenario is my wife is preparing food for us to eat for dinner. I am waiting wondering when food will be ready. With polling, I continuously bother her, asking, "when will food be ready?" With events, I can ask her to tell (notify) me, when food is ready, and I'll either take a nap (sleep) or play a video game (process something else).

The Headaches of Blocking Sockets

In Brunet, ACIS Lab's P2P software, we make heavy use of sockets, no doubt. Because UDP is more capable of traversing NATs than TCP, much of our focus has been on optimizing the UDP stack in the code. The concerns and issues we have had in using UDP include:

• Safety of sending and receiving on a UDP socket at the same time
• Do UDP sockets block
• Can there be concurrent senders / receivers on a UDP socket at the same time

On to the findings!

• Safety of sending and receiving on a UDP socket at the same time

The fear with sending and receiving at the same time is primarily based on lack of knowledge, the limited information provided on the Internet, and that you're dealing with what is presented as a single state. But let's think about it for a second, a socket is really like a pipe, well, rather two pipes, one for sending and one for receiving or better yet two queues, one queue waiting for the userspace to read the packet and another queue to send a packet across the network. Sure we multiplex the actual Ethernet device, well actually, we don't that's done by the OS. So why do we really need to be concerned about ordering when writing to a UDP socket? Maybe with a TCP socket, since that actually involves state, but UDP is stateless.

As it turns out, one need not worry about reading and writing from a socket at the same time. The conclusion was reached based upon two pieces of evidence, empirical evidence arrived through experimentation and and reading the grand TCP/IP Illustrated. With just this alone, we were able to boost the speed of our system 10x, reducing code complexity, and improving idle time execution.

• Do UDP sockets block

Once upon a time, I was a young naive boy working on a P2P VPN called IPOP. IPOP was working so very well, but one day out of the blue IPOP would keep freezing up. What was to blame? Was it latent bugs that were showing themselves at an inopportune time or did we make a mistake somewhere in our logic? So I searched high and far, I even did inspection of packets to find which one was causing the freezing of the code. It turned out to be the sending of a packet across a transport, strange, I thought to myself. We have been using this code for years and I have never seen this issue!

After some reading around, I discovered that I didn't know everything about UDP sockets, the worst of it was that UDP sockets actually have a send buffer!!! When the send buffer fills, it blocks! This is absolute nonsense, but what can you do? The only solution was to make UDP non-blocking somehow.

The two choices were to do it in the main loop of the system or alternatively to put it into another thread. Well, we were already paying a lock penalty to prevent multiple writers at the same time, so maybe its safest to just put everything into another thread. We made that choice, because that way, we could provide a slightly larger user space buffer. Guess what... the system doesn't stall any more! Quick caution, though, sometimes that queue can become very large, so its important to limit its size. We chose the modest value of only allowing 256 messages be stored into the queue and it seems to be working well for a node that receives messages from over 50 remote nodes somewhat aggressively.

• Can there be concurrent senders / receivers on a UDP socket at the same time

This is actually a very good question, which I can only base upon reading and not much more. According to Unix Network Programming, another of Stevens books, you can at least have multiple readers, but we haven't tested it as of now, the receiver is in a single, dedicated, thread.
Sending is kind of confusing and unaddressed in what I've read so far, in the next paragraph, I describe one issue, Connect, which makes it seem somewhat conclusive why it wouldn't work. Though it would be interesting to see what if anything happens if we allowed multiples of each, but unfortunately, we haven't spent much time focusing on the performance scalability of our P2P system.

At this point, I've addressed most of our concerns, I am still very interested in understanding better practices with sockets and how we could potentially improve our system. For example, one tidbit from Unix Network Programming was that a UDP socket actually calls Connect when sending a packet, apparently, this overhead is 1/3 of the actual time spent using the socket during a send. So if a user were to have a socket for each remote peer, performance could potentially be significantly better.

Returning to DTLS

Gosh was I ever wrong about DTLS, OpenSSL, and OpenSSL.NET. First some background, last spring, the project I had been working on, IPOP (IP over P2P) needed a security framework so that we could actually call it a VPN. So I did some research to determine, which security framework made the most sense. IPOP relies on unreliable paths, such as UDP and the P2P overlay, for communication. Thus a well accepted solution like SSL was out of the question, this led me to two possibilities IPsec and DTLS. I began reading as much as possible of the two frameworks, my conclusion was that DTLS was written for client-server applications and put more work on the server and it may require special logic to prevent duplicate connections if two peers attempt to create secure connections simulataneously. IPsec handles these cases, but is a beast and lacks, clean, portability across systems, thus supporting our VPN on various platforms would require special hooks or libraries for each system. Not very desirable, since we're a very small group. I settled creating a IPsec like implementation using only managed (C#) libraries that worked in Linux and Windows. Sometime late last year, I came to the realization that I am a PhD student and that this was probably not the best route, since my group lacked much knowledge regarding security systems. So I implemented a light-weight DTLS filter written in C#, but I never really spent much time figuring out how it fits in our P2P application.

Flash forward to a couple weeks ago. I started wondering, what if any important questions could be answered by what I have done and what sort of evaluations I could perform to gain some insights. So I went back to work and finished my DTLS integration into the P2P system. It was not pleasant. This time through, I made sure that I understood every little detail that was occurring in the system and applied that knowledge to make the system as simple as possible. Errors in my previous notes were the following:

1. DTLS does not handle cookies if you do not provide a generate and verify method
2. DTLS can have an assertion fail if certain packets are lost during the handshake
3. DTLS automatically handles retransmitting handshake messages

Determining the cookie was somewhat painful, since I had implemented a filter and the data structures were crossing a managed / unmanaged layer. In examples, DTLS uses the socket IP/Port set and a secret number to generate a cookie, though I didn't have the IP/Port. Luckily, it turns out that the HashCode of the Ssl.Handle (int ManagedSsl) remained constant across callbacks, unlike Ssl.GetHashCode() and Marshal.ReadInt32(Ssl.IntPtr). The former changed, while the latter was the same for every call, go figure. Thus to generate and verify, I took a 20-byte random number appended the HashCode of the Ssl.Handle and performed a sha1 hash.
One thing I was not very happy with the cookie generation and verification, is that, it does create state, which I thought was the soul reason for having it in the first place. The memory impact on my system (without taking OpenSSL into context) remains the same whether or not cookie generation is enabled. It would be nice to verify that cookies are actually being used intelligently, but no work has verified this.

For the life of me, I couldn't figure out why the handshake messages were failing. The code is very difficult to follow, but in the end, it appears that for some reason on the server side, when one of the timers expire, a few bytes appear ready for sending. The problem is, these bytes should never have been placed there. When performing a read on it, it causes an assertion that aborts the code. Let's hope an attacker doesn't figure out that pattern. Is anyone actually using OpenSSL DTLS in production level code? How did this not get detected earlier?

On the path, I encountered a few other headaches, which I will simply list as the details aren't very exciting:

1. OpenSSL does not support unencrypted PKCS#8 Keys via the API if they are in DER mode, but PEM mode works fine.
2. If an SSL state switches from client to server, it ignores messages, the only solution is to create a new one from scratch. This is after executing clear, reset, accept, and resetting the bios.
   
3. OpenSSL only supports a single PrivateKey / signed certificate per context (ssl_ctx), I guess that's fine for preshared keys, but it does limit the ability for a single user to obtain multiple certificates for different domains.

The final component deals with determining a client vs a server. All messages are prefixed with a local and remote identifier (think port). A client knows his local port but the remote port hasn't been sent by a server. If a server receives a message without a remote port (or rather his local port being set), he knows its a message from a client. Thus he creates a new server side component to handle the request. When the client receives a packet back, it will see that the server has set his port and all future messages will contain the port pair. Thus packets from the client to the server are of the form [client][server][data] and from the server to the client are of the form [server][client][data]. This allows multiplexing a single data channel to support multiple secure channels.

To summarize, I don't think DTLS is very well loved in the SSL community, there is literally like no documentation available for it. Most of what I learned, I did from the original paper, the RFCs, and looking at the code. I have attempted to communicate with the OpenSSL.NET guys to no avail. At this point, I have extended the system with a few hundred lines of code and will probably do a little more, since their current code needs to be compiled explicitly for Unix or Windows.

Still to be worked out is the cost of having (potentially) multiple secure connections, methods of reducing the duplication of secure connections due to lack of a P2P handshake, and potentially investigating potential security issues in the approach, especially when performing the handshake over the P2P overlay.

ICMP Source Quench

As with most PhD students, I have learned to run before crawling, meaning that I have worked on many systems in IP networking and P2P without having an expert knowledge in either field. To address this, I have begun a reading crusade beginning with a massive 1500 page book describing TCP: The TCP/IP Guide.

While reading it, I came across something that struck me... ICMP, or Internet control management protocol, has a feature known as source quench. Basically, it tells the receiver of the packet to slow down, though it requires the first 8 bytes of a sent packet. In UDP this means the entire header (ports, length, and checksum), while in TCP this is a smaller portion of the header (ports and sequence number). What struck me as odd was that no where in the book did it describe how well deployed this feature was and potential attacks made by using it.

So I turned to Google... and found this lengthy document. The basic summary was that most systems seem to have disabled TCP support for ICMP source quench, but what about UDP? I also found that it appears at least Windows CE does support ICMP source quench.

What interests me are, what OS' actually support this for UDP and TCP and do they verify UDP and TCP header mappings before reacting. Can users in a P2P system like Gnutella or BitTorrent who are downloading a file attack the sender of the data to slow down the rate at which he shares the files to others and thus improve their own bandwidth? I couldn't find any papers on Google or ACM, maybe this will be a neat little side trip for me.