Copy-on-Write using Ubuntu 9.10

Recently, I've been working on getting a package that enables copy-on-write for Ubuntu 9.10, the use-case would be Grid Appliance, a project that I have been working on for the past 3 and half years. Previously, I had used Debian, created a special kernel, and messed around with the init scripts. During which, I experimented with both UnionFS and AUFS, both of which required kernel patches. The former, UnionFS, was replaced with AUFS due to stability concerns.

To begin my research, I first investigated the possibility of a union filesystem in the modern kernel. To my surprise, it seems that kernel maintainers are uninterested in this feature unless it can be implemented in a truly transparent method. Thus AUFS, most likely, will not be included in the kernel and Ubuntu has or plans to phase it out in the near future. UnionFS shares an even worse fate, as most developers have moved towards AUFS for stability reasons. Luckily, I found something called UnionFS-Fuse, which implements the UnionFS system on top of Fuse, or file system in user-space as opposed to kernel file systems. I was first hesitant in this method, it occurred to me that Fuse ensures that I won't have to deal with kernel issues meaning that regardless of the direction taken by the kernel developers, I'll have a working union filesystem.

The next step was figuring out where to insert the cow init script. It turns out that Ubuntu 9.10 uses Upstart, a relational, event-based, threaded init system. This replaces the old init system, based off of System V, which provides an explicit, in-order init system. Upstart poses a challenge for cow filesystems, as the first step in Upstart is to mount all the file systems by issuing the "startup" event. To get around this challenge, I've appended a "pre-start script" command that executes the cow init script prior to starting the mounting of the other file systems. This appears to work as I've set the virtual machine disk to rdonly and the system boot perfectly*.

Yeah, it doesn't boot perfectly, because in Ubuntu 9.10 uses grub2 and checks whether or not the last startup sequence was ended successfully. This requires a non-trivial change that I am still figuring out. I can easily edit out the contents, but that won't work when an update occurs. I guess its time to start reading grub manipulation utilities to prevent that.

Working In And Around Licenses

Recently our group has been confronted with the issue of how to interoperate with non-GPL compatible code. Previously, a professor, whom I work with, has been considering this issue along with how to allow GPL incompatible or more specifically closed source application to interoperate with our P2P library, Brunet. At the root of the problem is the "viral" nature of the GPL. Namely that all derived works must also be GPLed, see this. Of particular interest in the GPLv2 is the definition of derived works and what consists of distribution.

During a recent discussion, one thought was that we could use a binary or conversion interface to bypass this issue and another was that we could use sockets, both of which were shot down by this message. Which made me wonder, how do other non-free applications link to GPL applications and what it falls under is distribution. Primarily, you cannot distribute non-GPL code that interacts with GPL code together, the intent is a derived work. Using a socket or a wrapper library effectively yields the same purpose, a circumvention of the GPL. Now what seems to be entirely possible is to define a binary interface or object interface licensed using a non-viral license, such as MIT/X11, use that interface to implement a GPL-incompatible application and distribute the two separately.

Before I go any further, there are a couple important things to consider here, who owns the code at hand and what is their interpretation of the GPL. The owners appear to be UCLA, some students at UF, and primarily the professor introduced at the beginning of this blog. Our first goal is try to get UCLA to allow a relicense of the software into MIT/X11 or even relinquish their right over the software, at which point, we could keep that license wrapped up and then relicense the software publicly as needed. More importantly that would mean the GPL interpretation used would primarily be the professor's, thereafter, it would be distributors such as students and even Linux distributions such as Debian, Ubuntu, and Redhat.

This leads to a more interesting question, what is generally accepted. Based upon the message I linked to above, it leads me to believe we either need to have a GPL friendly version or the application needs to work with out it. Furthermore, the application must be distributed separately and the GPL code cannot have auto-install or configuration scripts that bypass the GPL. Then users who install the non-GPL libraries would be not breaking the GPL in so long as they themselves do not distribute the set publicly. Of course, the auto-install / configure scripts could be part of the GPL-incompatible code.

This leads me into a more interesting question, can we distribute these in a VM? Given that the GPL allows use of "OS libraries" be mixed together, both GPL and GPL-incompatible, I believe that a VM Appliance can have such applications bundled together.

This all really boils down to what the intent of the GPL is and more specifically the license owners. My opinion is that no one wants to see their source code be eaten up and used without at least recognition and bug fixes being trickled into mainstream code. Closed source developers that desire stability would contribute both bug fixes and features that they deem in the critical path to the maintainers of Brunet, otherwise they would have to deal with maintaining complicated sets of patches.

Taking a step back, can someone stop me from distributing a library or application that links to MIT/X11 code, I think in all cases, the answer is no. Can someone stop me from bundling said binaries especially in the case of GPL-incompatibility with a GPL application, I think the answer is clearly yes. I am very curious how others see this issue. Most importantly, I really look forward to understanding all the points of view in the group. It has become very apparent to me that everyone has an opinion on this and I wouldn't be surprised that we don't find out whether or not we went the right or wrong way until someone else tells us.

My Impressions of DTLS and OpenSSL

So after a weekend of fun with DTLS, OpenSSL, and OpenSSL.NET, I am not entirely onboard with the idea of using it as the backend for our P2P Security system. So let me list my pros and cons of using DTLS:

Pros:

• Maintained by someone else
• FIPS certified
• Availble for all platforms
• Popular, well documented protocols

Cons:

• Lacks the level of documentation found for most of the other libraries I've used
• DTLS is scarcely mentioned on the website and it is unclear to me how well it is actually maintained
• DTLS (manual) renegotiation is broken
• The library doesn't really make it easy to determine when a SSL session should be aborted
• There is no async callbacks for when data is ready, which would be really helpful for handshake messages
• Platform dependent (i.e. library -- ABI -- is not portable, API is)
• The DTLSv1_methods lack the reliability layer that is actually in the DGRAM BIO
  

Given that there is an existing Security library in Brunet that actually handles the above cons lacking FIPS and support by others, it feels like I am between a rock and a hard place. My real goal was to get enough knowledge to add the details in my paper comparing my experiences with a standard DTLS implementation in comparison to a home brewed version. Since the transition to DTLS hasn't been entirely smooth sailing, I am a little hesitant to jump on board. I suppose two things need to be ensured to me, that renegotiation works and that messages are still transferable during renegotiation. While waiting, I may just test out what actually happens in systems that rely on DTLS, because I have a feeling that the libraries are significantly faster than .NET crypto libraries.

Note: I stand corrected, the DTLS bug is fixed in version 1.0.0 betas, though with a new added twist of segmentation faults!

OpenSSL-Net

After my post yesterday, my next task was to do the same in .Net. Boy was I rewarded when I used Google to search for "OpenSSL Net". I was presented with "OpenSSL-NET", a managed wrapper for OpenSSL! Interestingly enough, this wasn't the first time I have looked for such a project. In the past, the most I found was people looking for the same. What a reward, I don't have to implement a C# wrapper for OpenSSL :). Now on to the good stuff, I took the work I did yesterday and converted into a C# object oriented sample. To reproduce my efforts, I have packaged all my work here. For simplicity, I've included an OpenSSL-NET dll. One caveat, the current distribution of OpenSSL.NET won't work with the binary, I had to make a few tweaks, such as making the Ssl, SslContext, and some Enums linkable (i.e. internal to public) and then I added a couple of additional Ssl functions to the library. I've sent a patch to the manager and hopefully the changes get imported into the official library.

What does that leave next? Figuring out how to distribute OpenSSL with our libraries for Windows support. Windows presents an interesting problem as it apparently lacks an up-to-date OpenSSL distribution. So with that in mind, I'll probably attempt to integrate our current security stack and have OpenSSL DTLS as an option instead. That way, users who just want a simple to install system don't have to worry about installing OpenSSL and configuring our application to properly link to it.

Also another booger about OpenSSL is that the library names differ between Windows and Linux and there are a TON of functions called by the library. As such, the author has a library for Linux and Windows. One option to resolve this is to use a Windows version and created symbolic links in the local directory from the Linux ssl libraries to Windows ssl names.

I'm really looking forward to removing dependency of our system from our own security stack. It means that we don't have to worry (as much) about something so critical. I don't regret writing the original security stack, because I learned SO much and much of the framework or at least the ideas will be kept from the transition of the old stack to the new. Also I suspect the transition from managed security code to unmanaged will create a much faster system.

To assist in keeping track of my changes to the OpenSSL.NET code, I've put up a git repository at github.

Memory BIOS and OpenSSL

I spent the past 12 hours or so learning how to "use" OpenSSL with the goal of potentially using it to replace our existing security system. Why? Because security is hard, there are so many ridiculous attacks that can be made that probably have no real effect on anything, but it makes people nervous, like timing attacks that have the ability to decrypt the first few messages in a symmetric exchange. Research is ongoing to discover ways to break security and at the same time to improve protocols and algorithms. I think its all great, but I don't have time to worry about it. Its hard argument to make that creating a new system that takes the best parts of DTLS and IPsec is really the best pathway, if you can mess around with a well supported DTLS implementation to create an IPsec like system. While I didn't find something identical to that, I found that guys over in France have implemented a "P2P" VPN that uses DTLS in a non-standard way. So I suppose it is possible. With the remaining text in this entry, I'll discuss my recent experiment with OpenSSL and DTLS, which was rather quite interesting since I haven't coded in C/C++ in 4.5 years now :).

First off, OpenSSL documentation is horrible, it is spare and spread across the web. Most of the error messages don't really make sense. The requirements and order of operations for creating secure sessions aren't very clear. I used a mixture of sample code and their man pages to figure out what to do. Though there is really little text on DTLS. It turns out for our local testing purpose, it really doesn't matter which security protocol you use.

So onwards to how this whole thing works, we'll assume that you know how to make a self-signed certificate PEM file with an RSA PEM key file, which can be located in "server.pem". Now we can get on to the code, here's where ordering matters. Lesson 1) complete all operations with structures before moving on to others that are created from them.

To keep everything organized, and because I don't know SSL well enough, I created a structure so I can access everything I create:

typedef struct connection {
BIO* read;
BIO* write;
SSL* ssl;
SSL_CTX* ctx;
} connection;

The bios are the memory bios. The names for read and write are from the perspective of the SSL structure, i.e., read is used to write network traffic from, while write is used to read network traffic. The rest will become clear in the later parts of this post.

So here's how to initialize a server (minimized version):

connection *con = malloc(sizeof(connection));
con->ctx = SSL_CTX_new(DTLSv1_method());
 SSL_CTX_use_certificate_chain_file(con->ctx, "server.pem");
SSL_CTX_use_PrivateKey_file(con->ctx, "server.pem", SSL_FILETYPE_PEM);
SSL_CTX_use_certificate_file(con->ctx, "server.pem", SSL_FILETYPE_PEM);
SSL_CTX_set_options(con->ctx, SSL_OP_SINGLE_DH_USE);
SSL_CTX_set_session_id_context(con->ctx, sid, 4);

con->ssl = SSL_new(con->ctx);
SSL_set_accept_state(con->ssl);
SSL_set_verify(con->ssl, SSL_VERIFY_NONE, NULL);
SSL_set_session_id_context(con->ssl, sid, 4);

con->read = BIO_new(BIO_s_mem());
BIO_set_nbio(con->read, 1);
con->write = BIO_new(BIO_s_mem());
BIO_set_nbio(con->write, 1);
SSL_set_bio(con->ssl, con->read, con->write);

So what's interesting here? The most annoying issue was the ordering. You need to initialize the CTX with all those options before creating SSL or the SSL won't have the CTX options. That vexed me for an entire night, while I slept for 2 hours. The first two CTX options are clear, we need a certificate and a private key. The following option, SSL_CTX_set_session_id_context, needs to be set to some unique value, apparently not setting could cause a certificate exchange to fail, though I haven't experienced that issue. My favorite was the use of Diffie-Hellman, where I come from, the Diffie-Hellman exchange was handled in the background unless you really want to get in and dirty, not with OpenSSL, you either need to specify a Diffie-Hellman key file, a callback, or use the single use flag. Now we can create the server and since it is a server, we set it up for an accept state. The final components are the creation of memory bios, so that we can abstract the sending mechanism. Memory bios might be non-blocking, but it was explicit anywhere, it works if you don't make them non-blocking, but why risk it? Wow, looking back, that was kind of easy!

The client is even easier:

connection *con = malloc(sizeof(connection));
con->ctx = SSL_CTX_new(DTLSv1_method());
SSL_CTX_use_certificate_chain_file(con->ctx, "server.pem");
con->ssl = SSL_new(con->ctx);
SSL_set_connect_state(con->ssl);
SSL_set_verify(con->ssl, SSL_VERIFY_NONE, NULL);

con->read = BIO_new(BIO_s_mem());
BIO_set_nbio(con->read, 1);
con->write = BIO_new(BIO_s_mem());
BIO_set_nbio(con->write, 1);
SSL_set_bio(con->ssl, con->read, con->write);

We start with the same initialization of the CTX block and then for the SSL structure we set it to connect state. The rest is the same as the server.

So now we have usable client and server ssl structure, we need to do some sending between the two, that looks like this:

SSL_do_handshake(client->ssl);
handle_send(client, server);
handle_send(server, client);
handle_send(client, server);
handle_send(server, client);

The system needs to be started, so we start with a SSL_do_handshake on the client ssl, this causes the client to attempt to connect with the server. We use handle_send to facilitate this and abstract it from this component. After 2 round trips, both the client and server have authenticated.

Here's how handle_send works (minimized):

void handle_send(connection* sender, connection* receiver)
{
char buffer[1024];
int read = BIO_read(sender->write, buffer, 1024);
int written = read > 0 ? BIO_write(receiver->read, buffer, read) : -1;

if(written > 0) {
  if(!SSL_is_init_finished(receiver->ssl)) {
     SSL_do_handshake(receiver->ssl);
  } else {
    read = SSL_read(receiver->ssl, buffer, 1024);
    printf("Incoming message: %s", buffer);
  }
}

As mentioned above, the write bio contains encrypted data, which can be injected into another read bio to insert data into the SSL structure. If the init is finished and successful and we have data waiting, it most likely is a data message. We can use SSL_read to retrieve the decrypted data.

The last part is the fun part, where we can actually see a message go across:

strcpy(buffer, "HelloWorld!");

SSL_write(client->ssl, buffer, strlen(buffer));
handle_send(client, server);

SSL_write(server->ssl, buffer, strlen(buffer));
handle_send(server, client);

This causes both the client and server to send to each other the message "HelloWorld!", which will be printed to the screen. SSL_write is SSL_read's counterpart. It is used to insert unencrypted data into the SSL structure so that it can be read from the write bio as encrypted data.

Now we just need to clean up:

SSL_free(con->ssl);
SSL_CTX_free(con->ctx);
free(con);

There you have it, a clear and concise way to use memory bios! The full source code can be downloaded from here. Due to the girth of the code, the above code is signficantly minimized and not meant to be used as is. For example, functions were squashed and error handling was removed.

So I guess the answer is, yes SSL can be used as a part of an IPsec like system, if you're willing to implement the protocol handling for multiple and parallel sessions. The only other issue that may need to be addressed is the loss of fragmentation and resends due to not using the DGRAM bio, but that's furture work.

Reassembling the PDU

While I haven't written much lately, it has nothing to do with lack of productivity but rather too much on my plate. Since I've last updated I have:

1. Manuevered through the craziness of shepherded papers
2. Put together an elite team of peers to develop a NSDI 10 submission
3. Began a P2P student discussion group
4. Mentored a high school student
5. Organized an Archer trip that will have me visiting 3 schools in September
6. Started to learn how to hold my tongue when using it would bring no lasting benefit
7. Learning to deal with significant disappointment

Academically, I'm not quite where I need to be. There is much work to be done and not enough time to do it all. I often spread myself too thin by taking on far too many tasks for many people. Over the next month, as these tasks complete, I will need to refocus on my needs and focus on graduation.

The biggest thing I've contemplated as of late is the need of allies or at least peers and elders that will speak well of you. One person that I can look to that has some interesting level of respect is my advisor. He rarely speaks, yet people seem to go out of their way to talk with him. Clearly he is at an advantage, since he gets to remain his quiet self and does not have to become overly talkative to obtain respect. There is probably a lot more to the equation than what I can see, but I feel that it largely falls under respecting the other persons work and their knowledge.

In terms of research, I am in the heat of the NSDI work and have hit a few milestones, but due to events outside of my control, I will need to redouble my effort. The individuals that I am working with are steadfast, and I am sure that their contributions will make the breadeth of the paper amazing. I think it will be a very good read and informative for anyone that is interested in VPN technologies. The only tricky part about writing such a work is that there are many fields that are interested in similar virtual networks (communication) and thus there is no doubt going to be overlap that will not easily be identifiable. I think the only way to really deal with this issue is to make sure that the committee members who may potentially review the paper are appropriately commended for their previous work as it relates to the paper. I will admit though, that I am scared with a recent added burden that I may have too much on my hands to satisfactorily complete the paper. Perhaps my advisor or a fellow student will go above and beyond what previous coauthors have.

First conference level publication

Just the other day, I got some excellent news: My publication entitled "On the Design of Scalable, Self-Configuring Virtual Networks" was accepted into SuperComputing 09 with an acceptance rate of ~22.6%. Sadly it was shepherded, so I have to do some tweaks to it, but hopefully it won't be too painful. The only thing that may make it painful, is that all I have left is a tex file and none of the original images from the submission. The paper can be found on the sc09 web page:
http://scyourway.nacse.org/conference/view/pap152

Writing Application Entry Points

A long time ago, I spent a few days working on an option parser for C# tailored around my application, SimpleNode. The name was quite misleading as SimpleNode allowed users to make many nodes, act as user interface into the system, and probably a handful of other behaviors that have long since escaped me. It was largely unwieldly and probably not horribly useful as a learning tool nor very reusable. So I chopped everything out and went with thin applications that were tailored around objects that had simple, specific tasks. This allowed better integration at the cost of dumbing down the command-line interface and having multiple applications that had very similar behavior.

I typically desire to do things right the first time and to not come back later. When I don't get things quite right, I acknowledge the problem but add it to my priority queue, so I may or may not ever get back to it. Recently, we've had an increase of interest in our system along with the a stage 1 completion of my groupvpn, so I felt it fitting to improve the usability of our project. Lucky for me, one my peers in the lab is very interested in technologies and was able to point out NDesk.Options. I think the results are pretty good. What displeases me is that this sort of interface is probably pretty generic and so if a user wants a similar frontend, they'd have to copy and paste 100 lines of code to get the same thing, which may be ridiculous if what they do only requires a few lines, like this.

Originally, the posts title was going to be "Polymorphic Entry Points." I'd love to have something like that. I guess this can be achieved by adding helper methods to each class we instantiate that requires more than 5 lines, but I feel there needs to be a clear distinction between application entry point and classes as I think it makes it easier for users to understand how everything fits together. Maybe I'm wrong and I'm probably over thinking this :).

Dissertation Step 1

Main focus point: P2P VPN architecture with emphasis on scalablity, user-friendliness, and performance.

Scalability:

1. Effects of latency on VPN connectivity
2. Optimal peer count, cost per peer connection
3. Trusting a P2P infrastructure
4. VPN Router model
   

User-friendliness:

1. Ability to deploy and manage large, distributed infrastructures
2. GroupVPN - ability to use groups to manage VPN connectivity
3. Userspace based, non-root VPN solution
   

Performance:

1. Use of proximity neighbor and shortcut selection
2. Hybrid-based VPN model
3. Inverted P2P VPN model

Out with the old, in with the new.

On Friday at 6 PM EDT, my computer with components of varied ages all over 2.5 years did the unthinkable, died. I'm not really sure the entire process, the motherboards capacitors near the PSU plug were all blown and may have been leaking for a while and the PSU made a loud bang as I tried to see if it was broken. Using another PSU, I was able to boot the computer, but I had no USB nor fans working, I even replaced the broken CAPs to no avail. The PSU of course was dead, but it so happened to destroy two hard drives as well, both of the western digital first generation SATA 3.0. Luckily, I may be able to recover the data if it happens that only the boards are dead and the drives internals are fine.

Around 10 PM, I e-mailed my advisor with the situation and 1 hour later, he had a one line e-mail telling me to price out a machine and aim for less than 1K. Monday morning we ordered the machine, and Wednesday evening, I had a fully functioning system.

I will admit that it really sucks that I lost a lot of time, potentially lot's of data, and my own computer, but on the flip side, I am very happy to be blessed to have a considerate professor who recognizes my need for a high performance desktop computer. The machine is amazing, a new Core i7 920 with 6 GB of RAM. Even typing in commands at the terminal seemed faster than ever before.

As a lesson to this all, I have decided to employ a mechanism of maintaining the same (or nearly the same) home directory on all machines via rsync. When I work on a remote machine, I will begin my session with a pull from the desktop and when I finish, a push. To perform this, I've written to wrapper scripts to rsync, one called push and the other pull.

Push:
rsync -Cvauz --force --delete -e "ssh -i /home/name/.ssh/key" /home/name/ name@desktop:/home/name/

Pull:
rsync -Cvauz --force --delete -e "ssh -i /home/name/.ssh/key" name@desktop:/home/name/ /home/name/

C - uses a ~/.cvsignore, though I could just use an ignore file. By doing this, I can have files that aren't synced, like my mozilla firefox plugins and other cache files.

v - verbose, probably unnecessary
a - archive, keep owner, user, group, modification times, permissions, symbolic lnks, and devices, as well as recurse directories

u - update, skip files that are only newer on the receiver

z - compress files prior to transferring

force - overwrite files with directories and vice-versa

e ssh - how to execute the command

Notice the ending forward slash ("/"), that is necessary or it will actually copy the directory and place it at the destination (so you'd get /home/name/name)

Beyond backing up my data, changes to my configuration files will be moved around with little effort from myself.

Enabling Xml-Rpc for Mod-Mono in Apache

As part of my work, I am interested in enabling web services that reuse much of our current code base as well as in a similar language / environment. As such, I wanted to make it so that Xml-Rpc could be accessed via Mod-Mono and Apache. It seems a little scary at first, but what I did was break it down in to pieces.

1. Get mod-mono and xsp installed. I did this by source


2. Enable mod-mono, the compilation produced a /etc/apache2/mod_mono.conf that can be placed into mods-available and then symbolically linked to mods-enabled ( I haven't played with a2enmod, so that may be another solution).


3. Then in a directory on my website, I followed these directions.

Virtual Networking in the Cloud

Originally clouds provided public addresses for all resources in the cloud, for IPv4, this obviously doesn't scale well.  The alternative is placing a few on public network and the majority on a private network with each public address costing additional money motivating users to use private addressing.  Some potential solutions involve setting up complex static links via L2TP or GRE.  The problem is that for each one of these connections, it requires an operating system specific configuration.  Many users are unfamiliar with these protocols and according to wikipedia they aren't necessarily that easy to configure in Windows prior to Vista.

Now you could argue that most users won't use Windows with the cloud and that'd be a fair argument, so perhaps they could just use Linux and have automatically configured L2TP/GRE paths.  That could potentially work, though for a shared address space that would require a single L2TP/GRE server that requires being on a public address.  All nodes would have to route through that point in the public network.  Two cases where this is a big issue are:  1) two machines on the same physical network would have to route through the L2TP/GRE server for their "virtual" addressing and 2) two remote machines not connected on the same physical network nor to the public address would have to route through the public machine to communicate with each other, creating a huge potential bottleneck.

The problem lies in the desire of vendor lockin via centralization.  If all cloud vendors were to use a unified interface, users could at least minimize their overhead in working with different cloud providers.  Though cloud vendors may be not so excited about having a unified layer 2 network with others and even if they did it would have to be a selective process so that not all cloud vendors may provide this service.

What P2P Overlays and Network Virtualization can do to improve the situation:

• Migration between clouds and between a cloud and the users environment


• Separation of network spaces between resources in a cloud


• Private communication between multiple clouds and user resources


• Supporting a single address space across multiple sites in a single cloud

Enough for now, I'll discuss the above in future posts.