tag:blogger.com,1999:blog-23212805679132577772024-03-19T01:48:29.178-07:00David's BlogDavid Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.comBlogger39125tag:blogger.com,1999:blog-2321280567913257777.post-82388330623007880722015-11-23T06:56:00.001-08:002015-11-24T06:54:25.333-08:00Optimizing Server Interaction with Screen and Friends<p>
Since becoming part of industry, I've come to see the advantage of doing my work on a remote machine (a DevServer). Now historically, I had been this Linux nut, who loved using tiling window managers. My workflow has been launch a handful of terminals to begin working and launch more as needed. This was a rather mundane setup that just worked. Unfortunately, it just doesn't work in industry. I encountered several challenges:
</p>
<ul>
<li>Launching a terminal connects me only to my machine</li>
<li>Cannot work locally (challenges through the roof to make this even possible)</li>
<li>Accessing the server requires SSH, SSH requires the use of two factor auth (no auto login)</li>
<li>Long-lived sessions with SSH are impossible, it cannot withstand a computer suspend and rarely even a momentary loss of network connectivity</li>
</ul>
<p>
So the question became how can I reproduce, if not create an enhanced version of, my previous environment with minimal effort?
</p>
<ul>
<li>Find the simplest terminal emulator (<a href="http://st.suckless.org/">st</a>)</li>
<ul>
<li>For better colors, add the <a href="http://st.suckless.org/patches/solarized">solarized patch</a></li>
<li>I tried a few other patches, but they didn't seem to work on my system</li>
</ul>
<li>Use screen locally</li>
<ul>
<li>A keyboard only way to access scrollback buffer (content that was printed out that no longer fits on the screen)</li>
<li>Make the local screen respond to easier to access keys: Alt-a instead of Ctrl-a</li>
<li>Quicker access to my screen windows I'm looking for (windowlist -b) and make this easier Alt-a ' (instead of Alt-a ")</li>
</ul>
<li>Rename the terminal and screen window with the PWD and command executing inside (fun stuff with PROMPT_COMMAND)</li>
<li>Use screen remotely</li>
<ul>
<li>Always ssh into a long-lived state</li>
<li>Restart that state automatically, in case I manually close it out or the machine is rebooted</li>
<li>No need to SSH in for each additional window, just create new windows in the current screen session</li>
</ul>
<li>Use mosh</li>
<li>No need to deal with SSH flakiness -- automatic connection reestablishment after lossy network usage or suspend / resume</li>
<li>Mosh can be flaky off of our corporate network -- I can switch to SSH and resume my screen session there</li>
</ul>
<p>
The only thing really missing is that I would love to be able to create multiple distinct views of my remote screen session without multiple SSH sessions. Think of it this way, if I have 3 windows open on a remote screen session. I can only view one of them at a time unless I ssh in again and attach to that same session. Ideally, I could move the multiplexing locally, alas, I couldn't figure a clean way of moving the screen unix domain socket locally and have that local screen connect to it.
</p>
<p>
Now it is time for the useful code bits.
</p>
<p>
My .screenrc:
</p>
<pre>
vbell off
startup_message off
autodetach on
defscrollback 100000
shelltitle '$ |/home/davidiw bash'
#hardstatus string "%h"
#caption always "%{= kw} %-w%{= wk}%n*%t%{-}%+w%{= kw} %=%d %M %0c %{g}%H%{-}"
#hardstatus alwayslastline
escape ^Aa
register S ^A
bindkey "^[a" process S # On the remote machine, i set this to "^[s", so I don't have to type Alt-a a
bind "\'" windowlist -b
bind "\"" select
</pre>
<p>
Append this to .bashrc to get nice names for screen and xterm title's and to start screen with the default session (home):
</p>
<pre>
function make_title {
if [[ $BASH_COMMAND == 'trap "make_title" DEBUG' ]]; then
cmd="bash"
else
cmd=$BASH_COMMAND
fi
echo -ne "\033]0;$PWD $cmd\007"
echo -ne "\033k$PWD $cmd\033\\"
}
term=$TERM
case $term in
*screen*)
;&
*term*)
export TERM="xterm-256color"
export PROMPT_COMMAND='trap "make_title" DEBUG'
;;
esac
case $term in
*term*)
exec /bin/bash -c "screen -AUx home || screen -AUS home"
;;
esac
</pre>
<p>
Future work:
</p>
<p>
<ul>
<li>Shared local vim buffer</li>
<li>Shared remote vim buffer</li>
<li>A git repository to make reproduction easy</li>
<li>Fix for mosh to not break on non-corp networks</li>
<li>Clickable URLs in the Terminal</li>
</ul>
</p>David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-21638102309509754432015-03-19T08:28:00.005-07:002015-03-19T08:29:08.197-07:00GCC and UD2 instructionsA few colleagues and I are working on OS development. While most of the development has taken place on MacOS, I prefer Linux and primarily use rolling release distribution called <a href="https://www.archlinux.org/">Arch</a>. On the Mac, my colleagues obtained GCC 4.8 from mac ports and everything compiles just fine for them. However, having a rolling release version of Linux implies I will always have the latest and greatest versions on my system. Usually that is fine, sometimes not as in this scenario. At some point, GCC started introducing UD2 instructions instead of emitting errors. Now specifically, a UD2 instruction stands for undefined instruction and causes the system to halt. Why on earth would any compiler perform this function? It was absolutely baffling to see this type of behavior from a properly compiled program that was built using -Wall.<br />
<br />
So I did some searching in the assembly output to find where the UD2 instruction was being generated and found one in the following code snippet:<br />
<br />
<p>
static struct pci_func * alloc_pci_func(){
if (pci_dev_list.total_dev == N_PCI_DEV)
{
KERN_DEBUG ("Alloc pci_func from pci_dev_list error! no available \n");
return NULL;
}
return &pci_dev_list.dev[pci_dev_list.total_dev++];
}
</p>
<br />
Where do you think the problem is? My initial reaction was that maybe this is due to some fancy overflow detection not working quite right, notice that we increment total_dev but limit from incrementing it beyond N_PCI_DEV. This did not work. So I tried a slightly different method, I looked at our current optimization level and it happened to be -Os, or effectively -O2 with some tweaks for size of output. So I went to -O2 and then -O1, at -O2 the issue still existed whereas in -O1 it did not. Taking a peek at the list options enabled by -O2, I set the compilation to -O1 and began enabling -O2 options explicitly until I stumbled upon the problem: <b>-fisolate-erroneous-paths-attribute</b>. This flag happens to do the following: Detect paths which trigger erroneous or undefined behaviour due a NULL value being used in a way which is forbidden by a "returns_nonnull" or "nonnull" attribute. Isolate those paths from the main control flow and turn the statement with erroneous or undefined behaviour into a trap.
Brilliant, the others figured it was better to turn in return NULL is undefined behavior than warning us that maybe we should look into a different convention. Frankly, I'm not sure what the correct convention should be, perhaps a panic? But that seems a little bit harsh especially if the system can handle running out of limited resources. So to keep our -Os setting I also added the following compiler flag: <b>-fno-isolate-erroneous-paths-attribute</b>. Fortunately I found my bug issue, but it seems to be expected behavior from GCC. Mind you, this isn't the only example of a <a href="https://twitter.com/blackhole0173/status/306286002215993346">GCC UD2 issue</a>.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-86023036675092914062013-04-24T21:18:00.002-07:002013-04-24T21:18:31.524-07:00Race Conditions Resulting in Double FreeIn working with Qt and its slot and signals, I've encountered quite a few double free errors despite my use of QSharedPointer. One particularly puzzling situation, I have stumbled upon involves the following model:<br />
<br />
<ol>
<li>An object, say controller, owns another object, say agent</li>
<li>Controller subscribes to a event (signal) from agent called closing</li>
<li>If the agent issues a closing signal, the controller removes it from a list of known agents (effectively freeing it as a result of the QSharedPointer).</li>
<li>If the controller is closing, it calls close on the agents and then removes them from its list. </li>
</ol>
The double free occurs when the controller closes before the agents, effectively causing the agent to notify the controller it is being closed while both are being destructed. This probably occurs because the Qt signal disconnect code would happen in the base destructor (QObject), which would be called after the Session destructor. The easy resolution was to disconnect all signals except destroyed.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-37323751549753014792012-12-10T12:59:00.001-08:002012-12-10T12:59:08.047-08:00Message / Packet ParsingA common problem in designing and writing distributed systems is the handling of the wire protocol. To help in solving these problems both many programmers go it on their own writing their own serialization tools, while others trust third-party tools to ease their development. After doing a little bit of both I'm not convinced I prefer one approach over another.<br />
<br />
<b>The example:</b><br />
<br />
<br />
First let me provide an example of a message:<br />
<br />SampleMessage {<br />
int record; // unique id<br />
Type type; // some enumed field<br />
bytes message; // The data<br />
bytes signature; // Integrity and authenticity of this "SampleMessage"<br />
};<br />
<br />
<b>Custom made serializer:</b><br />
<br />
Using our own serializer, we could do the following assuming a SampleMessage msg:<br />
bytes buffer;<br />
<br />
<br />
buffer.append(msg.record());<br />
buffer.append(msg.type());<br />
buffer.append(msg.message().length());<br />
buffer.append(msg.message());<br />
buffer.append(msg.signature().length()); <br />
buffer.append(msg.signature());<br />
<br />
And then on the parsing side:<br />
bytes buffer;<br />
<br />
msg.set_record(buffer.read_int(0));<br />
msg.set_type(buffer.read_int(4));<br />
int length = buffer.read_int(8); <br />
<br />
msg.set_message(buffer.mid(12, length));<br />
int signature_length = buffer.read_int(12 + length); <br />
msg.set_signature(buffer.mid(12 + length + 4, signature_length));<br />
<br />
So the major caveats are the following: what is the size of Type, is it uniform across all platforms? Also we're making a lot of potentially unnecessary copies for what might be large datagrams.<br />
<br />
<b>Third-party serializers (without message definition):</b><br />
<br />
Alternatively, let's assume we have something like QDataStream:<br />
QByteArray data;<br />
QDataStream stream(&data, QIODevice::in);<br />
stream << msg.record() << msg.type() << msg.message() << msg.signature();<br />
// or maybe even <br />
stream << msg;<br />
<br />
For parsing:<br />
QDataStream stream(data);<br />
stream >> msg;<br />
// or maybe not...<br />
int record;<br />
Type type;<br />
QByteArray message;<br />
QByteArray signature;<br />
stream >> record >> type >> message >> signature;<br />
msg.set_record(record);<br />
...<br />
<br />
In this case, we just have to check that our output is sane or perhaps look at the QDataStream and ensure that it is still in good working order (Status != ReadPastEnd), but how do we check the signature matches the tuple (record, type, message) in any efficient manner?<br />
<br />
<b>Third-party serializers (with message definition):</b><br />
<br />
A completely different serializer, protobufs would work as such:<br />
std::stringstream stream(std::stringstream::out);<br />
msg.SerializeToOstream(&stream);<br />
string output = stream.str(); <br />
<br />
And on the return:<br />
std::stringstream stream(output, std::stringstream::out);<br />
msg.ParseFromIstream(&stream);<br />
<br />
Protobuf doesn't handle the signature issue any easier and requires both an external compiler and a library.<br />
<br />
<b>Thus far...</b><br />
<br />
Protobufs would be great if we could encapsulate a Message within a SignedMessage, then we *should* be able to get the original character array used for constructing the Message and verify that the signature is correct. Unfortunately that does not happen.<br />
<b> </b><br />
QByteArray does allow for constructing a QByteArray from another without doing a copy of the underlying array. However, we do not have the access we need from QDataStream to know where into the QByteArray to construct the base (unsigned) message.<br />
<br />
Using our own method allows us to have this fine grained control but at the cost of writing more expressive code and having more debugging routines.<br />
<br />
<b>Similar Packets</b><br />
<br />
Ideally we want to reduce our packet parsing as much as possible. So we can embed multiple packets in the same path. Using something like protobuf, where we must define the data we expect to be pushing around, makes it complicated for this arbitrary behavior. Requiring us to embed packets of one type as bytes in another or requiring this lower level packet to know about higher layer packets breaking modularity. The same could be said about QDataStream, but then again it allows us to avoid unnecessary copies. In either case, both scenarios feel unnatural. If we want our home grown packets to have these features, the code will start feeling bloated and potentially complex -- welcome to a whole new world of coding bugs.<br />
<br />
I'm still brainstorming on my conclusion and hopefully I'll update when I'm satisfied until then....David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-71445951582803804232012-07-03T19:50:00.002-07:002012-07-03T19:51:55.601-07:00Social KeysBack in my days at <a href="http://www.acis.ufl.edu/">Florida</a>, I worked on a project called <a href="http://socialvpn.org/">SocialVPN</a> with the intent of using <a href="http://www.facebook.com/">Facebook</a> as a medium for exchanging public keys amongst friends. Now I am revisiting this with another project called <a href="http://dedis.cs.yale.edu/2010/anon/">Dissent</a>, in which we want to use public keys from a group of users who may not be friends. Ignoring the why for each of these applications, let me describe the way things were, where they have gone, what I would like to see, and what could be minimally done.<br />
<br />
Back with SocialVPN, we created something called a "desktop application" to store the keys in Facebook. An application in this sense utilizes Facebook in some means to enhance their experience in another domain, such as a game, organizational things, searches, or identity... anything that can benefit from social identity or the contents of a Facebook profile. A desktop application, unlike a Web application, was a standalone application that did not require anything besides Facebook. Unfortunately, this flexibility, is probably why desktop applications did not live very long. Using Facebook application, we could store the keys within an application specific store accessible only to the application. Unfortunately, the applications required both the Application ID and Secret be embedded within the application, and thus a hacker could retrieve both of them, muck around in the application's data store, and perform man-in-the-middle attack. I suspect this was one of the many reasons why Facebook moved to Web Applications and away from Desktop Applications.<br />
<br />
One day, in the midst of success, this suddenly stopped working. It was horrible! We quickly created a Web application to do the same thing, but this time, I suppose it was actually "safe" (assuming we did not introduce any accidental bugs), but unfortunately, this meant the clients had to trust us. I did not want that! Anyway, we got fed up over all this nonsense and began investigating GoogleTalk and XMPP. Wow, that was wonderful, long story short, we found everything we needed to exchange keys between friends in a reasonable fashion without relying on a third-party services (besides Google, of course).<br />
<br />
Fast-forward several years, and again, we are considering the same task of inserting keys into Facebook. I had hoped that the years would have been good to developers and Facebook would have refined their APIs some how or another. Taking a step back, let me first explain what we really want. Ideally, a public key would be a first class citizen in the SN realm, such that under your info, there would be a special field where your key is stored. Perhaps the key would actually be signed by Facebook so it could be easily redistributed outside of Facebook, yet still trusted. In terms of API, perhaps the Facebook Graph API could be extend as such: https://graph.facebook.com/$userid?fields=public_key which would allow applications to retrive a base64 encoded key. Furthermore, this interface should be open to the public or as much as the profile is, so that the user can authenticate to members that are not friends, but they have some association, such as in a common group. Unfortunately, this does not exist, nor have I seen anything coming from SNs like it. I will admit that there was a position paper 2 years after our SocialVPN papers clamoring for SocialKeys, I should read it and update this entry...<br />
<br />
So where are we now? While Facebook's APIs have matured, so has their privacy measures. I suppose the APIs are largely the same from where we left off, we just never envisioned using Facebook's own internal data structures to store the key. Well, we did at one point consider using steganography to embed it within the user profile picture, but I think that can be challenging since the picture probably has been internally manipulated by Facebook, which would probably destroy traces of the public key. Other fields can also be populated, such as notes or picture albums, which can be searched by their subject or title, respectively, using FQL. Unfortunately, notes does not allow for an application to change the privacy setting and uses the default user setting for notes, while picture albums could potentially be used, the privacy setting cannot be looser than what the application is set to. By default applications are set to the profile as well, thus the only Facebook limited option that would be guaranteed to work, would require user interaction. Furthermore, an application must be granted the privilege of creating, editing, or deleting content. Once this privilege has been granted by a user, it must be manually deleted through the applications menu, not a very user-friendly interface for an application, and tin foil hat and many users should be wary of an application that needs constant write access for a one time (or occassional) operation.<br />
<br />
A potential solution that should always work, would be for us to have an application, that reads the key, prints a base64 version for the user to copy into a note, and then verifies that the user has done so correctly. This may in fact be a completely reasonable approach, but it does require greater user interaction than most casual users are interested in, which would certainly limit our application's appeal.<br />
<br />
Yet another issue that may come up in working with a profile is the issue with "hidden users," or users who show up in the group or have some presence, but their profile is only accessible to friends. If an application needs uniform access to profiles, it should be able to detect these types of users, warn them that their profile is too restrictive, and prevent the accidental use of their credentials to impact the rest of the group. For example, a friend of these users may see a different set of public keys than someone who is not.<br />
<br />
So thinking rather practical, what is one way we can accomplish this without any "additional" user interaction? Why not let an external app manage the data, back like we did with SocialVPN? These days Facebook has teamed with Heroku to provide a free service, so devs do not even need to manage or pay for the service. While this seems like a great solution, it changes ownership of the key from the profile, to the application. In the earlier iterations discussed above, the profile owned the key, independent of the application. In this system, the application owns the key, but links it to a profile. Granted this is probably more tin foil hat wearing than anything else, but because the profile key does not appear in the users profile, another user must trust the server that it is the correct public key. I also believe this is a bit stronger of an assumption that trusting the social network, which is inherent in the design anyway. A third-party developer is a stop gap, until social networks embrace social keys, perhaps I need to come up with persuasive arguments for why they should.<br />
<br />
So maybe I can get some feedback on this problem. That would be great. Alternatively, we may just go with the external application, since it is guaranteed to work, but also discuss the "safer" alternatives.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-17791613946463039382011-05-12T08:28:00.000-07:002011-05-15T18:57:39.443-07:00Recursive Virtualization Xen in KVMAs I wrap up my contributions to UF's, or more importantly Renato Figueiredo's, Virtual Computing course, I am putting together a Debian Xen image, which should work in Simics and a host of other VMMs. Because VMware is the least compatible, I always build my initial image in VMware and then quickly continue working on it in a chroot environment of course after converting it into a raw disk format. Anyway, in my most recent attempt, I was trying to figure out why Xen would not boot inside KVM, when it worked fine in VMware and Qemu.<br /><br />First a little background, Xen was compiled for 32-bit and running a 32-bit userspace, while my KVM install was 64-bit. As you may or may not be aware, x86-64 does not support privileged rings any more and instead relies on the paging modes (U/S) for privilege. Xen 32-bit relies on the rings for performance purposes. So you would think that KVM would support the rings as well. As it turns out, KVM 64-bit does not appear to virtualize the rings and thus Xen 32-bit does not work.<br /><br />While this is still somewhat of a hunch, I tested a Xen 64-bit, which does not use the privilege rings. In KVM 64-bit, it worked flawlessly. Leaving me two more things to check: determine if Xen 32-bit boots in KVM 32-bit and if a simple ring test boot code works in 32 or 64 bit KVM. The former would be useful for other's, while the latter would be a conclusive simple test to verify the functionality.<br /><br />Anyway, the point is, you can run Xen in KVM, if they are compiled for the same platform. I'd be curious to see if you can run a 64-bit VM in 32-bit KVM and if so, if the Xen 64-bit would work.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com1tag:blogger.com,1999:blog-2321280567913257777.post-89564952033841535912011-05-11T13:49:00.000-07:002011-05-11T14:02:11.636-07:00Dangers of IPTables NAT with DHCPFor <a href="http://www.archer-project.org">Archer</a>, I have setup a small private LAN for all the core resources to share with a common out into the main network, which connects to the Internet. I use Linux's IPTables via masquerade for NAT or routing the packets from the private network into the Internet and DHCP to automatically assign IP addresses to the resources. Recently, I extended the router to support other hosts to use it to bridge into the local network, which may or may not be responsiblef or a whole host of other issues. The two primary issues were routing of bridged packets through IPTables and DHCP messages traversing the bridge, both request and replies from both sides.<br /><br />The routing of bridged packets was particularly difficult to figure out. I had recently setup a web server, which had a public IP address and was bridged into the LAN. It was pingable and could ping other IP addresses within the LAN, but it couldn't ping the gateway. Furthermore, it was accessible from the Internet but could not initiate communication with the Internet. It was very strange behavior. I whipped out tcpdump and quickly noticed that packets it was sending out via the bridge were being picked up by IPtables masquerade and having their source address change. So that when the packet came back, it had no where to go. The resolution to this problem was pretty easy: add "-s ip_range/mask" to the masquerade iptables line. So this:<br /><span style="font-weight: bold;">iptables -t nat -A POSTROUTING -o bridged -j MASQUERADE </span>became<br /><span style="font-weight: bold;">iptables -t nat -A POSTROUTING -s 10.0.1.0/24 -o bridged -j MASQUERADE</span><br />where bridged is the network bridge into the LAN, i.e., a Linux bridge connecting eth0 to other devices wanting access to the LAN and 10.0.1.0/24 is the private address space. Now masquerade will only forward packets with the source address 10.0.1.0/24.<br /><br />Perhaps, doing this opened up the other problem, which was that resources on the LAN were somehow getting DHCP requests to the private network. This never makes people happy and worst of all, those individuals were losing their Internet connectivity. This could be related to forwarding of broadcast Ethernet packets as well. The resolution for this was a bit easier, simply drop all DHCP packets that come across the NAT:<br /><span style="font-weight: bold;">iptables -A FORWARD -p udp --dport 67 -i bridged -j DROP</span><br /><span style="font-weight: bold;">iptables -A FORWARD -p udp --dport 67 -o nat -j DROP</span><br />In this case, packets coming from the bridged device (-i bridged) are those on the LAN looking for IPs. The packets coming from the nat device (-o nat) are those on the private network looking for IPs. These rules effectively prevent the retransmission of dhcp packets. The reality is, I probably should prevent the transmission of multicast / broadcast messages from crossing the NAT, since there exists no gateway for packets to get into the private network.<br /><br />Anyway, this was a rather unfun discovery as I found NOWHERE online that discusses these two issues when configuring NATs. I hope this turns out to be useful for someone else out there.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com10tag:blogger.com,1999:blog-2321280567913257777.post-88303129493542567102011-05-10T18:28:00.000-07:002011-05-10T18:47:32.029-07:00Ubuntu Xen -> Debian XenAfter using the Ubuntu Xen setup for a class, I must admit I am very disappointed with it. Ubuntu is just really not that great as an OS for learning. While upstart may have some long term potential, nothing beats a sys-V or BSD mechanism for startup. The simplicity allows for use of chroot, which is critical when the resulting image or kernel will be used inside a virtual machine. This was the case for the Ubuntu-Xen, which was used inside Simics, VMware, or Qemu depending on the students project. Some other hiccups included:<br /><ul><li>Non-booting kernels when compiling additionals functionality into the kernel</li><li>Inability to use grub2 and having to install grub1</li><li>Default kernel didn't have devtmpfs included and Ubuntu relies on it</li></ul>I still think that apt is superior to yum. One thing to note is that even though Debian has Xen built-in, it is useless since this is for academics, where students will certainly be expected to modify the Xen and potentially the dom0 kernels.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-65203719455114257092011-05-03T18:07:00.001-07:002011-05-03T18:14:20.877-07:00The conclusion... is only a new beginning... or rather I am not quite there yet.<br /><br />Today was my dissertation defense. After approximately 23 years in academics, I am finished with the "student" title in my life. I have completed pre-school, elementary school, middle school, high school, undergraduate school, graduate school, and, finally now, graduate school. I am a PhD as of some 10 hours ago! I put the title "student" in quotes, because as we all know, we are always students in that we continue to grow and evolve as we become older and wiser. While the years have produced both good and bad memories, I am most blessed by the individuals that I have met along the way. I could go on and on, but for now, I just want to express the elation that after 6 long years as a graduate student, I can say that I'm done. While the path was certainly not friendly, I am just happy that I conquered the PhD and was not conquered by it!David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-15348163274540770872011-01-14T11:52:00.000-08:002011-01-14T11:57:08.291-08:00Ubuntu 10.04 + Xen + SimicsThe goal of my previous posts were to create a Xen 4.0.1 + Ubuntu 10.04 and run it on top of Simics 3. Well everything works great until I got to Simics and then I would get a dreaded triple fault due to MSR 0x8000020 or MSR_EFER, which appears to be a MSR for 64-bit CPUs. It does not exist on Pentium 4 machines, Simics x86 cpu, and it is unclear why that instruction was being executed. After figuring out the stack trace in Simics, I disassembled Linux and Xen and could not find a matching set of instructions. That meant the only other possibility was that grub was at fault. It seems that during the booting of a multiboot the instruction was being called. After installing grub1 onto the machine, the machine boots in Simics.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-71053395144680797932011-01-11T09:43:00.000-08:002011-01-11T10:12:50.231-08:00Creating a Xen DomU in UbuntuAs a follow up to my previous post, where I installed Xen onto Ubuntu 10.04, in this post, I will use debootstrap to create a DomU Ubuntu 10.04 sparse image.<br /><br />Begin by installing debootstrap: <span style="color: rgb(102, 102, 102);">apt-get install debootstrap</span><br /><br />Create a sparse image (1 GB) -- sparse is attractive as it doesn't use real disk space until the image has content: <span style="color: rgb(102, 102, 102);">dd if=/dev/null of=ubuntu.img bs=1M seek=1024</span><br /><br />Create an ext3 file system in the image: <span style="color: rgb(102, 102, 102);">mkfs.ext3 ubuntu.img</span><br /><br />Prepare for debootstrap:<br /><span style="color: rgb(102, 102, 102);">mkdir ubuntu</span><br /><span style="color: rgb(102, 102, 102);">mount -oloop ubuntu.img ubuntu</span><br /><br />Run debootstrap: <span style="color: rgb(102, 102, 102);">debootstrap lucid ubuntu http://us.archive.ubuntu.com/ubuntu/</span><br />Add the Xen console device to init: <span style="color: rgb(102, 102, 102);">sed 's/tty1/hvc0/' ubuntu/etc/init/tty1.conf > ubuntu/etc/init/hvc0.conf<br /><br /></span>Add network initialization to your image by adding the following or something similar to ubuntu/etc/network/interfaces:<br />auto eth0<br />iface eth0 inet dhcp<br /><br />Set a root password:<br /><span style="color: rgb(102, 102, 102);">chroot ubuntu</span><br /><span style="color: rgb(102, 102, 102);">passwd</span><br /><span style="color: rgb(102, 102, 102);">(set password)</span><br /><br />Clean up:<br /><span style="color: rgb(102, 102, 102);">umount ubuntu</span><br /><span style="color: rgb(102, 102, 102);">rmdir ubuntu</span><br /><br />Create a xen configuration file (ubuntu.cfg):<br /><br /><span style="color: rgb(102, 102, 102);">kernel = "/boot/vmlinuz-2.6.32.27"</span><br /><span style="color: rgb(102, 102, 102);">ramdisk = "/boot/initrd.img-2.6.32.27"</span><br /><span style="color: rgb(102, 102, 102);">memory = 64</span><br /><span style="color: rgb(102, 102, 102);">name = "ubuntu"</span><br /><span style="color: rgb(102, 102, 102);">root = "/dev/xvda ro"</span><br /><span style="color: rgb(102, 102, 102);">disk = [ "file:/root/ubuntu.img,xvda,w" ]</span><br /><span style="color: rgb(102, 102, 102);">vif = ['bridge=eth0']</span><br /><br />The above assumes your kernel is Xen kernel 2.6.32.27, the current working directory is "/root", and your default network device is "eth0". Adjust as necessary. Also the memory is configurable, the default listed above is 64M.<br /><br />Check that everything works -- assuming xen is on (verify xm list): <span style="color: rgb(102, 102, 102);">xm create -c ubuntu.cfg</span><br /><br />It should boot into a domain that you can log into as root.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-51995472546127715012010-12-06T06:39:00.000-08:002011-01-11T09:41:58.883-08:00Installing Xen 4.0.1 and testing on Ubuntu 10.04Several of my colleagues have recently had a great time trying to install Xen on Ubuntu. I've seen various locations where the task has been partially done, but the authors always leave something missing. I guess in their cases, the goal is to get their job done and not make a very generic tutorial. The goal of this tutorial is to be generic, so that the process can be repeated easily without going having to return to google to figure out what the contents of this tutorial are expressing.<br /><br />So here's a rough outline:<br /><p>Install or having a working Ubuntu 10.04. If you are doing a fresh install, I recommend using ext3 on your hard disks otherwise you'll have to reconfigure the Xen-Linux kernel.<br /></p>Install the necessary utilities to download, compile, and install Xen:<span style="color: rgb(102, 102, 102);">apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial build-essential make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg62-dev iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml libx11-dev</span><p></p><p>If you are using 32-bit install libc6-xen (This will get rid of the annoying TLS segmentation message): <span style="color: rgb(102, 102, 102);">apt-get install libc6-xen</span></p>If you are using 64-bit install 32-bit utils: <span style="color: rgb(102, 102, 102);">apt-get install gcc-multilib</span><br /><p>Download Xen, either get the Xen 4.0.1 source or get testing. If you want the 4.0.1 source, its faster to download the tar.gz than the entire mecurial repository!<br /></p><p>4.0.1 source: <span style="color: rgb(102, 102, 102);">wget http://bits.xensource.com/oss-xen/release/4.0.1/xen-4.0.1.tar.gz</span></p>Testing source:<span style="font-family:monospace;"> </span><span style="color: rgb(102, 102, 102);">hg clone http://xenbits.xen.org/xen-4.0-testing.hg</span><br /><p></p>Build Xen, this includes building all tools but not the Xen-Linux Kernel:<span style="font-family:monospace;"><span style="font-family:Georgia,serif;"><br /></span></span><span style="color: rgb(102, 102, 102);">cd into the Xen dir</span><br /><span style="color: rgb(102, 102, 102);">make xen</span><span style="color: rgb(102, 102, 102);font-family:monospace;" ><br /></span><span style="color: rgb(102, 102, 102);">make tools</span><span style="color: rgb(102, 102, 102);font-family:monospace;" ><span style="font-family:Georgia,serif;"><br /></span></span><span style="color: rgb(102, 102, 102);">make stubdom</span><span style="color: rgb(102, 102, 102);font-family:monospace;" ><br /></span><span style="color: rgb(102, 102, 102);">make install-xen<br />make install-tools PYTHON_PREFIX_ARG=<br /></span><span style="color: rgb(102, 102, 102);">make install-stubdom</span><br /><p>Build the Xen-Linux Kernel (if you need to configure the kernel see the bottom of this post):<br /></p> <span style="color: rgb(102, 102, 102);">make prep-kernels</span><br /><span style="color: rgb(102, 102, 102);"> Optionally configure the kernel especially if you are using ext4 (see below)</span><br /><span style="color: rgb(102, 102, 102);"> make kernels</span><br /><span style="color: rgb(102, 102, 102);"> make install-kernels</span><br /><br /><span style="color: rgb(0, 0, 0);"> Figure out kernel version by looking at the .config file in the kernel build path (typically starts with: build-linux-2.6-pvops_), so typing "head build-linux-2.6-pvops_x86_32/.config" in 32-bit case will print out the "Linux kernel version" near the top</span><span style="color: rgb(0, 0, 0);"> (the current version is 2.6.32.26)</span><br /><br /><span style="color: rgb(102, 102, 102);">update-initramfs -c -k [kernel version]</span><br /><br />Making Xend auto-start:<br /><br /><span style="color: rgb(102, 102, 102);">update-rc.d xend defaults 20 21</span><br /><br />Update Grub<p>Place this in /etc/grub/40_custom:</p><pre><span style="color: rgb(102, 102, 102);">menuentry "Xen-Testing - Linux 2.6.32.26" {</span><br /><span style="color: rgb(102, 102, 102);"> insmod ext2</span><br /><span style="color: rgb(102, 102, 102);"> set root="(hd0,1)</span>"<br /><span style="color: rgb(102, 102, 102);"> multiboot /boot/xen.gz dom0_mem=256M</span><br /><span style="color: rgb(102, 102, 102);"> module /boot/vmlinuz-2.6.32.26 ro nomodeset root=UUID=24aece87-9b12-4966-8c04-4bd59d5808ae</span><br /><span style="color: rgb(102, 102, 102);"> module /boot/initrd.img-2.6.32.26</span><br /><span style="color: rgb(102, 102, 102);">}<br /></span></pre>Note: you can obtain your disks partition (hd0,1) by running blkid on the device, the default device will usually be /dev/sda1 (e.g., blkid /dev/sda1). Also feel free to adjust dom0_mem<br /><br />Finally update grub:<br /><pre>update-grub</pre><p></p>Reboot into Xen. While testing you should leave the default boot into your regular kernel. Hold shift during reboot to access a grub menu and select Xen. Determine which grub entry Xen, hint, they are incremented in order from 0 to N. In my case, it was 6, so edit the /etc/default/grub and replace 0 with 6 on the line "GRUB_DEFAULT". The next step is to create a domU!David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com11tag:blogger.com,1999:blog-2321280567913257777.post-86700866691794448082010-10-04T14:21:00.000-07:002019-04-28T18:10:58.504-07:00The Difference Between Polling and Events(Modified from a presentation I gave a few years back...)<br />
<br />
The scenario is my wife is preparing food for us to eat for dinner. I am waiting wondering when food will be ready. With polling, I continuously bother her, asking, "when will food be ready?" With events, I can ask her to tell (notify) me, when food is ready, and I'll either take a nap (sleep) or play a video game (process something else).David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com1tag:blogger.com,1999:blog-2321280567913257777.post-90491418341625339102010-07-23T07:45:00.000-07:002010-07-23T08:10:53.186-07:00The Headaches of Blocking SocketsIn Brunet, ACIS Lab's P2P software, we make heavy use of sockets, no doubt. Because UDP is more capable of traversing NATs than TCP, much of our focus has been on optimizing the UDP stack in the code. The concerns and issues we have had in using UDP include:<br /><ul><li>Safety of sending and receiving on a UDP socket at the same time</li><li>Do UDP sockets block</li><li>Can there be concurrent senders / receivers on a UDP socket at the same time</li></ul>On to the findings!<br /><ul><li>Safety of sending and receiving on a UDP socket at the same time</li></ul>The fear with sending and receiving at the same time is primarily based on lack of knowledge, the limited information provided on the Internet, and that you're dealing with what is presented as a single state. But let's think about it for a second, a socket is really like a pipe, well, rather two pipes, one for sending and one for receiving or better yet two queues, one queue waiting for the userspace to read the packet and another queue to send a packet across the network. Sure we multiplex the actual Ethernet device, well actually, we don't that's done by the OS. So why do we really need to be concerned about ordering when writing to a UDP socket? Maybe with a TCP socket, since that actually involves state, but UDP is stateless.<br /><br />As it turns out, one need not worry about reading and writing from a socket at the same time. The conclusion was reached based upon two pieces of evidence, empirical evidence arrived through experimentation and and reading the grand TCP/IP Illustrated. With just this alone, we were able to boost the speed of our system 10x, reducing code complexity, and improving idle time execution.<br /><ul><li> Do UDP sockets block</li></ul>Once upon a time, I was a young naive boy working on a P2P VPN called IPOP. IPOP was working so very well, but one day out of the blue IPOP would keep freezing up. What was to blame? Was it latent bugs that were showing themselves at an inopportune time or did we make a mistake somewhere in our logic? So I searched high and far, I even did inspection of packets to find which one was causing the freezing of the code. It turned out to be the sending of a packet across a transport, strange, I thought to myself. We have been using this code for years and I have never seen this issue!<br /><br />After some reading around, I discovered that I didn't know everything about UDP sockets, the worst of it was that UDP sockets actually have a send buffer!!! When the send buffer fills, it blocks! This is absolute nonsense, but what can you do? The only solution was to make UDP non-blocking somehow.<br /><br />The two choices were to do it in the main loop of the system or alternatively to put it into another thread. Well, we were already paying a lock penalty to prevent multiple writers at the same time, so maybe its safest to just put everything into another thread. We made that choice, because that way, we could provide a slightly larger user space buffer. Guess what... the system doesn't stall any more! Quick caution, though, sometimes that queue can become very large, so its important to limit its size. We chose the modest value of only allowing 256 messages be stored into the queue and it seems to be working well for a node that receives messages from over 50 remote nodes somewhat aggressively.<br /><ul><li>Can there be concurrent senders / receivers on a UDP socket at the same time</li></ul>This is actually a very good question, which I can only base upon reading and not much more. According to Unix Network Programming, another of Stevens books, you can at least have multiple readers, but we haven't tested it as of now, the receiver is in a single, dedicated, thread. <br />Sending is kind of confusing and unaddressed in what I've read so far, in the next paragraph, I describe one issue, Connect, which makes it seem somewhat conclusive why it wouldn't work. Though it would be interesting to see what if anything happens if we allowed multiples of each, but unfortunately, we haven't spent much time focusing on the performance scalability of our P2P system.<br /><br />At this point, I've addressed most of our concerns, I am still very interested in understanding better practices with sockets and how we could potentially improve our system. For example, one tidbit from Unix Network Programming was that a UDP socket actually calls Connect when sending a packet, apparently, this overhead is 1/3 of the actual time spent using the socket during a send. So if a user were to have a socket for each remote peer, performance could potentially be significantly better.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com1tag:blogger.com,1999:blog-2321280567913257777.post-1106455652289181112010-05-03T10:44:00.000-07:002010-05-03T11:25:27.866-07:00Returning to DTLSGosh was I ever wrong about DTLS, OpenSSL, and OpenSSL.NET. First some background, last spring, the project I had been working on, IPOP (IP over P2P) needed a security framework so that we could actually call it a VPN. So I did some research to determine, which security framework made the most sense. IPOP relies on unreliable paths, such as UDP and the P2P overlay, for communication. Thus a well accepted solution like SSL was out of the question, this led me to two possibilities IPsec and DTLS. I began reading as much as possible of the two frameworks, my conclusion was that DTLS was written for client-server applications and put more work on the server and it may require special logic to prevent duplicate connections if two peers attempt to create secure connections simulataneously. IPsec handles these cases, but is a beast and lacks, clean, portability across systems, thus supporting our VPN on various platforms would require special hooks or libraries for each system. Not very desirable, since we're a very small group. I settled creating a IPsec like implementation using only managed (C#) libraries that worked in Linux and Windows. Sometime late last year, I came to the realization that I am a PhD student and that this was probably not the best route, since my group lacked much knowledge regarding security systems. So I implemented a light-weight DTLS filter written in C#, but I never really spent much time figuring out how it fits in our P2P application.<br /><br />Flash forward to a couple weeks ago. I started wondering, what if any important questions could be answered by what I have done and what sort of evaluations I could perform to gain some insights. So I went back to work and finished my DTLS integration into the P2P system. It was not pleasant. This time through, I made sure that I understood every little detail that was occurring in the system and applied that knowledge to make the system as simple as possible. Errors in my previous notes were the following:<br /><br /><ol><li>DTLS does not handle cookies if you do not provide a generate and verify method</li><li>DTLS can have an assertion fail if certain packets are lost during the handshake</li><li>DTLS automatically handles retransmitting handshake messages</li></ol>Determining the cookie was somewhat painful, since I had implemented a filter and the data structures were crossing a managed / unmanaged layer. In examples, DTLS uses the socket IP/Port set and a secret number to generate a cookie, though I didn't have the IP/Port. Luckily, it turns out that the HashCode of the Ssl.Handle (int ManagedSsl) remained constant across callbacks, unlike Ssl.GetHashCode() and Marshal.ReadInt32(Ssl.IntPtr). The former changed, while the latter was the same for every call, go figure. Thus to generate and verify, I took a 20-byte random number appended the HashCode of the Ssl.Handle and performed a sha1 hash.<br />One thing I was not very happy with the cookie generation and verification, is that, it does create state, which I thought was the soul reason for having it in the first place. The memory impact on my system (without taking OpenSSL into context) remains the same whether or not cookie generation is enabled. It would be nice to verify that cookies are actually being used intelligently, but no work has verified this.<br /><br />For the life of me, I couldn't figure out why the handshake messages were failing. The code is very difficult to follow, but in the end, it appears that for some reason on the server side, when one of the timers expire, a few bytes appear ready for sending. The problem is, these bytes should never have been placed there. When performing a read on it, it causes an assertion that aborts the code. Let's hope an attacker doesn't figure out that pattern. Is anyone actually using OpenSSL DTLS in production level code? How did this not get detected earlier?<br /><br />On the path, I encountered a few other headaches, which I will simply list as the details aren't very exciting:<br /><ol><li>OpenSSL does not support unencrypted PKCS#8 Keys via the API if they are in DER mode, but PEM mode works fine.</li><li>If an SSL state switches from client to server, it ignores messages, the only solution is to create a new one from scratch. This is after executing clear, reset, accept, and resetting the bios.<br /></li><li>OpenSSL only supports a single PrivateKey / signed certificate per context (ssl_ctx), I guess that's fine for preshared keys, but it does limit the ability for a single user to obtain multiple certificates for different domains.</li></ol>The final component deals with determining a client vs a server. All messages are prefixed with a local and remote identifier (think port). A client knows his local port but the remote port hasn't been sent by a server. If a server receives a message without a remote port (or rather his local port being set), he knows its a message from a client. Thus he creates a new server side component to handle the request. When the client receives a packet back, it will see that the server has set his port and all future messages will contain the port pair. Thus packets from the client to the server are of the form [client][server][data] and from the server to the client are of the form [server][client][data]. This allows multiplexing a single data channel to support multiple secure channels.<br /><br />To summarize, I don't think DTLS is very well loved in the SSL community, there is literally like no documentation available for it. Most of what I learned, I did from the original paper, the RFCs, and looking at the code. I have attempted to communicate with the OpenSSL.NET guys to no avail. At this point, I have extended the system with a few hundred lines of code and will probably do a little more, since their current code needs to be compiled explicitly for Unix or Windows.<br /><br />Still to be worked out is the cost of having (potentially) multiple secure connections, methods of reducing the duplication of secure connections due to lack of a P2P handshake, and potentially investigating potential security issues in the approach, especially when performing the handshake over the P2P overlay.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com6tag:blogger.com,1999:blog-2321280567913257777.post-27739025601578056502010-03-01T21:07:00.001-08:002010-03-01T21:23:36.891-08:00ICMP Source QuenchAs with most PhD students, I have learned to run before crawling, meaning that I have worked on many systems in IP networking and P2P without having an expert knowledge in either field. To address this, I have begun a reading crusade beginning with a massive 1500 page book describing TCP: <a href="http://www.tcpipguide.com/">The TCP/IP Guide</a>.<br /><br />While reading it, I came across something that struck me... ICMP, or Internet control management protocol, has a feature known as <a href="http://en.wikipedia.org/wiki/ICMP_Source_Quench">source quench</a>. Basically, it tells the receiver of the packet to slow down, though it requires the first 8 bytes of a sent packet. In UDP this means the entire header (ports, length, and checksum), while in TCP this is a smaller portion of the header (ports and sequence number). What struck me as odd was that no where in the book did it describe how well deployed this feature was and potential attacks made by using it.<br /><br />So I turned to Google... and found <a href="http://www.gont.com.ar/drafts/icmp-attacks/draft-ietf-tcpm-icmp-attacks-01.txt">this lengthy document</a>. The basic summary was that most systems seem to have disabled TCP support for ICMP source quench, but what about UDP? I also found that it appears at least <a href="http://msdn.microsoft.com/en-us/library/aa917439.aspx">Windows CE</a> does support ICMP source quench.<br /><br />What interests me are, what OS' actually support this for UDP and TCP and do they verify UDP and TCP header mappings before reacting. Can users in a P2P system like Gnutella or BitTorrent who are downloading a file attack the sender of the data to slow down the rate at which he shares the files to others and thus improve their own bandwidth? I couldn't find any papers on Google or ACM, maybe this will be a neat little side trip for me.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-50929343683669051272009-12-09T11:39:00.000-08:002009-12-09T13:01:01.940-08:00Copy-on-Write using Ubuntu 9.10Recently, I've been working on getting a package that enables copy-on-write for <a href="http://www.ubuntu.com/">Ubuntu 9.10</a>, the use-case would be <a href="http://www.grid-appliance.org">Grid Appliance</a>, a project that I have been working on for the past 3 and half years. Previously, I had used <a href="http://www.debian.org">Debian</a>, created a special kernel, and messed around with the init scripts. During which, I experimented with both <a href="http://www.fsl.cs.sunysb.edu/project-unionfs.html">UnionFS</a> and <a href="http://aufs.sourceforge.net/">AUFS</a>, both of which required kernel patches. The former, UnionFS, was replaced with AUFS due to stability concerns.<br /><br />To begin my research, I first investigated the possibility of a union filesystem in the modern kernel. To my surprise, it seems that kernel maintainers are uninterested in this feature unless it can be implemented in a truly transparent method. Thus AUFS, most likely, will not be included in the kernel and Ubuntu has or plans to phase it out in the near future. UnionFS shares an even worse fate, as most developers have moved towards AUFS for stability reasons. Luckily, I found something called <a href="http://podgorny.cz/moin/UnionFsFuse">UnionFS-Fuse</a>, which implements the UnionFS system on top of <a href="http://fuse.sourceforge.net/">Fuse</a>, or file system in user-space as opposed to kernel file systems. I was first hesitant in this method, it occurred to me that Fuse ensures that I won't have to deal with kernel issues meaning that regardless of the direction taken by the kernel developers, I'll have a working union filesystem.<br /><br />The next step was figuring out where to insert the cow init script. It turns out that Ubuntu 9.10 uses <a href="http://upstart.ubuntu.com/">Upstart</a>, a relational, event-based, threaded init system. This replaces the old init system, based off of System V, which provides an explicit, in-order init system. Upstart poses a challenge for cow filesystems, as the first step in Upstart is to mount all the file systems by issuing the "startup" event. To get around this challenge, I've appended a "pre-start script" command that executes the cow init script prior to starting the mounting of the other file systems. This appears to work as I've set the virtual machine disk to rdonly and the system boot perfectly*.<br /><br />Yeah, it doesn't boot perfectly, because in Ubuntu 9.10 uses <a href="http://www.gnu.org/software/grub/grub-2.en.html">grub2</a> and checks whether or not the last startup sequence was ended successfully. This requires a non-trivial change that I am still figuring out. I can easily edit out the contents, but that won't work when an update occurs. I guess its time to start reading grub manipulation utilities to prevent that.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-27902667037356337962009-11-08T22:38:00.000-08:002009-11-08T23:06:24.698-08:00Working In And Around LicensesRecently our group has been confronted with the issue of how to interoperate with non-GPL compatible code. Previously, a <a href="http://boykin.acis.ufl.edu/">professor</a>, whom I work with, has been considering this issue along with how to allow GPL incompatible or more specifically closed source application to interoperate with our P2P library, <a href="http://boykin.acis.ufl.edu/wiki/index.php/Brunet">Brunet</a>. At the root of the problem is the "viral" nature of the GPL. Namely that all derived works must also be GPLed, see <a href="http://www.gnu.org/licenses/gpl.html">this</a>. Of particular interest in the GPLv2 is the definition of derived works and what consists of distribution. <br /><br />During a recent discussion, one thought was that we could use a binary or conversion interface to bypass this issue and another was that we could use sockets, both of which were shot down by <a href="http://lists.gpl-violations.org/pipermail/legal/2009-June/001611.html">this message</a>. Which made me wonder, how do other non-free applications link to GPL applications and what it falls under is distribution. Primarily, you cannot distribute non-GPL code that interacts with GPL code together, the intent is a derived work. Using a socket or a wrapper library effectively yields the same purpose, a circumvention of the GPL. Now what seems to be entirely possible is to define a binary interface or object interface licensed using a non-viral license, such as MIT/X11, use that interface to implement a GPL-incompatible application and distribute the two separately.<br /><br />Before I go any further, there are a couple important things to consider here, who owns the code at hand and what is their interpretation of the GPL. The owners appear to be UCLA, some students at UF, and primarily the professor introduced at the beginning of this blog. Our first goal is try to get UCLA to allow a relicense of the software into MIT/X11 or even relinquish their right over the software, at which point, we could keep that license wrapped up and then relicense the software publicly as needed. More importantly that would mean the GPL interpretation used would primarily be the professor's, thereafter, it would be distributors such as students and even Linux distributions such as Debian, Ubuntu, and Redhat.<br /><br />This leads to a more interesting question, what is generally accepted. Based upon the message I linked to <a href="">above</a>, it leads me to believe we either need to have a GPL friendly version or the application needs to work with out it. Furthermore, the application must be distributed separately and the GPL code cannot have auto-install or configuration scripts that bypass the GPL. Then users who install the non-GPL libraries would be not breaking the GPL in so long as they themselves do not distribute the set publicly. Of course, the auto-install / configure scripts could be part of the GPL-incompatible code.<br /><br />This leads me into a more interesting question, can we distribute these in a VM? Given that the GPL allows use of "OS libraries" be mixed together, both GPL and GPL-incompatible, I believe that a VM Appliance can have such applications bundled together.<br /><br />This all really boils down to what the intent of the GPL is and more specifically the license owners. My opinion is that no one wants to see their source code be eaten up and used without at least recognition and bug fixes being trickled into mainstream code. Closed source developers that desire stability would contribute both bug fixes and features that they deem in the critical path to the maintainers of Brunet, otherwise they would have to deal with maintaining complicated sets of patches.<br /><br />Taking a step back, can someone stop me from distributing a library or application that links to MIT/X11 code, I think in all cases, the answer is no. Can someone stop me from bundling said binaries especially in the case of GPL-incompatibility with a GPL application, I think the answer is clearly yes. I am very curious how others see this issue. Most importantly, I really look forward to understanding all the points of view in the group. It has become very apparent to me that everyone has an opinion on this and I wouldn't be surprised that we don't find out whether or not we went the right or wrong way until someone else tells us.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-5490489431000396922009-11-01T22:10:00.000-08:002009-11-01T23:08:50.534-08:00My Impressions of DTLS and OpenSSLSo after a weekend of fun with DTLS, OpenSSL, and OpenSSL.NET, I am not entirely onboard with the idea of using it as the backend for our P2P Security system. So let me list my pros and cons of using DTLS:<br /><br />Pros:<br /><ul><li>Maintained by someone else</li><li>FIPS certified</li><li>Availble for all platforms</li><li>Popular, well documented protocols</li></ul>Cons:<br /><ul><li>Lacks the level of documentation found for most of the other libraries I've used</li><li>DTLS is scarcely mentioned on the website and it is unclear to me how well it is actually maintained</li><li>DTLS (manual) renegotiation is broken</li><li>The library doesn't really make it easy to determine when a SSL session should be aborted</li><li>There is no async callbacks for when data is ready, which would be really helpful for handshake messages</li><li>Platform dependent (i.e. library -- ABI -- is not portable, API is)</li><li>The DTLSv1_methods lack the reliability layer that is actually in the DGRAM BIO<br /></li></ul>Given that there is an existing Security library in Brunet that actually handles the above cons lacking FIPS and support by others, it feels like I am between a rock and a hard place. My real goal was to get enough knowledge to add the details in my paper comparing my experiences with a standard DTLS implementation in comparison to a home brewed version. Since the transition to DTLS hasn't been entirely smooth sailing, I am a little hesitant to jump on board. I suppose two things need to be ensured to me, that renegotiation works and that messages are still transferable during renegotiation. While waiting, I may just test out what actually happens in systems that rely on DTLS, because I have a feeling that the libraries are significantly faster than .NET crypto libraries.<br /><br />Note: I stand corrected, the DTLS bug is fixed in version 1.0.0 betas, though with a new added twist of segmentation faults!David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com1tag:blogger.com,1999:blog-2321280567913257777.post-21304097373162337212009-10-30T18:26:00.000-07:002009-10-31T10:11:21.239-07:00OpenSSL-NetAfter my post yesterday, my next task was to do the same in .Net. Boy was I rewarded when I used Google to search for "OpenSSL Net". I was presented with <a href="https://sourceforge.net/projects/openssl-net/">"OpenSSL-NET"</a>, a managed wrapper for OpenSSL! Interestingly enough, this wasn't the first time I have looked for such a project. In the past, the most I found was people looking for the same. What a reward, I don't have to implement a C# wrapper for OpenSSL :). Now on to the good stuff, I took the work I did <a href="http://davidiw.blogspot.com/2009/10/memory-bios-and-openssl.html">yesterday</a> and converted into a C# object oriented sample. To reproduce my efforts, I have packaged all my work <a href="http://www.grid-appliance.org/files/other/dtls-net.zip">here</a>. For simplicity, I've included an OpenSSL-NET dll. One caveat, the current distribution of OpenSSL.NET won't work with the binary, I had to make a few tweaks, such as making the Ssl, SslContext, and some Enums linkable (i.e. internal to public) and then I added a couple of additional Ssl functions to the library. I've sent a patch to the manager and hopefully the changes get imported into the official library.<br /><br />What does that leave next? Figuring out how to distribute OpenSSL with our libraries for Windows support. Windows presents an interesting problem as it apparently lacks an up-to-date OpenSSL distribution. So with that in mind, I'll probably attempt to integrate our current security stack and have OpenSSL DTLS as an option instead. That way, users who just want a simple to install system don't have to worry about installing OpenSSL and configuring our application to properly link to it.<br /><br />Also another booger about OpenSSL is that the library names differ between Windows and Linux and there are a TON of functions called by the library. As such, the author has a library for Linux and Windows. One option to resolve this is to use a Windows version and created symbolic links in the local directory from the Linux ssl libraries to Windows ssl names.<br /><br />I'm really looking forward to removing dependency of our system from our own security stack. It means that we don't have to worry (as much) about something so critical. I don't regret writing the original security stack, because I learned SO much and much of the framework or at least the ideas will be kept from the transition of the old stack to the new. Also I suspect the transition from managed security code to unmanaged will create a much faster system.<br /><br />To assist in keeping track of my changes to the OpenSSL.NET code, I've put up a git repository at <a href="http://github.com/davidiw/OpenSSL.NET">github</a>.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com9tag:blogger.com,1999:blog-2321280567913257777.post-27286991448878812152009-10-29T10:23:00.000-07:002009-10-29T11:08:53.754-07:00Memory BIOS and OpenSSLI spent the past 12 hours or so learning how to "use" OpenSSL with the goal of potentially using it to replace our existing security system. Why? Because security is hard, there are so many ridiculous attacks that can be made that probably have no real effect on anything, but it makes people nervous, like timing attacks that have the ability to decrypt the first few messages in a symmetric exchange. Research is ongoing to discover ways to break security and at the same time to improve protocols and algorithms. I think its all great, but I don't have time to worry about it. Its hard argument to make that creating a new system that takes the best parts of DTLS and IPsec is really the best pathway, if you can mess around with a well supported DTLS implementation to create an IPsec like system. While I didn't find something identical to that, I found that guys over in <a href="http://campagnol.sourceforge.net/">France</a> have implemented a "P2P" VPN that uses DTLS in a non-standard way. So I suppose it is possible. With the remaining text in this entry, I'll discuss my recent experiment with OpenSSL and DTLS, which was rather quite interesting since I haven't coded in C/C++ in 4.5 years now :).<br /><br />First off, OpenSSL documentation is horrible, it is spare and spread across the web. Most of the error messages don't really make sense. The requirements and order of operations for creating secure sessions aren't very clear. I used a mixture of sample code and their man pages to figure out what to do. Though there is really little text on DTLS. It turns out for our local testing purpose, it really doesn't matter which security protocol you use.<br /><br />So onwards to how this whole thing works, we'll assume that you know how to make a self-signed certificate PEM file with an RSA PEM key file, which can be located in "server.pem". Now we can get on to the code, here's where ordering matters. Lesson 1) complete all operations with structures before moving on to others that are created from them.<br /><br />To keep everything organized, and because I don't know SSL well enough, I created a structure so I can access everything I create:<br /><pre name="code" class="C"><br />typedef struct connection {<br />BIO* read;<br />BIO* write;<br />SSL* ssl;<br />SSL_CTX* ctx;<br />} connection;<br /></pre><br />The bios are the memory bios. The names for read and write are from the perspective of the SSL structure, i.e., read is used to write network traffic from, while write is used to read network traffic. The rest will become clear in the later parts of this post.<br /><br />So here's how to initialize a server (minimized version):<br /><pre><br />connection *con = malloc(sizeof(connection));<br />con->ctx = SSL_CTX_new(DTLSv1_method());<br /> SSL_CTX_use_certificate_chain_file(con->ctx, "server.pem");<br />SSL_CTX_use_PrivateKey_file(con->ctx, "server.pem", SSL_FILETYPE_PEM);<br />SSL_CTX_use_certificate_file(con->ctx, "server.pem", SSL_FILETYPE_PEM);<br />SSL_CTX_set_options(con->ctx, SSL_OP_SINGLE_DH_USE);<br />SSL_CTX_set_session_id_context(con->ctx, sid, 4);<br /><br />con->ssl = SSL_new(con->ctx);<br />SSL_set_accept_state(con->ssl);<br />SSL_set_verify(con->ssl, SSL_VERIFY_NONE, NULL);<br />SSL_set_session_id_context(con->ssl, sid, 4);<br /><br />con->read = BIO_new(BIO_s_mem());<br />BIO_set_nbio(con->read, 1);<br />con->write = BIO_new(BIO_s_mem());<br />BIO_set_nbio(con->write, 1);<br />SSL_set_bio(con->ssl, con->read, con->write);<br /></pre><br />So what's interesting here? The most annoying issue was the ordering. You need to initialize the CTX with all those options before creating SSL or the SSL won't have the CTX options. That vexed me for an entire night, while I slept for 2 hours. The first two CTX options are clear, we need a certificate and a private key. The following option, SSL_CTX_set_session_id_context, needs to be set to some unique value, apparently not setting could cause a certificate exchange to fail, though I haven't experienced that issue. My favorite was the use of Diffie-Hellman, where I come from, the Diffie-Hellman exchange was handled in the background unless you really want to get in and dirty, not with OpenSSL, you either need to specify a Diffie-Hellman key file, a callback, or use the single use flag. Now we can create the server and since it is a server, we set it up for an accept state. The final components are the creation of memory bios, so that we can abstract the sending mechanism. Memory bios might be non-blocking, but it was explicit anywhere, it works if you don't make them non-blocking, but why risk it? Wow, looking back, that was kind of easy!<br /><br />The client is even easier:<br /><pre><br />connection *con = malloc(sizeof(connection));<br />con->ctx = SSL_CTX_new(DTLSv1_method());<br />SSL_CTX_use_certificate_chain_file(con->ctx, "server.pem");<br />con->ssl = SSL_new(con->ctx);<br />SSL_set_connect_state(con->ssl);<br />SSL_set_verify(con->ssl, SSL_VERIFY_NONE, NULL);<br /><br />con->read = BIO_new(BIO_s_mem());<br />BIO_set_nbio(con->read, 1);<br />con->write = BIO_new(BIO_s_mem());<br />BIO_set_nbio(con->write, 1);<br />SSL_set_bio(con->ssl, con->read, con->write);<br /></pre><br />We start with the same initialization of the CTX block and then for the SSL structure we set it to connect state. The rest is the same as the server.<br /><br />So now we have usable client and server ssl structure, we need to do some sending between the two, that looks like this:<br /><pre><br />SSL_do_handshake(client->ssl);<br />handle_send(client, server);<br />handle_send(server, client);<br />handle_send(client, server);<br />handle_send(server, client);<br /></pre><br />The system needs to be started, so we start with a SSL_do_handshake on the client ssl, this causes the client to attempt to connect with the server. We use handle_send to facilitate this and abstract it from this component. After 2 round trips, both the client and server have authenticated.<br /><br />Here's how handle_send works (minimized):<br /><pre><br />void handle_send(connection* sender, connection* receiver)<br />{<br />char buffer[1024];<br />int read = BIO_read(sender->write, buffer, 1024);<br />int written = read > 0 ? BIO_write(receiver->read, buffer, read) : -1;<br /><br />if(written > 0) {<br /> if(!SSL_is_init_finished(receiver->ssl)) {<br /> SSL_do_handshake(receiver->ssl);<br /> } else {<br /> read = SSL_read(receiver->ssl, buffer, 1024);<br /> printf("Incoming message: %s", buffer);<br /> }<br />}<br /></pre><br />As mentioned above, the write bio contains encrypted data, which can be injected into another read bio to insert data into the SSL structure. If the init is finished and successful and we have data waiting, it most likely is a data message. We can use SSL_read to retrieve the decrypted data.<br /><br />The last part is the fun part, where we can actually see a message go across:<br /><pre><br />strcpy(buffer, "HelloWorld!");<br /><br />SSL_write(client->ssl, buffer, strlen(buffer));<br />handle_send(client, server);<br /><br />SSL_write(server->ssl, buffer, strlen(buffer));<br />handle_send(server, client);<br /></pre><br />This causes both the client and server to send to each other the message "HelloWorld!", which will be printed to the screen. SSL_write is SSL_read's counterpart. It is used to insert unencrypted data into the SSL structure so that it can be read from the write bio as encrypted data.<br /><br />Now we just need to clean up:<br /><pre><br />SSL_free(con->ssl);<br />SSL_CTX_free(con->ctx);<br />free(con);<br /></pre><br />There you have it, a clear and concise way to use memory bios! The full source code can be downloaded from <a href="http://www.grid-appliance/files/other/dtls_memory.zip">here</a>. Due to the girth of the code, the above code is signficantly minimized and not meant to be used as is. For example, functions were squashed and error handling was removed.<br /><br />So I guess the answer is, yes SSL can be used as a part of an IPsec like system, if you're willing to implement the protocol handling for multiple and parallel sessions. The only other issue that may need to be addressed is the loss of fragmentation and resends due to not using the DGRAM bio, but that's furture work.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com10tag:blogger.com,1999:blog-2321280567913257777.post-1841121465424956532009-08-03T06:06:00.000-07:002009-08-03T06:26:34.785-07:00Reassembling the PDUWhile I haven't written much lately, it has nothing to do with lack of productivity but rather too much on my plate. Since I've last updated I have:<br /><ol><li>Manuevered through the craziness of shepherded papers</li><li>Put together an elite team of peers to develop a NSDI 10 submission</li><li>Began a P2P student discussion group</li><li>Mentored a high school student</li><li>Organized an Archer trip that will have me visiting 3 schools in September</li><li>Started to learn how to hold my tongue when using it would bring no lasting benefit</li><li>Learning to deal with significant disappointment</li></ol>Academically, I'm not quite where I need to be. There is much work to be done and not enough time to do it all. I often spread myself too thin by taking on far too many tasks for many people. Over the next month, as these tasks complete, I will need to refocus on my needs and focus on graduation.<br /><br />The biggest thing I've contemplated as of late is the need of allies or at least peers and elders that will speak well of you. One person that I can look to that has some interesting level of respect is my advisor. He rarely speaks, yet people seem to go out of their way to talk with him. Clearly he is at an advantage, since he gets to remain his quiet self and does not have to become overly talkative to obtain respect. There is probably a lot more to the equation than what I can see, but I feel that it largely falls under respecting the other persons work and their knowledge.<br /><br />In terms of research, I am in the heat of the NSDI work and have hit a few milestones, but due to events outside of my control, I will need to redouble my effort. The individuals that I am working with are steadfast, and I am sure that their contributions will make the breadeth of the paper amazing. I think it will be a very good read and informative for anyone that is interested in VPN technologies. The only tricky part about writing such a work is that there are many fields that are interested in similar virtual networks (communication) and thus there is no doubt going to be overlap that will not easily be identifiable. I think the only way to really deal with this issue is to make sure that the committee members who may potentially review the paper are appropriately commended for their previous work as it relates to the paper. I will admit though, that I am scared with a recent added burden that I may have too much on my hands to satisfactorily complete the paper. Perhaps my advisor or a fellow student will go above and beyond what previous coauthors have.David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-4169628847164383002009-06-17T07:56:00.000-07:002009-06-17T08:01:27.961-07:00First conference level publicationJust the other day, I got some excellent news: My publication entitled "On the Design of Scalable, Self-Configuring Virtual Networks" was accepted into SuperComputing 09 with an acceptance rate of ~22.6%. Sadly it was shepherded, so I have to do some tweaks to it, but hopefully it won't be too painful. The only thing that may make it painful, is that all I have left is a tex file and none of the original images from the submission. The paper can be found on the sc09 web page:<br />http://scyourway.nacse.org/conference/view/pap152David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-71087762456529896452009-06-09T19:33:00.000-07:002009-06-09T20:13:33.034-07:00Writing Application Entry PointsA long time ago, I spent a few days working on an option parser for C# tailored around my application, SimpleNode. The name was quite misleading as SimpleNode allowed users to make many nodes, act as user interface into the system, and probably a handful of other behaviors that have long since escaped me. It was largely unwieldly and probably not horribly useful as a learning tool nor very reusable. So I chopped everything out and went with thin applications that were tailored around objects that had simple, specific tasks. This allowed better integration at the cost of dumbing down the command-line interface and having multiple applications that had very similar behavior.<br /><br />I typically desire to do things right the first time and to not come back later. When I don't get things quite right, I acknowledge the problem but add it to my priority queue, so I may or may not ever get back to it. Recently, we've had an increase of interest in our system along with the a stage 1 completion of my groupvpn, so I felt it fitting to improve the usability of our project. Lucky for me, <a href="http://blog.xujiangyan.net/">one my peers</a> in the lab is very interested in technologies and was able to point out <a href="http://www.ndesk.org/Options">NDesk.Options</a>. I think the <a href="http://github.com/davidiw/brunet/blob/6739d96d6691a00024a99a7358a8effb45800529/src/BasicNode/Main.cs">results</a> are pretty good. What displeases me is that this sort of interface is probably pretty generic and so if a user wants a similar frontend, they'd have to copy and paste 100 lines of code to get the same thing, which may be ridiculous if what they do only requires a few lines, like <a href="http://github.com/davidiw/brunet/blob/6739d96d6691a00024a99a7358a8effb45800529/src/BasicNode/Examples/HelloWorldNodeDataHandler.cs">this</a>.<br /><br />Originally, the posts title was going to be "Polymorphic Entry Points." I'd love to have something like that. I guess this can be achieved by adding helper methods to each class we instantiate that requires more than 5 lines, but I feel there needs to be a clear distinction between application entry point and classes as I think it makes it easier for users to understand how everything fits together. Maybe I'm wrong and I'm probably over thinking this :).David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com0tag:blogger.com,1999:blog-2321280567913257777.post-63000185567025145032009-06-04T11:21:00.000-07:002009-06-04T11:36:47.544-07:00Dissertation Step 1Main focus point: P2P VPN architecture with emphasis on scalablity, user-friendliness, and performance.<br /><br />Scalability:<br /><ol><li>Effects of latency on VPN connectivity</li><li>Optimal peer count, cost per peer connection</li><li>Trusting a P2P infrastructure</li><li>VPN Router model<br /></li></ol>User-friendliness:<br /><ol><li>Ability to deploy and manage large, distributed infrastructures</li><li>GroupVPN - ability to use groups to manage VPN connectivity</li><li>Userspace based, non-root VPN solution<br /></li></ol>Performance:<br /><ol><li>Use of proximity neighbor and shortcut selection</li><li>Hybrid-based VPN model</li><li>Inverted P2P VPN model</li></ol>David Wolinskyhttp://www.blogger.com/profile/06477494531387414917noreply@blogger.com11