Returning to DTLS

Gosh was I ever wrong about DTLS, OpenSSL, and OpenSSL.NET. First some background, last spring, the project I had been working on, IPOP (IP over P2P) needed a security framework so that we could actually call it a VPN. So I did some research to determine, which security framework made the most sense. IPOP relies on unreliable paths, such as UDP and the P2P overlay, for communication. Thus a well accepted solution like SSL was out of the question, this led me to two possibilities IPsec and DTLS. I began reading as much as possible of the two frameworks, my conclusion was that DTLS was written for client-server applications and put more work on the server and it may require special logic to prevent duplicate connections if two peers attempt to create secure connections simulataneously. IPsec handles these cases, but is a beast and lacks, clean, portability across systems, thus supporting our VPN on various platforms would require special hooks or libraries for each system. Not very desirable, since we're a very small group. I settled creating a IPsec like implementation using only managed (C#) libraries that worked in Linux and Windows. Sometime late last year, I came to the realization that I am a PhD student and that this was probably not the best route, since my group lacked much knowledge regarding security systems. So I implemented a light-weight DTLS filter written in C#, but I never really spent much time figuring out how it fits in our P2P application.

Flash forward to a couple weeks ago. I started wondering, what if any important questions could be answered by what I have done and what sort of evaluations I could perform to gain some insights. So I went back to work and finished my DTLS integration into the P2P system. It was not pleasant. This time through, I made sure that I understood every little detail that was occurring in the system and applied that knowledge to make the system as simple as possible. Errors in my previous notes were the following:

1. DTLS does not handle cookies if you do not provide a generate and verify method
2. DTLS can have an assertion fail if certain packets are lost during the handshake
3. DTLS automatically handles retransmitting handshake messages

Determining the cookie was somewhat painful, since I had implemented a filter and the data structures were crossing a managed / unmanaged layer. In examples, DTLS uses the socket IP/Port set and a secret number to generate a cookie, though I didn't have the IP/Port. Luckily, it turns out that the HashCode of the Ssl.Handle (int ManagedSsl) remained constant across callbacks, unlike Ssl.GetHashCode() and Marshal.ReadInt32(Ssl.IntPtr). The former changed, while the latter was the same for every call, go figure. Thus to generate and verify, I took a 20-byte random number appended the HashCode of the Ssl.Handle and performed a sha1 hash.
One thing I was not very happy with the cookie generation and verification, is that, it does create state, which I thought was the soul reason for having it in the first place. The memory impact on my system (without taking OpenSSL into context) remains the same whether or not cookie generation is enabled. It would be nice to verify that cookies are actually being used intelligently, but no work has verified this.

For the life of me, I couldn't figure out why the handshake messages were failing. The code is very difficult to follow, but in the end, it appears that for some reason on the server side, when one of the timers expire, a few bytes appear ready for sending. The problem is, these bytes should never have been placed there. When performing a read on it, it causes an assertion that aborts the code. Let's hope an attacker doesn't figure out that pattern. Is anyone actually using OpenSSL DTLS in production level code? How did this not get detected earlier?

On the path, I encountered a few other headaches, which I will simply list as the details aren't very exciting:

1. OpenSSL does not support unencrypted PKCS#8 Keys via the API if they are in DER mode, but PEM mode works fine.
2. If an SSL state switches from client to server, it ignores messages, the only solution is to create a new one from scratch. This is after executing clear, reset, accept, and resetting the bios.
   
3. OpenSSL only supports a single PrivateKey / signed certificate per context (ssl_ctx), I guess that's fine for preshared keys, but it does limit the ability for a single user to obtain multiple certificates for different domains.

The final component deals with determining a client vs a server. All messages are prefixed with a local and remote identifier (think port). A client knows his local port but the remote port hasn't been sent by a server. If a server receives a message without a remote port (or rather his local port being set), he knows its a message from a client. Thus he creates a new server side component to handle the request. When the client receives a packet back, it will see that the server has set his port and all future messages will contain the port pair. Thus packets from the client to the server are of the form [client][server][data] and from the server to the client are of the form [server][client][data]. This allows multiplexing a single data channel to support multiple secure channels.

To summarize, I don't think DTLS is very well loved in the SSL community, there is literally like no documentation available for it. Most of what I learned, I did from the original paper, the RFCs, and looking at the code. I have attempted to communicate with the OpenSSL.NET guys to no avail. At this point, I have extended the system with a few hundred lines of code and will probably do a little more, since their current code needs to be compiled explicitly for Unix or Windows.

Still to be worked out is the cost of having (potentially) multiple secure connections, methods of reducing the duplication of secure connections due to lack of a P2P handshake, and potentially investigating potential security issues in the approach, especially when performing the handshake over the P2P overlay.