Flash forward to a couple weeks ago. I started wondering, what if any important questions could be answered by what I have done and what sort of evaluations I could perform to gain some insights. So I went back to work and finished my DTLS integration into the P2P system. It was not pleasant. This time through, I made sure that I understood every little detail that was occurring in the system and applied that knowledge to make the system as simple as possible. Errors in my previous notes were the following:
- DTLS does not handle cookies if you do not provide a generate and verify method
- DTLS can have an assertion fail if certain packets are lost during the handshake
- DTLS automatically handles retransmitting handshake messages
One thing I was not very happy with the cookie generation and verification, is that, it does create state, which I thought was the soul reason for having it in the first place. The memory impact on my system (without taking OpenSSL into context) remains the same whether or not cookie generation is enabled. It would be nice to verify that cookies are actually being used intelligently, but no work has verified this.
For the life of me, I couldn't figure out why the handshake messages were failing. The code is very difficult to follow, but in the end, it appears that for some reason on the server side, when one of the timers expire, a few bytes appear ready for sending. The problem is, these bytes should never have been placed there. When performing a read on it, it causes an assertion that aborts the code. Let's hope an attacker doesn't figure out that pattern. Is anyone actually using OpenSSL DTLS in production level code? How did this not get detected earlier?
On the path, I encountered a few other headaches, which I will simply list as the details aren't very exciting:
- OpenSSL does not support unencrypted PKCS#8 Keys via the API if they are in DER mode, but PEM mode works fine.
- If an SSL state switches from client to server, it ignores messages, the only solution is to create a new one from scratch. This is after executing clear, reset, accept, and resetting the bios.
- OpenSSL only supports a single PrivateKey / signed certificate per context (ssl_ctx), I guess that's fine for preshared keys, but it does limit the ability for a single user to obtain multiple certificates for different domains.
To summarize, I don't think DTLS is very well loved in the SSL community, there is literally like no documentation available for it. Most of what I learned, I did from the original paper, the RFCs, and looking at the code. I have attempted to communicate with the OpenSSL.NET guys to no avail. At this point, I have extended the system with a few hundred lines of code and will probably do a little more, since their current code needs to be compiled explicitly for Unix or Windows.
Still to be worked out is the cost of having (potentially) multiple secure connections, methods of reducing the duplication of secure connections due to lack of a P2P handshake, and potentially investigating potential security issues in the approach, especially when performing the handshake over the P2P overlay.