Memory BIOS and OpenSSL

I spent the past 12 hours or so learning how to "use" OpenSSL with the goal of potentially using it to replace our existing security system. Why? Because security is hard, there are so many ridiculous attacks that can be made that probably have no real effect on anything, but it makes people nervous, like timing attacks that have the ability to decrypt the first few messages in a symmetric exchange. Research is ongoing to discover ways to break security and at the same time to improve protocols and algorithms. I think its all great, but I don't have time to worry about it. Its hard argument to make that creating a new system that takes the best parts of DTLS and IPsec is really the best pathway, if you can mess around with a well supported DTLS implementation to create an IPsec like system. While I didn't find something identical to that, I found that guys over in France have implemented a "P2P" VPN that uses DTLS in a non-standard way. So I suppose it is possible. With the remaining text in this entry, I'll discuss my recent experiment with OpenSSL and DTLS, which was rather quite interesting since I haven't coded in C/C++ in 4.5 years now :).

First off, OpenSSL documentation is horrible, it is spare and spread across the web. Most of the error messages don't really make sense. The requirements and order of operations for creating secure sessions aren't very clear. I used a mixture of sample code and their man pages to figure out what to do. Though there is really little text on DTLS. It turns out for our local testing purpose, it really doesn't matter which security protocol you use.

So onwards to how this whole thing works, we'll assume that you know how to make a self-signed certificate PEM file with an RSA PEM key file, which can be located in "server.pem". Now we can get on to the code, here's where ordering matters. Lesson 1) complete all operations with structures before moving on to others that are created from them.

To keep everything organized, and because I don't know SSL well enough, I created a structure so I can access everything I create:

typedef struct connection {
BIO* read;
BIO* write;
SSL* ssl;
SSL_CTX* ctx;
} connection;

The bios are the memory bios. The names for read and write are from the perspective of the SSL structure, i.e., read is used to write network traffic from, while write is used to read network traffic. The rest will become clear in the later parts of this post.

So here's how to initialize a server (minimized version):

connection *con = malloc(sizeof(connection));
con->ctx = SSL_CTX_new(DTLSv1_method());
 SSL_CTX_use_certificate_chain_file(con->ctx, "server.pem");
SSL_CTX_use_PrivateKey_file(con->ctx, "server.pem", SSL_FILETYPE_PEM);
SSL_CTX_use_certificate_file(con->ctx, "server.pem", SSL_FILETYPE_PEM);
SSL_CTX_set_options(con->ctx, SSL_OP_SINGLE_DH_USE);
SSL_CTX_set_session_id_context(con->ctx, sid, 4);

con->ssl = SSL_new(con->ctx);
SSL_set_accept_state(con->ssl);
SSL_set_verify(con->ssl, SSL_VERIFY_NONE, NULL);
SSL_set_session_id_context(con->ssl, sid, 4);

con->read = BIO_new(BIO_s_mem());
BIO_set_nbio(con->read, 1);
con->write = BIO_new(BIO_s_mem());
BIO_set_nbio(con->write, 1);
SSL_set_bio(con->ssl, con->read, con->write);

So what's interesting here? The most annoying issue was the ordering. You need to initialize the CTX with all those options before creating SSL or the SSL won't have the CTX options. That vexed me for an entire night, while I slept for 2 hours. The first two CTX options are clear, we need a certificate and a private key. The following option, SSL_CTX_set_session_id_context, needs to be set to some unique value, apparently not setting could cause a certificate exchange to fail, though I haven't experienced that issue. My favorite was the use of Diffie-Hellman, where I come from, the Diffie-Hellman exchange was handled in the background unless you really want to get in and dirty, not with OpenSSL, you either need to specify a Diffie-Hellman key file, a callback, or use the single use flag. Now we can create the server and since it is a server, we set it up for an accept state. The final components are the creation of memory bios, so that we can abstract the sending mechanism. Memory bios might be non-blocking, but it was explicit anywhere, it works if you don't make them non-blocking, but why risk it? Wow, looking back, that was kind of easy!

The client is even easier:

connection *con = malloc(sizeof(connection));
con->ctx = SSL_CTX_new(DTLSv1_method());
SSL_CTX_use_certificate_chain_file(con->ctx, "server.pem");
con->ssl = SSL_new(con->ctx);
SSL_set_connect_state(con->ssl);
SSL_set_verify(con->ssl, SSL_VERIFY_NONE, NULL);

con->read = BIO_new(BIO_s_mem());
BIO_set_nbio(con->read, 1);
con->write = BIO_new(BIO_s_mem());
BIO_set_nbio(con->write, 1);
SSL_set_bio(con->ssl, con->read, con->write);

We start with the same initialization of the CTX block and then for the SSL structure we set it to connect state. The rest is the same as the server.

So now we have usable client and server ssl structure, we need to do some sending between the two, that looks like this:

SSL_do_handshake(client->ssl);
handle_send(client, server);
handle_send(server, client);
handle_send(client, server);
handle_send(server, client);

The system needs to be started, so we start with a SSL_do_handshake on the client ssl, this causes the client to attempt to connect with the server. We use handle_send to facilitate this and abstract it from this component. After 2 round trips, both the client and server have authenticated.

Here's how handle_send works (minimized):

void handle_send(connection* sender, connection* receiver)
{
char buffer[1024];
int read = BIO_read(sender->write, buffer, 1024);
int written = read > 0 ? BIO_write(receiver->read, buffer, read) : -1;

if(written > 0) {
  if(!SSL_is_init_finished(receiver->ssl)) {
     SSL_do_handshake(receiver->ssl);
  } else {
    read = SSL_read(receiver->ssl, buffer, 1024);
    printf("Incoming message: %s", buffer);
  }
}

As mentioned above, the write bio contains encrypted data, which can be injected into another read bio to insert data into the SSL structure. If the init is finished and successful and we have data waiting, it most likely is a data message. We can use SSL_read to retrieve the decrypted data.

The last part is the fun part, where we can actually see a message go across:

strcpy(buffer, "HelloWorld!");

SSL_write(client->ssl, buffer, strlen(buffer));
handle_send(client, server);

SSL_write(server->ssl, buffer, strlen(buffer));
handle_send(server, client);

This causes both the client and server to send to each other the message "HelloWorld!", which will be printed to the screen. SSL_write is SSL_read's counterpart. It is used to insert unencrypted data into the SSL structure so that it can be read from the write bio as encrypted data.

Now we just need to clean up:

SSL_free(con->ssl);
SSL_CTX_free(con->ctx);
free(con);

There you have it, a clear and concise way to use memory bios! The full source code can be downloaded from here. Due to the girth of the code, the above code is signficantly minimized and not meant to be used as is. For example, functions were squashed and error handling was removed.

So I guess the answer is, yes SSL can be used as a part of an IPsec like system, if you're willing to implement the protocol handling for multiple and parallel sessions. The only other issue that may need to be addressed is the loss of fragmentation and resends due to not using the DGRAM bio, but that's furture work.