Got it... I think

Quick one here. I just realized, or made the decision, whatever. That I probably shouldn't try to insert the DANE check into the OpenSSL certificate checking path.

Instead what I'll do is I will add a separate file that has the DANE stuff and make it up to the application developer to run the DANE check as well, after the SSL handshake has happened.

That sure as hell made things neater. :)

More on the ingress point...

Found this article as well, and following that, I see that I should probably get in somewhere in the BIO_ piece of it. Somewhere in the vicinity of this code, or rather the library functions that are being called here.

bio = BIO_new_ssl_connect(ctx);
BIO_get_ssl(bio, & ssl);
SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);

and

/* Attempt to connect */
BIO_set_conn_hostname(bio, "hostname:port");

/* Verify the connection opened and perform the handshake */
if(BIO_do_connect(bio) <= 0) {
/* Handle failed connection */
}

if(SSL_get_verify_result(ssl) != X509_V_OK) {
/* Handle the failed verification */
}

The object bio should hold all that I need after the call to BIO_do_connect, as I believe that's where the TLS handshake takes place. Since that's probably the last function call that I know will be called by the client application, it seems that I should try to insert my code there.

Setting up the connection is done by functions in the crypto/bio/bss_conn.c file, but it's not clear to me where (or even if) it's validating the certificate. I don't think it is, it seems to handle the connection only.

Also was checking out the file ssl/s3_clnt.c - severely dense, but might be useful from a learning perspective.

Now time to sleep.

Looking for literature

That kind of sounded a little bit serious, but actually, it IS what I'm doing. I'm still struggling a little bit to grasp where I'm supposed to tie myself in but at least I'm starting to get a handle on how TLS works from the client point of view. Received a book from Amazon yesterday that's proving itself to be very helpful. "Implementing SSL/TLS" by Joshua Davies.

Also Googled a bit more and found a good set of articles written by Eric Rescorla, who I believe came up with SSL in the first place. It's a two-piece thing, can be found here and the second part here.

Trying to figure out where the ingress point will be

So one of my first problems is to figure out where I'm supposed to insert my work, which will do a DANE lookup and pass/fail the connection. The DANE lookup requires the connection's protocol and port so that a DNS lookup like:

_443._tcp.www.example.com

can be made. It seems that a good candidate for inserting my work will be somewhere south of BIO_s_connect(3). This specifically should give me access to the port number, whilst for the time being I think that the protocol can be assumed to be TCP.

One thing I need to validate though is that the BIO_* framework is actually used being used, and preferably that I can reasonably easily find a client implemented with that framework that I can trick into using a DANE-capable OpenSSL. So far my reference is this article.

More tk.

Edit: Just found this article that outlines how to write a client that uses BIO.

Getting started

Tonight I've downloaded and installed XCode. Now trying to build openssl but getting errors:

Undefined symbols:
"_ENGINE_load_gost", referenced from:
_ENGINE_load_builtin_engines in libcrypto.a(eng_all.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
make[2]: *** [link_app.] Error 1
make[1]: *** [openssl] Error 2
make: *** [build_apps] Error 1