Not sure the trade off line has been drawn in the correct place here. TLS should be implemented as a filter; the world isn't limited to files and (blocking) sockets. What about integration with event loops, for example? How does one test this API?
Indeed. Functions like tls_connect and tls_connect_fds don't belong at this level. A TLS library should do only TLS, and behave like a black box, from which you pull data and into which you can push data (in a non-blocking way).
Mixing in OS-related stuff only complicates the API.
It complicates the implementation, but I'd argue it simplifies the API... one of the things you want to get absolutely right is the binding between hostname and server certificate. If you separate the two, there's a good chance that the TLS code won't even know what hostname it's supposed to be talking to (and I think I've heard of two codebases that required some refactoring for the hostname to get to the right place). A tls_connect that takes a string forces you to do the right thing, and the primary design goal I want out of a TLS API is that it should force me to do the right thing.
But, I don't want to do TLS. I'm not in the encryption business. Instead, I just want to send data across the network.
The OS used to do the networking for me. It exposes the BSD sockets API, which is pretty simple and nice but is unfortunately no longer safe to use. Yet, the kernel doesn't offer anything else.
So you're right, but you don't go far enough. It should all be in the kernel. Instead of tls_config_set_cert_file() we should be using setsockopt(2). Until we can, however, we'll have to rely on user-space libraries.
> It exposes the BSD sockets API, which is pretty simple and nice but is unfortunately no longer safe to use.
This is the second time you've said something like this today. Explain what you mean by it.
> So you're right, but you don't go far enough. It should all be in the kernel. Instead of tls_config_set_cert_file() we should be using setsockopt(2). Until we can, however, we'll have to rely on user-space libraries.
I think what you mean is that it should be in the standard library. There is no reason it should have to be in the kernel.
1) Optimizing for the most common use case. 2) You can create file descriptors not backed by any actual network connection or file on the harddrive (man 2 pipe and man 2 socketpair), you can still build an API on top of that which operates purely on buffers.
Yes, you can, if your goal is massive bloat. If you want to have reasonable performance and scalability, a blocking API (that is to say, at least one OS thread per connection) and loads of IPC syscall overhead is about the worst thing you could do.
Why not both? A low level API where you create a context, then give it [clear|cipher]text and it gives you [cipher|clear]text, and and API that builds on top of it to provide socket communication? I don't see why the two are mutually exclusive. Now, your complaint about event loop integration is very valid. You should be able to use select/epoll/kqueue/etc. on TLS sockets without having to worry about what your libssl is doing.
I wrote exactly one program that uses libssl in C directly. It is a special-purpose web server that can serve files or stream content over an HTTPS connection [1]. It was a huge pain to get my event loop to play nice with TLS because TLS does more reads/writes where a plain HTTP connection would do just reads or just writes. I also had to enable various non-defaults (SSL_MODE_ENABLE_PARTIAL_WRITE | SSL_MODE_RELEASE_BUFFERS) to get it to behave more like a regular socket. Having a more standard way to support stuff like this would be great, and it does exist in other places. For example, Python wraps libssl in such a way that you can just wrap a plain socket into a TLS socket, and it pretty much just works. It'd be great if stuff like this existed at a lower level where this new API seems to live.
I'm not sure there's only one line to be drawn. There is absolutely a place for an API like this -- but that place is not as a replacement for OpenSSL.
The world has changed. We must now consider send(2) and recv(2) to be toxic. They are fundamentally broken and cannot be used safely. The problem is, what do we use in their place? Simple sockets should continue to be simple to use, even now that we need to layer encryption on top. OpenSSL is not a practical replacement for those functions. This new API is a pretty good attempt. It may not do everything that OpenSSL does, but it doesn't need to. It only needs to make the simple things easy. If you have a different use case, you should use a different high-level API.
I think the way to understand this API is to see it as a "view", a wrapper, an abstraction. It's like a database view that hides the dozen tricky joins necessary to answer a frequently asked question. Also, like a database view, it is one of potentially many, each with a different purpose.
As a wrapper, this API exposes only a subset of the full functionality, but it's a very useful subset. Perhaps 80% of the complexity of the OpenSSL API comes from only 20% of use cases. This API exposes only a small subset of functionality, but that's what makes it simple and easy to use and it still covers 80% of use cases. If you aren't in that 80%, then this API isn't for you. But that's ok!
It seems reasonable to expect multiple layers of cryptography libraries. We ought to have a set of low-level libraries that implement the fundamental primitives in a general and orthogonal way. It would be a sort of cryptography "kernel". Most developers wouldn't use it in the same way that most don't directly make syscalls into the OS kernel, but instead use a high-lever wrapper library. To open a file, for example, we often use a wrapper like the functions in a libc, or a wrapper of a wrapper like libsqlite3.
So, if an API doesn't cover your particular use case, that doesn't mean it's bad. You just need to find a different API. The uses for cryptography are myriad. It seems unreasonable to expect a single API to cover everything while remaining simple and easy to use.
> There is absolutely a place for an API like this -- but that place is not as a replacement for OpenSSL.
I disagree. I think it is exactly the place for an OpenSSL API replacement.
As is alluded to in this link (thanks cremno) [1], they want a critical mass of users before it makes sense to deprecate the libssl API. Deprecation seems necessary in order to perform the serious cleanup/reimplementation under the hood that they would like to do.
While new projects may have an easier time, they admit that adoption by existing projects is significant work. However, so many and large projects already exist that it seems that many of them need to be won over in order for a deprecation to be practical. Many existing projects (see other comments in this post) use this transport layer flexibility in the OpenSSL API, and not offering it with the new API may slow adoption.
Yes, like you say another API layer could be added, and it is a deeply subjective matter so let's agree to disagree... But composability is so important and only becomes more so as we tire of wheel reinventions leading to unnecessary bugs. Enabling modularity and reuse needs 1st class API attention. Being able to easily plug this new API as a source/sink onto the myriad of existing (and yet to be created) I/O frameworks would be a major win. This is my $.02 and humble appeal to the people doing the fantastic work on this lib. My thanks to all of you!
(I assume the parent is referring to) those primitives don't provide encryption by default, or offer any options to enable encryption. RFC 7258 considers this a vulnerability.
Yes, and BIOs are one of the parts that basically every major open source project uses:
Apache, Nginx, Node.js, curl, etc -- they all use the BIO API to provide TLS as a filter over their internal streaming APIs. Having personally worked on some of these projects, the current libtls API is not sufficient at all.
This isn't to say at all that OpenSSL's BIO API was great -- it could definitely use iteration all over -- and I hope libtls gets there, but it doesn't seem at all focused on realistic server workloads.
The only reason you need the BIO API is because it is the only way to push data into OpenSSL, there is no reason why this can't be done with buffers you manage.
For example: http://funcptr.net/2012/04/08/openssl-as-a-filter-(or-non-bl... is a blog post I wrote wherein I show how to use SSL as a filter. The only reason I use a BIO is to temporarily store the same data I just got from the network and have in an std::string... it's duplication.
From a perfectionist viewpoint, the decision's almost correct: TLS work on streams and fds are streams in POSIX. Although an ANSI C's FILE* would be a more portable thus wiser choice.
However, given that the world's flawed and FILEs (and fds) aren't proper objects that end-user can create and use with their own functions (like an implementation of read(2) that'd yield to an event loop until the data's there), guess it warrants yet another ad-hoc struct. Or a bunch of functions that work on some context struct and that you're supposed to pass data chunks by yourself. Like in classic OpenSSL API.
Not sure if this applies in this case, but perhaps this is an attempt to address a common problem with crypto, which is that you have to get a whole lot of stuff around the usage of the basic crypto primitives right to avoid inadvertently opening up sidechannel vulnerabilities, or weakening the crypto. By binding the crypto to particular ways of getting data in and out, perhaps it eliminates implementation risks?
Please correct me if I'm wrong, but I believe that the OpenSSL API allows one to implement the actual transport of the TLS protected data however one chooses, although it provides convenient socket/fd transports. Is this a possibility with this new API? Regardless of whether OpenSSL can do it today, I consider this a very valuable feature and allows for more flexible composition of the TLS implementation and the rest of the system.
* tls_connect() connects a client context to the server named by host. The port may be numeric or a service name. If it is NULL then a host of the format "hostname:port" is permitted.
* tls_connect_fds() connects a client context to a pair of existing file descriptors.
* tls_connect_socket() connects a client context to an already established socket connection.
I did read the linked page, but perhaps I'm misunderstanding something. To me, this looks like it still requires sockets or fds. I want to be able to grab the TLS-protected byte stream conveniently and transport it to the other endpoint however I see fit. This would make it possible to use this TLS implementation easier on top of I/O-frameworks that don't necessarily map immediately down to sockets/fds. Again, for improved composability and code reuse.
It is very much a kluge but you can use pipe(2); it will give you two fds both of which you control. Then you can read/write from the "local" end and serialise the traffic however you wish.
And I really like the new API. Kudos to the creators. I would really like to use this new API in a simple, straightforward and internally consistent way - and grab the byte stream after encryption to feed it into my existing I/O framework. Please forgive the sarcasm, but all my comments are derived from a will to see this new API succeed, which is why I don't want this use case to be overlooked while designing it. If I was happy staying with the original OpenSSL API I probably wouldn't have voiced my concern as fervently.
That sounds strange, as I believe we're in disagreement. I believe the flexibility should exist, and choosing the transport medium of already encrypted data should be an option for the developer. This is completely unrelated to the security critical configuration of the TLS protocol, i.e. certs, keys, ciphersuites, etc.
The ressl API does provide one noteworthy feature. Hostname verification. In order to make a secure TLS connection, you must do two things. Validate the certificate and its trust chain. Then verify that the hostname in the cert matches the hostname you've connected to. Lots of people don't do the latter because OpenSSL doesn't do that latter. You have to do it yourself, which requires knowing about things like CommonNames and SubjectAltNames. The good news is that popular bindings for languages like python and ruby include a function to verify the hostname. The bad news is if you pick a python or ruby project at random, they probably forget to do it. Another funny fact is that since everybody has to write this code themselves, everybody does it a little bit differently. Especially regarding handling of wildcard certificates and everybody's favorite, embedded nul bytes. Hostname verification is on by default in ressl, and the API is designed so that you always provide a hostname; there's no way to accidentally call the function that doesn't do verification.
It's probably still early to be using this, unless it's of particular interest (interest is encouraged, of course!). There's a reason the server API isn't documented, for example.
So wait until the API is used by relayd, opensmtpd and httpd and see how that works out? I assume that in the end the goal would be to use the libtls for all of the home grown OpenBSD tools?
How does one poll for socket activity using this API, given that a read on a socket doesn't necessarily mean that decrypted data is available, and yet that socket writes may be required?
Other TLS libraries tend to have an OpenSSL compatibility layer. Now here’s yet another API for them to write compatibility layers for. This is a bad trend.
I’m thinking that there ought to be an API defined by an IETF working group and specified in an RFC (like the IPv6 API C interface is), with obvious room for expansion for library-specific features. Then all TLS libraries could implement that API and end the madness.
Do the defaults strike the best balance between security, PFS and compatibility? Because I think that's really the problem with the TLS libraries I've used, and introducing a new API is a good opportunity to fix that. What happens, for example, if I don't call
tls_config_set_ciphers()?
Host and cert verification by default is good. You can use tls_config_set_ciphers to set a cipher string, but I don't see if they disable unsafe ciphers by default.
