I've been looking into what options exist and what their pros and cons are for switching LuaSec from OpenSSL to something else (mostly in the context of Prosody's TLS support). These are my pros and cons so far, but it's a pretty specific application, so not all points may be important for others.
Pros:
* PolarSSL seems to be very clear code, well modularized and a much more sane API than OpenSSL.
* Supports setting cipher orders by TLS version.
* Cipher support is good (all the new ECC and SHA2 stuff is included), but it doesn't support SSLv2 or EXP ciphers (I'm counting this as a "pro", though).
Cons:
* Not quite as much support for working with X509 certificates. It parses the extension fields it knows about, but if you want to know more than those, you'll have to dig into ASN.1 yourself.
* Related to the previous point: I couldn't find support in converting different ASN.1 string formats. Probably will have to depend on another library to do that.
* I found a buffer overflow bug within a few hours of working with it [1], which is really not a good sign for a TLS library. However, it did get acknowledged within 24 hours (I've got an OpenSSL patch which has been sitting on their bugtracker without any reply for 6 months).
* Setting a different curve for ECC requires a compile time flag to be set, which is disabled by default.
* Using the server's cipher order or the client's is also a compile time flag, can not be set at run time.
Pros:
* PolarSSL seems to be very clear code, well modularized and a much more sane API than OpenSSL.
* Supports setting cipher orders by TLS version.
* Cipher support is good (all the new ECC and SHA2 stuff is included), but it doesn't support SSLv2 or EXP ciphers (I'm counting this as a "pro", though).
Cons:
* Not quite as much support for working with X509 certificates. It parses the extension fields it knows about, but if you want to know more than those, you'll have to dig into ASN.1 yourself.
* Related to the previous point: I couldn't find support in converting different ASN.1 string formats. Probably will have to depend on another library to do that.
* I found a buffer overflow bug within a few hours of working with it [1], which is really not a good sign for a TLS library. However, it did get acknowledged within 24 hours (I've got an OpenSSL patch which has been sitting on their bugtracker without any reply for 6 months).
* Setting a different curve for ECC requires a compile time flag to be set, which is disabled by default.
* Using the server's cipher order or the client's is also a compile time flag, can not be set at run time.
[1] = https://github.com/polarssl/polarssl/issues/83