Stick to the existing protocols. TLS for session control. ESP for bulk traffic. IKE - for the ESP key exchange or piggy-back on TLS instead. Multiplex as needed.

Which libraries to use is secondary.