Seems like a good idea, but it also seems like a lot of IPC. The author suggests that crypto costs more than IPC, so it doesn't matter much, which also seems reasonable, and I suppose most related DOS attacks are mitigated by disallowing client-initiated renegotiation anyway.
> Q. How much is the overhead?
> Virtually none.
> On my Linux VM running on Core i7 @ 2.4GHz (MacBook Pro 15" Late 2013)...
As an implementor using OpenSSL, I like this. It has a simple API, and is clearly documented.
I wish all security improvements were as simple and easy.
Someone should make an LD_PRELOAD hack for this. I might try throwing one together this weekend if someone doesn't beat me to it.
How is it different than using a SoftHSM?
I applaud the effort but it does not address the elephant in the room: openssl engines are synchronous. If you use openssl in an event-loop style server (coincidentally, OP's own web server h2o appears to be event-loop based), this means that the whole server blocks if the RSA operation blocks.
For a hardware accelerator that may not be so bad but if you are trying to separate the HSM via the network, to minimize what an attacker can do after compromising the httpd, then every packet loss or an outage or network delay (or, more generally, any latency) would block the whole httpd.
Cloudflare proclaimed a while ago that they had a way to do essentially this (with nginx+openssl) and they said their solution was non-blocking, but they did not publish the code as far as I can tell.
I think if one wanted to solve this problem properly, larger architectural changes to openssl would be necessary. Please correct me if I'm wrong!
EDIT: Also, if you move out the RSA operation, ideally you'd want to distribute the work over more than one CPU core. If the operation is synchronous, you can't really do that.