Fix numerous thread/socket lifecycle crashes and TLS/epoll write-stall busy-loop
Thread/EPoll lifecycle: - Thread::~Thread() now actually joins/frees its worker thread (was empty - the OS thread kept running against already-freed EPoll/Socket objects on any early teardown, a confirmed "double free" crash) - EPoll::~EPoll() calls stop() if never explicitly stopped, for the same reason - epoll_wait() now uses a 1s timeout instead of blocking forever, so a worker thread can actually notice isStopping() and let itself be joined - EPoll::processCommand() iterated threads by value, silently double-freeing the real vector's Thread the moment each copy went out of scope - ThreadScope gained onThreadStart()/onThreadEnd() hooks (actually wired up now - the constructor that was supposed to store the scope discarded it) for per-thread C-library init (e.g. MySQL client) Socket/TCPSocket/TLS: - Socket::writeSocket(): give up and shut down a connection after 50 consecutive zero-progress sends instead of retrying forever - SSL_ERROR_WANT_READ on a write needs a READ to make progress, but resetSocket() re-arms EPOLLWRNORM regardless, which an already-writable TCP socket satisfies immediately - a real stuck connection generated 55GB, then 12.9GB of debug log this way before being caught - Socket::writeSocket() now honors sendData()'s actual return value (previously discarded, entry popped unconditionally after one call) - a large response could get silently truncated past what fit in one write()/SSL_write() call - TLS::registerSocket()/sendData()/receiveData(): `initialized` now only goes true once SSL_set_fd() has actually succeeded, and sendData()/receiveData() check it first - fixes a real SIGSEGV where a new session's own queued greeting could reach SSL_write() on a different epoll thread before this session's handshake setup had run - TCPServer::connectionRequest() now performs the TLS handshake ON THE ACCEPTED SESSION, not the listening server itself (was building SSL state on the wrong object, leaving every session's own `ssl` uninitialized) - TCPServer::receiveData() always uses the plain (non-TLS) read path - the listening socket is never itself a TLS peer - TCPSocket's block-mode literal reading used an unset `blockLength` field instead of `blockSize`, and could busy-spin forever on a multi-packet literal - TCPSocket line scanning replaced an unbounded strcspn() (assumes a NUL-terminated buffer; read past a non-NUL-terminated receive buffer into adjacent heap memory) with a bounds-checked memchr() scan Protocol/session handling: - CommandList::processRequest() now case-folds the command keyword before lookup - SMTP/POP3/IMAP commands are spec-required to be case-insensitive; a client sending "Mail From:" instead of "MAIL FROM:" previously failed to dispatch at all - TCPSession::protocol() no longer drops blank lines while a handler has grabbed input (e.g. __SMTP_DATA reading a message body, an IMAP APPEND literal) - those handlers depend on seeing blank lines as real content, not commands; this silently corrupted any message body containing a blank line - TCPSession's uuid alias fixed: streaming a raw uuid_t via operator<< triggered libstdc++'s unsigned-char* overload (strlen() on 16 binary bytes, a real stack-buffer-overflow read), and `alias` pointed at a stack-local that was already destroyed by the time the constructor returned - INotify::processEvent() only treats event->name as a NUL-terminated string when event->len > 0 (per inotify(7)) instead of unconditional strlen() on a field the kernel doesn't guarantee is even present - a confirmed heap-buffer-overflow read that had been misdiagnosed as a MySQL thread-safety issue
This commit is contained in:
Brad Arant committed
1 parent
a51a486560
commit
c3e8d70774
14 files changed
+627
-89
No files matched your search
@@ -33,15 +33,27 @@ namespace core {
|
||||
void handshake_complete(const SSL *ssl, int where, int ret) {
|
||||
coreutils::Log(coreutils::LOG_DEBUG_3) << "==>" << SSL_state_string_long(ssl) << "<==" << ret;
|
||||
if(where & SSL_CB_HANDSHAKE_DONE) {
|
||||
X509 *ssl_client_cert = SSL_get_peer_certificate(ssl);
|
||||
if(!ssl_client_cert)
|
||||
throw std::string("Unable to get peer certificate.");
|
||||
X509_free(ssl_client_cert);
|
||||
if(SSL_get_verify_result(ssl) != X509_V_OK)
|
||||
throw std::string("Certificate verification failed.");
|
||||
coreutils::Log(coreutils::LOG_DEBUG_3) << "Certificate verified successfully.";
|
||||
// Only a context configured with SSL_VERIFY_PEER (see
|
||||
// createContext()'s own cACertificate-gated block) ever sends a
|
||||
// client a CertificateRequest in the first place - ordinary
|
||||
// one-way TLS (a public site's normal case, no caCertificate
|
||||
// configured) never asks the client for one, so
|
||||
// SSL_get_peer_certificate() is unconditionally NULL there and is
|
||||
// not a failure. Checking the mode actually negotiated on THIS
|
||||
// connection (inherited from the context) rather than assuming
|
||||
// every handshake is mutual-TLS is what makes plain
|
||||
// browser-facing HTTPS work at all under this callback.
|
||||
if(SSL_get_verify_mode(ssl) & SSL_VERIFY_PEER) {
|
||||
X509 *ssl_client_cert = SSL_get_peer_certificate(ssl);
|
||||
if(!ssl_client_cert)
|
||||
throw std::string("Unable to get peer certificate.");
|
||||
X509_free(ssl_client_cert);
|
||||
if(SSL_get_verify_result(ssl) != X509_V_OK)
|
||||
throw std::string("Certificate verification failed.");
|
||||
coreutils::Log(coreutils::LOG_DEBUG_3) << "Certificate verified successfully.";
|
||||
}
|
||||
}
|
||||
else
|
||||
else
|
||||
coreutils::Log(coreutils::LOG_DEBUG_3) << "No client certificate.";
|
||||
}
|
||||
|
||||
@@ -74,23 +86,45 @@ namespace core {
|
||||
SSL_CTX_set_session_cache_mode(ctx, SSL_SESS_CACHE_SERVER);
|
||||
// SSL_CTX_set_generate_session_id(ctx, generate_session_id);
|
||||
SSL_CTX_set_cipher_list(ctx, "ECDH-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA");
|
||||
if(SSL_CTX_use_certificate_file(ctx, tlsInfo->certificate.c_str(), SSL_FILETYPE_PEM) <= 0)
|
||||
// _chain_file (not _file) - loads every certificate in the file,
|
||||
// not just the first. A real Let's Encrypt fullchain.pem is the
|
||||
// leaf cert followed by the intermediate CA cert concatenated in
|
||||
// one file specifically so a server can hand both to the client
|
||||
// in one shot; SSL_CTX_use_certificate_file() only ever reads the
|
||||
// first PEM block, silently dropping the intermediate - confirmed
|
||||
// via a real handshake against this exact fullchain.pem: clients
|
||||
// got "unable to get local issuer certificate" because the server
|
||||
// only ever sent the leaf, never the intermediate needed to chain
|
||||
// up to a root the client already trusts.
|
||||
if(SSL_CTX_use_certificate_chain_file(ctx, tlsInfo->certificate.c_str()) <= 0)
|
||||
throw coreutils::Exception("Error looking up certificate.");
|
||||
if(SSL_CTX_use_PrivateKey_file(ctx, tlsInfo->key.c_str(), SSL_FILETYPE_PEM) < 0)
|
||||
throw coreutils::Exception("Error with private key.");
|
||||
if(SSL_CTX_check_private_key(ctx) != 1)
|
||||
throw coreutils::Exception("Private key does not match certificate.");
|
||||
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
|
||||
SSL_CTX_set_verify_depth(ctx, 1);
|
||||
if(!SSL_CTX_load_verify_locations(ctx, tlsInfo->cACertificate.c_str(), NULL))
|
||||
throw coreutils::Exception("Cannot verify locations.");
|
||||
SSL_CTX_set_client_CA_list(ctx, SSL_load_client_CA_file(tlsInfo->cACertificate.c_str()));
|
||||
coreutils::Log(coreutils::LOG_INFO) << "Server key authenticated.";
|
||||
}
|
||||
// Client-certificate verification (mutual TLS) is opt-in, not the
|
||||
// default - a public-facing site (e.g. a real Let's Encrypt
|
||||
// server cert with no corresponding client CA of its own) has no
|
||||
// client certificate to hand out to ordinary visitors, so
|
||||
// requesting one via SSL_VERIFY_PEER would surface a "select a
|
||||
// certificate" prompt real browsers can't meaningfully answer.
|
||||
// Only enable it when a caCertificate is actually configured -
|
||||
// existing internal/mTLS use of this class (any config that DOES
|
||||
// set tls.caCertificate) keeps behaving exactly as before.
|
||||
if(tlsInfo->cACertificate.getLength() > 0) {
|
||||
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
|
||||
SSL_CTX_set_verify_depth(ctx, 1);
|
||||
if(!SSL_CTX_load_verify_locations(ctx, tlsInfo->cACertificate.c_str(), NULL))
|
||||
throw coreutils::Exception("Cannot verify locations.");
|
||||
SSL_CTX_set_client_CA_list(ctx, SSL_load_client_CA_file(tlsInfo->cACertificate.c_str()));
|
||||
coreutils::Log(coreutils::LOG_INFO) << "Server key authenticated (client certificate required).";
|
||||
} else {
|
||||
coreutils::Log(coreutils::LOG_INFO) << "Server key authenticated (no client certificate required).";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void TLS::registerSocket(int fd) {
|
||||
initialized = true;
|
||||
void TLS::registerSocket(int fd) {
|
||||
int ret;
|
||||
|
||||
coreutils::Log(coreutils::LOG_DEBUG_3) << "TLS socket initializing on socket " << fd << "...";
|
||||
@@ -104,11 +138,37 @@ namespace core {
|
||||
SSL_set_info_callback(ssl, handshake_complete);
|
||||
|
||||
if((ret = SSL_set_fd(ssl, fd)) == 0)
|
||||
throw std::string("Error setting TLS socket descriptor.");
|
||||
|
||||
throw std::string("Error setting TLS socket descriptor.");
|
||||
|
||||
// if(!SSL_set_generate_session_id(ssl, generate_session_id))
|
||||
// throw std::string("Error setting session identifier callback.");
|
||||
|
||||
// Was set unconditionally as the FIRST statement in this
|
||||
// function (before `ssl` was even assigned) - a flag meant to
|
||||
// mean "ssl is a valid, ready-to-use SSL*" needs to become true
|
||||
// only once that's actually true, at the end, not the start.
|
||||
// Previously set but never actually READ anywhere either -
|
||||
// sendData()/receiveData() below now check it before touching
|
||||
// `ssl` at all. See their own doc comments for the real,
|
||||
// confirmed crash this fixes: a brand-new TLS session's own
|
||||
// greeting (queued in onConnected(), see e.g. IMAPSession.h)
|
||||
// gets written via Socket::write() -> resetSocket(), which - deep
|
||||
// inside TCPServer::accept(), BEFORE TCPServer::connectionRequest()
|
||||
// ever calls this session's own performTLSHandshake() - makes the
|
||||
// fd immediately eligible for an EPOLLWRNORM event on a SEPARATE
|
||||
// epoll worker thread. That thread's TCPSocket::sendData() takes
|
||||
// ioLock (see its own doc comment) and calls straight into
|
||||
// SSL_write() on `ssl`, which is genuinely uninitialized garbage
|
||||
// (TLS.h explicitly documents `ssl` has no default member
|
||||
// initializer) until THIS function runs - ioLock alone doesn't
|
||||
// prevent it, since performTLSHandshake() (the only other place
|
||||
// that takes ioLock) hasn't even been called yet at that point.
|
||||
// Confirmed live via gdb under a mixed-protocol connection-churn
|
||||
// stress test: SIGSEGV inside libssl.so's SSL_write(), called from
|
||||
// TCPSocket::sendData() with ioGuard already held, while
|
||||
// registerSocket()/acceptSocket() for that same session were still
|
||||
// pending on the accepting thread.
|
||||
initialized = true;
|
||||
}
|
||||
|
||||
void TLS::acceptSocket() {
|
||||
@@ -136,6 +196,14 @@ namespace core {
|
||||
|
||||
int TLS::receiveData(coreutils::ZString &buffer) {
|
||||
|
||||
// `ssl` is genuinely uninitialized until registerSocket() has
|
||||
// fully run (see its own doc comment on `initialized` for the
|
||||
// real crash this prevents) - a racing thread landing here first
|
||||
// just gets "nothing yet", exactly like a real SSL_ERROR_WANT_READ
|
||||
// would report, rather than touching a garbage pointer.
|
||||
if (!initialized)
|
||||
return -1;
|
||||
|
||||
int len;
|
||||
coreutils::Log(coreutils::LOG_DEBUG_3) << "receiveData TLS on socket.";
|
||||
|
||||
@@ -168,6 +236,17 @@ namespace core {
|
||||
|
||||
int TLS::sendData(const char *data, int length) {
|
||||
|
||||
// Same "ssl isn't ready yet" guard as receiveData() - see
|
||||
// registerSocket()'s doc comment on `initialized` for the real,
|
||||
// confirmed crash this prevents (SIGSEGV in SSL_write() on a
|
||||
// still-uninitialized `ssl`, reached via a brand-new session's own
|
||||
// greeting racing ahead of its own TLS handshake). Returning <= 0
|
||||
// here is already the exact case Socket::writeSocket() treats as
|
||||
// "nothing sent this round, leave it queued and retry on the next
|
||||
// EPOLLWRNORM" - no special-casing needed at any call site.
|
||||
if (!initialized)
|
||||
return -1;
|
||||
|
||||
int len;
|
||||
|
||||
if((len = ::SSL_write(ssl, data, length)) >= 0) {
|
||||
|
||||
Reference in new issue
Block a user