Fix numerous thread/socket lifecycle crashes and TLS/epoll write-stall busy-loop

Thread/EPoll lifecycle:
- Thread::~Thread() now actually joins/frees its worker thread (was
  empty - the OS thread kept running against already-freed EPoll/Socket
  objects on any early teardown, a confirmed "double free" crash)
- EPoll::~EPoll() calls stop() if never explicitly stopped, for the
  same reason
- epoll_wait() now uses a 1s timeout instead of blocking forever, so a
  worker thread can actually notice isStopping() and let itself be
  joined
- EPoll::processCommand() iterated threads by value, silently
  double-freeing the real vector's Thread the moment each copy went
  out of scope
- ThreadScope gained onThreadStart()/onThreadEnd() hooks (actually
  wired up now - the constructor that was supposed to store the scope
  discarded it) for per-thread C-library init (e.g. MySQL client)

Socket/TCPSocket/TLS:
- Socket::writeSocket(): give up and shut down a connection after 50
  consecutive zero-progress sends instead of retrying forever -
  SSL_ERROR_WANT_READ on a write needs a READ to make progress, but
  resetSocket() re-arms EPOLLWRNORM regardless, which an
  already-writable TCP socket satisfies immediately - a real stuck
  connection generated 55GB, then 12.9GB of debug log this way before
  being caught
- Socket::writeSocket() now honors sendData()'s actual return value
  (previously discarded, entry popped unconditionally after one call) -
  a large response could get silently truncated past what fit in one
  write()/SSL_write() call
- TLS::registerSocket()/sendData()/receiveData(): `initialized` now
  only goes true once SSL_set_fd() has actually succeeded, and
  sendData()/receiveData() check it first - fixes a real SIGSEGV where
  a new session's own queued greeting could reach SSL_write() on a
  different epoll thread before this session's handshake setup had run
- TCPServer::connectionRequest() now performs the TLS handshake ON THE
  ACCEPTED SESSION, not the listening server itself (was building SSL
  state on the wrong object, leaving every session's own `ssl`
  uninitialized)
- TCPServer::receiveData() always uses the plain (non-TLS) read path -
  the listening socket is never itself a TLS peer
- TCPSocket's block-mode literal reading used an unset `blockLength`
  field instead of `blockSize`, and could busy-spin forever on a
  multi-packet literal
- TCPSocket line scanning replaced an unbounded strcspn() (assumes a
  NUL-terminated buffer; read past a non-NUL-terminated receive buffer
  into adjacent heap memory) with a bounds-checked memchr() scan

Protocol/session handling:
- CommandList::processRequest() now case-folds the command keyword
  before lookup - SMTP/POP3/IMAP commands are spec-required to be
  case-insensitive; a client sending "Mail From:" instead of
  "MAIL FROM:" previously failed to dispatch at all
- TCPSession::protocol() no longer drops blank lines while a handler
  has grabbed input (e.g. __SMTP_DATA reading a message body, an IMAP
  APPEND literal) - those handlers depend on seeing blank lines as
  real content, not commands; this silently corrupted any message body
  containing a blank line
- TCPSession's uuid alias fixed: streaming a raw uuid_t via operator<<
  triggered libstdc++'s unsigned-char* overload (strlen() on 16 binary
  bytes, a real stack-buffer-overflow read), and `alias` pointed at a
  stack-local that was already destroyed by the time the constructor
  returned
- INotify::processEvent() only treats event->name as a NUL-terminated
  string when event->len > 0 (per inotify(7)) instead of unconditional
  strlen() on a field the kernel doesn't guarantee is even present -
  a confirmed heap-buffer-overflow read that had been misdiagnosed as
  a MySQL thread-safety issue
This commit is contained in:
Brad Arant committed 2026-09-03 05:17:41 +00:00
1 parent a51a486560
commit c3e8d70774
14 files changed
+627 -89

No files matched your search

+91 -10
View File
@@ -3,6 +3,8 @@
#include "Log.h"
#include "Exception.h"
#include "errno.h"
#include <algorithm>
#include <cstring>
namespace core {
@@ -38,7 +40,15 @@ namespace core {
// session (set in TLS::createContext(), inherited by sessions via
// TLS(SSL_CTX*)), safely NULL by default otherwise (see TLS.h).
if(ctx) {
int len = TLS::receiveData(buffer);
int len;
{
// Held only around the raw SSL_read call - released before
// onDataReceived() so the request-handling chain it triggers
// (which reaches Socket::write()/outlock) never runs while
// ioLock is held. See ioLock's own doc comment in TCPSocket.h.
std::lock_guard<std::mutex> ioGuard(ioLock);
len = TLS::receiveData(buffer);
}
if(len >= 0) {
coreutils::ZString zbuffer(buffer.getData(), len);
onDataReceived(zbuffer);
@@ -49,12 +59,39 @@ namespace core {
}
int TCPSocket::sendData(const char *data, int length) {
if(ctx)
return TLS::sendData(data, length);
else
if(ctx) {
std::lock_guard<std::mutex> ioGuard(ioLock);
return TLS::sendData(data, length);
} else
return Socket::sendData(data, length);
}
void TCPSocket::performTLSHandshake() {
// registerSocket() makes this fd eligible for epoll dispatch to ANY
// worker thread immediately. ioLock MUST be acquired BEFORE that
// call, not after - a first attempt here took the lock AFTER
// registerSocket(), which still left a real window: another thread
// could win the race for ioLock (via a read/write event on this
// same brand-new connection) and have SSL_read/SSL_write silently
// complete the handshake on OpenSSL's own initiative, and only
// THEN would this thread acquire ioLock and run its own, now
// redundant, acceptSocket() (SSL_accept) against an already-
// negotiated SSL object - confirmed via gdb `thread apply all bt`
// during a live crash: one thread mid-SSL_read, a second thread
// simultaneously inside performTLSHandshake, both on the identical
// object address. Acquiring ioLock first closes that window
// entirely: no other thread can even begin touching this
// connection's SSL object until this thread's own handshake
// attempt has released the lock. acceptSocket() only attempts
// SSL_accept() once and may leave the handshake incomplete
// (WANT_READ/WANT_WRITE) - that's fine, OpenSSL transparently
// finishes a pending handshake on the next SSL_read/SSL_write,
// which ioLock already serializes against everything else.
std::lock_guard<std::mutex> ioGuard(ioLock);
registerSocket(getDescriptor());
acceptSocket();
}
void TCPSocket::onDataReceived(coreutils::ZString &data) {
if (data.getLength() > 0) {
@@ -63,28 +100,72 @@ namespace core {
lineBufferSize += data.getLength();
while (lineBufferSize > 0) {
if (blockSize == 0) {
lineLength = strcspn(lineBuffer, "\r\n");
// strcspn() treats its argument as a NUL-terminated C
// string and keeps scanning past the end looking for one -
// lineBuffer is a raw byte buffer sized via realloc() to
// exactly lineBufferSize bytes, with no NUL terminator
// appended anywhere. Whenever the buffered data doesn't yet
// contain a CRLF (an ordinary, frequent case - a line split
// across TCP reads, or mid-way through a long one, e.g. a
// large email body), strcspn() read straight past
// lineBufferSize into whatever heap memory happened to
// follow, hunting for a byte that happened to match '\r',
// '\n', or an accidental '\0' - producing a lineLength
// larger than the buffer actually holds. Confirmed live via
// gdb: SIGSEGV inside a LATER, unrelated realloc() call on
// this same lineBuffer, well downstream of where the real
// corruption happened (lineBufferSize -= lineLength
// underflows when lineLength > lineBufferSize, and that
// negative value gets handed to realloc() as a size_t -
// realloc(ptr, SIZE_MAX-ish) - during a real inbound SMTP
// message with a long quoted-printable line). memchr() is
// the bounded equivalent - it never looks past the length
// given, so "no CRLF yet" correctly comes back as "not
// found" instead of reading off the end of the allocation.
char *cr = (char *)memchr(lineBuffer, '\r', lineBufferSize);
char *nl = (char *)memchr(lineBuffer, '\n', lineBufferSize);
char *found = (cr && nl) ? std::min(cr, nl) : (cr ? cr : nl);
lineLength = found ? (int)(found - lineBuffer) : lineBufferSize;
if (lineLength == lineBufferSize)
break;
coreutils::ZString zLine(lineBuffer, lineLength);
onLineReceived(zLine);
// lineLength < lineBufferSize is guaranteed by the break
// above, so this first index is always in bounds; bump
// past a lone '\r' only if there's still at least one more
// buffered byte to safely check for a following '\n' -
// otherwise (a bare trailing '\r' with nothing buffered
// after it yet) leave it for the next read to resolve,
// rather than reading lineBuffer[lineBufferSize], one past
// the allocation.
if (lineBuffer[lineLength] == '\r')
++lineLength;
if (lineBuffer[lineLength] == '\n')
if (lineLength < lineBufferSize && lineBuffer[lineLength] == '\n')
++lineLength;
lineBufferSize -= lineLength;
if (lineBufferSize > 0)
memmove(lineBuffer, lineBuffer + lineLength, lineBufferSize);
lineBuffer = (char *)realloc(lineBuffer, lineBufferSize);
}
else if (lineBufferSize >= blockLength) {
coreutils::ZString zBlock(lineBuffer, blockLength);
else if (lineBufferSize >= blockSize) {
coreutils::ZString zBlock(lineBuffer, blockSize);
onBlockReceived(zBlock);
lineBufferSize -= blockLength;
lineBufferSize -= blockSize;
if (lineBufferSize > 0)
memmove(lineBuffer, lineBuffer + blockLength, lineBufferSize);
memmove(lineBuffer, lineBuffer + blockSize, lineBufferSize);
lineBuffer = (char *)realloc(lineBuffer, lineBufferSize);
}
else
// Not enough buffered data yet for a full block - stop and
// wait for the next read, exactly like the line-mode
// branch's own "no complete line yet" break above.
// Without this, a block spanning more than one TCP read
// (any IMAP APPEND literal bigger than a single read()'s
// worth of socket buffer) sends this EPoll worker thread
// into an unconditional busy-spin - lineBufferSize never
// changes, so the while(lineBufferSize > 0) loop above
// never yields back to let more data arrive.
break;
}
}
}