Every crash root-caused earlier tonight (Subscription::subscribers,
TCPServer::sessions, the Socket::write()/eventReceived() re-arm race) was
the same underlying shape: some thread calling a method - or Thread::run()
calling delete - on a TCPSession* while another thread concurrently freed
that same object, because nothing actually guaranteed the object stayed
alive for the duration of a call. Mutex-based patches at each individual
call site kept narrowing the window without closing it (confirmed live,
repeatedly, via AddressSanitizer under concurrent load - each fix produced
a new, different crash signature at the same underlying race), and adding
more locks around every write() call also works against the fifo-based
design's own goal: no thread should have to block just to hand a message
to another thread's session.
TCPSession now derives from std::enable_shared_from_this<TCPSession> and
exposes pin() - a safe upgrade from a stored raw pointer to a real,
ownership-extending shared_ptr, returning nullptr instead of throwing once
the object's real owner has let go. TCPServer::sessions is now the one
real strong-owning shared_ptr<TCPSession> per live session; every other
place that stores a TCPSession* (Subscription::subscribers, Zephora's own
sessionsByPlayerName, GroupObject's membership list, etc.) is just a
non-owning observer and must pin() before touching the object.
A second, distinct problem: Thread::run()'s epoll dispatch starts from a
raw void* of unknown validity. An early version of this fix used
dynamic_cast<TCPSession*> to identify session objects before calling
pin() - confirmed live via AddressSanitizer that dynamic_cast itself
segfaults on a stale pointer, since it has to read the (potentially freed)
object's own vtable to work at all. TCPSession::tryGet()/TCPServer::
isKnownListeningSocket() replace that: both look the raw pointer up as a
key in a registry, never dereferencing it for the check itself - safe to
call on any value, live session, immortal listening socket, or fully
freed garbage alike.
Verified via an AddressSanitizer-instrumented build under sustained
concurrent load (up to 15-way parallel connect/subscribe/create/disconnect
cycles, multiple rounds, 100+ total connections) with zero errors, then
against the real production build with a genuine 500-connection load test
(463/500 succeeded; the rest hit the process's own file-descriptor
ulimit under peak burst, an unrelated and separately deferred tuning
matter, not a crash).
Files touched: TCPSession.h (enable_shared_from_this, pin(), the tryGet()
registry), TCPServer.h/.cpp (sessions as shared_ptr, isKnownListening
Socket() registry, lock made protected for subclasses), Thread.cpp (the
whole dispatch rewrite), Subscription.cpp/SubscriptionManager.cpp (every
subscriber write() site now pin()s first), ConsoleServer.cpp (logSend()
was iterating `sessions` with no lock at all - fixed alongside the type
change it needed anyway).
Socket::write() unconditionally called resetSocket() (via EPOLL_CTL_MOD)
from whatever thread called it, whenever it could grab `lock` - including
while this same socket's OWN eventReceived() call was still mid-flight on
another thread (past receiveData()'s lock_guard, but before its own
trailing resetSocket()/return). Re-arming makes the fd immediately
eligible for a third thread to pick up a fresh event and, if it decides
to shut the connection down, delete the object outright (Thread::run()
does exactly that on a false return) - while the original thread is still
touching it. A real heap-corruption crash inside libc's malloc internals
was confirmed live via dmesg under concurrent load, consistent with
exactly this: a socket freed while another thread still held a reference
to it.
inHandler existed as a flag already but only fed a diagnostic log line -
never actually gated anything. Made it a real std::atomic<bool> (was
`volatile`, which gives no actual atomicity or cross-thread visibility)
and used it two ways:
- write() now skips its own resetSocket() call entirely while inHandler
is true - that call's own eventReceived() will re-arm once it's
actually done, seeing the fifo entry write() just queued.
- eventReceived() itself was clearing inHandler well before its own
resetSocket() call (several statements up, right after EPOLLIN/
EPOLLWRNORM handling) - moved the clear to immediately precede that
call, closing the exposed window down to the minimum achievable
without restructuring how write() re-arms sockets altogether.
This is a distinct bug from the two subscriber/session-list races fixed
earlier tonight (SubscriptionManager.cpp, TCPServer.cpp) - found while
stress-testing those fixes turned up a third, different crash signature
(heap corruption, not the null-EPoll-reference pattern the earlier two
produced).
shutdown() only sets shutDown=true and expects the fifo to drain
normally before eventReceived()'s keepAlive check
(!shutDown || (fifo.size() > 0 && !readHangup)) lets the connection
close - correct for a graceful QUIT/LOGOUT reply that just needs a few
more successful writes to flush, but exactly backwards for the give-up
path: the whole reason that path was reached is that the front of the
fifo can never be sent. Leaving it queued meant keepAlive stayed true
forever, resetSocket() kept re-arming EPOLLWRNORM because fifo wasn't
empty, the very next write attempt hit the identical stall
immediately, and shutdown() got called again - forever, at whatever
rate the kernel could fire EPOLLWRNORM.
Confirmed live on a real internet-facing server (Zephora's game
server, an ordinary port-scanner connection was enough to trigger it):
one socket logged "consecutive writes made zero progress - giving up
and shutting down" 2.4 million+ times in under 4 milliseconds,
pegging a worker thread at 100% CPU indefinitely. The give-up path now
clears the undeliverable fifo entry before calling shutdown(), so
keepAlive correctly evaluates to just !shutDown (now false) on the
next cycle and the connection actually tears down.
A TLS sendData() failing with SSL_ERROR_WANT_READ (mid-handshake
application data, or a post-handshake session ticket exchange) has
nothing to do with write-readiness, but resetSocket() kept re-arming
EPOLLWRNORM anyway - an already-writable TCP socket fires that
immediately, so the retry loop spun the CPU as fast as it could
until the connection was eventually dropped, without ever reading
the peer's first command. Confirmed via a 15GB+ debug log from one
stuck connection. sendNeedsRead() lets Socket now tell the
difference and wait for a real EPOLLIN instead of busy-retrying.
Thread/EPoll lifecycle:
- Thread::~Thread() now actually joins/frees its worker thread (was
empty - the OS thread kept running against already-freed EPoll/Socket
objects on any early teardown, a confirmed "double free" crash)
- EPoll::~EPoll() calls stop() if never explicitly stopped, for the
same reason
- epoll_wait() now uses a 1s timeout instead of blocking forever, so a
worker thread can actually notice isStopping() and let itself be
joined
- EPoll::processCommand() iterated threads by value, silently
double-freeing the real vector's Thread the moment each copy went
out of scope
- ThreadScope gained onThreadStart()/onThreadEnd() hooks (actually
wired up now - the constructor that was supposed to store the scope
discarded it) for per-thread C-library init (e.g. MySQL client)
Socket/TCPSocket/TLS:
- Socket::writeSocket(): give up and shut down a connection after 50
consecutive zero-progress sends instead of retrying forever -
SSL_ERROR_WANT_READ on a write needs a READ to make progress, but
resetSocket() re-arms EPOLLWRNORM regardless, which an
already-writable TCP socket satisfies immediately - a real stuck
connection generated 55GB, then 12.9GB of debug log this way before
being caught
- Socket::writeSocket() now honors sendData()'s actual return value
(previously discarded, entry popped unconditionally after one call) -
a large response could get silently truncated past what fit in one
write()/SSL_write() call
- TLS::registerSocket()/sendData()/receiveData(): `initialized` now
only goes true once SSL_set_fd() has actually succeeded, and
sendData()/receiveData() check it first - fixes a real SIGSEGV where
a new session's own queued greeting could reach SSL_write() on a
different epoll thread before this session's handshake setup had run
- TCPServer::connectionRequest() now performs the TLS handshake ON THE
ACCEPTED SESSION, not the listening server itself (was building SSL
state on the wrong object, leaving every session's own `ssl`
uninitialized)
- TCPServer::receiveData() always uses the plain (non-TLS) read path -
the listening socket is never itself a TLS peer
- TCPSocket's block-mode literal reading used an unset `blockLength`
field instead of `blockSize`, and could busy-spin forever on a
multi-packet literal
- TCPSocket line scanning replaced an unbounded strcspn() (assumes a
NUL-terminated buffer; read past a non-NUL-terminated receive buffer
into adjacent heap memory) with a bounds-checked memchr() scan
Protocol/session handling:
- CommandList::processRequest() now case-folds the command keyword
before lookup - SMTP/POP3/IMAP commands are spec-required to be
case-insensitive; a client sending "Mail From:" instead of
"MAIL FROM:" previously failed to dispatch at all
- TCPSession::protocol() no longer drops blank lines while a handler
has grabbed input (e.g. __SMTP_DATA reading a message body, an IMAP
APPEND literal) - those handlers depend on seeing blank lines as
real content, not commands; this silently corrupted any message body
containing a blank line
- TCPSession's uuid alias fixed: streaming a raw uuid_t via operator<<
triggered libstdc++'s unsigned-char* overload (strlen() on 16 binary
bytes, a real stack-buffer-overflow read), and `alias` pointed at a
stack-local that was already destroyed by the time the constructor
returned
- INotify::processEvent() only treats event->name as a NUL-terminated
string when event->len > 0 (per inotify(7)) instead of unconditional
strlen() on a field the kernel doesn't guarantee is even present -
a confirmed heap-buffer-overflow read that had been misdiagnosed as
a MySQL thread-safety issue
The previous commit picked up CRLF line endings from a Windows-side
working copy used during development. Converting to plain LF to match
the rest of this Linux-native repo - content unchanged, this is purely
a line-ending normalization (932 insertions/932 deletions, 1:1).
TCPSocket inherits from both Socket and TLS, both declaring receiveData()
as unrelated sibling methods - only Socket::receiveData was virtual, so
TLS::receiveData (SSL_read) was never reached regardless of TLS config,
and even when reached its result was never forwarded to onDataReceived
(that call was commented out). Added an explicit TCPSocket::receiveData
override dispatching to TLS::receiveData or Socket::receiveData based on
tlsInfo, and completed TLS::receiveData to actually return/forward the
decrypted bytes.
Write path did not exist at all. Added Socket::sendData() as a new
virtual hook mirroring receiveData, routed Socket::writeSocket() through
it instead of a raw ::write(), added TLS::sendData() (SSL_write, same
error-handling style as the existing TLS methods), and TCPSocket::sendData()
dispatching the same way the read side does.
Also fixed: a queued write in writeSocket() was being popped off the fifo
as if sent even when shutDown was true and nothing was actually written;
removed a dead empty if(tlsInfo){} stub in onDataReceived, no longer
needed now dispatch happens one level up; replaced leftover debug
std::cout lines with proper Log() calls matching the rest of the file.
Verified via full local compile: all 20 .cpp files build clean, link
into libServerCore.a, and nm confirms the new/fixed symbols are correctly
defined in the right object files.
Matt Arant
barant