Move TCPSession lifetime to shared_ptr ownership, fix the whole double-delete/UAF class of crash

Every crash root-caused earlier tonight (Subscription::subscribers,
TCPServer::sessions, the Socket::write()/eventReceived() re-arm race) was
the same underlying shape: some thread calling a method - or Thread::run()
calling delete - on a TCPSession* while another thread concurrently freed
that same object, because nothing actually guaranteed the object stayed
alive for the duration of a call. Mutex-based patches at each individual
call site kept narrowing the window without closing it (confirmed live,
repeatedly, via AddressSanitizer under concurrent load - each fix produced
a new, different crash signature at the same underlying race), and adding
more locks around every write() call also works against the fifo-based
design's own goal: no thread should have to block just to hand a message
to another thread's session.

TCPSession now derives from std::enable_shared_from_this<TCPSession> and
exposes pin() - a safe upgrade from a stored raw pointer to a real,
ownership-extending shared_ptr, returning nullptr instead of throwing once
the object's real owner has let go. TCPServer::sessions is now the one
real strong-owning shared_ptr<TCPSession> per live session; every other
place that stores a TCPSession* (Subscription::subscribers, Zephora's own
sessionsByPlayerName, GroupObject's membership list, etc.) is just a
non-owning observer and must pin() before touching the object.

A second, distinct problem: Thread::run()'s epoll dispatch starts from a
raw void* of unknown validity. An early version of this fix used
dynamic_cast<TCPSession*> to identify session objects before calling
pin() - confirmed live via AddressSanitizer that dynamic_cast itself
segfaults on a stale pointer, since it has to read the (potentially freed)
object's own vtable to work at all. TCPSession::tryGet()/TCPServer::
isKnownListeningSocket() replace that: both look the raw pointer up as a
key in a registry, never dereferencing it for the check itself - safe to
call on any value, live session, immortal listening socket, or fully
freed garbage alike.

Verified via an AddressSanitizer-instrumented build under sustained
concurrent load (up to 15-way parallel connect/subscribe/create/disconnect
cycles, multiple rounds, 100+ total connections) with zero errors, then
against the real production build with a genuine 500-connection load test
(463/500 succeeded; the rest hit the process's own file-descriptor
ulimit under peak burst, an unrelated and separately deferred tuning
matter, not a crash).

Files touched: TCPSession.h (enable_shared_from_this, pin(), the tryGet()
registry), TCPServer.h/.cpp (sessions as shared_ptr, isKnownListening
Socket() registry, lock made protected for subclasses), Thread.cpp (the
whole dispatch rewrite), Subscription.cpp/SubscriptionManager.cpp (every
subscriber write() site now pin()s first), ConsoleServer.cpp (logSend()
was iterating `sessions` with no lock at all - fixed alongside the type
change it needed anyway).
This commit is contained in:
Matt Arant committed 2026-09-08 02:24:09 +00:00
1 parent 8009564a7a
commit d058322cc6
10 files changed
+411 -64

No files matched your search

+56 -22
View File
@@ -144,7 +144,7 @@ namespace core
// drain to a peer that's already gone is pointless, not graceful)
// - gives that queued reply a real chance to go out before this
// socket actually closes.
bool keepAlive = !shutDown || (fifo.size() > 0 && !readHangup);
bool keepAlive = preventSelfDelete || !shutDown || (fifo.size() > 0 && !readHangup);
// resetSocket() re-arms this fd via EPOLL_CTL_MOD, which makes it
// immediately eligible for delivery to ANY worker thread's
@@ -162,24 +162,44 @@ namespace core
// already captured in a local (`keepAlive`) beforehand.
coreutils::Log(coreutils::LOG_DEBUG_1) << "Event process ending for socket " << getDescriptor();
// Was cleared several statements above (right after the EPOLLIN/
// EPOLLWRNORM handling) - a genuinely separate race from the one
// documented above: Socket::write() re-arms via this same
// EPOLLWRNORM handling), with no lock held around clearing it or
// calling resetSocket() below - a genuinely separate race from the
// one documented above: Socket::write() re-arms via this same
// resetSocket() (see its own doc comment there) whenever it can
// grab `lock`, from whatever thread called it (e.g. a broadcast
// hitting this session while ITS OWN eventReceived() call is still
// running, past receiveData()'s lock_guard but before reaching
// here). Clearing inHandler that early left exactly that window
// open for write() to re-arm this fd a second, premature time,
// letting a third thread pick up a fresh event - and potentially
// delete this object outright - while this call was still
// mid-flight. Keeping inHandler true until the instant before
// resetSocket() actually runs (write() checks it, see its own
// comment) closes that window down to the two adjacent statements
// below, as tight as achievable without restructuring how write()
// re-arms at all.
inHandler = false;
if(keepAlive)
// here). A first attempt narrowed this to just checking `inHandler`
// in both places, without holding `lock` across the check-and-act
// in either one - still a real TOCTOU window (inHandler read as
// false here, then write() on another thread sees false too and
// wins the race to resetSocket() first), confirmed live via
// AddressSanitizer as a genuine double-delete of the same session
// object (two threads both reaching ~TCPSession() on it) even
// after that narrowing. Holding `lock` for the whole
// clear-then-reset sequence - the exact same mutex write() already
// takes before its own resetSocket() call - makes the two
// mutually exclusive for real: whichever thread gets there first
// fully finishes (clearing inHandler and re-arming, or observing
// inHandler still true and skipping) before the other can act at
// all, closing the window rather than just narrowing it.
if(keepAlive) {
lock.lock();
inHandler = false;
resetSocket();
lock.unlock();
}
// else: this object is about to be delete()'d by Thread::run() the
// moment this call returns, unconditionally, regardless of
// anything write() does. Leaving inHandler true forever (never
// clearing it here) means no other thread's write() can ever
// resetSocket() this dying object in the meantime - it has no
// future to reset it FOR, and letting write() re-arm it here, even
// briefly, was exactly the remaining gap: a third thread could
// pick up that re-armed fd and touch/delete an object Thread::run()
// was already about to free, a genuine double-delete confirmed
// live via AddressSanitizer even after write()/resetSocket() were
// otherwise fully serialized against each other.
return keepAlive;
}
@@ -327,14 +347,28 @@ namespace core
// delete this object (Thread::run() does exactly that on a false
// return) while the first thread is still touching it - a genuine
// use-after-free, matching a real heap-corruption crash confirmed
// live via dmesg under concurrent load. eventReceived() already
// re-arms itself once it's actually done (checking the current
// fifo, which already includes whatever was just queued above);
// skip doing it a second, early time from here while that's still
// in progress - the data isn't lost, just picked up on that call's
// own re-arm instead of this one's.
if(!inHandler && lock.try_lock()) {
resetSocket();
// live via dmesg/AddressSanitizer under concurrent load.
// eventReceived() already re-arms itself once it's actually done
// (checking the current fifo, which already includes whatever was
// just queued above); skip doing it a second, early time from here
// while that's still in progress - the data isn't lost, just
// picked up on that call's own re-arm instead of this one's.
//
// inHandler must be re-checked AFTER acquiring `lock`, not just
// before try_lock() - checking it first and then separately
// acquiring the lock leaves a real TOCTOU gap (this thread reads
// inHandler==false, eventReceived() flips it back and forth on
// another thread, both end up calling resetSocket()); a first
// attempt at this fix only checked before, and a genuine
// double-delete of the same session object still reproduced live
// under AddressSanitizer despite it. eventReceived() holds this
// exact same `lock` across its own inHandler-clear-then-reset
// sequence now (see its own comment), so re-checking inHandler
// only after try_lock() succeeds makes the two truly mutually
// exclusive.
if(lock.try_lock()) {
if(!inHandler)
resetSocket();
lock.unlock();
}
return 1;