Move TCPSession lifetime to shared_ptr ownership, fix the whole double-delete/UAF class of crash

Every crash root-caused earlier tonight (Subscription::subscribers,
TCPServer::sessions, the Socket::write()/eventReceived() re-arm race) was
the same underlying shape: some thread calling a method - or Thread::run()
calling delete - on a TCPSession* while another thread concurrently freed
that same object, because nothing actually guaranteed the object stayed
alive for the duration of a call. Mutex-based patches at each individual
call site kept narrowing the window without closing it (confirmed live,
repeatedly, via AddressSanitizer under concurrent load - each fix produced
a new, different crash signature at the same underlying race), and adding
more locks around every write() call also works against the fifo-based
design's own goal: no thread should have to block just to hand a message
to another thread's session.

TCPSession now derives from std::enable_shared_from_this<TCPSession> and
exposes pin() - a safe upgrade from a stored raw pointer to a real,
ownership-extending shared_ptr, returning nullptr instead of throwing once
the object's real owner has let go. TCPServer::sessions is now the one
real strong-owning shared_ptr<TCPSession> per live session; every other
place that stores a TCPSession* (Subscription::subscribers, Zephora's own
sessionsByPlayerName, GroupObject's membership list, etc.) is just a
non-owning observer and must pin() before touching the object.

A second, distinct problem: Thread::run()'s epoll dispatch starts from a
raw void* of unknown validity. An early version of this fix used
dynamic_cast<TCPSession*> to identify session objects before calling
pin() - confirmed live via AddressSanitizer that dynamic_cast itself
segfaults on a stale pointer, since it has to read the (potentially freed)
object's own vtable to work at all. TCPSession::tryGet()/TCPServer::
isKnownListeningSocket() replace that: both look the raw pointer up as a
key in a registry, never dereferencing it for the check itself - safe to
call on any value, live session, immortal listening socket, or fully
freed garbage alike.

Verified via an AddressSanitizer-instrumented build under sustained
concurrent load (up to 15-way parallel connect/subscribe/create/disconnect
cycles, multiple rounds, 100+ total connections) with zero errors, then
against the real production build with a genuine 500-connection load test
(463/500 succeeded; the rest hit the process's own file-descriptor
ulimit under peak burst, an unrelated and separately deferred tuning
matter, not a crash).

Files touched: TCPSession.h (enable_shared_from_this, pin(), the tryGet()
registry), TCPServer.h/.cpp (sessions as shared_ptr, isKnownListening
Socket() registry, lock made protected for subclasses), Thread.cpp (the
whole dispatch rewrite), Subscription.cpp/SubscriptionManager.cpp (every
subscriber write() site now pin()s first), ConsoleServer.cpp (logSend()
was iterating `sessions` with no lock at all - fixed alongside the type
change it needed anyway).
This commit is contained in:
Matt Arant committed 2026-09-08 02:24:09 +00:00
1 parent 8009564a7a
commit d058322cc6
10 files changed
+411 -64

No files matched your search

+60 -7
View File
@@ -1,5 +1,6 @@
#include "Thread.h"
#include "EPoll.h"
#include "TCPServer.h"
#include <sys/epoll.h>
namespace core {
@@ -100,14 +101,66 @@ namespace core {
} else if(rc > 0) {
for(int ix = 0; ix < rc; ++ix) {
++count;
if(((Socket *)events[ix].data.ptr)->eventReceived(events[ix], ++ePoll.eventId)) {
// coreutils::Log(coreutils::LOG_DEBUG_4) << "return true";
// ePoll.resetSocket((Socket *)events[ix].data.ptr);
// events[ix].data.ptr is a raw pointer of UNKNOWN validity at
// this point - it could be a live TCPSession, a listening
// TCPServer/ConsoleServer (immortal, never shared_ptr-owned),
// or - the case this whole block exists to handle safely - a
// stale/duplicate event for a TCPSession something else
// already fully destroyed and freed on another thread.
//
// A first version of this fix used dynamic_cast<TCPSession*>
// to tell these apart, and a version before that cast
// straight to Socket* unconditionally (the original code).
// BOTH require reading the pointed-to object's own memory
// (its vtable, at minimum) to work - confirmed live via
// AddressSanitizer that dynamic_cast itself segfaults on a
// stale pointer, before any pin()-based safety logic even
// runs. TCPSession::tryGet() and TCPServer::isKnownListening
// Socket() below never dereference this raw pointer for their
// own check - only compare its address against a registry -
// so they're safe to call on literally any value here,
// known-good or not.
void *rawPtr = events[ix].data.ptr;
std::shared_ptr<TCPSession> pinned = TCPSession::tryGet(rawPtr);
if(pinned) {
if(pinned->eventReceived(events[ix], ++ePoll.eventId)) {
// keep alive - pinned goes out of scope at the end of
// this iteration, releasing this thread's temporary
// reference; real ownership (TCPServer::sessions) is
// unaffected.
} else {
pinned->shutDown = true;
// Was `delete (Socket *)events[ix].data.ptr;`
// unconditionally, for every socket type, including
// real sessions another thread might be mid-call on
// right now via its own pin()'d reference. Dropping
// the one real owning reference is what actually
// frees the object - immediately, if this thread's
// own `pinned` (about to go out of scope) was the
// last one outstanding; safely deferred otherwise.
pinned->server->removeFromSessionList(pinned.get());
}
} else if(TCPServer::isKnownListeningSocket(rawPtr)) {
// Known-immortal (see its own doc comment) - safe to call
// directly, unchanged from the original behavior. A
// listening socket's own eventReceived() returning false
// isn't something that happens in normal operation, but
// preserved as-is for parity with the pre-existing code.
Socket *rawSocket = (Socket *)rawPtr;
if(rawSocket->eventReceived(events[ix], ++ePoll.eventId)) {
// keep alive
} else {
rawSocket->shutDown = true;
delete rawSocket;
}
} else {
((Socket *)events[ix].data.ptr)->shutDown = true;
// coreutils::Log(coreutils::LOG_DEBUG_4) << "return false";
delete (Socket *)events[ix].data.ptr;
}
// Neither registry recognizes this pointer - almost
// certainly a stale/duplicate epoll event for a
// TCPSession that has already been fully destroyed and
// unregistered elsewhere. There is nothing safe to do
// with it except leave it alone entirely.
coreutils::Log(coreutils::LOG_DEBUG_1) << "Ignoring stale/unrecognized epoll event (pointer " << rawPtr << ")";
}
}
}
}