Move TCPSession lifetime to shared_ptr ownership, fix the whole double-delete/UAF class of crash
Every crash root-caused earlier tonight (Subscription::subscribers, TCPServer::sessions, the Socket::write()/eventReceived() re-arm race) was the same underlying shape: some thread calling a method - or Thread::run() calling delete - on a TCPSession* while another thread concurrently freed that same object, because nothing actually guaranteed the object stayed alive for the duration of a call. Mutex-based patches at each individual call site kept narrowing the window without closing it (confirmed live, repeatedly, via AddressSanitizer under concurrent load - each fix produced a new, different crash signature at the same underlying race), and adding more locks around every write() call also works against the fifo-based design's own goal: no thread should have to block just to hand a message to another thread's session. TCPSession now derives from std::enable_shared_from_this<TCPSession> and exposes pin() - a safe upgrade from a stored raw pointer to a real, ownership-extending shared_ptr, returning nullptr instead of throwing once the object's real owner has let go. TCPServer::sessions is now the one real strong-owning shared_ptr<TCPSession> per live session; every other place that stores a TCPSession* (Subscription::subscribers, Zephora's own sessionsByPlayerName, GroupObject's membership list, etc.) is just a non-owning observer and must pin() before touching the object. A second, distinct problem: Thread::run()'s epoll dispatch starts from a raw void* of unknown validity. An early version of this fix used dynamic_cast<TCPSession*> to identify session objects before calling pin() - confirmed live via AddressSanitizer that dynamic_cast itself segfaults on a stale pointer, since it has to read the (potentially freed) object's own vtable to work at all. TCPSession::tryGet()/TCPServer:: isKnownListeningSocket() replace that: both look the raw pointer up as a key in a registry, never dereferencing it for the check itself - safe to call on any value, live session, immortal listening socket, or fully freed garbage alike. Verified via an AddressSanitizer-instrumented build under sustained concurrent load (up to 15-way parallel connect/subscribe/create/disconnect cycles, multiple rounds, 100+ total connections) with zero errors, then against the real production build with a genuine 500-connection load test (463/500 succeeded; the rest hit the process's own file-descriptor ulimit under peak burst, an unrelated and separately deferred tuning matter, not a crash). Files touched: TCPSession.h (enable_shared_from_this, pin(), the tryGet() registry), TCPServer.h/.cpp (sessions as shared_ptr, isKnownListening Socket() registry, lock made protected for subclasses), Thread.cpp (the whole dispatch rewrite), Subscription.cpp/SubscriptionManager.cpp (every subscriber write() site now pin()s first), ConsoleServer.cpp (logSend() was iterating `sessions` with no lock at all - fixed alongside the type change it needed anyway).
This commit is contained in:
1 parent
8009564a7a
commit
d058322cc6
10 files changed
+411
-64
No files matched your search
+56
-22
@@ -144,7 +144,7 @@ namespace core
|
||||
// drain to a peer that's already gone is pointless, not graceful)
|
||||
// - gives that queued reply a real chance to go out before this
|
||||
// socket actually closes.
|
||||
bool keepAlive = !shutDown || (fifo.size() > 0 && !readHangup);
|
||||
bool keepAlive = preventSelfDelete || !shutDown || (fifo.size() > 0 && !readHangup);
|
||||
|
||||
// resetSocket() re-arms this fd via EPOLL_CTL_MOD, which makes it
|
||||
// immediately eligible for delivery to ANY worker thread's
|
||||
@@ -162,24 +162,44 @@ namespace core
|
||||
// already captured in a local (`keepAlive`) beforehand.
|
||||
coreutils::Log(coreutils::LOG_DEBUG_1) << "Event process ending for socket " << getDescriptor();
|
||||
// Was cleared several statements above (right after the EPOLLIN/
|
||||
// EPOLLWRNORM handling) - a genuinely separate race from the one
|
||||
// documented above: Socket::write() re-arms via this same
|
||||
// EPOLLWRNORM handling), with no lock held around clearing it or
|
||||
// calling resetSocket() below - a genuinely separate race from the
|
||||
// one documented above: Socket::write() re-arms via this same
|
||||
// resetSocket() (see its own doc comment there) whenever it can
|
||||
// grab `lock`, from whatever thread called it (e.g. a broadcast
|
||||
// hitting this session while ITS OWN eventReceived() call is still
|
||||
// running, past receiveData()'s lock_guard but before reaching
|
||||
// here). Clearing inHandler that early left exactly that window
|
||||
// open for write() to re-arm this fd a second, premature time,
|
||||
// letting a third thread pick up a fresh event - and potentially
|
||||
// delete this object outright - while this call was still
|
||||
// mid-flight. Keeping inHandler true until the instant before
|
||||
// resetSocket() actually runs (write() checks it, see its own
|
||||
// comment) closes that window down to the two adjacent statements
|
||||
// below, as tight as achievable without restructuring how write()
|
||||
// re-arms at all.
|
||||
inHandler = false;
|
||||
if(keepAlive)
|
||||
// here). A first attempt narrowed this to just checking `inHandler`
|
||||
// in both places, without holding `lock` across the check-and-act
|
||||
// in either one - still a real TOCTOU window (inHandler read as
|
||||
// false here, then write() on another thread sees false too and
|
||||
// wins the race to resetSocket() first), confirmed live via
|
||||
// AddressSanitizer as a genuine double-delete of the same session
|
||||
// object (two threads both reaching ~TCPSession() on it) even
|
||||
// after that narrowing. Holding `lock` for the whole
|
||||
// clear-then-reset sequence - the exact same mutex write() already
|
||||
// takes before its own resetSocket() call - makes the two
|
||||
// mutually exclusive for real: whichever thread gets there first
|
||||
// fully finishes (clearing inHandler and re-arming, or observing
|
||||
// inHandler still true and skipping) before the other can act at
|
||||
// all, closing the window rather than just narrowing it.
|
||||
if(keepAlive) {
|
||||
lock.lock();
|
||||
inHandler = false;
|
||||
resetSocket();
|
||||
lock.unlock();
|
||||
}
|
||||
// else: this object is about to be delete()'d by Thread::run() the
|
||||
// moment this call returns, unconditionally, regardless of
|
||||
// anything write() does. Leaving inHandler true forever (never
|
||||
// clearing it here) means no other thread's write() can ever
|
||||
// resetSocket() this dying object in the meantime - it has no
|
||||
// future to reset it FOR, and letting write() re-arm it here, even
|
||||
// briefly, was exactly the remaining gap: a third thread could
|
||||
// pick up that re-armed fd and touch/delete an object Thread::run()
|
||||
// was already about to free, a genuine double-delete confirmed
|
||||
// live via AddressSanitizer even after write()/resetSocket() were
|
||||
// otherwise fully serialized against each other.
|
||||
return keepAlive;
|
||||
}
|
||||
|
||||
@@ -327,14 +347,28 @@ namespace core
|
||||
// delete this object (Thread::run() does exactly that on a false
|
||||
// return) while the first thread is still touching it - a genuine
|
||||
// use-after-free, matching a real heap-corruption crash confirmed
|
||||
// live via dmesg under concurrent load. eventReceived() already
|
||||
// re-arms itself once it's actually done (checking the current
|
||||
// fifo, which already includes whatever was just queued above);
|
||||
// skip doing it a second, early time from here while that's still
|
||||
// in progress - the data isn't lost, just picked up on that call's
|
||||
// own re-arm instead of this one's.
|
||||
if(!inHandler && lock.try_lock()) {
|
||||
resetSocket();
|
||||
// live via dmesg/AddressSanitizer under concurrent load.
|
||||
// eventReceived() already re-arms itself once it's actually done
|
||||
// (checking the current fifo, which already includes whatever was
|
||||
// just queued above); skip doing it a second, early time from here
|
||||
// while that's still in progress - the data isn't lost, just
|
||||
// picked up on that call's own re-arm instead of this one's.
|
||||
//
|
||||
// inHandler must be re-checked AFTER acquiring `lock`, not just
|
||||
// before try_lock() - checking it first and then separately
|
||||
// acquiring the lock leaves a real TOCTOU gap (this thread reads
|
||||
// inHandler==false, eventReceived() flips it back and forth on
|
||||
// another thread, both end up calling resetSocket()); a first
|
||||
// attempt at this fix only checked before, and a genuine
|
||||
// double-delete of the same session object still reproduced live
|
||||
// under AddressSanitizer despite it. eventReceived() holds this
|
||||
// exact same `lock` across its own inHandler-clear-then-reset
|
||||
// sequence now (see its own comment), so re-checking inHandler
|
||||
// only after try_lock() succeeds makes the two truly mutually
|
||||
// exclusive.
|
||||
if(lock.try_lock()) {
|
||||
if(!inHandler)
|
||||
resetSocket();
|
||||
lock.unlock();
|
||||
}
|
||||
return 1;
|
||||
|
||||
Reference in new issue
Block a user