Move TCPSession lifetime to shared_ptr ownership, fix the whole double-delete/UAF class of crash
Every crash root-caused earlier tonight (Subscription::subscribers, TCPServer::sessions, the Socket::write()/eventReceived() re-arm race) was the same underlying shape: some thread calling a method - or Thread::run() calling delete - on a TCPSession* while another thread concurrently freed that same object, because nothing actually guaranteed the object stayed alive for the duration of a call. Mutex-based patches at each individual call site kept narrowing the window without closing it (confirmed live, repeatedly, via AddressSanitizer under concurrent load - each fix produced a new, different crash signature at the same underlying race), and adding more locks around every write() call also works against the fifo-based design's own goal: no thread should have to block just to hand a message to another thread's session. TCPSession now derives from std::enable_shared_from_this<TCPSession> and exposes pin() - a safe upgrade from a stored raw pointer to a real, ownership-extending shared_ptr, returning nullptr instead of throwing once the object's real owner has let go. TCPServer::sessions is now the one real strong-owning shared_ptr<TCPSession> per live session; every other place that stores a TCPSession* (Subscription::subscribers, Zephora's own sessionsByPlayerName, GroupObject's membership list, etc.) is just a non-owning observer and must pin() before touching the object. A second, distinct problem: Thread::run()'s epoll dispatch starts from a raw void* of unknown validity. An early version of this fix used dynamic_cast<TCPSession*> to identify session objects before calling pin() - confirmed live via AddressSanitizer that dynamic_cast itself segfaults on a stale pointer, since it has to read the (potentially freed) object's own vtable to work at all. TCPSession::tryGet()/TCPServer:: isKnownListeningSocket() replace that: both look the raw pointer up as a key in a registry, never dereferencing it for the check itself - safe to call on any value, live session, immortal listening socket, or fully freed garbage alike. Verified via an AddressSanitizer-instrumented build under sustained concurrent load (up to 15-way parallel connect/subscribe/create/disconnect cycles, multiple rounds, 100+ total connections) with zero errors, then against the real production build with a genuine 500-connection load test (463/500 succeeded; the rest hit the process's own file-descriptor ulimit under peak burst, an unrelated and separately deferred tuning matter, not a crash). Files touched: TCPSession.h (enable_shared_from_this, pin(), the tryGet() registry), TCPServer.h/.cpp (sessions as shared_ptr, isKnownListening Socket() registry, lock made protected for subclasses), Thread.cpp (the whole dispatch rewrite), Subscription.cpp/SubscriptionManager.cpp (every subscriber write() site now pin()s first), ConsoleServer.cpp (logSend() was iterating `sessions` with no lock at all - fixed alongside the type change it needed anyway).
This commit is contained in:
1 parent
8009564a7a
commit
d058322cc6
10 files changed
+411
-64
No files matched your search
+60
-7
@@ -1,5 +1,6 @@
|
||||
#include "Thread.h"
|
||||
#include "EPoll.h"
|
||||
#include "TCPServer.h"
|
||||
#include <sys/epoll.h>
|
||||
|
||||
namespace core {
|
||||
@@ -100,14 +101,66 @@ namespace core {
|
||||
} else if(rc > 0) {
|
||||
for(int ix = 0; ix < rc; ++ix) {
|
||||
++count;
|
||||
if(((Socket *)events[ix].data.ptr)->eventReceived(events[ix], ++ePoll.eventId)) {
|
||||
// coreutils::Log(coreutils::LOG_DEBUG_4) << "return true";
|
||||
// ePoll.resetSocket((Socket *)events[ix].data.ptr);
|
||||
// events[ix].data.ptr is a raw pointer of UNKNOWN validity at
|
||||
// this point - it could be a live TCPSession, a listening
|
||||
// TCPServer/ConsoleServer (immortal, never shared_ptr-owned),
|
||||
// or - the case this whole block exists to handle safely - a
|
||||
// stale/duplicate event for a TCPSession something else
|
||||
// already fully destroyed and freed on another thread.
|
||||
//
|
||||
// A first version of this fix used dynamic_cast<TCPSession*>
|
||||
// to tell these apart, and a version before that cast
|
||||
// straight to Socket* unconditionally (the original code).
|
||||
// BOTH require reading the pointed-to object's own memory
|
||||
// (its vtable, at minimum) to work - confirmed live via
|
||||
// AddressSanitizer that dynamic_cast itself segfaults on a
|
||||
// stale pointer, before any pin()-based safety logic even
|
||||
// runs. TCPSession::tryGet() and TCPServer::isKnownListening
|
||||
// Socket() below never dereference this raw pointer for their
|
||||
// own check - only compare its address against a registry -
|
||||
// so they're safe to call on literally any value here,
|
||||
// known-good or not.
|
||||
void *rawPtr = events[ix].data.ptr;
|
||||
std::shared_ptr<TCPSession> pinned = TCPSession::tryGet(rawPtr);
|
||||
if(pinned) {
|
||||
if(pinned->eventReceived(events[ix], ++ePoll.eventId)) {
|
||||
// keep alive - pinned goes out of scope at the end of
|
||||
// this iteration, releasing this thread's temporary
|
||||
// reference; real ownership (TCPServer::sessions) is
|
||||
// unaffected.
|
||||
} else {
|
||||
pinned->shutDown = true;
|
||||
// Was `delete (Socket *)events[ix].data.ptr;`
|
||||
// unconditionally, for every socket type, including
|
||||
// real sessions another thread might be mid-call on
|
||||
// right now via its own pin()'d reference. Dropping
|
||||
// the one real owning reference is what actually
|
||||
// frees the object - immediately, if this thread's
|
||||
// own `pinned` (about to go out of scope) was the
|
||||
// last one outstanding; safely deferred otherwise.
|
||||
pinned->server->removeFromSessionList(pinned.get());
|
||||
}
|
||||
} else if(TCPServer::isKnownListeningSocket(rawPtr)) {
|
||||
// Known-immortal (see its own doc comment) - safe to call
|
||||
// directly, unchanged from the original behavior. A
|
||||
// listening socket's own eventReceived() returning false
|
||||
// isn't something that happens in normal operation, but
|
||||
// preserved as-is for parity with the pre-existing code.
|
||||
Socket *rawSocket = (Socket *)rawPtr;
|
||||
if(rawSocket->eventReceived(events[ix], ++ePoll.eventId)) {
|
||||
// keep alive
|
||||
} else {
|
||||
rawSocket->shutDown = true;
|
||||
delete rawSocket;
|
||||
}
|
||||
} else {
|
||||
((Socket *)events[ix].data.ptr)->shutDown = true;
|
||||
// coreutils::Log(coreutils::LOG_DEBUG_4) << "return false";
|
||||
delete (Socket *)events[ix].data.ptr;
|
||||
}
|
||||
// Neither registry recognizes this pointer - almost
|
||||
// certainly a stale/duplicate epoll event for a
|
||||
// TCPSession that has already been fully destroyed and
|
||||
// unregistered elsewhere. There is nothing safe to do
|
||||
// with it except leave it alone entirely.
|
||||
coreutils::Log(coreutils::LOG_DEBUG_1) << "Ignoring stale/unrecognized epoll event (pointer " << rawPtr << ")";
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user