Every crash root-caused earlier tonight (Subscription::subscribers,
TCPServer::sessions, the Socket::write()/eventReceived() re-arm race) was
the same underlying shape: some thread calling a method - or Thread::run()
calling delete - on a TCPSession* while another thread concurrently freed
that same object, because nothing actually guaranteed the object stayed
alive for the duration of a call. Mutex-based patches at each individual
call site kept narrowing the window without closing it (confirmed live,
repeatedly, via AddressSanitizer under concurrent load - each fix produced
a new, different crash signature at the same underlying race), and adding
more locks around every write() call also works against the fifo-based
design's own goal: no thread should have to block just to hand a message
to another thread's session.
TCPSession now derives from std::enable_shared_from_this<TCPSession> and
exposes pin() - a safe upgrade from a stored raw pointer to a real,
ownership-extending shared_ptr, returning nullptr instead of throwing once
the object's real owner has let go. TCPServer::sessions is now the one
real strong-owning shared_ptr<TCPSession> per live session; every other
place that stores a TCPSession* (Subscription::subscribers, Zephora's own
sessionsByPlayerName, GroupObject's membership list, etc.) is just a
non-owning observer and must pin() before touching the object.
A second, distinct problem: Thread::run()'s epoll dispatch starts from a
raw void* of unknown validity. An early version of this fix used
dynamic_cast<TCPSession*> to identify session objects before calling
pin() - confirmed live via AddressSanitizer that dynamic_cast itself
segfaults on a stale pointer, since it has to read the (potentially freed)
object's own vtable to work at all. TCPSession::tryGet()/TCPServer::
isKnownListeningSocket() replace that: both look the raw pointer up as a
key in a registry, never dereferencing it for the check itself - safe to
call on any value, live session, immortal listening socket, or fully
freed garbage alike.
Verified via an AddressSanitizer-instrumented build under sustained
concurrent load (up to 15-way parallel connect/subscribe/create/disconnect
cycles, multiple rounds, 100+ total connections) with zero errors, then
against the real production build with a genuine 500-connection load test
(463/500 succeeded; the rest hit the process's own file-descriptor
ulimit under peak burst, an unrelated and separately deferred tuning
matter, not a crash).
Files touched: TCPSession.h (enable_shared_from_this, pin(), the tryGet()
registry), TCPServer.h/.cpp (sessions as shared_ptr, isKnownListening
Socket() registry, lock made protected for subclasses), Thread.cpp (the
whole dispatch rewrite), Subscription.cpp/SubscriptionManager.cpp (every
subscriber write() site now pin()s first), ConsoleServer.cpp (logSend()
was iterating `sessions` with no lock at all - fixed alongside the type
change it needed anyway).
processCommand() read/mutated the `subscriptions` map and individual
Subscription objects' `subscribers` vectors (via subscribe()/unsubscribe()/
event()/the "publish" branch's own insert()) without taking `lock` at all,
even though add() and removeSessionSubscriptions() already do take it for
the exact same state. removeSessionSubscriptions() runs from ~TCPSession(),
on whatever thread happens to be tearing down a *different* connection, so
one thread erasing a session from a Subscription's subscribers vector could
run concurrently with another thread iterating (event()) or appending to
(subscribe()) that same vector - a plain unsynchronized data race.
Reproduced live while testing an unrelated ItemDataBase fix: rapid connect/
subscribe/disconnect cycles against the "createItem" subscription produced
a real segfault, confirmed via gdb - Socket::write() -> resetSocket()
dereferencing a corrupted EPoll reference, consistent with iterating a
vector that another thread was concurrently erasing from.
Wrapped the whole method body in a std::lock_guard on the same `lock`
mutex SubscriptionManager already has. Verified no reentrancy risk: nothing
reachable from processCommand() (including subscription handlers' own
process()/onSubscribe() implementations) calls back into processCommand(),
add(), or removeSessionSubscriptions() on the same thread, and `lock` is a
plain non-recursive std::mutex, so this can't deadlock.
Verified fix: the exact repro sequence that segfaulted the server twice
now runs clean, server stays up.
Not touched: Subscription::sendToAll() has no live callers (dead code);
dynamically `new`'d *OWNER subscriptions are still never delete()'d when
unpublished (a pre-existing leak, not a crash) - both out of scope here.
SubscriptionManager::processCommand()'s "invite" branch looked up its
target via getSessionByAlias(&invitee), where invitee is a std::string
local to that function call. getSessionByAlias(void*) compares by
pointer identity against each session's own alias - the address of a
brand-new local variable can never equal any of those, so the lookup
was guaranteed to return NULL every time, and the result was
dereferenced unconditionally right after. Any client sending a bare
"invite:" command crashed the entire process, not just its own
connection - same class of bug as the two GuildManager crashes fixed
earlier this session, just found afterward while documenting this
file's own mechanics.
This generic ServerCore layer has no name-based session lookup
available (that only exists as a Zephora-specific addition on top of
TCPServer), so this stops the crash with a null check rather than
attempting a real name-based fix, which would need a larger change.
Also includes this file's pending brace-style reformat (K&R -> Allman)
that was sitting uncommitted before this fix - bundled in rather than
left stranded, no functional change from that part.
subscriptions[id] via operator[] default-constructed (NULL) and
inserted an entry for any unregistered id, and the very next line
dereferenced it unconditionally - any client referencing a
subscription id that was never created (e.g. Zephora client sending
subscribe:invite for a channel nobody ever Publish()d) segfaulted the
entire process, not just that connection. Use find() and return
gracefully instead.
tls and master diverged from a July 2024 fork point and never
reconciled - tls carries ~19 months of real socket/TLS stability
fixes (uninitialized tlsInfo, TCPServer::connectionRequest()
unconditionally initializing TLS, the WANT_READ write busy-spin, the
console authentication fix from this session), while master
independently picked up a Subscription/SubscriptionManager rework
(MString-based ownership tracking: owner/ownership/selection/time
fields and a working subscriptionOwner() method) that tls never had.
Every file both branches touched was resolved by keeping tls's side
entirely (Command.h, Socket.cpp, TCPServer.h, TCPSession.cpp/h,
TCPSocket.h were pure reformatting on master's side, or an
alternate/superseded fix for a bug tls already fixed a different way -
e.g. master's TCPSession uuid/alias handling vs tls's existing
uuid_unparse-based fix), with two deliberate exceptions:
- TCPServer.cpp's constructor was missing the
commands.add(subscriptions, "publish"/"subscribe"/"catalog"/...)
wiring master had - without it, SubscriptionManager existed but was
completely unreachable through the normal command dispatch. Restored
on top of tls's TLS-aware constructor.
- SubscriptionManager.cpp's "invite" command handler had a real
incompatibility: master's version assigned into an `invitee` member
typed as coreutils::MString via a stray `(coreutils::MString *)`
cast that doesn't match the member's actual (non-pointer) type, and
called session.server.getSessionByAlias() against a TCPServer
reference - tls made TCPSession::server a pointer (part of removing
the separate TCPSession2 class), so that call site no longer
compiles as written. Took tls's cleaner local-variable version
instead (functionally identical - looks up a session by alias and
forwards an invite message) and removed the now-unused `invitee`
member from SubscriptionManager.h.
One additional compile fix needed after resolving conflicts:
Subscription::subscriptionOwners is an MString (master's ownership
design), but TCPSession::alias is void* (tls's identity mechanism,
unchanged and depended on elsewhere) - MString has an implicit
constructor from void* already used successfully elsewhere in this
same file (the "publish" handler's Subscription constructor call),
but operator== doesn't accept the same implicit conversion, so
removeSessionSubscriptions()'s ownership check needed an explicit
coreutils::MString(session.alias) construction to compile.
Also drops output/main (a stray checked-in compiled binary, already
removed on tls) and the doxygen docs/html output (already removed on
tls, was carried on master since the fork).
Verified: full ServerCore rebuild (./compile) succeeds with zero
errors or warnings beyond the one compile fix documented above.
Matt Arant