Merging upstream TCPServer.h changed lock from std::mutex to
std::recursive_mutex (the ConsoleServer self-deadlock fix); this call
site still hardcoded std::lock_guard<std::mutex>, which no longer
compiles since lock_guard cannot bind to a different mutex type. Uses
the same class-template-argument-deduction form every other lock_guard
in this file already uses.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TCPServer accepted a connection and never revisited it again once it
stopped generating epoll events - a connection that completes the TCP
handshake and then goes permanently silent (an incomplete TLS
handshake, a port scanner/prober) sat in sessions forever, one
TCPSession + one fd each. sweepStaleSessions() now drops any session
that has gone 30+ seconds without a single byte of real data ever
passing through onDataReceived(), run each time a new connection
arrives rather than needing a dedicated timer. hasReceivedData latches
permanently true on a sessions first real byte, so an ordinary
connected-but-briefly-idle client is never at risk.
Also raised listen()s backlog from 20 to 128, so a burst of connection
attempts has more headroom before the kernel starts dropping SYNs.
Verified live on zephora.barant.com: a silent connection held open was
reaped within the grace window while a real concurrent connection was
left untouched, with the service remaining healthy throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- INotify/Thread epoll dispatch was silently dropping every live
core::INotify event (an isKnownINotify registry fixes it) - this had
broken mail delivery for over a week, masked by routine restarts.
- TCPServer::lock self-deadlocked every admin-console connection via a
plain std::mutex being re-entered; switched to recursive_mutex.
- TLS.cpp/h: SNI-based certificate selection (addSNIContext), the
groundwork for multi-tenant TLS (qualityiron.vegas alongside
barant.com on one APIServer process).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every crash root-caused earlier tonight (Subscription::subscribers,
TCPServer::sessions, the Socket::write()/eventReceived() re-arm race) was
the same underlying shape: some thread calling a method - or Thread::run()
calling delete - on a TCPSession* while another thread concurrently freed
that same object, because nothing actually guaranteed the object stayed
alive for the duration of a call. Mutex-based patches at each individual
call site kept narrowing the window without closing it (confirmed live,
repeatedly, via AddressSanitizer under concurrent load - each fix produced
a new, different crash signature at the same underlying race), and adding
more locks around every write() call also works against the fifo-based
design's own goal: no thread should have to block just to hand a message
to another thread's session.
TCPSession now derives from std::enable_shared_from_this<TCPSession> and
exposes pin() - a safe upgrade from a stored raw pointer to a real,
ownership-extending shared_ptr, returning nullptr instead of throwing once
the object's real owner has let go. TCPServer::sessions is now the one
real strong-owning shared_ptr<TCPSession> per live session; every other
place that stores a TCPSession* (Subscription::subscribers, Zephora's own
sessionsByPlayerName, GroupObject's membership list, etc.) is just a
non-owning observer and must pin() before touching the object.
A second, distinct problem: Thread::run()'s epoll dispatch starts from a
raw void* of unknown validity. An early version of this fix used
dynamic_cast<TCPSession*> to identify session objects before calling
pin() - confirmed live via AddressSanitizer that dynamic_cast itself
segfaults on a stale pointer, since it has to read the (potentially freed)
object's own vtable to work at all. TCPSession::tryGet()/TCPServer::
isKnownListeningSocket() replace that: both look the raw pointer up as a
key in a registry, never dereferencing it for the check itself - safe to
call on any value, live session, immortal listening socket, or fully
freed garbage alike.
Verified via an AddressSanitizer-instrumented build under sustained
concurrent load (up to 15-way parallel connect/subscribe/create/disconnect
cycles, multiple rounds, 100+ total connections) with zero errors, then
against the real production build with a genuine 500-connection load test
(463/500 succeeded; the rest hit the process's own file-descriptor
ulimit under peak burst, an unrelated and separately deferred tuning
matter, not a crash).
Files touched: TCPSession.h (enable_shared_from_this, pin(), the tryGet()
registry), TCPServer.h/.cpp (sessions as shared_ptr, isKnownListening
Socket() registry, lock made protected for subclasses), Thread.cpp (the
whole dispatch rewrite), Subscription.cpp/SubscriptionManager.cpp (every
subscriber write() site now pin()s first), ConsoleServer.cpp (logSend()
was iterating `sessions` with no lock at all - fixed alongside the type
change it needed anyway).
Socket::write() unconditionally called resetSocket() (via EPOLL_CTL_MOD)
from whatever thread called it, whenever it could grab `lock` - including
while this same socket's OWN eventReceived() call was still mid-flight on
another thread (past receiveData()'s lock_guard, but before its own
trailing resetSocket()/return). Re-arming makes the fd immediately
eligible for a third thread to pick up a fresh event and, if it decides
to shut the connection down, delete the object outright (Thread::run()
does exactly that on a false return) - while the original thread is still
touching it. A real heap-corruption crash inside libc's malloc internals
was confirmed live via dmesg under concurrent load, consistent with
exactly this: a socket freed while another thread still held a reference
to it.
inHandler existed as a flag already but only fed a diagnostic log line -
never actually gated anything. Made it a real std::atomic<bool> (was
`volatile`, which gives no actual atomicity or cross-thread visibility)
and used it two ways:
- write() now skips its own resetSocket() call entirely while inHandler
is true - that call's own eventReceived() will re-arm once it's
actually done, seeing the fifo entry write() just queued.
- eventReceived() itself was clearing inHandler well before its own
resetSocket() call (several statements up, right after EPOLLIN/
EPOLLWRNORM handling) - moved the clear to immediately precede that
call, closing the exposed window down to the minimum achievable
without restructuring how write() re-arms sockets altogether.
This is a distinct bug from the two subscriber/session-list races fixed
earlier tonight (SubscriptionManager.cpp, TCPServer.cpp) - found while
stress-testing those fixes turned up a third, different crash signature
(heap corruption, not the null-EPoll-reference pattern the earlier two
produced).
connectionRequest()/removeFromSessionList() already lock `lock` around
every mutation of `sessions` (a session accepting or disconnecting), but
every read of it - sendToAll() (all three overloads), getSessionByAlias(),
and the debug-console processCommand() - iterated it with no lock at all.
Exact same bug shape as Subscription::subscribers, fixed earlier today in
SubscriptionManager.cpp, just in a completely separate container.
Confirmed live via dmesg: a real segfault at a stable, deterministic
instruction offset inside the binary, hit twice across separate process
launches tonight - Socket::write() -> resetSocket() dereferencing a null
EPoll reference, the same crash signature as the earlier Subscription fix,
consistent with iterating `sessions` while another thread concurrently
erased and freed one of its entries. sendToAll() backs most real gameplay
broadcasts (combat, position, chat), so this is very likely the actual
dominant cause of tonight's crash - not just the ItemDataBase-specific
subscriber list the earlier fix addressed.
Took the existing `lock` mutex around all five read sites. No new
reentrancy risk: SubscriptionManager's "invite" verb already calls
getSessionByAlias() while holding SubscriptionManager::lock, so the two
mutexes now nest in one fixed order (SubscriptionManager::lock, then
TCPServer::lock) - nothing acquires them in the reverse order anywhere
in this codebase, so no AB-BA deadlock risk.
processCommand() read/mutated the `subscriptions` map and individual
Subscription objects' `subscribers` vectors (via subscribe()/unsubscribe()/
event()/the "publish" branch's own insert()) without taking `lock` at all,
even though add() and removeSessionSubscriptions() already do take it for
the exact same state. removeSessionSubscriptions() runs from ~TCPSession(),
on whatever thread happens to be tearing down a *different* connection, so
one thread erasing a session from a Subscription's subscribers vector could
run concurrently with another thread iterating (event()) or appending to
(subscribe()) that same vector - a plain unsynchronized data race.
Reproduced live while testing an unrelated ItemDataBase fix: rapid connect/
subscribe/disconnect cycles against the "createItem" subscription produced
a real segfault, confirmed via gdb - Socket::write() -> resetSocket()
dereferencing a corrupted EPoll reference, consistent with iterating a
vector that another thread was concurrently erasing from.
Wrapped the whole method body in a std::lock_guard on the same `lock`
mutex SubscriptionManager already has. Verified no reentrancy risk: nothing
reachable from processCommand() (including subscription handlers' own
process()/onSubscribe() implementations) calls back into processCommand(),
add(), or removeSessionSubscriptions() on the same thread, and `lock` is a
plain non-recursive std::mutex, so this can't deadlock.
Verified fix: the exact repro sequence that segfaulted the server twice
now runs clean, server stays up.
Not touched: Subscription::sendToAll() has no live callers (dead code);
dynamically `new`'d *OWNER subscriptions are still never delete()'d when
unpublished (a pre-existing leak, not a crash) - both out of scope here.
SubscriptionManager::processCommand()'s "invite" branch looked up its
target via getSessionByAlias(&invitee), where invitee is a std::string
local to that function call. getSessionByAlias(void*) compares by
pointer identity against each session's own alias - the address of a
brand-new local variable can never equal any of those, so the lookup
was guaranteed to return NULL every time, and the result was
dereferenced unconditionally right after. Any client sending a bare
"invite:" command crashed the entire process, not just its own
connection - same class of bug as the two GuildManager crashes fixed
earlier this session, just found afterward while documenting this
file's own mechanics.
This generic ServerCore layer has no name-based session lookup
available (that only exists as a Zephora-specific addition on top of
TCPServer), so this stops the crash with a null check rather than
attempting a real name-based fix, which would need a larger change.
Also includes this file's pending brace-style reformat (K&R -> Allman)
that was sitting uncommitted before this fix - bundled in rather than
left stranded, no functional change from that part.
subscriptions[id] via operator[] default-constructed (NULL) and
inserted an entry for any unregistered id, and the very next line
dereferenced it unconditionally - any client referencing a
subscription id that was never created (e.g. Zephora client sending
subscribe:invite for a channel nobody ever Publish()d) segfaulted the
entire process, not just that connection. Use find() and return
gracefully instead.
shutdown() only sets shutDown=true and expects the fifo to drain
normally before eventReceived()'s keepAlive check
(!shutDown || (fifo.size() > 0 && !readHangup)) lets the connection
close - correct for a graceful QUIT/LOGOUT reply that just needs a few
more successful writes to flush, but exactly backwards for the give-up
path: the whole reason that path was reached is that the front of the
fifo can never be sent. Leaving it queued meant keepAlive stayed true
forever, resetSocket() kept re-arming EPOLLWRNORM because fifo wasn't
empty, the very next write attempt hit the identical stall
immediately, and shutdown() got called again - forever, at whatever
rate the kernel could fire EPOLLWRNORM.
Confirmed live on a real internet-facing server (Zephora's game
server, an ordinary port-scanner connection was enough to trigger it):
one socket logged "consecutive writes made zero progress - giving up
and shutting down" 2.4 million+ times in under 4 milliseconds,
pegging a worker thread at 100% CPU indefinitely. The give-up path now
clears the undeliverable fifo entry before calling shutdown(), so
keepAlive correctly evaluates to just !shutDown (now false) on the
next cycle and the connection actually tears down.
CommandList::processRequest() upper-cases the incoming keyword before
looking it up, but add() stored whatever case the caller passed in
verbatim - any registration not already uppercase could then never be
matched by any input at all, since upper-casing a non-uppercase string
cannot produce a match against a non-uppercase key. This was invisible
on BARANTMail/APIServer (their own commands - SMTP, POP3/IMAP, the
admin console - all happen to already be registered uppercase) but
confirmed, via a full Zephora game client/server protocol comparison,
to make literally every Zephora game command unreachable, including
the base TCPServer's own publish/subscribe/event/etc. registrations
that the whole subscription layer depends on. Folding at add() time
keeps registration and lookup permanently in sync regardless of
caller case; already-uppercase callers are unaffected (idempotent).
The merge commit (3609ed6) used git add -A to handle the large number
of deletes/renames from conflict resolution, which also picked up 22
untracked *.pre-* files - local safety-copy backups made before
editing tracked files, never meant to be committed. Untracked (kept on
disk) rather than deleted, and added *.pre-* to .gitignore so this
does not recur.
tls and master diverged from a July 2024 fork point and never
reconciled - tls carries ~19 months of real socket/TLS stability
fixes (uninitialized tlsInfo, TCPServer::connectionRequest()
unconditionally initializing TLS, the WANT_READ write busy-spin, the
console authentication fix from this session), while master
independently picked up a Subscription/SubscriptionManager rework
(MString-based ownership tracking: owner/ownership/selection/time
fields and a working subscriptionOwner() method) that tls never had.
Every file both branches touched was resolved by keeping tls's side
entirely (Command.h, Socket.cpp, TCPServer.h, TCPSession.cpp/h,
TCPSocket.h were pure reformatting on master's side, or an
alternate/superseded fix for a bug tls already fixed a different way -
e.g. master's TCPSession uuid/alias handling vs tls's existing
uuid_unparse-based fix), with two deliberate exceptions:
- TCPServer.cpp's constructor was missing the
commands.add(subscriptions, "publish"/"subscribe"/"catalog"/...)
wiring master had - without it, SubscriptionManager existed but was
completely unreachable through the normal command dispatch. Restored
on top of tls's TLS-aware constructor.
- SubscriptionManager.cpp's "invite" command handler had a real
incompatibility: master's version assigned into an `invitee` member
typed as coreutils::MString via a stray `(coreutils::MString *)`
cast that doesn't match the member's actual (non-pointer) type, and
called session.server.getSessionByAlias() against a TCPServer
reference - tls made TCPSession::server a pointer (part of removing
the separate TCPSession2 class), so that call site no longer
compiles as written. Took tls's cleaner local-variable version
instead (functionally identical - looks up a session by alias and
forwards an invite message) and removed the now-unused `invitee`
member from SubscriptionManager.h.
One additional compile fix needed after resolving conflicts:
Subscription::subscriptionOwners is an MString (master's ownership
design), but TCPSession::alias is void* (tls's identity mechanism,
unchanged and depended on elsewhere) - MString has an implicit
constructor from void* already used successfully elsewhere in this
same file (the "publish" handler's Subscription constructor call),
but operator== doesn't accept the same implicit conversion, so
removeSessionSubscriptions()'s ownership check needed an explicit
coreutils::MString(session.alias) construction to compile.
Also drops output/main (a stray checked-in compiled binary, already
removed on tls) and the doxygen docs/html output (already removed on
tls, was carried on master since the fork).
Verified: full ServerCore rebuild (./compile) succeeds with zero
errors or warnings beyond the one compile fix documented above.
ConsoleSession's login flow never actually checked anything: the
entered username was discarded and the password was never compared,
so any input (including blank) reached the command prompt. Confirmed
live via an unauthenticated login against BARANTMail's admin console
on port 1027, which was reachable externally. ConsoleServer now takes
an optional credentialChecker callback consulted by the WAIT_PASSWORD
state; omitting it preserves the old accept-anything behavior for any
other use of this class.
Separately, the console never prompted until the client's first line
arrived - that line was silently discarded just to kick the state
machine off, shifting every subsequent line one slot early. Harmless
under the old accept-anything login, but it broke real credential
checking outright (the typed password was compared against the typed
username, etc). ConsoleSession::onConnected() now starts the state
machine immediately on connect instead.
CommandList::processCommand() was a no-op stub - the intended
"typing help lists the registered commands" body was commented out,
and even referenced a Command::getName() that was never added to the
base class. The map's own key from add() already is the registered
name, so this just walks it directly.
A TLS sendData() failing with SSL_ERROR_WANT_READ (mid-handshake
application data, or a post-handshake session ticket exchange) has
nothing to do with write-readiness, but resetSocket() kept re-arming
EPOLLWRNORM anyway - an already-writable TCP socket fires that
immediately, so the retry loop spun the CPU as fast as it could
until the connection was eventually dropped, without ever reading
the peer's first command. Confirmed via a 15GB+ debug log from one
stuck connection. sendNeedsRead() lets Socket now tell the
difference and wait for a real EPOLLIN instead of busy-retrying.
Thread/EPoll lifecycle:
- Thread::~Thread() now actually joins/frees its worker thread (was
empty - the OS thread kept running against already-freed EPoll/Socket
objects on any early teardown, a confirmed "double free" crash)
- EPoll::~EPoll() calls stop() if never explicitly stopped, for the
same reason
- epoll_wait() now uses a 1s timeout instead of blocking forever, so a
worker thread can actually notice isStopping() and let itself be
joined
- EPoll::processCommand() iterated threads by value, silently
double-freeing the real vector's Thread the moment each copy went
out of scope
- ThreadScope gained onThreadStart()/onThreadEnd() hooks (actually
wired up now - the constructor that was supposed to store the scope
discarded it) for per-thread C-library init (e.g. MySQL client)
Socket/TCPSocket/TLS:
- Socket::writeSocket(): give up and shut down a connection after 50
consecutive zero-progress sends instead of retrying forever -
SSL_ERROR_WANT_READ on a write needs a READ to make progress, but
resetSocket() re-arms EPOLLWRNORM regardless, which an
already-writable TCP socket satisfies immediately - a real stuck
connection generated 55GB, then 12.9GB of debug log this way before
being caught
- Socket::writeSocket() now honors sendData()'s actual return value
(previously discarded, entry popped unconditionally after one call) -
a large response could get silently truncated past what fit in one
write()/SSL_write() call
- TLS::registerSocket()/sendData()/receiveData(): `initialized` now
only goes true once SSL_set_fd() has actually succeeded, and
sendData()/receiveData() check it first - fixes a real SIGSEGV where
a new session's own queued greeting could reach SSL_write() on a
different epoll thread before this session's handshake setup had run
- TCPServer::connectionRequest() now performs the TLS handshake ON THE
ACCEPTED SESSION, not the listening server itself (was building SSL
state on the wrong object, leaving every session's own `ssl`
uninitialized)
- TCPServer::receiveData() always uses the plain (non-TLS) read path -
the listening socket is never itself a TLS peer
- TCPSocket's block-mode literal reading used an unset `blockLength`
field instead of `blockSize`, and could busy-spin forever on a
multi-packet literal
- TCPSocket line scanning replaced an unbounded strcspn() (assumes a
NUL-terminated buffer; read past a non-NUL-terminated receive buffer
into adjacent heap memory) with a bounds-checked memchr() scan
Protocol/session handling:
- CommandList::processRequest() now case-folds the command keyword
before lookup - SMTP/POP3/IMAP commands are spec-required to be
case-insensitive; a client sending "Mail From:" instead of
"MAIL FROM:" previously failed to dispatch at all
- TCPSession::protocol() no longer drops blank lines while a handler
has grabbed input (e.g. __SMTP_DATA reading a message body, an IMAP
APPEND literal) - those handlers depend on seeing blank lines as
real content, not commands; this silently corrupted any message body
containing a blank line
- TCPSession's uuid alias fixed: streaming a raw uuid_t via operator<<
triggered libstdc++'s unsigned-char* overload (strlen() on 16 binary
bytes, a real stack-buffer-overflow read), and `alias` pointed at a
stack-local that was already destroyed by the time the constructor
returned
- INotify::processEvent() only treats event->name as a NUL-terminated
string when event->len > 0 (per inotify(7)) instead of unconditional
strlen() on a field the kernel doesn't guarantee is even present -
a confirmed heap-buffer-overflow read that had been misdiagnosed as
a MySQL thread-safety issue
TCPSocket's SSL_CTX*-taking constructor - the one every accepted
TCPSession actually goes through (TCPSession(EPoll&, TCPServer*, ...)
passes server->ctx into it) - never initialized the `tlsInfo` member,
and TCPSocket.h gave it no default either. Every session's `tlsInfo`
was uninitialized garbage.
receiveData()/sendData() key their TLS-vs-plain dispatch on
`if(tlsInfo)`. Since that garbage value is non-zero often enough on a
real stack, sessions on a plain, non-TLS server were intermittently
routed through TLS::receiveData()/sendData() (SSL_read()/SSL_write())
against an also-uninitialized SSL*, segfaulting inside libssl. Confirmed
via dmesg: repeated "general protection fault ... in libssl.so.3",
signal 11, non-deterministic timing consistent with depending on
whatever garbage happened to be on the stack for a given session.
Fixed by dispatching on `ctx` instead of `tlsInfo`. `ctx` (TLS.h) now
defaults to NULL (previously also uninitialized) and is the value that
actually gets set for real in TLS::createContext() and correctly
inherited by every session via TLS(SSL_CTX*) - unlike `tlsInfo`, which
is only ever set on the listening TCPServer's own constructor and never
propagated to sessions at all. For the previously-working TLS-enabled
case this is a no-op: ctx and tlsInfo were always both set (or both
unset) together on the listening server, so the observable behavior
there is unchanged - only the previously-undefined session-level
behavior is fixed.
Verified live: restarted the APIServer test instance (a plain, non-TLS
TCPServer - the exact configuration that was crashing), ran 100
sequential requests with no failures, and confirmed no new libssl
segfaults in dmesg since the fix (three separate pre-fix crash
incidents are all still visible in the dmesg history for comparison).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
connectionRequest() had a literal "if(true) { registerSocket(...);
acceptSocket(); }" guarding the TLS handshake setup for every accepted
connection, regardless of whether this TCPServer was ever given a
TLSInfo. TLS::createContext() only sets `ctx` when tlsInfo is non-NULL
(correctly guarded there) - so for any plain, non-TLS TCPServer, `ctx`
is left uninitialized, and this unconditional call reached
TLS::registerSocket() -> SSL_new(ctx) with garbage, segfaulting the
whole process on the very first connection.
Found while building APIServer (a new non-TLS HTTP server on top of
TCPServer) - confirmed via gdb (SIGSEGV inside SSL_new, called from
TLS::registerSocket, called from this exact line) that this affects ANY
non-TLS ServerCore-based server's first accepted connection, not
anything specific to the new code being tested against it.
Fixed by checking `tlsInfo` (already a TCPSocket-protected member this
class inherits) instead of the unconditional `true` - matches the guard
TLS::createContext() already uses correctly for the same distinction.
Verified live: before the fix, a non-TLS ApiServer segfaulted on the
first curl request every time; after, GET/HEAD/POST requests across
several real routes complete correctly and the server stays up for
repeated requests, including one that deliberately throws (confirming
the crash is really gone, not just less frequent).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous commit picked up CRLF line endings from a Windows-side
working copy used during development. Converting to plain LF to match
the rest of this Linux-native repo - content unchanged, this is purely
a line-ending normalization (932 insertions/932 deletions, 1:1).
TCPSocket inherits from both Socket and TLS, both declaring receiveData()
as unrelated sibling methods - only Socket::receiveData was virtual, so
TLS::receiveData (SSL_read) was never reached regardless of TLS config,
and even when reached its result was never forwarded to onDataReceived
(that call was commented out). Added an explicit TCPSocket::receiveData
override dispatching to TLS::receiveData or Socket::receiveData based on
tlsInfo, and completed TLS::receiveData to actually return/forward the
decrypted bytes.
Write path did not exist at all. Added Socket::sendData() as a new
virtual hook mirroring receiveData, routed Socket::writeSocket() through
it instead of a raw ::write(), added TLS::sendData() (SSL_write, same
error-handling style as the existing TLS methods), and TCPSocket::sendData()
dispatching the same way the read side does.
Also fixed: a queued write in writeSocket() was being popped off the fifo
as if sent even when shutDown was true and nothing was actually written;
removed a dead empty if(tlsInfo){} stub in onDataReceived, no longer
needed now dispatch happens one level up; replaced leftover debug
std::cout lines with proper Log() calls matching the rest of the file.
Verified via full local compile: all 20 .cpp files build clean, link
into libServerCore.a, and nm confirms the new/fixed symbols are correctly
defined in the right object files.
