Merge branch 'master' of git.barant.com:barant/ServerCore

TCPServer.h

@@ -207,7 +207,33 @@ namespace core {
       // able to synchronize against the exact same mutex
       // connectionRequest()/removeFromSessionList() already use, rather
       // than reading a shared_ptr vector with no locking of its own.
-      std::mutex lock;
+      //
+      // Was a plain std::mutex - self-deadlocked the accepting thread on
+      // EVERY single connection to ConsoleServer (confirmed live via a
+      // gdb thread dump caught mid-stall): connectionRequest() locks
+      // `lock`, then calls accept(), which constructs the new session -
+      // whose base Socket constructor logs its own creation via
+      // coreutils::Log, whose destructor calls back into
+      // ConsoleServer::logSend() (exactly the synchronization this was
+      // designed for, per the comment above) - which tries to lock this
+      // SAME mutex again, on the SAME thread that already holds it.
+      // std::mutex is not reentrant, so that thread deadlocked
+      // permanently, every time, without ever completing the accept -
+      // this is why the admin console never responded to anything.
+      // Since all of a server's listening sockets share one fixed-size
+      // EPoll worker-thread pool, each such deadlocked thread is gone
+      // for good - enough accumulated connection attempts (even
+      // sporadic ones, e.g. internet-wide port scanners probing every
+      // port) eventually exhausts every worker thread, at which point
+      // the WHOLE server - every listening socket, not just the
+      // console - stops accepting anything new. This is the actual
+      // root cause of BARANTMail's own multi-day-recurring "all ports
+      // simultaneously stop accepting connections" incidents - nothing
+      // to do with DNS/SPF lookup timing, which was a real, separate,
+      // worthwhile fix but not this bug. recursive_mutex lets the same
+      // thread re-enter safely (identical behavior to std::mutex for
+      // every other thread) without changing the locking design at all.
+      std::recursive_mutex lock;
 
     public:
       ///