Brad Arant
2676aaf84c
A TLS sendData() failing with SSL_ERROR_WANT_READ (mid-handshake application data, or a post-handshake session ticket exchange) has nothing to do with write-readiness, but resetSocket() kept re-arming EPOLLWRNORM anyway - an already-writable TCP socket fires that immediately, so the retry loop spun the CPU as fast as it could until the connection was eventually dropped, without ever reading the peer's first command. Confirmed via a 15GB+ debug log from one stuck connection. sendNeedsRead() lets Socket now tell the difference and wait for a real EPOLLIN instead of busy-retrying.
263 lines
9.4 KiB
C++
263 lines
9.4 KiB
C++
#ifndef __Socket_h__
|
|
#define __Socket_h__
|
|
|
|
#include "ZString.h"
|
|
#include <queue>
|
|
#include <mutex>
|
|
#include <sys/epoll.h>
|
|
|
|
namespace core {
|
|
|
|
class EPoll;
|
|
|
|
///
|
|
/// Socket
|
|
///
|
|
/// The core component to managing a socket.
|
|
///
|
|
/// Hooks into the EPoll through the registration and unregistration process and provides a
|
|
/// communication socket of the specified protocol type. This object provides for all receiving
|
|
/// data threading through use of the EPoll object and also provides buffering for output data
|
|
/// requests to the socket.
|
|
///
|
|
/// A program using a socket object can request to open a socket (network or device) and
|
|
/// communicate through the streambuffer interface of the socket object.
|
|
///
|
|
/// The socket side of the Socket accepts EPOLLIN event and will maintain the data in a buffer
|
|
/// for the stream readers to read. A onDataReceived event is then sent with the data received in
|
|
/// the buffer that can be read through the stream. Only sockets that send events to epoll can be
|
|
/// used with this object.
|
|
///
|
|
/// When writing to the stream the data is written into a buffer and a EPOLLOUT is scheduled. Upon
|
|
/// receiving the EPOLLOUT event then the buffer is written to the socket output.
|
|
///
|
|
|
|
class Socket {
|
|
|
|
public:
|
|
|
|
///
|
|
/// Constructor
|
|
///
|
|
/// @param ePoll The EPoll socket descriptor.
|
|
/// @param text A title for this socket.
|
|
///
|
|
|
|
Socket(EPoll &ePoll, std::string text = "");
|
|
|
|
///
|
|
/// Destructor
|
|
///
|
|
|
|
virtual ~Socket();
|
|
|
|
///
|
|
/// Use the shutdown() method to terminate the socket connection and remove resources.
|
|
/// This method is provided to ensure that all destructors are called for all inherited
|
|
/// objects with a virtual destructor.
|
|
///
|
|
|
|
void shutdown(std::string text = "unknown");
|
|
|
|
///
|
|
/// setDescriptor establishes the file descriptor for the socket and registers the socket
|
|
/// on the EPoll controller. setDescriptor will invoke the onRegister() event.
|
|
///
|
|
|
|
void setDescriptor(int descriptor); ///<Set the descriptor for the socket.
|
|
|
|
int getDescriptor(); ///< Get the descriptor for the socket.
|
|
|
|
///
|
|
/// The event received from epoll is sent through the eventReceived
|
|
/// method which will parse the event and call the read and write
|
|
/// callbacks on the socket.
|
|
///
|
|
/// This method is called by the BMAEPoll object and should not be called
|
|
/// from any user extended classes unless an epoll event is being
|
|
/// simulated.
|
|
///
|
|
/// The return value of false will delete the socket object causing the destructors to run.
|
|
/// The return value of true will enable the socket on ePoll to receive more events.
|
|
///
|
|
|
|
bool eventReceived(struct epoll_event event, long long eventId); ///< Parse epoll event and call specified callbacks.
|
|
|
|
///
|
|
/// Write data to the socket.
|
|
///
|
|
|
|
int write(std::string data);
|
|
void write(char *buffer, int length);
|
|
|
|
void output(std::stringstream &out);
|
|
|
|
///
|
|
/// The onRegister method is called before the socket is registered with
|
|
/// ePoll so objects extending the Socket definition can initialize the socket
|
|
/// before receiving events. Evoked when the
|
|
/// descriptor is set using setDescriptor for the socket.
|
|
///
|
|
|
|
virtual void onRegister(); ///< Called before the socket has registered with the epoll processing.
|
|
virtual void onRegistered(); ///< Called after the socket has been registered with epoll processing.
|
|
|
|
virtual void onUnregister();
|
|
|
|
///
|
|
/// ePoll and socket communcation events will be stopped. The default method will
|
|
/// close the socket and clean up the connection. If this is overridden by an
|
|
/// extended object then the object should call this method to clean the socket up.
|
|
///
|
|
|
|
virtual void onUnregistered(); ///< Called when the socket has finished unregistering for the epoll processing.
|
|
|
|
bool needsToWrite();
|
|
|
|
bool reset = false;
|
|
|
|
volatile bool shutDown = false;
|
|
|
|
void enableSocket();
|
|
void disableSocket();
|
|
|
|
protected:
|
|
|
|
EPoll &ePoll; // The EPoll control object.
|
|
|
|
void setBufferSize(int length);
|
|
|
|
int getBufferSize();
|
|
|
|
///
|
|
/// The onConnected method is called when the socket is ready to communicate.
|
|
/// Writing to the socket can begin on this call to initiate a contact with the
|
|
/// remote device.
|
|
///
|
|
|
|
// virtual void onConnected(); ///< Called when socket is open and ready to communicate.
|
|
|
|
///
|
|
/// The connectionRequest method is called to instantiate a new session handler object
|
|
/// and form a new connection to a listening server.
|
|
///
|
|
|
|
virtual void connectionRequest();
|
|
|
|
///
|
|
///
|
|
///
|
|
|
|
// virtual void onDisconnected(); ///< Called when socket is closing and no longer ready to communicate.
|
|
|
|
///
|
|
/// The onDataReceived method is called when the socket has received an event from
|
|
/// epoll and there is data ready to be read from the socket. The default handler
|
|
/// will pull the data and put it into the streambuf for the socket. EPOLLIN
|
|
///
|
|
/// @param data the data that has been received from the socket.
|
|
///
|
|
|
|
virtual void onDataReceived(std::string data); ///< Called when data is received from the socket.
|
|
|
|
///
|
|
///
|
|
///
|
|
|
|
virtual void onDataReceived(coreutils::ZString &data);
|
|
|
|
///
|
|
/// receiveData will read the data from the socket and place it in the socket buffer.
|
|
/// TLS layer overrides this to be able to read from SSL.
|
|
///
|
|
|
|
virtual void receiveData(coreutils::ZString &buffer);
|
|
|
|
///
|
|
/// sendData writes data to the socket descriptor. TLS layer overrides this to
|
|
/// write through SSL instead of a raw socket write.
|
|
///
|
|
/// @return The number of bytes written, or -1 on error.
|
|
///
|
|
|
|
virtual int sendData(const char *data, int length);
|
|
|
|
///
|
|
/// Overridden by TCPSocket to report whether the last sendData()
|
|
/// call failed specifically because the TLS layer needs a READ to
|
|
/// make progress (SSL_ERROR_WANT_READ), as opposed to any other
|
|
/// reason - see waitingOnReadToWrite's own doc comment for why
|
|
/// writeSocket()/resetSocket() need to know this. Always false for
|
|
/// a plain (non-TLS) socket, which has no such distinction.
|
|
///
|
|
|
|
virtual bool sendNeedsRead() { return false; }
|
|
|
|
private:
|
|
|
|
std::string text;
|
|
int descriptor = -1;
|
|
std::mutex outlock;
|
|
bool readHangup = false;
|
|
volatile bool inHandler = false;
|
|
|
|
//-------------------------------------------------------------------------------------
|
|
// the writeSocket is called when epoll has received a write request for a socket.
|
|
// Writing data to this socket is queued in the streambuf and permission is requested
|
|
// to write to the socket. This routine handles the writing of the streambuf data
|
|
// buffer to the socket.
|
|
//-------------------------------------------------------------------------------------
|
|
|
|
void writeSocket();
|
|
|
|
// int_type underflow();
|
|
// int_type uflow();
|
|
// int_type pbackfail(int_type ch);
|
|
// streamsize showmanyc();
|
|
|
|
char *buffer; // This is a pointer to the managed buffer space.
|
|
int length; // This is the length of the buffer.
|
|
|
|
std::queue<std::string> fifo;
|
|
|
|
// Counts consecutive writeSocket() calls that sent zero bytes for
|
|
// the SAME queued fifo entry (SSL_ERROR_WANT_READ from
|
|
// TLS::sendData() is the confirmed real case - see writeSocket()'s
|
|
// own doc comment). resetSocket() re-arms EPOLLWRNORM whenever the
|
|
// fifo is non-empty regardless of WHY the last send failed, and an
|
|
// already-writable TCP socket (the common case) makes EPOLLWRNORM
|
|
// fire again essentially immediately - so a send that keeps
|
|
// failing for a reason unrelated to socket writability (like SSL
|
|
// needing a read first) spins this cycle as fast as the CPU can
|
|
// run it, forever, with zero actual progress. Confirmed live: one
|
|
// stuck connection generated a 55GB, then a 12.9GB debug log
|
|
// before being noticed, at roughly 20 log lines per millisecond.
|
|
// Reset to 0 on any successful send (even a partial one - real
|
|
// progress happened); past the threshold, the connection is
|
|
// clearly never going to un-stick itself and gets shut down.
|
|
int consecutiveWriteStalls = 0;
|
|
|
|
// Set by writeSocket() when the last failed send specifically
|
|
// needed a read to proceed (sendNeedsRead() true) rather than
|
|
// being genuinely stalled on write-readiness. resetSocket() skips
|
|
// adding EPOLLWRNORM to the re-armed event mask while this is
|
|
// true - re-arming it anyway just spins the CPU as fast as it can
|
|
// loop, since an already-writable TCP socket keeps firing
|
|
// EPOLLWRNORM immediately while the actual blocker (more bytes
|
|
// from the peer) has nothing to do with write-readiness at all.
|
|
// Cleared on any EPOLLIN event (see eventReceived()) so the write
|
|
// gets a fresh attempt via that same event's own trailing
|
|
// resetSocket() call, once there's real reason to expect progress.
|
|
bool waitingOnReadToWrite = false;
|
|
|
|
void resetSocket();
|
|
|
|
std::mutex lock;
|
|
|
|
};
|
|
|
|
}
|
|
|
|
#endif
|
|
|