JString::operator=(ZString) already stripped whitespace between JSON
structural tokens before parsing, but JFile constructed via MFile file-
loading bypassed that path entirely, so any pretty-printed/hand-edited
JSON file silently failed every locate()/exists()/get*() call. Widen
removeWhitespace() to protected and call it from JFile constructor.
Stage 2 of the APIServer plan. This codebase has only ever sat behind a
CGI gateway - IMFRequest/IMFResponse/IMFMessage predate the
StreamReader/Parser rebuild and are stubs (IMFResponse's header-emission
calls addHeader()/output(), neither of which exist on the class;
IMFRequest is a fragile non-resumable 3-token line split). Nothing in
this repo has ever actually parsed a raw HTTP request off a socket.
HTTPRequest : public Parser fills that gap the same way every other
Parser here is built: REQUEST_LINE -> HEADERS -> BODY -> DONE, chaining
LineParser (request line) and IMFHeader (headers, reused as-is - HTTP
header syntax is IMF-derived, nothing email-specific in IMFHeader
itself) via checkParse() over one shared StreamReader, exactly the
pattern CGIFormData already established for chaining IMFHeader/
IMFBoundary. The BODY state dispatches on Content-Type as CGIFormData
itself does for CGI bodies - application/x-www-form-urlencoded and
multipart/form-data delegate directly to CGIFormData (reused, not
reimplemented), anything else is captured as a raw body (e.g. JSON).
Query string is split off the path before anything else sees it.
getHeader() does a case-insensitive lookup (HTTP header names are
case-insensitive per RFC 7230, unlike IMFHeader's own std::map, which
compares byte-for-byte) since a route handler constantly needs this and
getting it wrong would be a silent framing bug, not a cosmetic one.
The boundary-extraction code mirrors __jet.cpp's own fix exactly (same
cursor-mutation trap: ZString::find() leaves the searched string's
cursor at the match, and CGIFormData's constructor copy-constructs its
own contentType member from whatever's passed in, inheriting that same
moved cursor unless reset() first - would otherwise silently miss its
own startsWith() dispatch checks).
HTTPResponse is a small, new, output-only builder (status line + a real
headers container + body, Content-Length auto-computed unless set
explicitly) - deliberately not patched from IMFResponse, which has
nothing safe to extend.
v1 limitations, flagged not fixed (matching the standard already set by
other Parser subclasses here for known-but-out-of-scope gaps): chunked
Transfer-Encoding isn't supported, and repeated header names last-wins
(inherited from IMFHeader's own std::map).
Verified live via testing/http_test.cpp (34 assertions, new - now wired
into testing/compile): byte-at-a-time GET with a query string and
case-insensitive header lookup; a GET with no query string; a POST with
an urlencoded body deliberately split mid-body (not just mid-header)
across two feed() calls, confirming the CGIFormData delegation resumes
correctly; a raw JSON POST body; testing/http_request.data (a real
fixture, previously unused - normalized from its stored LF line endings
to real CRLF, since HTTP requires CRLF and IMFHeader enforces that
strictly) fed byte-at-a-time; and HTTPResponse::serialize() both with an
auto-computed and an explicitly-set Content-Length (no duplicate
header). Also reran the full existing testing/compile suite
(zstring/mstring/jstring/mfile/jfile) against the rebuilt libCoreUtils.a
to confirm no regression from the two new files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
operator=(MString)/operator=(ZString)/operator=(std::string)/operator=(const
char*)/operator=(char) all replaced the buffer's content via setSize()+
memcpy() but never reset cursor back to the start - setSize() preserves
the *offset* the cursor was already at (correct for its own use, growing
a buffer while keeping a read position stable), so a full content
replacement left the cursor sitting at whatever offset it had into the
*previous* content, not the new content. operator=(double)/operator=(int)
already did `cursor = data;` after the same setSize()+memcpy() pattern,
confirming this was the intended behavior everywhere - just missing from
the other five overloads.
This is invisible to JetCore's interpreter, which never reassigns an
already-read-from MString (each tag is a freshly parsed C++ object every
time its container is scanned, so a keyword's MString is never both read
and then reassigned on the same instance). It surfaced in a prototype JET
compiler (JetCore/jetc) built this session, which intentionally reuses
one persistent keyword slot across loop iterations instead of
reconstructing everything from text every pass - exactly the case this
bug needed. A <while expr="..."> loop reusing a slot for its condition
worked for two iterations (matching a decoy pattern: the second
assignment happened to land the stale cursor at the exact end of the
new, same-length content) and threw "operand is not valid" on the third,
since a keyword reassigned inside the loop body left the cursor already
at eod() from the read.
Verified via jstring_test/mstring_test/zstring_test/parser_test (no
regressions), JetCore's tests/posttest.sh and the two-process JSON
session persistence test (no regressions), and the jetc prototype's two
demo templates - both now produce byte-identical output to jet-2.0's
interpreter, including a <while> loop that mutates and rereads the same
keyword slot every iteration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
JFile.cpp: JFile and MFile both "virtual public MString" a single shared
base, so a virtual base is initialized only by the *most derived* class's
constructor. JFile's own constructor never mentions MString, so
JString's own "{}" default initializer never ran - a JFile backed by a
new or nonexistent file started as a raw empty string instead of a valid
empty JSON object. Every subsequent locate()/set() call then silently
misbehaved (no leading "{" for ifNext("{") to match, so writes landed at
offset 0 with no braces or separators - fields ended up prepended in
reverse order, concatenated with no delimiters at all). Fixed by seeding
"{}" via append (matching MFile's own load pattern) whenever the loaded
content is empty.
MString.cpp: several ZString-source copy paths (the ZString/MString copy
constructors, both assignment operators, operator<<(ZString), insert(),
write()) called zstring.getData() (the object's absolute start) paired
with zstring.getLength() (which is documented/implemented as "remaining
bytes from cursor to end", i.e. length - (cursor - data)). Pairing an
absolute start with a cursor-relative length silently corrupts any
cursor-advanced source: the copy starts from the wrong (too-early)
position but is truncated by exactly the amount the cursor had already
advanced, keeping unwanted leading bytes while dropping trailing ones.
Found via the CGIFormData-style resumability testing pattern used
elsewhere this session, triggered by JetCore's new "@name" session
variable feature stripping a sigil prefix via ifNext() before streaming
the remainder into an MString. Fixed by pairing zstring.getCursor()
(cursor-relative start) with zstring.getLength() consistently everywhere
this pattern appeared.
Verified via the existing jstring_test/mstring_test/zstring_test/
parser_test suites (all pass, no output changes) plus a new isolated
JFile round-trip test proving persistence across two separate process
instances sharing one file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
JString::Proxy offers several implicit conversion operators (ZString,
MString, std::string, double, bool) so a single $[...]-style access can
be used wherever any of those types is expected. That flexibility made
"MString << Proxy" genuinely ambiguous: operator ZString() and
operator MString() (via its own implicit upcast to ZString) both satisfy
MString::operator<<(ZString), and operator double()/operator bool() both
satisfy MString::operator<<(int) via a standard numeric conversion, so
the compiler had no basis to prefer one conversion path over another.
Fixed the same way Proxy already solves this exact problem for
std::cout << proxy - a dedicated friend operator<<(MString&, const
Proxy&) that needs no conversion at all, so it's always the preferred
candidate. Verified by actually running testing/jstring_test.cpp (not
just compiling it) - the line that used to fail to compile now correctly
prints "this is a test: this is a comment".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CGIFormData now inherits Parser instead of MString and parses a CGI
request body (urlencoded or multipart/form-data) resumably against the
shared StreamReader, exposing the result via operator[] - matching every
other Parser in this redesign. multipart/form-data reuses IMFBoundary and
IMFHeader via checkParse() (chained through the same StreamReader) for
each part's boundary line and header block; urlencoded bodies wait for
the full Content-Length before delegating to the existing
CGIFormattedData parser. File-upload parts (Content-Disposition with a
filename= parameter) are recognized and skipped, matching this
codebase's convention that uploads get a separate mechanism.
Values are stored under an indexed "name:N" key (N = 0-based occurrence
count for that name), matching the :N-suffix convention JetCore's
Tag::getVariable already applies to its $[:name] CGI variable syntax
(defaulting to :0) - this also naturally supports repeated field names
(checkboxes, multi-select). operator[] applies the same :0 default so a
bare name still works when called directly, not just through that
convention layer.
Two real resumability bugs found and fixed via a byte-at-a-time test
(feeding one byte per StreamReader::feed() call, the harshest possible
fragmentation):
- CGIFormData's opening boundary marker ("--boundary", no leading CRLF
per RFC 2046 for the first boundary) crammed "find+consume the marker"
and "disambiguate -- vs CRLF" into one state; when the disambiguation
needed more data, INCOMPLETE was returned without recording that the
marker was already consumed, so retrying re-searched for it past where
it had already been consumed, silently skipping content forever. Split
into two states (INIT / OPENING_TAIL), the same fix already applied
once to IMFBoundary's PREFIX/TAIL split.
- IMFHeader's FIELD state checked eod() (a 1-byte availability check)
immediately before an ifNext("\r\n") call that needs 2 bytes - a lone
trailing \r with no \n yet available was wrongly treated as malformed
instead of "wait for more data". Fixed to check getLength() < 2.
Verified via a dedicated test (64 assertions total across this and the
earlier Parser/StreamReader test) covering urlencoded, multipart with a
skipped file upload, and the byte-at-a-time stress case - plus a real
end-to-end run through jet-2.0 with a simulated CGI multipart POST
request (see the matching JetCore commit).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Introduces StreamReader to accumulate socket data delivered in arbitrary
chunks into one buffer, so a Parser can build an object (a line, a header
block, a MIME boundary) that spans any number of packets without losing
its place. Parser is redesigned around this: no longer inherits MString
(was redundant with its own buffer member), takes a StreamReader&, and
drives a resumable COMPLETE/INCOMPLETE/DATA_AVAIL state-machine protocol
via doParse(). LineParser, IMFHeader, and IMFBoundary are rewritten as
real, working resumable parsers against the shared buffer instead of the
previous non-compiling stubs (mismatched constructors/return types, a
syntax error, checkParse() never implemented).
IMFFormData is left as the stub it already was, just restored to
compiling against the new IMFHeader signature - fully wiring it up needs
IMFMultipart/IMFText migrated to StreamReader too, out of scope here.
Verified with real runtime tests (not just compilation) covering data
split mid-line, mid-CRLF, mid-header-fold, and mid-boundary-marker across
separate feed() calls, plus StreamReader::compact(). Along the way fixed
a header-value bug (space after ':' wasn't skipped) and a double
cursor-offset subtraction bug in two places caused by forgetting that
ZString::getLength() is already cursor-relative, not the raw buffer size.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- CGIFormData: fix broken setSize()+read() double-length bug that
corrupted every parsed body with a leading run of garbage bytes
(setSize(contentLength) pre-set length before read()/write()
appended on top of it); switch content-type checks from exact
equality to startsWith() since a real multipart Content-Type
always carries a boundary= parameter; implement the previously
empty multipart/form-data branch (ordinary fields only, file
uploads deferred to a separate upload mechanism per TODO.txt);
restore operator[] real lookup logic (was hardcoded to echo the
key back); default-initialize formdata to NULL; fix mismatched
include guard (leftover from an IMFFormData.h copy-paste); drop
the unused IMFHeader.h include that pulled in a broken,
unrelated StreamReader.h dependency.
- CGIFormattedData: parameterize the split delimiter (was hardcoded
to &, which silently broke cookie parsing since HTTP_COOKIE uses
; not &) and trim whitespace on each split token.
- MString::toCGI()/fromCGI(): were unimplemented stubs (always
returned an empty string); implemented proper CGI percent-encode
/ percent-decode + space plus-encoding, and fixed the one caller
that discarded fromCGIs return value instead of using it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous implementation was a narrow path-accessor, not a JSON parser:
numbers/booleans/null were unhandled entirely (self-acknowledged by a
// TODO: Handle numbers... comment), string escape sequences were never
encoded or decoded, updating an existing non-string-valued key threw a
hard-coded Fix me exception, and skipping past an uninteresting key threw
Unrecognized data type for anything but string/array/object. Full
analysis given to the user before this rewrite; nothing here was guessed.
Added:
- JSONType enum + getType()/exists()/isNull() for type introspection.
- Typed read access: getString() (escape-decoded), getNumber(), getInt(),
getBool() - each validates the value at the path is actually that type.
- Typed write access: setString() (escapes automatically), setNumber(),
setInt(), setBool(), setNull() - and Proxy operator= overloads for
double/int/long/bool so produces a real JSON number, not
a quoted string the way the old single MString-typed operator= did.
- jsonEscape()/jsonUnescape(): full escape codec including \uXXXX and
UTF-16 surrogate pairs (astral characters, e.g. emoji) decoded to
correct UTF-8.
- skipValue()/skipString()/skipContainer()/skipNumber() and their
capture (span-returning) counterparts: escape-aware structural
skipping, so a string value containing a literal { } [ ] , or : can
no longer desync object/array nesting depth - this affected both
locate()s internal key-skipping and find()s array/object extraction
(the latter still relied on the base ZString::getContainer(), which
has this exact bug and was not touched - see the ZString commit in
this same session - so a JString-specific escape-aware capture was
added instead of depending on it for this purpose).
- set() rewritten to use skipValue() to correctly locate and replace an
EXISTING values full span regardless of its current type, instead of
only handling the case where the existing value was already a quoted
string.
Bugs found and fixed along the way, surfaced by writing real tests
against realistic JSON rather than just getting it to compile:
- The array-index-then-further-path case (e.g. b[2].c) always returned
true immediately after reaching the index, so any path segment past
the index was silently ignored - dead code after an unconditional
return/throw meant the intended continuation logic could never run.
- The array selector syntax (label[key=value]) was only reachable as a
top-level special case, never as part of a normal nested path (e.g.
items[id=x].val) - unified both under the same bracket handling.
- Once reachable, the selector-match subkey check (find(key) on each
array element) always searched from the document root, not the
current array element, because find() unconditionally calls reset().
Split into public find() (resets) and internal findRelative() (does
not), and pointed both selector-matching call sites at the latter.
- path[0].getTokenExclude(...) in the array-selector path parser
indexed into a list vector that split() never populated - undefined
behavior. Removed the erroneous [0].
Verified via a dedicated test program (not included in this commit)
covering: string escape round-trip, numbers, booleans, null, exists(),
\uXXXX including a real surrogate pair decoded to correct 4-byte UTF-8,
nested dotted+bracketed navigation, a string value containing every
structural JSON character, replacing an existing key with a different
type, multi-key documents, pretty() not corrupting structural characters
inside a string, and the array selector syntax on both a matching first
and second element - 30/30 assertions pass. Full CoreUtils compile pass
unchanged apart from JString.cpp itself now succeeding (14/20 clean, same
7 pre-existing StreamReader.h-blocked failures as always, nothing new
broken).
operator< computed end2 as valuex.getData() + valuex.getLength() - mixing
an absolute pointer (getData(), the buffer start) with a cursor-relative
length (getLength(), remaining-from-cursor). For any ZString with a
non-zero cursor offset this produces an end pointer short of the true
buffer end by exactly the amount already consumed, truncating the
comparison early. Fixed to use valuex.data + valuex.length (the true
end), matching how end1 was already computed for *this. operator<=,
>=, and > were separately broken (flagged by the codes own TODO
comment) - they compared via strncmp from data (ignoring cursor
entirely), inconsistent with operator< and with each other. Replaced
all three with derivations from the now-correct operator< (a <= b is
!(b < a), etc.) - guarantees trichotomy by construction instead of
three independently-written comparison loops.
substring(start, len): when the requested length overran the buffer,
computed newlen as endChar - end (the overrun amount) instead of
end - startChar (the actual remaining valid length) - backwards, and
stored into a char (max 127), silently wrapping for longer strings.
Fixed the formula and widened to size_t.
trimCRLF(): read *(data + length - 1) unconditionally, including when
length is already 0 - one byte before the buffer on an empty string.
Guarded both checks on length > 0.
Verified: full local compile (identical result to before - 13/20 clean,
same pre-existing StreamReader.h-blocked failures, nothing new broken);
a dedicated runtime test exercising all three fixes plus a trichotomy
check across multiple cursor offsets - all 13 assertions pass. Test
file not included in this commit.
__FILE__/__LINE__ are preprocessor macros - as default argument values
in the header they expand once, at the header declaration site, not at
each call site. Every throw relying on the defaults (all but File.cpp,
which passed them explicitly) reported Exception.h itself instead of
the real error location.
Switched the defaults to std::source_location::current(), which the
C++20 standard specifically defines to evaluate at the call site when
used as a default argument - unlike macros. Kept the same 4-parameter
signature so every existing call site, including ones that pass
__FILE__/__LINE__ explicitly, is unaffected.
Verified two ways: full-repo compile pass (identical result to before,
13/20 clean, same pre-existing StreamReader.h-blocked failures, nothing
new broken); a real runtime test - a throw relying on defaults from a
known line correctly reported that exact file/line via the caught
exceptions .file/.line members. Test file not included in this commit.
The Exception on the odd-length-input path was constructed but never
thrown (missing throw keyword) - the temporary was destroyed with no
effect, so execution fell through to an out-of-bounds nextChar() read
past the exhausted buffer instead of raising an error. Added the
missing throw.
Verified via full local compile: MString.cpp builds clean standalone;
a full-repo pass shows only the pre-existing, already-known
StreamReader.h-missing failures (7 files, unrelated to this change),
confirmed via each error log individually.
Brad Arant