Used to live-verify APIServer end to end - one small template per
distinct code path, matching demo.jet/demo2.jet's existing role as
worked examples rather than throwaway scratch files:
- demo4.jet: "#api GET /users/{id}" + $[id] - path param capture.
- demo5.jet: "#api POST /echo" + $[:message] - urlencoded form body via
CGIFormData, reached through the same $[:name] CGI convention the
interpreter already uses.
- demo6.jet: "#api GET /crash" + $[:message] with no body present -
deliberately throws (Tag::getVariable's cgiData-is-null error),
verifying a render-time exception turns into a clean response instead
of taking the server down.
- demo7.jet: "#api GET /search" + $[q] - query string parsing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 1 of the APIServer plan: a compiled --shared .so currently exports
only a bare jetc_render_<name>(Global&, MString&) with no way for a host
to discover what HTTP method+path it wants to serve. Adds a leading
"#api METHOD /path" directive (one or more, parsed the same way jetc
already strips a leading "#!" shebang line - not modeled as a new
AST/tag-attribute concept, since jetc's parser has no existing notion of
the enclosing <jet> tag's attributes to hang this off of).
In --shared mode, JetCodegen now also emits a second exported function
alongside the render entry point:
struct JetApiRoute { const char *method; const char *path; };
extern "C" const JetApiRoute *jetc_api_<name>(int *count) { ... }
<name> is derived the same way for both symbols (sanitizeIdent(
baseNameNoExt(inputPath))), so a future host loader can find both by
convention from just the .so's filename - no name needs to be embedded
in the metadata itself, and the routes a module declares can never drift
out of sync with the render function that implements them (the failure
mode a sidecar manifest file would have).
Malformed directives (missing method/path, path without a leading '/')
are a hard compile error; #api directives present without --shared print
a non-fatal warning (the metadata has nowhere to go without a host to
load a .so, but the render-only executable/CGI path still works fine).
Verified live, not just compiled: rebuilt jetc, compiled demo3.jet (a new
worked example with two #api lines sharing one handler) via
build-shared.sh into a real demo3.so, and added api_test.cpp (mirrors
host_test.cpp's dlopen()/dlsym() pattern) confirming jetc_api_demo3
returns the expected two routes at runtime. Re-ran host_test against the
same .so to confirm the render path is unaffected. Confirmed demo.jet/
demo2.jet (no #api lines) emit no JetApiRoute code at all - no regression
for templates that don't declare routes.
Also fixed a gitignore gap surfaced while committing this: an earlier
"jetc/*.cpp" ignore rule (meant for generated *.compiled.cpp output) was
a blanket pattern that silently hid this commit's own api_test.cpp from
git status. Narrowed to "jetc/*.compiled.cpp" (the actual generated-file
naming, per build-compiled.sh/build-shared.sh) and added explicit ignores
for the host_test/api_test/demo3 binaries plus two known stale generated
leftovers (demo.cpp/demo2.cpp) that predate this session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TODO.txt: drop the stale "Fix the variable retriever..." bug-list entry -
long since fixed. tests/testjet.jet: cgi="true" -> cgi="yes", matching the
deliberate "yes" convention already used elsewhere (__jet.cpp checks
resolveKeyword("cgi") == "yes", not "true"). tests/dump.txt: regenerated
output from re-running the test after the above.
.gitignore: jetc/ was accumulating untracked build output (compiled .cpp,
binaries, .pic.o, .so, libCoreUtils_pic.a) alongside its real tracked
source - ignore the generated artifacts, keep the two worked-example
.jet templates explicitly un-ignored since *.cpp/no-extension patterns
would otherwise also catch demo.jet/demo2.jet's own extension-less
compiled binaries without excluding the .jet sources themselves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
renderVariableName() built the variable name by whitelisting an
identifier charset (letters/digits/._-) via getTokenInclude(), then
threw "invalid variable name." on the first character outside that set
- including a plain space. Real-world CGI form field names aren't
restricted to identifier characters (an HTML <input name="first name">
is completely valid), so $[:first name] failed outright instead of
looking up the field as sent.
Rewrote to exclude only the genuinely structural characters (']', ';',
and the start of a nested $[.../#[...substitution) via getTokenExclude,
so everything else between the brackets - spaces included - is consumed
as part of the name before any lookup happens, rather than stopping
early and erroring. This also let the separate ':' handling branch be
dropped entirely: since ':' is no longer excluded, it now flows through
the same sweep naturally (needed for both the CGI sigil name itself and
the "$[:name:2]" explicit-index suffix).
Verified live: $[:first name] against a real multipart CGI body with a
field literally named "first name" now resolves correctly (previously
threw before ever reaching the lookup). A stray, non-nested '$' or '#'
in a name is kept as literal text rather than erroring. Re-verified
existing sigils are unaffected - plain $[x], $[!x] (forced-global), and
tests/posttest.sh's ^session/:.name3 lookups all still resolve exactly
as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
When resolving a bare $[var]/#[var] reference, Operand set .boolean =
true whenever the value's text was EITHER "true" OR "false" -
effectively "boolean = looks-boolean-ish" rather than reflecting which
of the two it actually was. <if expr="..">/<while expr="..">check
exactly this .boolean field (see __if.cpp/__while.cpp), so <if
expr="$[flag]"> took the true branch even when flag held the literal
text "false".
Fixed to boolean = (string == "true").
Verified live: a variable set to value="false" now correctly takes an
<if expr="$[var]">'s false branch, and value="true" still correctly
takes the true branch. Re-ran tests/posttest.sh - unaffected, no
regression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CONTENT_TYPE was "multipart/form-data" with no boundary= parameter - a
real browser/curl request always includes one. Without it,
__jet.cpp's contentType.find("boundary=") returns -1, so boundary stays
a default-constructed empty ZString. CGIFormData's opening-marker search
then becomes just "--" (boundary appended to it), which matches
trivially at the very start of the body text, so the parser immediately
reports Status::COMPLETE having consumed only those two bytes and parsed
zero fields - no error, no exception, just silently empty. Every
$[:name] lookup returned blank as a result, not just $[:name3] (verified
- all six of post.example.formdata's fields were affected identically).
Fixed CONTENT_TYPE to carry the real boundary value used in
post.example.formdata ("----WebKitFormBoundaryA1LWIjMAIsdFW9XA", matching
each part's "--" + boundary delimiter line), and corrected
CONTENT_LENGTH (619 -> 623, the fixture's real current byte count - it
had grown since 619 was set, from prior edits giving each form field a
distinct name instead of several duplicate "name" fields).
Also includes the already-in-progress edits to post.example.formdata
(distinct field names name/name2.../name6, replacing several duplicate
"name" entries), post.example.urlencoded (matching field names), and
testpost.jet (added a $[:name3] line) that were sitting uncommitted -
these are what surfaced the bug in the first place, not a new change
made alongside this fix.
Verified live: all six $[:name]..$[:name6] now resolve to their exact
raw multipart values (1 test / 2&test / 3+test / 4 test / 5 / 6 - no
CGI-decoding applied, correct for multipart bodies, unlike the
urlencoded-body path which does decode).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For the server you're about to build: `jetc --shared template.jet out.cpp`
now emits no main() at all - instead the render logic is exported as
`extern "C" void jetc_render_<template>(jet::Global &global,
coreutils::MString &out)`, one call per request, meant to be dlsym()'d
by a long-running host process instead of spawning a jet-2.0 process (or
even a separate compiled executable) per request. extern "C" here only
fixes the symbol's name mangling for a stable dlsym() target - the
parameter types are still real C++ classes, safe to pass across the
dlopen() boundary since the host and the .so share the same
CoreUtils/JetCore headers and toolchain (a same-ABI plugin boundary, not
a foreign C one). coreutils::Exception propagates out of the call
uncaught, so the host decides how to handle a render failure rather than
the module unilaterally deciding (the executable mode's try/catch prints
to stdout and exits, which is CGI-appropriate but wrong for a persistent
server). The exported symbol name is derived from the input filename
(jetc.cpp's baseNameNoExt+sanitizeIdent) so multiple compiled templates'
.so's stay collision-free even under dlopen(..., RTLD_GLOBAL).
jetc/build-shared.sh builds the .so: JetCore's 22 tag classes are still
force-linked directly (Tag::interpretSpan's fallback path can dispatch
to any of them at runtime, regardless of what a given template
statically references), but CoreUtils objects are compiled -fPIC and
archived into a throwaway libCoreUtils_pic.a instead of being linked
directly - libCoreUtils.a itself isn't position-independent, so linking
it into a .so fails outright ("relocation ... can not be used when
making a shared object"), and force-linking every CoreUtils .cpp
directly (rather than archiving, so the linker only pulls in what's
actually referenced) surfaced an unrelated, pre-existing incomplete
class - IMFMessage's vtable is never emitted anywhere, presumably a
partially-migrated class (see project memory on IMFMultipart/IMFText
still being fd-based) - as an unresolved dlopen()-time symbol, even
though nothing here uses it. A normal archive-based link (like the
existing libCoreUtils.a already does for jet-2.0) never pulls IMFMessage
in at all, so building libCoreUtils_pic.a the same way sidesteps it
without touching the unrelated bug itself.
jetc/host_test.cpp is a minimal stand-in for the real server: dlopen()s
a compiled .so, dlsym()s its entry point, and calls it twice with fresh
jet::Global objects (simulating two separate requests against one loaded
module) to prove the .so is genuinely reusable, not single-use. Verified
both demo templates (demo.jet, demo2.jet - the latter covering nested
if/for, a stateful while loop, and the interpretSpan fallback path) build
as .so's and produce the same byte-identical output as the executable
and interpreter paths, across two calls each. Also re-verified the
existing (non-shared) executable build path still matches jet-2.0's
output after the Codegen::generate() signature change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jetc (JetCore/jetc/) compiles a subset of JET template syntax - literal
text, $[...]/#[...] variable output, <set>, <if>/<else>, <for>, <while> -
into a standalone C++ program that links against CoreUtils and the same
JetCore tag-class library jet-2.0 itself uses. The static tag/text
structure becomes real C++ control flow (if/for/while), resolved once at
build time instead of being re-scanned character-by-character from raw
source on every request and, for loop bodies, on every single iteration
(confirmed by reading Tag::parseContainer - it re-tokenizes its container
text from scratch every time it runs, since a tag's parse and its
execution are the same pass in this codebase). Anything genuinely
dynamic - variable values, expr/value comparisons, loop bounds - still
calls into the existing runtime (Tag::getVariable/resolveKeyword,
jet::Operand) exactly as the interpreter does; compiling doesn't try to
eliminate that; only the repeated re-parsing of the template's own shape.
A <if type="..."> or <while type="..."> comparison type is resolved to
its C++ operator once at compile time (type is a literal attribute value
in practice, not runtime data) instead of being re-checked against six
string constants on every evaluation.
Any tag jetc doesn't have a codegen rule for (mysql, sql, call, include,
a <set> with a container body, etc.) is captured as its exact raw source
span and handed to the existing interpreter at compiled-program runtime
via the new Tag::interpretSpan(), so an unsupported construct degrades
gracefully instead of failing to compile - a real, if narrower, program
rather than requiring full tag coverage up front.
Tag.h/.cpp: made assignVariable public (parallel to the already-public
getVariable - generated code needs a write path without being a Tag
subclass) and added interpretSpan() as the fallback's entry point into
parseContainer.
Verified live: two representative templates (jetc/demo.jet, an if/for/set
walkthrough; jetc/demo2.jet, a nested <if> inside a <for>, a <while> that
mutates and rereads its own condition variable across iterations, and a
fallback-interpreted <comment> tag) both compile, link, run standalone,
and produce byte-identical output to jet-2.0 running the same source.
Building and running jetc/demo2.jet's <while> loop surfaced a real,
previously-unknown MString bug (fixed separately, see CoreUtils commit
6e4e463) along with a separate, already-existing bug in __while.cpp's
value1/value2/type form (unrelated to this compiler, not fixed here -
see project memory).
jetc/build-compiled.sh builds the compiler and a given .jet file's
compiled binary in one step; jetc/demo.jet and demo2.jet are kept as
worked examples. Known limitations: <set> with a container body isn't
modeled (falls back to interpretation); a fallback-interpreted tag
nested inside compiled control flow uses the template's single top-level
scope for parent/local context rather than a faithful per-level chain,
which only matters for legacy tags that rely on nested "local" scoping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The @ sigil previously read through global.getSession(global.sessionId),
a MySQL-session binding that was already confirmed dead this session -
Global::sessionId is never assigned anywhere in the codebase, so any
$[@name] read would always throw. The design goal: persist session state
as a JSON file at sessiondir/<sessionId>.json, using CoreUtils's existing
JFile (an MFile+JString combination with load-on-construct/save-on-
destruct RAII semantics), reusing @ rather than introducing a new sigil
since the old @ usage was unreachable in practice anyway.
Global.h/.cpp: added Global::sessionFile (a JFile*, NULL unless a request
is in cgi mode with sessiondir set). Its destructor now explicitly
deletes sessionFile - JFile's actual disk write happens in MFile's
destructor via update(), so this has to run for persistence to happen at
all (a raw `new`'d pointer that's never deleted, as cgiData still is,
would never save anything back to disk).
__jet.cpp: the sessiondir block now builds the session file path from the
sessiondir keyword and the active session id (existing cookie or newly
issued) and constructs global.sessionFile there, for both branches. The
"pull sessionfile from sessiondir" comment is now implemented - JFile's
own constructor already handles "file doesn't exist yet" by loading as
empty, so one construction point serves both a resumed and a brand-new
session.
Tag.cpp: getVariable's "@" branch now reads via (*global.sessionFile)[name]
with a clear error if no session is active, instead of the dead MySQL
lookup. Added Tag::assignVariable (declared in Tag.h), a single write
path both storeVariable overloads now funnel through, so a "@name"-
prefixed target (e.g. <set name="@x" value="y"/>) is routed to the
session file instead of the normal global/local/parent variables maps -
without this, the read side would have been read-only, which isn't a
very useful session mechanism.
This surfaced two real, pre-existing CoreUtils bugs, both fixed
separately (see CoreUtils commit 3e6e410): JFile's virtual-base diamond
skipping JString's own empty-object seed, and MString's ZString-source
copy paths pairing an absolute data pointer with a cursor-relative
length - both were silently corrupting any cursor-advanced ZString
(exactly what assignVariable produces when it strips the "@" prefix via
ifNext() before storing).
Verified live: two genuinely separate jet-2.0 process invocations
sharing one session cookie, where the first writes two session
variables and the second - a fresh process with no shared memory - reads
both back correctly from the JSON file on disk. Also re-ran
tests/posttest.sh (unchanged output, no regression) and the full
CoreUtils test suite (no regressions).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
$[^session] (a cookie lookup, global.cookies.data[name]) was always
empty because __jet.cpp's cgi keyword check compares against "yes" (a
deliberate choice from an earlier session, not the codebase-wide
true/false convention used by filterblanklines/trimlines/eval etc.) -
testpost.jet had cgi="true", a stale mismatch that predates that
decision, so the entire cgi block (including reading HTTP_COOKIE into
global.cookies) never ran at all. Fixed the test fixture to say
cgi="yes" to match.
Verified live via tests/posttest.sh against the current build:
$[^session] now correctly prints 18fd56a83f (posttest.sh's simulated
HTTP_COOKIE session id), and dump.txt's *** COOKIES *** section shows
session=[18fd56a83f] instead of empty. dump.txt committed with this
fresh, correct output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Global now owns a long-lived StreamReader (cgiReader) that CGIFormData
(a Parser) holds a reference to - declared before cgiData so it's
constructed first and destroyed last. __jet.cpp's POST handling
constructs CGIFormData against it and feeds the request body off stdin
in 4KB chunks, driving doParse() after each feed rather than blocking to
read the whole body up front - the same feed()/doParse() pattern a
socket-fed StreamReader would use, keeping memory bounded for large
multipart uploads.
Also fixes two real, pre-existing bugs surfaced while getting an actual
end-to-end CGI request working through jet-2.0 (not just compiling):
- Tag::Tag's opening-tag parser was gated on "parent && in.ifNext(\"<\")",
but jet-2.0.cpp's main() constructs the top-level __jet with
parent=NULL - meaning the outer <jet ...> tag's own attributes
(cgi="yes" and everything else) were never actually parsed at all, just
echoed as literal passthrough content. parent=NULL is used at exactly
one call site in the whole codebase, so dropping the "parent &&"
requirement only changes behavior for that one top-level case - every
nested tag construction already passes a real parent.
- __jet.cpp's boundary-extraction code calls contentType.find("boundary=")
to locate the multipart boundary parameter, but find() repositions
contentType's own cursor to the match as a side effect. That same
cursor-shifted contentType then got passed into CGIFormData's
constructor, whose startsWith("multipart/form-data")/
startsWith("application/x-www-form-urlencoded") checks are
cursor-relative and so silently matched neither, falling through to an
unrecognized content type that parsed nothing (reported COMPLETE
immediately, no error, just empty results). Fixed with
contentType.reset() before the CGIFormData construction.
Verified with a real end-to-end run: jet-2.0 invoked with simulated CGI
environment variables (REQUEST_METHOD=POST, CONTENT_TYPE with a
boundary=, CONTENT_LENGTH) and a genuine multipart/form-data body
(including a file-upload part) piped over stdin, using $[:name] in a
.jet script - correctly returns each field's value and correctly leaves
the skipped upload field empty.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- __jet.cpp: the cgi keyword check compared against "true" but the
documented/intended value is "yes", so <jet cgi="yes" ...> never
actually turned CGI mode on; fixed. Also replaced the hardcoded
fake multipart boundary ("xxxx") with real extraction from the
Content-Type header's boundary= parameter.
- Global.h: default-initialize cgiData to NULL (a GET request with
cgi="yes" never allocates it, so it was previously left
dangling); Global.cpp now sets the cookie jar's delimiter to ";"
to match RFC 6265, since it defaulted to CGIFormattedData's
&-based form-data delimiter.
- Global.cpp: fix addSession()'s missing throw that let it silently
continue past a duplicate session id.
- Tag.cpp: fix the CGI-variable lookup branch, which failed to
compile (cgiData is a pointer; needed to dereference it before
indexing) and added a null check so a CGI request that never
populated cgiData (bad Content-Type, or the null-safety gap
above) throws a clear error instead of crashing; implemented the
previously-empty session-variable lookup branch; left the
unreachable duplicate environment-variable branch as-is with a
comment, since fixing its sigil is a user-facing template-syntax
decision, not a bug fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
barant