<comment> had no special-case in jetc's compiler at all - it fell
through to parseFallback(), the generic unknown-tag path, which hunts
for a tag's own closing tag via scanBody()/skipQuotedValue(), the same
quote-aware, backslash-escape-interpreting scanning real Jet attribute
syntax needs. A comment containing an explanatory " sequence (a JSON
example in prose, say) or a literal <else>/<if>-looking word - both
already confirmed, separately, to corrupt jetc when they turn up
inside a JS // comment - hit the exact same corruption here too, just
triggered from inside a <comment> block instead.
Added skipCommentTag(), which only tracks same-tag-name nesting depth
via the existing case-insensitive matchTagOpen/matchTagClose and
otherwise ignores every character - no quote tracking, no other-tag
recognition, nothing a comment's own body could ever trip over. Wired
into parseNodes() alongside the other recognized top-level tags;
unlike those, it pushes no node at all; the comment contributes
nothing to the compiled output, matching the runtime interpreters own
__comment behavior but now safe at compile time too.
Verified directly: a scratch .jet with five <comment> blocks (plain,
a bare quote, a backslash-quote JSON example, literal <else>/<if>
text, and nested comments with their own quotes) all compiled clean
and left zero trace in the generated code, with the <if> immediately
following them compiling correctly too - confirming nothing after a
tricky comment gets corrupted either.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Jet interpreter (Tag.cpp, __call.cpp, __imap*.cpp/h, __smtpsend.cpp/h,
__imapemptyfolder.cpp/h - new): assorted parser/tag fixes found across
this work - a stray literal-tag-in-comment and backslash-quote parser
corruption class, a dangling-pointer fix, a __call.cpp fd leak, IMAP
folder-management tags, multi-account/per-host IMAP support, and a
smtpsend Message-ID fix.
barant.com site (jetc/barant-site/):
- A site-wide persistent music player (header band, playlist/queue,
stereo FFT EQ, session-persisted state) fed by new library/
ianslibrary/music song-list subpages.
- /account split into info/phones/invoices subpages; a full invoicing
and Square payment system (create/edit/copy/pay/email invoice);
admin invoice management and mail-account management subpages.
- /services split into overview/hosting/siptrunking/documents
subpages, retiring the old standalone pages.
- SIPPHONE.tag: a Jet 2.0 WebRTC softphone tag.
- main.css/spa.js/email.js: the SPA loading indicator, the music
player band styling, and assorted fixes accumulated alongside this
work.
qualityiron-site/: the qualityiron.vegas job/contractor/customer/
document management app, ported from its original Jet 1.4 codebase to
Jet 2.0 as barant.coms APIServers second multi-tenant site.
wcwc/wcwc.jet: the WCWC site Jet source (wcwc.jtc build output not
included).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- __mysql.cpp: <sql> never checked mysql_real_query's own return code,
so a genuine query failure was indistinguishable from "ran fine, zero
rows" - <try>/<catch> around a failing <sql> never actually caught
anything anywhere. Now throws on a real failure.
- spa.js: dynamically re-injected scripts now run in true sequential
order (wait for each external script's load event before running the
next), fixing checkout's Square card field never building on the
first client-side navigation to /checkout (only after a refresh) -
async=false alone only orders multiple external scripts relative to
each other, it doesn't delay a plain inline sibling.
- /vst split into a real overview + /vst/modular1 subpage with a left
menu, reusing opensource.page's existing subpage-tier pattern
(opensource.menu/main.css's submenuitem-active highlighting, and
several other subpage-tier retirements, were already finished earlier
this session but never committed - included here too).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
listMessages() only ever fetched everything then kept the newest
limit (default 50) messages, with no offset parameter and no way for
the client to know how many messages existed beyond what it was
shown - older mail was completely unreachable through the webmail UI.
__imap::listMessages() now takes an offset (counted back from the
newest message) and returns {"total":N,"messages":[...]} instead of
a bare array, so the client can page backward/forward and know when
it has reached either end. __imaplist tag and /email/list both pass
offset through; /email/search is unchanged (search results are not
paginated). email.js tracks currentOffset (reset alongside
currentFolder/currentUid at every existing reset site), and renders
Prev/Next controls next to the unread count showing the current
1-based range and total.
htmleditor.page previously used hardcoded blue/gray Windows-95-era
inline styles (#678, #abc, #808080, #def) totally disconnected from
the site current dark/gold Bond theme (main.css --bond-* variables) -
this is what showed up as "the old page with the blue stylesheet"
when opening a file to edit. Restyled the window chrome, toolbar,
title bar, and buttons to use the same custom properties/.button
class the rest of the site uses; left the #mainpage content-preview
canvas white since it renders the actual page being edited.
email.js: removed the Disconnect button/handler and its unused icon
from the email toolbar per request. Bumped its cache-buster in
admin-email.subpage.
New __imap*/__smtpsend/__eval Jet tags give templates real IMAP
(list/message/flag/delete/move/search/folders/accounts) and SMTP-send
capability, backing the new /email webmail client (email.jet/
email.menu, assets/images/email.js + newemail.js) and its
mainpagetemplate.jet-based page shell. Most of this tag family's
source was already being built into the live site (Tag.cpp already
registered __imap/__imaplist/__imapmessage/__imapflag/__imapdelete/
__eval) but had never actually been committed; this also registers
and commits the remaining five (__imapmove/__imapfolders/
__imapsearch/__smtpsend/__imapaccounts).
admin.page/admin.menu gain a real subpage tier (admin-email,
admin-newemail, admin-invoices, admin-products, admin-users) replacing
several previously top-level pages, mirrored by a similar
opensource.page subpage tier (opensource-overview/asm1802/barantmail/
jet/serverframework/synthmax) - old top-level pages retired via git
rename (asm1802.page, jet.page, serverframework.page,
admin_invoices.page); barantmail.page is dropped outright, folded into
opensource-barantmail.subpage.
New storefront (store.page, store-createstore.subpage,
barantstore.page) and cart flow (cart.jet, updatecartqty.proc,
removecartitem.proc, editproduct.page/selectproduct.proc/
update_product.proc for inventory maintenance against the barant
database).
New document upload/download manager (documents.jet, documents_api.jet,
documents_upload.proc, documents_delete.proc,
assets/images/documents.js + documents-page.js) with a persistent
floating panel, and a rich-text page editor (htmleditor.jet/.page,
createhtmleditor.proc, savehtmleditor.proc, assets/images/htmleditor.js
+ its icon/logo assets).
SESSIONCTL.tag: login now redirects back to the originally-requested
page instead of always landing on /account ($[start:0] vs $[start]
variable-name mismatch).
main.css/menu.bar/invoices.menu/jet_overview.section/login.menu:
styling and navigation updates for the above; new site assets (fonts,
images, audio) referenced by them.
jetc/build-assets.sh, list_jet_files.sh, lowercase_jet.py,
strip_fragments.py: small dev tooling used while building the above.
JetCore engine (Tag.cpp/.h, Global.cpp/.h, __*.cpp tag classes):
- Numerous __mysql.cpp connection-pooling correctness fixes (stale
result sets, double-indirection sessionid bugs, field caching,
pool keying) and a session-registration-before-connection-confirmed
ordering fix (dangling pointer on a failed connect)
- Tag.cpp: large body of fixes to the tag-dispatch/container-scanning
logic (bare-name fallback, database aliasing, single-quote handling,
@-prefix session variable resolution, TAGDEF caching, nested split-tag
scanning for if/while/until/ifrow so an inner one doesn't corrupt an
outer one's own boundary scan) plus the process-first rendering
order fix (a <process> now runs before <checksession>/page
resolution, fixing a real "blank page after successful login" bug)
- Global.h/.cpp: added imapSessions (parallel to the existing __mysql*
session map) for the new <imap> tag family
- __whiledir.cpp/__include.cpp: caching and directory-name-aliasing
fixes
- __call.cpp/.h: fixed a dangling-pointer bug
- __write.cpp: fixed an O_WRONLY-only open that couldn't create a new
file
- __ifrow.cpp/__sql.cpp: fixed the same sessionid double-indirection
bug __mysql.cpp had (resolveKeyword's result was being used as a map
key lookup instead of the resolved value itself)
jetc compiler (jetc/JetParser.cpp/.h, JetCodegen.cpp/.h, JetAST.h,
jetc.cpp, build-shared.sh, build-compiled.sh):
- Parser fixes: inline <include>, nested-quote handling, void-tag
recognition
- Codegen fixes and expanded AST support
- build-shared.sh: atomic rename into place for a freshly-built .jtc
(a live server dlopen()ing the same path mid-rebuild was corrupting
its own already-mapped view of the old file) and a staleness check
so a cached JetCore .o doesn't silently keep serving yesterday's
fixes forever
- Added -lssl to both build scripts' link lines (needed once the new
<imap> tag's client-side TLS code existed - previously nothing in
this codebase used OpenSSL's SSL/TLS client API, only libcrypto)
Site content (jetc/barant-site/): SPA (fetch-and-swap navigation)
rollout across essentially every .page/.jet/.section/.tag/.proc/.method
file - dynamic-content fixes, fragment-mode stripping, native
$[@name] session variables replacing older mechanisms, and this
session's own checkout.page redesign, login start-param redirect fix,
and per-page side-menu (menu.bar/$[PAGE].menu) refresh support.
Completed the earlier main.css/control.css move from assets/ to
assets/images/ (git still showed the old paths as tracked-but-deleted
until now).
Deliberately left out of this commit: dozens of .pre-<description>-
<date> manual backup files (this session's and earlier ones' own
before/after snapshots, not meant for permanent history), compiled
.jtc build outputs, ad hoc test binaries and scripts under jetc/
(login_asan, wcwc_test, optest2_bin, etc.), and jetc/__pycache__/ -
none of these are meant to be tracked; a .gitignore covering them
would be worth adding separately.
Three correctness fixes to the --shared build path:
- Only skipped rebuilding a .pic.o when it existed at all, never checking
whether the source .cpp was newer - confirmed live: Tag.pic.o (and 6
other JetCore objects) sat uncompiled since 2026-08-25 while Tag.cpp
itself picked up same-day fixes, so every .jtc silently ran yesterday's
JetCore code.
- Only re-archive libCoreUtils_pic.a when a member object actually changed,
instead of unconditionally on every build.
- Link to a temp file in the same directory, then rename() it into place,
instead of linking directly onto the live .jtc path. A running APIServer
may already have that exact path dlopen()ed (hot-reload watches this
directory) - overwriting it in place corrupts the live mapping's code
pages out from under the process, causing a dlclose() SIGSEGV the next
time it reloads. rename() never touches the old inode, so anything with
it still mapped keeps seeing the unmodified original.
The env-var branch in Tag::getVariable was dead code - it checked for "%",
but that sigil was already claimed by the keyword-lookup branch earlier in
the same if/else chain, so it could never be reached. Switched to "&"
(unused by any other sigil) per explicit user decision.
Also fixed a real crash risk while making this reachable: getenv() returns
NULL for an unset variable, and MString's const char* constructor calls
strlen() unconditionally - now throws a clean "environment variable is not
set." error instead.
Verified via real jet-2.0 runs: a set variable reads and applies modifiers
(;toupper) correctly, an unset one throws cleanly rather than crashing.
Full regression suite and 30/30 compile check still pass.
A RAND_bytes()-generated 160-bit session id colliding with an existing session file is astronomically unlikely, but the check is cheap and removes any dependence on that probability - regenerates (up to 5 attempts, then throws) rather than ever reusing an existing sessions file.
SHA1("JETSESSION" + time(0)) meant two sessions started in the same second got the identical id, and the whole 160-bit value was derivable from a public, second-granularity input - not a real session identifier. Replaced with 20 bytes straight from OpenSSL RAND_bytes(), already linked via -lcrypto. Verified via a real jet-2.0 test: session get/set/persist across separate processes still works via the cookie, and two back-to-back new-session requests now get genuinely different ids.
Parsed its arguments correctly but had an empty body (a literal
"// TODO: Implement DATEDIFF." comment), never setting a result - isNumber
was left uninitialized on this path.
Matches MySQL's own DATEDIFF(expr1, expr2): expr1 - expr2 expressed as a
whole number of days, ignoring time-of-day. Dates parse as YYYY-MM-DD via
strptime (a longer "YYYY-MM-DD HH:MM:SS" value parses fine too, since
strptime only consumes what the format string asks for) and are
interpreted as UTC via timegm() rather than mktime(), so the result
doesn't shift with the server's own timezone. An unparseable date throws
a clear, specific error instead of producing garbage.
Verified via real jet-2.0 runs: datediff('2026-08-26','2026-08-20') -> 6,
the reverse -> -6, same date -> 0, a full non-leap year -> 365, and an
invalid date string throws cleanly rather than crashing or silently
returning nonsense. Full 9-script regression suite still passes.
Both matched their function name but had empty branch bodies, silently
producing nothing regardless of arguments. reverse(str) follows the same
single-argument shape as trim/toupper/tolower; concat(a, b, ...) follows
the same variadic comma-separated shape as max/min.
reverse() converts to std::string and reverses in place via the already-
imported <algorithm> (std::reverse), then assigns back through MString's
std::string constructor. concat() starts from the first argument and
appends each subsequent one via MString's own operator<<(ZString).
Verified via real jet-2.0 runs: reverse('hello') -> "olleh",
concat('a','b','c') -> "abc", concat('x','y') -> "xy". Full 9-script
regression suite (custom tag scoping, dump tag tree, try/catch, if/else,
keyword sigil) still passes unchanged.
Found while writing a new Expressions Reference chapter for the manual,
where every function got run for real before being documented rather
than trusted from reading the source alone - substring('helloworld', 7)
threw "operand is not valid." immediately after producing the correct
answer, confirmed via debug tracing showing an extra Operand constructed
on empty input right after the 2-arg case finished.
Root cause: a missing pair of braces around the 3-argument-only branch
meant Operand parm3(in, tag) and its own closing-paren check ran
unconditionally, even after the 2-arg form's own "if(in.ifNext(')'))"
check had already matched and produced string = parm1.string.substring(
parm2.string.asInteger()) - so it immediately tried to parse a third
argument from whatever was left (nothing, since the real closing paren
was already consumed), and construction on empty input always throws.
The 3-argument form (substring(str, start, length)) was unaffected,
since its own path never hit the first if-branch's early exit.
Fixed by properly scoping the 3-arg branch inside an else, so it only
runs when the 2-arg form's own closing paren wasn't found. Verified both
forms with real jet-2.0 runs: substring('helloworld', 7) -> "rld",
substring('helloworld', 0, 5) -> "hello".
expansion terminated the whole process instead of reporting cleanly
Found while verifying documentation for #[name] vs $[%name] (writing the
new tag-hierarchy section surfaced that my own published example was
factually wrong about which sigil reads a keyword attribute - #[name]
actually reads local->variables, a separate store; $[%name] is the real
keyword sigil, confirmed against Tag::getVariable directly). Referencing
an uninitialized #[name] from inside a custom tag's own template crashed
the process outright (SIGABRT) rather than reporting a normal error -
confirmed via a real gdb backtrace, not guessed:
1. Tag::getVariable's "#[" branch and Tag::assignVariable's "local"/
"parent" scope branches all dereferenced `local` unconditionally.
`local` is genuinely NULL at the true top level (jet-2.0.cpp's own
`new jet::__jet(data, out, global, NULL, NULL)`) and stays NULL through
any tag that doesn't establish its own local scope (most don't -
<if>/<while>/<until> are the exception, passing `this` as local via
their splitTagName constructor path). Added NULL checks with clear
messages instead of an unconditional dereference.
2. The real root cause of the actual crash observed, though: jet-2.0.cpp's
own top-level catch(coreutils::Exception) handler unconditionally calls
data.setCursor(global.errorCursor) to show precise source-line context
- but global.errorCursor is updated by every Tag's own construction, so
it can (and, for an error originating inside a custom tag's own
expanded template - container3 in __dotag.cpp - always does) point
into a completely different ZString buffer than `data` (the top-level
script). ZString::setCursor() correctly rejects a cursor outside its
own buffer by throwing - but that throw happens while the ORIGINAL
exception is still being handled inside this same catch block, and a
second exception escaping an active catch block calls std::terminate()
immediately, regardless of any enclosing try/catch. Confirmed via gdb:
the abort's own stack showed __cxa_throw called from
ZString::setCursor, called from the catch block itself.
Fixed by wrapping the precise-location diagnostics in their own
try/catch, falling back to just the error text (already printed first)
plus a note that precise context isn't available, rather than crashing
the whole process. The error text is the useful part either way - the
line-context is a bonus only available when the cursor genuinely
belongs to the top-level buffer.
Verified via a full regression pass (9 real jet-2.0 runs covering custom
tag scoping at 1/2/3 nesting levels, the dump tag tree, <try>/<catch>,
<if>/<else>, and the corrected $[%name] keyword sigil) - all still work,
plus the specific crash reproduction now reports a clean error instead of
aborting.
two real, previously-latent bugs found while building this
Custom tag definitions used to be entirely flat: every <tag name="..."> -
regardless of where in the script it appeared - registered into a single
global.tags map, visible everywhere for the rest of the script. Now a
<tag> defined directly inside another custom tag's own invocation (i.e.
nested inside a __dotag instance's own expansion, not just sitting inside
an <if>/<for>/<while> that happens to be there - those remain transparent)
is scoped to that invocation: visible only for the rest of that specific
expansion, gone once it finishes. Definitions at the true top level (or
inside control-flow tags that are themselves at the top level) still
register globally, unchanged.
Implementation: Tag gained isCustomTagBoundary (true only on __dotag
instances) and localTags (populated only on boundary instances) - both
public rather than protected, since __tag.cpp needs to read/write them
through an arbitrary ancestor Tag* found by walking parent, and C++
protected access doesn't extend to another object reached through a
base-typed pointer. findTagDefinition() walks from `this` up through
`parent` looking for the nearest boundary whose localTags has the name,
falling back to global.tags - used by both ifTagDefined() (is <name>
visible right here?) and __dotag's own constructor (fetch the definition
to expand). __tag.cpp's registration does the mirror walk starting from
`parent` (skip self, a <tag> definition is never itself a boundary).
Two real bugs found and fixed along the way, neither introduced by this
feature, both confirmed via a hung/broken real jet-2.0 run before
diagnosis rather than assumed:
1. __tag.cpp's own base Tag(...) call passed (this, this) as its own
parent/local instead of the parent/local it actually received - made
the new scope-walk loop on itself forever (since <tag>'s own `parent`
would just point at itself). Fixed to pass through the real received
parent/local; nothing else in this class depended on the old
self-referential values (a <tag> definition's own container is never
evaluated at definition time - evaluate=false - so parent/local were
never used for anything else here).
2. __dotag never overrode hasContainer, which Tag::processContainer gates
real rendering on - and the base constructor sets hasContainer from
whether THIS INVOCATION was self-closing (<name /> vs <name>...</name>),
which is the right question for capturing an invocation's own nested
content, but the wrong one for whether the tag's OWN DEFINITION should
render. Net effect, confirmed via a real run: <name /> always silently
rendered nothing for ANY custom tag - only the <name></name> form ever
worked. Fixed by forcing hasContainer=true in __dotag's constructor -
by the time it's constructed a valid definition is guaranteed (dispatch
only builds __dotag after ifTagDefined() already confirmed one exists),
so there's always something to render regardless of invocation syntax.
<dump> updated to match: its "DEFINED TAGS" section now renders the real
tree currently in effect (global at the root, each enclosing custom-tag
invocation nested one level deeper, outermost first) instead of a single
flat list - if there's no active local scope, that's just the root with
nothing nested under it.
Verified via real jet-2.0 runs, not just compilation: a tag defined inside
another tag's body is invokable from within that body and correctly NOT
recognized (falls through to literal-text echo) outside it; the same
holds through 3 levels of nesting; the dump tree correctly shows nested
levels indented under their own "(inside X)" header. Full regression pass
on <try>/<catch>, <if>/<else>, <while>, and dump's own flat top-level
output - all still work unchanged.
Custom <tag name=...> definitions have no scope levels the way variables
do - __tag.cpp always registers into the single global.tags map regardless
of where in the script the definition sits, so this is one flat list, not
broken out by global/local/parent like the existing variable/keyword
sections. Verified end-to-end via a real jet-2.0 run: two tags defined
before <dump> both appear correctly in the DEFINED TAGS section.
split-container mechanism
<try>...container...<catch>...container2...</catch-less, matching <else>'s
own convention> intercepts any error that would otherwise propagate up and
halt the whole script - a coreutils::Exception (what every other tag
actually throws), any other std::exception (e.g. the
std::filesystem::filesystem_error <whiledir> can raise), or anything else.
Optional name=/scope= keywords bind the caught error's message to a
variable for the catch block to reference, matching <for>'s own
name=/scope= convention. New class named __trytag rather than __try: <ios>
transitively defines a preprocessor macro "#define __try try" (an SEH-
compatibility shim pulled in on this toolchain) that silently rewrites the
identifier __try to the keyword try wherever it appears, breaking
compilation - confirmed via `g++ -E -dM` before picking this name. The JET
tag name itself is still plain <try>.
Reuses the same container/container2 split mechanism <if>/<else> already
has (Tag's constructor takes an optional splitTagName), which surfaced a
real, previously-latent bug: the code that consumes the split tag's own
opening delimiter after a match was hardcoded to the literal string
"<else>" instead of being built from splitTagName. This happened to work
for every existing split-tag user (<if>, <while>, <until> all use "else"),
but for <try>'s "catch" the hardcoded ifNext("<else>") never matched, so
the cursor never advanced past <catch> and the outer scan loop matched the
same unconsumed <catch> forever - a genuine infinite loop, not just wrong
output, confirmed via a real hung jet-2.0 run before diagnosis. Fixed by
building the actual expected delimiter ("<" + splitTagName + ">")
dynamically instead.
Verified via real jet-2.0 script execution (not just compilation): a
caught <read> failure correctly binds its error text to a name= variable
and skips straight to the catch block, discarding whatever partial output
the failed attempt had already written; an untriggered <try> runs its own
content normally and skips <catch> entirely; a <try> with no <catch> at
all silently swallows the error and the script continues past it. Also
regression-tested <if>/<else> and <while> (both depend on the exact same
fixed code path) to confirm nothing broke for the existing "else" case.
Its source uses <whiledir dir=... name=...>, but the tags real keyword is
path= (confirmed in __whiledir.cpp) - dir/name are not recognized, so this
throws whiledir tag must specify a path. at runtime, same as barant.com own
mismatched source would. Converted through the normal pipeline rather than
omitted, same treatment as vst/modular1/modular1_patches: verified it fails
with a clean HTTP 500 via APIServers existing exception-catching path, and
the server stays alive and responsive throughout.
<database> was SESSIONCTL.tag's own JET-1.4-only concept - the new JetCore
port never implemented it. The real, dispatched tag for opening a MySQL
connection is <mysql> (confirmed in Tag.cpp's dispatch chain), so these 3
product pages - the ones deferred in the previous conversion pass because
they pull live rows via <sql> - now wrap their content in a real
<mysql host="localhost" database="barant" user="barant" password="...">
matching barant.com's own config.cfg, with <retain> (a SESSIONCTL-only
form-echo helper with no analog here) stripped.
host="localhost" is correct per config.cfg but only reachable from
barant.com itself - a locally-run .jtc genuinely cannot connect from here.
That's intentional: the connection fails at runtime exactly as if the
database were down, and APIServer's existing catch(coreutils::Exception)
path (ApiSession.cpp) already turns that into a clean 500 rather than a
crash - verified live: all 3 pages return HTTP 500 with mysql's own
"database and host parameters are not valid." message, and the server
process stays up and responsive throughout.
The real DB password is deliberately not a literal in this file - it's
read from BARANT_DB_PASSWORD at conversion time, and the resulting .jet
output for these 3 pages is gitignored rather than committed, so the
credential never enters git history even though it does end up baked into
the locally-generated, locally-deployed .jtc.
Pulls .page files and shared template infrastructure (mainpagetemplate.temp,
menu.bar, barant.tag, opensource.menu, cart.menu, login.menu) read-only from
barant.com via SFTP into jetc/barant-site/src, then convert.py mechanically
flattens them into self-contained .jet files: linkbutton becomes a plain
anchor link (the old SESSIONCTL PAGE/PROCESS form-postback model has no
equivalent here - each page is its own real #api route now, not a shared
single-form state machine), sectionframe is expanded per its barant.tag
definition, /images/* references are fetched and inlined as data URIs
(APIServer has no static-file serving), and session-gated chrome (cart,
account/admin, login.menu) is resolved to its anonymous-visitor default
state, since no session/DB backend is wired into this conversion.
17 pages convert and serve cleanly end-to-end (verified via a local
apiserver-barant-site.json instance on :8081, kept separate from the
existing demo server on :8080 to avoid a /main route collision with the
earlier hand-cloned barant.jet). Two pages deliberately out of scope for
this pass: davidarant.page (whiledir filesystem iteration) and the
per-topic sub-pages of jet.page (dispatches on an ARGS query param this
conversion does not populate - only the default overview state converts).
vst.page/modular1.page/modular1_patches.page also deferred - they pull
live product rows via sql, a real backend dependency beyond "pages and
their template" scope.
Nothing was written back to barant.com - SFTP access was read-only
throughout, per explicit instruction to convert locally without touching
the original site.
The two <img> tags used relative /images/... paths, which 404 through
APIServer (no static-file serving) exactly like the stylesheets did.
The three @font-face src urls were absolute https://barant.com/fonts/...
references - technically reachable, but barant.com does not send an
Access-Control-Allow-Origin header for them, so browsers block loading
them cross-origin from a different serving host. Fetched all five and
embedded as base64 data: URIs so the served page is fully self-contained.
APIServer has no static-file serving, only #api-declared routes, so the
original relative /css/main.css and /css/control.css <link> tags 404d.
Fetched both stylesheets from barant.com and embedded them directly in
a <style> block instead.
Fetched the live raw HTML from https://barant.com/ (rendered by the
user's own separate JET-1.4 C interpreter, unrelated to this JetCore/
jetc project - see the barant.com project memory) and embedded it as
literal content in a jetc template (#api GET /barant), as a real-world
stress test of the --shared compiler against markup nobody wrote with
jetc in mind.
Needed several fixes to compile, all discovered by reading jetc's own
JetParser.cpp rather than guessing, and each one either genuinely
DOM-neutral or an actual bug fix in barant.com's own markup:
- Void elements (<meta>/<link>/<input>/<img>/<br>) had to be
self-closed. jetc's parser treats any unrecognized <tag ...> as a
generic custom tag needing a matching </tag> (mirroring the
interpreter's own extensible custom-tag model) - it has no built-in
notion of HTML's void-element list. Self-closing is standard HTML5
syntax with zero effect on the parsed DOM either way.
- <!DOCTYPE html> and HTML comments (<!--...-->) aren't understood at
all - confirmed via a minimal repro that even a bare <!-- comment -->
breaks tag tokenization (the "<!" prefix isn't special-cased).
Routed the doctype through a <set name="x" value="<!DOCTYPE html>"/>
+ $[x] pair instead - a <set> attribute value is a plain quoted
string, never re-parsed as nested markup, verified via host_test to
render byte-identical to the literal text. The one HTML comment
(a Google Analytics label) was just removed outright - comments have
no rendered/DOM effect, so this is neither a workaround nor a
meaningful change, and the actual <script> tags it was labeling are
untouched and verified still present.
- A genuine real bug in barant.com's own live markup: `<table
class="menubar" cellspacing="0" cellpadding="0" width: 100%">` (line
28) - a stray unpaired `"` with no matching partner (missing `=` and
opening quote before `100%`). jetc's attribute scanner toggles into
"inside a quoted value" mode on any `"` and treats everything
(including the real closing `>`) as opaque until the next `"` - a
single unpaired quote anywhere derails tokenization for the rest of
the file, which is exactly what "unterminated tag near end of input"
turned out to mean here. Browsers silently tolerate this malformed
attribute (hence it looking fine live); jetc's simpler tokenizer
doesn't. Fixed to width="100%", the obvious intent.
- Two stray trailing </if></if> lines (a leftover artifact from
barant.com's own JET-1.4 template engine, unrelated to anything here)
were dropped - dead, invisible in any browser, and incompatible with
jetc's tag-matching (no corresponding <if> anywhere).
Verified: diffed the actual rendered output (via host_test) against the
original fetched HTML - every difference maps exactly to one of the
changes above, nothing else moved. Deployed live into the running
APIServer test instance (GET /barant) and confirmed Content-Length
matches the diffed render exactly (13944 bytes).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Purely a naming-convention change per user request, not a format
change - the output is still a real ELF shared object (-shared -fPIC),
still dlopen()-able exactly as before. .jtc ("jet compiled") marks it as
an APIServer-loadable content module rather than a generic library.
build-shared.sh: usage text and STEM derivation
(basename "$OUTPUT" .so -> .jtc) both updated - the STEM change is the
one functional piece here, everything else in this commit is comments/
usage strings. Doc comments in jetc.cpp/JetCodegen.h/host_test.cpp/
api_test.cpp updated to match.
Verified live: rebuilt jetc and all five demo*.jtc modules, reran
api_test/host_test against demo3.jtc (still resolves jetc_api_demo3/
jetc_render_demo3 and renders correctly) - the extension is purely
cosmetic to dlopen(), as expected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Used to live-verify APIServer end to end - one small template per
distinct code path, matching demo.jet/demo2.jet's existing role as
worked examples rather than throwaway scratch files:
- demo4.jet: "#api GET /users/{id}" + $[id] - path param capture.
- demo5.jet: "#api POST /echo" + $[:message] - urlencoded form body via
CGIFormData, reached through the same $[:name] CGI convention the
interpreter already uses.
- demo6.jet: "#api GET /crash" + $[:message] with no body present -
deliberately throws (Tag::getVariable's cgiData-is-null error),
verifying a render-time exception turns into a clean response instead
of taking the server down.
- demo7.jet: "#api GET /search" + $[q] - query string parsing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 1 of the APIServer plan: a compiled --shared .so currently exports
only a bare jetc_render_<name>(Global&, MString&) with no way for a host
to discover what HTTP method+path it wants to serve. Adds a leading
"#api METHOD /path" directive (one or more, parsed the same way jetc
already strips a leading "#!" shebang line - not modeled as a new
AST/tag-attribute concept, since jetc's parser has no existing notion of
the enclosing <jet> tag's attributes to hang this off of).
In --shared mode, JetCodegen now also emits a second exported function
alongside the render entry point:
struct JetApiRoute { const char *method; const char *path; };
extern "C" const JetApiRoute *jetc_api_<name>(int *count) { ... }
<name> is derived the same way for both symbols (sanitizeIdent(
baseNameNoExt(inputPath))), so a future host loader can find both by
convention from just the .so's filename - no name needs to be embedded
in the metadata itself, and the routes a module declares can never drift
out of sync with the render function that implements them (the failure
mode a sidecar manifest file would have).
Malformed directives (missing method/path, path without a leading '/')
are a hard compile error; #api directives present without --shared print
a non-fatal warning (the metadata has nowhere to go without a host to
load a .so, but the render-only executable/CGI path still works fine).
Verified live, not just compiled: rebuilt jetc, compiled demo3.jet (a new
worked example with two #api lines sharing one handler) via
build-shared.sh into a real demo3.so, and added api_test.cpp (mirrors
host_test.cpp's dlopen()/dlsym() pattern) confirming jetc_api_demo3
returns the expected two routes at runtime. Re-ran host_test against the
same .so to confirm the render path is unaffected. Confirmed demo.jet/
demo2.jet (no #api lines) emit no JetApiRoute code at all - no regression
for templates that don't declare routes.
Also fixed a gitignore gap surfaced while committing this: an earlier
"jetc/*.cpp" ignore rule (meant for generated *.compiled.cpp output) was
a blanket pattern that silently hid this commit's own api_test.cpp from
git status. Narrowed to "jetc/*.compiled.cpp" (the actual generated-file
naming, per build-compiled.sh/build-shared.sh) and added explicit ignores
for the host_test/api_test/demo3 binaries plus two known stale generated
leftovers (demo.cpp/demo2.cpp) that predate this session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TODO.txt: drop the stale "Fix the variable retriever..." bug-list entry -
long since fixed. tests/testjet.jet: cgi="true" -> cgi="yes", matching the
deliberate "yes" convention already used elsewhere (__jet.cpp checks
resolveKeyword("cgi") == "yes", not "true"). tests/dump.txt: regenerated
output from re-running the test after the above.
.gitignore: jetc/ was accumulating untracked build output (compiled .cpp,
binaries, .pic.o, .so, libCoreUtils_pic.a) alongside its real tracked
source - ignore the generated artifacts, keep the two worked-example
.jet templates explicitly un-ignored since *.cpp/no-extension patterns
would otherwise also catch demo.jet/demo2.jet's own extension-less
compiled binaries without excluding the .jet sources themselves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
renderVariableName() built the variable name by whitelisting an
identifier charset (letters/digits/._-) via getTokenInclude(), then
threw "invalid variable name." on the first character outside that set
- including a plain space. Real-world CGI form field names aren't
restricted to identifier characters (an HTML <input name="first name">
is completely valid), so $[:first name] failed outright instead of
looking up the field as sent.
Rewrote to exclude only the genuinely structural characters (']', ';',
and the start of a nested $[.../#[...substitution) via getTokenExclude,
so everything else between the brackets - spaces included - is consumed
as part of the name before any lookup happens, rather than stopping
early and erroring. This also let the separate ':' handling branch be
dropped entirely: since ':' is no longer excluded, it now flows through
the same sweep naturally (needed for both the CGI sigil name itself and
the "$[:name:2]" explicit-index suffix).
Verified live: $[:first name] against a real multipart CGI body with a
field literally named "first name" now resolves correctly (previously
threw before ever reaching the lookup). A stray, non-nested '$' or '#'
in a name is kept as literal text rather than erroring. Re-verified
existing sigils are unaffected - plain $[x], $[!x] (forced-global), and
tests/posttest.sh's ^session/:.name3 lookups all still resolve exactly
as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
When resolving a bare $[var]/#[var] reference, Operand set .boolean =
true whenever the value's text was EITHER "true" OR "false" -
effectively "boolean = looks-boolean-ish" rather than reflecting which
of the two it actually was. <if expr="..">/<while expr="..">check
exactly this .boolean field (see __if.cpp/__while.cpp), so <if
expr="$[flag]"> took the true branch even when flag held the literal
text "false".
Fixed to boolean = (string == "true").
Verified live: a variable set to value="false" now correctly takes an
<if expr="$[var]">'s false branch, and value="true" still correctly
takes the true branch. Re-ran tests/posttest.sh - unaffected, no
regression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CONTENT_TYPE was "multipart/form-data" with no boundary= parameter - a
real browser/curl request always includes one. Without it,
__jet.cpp's contentType.find("boundary=") returns -1, so boundary stays
a default-constructed empty ZString. CGIFormData's opening-marker search
then becomes just "--" (boundary appended to it), which matches
trivially at the very start of the body text, so the parser immediately
reports Status::COMPLETE having consumed only those two bytes and parsed
zero fields - no error, no exception, just silently empty. Every
$[:name] lookup returned blank as a result, not just $[:name3] (verified
- all six of post.example.formdata's fields were affected identically).
Fixed CONTENT_TYPE to carry the real boundary value used in
post.example.formdata ("----WebKitFormBoundaryA1LWIjMAIsdFW9XA", matching
each part's "--" + boundary delimiter line), and corrected
CONTENT_LENGTH (619 -> 623, the fixture's real current byte count - it
had grown since 619 was set, from prior edits giving each form field a
distinct name instead of several duplicate "name" fields).
Also includes the already-in-progress edits to post.example.formdata
(distinct field names name/name2.../name6, replacing several duplicate
"name" entries), post.example.urlencoded (matching field names), and
testpost.jet (added a $[:name3] line) that were sitting uncommitted -
these are what surfaced the bug in the first place, not a new change
made alongside this fix.
Verified live: all six $[:name]..$[:name6] now resolve to their exact
raw multipart values (1 test / 2&test / 3+test / 4 test / 5 / 6 - no
CGI-decoding applied, correct for multipart bodies, unlike the
urlencoded-body path which does decode).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For the server you're about to build: `jetc --shared template.jet out.cpp`
now emits no main() at all - instead the render logic is exported as
`extern "C" void jetc_render_<template>(jet::Global &global,
coreutils::MString &out)`, one call per request, meant to be dlsym()'d
by a long-running host process instead of spawning a jet-2.0 process (or
even a separate compiled executable) per request. extern "C" here only
fixes the symbol's name mangling for a stable dlsym() target - the
parameter types are still real C++ classes, safe to pass across the
dlopen() boundary since the host and the .so share the same
CoreUtils/JetCore headers and toolchain (a same-ABI plugin boundary, not
a foreign C one). coreutils::Exception propagates out of the call
uncaught, so the host decides how to handle a render failure rather than
the module unilaterally deciding (the executable mode's try/catch prints
to stdout and exits, which is CGI-appropriate but wrong for a persistent
server). The exported symbol name is derived from the input filename
(jetc.cpp's baseNameNoExt+sanitizeIdent) so multiple compiled templates'
.so's stay collision-free even under dlopen(..., RTLD_GLOBAL).
jetc/build-shared.sh builds the .so: JetCore's 22 tag classes are still
force-linked directly (Tag::interpretSpan's fallback path can dispatch
to any of them at runtime, regardless of what a given template
statically references), but CoreUtils objects are compiled -fPIC and
archived into a throwaway libCoreUtils_pic.a instead of being linked
directly - libCoreUtils.a itself isn't position-independent, so linking
it into a .so fails outright ("relocation ... can not be used when
making a shared object"), and force-linking every CoreUtils .cpp
directly (rather than archiving, so the linker only pulls in what's
actually referenced) surfaced an unrelated, pre-existing incomplete
class - IMFMessage's vtable is never emitted anywhere, presumably a
partially-migrated class (see project memory on IMFMultipart/IMFText
still being fd-based) - as an unresolved dlopen()-time symbol, even
though nothing here uses it. A normal archive-based link (like the
existing libCoreUtils.a already does for jet-2.0) never pulls IMFMessage
in at all, so building libCoreUtils_pic.a the same way sidesteps it
without touching the unrelated bug itself.
jetc/host_test.cpp is a minimal stand-in for the real server: dlopen()s
a compiled .so, dlsym()s its entry point, and calls it twice with fresh
jet::Global objects (simulating two separate requests against one loaded
module) to prove the .so is genuinely reusable, not single-use. Verified
both demo templates (demo.jet, demo2.jet - the latter covering nested
if/for, a stateful while loop, and the interpretSpan fallback path) build
as .so's and produce the same byte-identical output as the executable
and interpreter paths, across two calls each. Also re-verified the
existing (non-shared) executable build path still matches jet-2.0's
output after the Codegen::generate() signature change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jetc (JetCore/jetc/) compiles a subset of JET template syntax - literal
text, $[...]/#[...] variable output, <set>, <if>/<else>, <for>, <while> -
into a standalone C++ program that links against CoreUtils and the same
JetCore tag-class library jet-2.0 itself uses. The static tag/text
structure becomes real C++ control flow (if/for/while), resolved once at
build time instead of being re-scanned character-by-character from raw
source on every request and, for loop bodies, on every single iteration
(confirmed by reading Tag::parseContainer - it re-tokenizes its container
text from scratch every time it runs, since a tag's parse and its
execution are the same pass in this codebase). Anything genuinely
dynamic - variable values, expr/value comparisons, loop bounds - still
calls into the existing runtime (Tag::getVariable/resolveKeyword,
jet::Operand) exactly as the interpreter does; compiling doesn't try to
eliminate that; only the repeated re-parsing of the template's own shape.
A <if type="..."> or <while type="..."> comparison type is resolved to
its C++ operator once at compile time (type is a literal attribute value
in practice, not runtime data) instead of being re-checked against six
string constants on every evaluation.
Any tag jetc doesn't have a codegen rule for (mysql, sql, call, include,
a <set> with a container body, etc.) is captured as its exact raw source
span and handed to the existing interpreter at compiled-program runtime
via the new Tag::interpretSpan(), so an unsupported construct degrades
gracefully instead of failing to compile - a real, if narrower, program
rather than requiring full tag coverage up front.
Tag.h/.cpp: made assignVariable public (parallel to the already-public
getVariable - generated code needs a write path without being a Tag
subclass) and added interpretSpan() as the fallback's entry point into
parseContainer.
Verified live: two representative templates (jetc/demo.jet, an if/for/set
walkthrough; jetc/demo2.jet, a nested <if> inside a <for>, a <while> that
mutates and rereads its own condition variable across iterations, and a
fallback-interpreted <comment> tag) both compile, link, run standalone,
and produce byte-identical output to jet-2.0 running the same source.
Building and running jetc/demo2.jet's <while> loop surfaced a real,
previously-unknown MString bug (fixed separately, see CoreUtils commit
6e4e463) along with a separate, already-existing bug in __while.cpp's
value1/value2/type form (unrelated to this compiler, not fixed here -
see project memory).
jetc/build-compiled.sh builds the compiler and a given .jet file's
compiled binary in one step; jetc/demo.jet and demo2.jet are kept as
worked examples. Known limitations: <set> with a container body isn't
modeled (falls back to interpretation); a fallback-interpreted tag
nested inside compiled control flow uses the template's single top-level
scope for parent/local context rather than a faithful per-level chain,
which only matters for legacy tags that rely on nested "local" scoping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The @ sigil previously read through global.getSession(global.sessionId),
a MySQL-session binding that was already confirmed dead this session -
Global::sessionId is never assigned anywhere in the codebase, so any
$[@name] read would always throw. The design goal: persist session state
as a JSON file at sessiondir/<sessionId>.json, using CoreUtils's existing
JFile (an MFile+JString combination with load-on-construct/save-on-
destruct RAII semantics), reusing @ rather than introducing a new sigil
since the old @ usage was unreachable in practice anyway.
Global.h/.cpp: added Global::sessionFile (a JFile*, NULL unless a request
is in cgi mode with sessiondir set). Its destructor now explicitly
deletes sessionFile - JFile's actual disk write happens in MFile's
destructor via update(), so this has to run for persistence to happen at
all (a raw `new`'d pointer that's never deleted, as cgiData still is,
would never save anything back to disk).
__jet.cpp: the sessiondir block now builds the session file path from the
sessiondir keyword and the active session id (existing cookie or newly
issued) and constructs global.sessionFile there, for both branches. The
"pull sessionfile from sessiondir" comment is now implemented - JFile's
own constructor already handles "file doesn't exist yet" by loading as
empty, so one construction point serves both a resumed and a brand-new
session.
Tag.cpp: getVariable's "@" branch now reads via (*global.sessionFile)[name]
with a clear error if no session is active, instead of the dead MySQL
lookup. Added Tag::assignVariable (declared in Tag.h), a single write
path both storeVariable overloads now funnel through, so a "@name"-
prefixed target (e.g. <set name="@x" value="y"/>) is routed to the
session file instead of the normal global/local/parent variables maps -
without this, the read side would have been read-only, which isn't a
very useful session mechanism.
This surfaced two real, pre-existing CoreUtils bugs, both fixed
separately (see CoreUtils commit 3e6e410): JFile's virtual-base diamond
skipping JString's own empty-object seed, and MString's ZString-source
copy paths pairing an absolute data pointer with a cursor-relative
length - both were silently corrupting any cursor-advanced ZString
(exactly what assignVariable produces when it strips the "@" prefix via
ifNext() before storing).
Verified live: two genuinely separate jet-2.0 process invocations
sharing one session cookie, where the first writes two session
variables and the second - a fresh process with no shared memory - reads
both back correctly from the JSON file on disk. Also re-ran
tests/posttest.sh (unchanged output, no regression) and the full
CoreUtils test suite (no regressions).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
$[^session] (a cookie lookup, global.cookies.data[name]) was always
empty because __jet.cpp's cgi keyword check compares against "yes" (a
deliberate choice from an earlier session, not the codebase-wide
true/false convention used by filterblanklines/trimlines/eval etc.) -
testpost.jet had cgi="true", a stale mismatch that predates that
decision, so the entire cgi block (including reading HTTP_COOKIE into
global.cookies) never ran at all. Fixed the test fixture to say
cgi="yes" to match.
Verified live via tests/posttest.sh against the current build:
$[^session] now correctly prints 18fd56a83f (posttest.sh's simulated
HTTP_COOKIE session id), and dump.txt's *** COOKIES *** section shows
session=[18fd56a83f] instead of empty. dump.txt committed with this
fresh, correct output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Global now owns a long-lived StreamReader (cgiReader) that CGIFormData
(a Parser) holds a reference to - declared before cgiData so it's
constructed first and destroyed last. __jet.cpp's POST handling
constructs CGIFormData against it and feeds the request body off stdin
in 4KB chunks, driving doParse() after each feed rather than blocking to
read the whole body up front - the same feed()/doParse() pattern a
socket-fed StreamReader would use, keeping memory bounded for large
multipart uploads.
Also fixes two real, pre-existing bugs surfaced while getting an actual
end-to-end CGI request working through jet-2.0 (not just compiling):
- Tag::Tag's opening-tag parser was gated on "parent && in.ifNext(\"<\")",
but jet-2.0.cpp's main() constructs the top-level __jet with
parent=NULL - meaning the outer <jet ...> tag's own attributes
(cgi="yes" and everything else) were never actually parsed at all, just
echoed as literal passthrough content. parent=NULL is used at exactly
one call site in the whole codebase, so dropping the "parent &&"
requirement only changes behavior for that one top-level case - every
nested tag construction already passes a real parent.
- __jet.cpp's boundary-extraction code calls contentType.find("boundary=")
to locate the multipart boundary parameter, but find() repositions
contentType's own cursor to the match as a side effect. That same
cursor-shifted contentType then got passed into CGIFormData's
constructor, whose startsWith("multipart/form-data")/
startsWith("application/x-www-form-urlencoded") checks are
cursor-relative and so silently matched neither, falling through to an
unrecognized content type that parsed nothing (reported COMPLETE
immediately, no error, just empty results). Fixed with
contentType.reset() before the CGIFormData construction.
Verified with a real end-to-end run: jet-2.0 invoked with simulated CGI
environment variables (REQUEST_METHOD=POST, CONTENT_TYPE with a
boundary=, CONTENT_LENGTH) and a genuine multipart/form-data body
(including a file-upload part) piped over stdin, using $[:name] in a
.jet script - correctly returns each field's value and correctly leaves
the skipped upload field empty.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- __jet.cpp: the cgi keyword check compared against "true" but the
documented/intended value is "yes", so <jet cgi="yes" ...> never
actually turned CGI mode on; fixed. Also replaced the hardcoded
fake multipart boundary ("xxxx") with real extraction from the
Content-Type header's boundary= parameter.
- Global.h: default-initialize cgiData to NULL (a GET request with
cgi="yes" never allocates it, so it was previously left
dangling); Global.cpp now sets the cookie jar's delimiter to ";"
to match RFC 6265, since it defaulted to CGIFormattedData's
&-based form-data delimiter.
- Global.cpp: fix addSession()'s missing throw that let it silently
continue past a duplicate session id.
- Tag.cpp: fix the CGI-variable lookup branch, which failed to
compile (cgiData is a pointer; needed to dereference it before
indexing) and added a null check so a CGI request that never
populated cgiData (bad Content-Type, or the null-safety gap
above) throws a clear error instead of crashing; implemented the
previously-empty session-variable lookup branch; left the
unreachable duplicate environment-variable branch as-is with a
comment, since fixing its sigil is a user-facing template-syntax
decision, not a bug fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
