Multi-day accumulated fixes: tag dispatch, session/pooling correctness, SPA rollout, jetc compiler
JetCore engine (Tag.cpp/.h, Global.cpp/.h, __*.cpp tag classes): - Numerous __mysql.cpp connection-pooling correctness fixes (stale result sets, double-indirection sessionid bugs, field caching, pool keying) and a session-registration-before-connection-confirmed ordering fix (dangling pointer on a failed connect) - Tag.cpp: large body of fixes to the tag-dispatch/container-scanning logic (bare-name fallback, database aliasing, single-quote handling, @-prefix session variable resolution, TAGDEF caching, nested split-tag scanning for if/while/until/ifrow so an inner one doesn't corrupt an outer one's own boundary scan) plus the process-first rendering order fix (a <process> now runs before <checksession>/page resolution, fixing a real "blank page after successful login" bug) - Global.h/.cpp: added imapSessions (parallel to the existing __mysql* session map) for the new <imap> tag family - __whiledir.cpp/__include.cpp: caching and directory-name-aliasing fixes - __call.cpp/.h: fixed a dangling-pointer bug - __write.cpp: fixed an O_WRONLY-only open that couldn't create a new file - __ifrow.cpp/__sql.cpp: fixed the same sessionid double-indirection bug __mysql.cpp had (resolveKeyword's result was being used as a map key lookup instead of the resolved value itself) jetc compiler (jetc/JetParser.cpp/.h, JetCodegen.cpp/.h, JetAST.h, jetc.cpp, build-shared.sh, build-compiled.sh): - Parser fixes: inline <include>, nested-quote handling, void-tag recognition - Codegen fixes and expanded AST support - build-shared.sh: atomic rename into place for a freshly-built .jtc (a live server dlopen()ing the same path mid-rebuild was corrupting its own already-mapped view of the old file) and a staleness check so a cached JetCore .o doesn't silently keep serving yesterday's fixes forever - Added -lssl to both build scripts' link lines (needed once the new <imap> tag's client-side TLS code existed - previously nothing in this codebase used OpenSSL's SSL/TLS client API, only libcrypto) Site content (jetc/barant-site/): SPA (fetch-and-swap navigation) rollout across essentially every .page/.jet/.section/.tag/.proc/.method file - dynamic-content fixes, fragment-mode stripping, native $[@name] session variables replacing older mechanisms, and this session's own checkout.page redesign, login start-param redirect fix, and per-page side-menu (menu.bar/$[PAGE].menu) refresh support. Completed the earlier main.css/control.css move from assets/ to assets/images/ (git still showed the old paths as tracked-but-deleted until now). Deliberately left out of this commit: dozens of .pre-<description>- <date> manual backup files (this session's and earlier ones' own before/after snapshots, not meant for permanent history), compiled .jtc build outputs, ad hoc test binaries and scripts under jetc/ (login_asan, wcwc_test, optest2_bin, etc.), and jetc/__pycache__/ - none of these are meant to be tracked; a .gitignore covering them would be worth adding separately.
This commit is contained in:
Brad Arant committed
1 parent
54b6e522f9
commit
b0cadc3f08
162 files changed
+3567
-2887
No files matched your search
+173
-20
@@ -12,28 +12,27 @@ IMG_CACHE = ROOT + "/assets/images"
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
os.makedirs(IMG_CACHE, exist_ok=True)
|
||||
|
||||
IMG_MIME = {
|
||||
".png": "image/png", ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
|
||||
".gif": "image/gif", ".svg": "image/svg+xml", ".webp": "image/webp",
|
||||
".ico": "image/x-icon",
|
||||
}
|
||||
|
||||
|
||||
def resolve_images(html):
|
||||
"""Find any remaining /images/<file> reference (src="..." or url('...'))
|
||||
and inline it as a data: URI, fetching+caching from barant.com on first
|
||||
use - APIServer has no static-file serving, same reason the chrome logo
|
||||
and page CSS/fonts needed inlining earlier this session."""
|
||||
"""Find any /images/<file> or images/<file> reference (src="...",
|
||||
href="...", or CSS url('...')) and repoint it at the live APIServer
|
||||
GET /assets/{filename} route instead of inlining it as a data: URI -
|
||||
fetching+caching from barant.com into the same assets/images/ cache
|
||||
directory on first use (unchanged), just no longer embedding the raw
|
||||
bytes into every page's own HTML (undoes the earlier base64-inlining
|
||||
approach now that a real static-asset module exists to serve these).
|
||||
Matches both the absolute form (/images/x.png) and the relative form
|
||||
(images/x.png, no leading slash) - the previous version of this
|
||||
function only matched the absolute form, silently leaving several
|
||||
real <img src="images/.."> references un-rewritten and pointing
|
||||
nowhere (APIServer has no /images/ route at all)."""
|
||||
def repl(m):
|
||||
path = m.group(1) # e.g. images/grid.png
|
||||
fname = path.split("/")[-1]
|
||||
ext = os.path.splitext(fname)[1].lower()
|
||||
mime = IMG_MIME.get(ext, "application/octet-stream")
|
||||
local_path = IMG_CACHE + "/" + fname
|
||||
if not os.path.exists(local_path):
|
||||
subprocess.run(["curl", "-s", "-o", local_path, "https://barant.com/" + path], check=True)
|
||||
return data_uri(local_path, mime)
|
||||
html = re.sub(r'/images/([A-Za-z0-9_.-]+)(?=["\'])', repl, html)
|
||||
return "/assets/" + fname
|
||||
html = re.sub(r'/?images/([A-Za-z0-9_.-]+)(?=["\'])', repl, html)
|
||||
return html
|
||||
|
||||
|
||||
@@ -199,6 +198,72 @@ opensource_menu = flatten_linkbuttons(opensource_menu)
|
||||
|
||||
SIDEBAR_HTML = opensource_menu
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SPA bootstrap script: present once, in the chrome, on every FULL page load
|
||||
# (never in a fragment-only response, which has no <head>/<script> of its
|
||||
# own). Intercepts clicks on internal <a href="/..."> links and swaps just
|
||||
# #jetc-content via fetch('/name?fragment=1') instead of a full navigation;
|
||||
# also drives the browser back/forward buttons via popstate. See the
|
||||
# FULL_PAGE_TMPL / FRAGMENT_TMPL split in convert_page() for how a single
|
||||
# route serves both the full page and the fragment-only response, decided
|
||||
# by the ?fragment=1 query flag.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Pages excluded from client-side fragment fetching entirely - a click on
|
||||
# one of these falls through to an ORDINARY browser navigation (hitting
|
||||
# the full-page route, confirmed safe) instead of being intercepted.
|
||||
# Real, deterministic finding: <name>?fragment=1 for a MYSQL_PAGES page
|
||||
# crashes the whole apiserver process on the very FIRST such request
|
||||
# (reproduced repeatedly - not state-dependent, not "two failures in a
|
||||
# row"), while the exact same page's full-page route (no ?fragment=1)
|
||||
# fails cleanly with the expected "database and host parameters are not
|
||||
# valid." and the server survives. Root cause not isolated (a <mysql>
|
||||
# connection failure unwinding through the smaller <tag name="_frag_X">
|
||||
# definition specifically, vs surviving fine through the larger
|
||||
# <tag name="_full_X"> one, for reasons not chased down further given
|
||||
# scope) - this is a mitigation, not a fix, and only matters while these
|
||||
# pages fail (no real DB password configured); worth re-testing once a
|
||||
# real DB is wired in, since a successful <mysql> connection takes a
|
||||
# completely different code path that may not hit this at all.
|
||||
FRAGMENT_EXCLUDED_PAGES = sorted(MYSQL_PAGES)
|
||||
|
||||
SPA_SCRIPT = """<script>
|
||||
(function() {
|
||||
var FRAGMENT_EXCLUDED = %s;
|
||||
function pageNameFromPath(path) {
|
||||
var name = path.replace(/^\\//, '').replace(/\\.page$/, '');
|
||||
return name || 'main';
|
||||
}
|
||||
function loadPage(name, push) {
|
||||
fetch('/' + name + '?fragment=1').then(function(resp) {
|
||||
if (!resp.ok) throw new Error('HTTP ' + resp.status);
|
||||
return resp.text();
|
||||
}).then(function(html) {
|
||||
var el = document.getElementById('jetc-content');
|
||||
if (el) el.innerHTML = html;
|
||||
if (push) history.pushState({page: name}, '', '/' + name);
|
||||
}).catch(function(err) {
|
||||
console.error('jetc-spa: failed to load page', name, err);
|
||||
});
|
||||
}
|
||||
document.addEventListener('click', function(ev) {
|
||||
if (ev.defaultPrevented || ev.button !== 0 || ev.metaKey || ev.ctrlKey || ev.shiftKey || ev.altKey) return;
|
||||
var target = ev.target.nodeType === 1 ? ev.target : ev.target.parentElement;
|
||||
var a = target ? target.closest('a') : null;
|
||||
if (!a) return;
|
||||
if (a.target === '_blank' || a.hasAttribute('download')) return;
|
||||
var href = a.getAttribute('href') || '';
|
||||
if (href.indexOf(':') !== -1 || href.indexOf('//') === 0 || href.indexOf('/') !== 0) return;
|
||||
if (FRAGMENT_EXCLUDED.indexOf(pageNameFromPath(href)) !== -1) return;
|
||||
ev.preventDefault();
|
||||
loadPage(pageNameFromPath(href), true);
|
||||
});
|
||||
window.addEventListener('popstate', function() {
|
||||
loadPage(pageNameFromPath(location.pathname), false);
|
||||
});
|
||||
})();
|
||||
</script>""" % (repr(FRAGMENT_EXCLUDED_PAGES).replace("'", '"'))
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Build the page chrome from mainpagetemplate.temp: the part of <template>
|
||||
# before <contents> and the part after.
|
||||
@@ -208,6 +273,39 @@ temp_raw = read(SRC + "/mainpagetemplate.temp")
|
||||
m = re.search(r'<template>(.*)</template>', temp_raw, re.DOTALL)
|
||||
template_body = m.group(1)
|
||||
|
||||
# mainpagetemplate.temp's own <tag name="title">...</tag> helper is dead
|
||||
# code in this conversion - title substitution happens via a plain Python
|
||||
# string-replace on <exclude>$[_title]</exclude> below, never by actually
|
||||
# invoking this helper. Stripped as genuinely dead code (not a bug fix -
|
||||
# see below for the real one).
|
||||
template_body = drop_tag_block(template_body, "title")
|
||||
|
||||
# Real bug found and isolated via a minimal standalone repro (bisecting
|
||||
# down from "the whole chrome throws 'comment must have a container'"):
|
||||
# a <comment>...</comment> block, when it ends up nested inside one of
|
||||
# this script's own <tag name="_full_NAME">/<tag name="_frag_NAME">
|
||||
# definitions (see convert_page() below) - needed so the fragment/full
|
||||
# <if> branches can hold real HTML content without jetc's compile-time
|
||||
# <if> body parser choking on it - fails at RUNTIME with "comment must
|
||||
# have a container", even though the exact same <comment> block renders
|
||||
# fine as ordinary top-level content. Reproduced with a 4-line synthetic
|
||||
# case (<tag name="x">text<comment>plain text, no nested tags</comment>
|
||||
# text</tag>) - so this is a genuine jetc/JetCore bug in how a custom tag
|
||||
# DEFINITION'S stored text gets re-processed at <dotag> invocation time,
|
||||
# not something specific to this page's content. Flagged for the project,
|
||||
# not fixed here (root cause not isolated further - the base Tag class's
|
||||
# container-boundary bookkeeping is shared by many tags and risky to
|
||||
# change without full regression coverage).
|
||||
#
|
||||
# Worked around instead: this specific <comment> block wraps content
|
||||
# gated on $[_login_id] (show a logged-in session var menu) - the EXACT
|
||||
# same "session-gated, always empty for an anonymous visitor" category
|
||||
# already established as this conversion's design for cart.menu/login.menu/
|
||||
# the ACCOUNT+ADMIN nav tabs (see below) - so stripping it entirely is
|
||||
# consistent with the rest of this file, not a new compromise.
|
||||
template_body = re.sub(r'<comment>\s*<if value1="\$\[_login_id\]".*?</if>\s*</comment>',
|
||||
'', template_body, flags=re.DOTALL)
|
||||
|
||||
# Substitute the chrome-level includes/placeholders that are now fixed for
|
||||
# every page in this conversion (no per-page menubar/menu overrides exist -
|
||||
# confirmed by grepping every source .page file for _menubar/_menu overrides
|
||||
@@ -221,7 +319,13 @@ template_body = re.sub(r'<if value1="\$\[_menu\]" value2="" type="eq">\s*<includ
|
||||
CONTENTS_MARKER = "<contents>"
|
||||
idx = template_body.index(CONTENTS_MARKER)
|
||||
CHROME_BEFORE = resolve_images(template_body[:idx])
|
||||
CHROME_AFTER = resolve_images(template_body[idx + len(CONTENTS_MARKER):])
|
||||
CHROME_AFTER = resolve_images(template_body[idx + len(CONTENTS_MARKER):]) + SPA_SCRIPT
|
||||
|
||||
FULL_PAGE_TMPL = (
|
||||
'<set name="jetc_doctype" value="<!DOCTYPE html>" />$[jetc_doctype]'
|
||||
'<html lang="en"><head><title>%s</title>%s</head><body>'
|
||||
'%s<div id="jetc-content">%s</div>%s</body></html>'
|
||||
)
|
||||
|
||||
|
||||
def convert_page(name):
|
||||
@@ -262,11 +366,60 @@ def convert_page(name):
|
||||
|
||||
chrome_before = CHROME_BEFORE.replace('<exclude>$[_title]</exclude>', page_title)
|
||||
|
||||
full_page = FULL_PAGE_TMPL % (page_title, STYLE_BLOCK, chrome_before, contents, CHROME_AFTER)
|
||||
|
||||
if name in MYSQL_PAGES:
|
||||
# Real, reproducible server crash (heap corruption - "free():
|
||||
# invalid pointer" seen live) when a <mysql> connection failure
|
||||
# unwinds through the smaller fragment-mode <tag name="_frag_X">
|
||||
# definition specifically - the exact same failure through the
|
||||
# larger _full_X definition is fine (clean caught exception, 500).
|
||||
# The client-side click handler already skips fetching these pages
|
||||
# as fragments (FRAGMENT_EXCLUDED, above), but that only protects
|
||||
# normal clicking - a stale already-open tab (running JS from
|
||||
# before that fix shipped) or anyone hitting the URL directly
|
||||
# would still trigger it. This is the real, server-side fix: the
|
||||
# fragment response for these pages never attempts the <mysql>
|
||||
# connection at all, so the crash is unreachable regardless of
|
||||
# how the request arrives. Re-test once a real DB password makes
|
||||
# these pages actually work - a successful connection takes a
|
||||
# different code path that may not need this at all.
|
||||
fragment_page = ('<div id="jetc-content"><p>This page requires a full load - '
|
||||
'<a href="/%s">view it directly</a>.</p></div>') % name
|
||||
else:
|
||||
fragment_page = '<div id="jetc-content">%s</div>' % contents
|
||||
|
||||
# A direct/full-page load and an SPA-driven fragment fetch both hit this
|
||||
# SAME route (/name) - distinguished by ?fragment=1 in the query string,
|
||||
# per the user's explicit choice over a separate /name.page route. A
|
||||
# normal load has no query string at all, so $[fragment] is simply
|
||||
# unset - referencing an unset global variable throws ("global variable
|
||||
# is not initialized.").
|
||||
#
|
||||
# Two real jetc bugs found and worked around while building this (both
|
||||
# confirmed via direct minimal reproduction, not guessed):
|
||||
# 1. <try> safely captures the possibly-unset $[fragment] into
|
||||
# fragflag (always "1" or "" afterward) - but <try> must wrap ONLY
|
||||
# the <set>, never an <if>. Nesting <if>/<else> INSIDE <try> silently
|
||||
# drops the <else> branch's output specifically when compiled
|
||||
# through jetc (works fine in jet-2.0's own interpreter; the bug is
|
||||
# jetc-compiled-only). The branching <if> lives entirely OUTSIDE/
|
||||
# AFTER <try> instead.
|
||||
# 2. A large/complex body (real inlined HTML+CSS+JS, hundreds of KB)
|
||||
# placed directly as an <if>'s branch content fails to COMPILE at
|
||||
# all ("unterminated <html> - no matching close tag found") - jetc's
|
||||
# own compile-time <if>/<else> body scanner isn't robust against
|
||||
# real-world markup at this scale. Worked around by keeping the
|
||||
# <if>'s own branches tiny (self-closing custom-tag invocations)
|
||||
# and moving the real content into <tag name="..."> DEFINITIONS
|
||||
# (which jetc never compile-time-parses at all - stored as opaque
|
||||
# raw text, exactly like a <try> body, and only actually rendered
|
||||
# via <dotag> at runtime).
|
||||
body = (
|
||||
'<set name="jetc_doctype" value="<!DOCTYPE html>" />$[jetc_doctype]'
|
||||
'<html lang="en"><head><title>%s</title>%s</head><body>'
|
||||
'%s%s%s</body></html>'
|
||||
) % (page_title, STYLE_BLOCK, chrome_before, contents, CHROME_AFTER)
|
||||
'<try name="_fragerr"><set name="fragflag" expr="$[fragment]" /><catch><set name="fragflag" value="" /></try>'
|
||||
'<tag name="_frag_%s">%s</tag><tag name="_full_%s">%s</tag>'
|
||||
'<if value1="$[fragflag]" value2="1" type="eq"><_frag_%s /><else><_full_%s /></if>'
|
||||
) % (name, fragment_page, name, full_page, name, name)
|
||||
|
||||
jet_text = "#api GET /%s\n%s\n" % (name, body)
|
||||
|
||||
|
||||
Reference in new issue
Block a user