Add jetc --lint-sql to find <sql> that pastes values (phase 2)

Implements phase 2 of docs/design-sql-bound-parameters.md. jetc --lint-sql
[--strict] <file|dir>... lists each <sql> whose statement contains a $[...]
or #[...] reference instead of a <bind>, with file:line and where each value
comes from. Row values ($[.col], $[session.col], $[?N], $[#]) are only
reported with --strict. References in <bind>, <if>, <set> and the other
attribute-only tags, <comment> and |> <| are skipped. A custom tag defined
as <sql><container></sql> (barant.com's <showtable>) is checked at each use.
Exit status is 0 for none found, 1 for findings, 2 for an unreadable path.

- jetc/JetLint.*: the scanner; jetc.cpp: the option
- build scripts and runbindtests.sh: link JetLint.cpp
- tests/runlinttests.sh and tests/lintsql/: fixtures and expected output
- docs: worklist for the local copies of both sites, design doc status,
  manual note on the lint

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
brad ArantandClaude Opus 5.5 committed 2026-11-20 20:29:28 -08:00
1 parent fd81bb877f
commit 2847ce5fa7
22 files changed
+665 -8

No files matched your search

BIN
View File
Binary file not shown.
+10
View File
@@ -443,6 +443,16 @@ outside an sql tag, a bind with no value, an unknown type, and a
statement whose number of ? placeholders differs from its number of statement whose number of ? placeholders differs from its number of
bind tags. The error text never contains the bound value itself. bind tags. The error text never contains the bound value itself.
To find the sql tags that still write values into the statement text,
run the jet compiler's lint over a site:
\begin{verbatim}
jetc --lint-sql /path/to/site
\end{verbatim}
It prints the file and line of each one, with the variables it uses.
Add \texttt{-{}-strict} to include values from the rows of an earlier query.
\section{call} \section{call}
Use the call tag to evoke an executable file from a jet script. Use the call tag to evoke an executable file from a jet script.
+16 -3
View File
@@ -1,6 +1,6 @@
# Design: bound parameters for `<sql>` # Design: bound parameters for `<sql>`
**Status:** draft for review · **Author:** Brad Arant (drafted with Claude) · **Date:** 2026-10-23 **Status:** phases 1 and 2 implemented (2026-11-20) · **Author:** Brad Arant (drafted with Claude) · **Date:** 2026-10-23
**Applies to:** JetCore (`__sql`, `__mysql`, `Tag.cpp` tag dispatch, `jetc` parser/codegen) **Applies to:** JetCore (`__sql`, `__mysql`, `Tag.cpp` tag dispatch, `jetc` parser/codegen)
## 1. The problem ## 1. The problem
@@ -184,6 +184,19 @@ Once pages are migrated, make the unsafe form an error instead of a habit:
1. **`jetc --lint-sql`** - lists every `<sql>` whose body contains `$[...]` 1. **`jetc --lint-sql`** - lists every `<sql>` whose body contains `$[...]`
other than `$[.column]` (row values from an outer query), with file:line. other than `$[.column]` (row values from an outer query), with file:line.
This is the migration worklist (§8) and a CI check. This is the migration worklist (§8) and a CI check.
As implemented (`jetc/JetLint.cpp`): `jetc --lint-sql [--strict] <file|dir>...`
scans every file under the given paths. It reports `$[...]` and local
`#[...]` references, with the kind of each (`global`, `cgi`, `session`,
`env`, `keyword`, `local`). Row values (`$[.col]`, `$[session.col]`,
`$[?N]`, `$[#]`, `$[?#]`) are not reported unless `--strict` is given.
References in the attributes of `<bind>`, `<if>`, `<else>`, `<ifrow>`,
`<while>`, `<whilerow>`, `<until>`, `<for>` and `<set>` are not reported,
since those never become statement text; nor is anything inside
`<comment>` or `|> <|`. A custom tag defined as `<sql ...><container></sql>`
(barant.com's `<showtable>`) takes its statement from the caller's content,
so each use of it is checked like an `<sql>`. Exit status: 0 nothing found,
1 findings, 2 a path could not be read. Tests: `tests/runlinttests.sh`.
2. **Per-site strict mode** - an APIServer site setting 2. **Per-site strict mode** - an APIServer site setting
`"sqlInterpolation": "warn" | "error"`. In `error` mode `<sql>` refuses to run `"sqlInterpolation": "warn" | "error"`. In `error` mode `<sql>` refuses to run
if its body substituted any `$[...]` except `$[.column]`; `warn` logs it. if its body substituted any `$[...]` except `$[.column]`; `warn` logs it.
@@ -196,8 +209,8 @@ Once pages are migrated, make the unsafe form an error instead of a habit:
| Phase | What | Notes | | Phase | What | Notes |
|---|---|---| |---|---|---|
| 1 | Implement `<bind>`, escaping, utf8mb4 pinning, tests | JetCore only; no page changes | | 1 | Implement `<bind>`, escaping, utf8mb4 pinning, tests | JetCore only; no page changes. Done 2026-11-20 |
| 2 | `jetc --lint-sql`; run on both sites | produces the worklist (≈200 queries) | | 2 | `jetc --lint-sql`; run on both sites | produces the worklist (≈200 queries). Done 2026-11-20 on the local copies: see [sql-lint-worklist.md](sql-lint-worklist.md) |
| 3 | Convert the high-risk pages first | login/signup/password reset, cart/checkout/payment (`pp.jet`), admin procs, ClockPlanet `mobile_api.jet` and auth | | 3 | Convert the high-risk pages first | login/signup/password reset, cart/checkout/payment (`pp.jet`), admin procs, ClockPlanet `mobile_api.jet` and auth |
| 4 | Convert the rest; turn on `warn` | logs show anything missed | | 4 | Convert the rest; turn on `warn` | logs show anything missed |
| 5 | Switch both sites to `error` | unsafe queries can no longer ship | | 5 | Switch both sites to `error` | unsafe queries can no longer ship |
+166
View File
@@ -0,0 +1,166 @@
# `<sql>` migration worklist
Output of `jetc --lint-sql` (phase 2 of [design-sql-bound-parameters.md](design-sql-bound-parameters.md)),
generated 2026-11-20 from the copies of the two sites in `~/Development`.
Each line is a `<sql>` (or a use of a custom tag that wraps one, such as
barant.com's `<showtable>`) whose statement pastes in a variable instead of
using `<bind>`. The kind in brackets is where the value comes from: `global`
includes request parameters, `cgi` is a POST body field, `session` is
`$[@name]`, `local` is `#[name]`.
These local copies are not the whole of either live site. ClockPlanet in
particular has only billing and a few site pages here (no `mobile_api.jet`), so
rerun the lint on the live site trees for the full list:
```
jetc --lint-sql /path/to/site
```
Phase 3 starts with login/signup/password reset, cart/checkout/payment and
admin procs (see the design doc §8).
## barant.com (`JetCore/jetc/barant-site`)
```
./src/SIPPHONE.tag:178: <sql> pastes $[_login_id] (global)
./src/account-info.subpage:3: <sql> pastes $[_login_id] (global)
./src/account-info.subpage:14: <sql> pastes $[_login_id] (global)
./src/account-info.subpage:28: <sql> pastes $[_login_id] (global)
./src/account-info.subpage:54: <sql> pastes $[_login_id] (global)
./src/account-invoices.subpage:3: <sql> pastes $[@_invoice_id] (session)
./src/account-invoices.subpage:14: <showtable> (wraps sql) pastes $[_login_id] (global)
./src/accountconfirm.page:14: <sql> pastes $[uuid] (global)
./src/accountconfirm.page:15: <sql> pastes $[uuid] (global)
./src/accountconfirm.page:16: <sql> pastes $[id] (global)
./src/add_track.page:10: <sql> pastes $[collection_id] (global)
./src/add_track.proc:5: <sql> pastes $[login_id] (global), $[collection_id] (global), $[name] (global), $[artist] (global), $[description] (global), $[price] (global), $[filedata_binary] (global)
./src/addtasknote.page:12: <sql> pastes $[_task_id] (global)
./src/addtasknote.proc:3: <sql> pastes $[_task_id] (global), $[_login_id] (global), $[noteb] (global), $[task_status_id] (global)
./src/addtocart.proc:1: <sql> pastes $[_session_id] (global), $[sku] (global)
./src/addtocart.proc:4: <sql> pastes $[_session_id] (global), $[sku] (global), $[qty] (global)
./src/admin-mailaccounts.subpage:19: <sql> pastes $[@_login_id] (session)
./src/barant.tag:70: <sql> pastes #[table] (local), #[where] (local), #[sortfield] (local)
./src/cart.menu:2: <sql> pastes $[_session_id] (global)
./src/cart.page:17: <sql> pastes $[_session_id] (global)
./src/cart.page:34: <sql> pastes $[_session_id] (global)
./src/cart.page:67: <sql> pastes $[_session_id] (global)
./src/changeprojecticon.proc:3: <sql> pastes $[datax] (global)
./src/changeprojecticon.proc:9: <sql> pastes $[image_id] (global), $[_project_id] (global)
./src/checkout.page:65: <sql> pastes $[_session_id] (global)
./src/checkout.page:75: <sql> pastes $[_session_id] (global)
./src/copy_invoice.jet:14: <sql> pastes $[source_invoice_id] (global)
./src/copy_invoice.jet:19: <sql> pastes $[new_invoice_id] (global), $[source_invoice_id] (global)
./src/create_invoice.jet:14: <sql> pastes $[user_id] (global)
./src/createaccount.proc:17: <sql> pastes $[email:0] (global)
./src/createaccount.proc:28: <sql> pastes $[name] (global), $[company_name] (global), $[email] (global), $[password] (global)
./src/createaccount.proc:46: <sql> pastes $[id] (global), $[link] (global)
./src/createproject.proc:2: <sql> pastes $[_login_id] (global), $[text] (global), $[description] (global)
./src/createreleasetarget.proc:4: <sql> pastes $[_project_id] (global), $[release] (global), $[descriptionx] (global)
./src/createrequest.proc:4: <sql> pastes $[_project_id] (global), $[_login_id] (global), $[priority] (global), $[release_id] (global), $[textx] (global), $[descriptionx] (global)
./src/createtask.proc:5: <sql> pastes $[_request_id] (global), $[_login_id] (global), $[textx] (global), $[descriptionx] (global)
./src/deleterequest.proc:3: <sql> pastes $[_request_id] (global)
./src/documents.jet:2: <sql> pastes $[id] (global), $[@_login_id] (session)
./src/documents_api.jet:17: <sql> pastes $[@_login_id] (session), $[name] (global)
./src/documents_api.jet:29: <sql> pastes $[id] (global), $[@_login_id] (session)
./src/documents_api.jet:31: <sql> pastes $[id] (global), $[@_login_id] (session)
./src/documents_api.jet:43: <sql> pastes $[@_login_id] (session)
./src/documents_delete.proc:4: <sql> pastes $[delete_id] (global), $[@_login_id] (session)
./src/documents_delete.proc:6: <sql> pastes $[delete_id] (global), $[@_login_id] (session)
./src/documents_upload.proc:4: <sql> pastes $[@_login_id] (session), $[name] (global)
./src/editinvoice.page:180: <sql> pastes $[@_invoice_id] (session)
./src/editinvoice.page:234: <sql> pastes $[@_invoice_id] (session)
./src/editproduct.page:8: <sql> pastes $[_product_sku] (global)
./src/editrequest.page:7: <sql> pastes $[_request_id] (global)
./src/edittask.page:12: <sql> pastes $[_task_id] (global)
./src/invoice.method:1: <sql> pastes $[@_invoice_id] (session)
./src/invoice.method:28: <sql> pastes $[@_invoice_id] (session)
./src/invoice.method:51: <sql> pastes $[@_invoice_id] (session)
./src/invoices.menu:2: <sql> pastes $[_user_id] (global)
./src/login.method:1: <sql> pastes $[login_email] (global), $[login_password] (global)
./src/login.method:16: <sql> pastes $[login_id] (global)
./src/login.method:17: <sql> pastes $[login_id] (global)
./src/mailaccounts.jet:25: <sql> pastes $[@_login_id] (session), $[mail_email] (global), $[mail_host] (global), $[mail_password] (global)
./src/mailaccounts.jet:38: <sql> pastes $[account_id] (global), $[@_login_id] (session)
./src/make_invoice.method:5: <sql> pastes $[_session_id] (global)
./src/make_invoice.method:47: <sql> pastes $[_login_id] (global), $[total] (global)
./src/make_invoice.method:62: <sql> pastes $[invoice_id] (global), $[total] (global), $[card_brand] (global), $[last_4_digits] (global), $[exp_month] (global), $[exp_year] (global), $[payment_id] (global), $[tranonce] (global)
./src/make_invoice.method:74: <sql> pastes $[_session_id] (global)
./src/make_invoice.method:81: <sql> pastes $[invoice_id] (global)
./src/make_invoice.method:96: <sql> pastes $[license] (global)
./src/make_invoice.method:97: <sql> pastes $[_login_id] (global), $[invoice_id] (global), $[license] (global), $[key] (global)
./src/make_invoice.method:110: <sql> pastes $[_session_id] (global)
./src/memberhometemplate.temp:25: <sql> pastes $[login_id] (global)
./src/menu.bar:18: <sql> pastes $[_session_id] (global)
./src/musicianhome.page:13: <sql> pastes $[login_id] (global)
./src/musicianhome.page:35: <sql> pastes $[login_id] (global)
./src/myprojects.page:15: <showtable> (wraps sql) pastes $[_login_id] (global)
./src/myworklist.page:10: <showtable> (wraps sql) pastes $[_login_id] (global)
./src/newcollection.proc:11: <sql> pastes $[login_id] (global), $[name] (global), $[artist] (global), $[description] (global), $[price] (global)
./src/payinvoice.page:7: <sql> pastes $[_invoice_id] (global)
./src/phones.section:1: <sql> pastes $[@_login_id] (session)
./src/phones.section:9: <showtable> (wraps sql) pastes $[@_login_id] (session)
./src/processorder.proc:4: <sql> pastes $[_login_id] (global)
./src/processorder.proc:7: <sql> pastes $[_session_id] (global)
./src/processorder.proc:12: <sql> pastes $[invoice_id] (global)
./src/processorder.proc:27: <sql> pastes $[license] (global)
./src/processorder.proc:28: <sql> pastes $[_login_id] (global), $[invoice_id] (global), $[license] (global), $[key] (global)
./src/processorder.proc:41: <sql> pastes $[_session_id] (global)
./src/project.page:16: <sql> pastes $[_project_id] (global)
./src/project.page:37: <showtable> (wraps sql) pastes $[_project_id] (global)
./src/project.tag:3: <sql> pastes $[_project_id] (global)
./src/projects.menu:1: <sql> pastes $[_login_id] (global)
./src/projman.bar:15: <sql> pastes $[_session_id] (global)
./src/registermusician.proc:2: <sql> pastes $[login_id] (global)
./src/removecartitem.proc:1: <sql> pastes $[_session_id] (global), $[sku] (global)
./src/removefromcart.proc:1: <sql> pastes $[_session_id] (global)
./src/request.page:12: <sql> pastes $[_request_id] (global)
./src/request.page:43: <showtable> (wraps sql) pastes $[_request_id] (global)
./src/selectproject.proc:2: <sql> pastes $[project_id] (global)
./src/selecttask.proc:2: <sql> pastes $[task_id] (global)
./src/send_invoice_email.method:5: <sql> pastes $[_send_invoice_id] (global)
./src/task.page:12: <sql> pastes $[_request_id] (global)
./src/task.page:35: <sql> pastes $[_task_id] (global)
./src/taskassignresource.page:15: <sql> pastes $[_task_id] (global)
./src/taskassignresource.proc:5: <sql> pastes $[_task_id] (global), $[user_id] (global), $[estimate_hours] (global)
./src/update_invoice.proc:3: <sql> pastes $[@_invoice_id] (session)
./src/update_invoice.proc:5: <sql> pastes $[@_invoice_id] (session), $[sku$[ix]] (global), $[quantity$[ix]] (global), $[description$[ix]] (global), $[textarea$[ix]] (global), $[price$[ix]] (global), $[total$[ix]] (global), $[tax_amount$[ix]] (global), $[type$[ix]] (global)
./src/update_invoice.proc:8: <sql> pastes $[invoice_total] (global), $[user_id] (global), $[@_invoice_id] (session)
./src/update_product.proc:2: <sql> pastes $[type] (global), $[description] (global), $[text] (global), $[price] (global), $[image] (global), $[_product_sku] (global)
./src/updatecartqty.proc:2: <sql> pastes $[_session_id] (global), $[sku] (global)
./src/updatecartqty.proc:4: <sql> pastes $[qty] (global), $[_session_id] (global), $[sku] (global)
./src/updaterequest.proc:6: <sql> pastes $[textx] (global), $[descriptionx] (global), $[priority] (global), $[owner_id] (global), $[release_id] (global), $[_request_id] (global)
./src/upload_modular1_1_preset.proc:5: <sql> pastes $[title] (global), $[textx] (global), $[preset_datax] (global), $[audio_samplex] (global)
lint-sql: 108 of 139 <sql> blocks (and uses of tags that wrap <sql>) paste values into the statement, in 64 of 76 files (global 174, local 3, session 22)
```
## ClockPlanet (`~/Development/ClockPlanet`)
```
./billing/account.page:9: <sql> pastes $[_acct_bid] (global)
./billing/account.page:46: <sql> pastes $[_acct_bid] (global), $[_acct_req] (global)
./billing/account.page:157: <sql> pastes $[_acct_bid] (global)
./billing/account.page:159: <sql> pastes $[_acct_bid] (global)
./billing/billing_api.jet:28: <sql> pastes $[_mail_key] (global)
./billing/billing_api.jet:50: <sql> pastes $[_mail_key] (global)
./billing/billing_api.jet:53: <sql> pastes $[_notice_business] (global)
./billing/dobilling_pay.proc:16: <sql> pastes $[_bp_bid] (global)
./billing/dobilling_settings.proc:7: <sql> pastes $[_bs_bid] (global), $[_bs_low] (global), $[_bs_rep] (global)
./billing/dobilling_settings.proc:14: <sql> pastes $[_bs_low] (global), $[_bs_rep] (global), $[_bs_bid] (global)
./billing/doemployee_add.proc:6: <sql> pastes $[@_login_business_id] (session), $[name] (global), $[role] (global), $[phone] (global), $[mode] (global), $[invitecode] (global)
./billing/invoice.method:1: <sql> pastes $[_cp_invoice_id] (global)
./billing/invoice.method:33: <sql> pastes $[_cp_invoice_id] (global)
./billing/invoice.method:52: <sql> pastes $[_cp_invoice_id] (global)
./billing/send_invoice_email.method:3: <sql> pastes $[_send_invoice_id] (global)
./site-live/billing-current.txt:10: <sql> pastes $[_acct_bid] (global)
./site-live/billing-current.txt:49: <sql> pastes $[_acct_bid] (global), $[_acct_req] (global)
./site-live/billing-current.txt:114: <sql> pastes $[_acct_bid] (global)
./site-live/billing-current.txt:116: <sql> pastes $[_acct_bid] (global)
./site-live/billing-current.txt:252: <sql> pastes $[_mail_key] (global)
./site-live/billing-current.txt:272: <sql> pastes $[_mail_key] (global)
./site-live/billing-current.txt:275: <sql> pastes $[_notice_business] (global)
./site-live/billing-current.txt:313: <sql> pastes $[@_login_business_id] (session), $[name] (global), $[role] (global), $[phone] (global), $[mode] (global), $[invitecode] (global)
./site-live/sites.page:12: <sql> pastes $[@_login_business_id] (session)
./site-live/sites.page:32: <sql> pastes $[@_login_business_id] (session)
./site-live/sites.page:36: <sql> pastes $[@_login_business_id] (session)
lint-sql: 26 of 28 <sql> blocks paste values into the statement, in 9 of 10 files (global 37, session 5)
```
+322
View File
@@ -0,0 +1,322 @@
#include "JetLint.h"
#include <algorithm>
#include <cctype>
#include <filesystem>
#include <fstream>
#include <map>
#include <sstream>
// See JetLint.h. This is a plain text scan of the page source rather than a
// use of jetc's Parser: the parser keeps no line numbers, compiles some
// content into fallback spans, and stops at the first error, while a lint
// run over a whole site has to report every <sql> in every file.
namespace jetc {
namespace {
bool startsWithNoCase(const std::string &s, size_t pos, const char *word) {
for (size_t i = 0; word[i]; ++i)
if (pos + i >= s.size() || std::tolower((unsigned char)s[pos + i]) != word[i]) return false;
return true;
}
bool isTagEndChar(char c) {
return c == '>' || c == '/' || std::isspace((unsigned char)c);
}
// "<name" at pos, followed by whitespace, '>' or '/'.
bool isOpenTag(const std::string &s, size_t pos, const std::string &name) {
if (s[pos] != '<' || !startsWithNoCase(s, pos + 1, name.c_str())) return false;
size_t after = pos + 1 + name.size();
return after < s.size() && isTagEndChar(s[after]);
}
// "</name" + optional whitespace + ">" at pos. Returns the position
// after the '>', or npos.
size_t closeTagEnd(const std::string &s, size_t pos, const std::string &name) {
if (s.compare(pos, 2, "</") != 0 || !startsWithNoCase(s, pos + 2, name.c_str())) return std::string::npos;
size_t i = pos + 2 + name.size();
while (i < s.size() && std::isspace((unsigned char)s[i])) ++i;
return (i < s.size() && s[i] == '>') ? i + 1 : std::string::npos;
}
// pos is at '<'. Returns the position of the '>' ending the opening
// tag, skipping quoted attribute values, or npos.
size_t openTagEnd(const std::string &s, size_t pos) {
char quote = 0;
for (size_t i = pos + 1; i < s.size(); ++i) {
char c = s[i];
if (quote) { if (c == quote) quote = 0; }
else if (c == '"' || c == '\'') quote = c;
else if (c == '>') return i;
}
return std::string::npos;
}
// pos is at "$[" or "#[". Returns the position after the matching
// ']' (nested references included), or npos.
size_t refEnd(const std::string &s, size_t pos) {
int depth = 0;
for (size_t i = pos; i < s.size(); ++i) {
if ((s[i] == '$' || s[i] == '#') && i + 1 < s.size() && s[i + 1] == '[') { ++depth; ++i; }
else if (s[i] == ']' && --depth == 0) return i + 1;
}
return std::string::npos;
}
// What a reference reads from, from its type designator (see
// Tag::getVariable). "row" is a column of an earlier query.
std::string refKind(const std::string &ref) {
if (ref[0] == '#') return "local";
char d = ref.size() > 2 ? ref[2] : 0;
switch (d) {
case ':': return "cgi";
case '@': return "session";
case '&': return "env";
case '%': return "keyword";
case '!': return "global";
case '.': case '?': case '#': return "row";
}
// $[session.column] - a dot outside any nested reference.
int depth = 0;
for (size_t i = 2; i + 1 < ref.size(); ++i) {
if (ref[i] == '[') ++depth;
else if (ref[i] == ']') --depth;
else if (ref[i] == '.' && depth == 0) return "row";
}
return "global";
}
int lineAt(const std::string &s, size_t pos) {
return 1 + (int)std::count(s.begin(), s.begin() + pos, '\n');
}
// Tags inside a <sql> whose attributes never become statement text:
// <bind> passes its value safely, and the rest only decide what the
// body renders.
const char *const attributeOnlyTags[] = {
"bind", "if", "else", "ifrow", "while", "whilerow", "until", "for", "set",
};
}
void lintSqlText(const std::string &file, const std::string &s, bool strict,
const std::vector<std::string> &wrappers,
std::vector<LintFinding> &findings, LintStats &stats) {
std::vector<std::string> names{"sql"};
names.insert(names.end(), wrappers.begin(), wrappers.end());
bool counted = false;
size_t pos = 0;
while ((pos = s.find('<', pos)) != std::string::npos) {
std::string name;
for (const auto &n : names)
if (isOpenTag(s, pos, n)) { name = n; break; }
if (name.empty()) { ++pos; continue; }
if (!counted) { ++stats.files; counted = true; }
++stats.sqlBlocks;
size_t tagStart = pos;
size_t gt = openTagEnd(s, pos);
if (gt == std::string::npos) break;
pos = gt + 1;
if (s[gt - 1] == '/') continue; // <sql ... /> has no statement
LintFinding finding{file, lineAt(s, tagStart), name, {}};
int depth = 1;
size_t i = pos;
while (i < s.size()) {
size_t end;
if (s[i] == '<') {
if ((end = closeTagEnd(s, i, name)) != std::string::npos) {
i = end;
if (--depth == 0) break;
continue;
}
if (isOpenTag(s, i, name)) {
// A nested block of the same tag is its own block; it is
// reported when the outer scan reaches it.
++depth;
size_t e = openTagEnd(s, i);
i = (e == std::string::npos) ? s.size() : e + 1;
continue;
}
if (isOpenTag(s, i, "comment")) {
size_t c = s.find("</comment>", i);
i = (c == std::string::npos) ? s.size() : c + 10;
continue;
}
bool skipped = false;
for (const char *tag : attributeOnlyTags)
if (isOpenTag(s, i, tag)) {
size_t e = openTagEnd(s, i);
i = (e == std::string::npos) ? s.size() : e + 1;
skipped = true;
break;
}
if (skipped) continue;
} else if (s.compare(i, 2, "|>") == 0) {
// |> ... <| is never parsed, so nothing in it is a reference.
size_t c = s.find("<|", i + 2);
i = (c == std::string::npos) ? s.size() : c + 2;
continue;
} else if ((s[i] == '$' || s[i] == '#') && i + 1 < s.size() && s[i + 1] == '[') {
end = refEnd(s, i);
if (end == std::string::npos) end = s.size();
LintRef ref{s.substr(i, end - i), ""};
ref.kind = refKind(ref.text);
i = end;
if (ref.kind == "row" && !strict) continue;
bool seen = false;
for (auto &r : finding.refs) seen = seen || r.text == ref.text;
if (!seen) finding.refs.push_back(ref);
continue;
}
++i;
}
if (!finding.refs.empty()) {
++stats.flaggedBlocks;
findings.push_back(finding);
}
// Carry on from the end of this <sql>'s opening tag, not its body,
// so a nested <sql> is still counted and checked.
}
}
namespace {
bool fileMentions(const std::string &text, const std::vector<std::string> &names) {
for (size_t p = 0; (p = text.find('<', p)) != std::string::npos; ++p)
for (const auto &n : names)
if (isOpenTag(text, p, n)) return true;
return false;
}
bool readFile(const std::string &path, std::string &text) {
std::ifstream in(path, std::ios::binary);
if (!in) return false;
std::ostringstream ss;
ss << in.rdbuf();
text = ss.str();
return true;
}
}
void findSqlWrappers(const std::string &s, std::vector<std::string> &wrappers) {
for (size_t pos = 0; (pos = s.find('<', pos)) != std::string::npos; ++pos) {
if (!isOpenTag(s, pos, "tag")) continue;
size_t gt = openTagEnd(s, pos);
if (gt == std::string::npos) return;
std::string open = s.substr(pos, gt - pos);
// name="X" (or name='X')
std::string lower = open;
for (auto &c : lower) c = (char)std::tolower((unsigned char)c);
size_t n = lower.find("name=");
if (n == std::string::npos || n + 5 >= open.size()) continue;
char q = open[n + 5];
if (q != '"' && q != '\'') continue;
size_t e = open.find(q, n + 6);
if (e == std::string::npos) continue;
std::string name = open.substr(n + 6, e - n - 6);
// Tag names match without regard to case (isOpenTag lowercases).
for (auto &c : name) c = (char)std::tolower((unsigned char)c);
size_t close = std::string::npos;
for (size_t c = gt; (c = s.find("</", c)) != std::string::npos; ++c)
if (closeTagEnd(s, c, "tag") != std::string::npos) { close = c; break; }
std::string body = s.substr(gt + 1, (close == std::string::npos ? s.size() : close) - gt - 1);
// An <sql> in the definition whose statement is the caller's content.
for (size_t p = 0; (p = body.find('<', p)) != std::string::npos; ++p) {
if (!isOpenTag(body, p, "sql")) continue;
size_t bgt = openTagEnd(body, p);
if (bgt == std::string::npos) break;
size_t end = std::string::npos;
for (size_t c = bgt; (c = body.find("</", c)) != std::string::npos; ++c)
if (closeTagEnd(body, c, "sql") != std::string::npos) { end = c; break; }
std::string stmt = body.substr(bgt + 1, (end == std::string::npos ? body.size() : end) - bgt - 1);
bool hasContainer = false;
for (size_t c = 0; (c = stmt.find('<', c)) != std::string::npos; ++c)
hasContainer = hasContainer || isOpenTag(stmt, c, "container");
if (hasContainer && !name.empty() &&
std::find(wrappers.begin(), wrappers.end(), name) == wrappers.end()) {
wrappers.push_back(name);
break;
}
}
}
}
int lintSql(const std::vector<std::string> &paths, bool strict, std::ostream &out, std::ostream &err) {
namespace fs = std::filesystem;
std::vector<std::string> files;
bool error = false;
for (const auto &p : paths) {
std::error_code ec;
if (fs::is_directory(p, ec)) {
std::vector<std::string> found;
for (auto it = fs::recursive_directory_iterator(p, fs::directory_options::skip_permission_denied, ec);
it != fs::recursive_directory_iterator(); it.increment(ec)) {
if (ec) break;
std::string name = it->path().filename().string();
if (it->is_directory(ec) && !name.empty() && name[0] == '.') { it.disable_recursion_pending(); continue; }
if (it->is_regular_file(ec)) found.push_back(it->path().string());
}
std::sort(found.begin(), found.end());
files.insert(files.end(), found.begin(), found.end());
} else if (fs::is_regular_file(p, ec)) {
files.push_back(p);
} else {
err << "jetc: cannot open " << p << std::endl;
error = true;
}
}
// Two passes: a custom tag defined anywhere in the scanned files as
// <sql ...><container></sql> takes its statement from the caller's
// content, so each use of it is checked like an <sql>.
std::vector<std::string> texts(files.size());
std::vector<bool> readOk(files.size(), false);
std::vector<std::string> wrappers;
for (size_t i = 0; i < files.size(); ++i) {
if (!readFile(files[i], texts[i])) { err << "jetc: cannot read " << files[i] << std::endl; error = true; continue; }
readOk[i] = true;
findSqlWrappers(texts[i], wrappers);
}
std::vector<std::string> names{"sql"};
names.insert(names.end(), wrappers.begin(), wrappers.end());
std::vector<LintFinding> findings;
LintStats stats;
for (size_t i = 0; i < files.size(); ++i) {
if (!readOk[i] || !fileMentions(texts[i], names)) continue;
lintSqlText(files[i], texts[i], strict, wrappers, findings, stats);
}
std::map<std::string, int> byKind;
std::map<std::string, bool> flaggedFiles;
for (const auto &f : findings) {
flaggedFiles[f.file] = true;
out << f.file << ":" << f.line << ": <" << f.tag << ">"
<< (f.tag == "sql" ? "" : " (wraps sql)") << " pastes ";
for (size_t i = 0; i < f.refs.size(); ++i) {
out << (i ? ", " : "") << f.refs[i].text << " (" << f.refs[i].kind << ")";
++byKind[f.refs[i].kind];
}
out << "\n";
}
out << "lint-sql: " << stats.flaggedBlocks << " of " << stats.sqlBlocks
<< " <sql> blocks" << (wrappers.empty() ? "" : " (and uses of tags that wrap <sql>)")
<< " paste values into the statement, in " << flaggedFiles.size()
<< " of " << stats.files << " files";
if (!byKind.empty()) {
out << " (";
bool first = true;
for (auto &[kind, n] : byKind) { out << (first ? "" : ", ") << kind << " " << n; first = false; }
out << ")";
}
out << std::endl;
if (error) return 2;
return findings.empty() ? 0 : 1;
}
}
+57
View File
@@ -0,0 +1,57 @@
#ifndef __JetLint_h__
#define __JetLint_h__
#include <string>
#include <vector>
#include <ostream>
// jetc --lint-sql: phase 2 of docs/design-sql-bound-parameters.md.
//
// Lists every <sql> whose body pastes a variable into the statement text
// ($[...] or #[...]) instead of passing it with <bind>. These are the
// queries to migrate. A value from a row of an earlier query
// ($[.column], $[session.column], $[?N], $[#]) is not reported, as the
// design doc specifies, unless strict is set.
namespace jetc {
struct LintRef {
std::string text; // the reference as written, e.g. "$[:email]"
std::string kind; // cgi, global, local, session, env, keyword, row, ...
};
struct LintFinding {
std::string file;
int line; // line of the <sql tag
std::string tag; // "sql", or a custom tag that wraps <sql><container></sql>
std::vector<LintRef> refs;
};
struct LintStats {
int files = 0; // files that contain at least one <sql>
int sqlBlocks = 0;
int flaggedBlocks = 0;
};
// Scans one file's text. Appends to findings and stats.
// wrappers: custom tag names whose uses count as <sql> (see
// findSqlWrappers).
void lintSqlText(const std::string &file, const std::string &text, bool strict,
const std::vector<std::string> &wrappers,
std::vector<LintFinding> &findings, LintStats &stats);
// Adds to wrappers each custom tag defined in text (<tag name="X">)
// whose definition runs the caller's content as its statement:
// <sql ...><container></sql>.
void findSqlWrappers(const std::string &text, std::vector<std::string> &wrappers);
// Scans files and directories (recursively; every regular file,
// whatever its extension). Tag definitions are collected from all of
// them first, so a wrapper defined in one file is found in all.
// Prints one line per finding and a summary. Returns 0 when nothing is found, 1 when something is,
// 2 on an error (unreadable path).
int lintSql(const std::vector<std::string> &paths, bool strict, std::ostream &out, std::ostream &err);
}
#endif
+1 -1
View File
@@ -21,7 +21,7 @@ OUTNAME="$2"
if [ ! -x "$HERE/jetc" ]; then if [ ! -x "$HERE/jetc" ]; then
echo "Building jetc..." echo "Building jetc..."
g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" "$HERE/JetLint.cpp"
fi fi
echo "Compiling $INPUT -> $OUTNAME.compiled.cpp" echo "Compiling $INPUT -> $OUTNAME.compiled.cpp"
+1 -1
View File
@@ -30,7 +30,7 @@ STEM="$(basename "$OUTPUT" .jtc)"
if [ ! -x "$HERE/jetc" ]; then if [ ! -x "$HERE/jetc" ]; then
echo "Building jetc..." echo "Building jetc..."
g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" "$HERE/JetLint.cpp"
fi fi
echo "Compiling $INPUT -> $STEM.compiled.cpp (shared mode)" echo "Compiling $INPUT -> $STEM.compiled.cpp (shared mode)"
+13 -2
View File
@@ -1,5 +1,6 @@
#include "JetParser.h" #include "JetParser.h"
#include "JetCodegen.h" #include "JetCodegen.h"
#include "JetLint.h"
#include <fstream> #include <fstream>
#include <sstream> #include <sstream>
#include <iostream> #include <iostream>
@@ -26,17 +27,27 @@ namespace {
int main(int argc, char **argv) { int main(int argc, char **argv) {
bool shared = false; bool shared = false;
bool lintSql = false;
bool strict = false;
std::vector<std::string> pos; std::vector<std::string> pos;
for (int i = 1; i < argc; ++i) { for (int i = 1; i < argc; ++i) {
std::string a = argv[i]; std::string a = argv[i];
if (a == "--shared") shared = true; if (a == "--shared") shared = true;
else if (a == "--lint-sql") lintSql = true;
else if (a == "--strict") strict = true;
else pos.push_back(a); else pos.push_back(a);
} }
if (pos.size() < 2) { if (lintSql && !pos.empty())
return jetc::lintSql(pos, strict, std::cout, std::cerr);
if (pos.size() < 2 || lintSql || strict) {
std::cerr << "usage: jetc [--shared] <input.jet> <output.cpp>" << std::endl; std::cerr << "usage: jetc [--shared] <input.jet> <output.cpp>" << std::endl;
std::cerr << " jetc --lint-sql [--strict] <file|directory>..." << std::endl;
std::cerr << " --shared emit a dlsym()-able entry point (see build-shared.sh)" << std::endl; std::cerr << " --shared emit a dlsym()-able entry point (see build-shared.sh)" << std::endl;
std::cerr << " instead of a standalone main()." << std::endl; std::cerr << " instead of a standalone main()." << std::endl;
return 1; std::cerr << " --lint-sql list every <sql> that pastes a variable into the statement" << std::endl;
std::cerr << " instead of using <bind> (exit 1 if any, 0 if none)." << std::endl;
std::cerr << " --strict with --lint-sql, also report row values ($[.column] etc.)." << std::endl;
return lintSql ? 2 : 1;
} }
const std::string &inputPath = pos[0]; const std::string &inputPath = pos[0];
const std::string &outputPath = pos[1]; const std::string &outputPath = pos[1];
+1
View File
@@ -0,0 +1 @@
lint-sql: 0 of 1 <sql> blocks paste values into the statement, in 0 of 1 files
+1
View File
@@ -0,0 +1 @@
<sql>select * from t where id=?<bind value="$[id]" type="int"></sql>
+2
View File
@@ -0,0 +1,2 @@
jetc: cannot open no-such-dir
lint-sql: 0 of 1 <sql> blocks paste values into the statement, in 0 of 1 files
+6
View File
@@ -0,0 +1,6 @@
site/kinds.page:1: <sql> pastes $[plain] (global), $[:form] (cgi), $[@sess] (session)
site/kinds.page:2: <sql> pastes $[&HTTP_HOST] (env), $[%kw] (keyword), $[!g] (global), #[loc] (local)
site/kinds.page:3: <sql> pastes $[late] (global)
site/kinds.page:6: <sql> pastes $[name$[ix]] (global)
site/uses.page:1: <showtable> (wraps sql) pastes $[_login_id] (global)
lint-sql: 5 of 11 <sql> blocks (and uses of tags that wrap <sql>) paste values into the statement, in 2 of 4 files (cgi 1, env 1, global 5, keyword 1, local 1, session 1)
+1
View File
@@ -0,0 +1 @@
<sql>select $[hidden]</sql>
+6
View File
@@ -0,0 +1,6 @@
<sql>select * from t where a='$[plain]' and b=$[:form] and c='$[@sess]'</sql>
<sql>select '$[&HTTP_HOST]', '$[%kw]', '$[!g]', '#[loc]'</sql>
<SQL sessionid="#[sid]">select 1
from t
where x=$[late] and y=$[late]</SQL>
<sql>select * from a where k=$[name$[ix]]</sql>
+1
View File
@@ -0,0 +1 @@
no queries here $[x]
+11
View File
@@ -0,0 +1,11 @@
<sql>select $[.col], $[projman.col], $[?1], $[#], $[?#] from t</sql>
<sql>select * from t where a=? and b=?
<bind value="$[email]">
<if value1="$[x]" value2="1"><bind value="$[y]" type="int"></if>
<set name="z" value="$[z]">
<comment> uses $[nothing] </comment>
</sql>
<sql>select '|>$[not_parsed]<|'</sql>
<sql sessionid="x"/>
<sqlx>$[not_sql]</sqlx>
$[outside_sql]
+4
View File
@@ -0,0 +1,4 @@
<tag name="showTable" keywords="sessionid">
<sql sessionid="#[sessionid]"><container></sql>
</tag>
<tag name="plain"><b><container></b></tag>
+3
View File
@@ -0,0 +1,3 @@
<showtable sessionid="1">select * from t where u=$[_login_id]</showtable>
<showtable>select * from t where u=$[.id]</showtable>
<plain>$[not_sql]</plain>
+8
View File
@@ -0,0 +1,8 @@
site/kinds.page:1: <sql> pastes $[plain] (global), $[:form] (cgi), $[@sess] (session)
site/kinds.page:2: <sql> pastes $[&HTTP_HOST] (env), $[%kw] (keyword), $[!g] (global), #[loc] (local)
site/kinds.page:3: <sql> pastes $[late] (global)
site/kinds.page:6: <sql> pastes $[name$[ix]] (global)
site/safe.page:1: <sql> pastes $[.col] (row), $[projman.col] (row), $[?1] (row), $[#] (row), $[?#] (row)
site/uses.page:1: <showtable> (wraps sql) pastes $[_login_id] (global)
site/uses.page:2: <showtable> (wraps sql) pastes $[.id] (row)
lint-sql: 7 of 11 <sql> blocks (and uses of tags that wrap <sql>) paste values into the statement, in 3 of 4 files (cgi 1, env 1, global 5, keyword 1, local 1, row 6, session 1)
+1 -1
View File
@@ -36,7 +36,7 @@ else
fi fi
echo "== jetc ==" echo "== jetc =="
g++ -g -std=c++23 -o "$WORK/jetc" "$JC/jetc/jetc.cpp" "$JC/jetc/JetParser.cpp" "$JC/jetc/JetCodegen.cpp" || exit 1 g++ -g -std=c++23 -o "$WORK/jetc" "$JC/jetc/jetc.cpp" "$JC/jetc/JetParser.cpp" "$JC/jetc/JetCodegen.cpp" "$JC/jetc/JetLint.cpp" || exit 1
# A bare <bind> must not swallow the content after it looking for </bind>. # A bare <bind> must not swallow the content after it looking for </bind>.
printf 'A<sql>select ?<bind value="$[x]"></sql>B<bind value="y">C\n' > "$WORK/void.jet" printf 'A<sql>select ?<bind value="$[x]"></sql>B<bind value="y">C\n' > "$WORK/void.jet"
"$WORK/jetc" "$WORK/void.jet" "$WORK/void.cpp" >/dev/null || fail=1 "$WORK/jetc" "$WORK/void.jet" "$WORK/void.cpp" >/dev/null || fail=1
+34
View File
@@ -0,0 +1,34 @@
#!/bin/bash
# Tests for jetc --lint-sql (docs/design-sql-bound-parameters.md, phase 2).
# Runs the lint over tests/lintsql/ and compares with the .expected files.
# Needs no database.
set -u
T="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
JC="$(cd "$T/.." && pwd)"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
fail=0
g++ -g -std=c++23 -o "$WORK/jetc" "$JC/jetc/jetc.cpp" "$JC/jetc/JetParser.cpp" "$JC/jetc/JetCodegen.cpp" "$JC/jetc/JetLint.cpp" || exit 1
# check NAME EXPECTED_EXIT ARGS... - output is compared with lintsql/NAME.expected
check() {
local name="$1" want="$2"; shift 2
(cd "$T/lintsql" && "$WORK/jetc" --lint-sql "$@") > "$WORK/$name.out" 2>&1
local got=$?
if [ "$got" != "$want" ]; then
echo "FAIL $name: exit $got, expected $want"; fail=1
elif ! diff -u "$T/lintsql/$name.expected" "$WORK/$name.out"; then
echo "FAIL $name: output differs"; fail=1
else
echo "PASS $name"
fi
}
check site 1 site
check strict 1 --strict site
check clean 0 clean
check missing 2 clean no-such-dir
[ $fail = 0 ] && echo "ALL PASSED" || echo "FAILURES"
exit $fail