diff --git a/docs/JetCore.pdf b/docs/JetCore.pdf index b7498ed..aae1827 100644 Binary files a/docs/JetCore.pdf and b/docs/JetCore.pdf differ diff --git a/docs/JetCore.tex b/docs/JetCore.tex index 54eb713..257bba4 100644 --- a/docs/JetCore.tex +++ b/docs/JetCore.tex @@ -443,6 +443,16 @@ outside an sql tag, a bind with no value, an unknown type, and a statement whose number of ? placeholders differs from its number of bind tags. The error text never contains the bound value itself. +To find the sql tags that still write values into the statement text, +run the jet compiler's lint over a site: + +\begin{verbatim} +jetc --lint-sql /path/to/site +\end{verbatim} + +It prints the file and line of each one, with the variables it uses. +Add \texttt{-{}-strict} to include values from the rows of an earlier query. + \section{call} Use the call tag to evoke an executable file from a jet script. diff --git a/docs/design-sql-bound-parameters.md b/docs/design-sql-bound-parameters.md index 1f5ddfb..3d0b940 100644 --- a/docs/design-sql-bound-parameters.md +++ b/docs/design-sql-bound-parameters.md @@ -1,6 +1,6 @@ # Design: bound parameters for `` -**Status:** draft for review · **Author:** Brad Arant (drafted with Claude) · **Date:** 2026-10-23 +**Status:** phases 1 and 2 implemented (2026-11-20) · **Author:** Brad Arant (drafted with Claude) · **Date:** 2026-10-23 **Applies to:** JetCore (`__sql`, `__mysql`, `Tag.cpp` tag dispatch, `jetc` parser/codegen) ## 1. The problem @@ -184,6 +184,19 @@ Once pages are migrated, make the unsafe form an error instead of a habit: 1. **`jetc --lint-sql`** - lists every `` whose body contains `$[...]` other than `$[.column]` (row values from an outer query), with file:line. This is the migration worklist (§8) and a CI check. + + As implemented (`jetc/JetLint.cpp`): `jetc --lint-sql [--strict] ...` + scans every file under the given paths. It reports `$[...]` and local + `#[...]` references, with the kind of each (`global`, `cgi`, `session`, + `env`, `keyword`, `local`). Row values (`$[.col]`, `$[session.col]`, + `$[?N]`, `$[#]`, `$[?#]`) are not reported unless `--strict` is given. + References in the attributes of ``, ``, ``, ``, + ``, ``, ``, `` and `` are not reported, + since those never become statement text; nor is anything inside + `` or `|> <|`. A custom tag defined as `` + (barant.com's ``) takes its statement from the caller's content, + so each use of it is checked like an ``. Exit status: 0 nothing found, + 1 findings, 2 a path could not be read. Tests: `tests/runlinttests.sh`. 2. **Per-site strict mode** - an APIServer site setting `"sqlInterpolation": "warn" | "error"`. In `error` mode `` refuses to run if its body substituted any `$[...]` except `$[.column]`; `warn` logs it. @@ -196,8 +209,8 @@ Once pages are migrated, make the unsafe form an error instead of a habit: | Phase | What | Notes | |---|---|---| -| 1 | Implement ``, escaping, utf8mb4 pinning, tests | JetCore only; no page changes | -| 2 | `jetc --lint-sql`; run on both sites | produces the worklist (≈200 queries) | +| 1 | Implement ``, escaping, utf8mb4 pinning, tests | JetCore only; no page changes. Done 2026-11-20 | +| 2 | `jetc --lint-sql`; run on both sites | produces the worklist (≈200 queries). Done 2026-11-20 on the local copies: see [sql-lint-worklist.md](sql-lint-worklist.md) | | 3 | Convert the high-risk pages first | login/signup/password reset, cart/checkout/payment (`pp.jet`), admin procs, ClockPlanet `mobile_api.jet` and auth | | 4 | Convert the rest; turn on `warn` | logs show anything missed | | 5 | Switch both sites to `error` | unsafe queries can no longer ship | diff --git a/docs/sql-lint-worklist.md b/docs/sql-lint-worklist.md new file mode 100644 index 0000000..f160e99 --- /dev/null +++ b/docs/sql-lint-worklist.md @@ -0,0 +1,166 @@ +# `` migration worklist + +Output of `jetc --lint-sql` (phase 2 of [design-sql-bound-parameters.md](design-sql-bound-parameters.md)), +generated 2026-11-20 from the copies of the two sites in `~/Development`. +Each line is a `` (or a use of a custom tag that wraps one, such as +barant.com's ``) whose statement pastes in a variable instead of +using ``. The kind in brackets is where the value comes from: `global` +includes request parameters, `cgi` is a POST body field, `session` is +`$[@name]`, `local` is `#[name]`. + +These local copies are not the whole of either live site. ClockPlanet in +particular has only billing and a few site pages here (no `mobile_api.jet`), so +rerun the lint on the live site trees for the full list: + +``` +jetc --lint-sql /path/to/site +``` + +Phase 3 starts with login/signup/password reset, cart/checkout/payment and +admin procs (see the design doc §8). + +## barant.com (`JetCore/jetc/barant-site`) + +``` +./src/SIPPHONE.tag:178: pastes $[_login_id] (global) +./src/account-info.subpage:3: pastes $[_login_id] (global) +./src/account-info.subpage:14: pastes $[_login_id] (global) +./src/account-info.subpage:28: pastes $[_login_id] (global) +./src/account-info.subpage:54: pastes $[_login_id] (global) +./src/account-invoices.subpage:3: pastes $[@_invoice_id] (session) +./src/account-invoices.subpage:14: (wraps sql) pastes $[_login_id] (global) +./src/accountconfirm.page:14: pastes $[uuid] (global) +./src/accountconfirm.page:15: pastes $[uuid] (global) +./src/accountconfirm.page:16: pastes $[id] (global) +./src/add_track.page:10: pastes $[collection_id] (global) +./src/add_track.proc:5: pastes $[login_id] (global), $[collection_id] (global), $[name] (global), $[artist] (global), $[description] (global), $[price] (global), $[filedata_binary] (global) +./src/addtasknote.page:12: pastes $[_task_id] (global) +./src/addtasknote.proc:3: pastes $[_task_id] (global), $[_login_id] (global), $[noteb] (global), $[task_status_id] (global) +./src/addtocart.proc:1: pastes $[_session_id] (global), $[sku] (global) +./src/addtocart.proc:4: pastes $[_session_id] (global), $[sku] (global), $[qty] (global) +./src/admin-mailaccounts.subpage:19: pastes $[@_login_id] (session) +./src/barant.tag:70: pastes #[table] (local), #[where] (local), #[sortfield] (local) +./src/cart.menu:2: pastes $[_session_id] (global) +./src/cart.page:17: pastes $[_session_id] (global) +./src/cart.page:34: pastes $[_session_id] (global) +./src/cart.page:67: pastes $[_session_id] (global) +./src/changeprojecticon.proc:3: pastes $[datax] (global) +./src/changeprojecticon.proc:9: pastes $[image_id] (global), $[_project_id] (global) +./src/checkout.page:65: pastes $[_session_id] (global) +./src/checkout.page:75: pastes $[_session_id] (global) +./src/copy_invoice.jet:14: pastes $[source_invoice_id] (global) +./src/copy_invoice.jet:19: pastes $[new_invoice_id] (global), $[source_invoice_id] (global) +./src/create_invoice.jet:14: pastes $[user_id] (global) +./src/createaccount.proc:17: pastes $[email:0] (global) +./src/createaccount.proc:28: pastes $[name] (global), $[company_name] (global), $[email] (global), $[password] (global) +./src/createaccount.proc:46: pastes $[id] (global), $[link] (global) +./src/createproject.proc:2: pastes $[_login_id] (global), $[text] (global), $[description] (global) +./src/createreleasetarget.proc:4: pastes $[_project_id] (global), $[release] (global), $[descriptionx] (global) +./src/createrequest.proc:4: pastes $[_project_id] (global), $[_login_id] (global), $[priority] (global), $[release_id] (global), $[textx] (global), $[descriptionx] (global) +./src/createtask.proc:5: pastes $[_request_id] (global), $[_login_id] (global), $[textx] (global), $[descriptionx] (global) +./src/deleterequest.proc:3: pastes $[_request_id] (global) +./src/documents.jet:2: pastes $[id] (global), $[@_login_id] (session) +./src/documents_api.jet:17: pastes $[@_login_id] (session), $[name] (global) +./src/documents_api.jet:29: pastes $[id] (global), $[@_login_id] (session) +./src/documents_api.jet:31: pastes $[id] (global), $[@_login_id] (session) +./src/documents_api.jet:43: pastes $[@_login_id] (session) +./src/documents_delete.proc:4: pastes $[delete_id] (global), $[@_login_id] (session) +./src/documents_delete.proc:6: pastes $[delete_id] (global), $[@_login_id] (session) +./src/documents_upload.proc:4: pastes $[@_login_id] (session), $[name] (global) +./src/editinvoice.page:180: pastes $[@_invoice_id] (session) +./src/editinvoice.page:234: pastes $[@_invoice_id] (session) +./src/editproduct.page:8: pastes $[_product_sku] (global) +./src/editrequest.page:7: pastes $[_request_id] (global) +./src/edittask.page:12: pastes $[_task_id] (global) +./src/invoice.method:1: pastes $[@_invoice_id] (session) +./src/invoice.method:28: pastes $[@_invoice_id] (session) +./src/invoice.method:51: pastes $[@_invoice_id] (session) +./src/invoices.menu:2: pastes $[_user_id] (global) +./src/login.method:1: pastes $[login_email] (global), $[login_password] (global) +./src/login.method:16: pastes $[login_id] (global) +./src/login.method:17: pastes $[login_id] (global) +./src/mailaccounts.jet:25: pastes $[@_login_id] (session), $[mail_email] (global), $[mail_host] (global), $[mail_password] (global) +./src/mailaccounts.jet:38: pastes $[account_id] (global), $[@_login_id] (session) +./src/make_invoice.method:5: pastes $[_session_id] (global) +./src/make_invoice.method:47: pastes $[_login_id] (global), $[total] (global) +./src/make_invoice.method:62: pastes $[invoice_id] (global), $[total] (global), $[card_brand] (global), $[last_4_digits] (global), $[exp_month] (global), $[exp_year] (global), $[payment_id] (global), $[tranonce] (global) +./src/make_invoice.method:74: pastes $[_session_id] (global) +./src/make_invoice.method:81: pastes $[invoice_id] (global) +./src/make_invoice.method:96: pastes $[license] (global) +./src/make_invoice.method:97: pastes $[_login_id] (global), $[invoice_id] (global), $[license] (global), $[key] (global) +./src/make_invoice.method:110: pastes $[_session_id] (global) +./src/memberhometemplate.temp:25: pastes $[login_id] (global) +./src/menu.bar:18: pastes $[_session_id] (global) +./src/musicianhome.page:13: pastes $[login_id] (global) +./src/musicianhome.page:35: pastes $[login_id] (global) +./src/myprojects.page:15: (wraps sql) pastes $[_login_id] (global) +./src/myworklist.page:10: (wraps sql) pastes $[_login_id] (global) +./src/newcollection.proc:11: pastes $[login_id] (global), $[name] (global), $[artist] (global), $[description] (global), $[price] (global) +./src/payinvoice.page:7: pastes $[_invoice_id] (global) +./src/phones.section:1: pastes $[@_login_id] (session) +./src/phones.section:9: (wraps sql) pastes $[@_login_id] (session) +./src/processorder.proc:4: pastes $[_login_id] (global) +./src/processorder.proc:7: pastes $[_session_id] (global) +./src/processorder.proc:12: pastes $[invoice_id] (global) +./src/processorder.proc:27: pastes $[license] (global) +./src/processorder.proc:28: pastes $[_login_id] (global), $[invoice_id] (global), $[license] (global), $[key] (global) +./src/processorder.proc:41: pastes $[_session_id] (global) +./src/project.page:16: pastes $[_project_id] (global) +./src/project.page:37: (wraps sql) pastes $[_project_id] (global) +./src/project.tag:3: pastes $[_project_id] (global) +./src/projects.menu:1: pastes $[_login_id] (global) +./src/projman.bar:15: pastes $[_session_id] (global) +./src/registermusician.proc:2: pastes $[login_id] (global) +./src/removecartitem.proc:1: pastes $[_session_id] (global), $[sku] (global) +./src/removefromcart.proc:1: pastes $[_session_id] (global) +./src/request.page:12: pastes $[_request_id] (global) +./src/request.page:43: (wraps sql) pastes $[_request_id] (global) +./src/selectproject.proc:2: pastes $[project_id] (global) +./src/selecttask.proc:2: pastes $[task_id] (global) +./src/send_invoice_email.method:5: pastes $[_send_invoice_id] (global) +./src/task.page:12: pastes $[_request_id] (global) +./src/task.page:35: pastes $[_task_id] (global) +./src/taskassignresource.page:15: pastes $[_task_id] (global) +./src/taskassignresource.proc:5: pastes $[_task_id] (global), $[user_id] (global), $[estimate_hours] (global) +./src/update_invoice.proc:3: pastes $[@_invoice_id] (session) +./src/update_invoice.proc:5: pastes $[@_invoice_id] (session), $[sku$[ix]] (global), $[quantity$[ix]] (global), $[description$[ix]] (global), $[textarea$[ix]] (global), $[price$[ix]] (global), $[total$[ix]] (global), $[tax_amount$[ix]] (global), $[type$[ix]] (global) +./src/update_invoice.proc:8: pastes $[invoice_total] (global), $[user_id] (global), $[@_invoice_id] (session) +./src/update_product.proc:2: pastes $[type] (global), $[description] (global), $[text] (global), $[price] (global), $[image] (global), $[_product_sku] (global) +./src/updatecartqty.proc:2: pastes $[_session_id] (global), $[sku] (global) +./src/updatecartqty.proc:4: pastes $[qty] (global), $[_session_id] (global), $[sku] (global) +./src/updaterequest.proc:6: pastes $[textx] (global), $[descriptionx] (global), $[priority] (global), $[owner_id] (global), $[release_id] (global), $[_request_id] (global) +./src/upload_modular1_1_preset.proc:5: pastes $[title] (global), $[textx] (global), $[preset_datax] (global), $[audio_samplex] (global) +lint-sql: 108 of 139 blocks (and uses of tags that wrap ) paste values into the statement, in 64 of 76 files (global 174, local 3, session 22) +``` + +## ClockPlanet (`~/Development/ClockPlanet`) + +``` +./billing/account.page:9: pastes $[_acct_bid] (global) +./billing/account.page:46: pastes $[_acct_bid] (global), $[_acct_req] (global) +./billing/account.page:157: pastes $[_acct_bid] (global) +./billing/account.page:159: pastes $[_acct_bid] (global) +./billing/billing_api.jet:28: pastes $[_mail_key] (global) +./billing/billing_api.jet:50: pastes $[_mail_key] (global) +./billing/billing_api.jet:53: pastes $[_notice_business] (global) +./billing/dobilling_pay.proc:16: pastes $[_bp_bid] (global) +./billing/dobilling_settings.proc:7: pastes $[_bs_bid] (global), $[_bs_low] (global), $[_bs_rep] (global) +./billing/dobilling_settings.proc:14: pastes $[_bs_low] (global), $[_bs_rep] (global), $[_bs_bid] (global) +./billing/doemployee_add.proc:6: pastes $[@_login_business_id] (session), $[name] (global), $[role] (global), $[phone] (global), $[mode] (global), $[invitecode] (global) +./billing/invoice.method:1: pastes $[_cp_invoice_id] (global) +./billing/invoice.method:33: pastes $[_cp_invoice_id] (global) +./billing/invoice.method:52: pastes $[_cp_invoice_id] (global) +./billing/send_invoice_email.method:3: pastes $[_send_invoice_id] (global) +./site-live/billing-current.txt:10: pastes $[_acct_bid] (global) +./site-live/billing-current.txt:49: pastes $[_acct_bid] (global), $[_acct_req] (global) +./site-live/billing-current.txt:114: pastes $[_acct_bid] (global) +./site-live/billing-current.txt:116: pastes $[_acct_bid] (global) +./site-live/billing-current.txt:252: pastes $[_mail_key] (global) +./site-live/billing-current.txt:272: pastes $[_mail_key] (global) +./site-live/billing-current.txt:275: pastes $[_notice_business] (global) +./site-live/billing-current.txt:313: pastes $[@_login_business_id] (session), $[name] (global), $[role] (global), $[phone] (global), $[mode] (global), $[invitecode] (global) +./site-live/sites.page:12: pastes $[@_login_business_id] (session) +./site-live/sites.page:32: pastes $[@_login_business_id] (session) +./site-live/sites.page:36: pastes $[@_login_business_id] (session) +lint-sql: 26 of 28 blocks paste values into the statement, in 9 of 10 files (global 37, session 5) +``` diff --git a/jetc/JetLint.cpp b/jetc/JetLint.cpp new file mode 100644 index 0000000..5dae86c --- /dev/null +++ b/jetc/JetLint.cpp @@ -0,0 +1,322 @@ +#include "JetLint.h" +#include +#include +#include +#include +#include +#include + +// See JetLint.h. This is a plain text scan of the page source rather than a +// use of jetc's Parser: the parser keeps no line numbers, compiles some +// content into fallback spans, and stops at the first error, while a lint +// run over a whole site has to report every in every file. + +namespace jetc { + + namespace { + + bool startsWithNoCase(const std::string &s, size_t pos, const char *word) { + for (size_t i = 0; word[i]; ++i) + if (pos + i >= s.size() || std::tolower((unsigned char)s[pos + i]) != word[i]) return false; + return true; + } + + bool isTagEndChar(char c) { + return c == '>' || c == '/' || std::isspace((unsigned char)c); + } + + // "' or '/'. + bool isOpenTag(const std::string &s, size_t pos, const std::string &name) { + if (s[pos] != '<' || !startsWithNoCase(s, pos + 1, name.c_str())) return false; + size_t after = pos + 1 + name.size(); + return after < s.size() && isTagEndChar(s[after]); + } + + // "" at pos. Returns the position + // after the '>', or npos. + size_t closeTagEnd(const std::string &s, size_t pos, const std::string &name) { + if (s.compare(pos, 2, "') ? i + 1 : std::string::npos; + } + + // pos is at '<'. Returns the position of the '>' ending the opening + // tag, skipping quoted attribute values, or npos. + size_t openTagEnd(const std::string &s, size_t pos) { + char quote = 0; + for (size_t i = pos + 1; i < s.size(); ++i) { + char c = s[i]; + if (quote) { if (c == quote) quote = 0; } + else if (c == '"' || c == '\'') quote = c; + else if (c == '>') return i; + } + return std::string::npos; + } + + // pos is at "$[" or "#[". Returns the position after the matching + // ']' (nested references included), or npos. + size_t refEnd(const std::string &s, size_t pos) { + int depth = 0; + for (size_t i = pos; i < s.size(); ++i) { + if ((s[i] == '$' || s[i] == '#') && i + 1 < s.size() && s[i + 1] == '[') { ++depth; ++i; } + else if (s[i] == ']' && --depth == 0) return i + 1; + } + return std::string::npos; + } + + // What a reference reads from, from its type designator (see + // Tag::getVariable). "row" is a column of an earlier query. + std::string refKind(const std::string &ref) { + if (ref[0] == '#') return "local"; + char d = ref.size() > 2 ? ref[2] : 0; + switch (d) { + case ':': return "cgi"; + case '@': return "session"; + case '&': return "env"; + case '%': return "keyword"; + case '!': return "global"; + case '.': case '?': case '#': return "row"; + } + // $[session.column] - a dot outside any nested reference. + int depth = 0; + for (size_t i = 2; i + 1 < ref.size(); ++i) { + if (ref[i] == '[') ++depth; + else if (ref[i] == ']') --depth; + else if (ref[i] == '.' && depth == 0) return "row"; + } + return "global"; + } + + int lineAt(const std::string &s, size_t pos) { + return 1 + (int)std::count(s.begin(), s.begin() + pos, '\n'); + } + + // Tags inside a whose attributes never become statement text: + // passes its value safely, and the rest only decide what the + // body renders. + const char *const attributeOnlyTags[] = { + "bind", "if", "else", "ifrow", "while", "whilerow", "until", "for", "set", + }; + + } + + void lintSqlText(const std::string &file, const std::string &s, bool strict, + const std::vector &wrappers, + std::vector &findings, LintStats &stats) { + std::vector names{"sql"}; + names.insert(names.end(), wrappers.begin(), wrappers.end()); + bool counted = false; + size_t pos = 0; + while ((pos = s.find('<', pos)) != std::string::npos) { + std::string name; + for (const auto &n : names) + if (isOpenTag(s, pos, n)) { name = n; break; } + if (name.empty()) { ++pos; continue; } + if (!counted) { ++stats.files; counted = true; } + ++stats.sqlBlocks; + size_t tagStart = pos; + size_t gt = openTagEnd(s, pos); + if (gt == std::string::npos) break; + pos = gt + 1; + if (s[gt - 1] == '/') continue; // has no statement + + LintFinding finding{file, lineAt(s, tagStart), name, {}}; + int depth = 1; + size_t i = pos; + while (i < s.size()) { + size_t end; + if (s[i] == '<') { + if ((end = closeTagEnd(s, i, name)) != std::string::npos) { + i = end; + if (--depth == 0) break; + continue; + } + if (isOpenTag(s, i, name)) { + // A nested block of the same tag is its own block; it is + // reported when the outer scan reaches it. + ++depth; + size_t e = openTagEnd(s, i); + i = (e == std::string::npos) ? s.size() : e + 1; + continue; + } + if (isOpenTag(s, i, "comment")) { + size_t c = s.find("", i); + i = (c == std::string::npos) ? s.size() : c + 10; + continue; + } + bool skipped = false; + for (const char *tag : attributeOnlyTags) + if (isOpenTag(s, i, tag)) { + size_t e = openTagEnd(s, i); + i = (e == std::string::npos) ? s.size() : e + 1; + skipped = true; + break; + } + if (skipped) continue; + } else if (s.compare(i, 2, "|>") == 0) { + // |> ... <| is never parsed, so nothing in it is a reference. + size_t c = s.find("<|", i + 2); + i = (c == std::string::npos) ? s.size() : c + 2; + continue; + } else if ((s[i] == '$' || s[i] == '#') && i + 1 < s.size() && s[i + 1] == '[') { + end = refEnd(s, i); + if (end == std::string::npos) end = s.size(); + LintRef ref{s.substr(i, end - i), ""}; + ref.kind = refKind(ref.text); + i = end; + if (ref.kind == "row" && !strict) continue; + bool seen = false; + for (auto &r : finding.refs) seen = seen || r.text == ref.text; + if (!seen) finding.refs.push_back(ref); + continue; + } + ++i; + } + if (!finding.refs.empty()) { + ++stats.flaggedBlocks; + findings.push_back(finding); + } + // Carry on from the end of this 's opening tag, not its body, + // so a nested is still counted and checked. + } + } + + namespace { + + bool fileMentions(const std::string &text, const std::vector &names) { + for (size_t p = 0; (p = text.find('<', p)) != std::string::npos; ++p) + for (const auto &n : names) + if (isOpenTag(text, p, n)) return true; + return false; + } + + bool readFile(const std::string &path, std::string &text) { + std::ifstream in(path, std::ios::binary); + if (!in) return false; + std::ostringstream ss; + ss << in.rdbuf(); + text = ss.str(); + return true; + } + + } + + void findSqlWrappers(const std::string &s, std::vector &wrappers) { + for (size_t pos = 0; (pos = s.find('<', pos)) != std::string::npos; ++pos) { + if (!isOpenTag(s, pos, "tag")) continue; + size_t gt = openTagEnd(s, pos); + if (gt == std::string::npos) return; + std::string open = s.substr(pos, gt - pos); + // name="X" (or name='X') + std::string lower = open; + for (auto &c : lower) c = (char)std::tolower((unsigned char)c); + size_t n = lower.find("name="); + if (n == std::string::npos || n + 5 >= open.size()) continue; + char q = open[n + 5]; + if (q != '"' && q != '\'') continue; + size_t e = open.find(q, n + 6); + if (e == std::string::npos) continue; + std::string name = open.substr(n + 6, e - n - 6); + // Tag names match without regard to case (isOpenTag lowercases). + for (auto &c : name) c = (char)std::tolower((unsigned char)c); + size_t close = std::string::npos; + for (size_t c = gt; (c = s.find(" in the definition whose statement is the caller's content. + for (size_t p = 0; (p = body.find('<', p)) != std::string::npos; ++p) { + if (!isOpenTag(body, p, "sql")) continue; + size_t bgt = openTagEnd(body, p); + if (bgt == std::string::npos) break; + size_t end = std::string::npos; + for (size_t c = bgt; (c = body.find(" &paths, bool strict, std::ostream &out, std::ostream &err) { + namespace fs = std::filesystem; + std::vector files; + bool error = false; + for (const auto &p : paths) { + std::error_code ec; + if (fs::is_directory(p, ec)) { + std::vector found; + for (auto it = fs::recursive_directory_iterator(p, fs::directory_options::skip_permission_denied, ec); + it != fs::recursive_directory_iterator(); it.increment(ec)) { + if (ec) break; + std::string name = it->path().filename().string(); + if (it->is_directory(ec) && !name.empty() && name[0] == '.') { it.disable_recursion_pending(); continue; } + if (it->is_regular_file(ec)) found.push_back(it->path().string()); + } + std::sort(found.begin(), found.end()); + files.insert(files.end(), found.begin(), found.end()); + } else if (fs::is_regular_file(p, ec)) { + files.push_back(p); + } else { + err << "jetc: cannot open " << p << std::endl; + error = true; + } + } + + // Two passes: a custom tag defined anywhere in the scanned files as + // takes its statement from the caller's + // content, so each use of it is checked like an . + std::vector texts(files.size()); + std::vector readOk(files.size(), false); + std::vector wrappers; + for (size_t i = 0; i < files.size(); ++i) { + if (!readFile(files[i], texts[i])) { err << "jetc: cannot read " << files[i] << std::endl; error = true; continue; } + readOk[i] = true; + findSqlWrappers(texts[i], wrappers); + } + std::vector names{"sql"}; + names.insert(names.end(), wrappers.begin(), wrappers.end()); + + std::vector findings; + LintStats stats; + for (size_t i = 0; i < files.size(); ++i) { + if (!readOk[i] || !fileMentions(texts[i], names)) continue; + lintSqlText(files[i], texts[i], strict, wrappers, findings, stats); + } + + std::map byKind; + std::map flaggedFiles; + for (const auto &f : findings) { + flaggedFiles[f.file] = true; + out << f.file << ":" << f.line << ": <" << f.tag << ">" + << (f.tag == "sql" ? "" : " (wraps sql)") << " pastes "; + for (size_t i = 0; i < f.refs.size(); ++i) { + out << (i ? ", " : "") << f.refs[i].text << " (" << f.refs[i].kind << ")"; + ++byKind[f.refs[i].kind]; + } + out << "\n"; + } + out << "lint-sql: " << stats.flaggedBlocks << " of " << stats.sqlBlocks + << " blocks" << (wrappers.empty() ? "" : " (and uses of tags that wrap )") + << " paste values into the statement, in " << flaggedFiles.size() + << " of " << stats.files << " files"; + if (!byKind.empty()) { + out << " ("; + bool first = true; + for (auto &[kind, n] : byKind) { out << (first ? "" : ", ") << kind << " " << n; first = false; } + out << ")"; + } + out << std::endl; + + if (error) return 2; + return findings.empty() ? 0 : 1; + } + +} diff --git a/jetc/JetLint.h b/jetc/JetLint.h new file mode 100644 index 0000000..104c8b8 --- /dev/null +++ b/jetc/JetLint.h @@ -0,0 +1,57 @@ +#ifndef __JetLint_h__ +#define __JetLint_h__ + +#include +#include +#include + +// jetc --lint-sql: phase 2 of docs/design-sql-bound-parameters.md. +// +// Lists every whose body pastes a variable into the statement text +// ($[...] or #[...]) instead of passing it with . These are the +// queries to migrate. A value from a row of an earlier query +// ($[.column], $[session.column], $[?N], $[#]) is not reported, as the +// design doc specifies, unless strict is set. + +namespace jetc { + + struct LintRef { + std::string text; // the reference as written, e.g. "$[:email]" + std::string kind; // cgi, global, local, session, env, keyword, row, ... + }; + + struct LintFinding { + std::string file; + int line; // line of the + std::vector refs; + }; + + struct LintStats { + int files = 0; // files that contain at least one + int sqlBlocks = 0; + int flaggedBlocks = 0; + }; + + // Scans one file's text. Appends to findings and stats. + // wrappers: custom tag names whose uses count as (see + // findSqlWrappers). + void lintSqlText(const std::string &file, const std::string &text, bool strict, + const std::vector &wrappers, + std::vector &findings, LintStats &stats); + + // Adds to wrappers each custom tag defined in text () + // whose definition runs the caller's content as its statement: + // . + void findSqlWrappers(const std::string &text, std::vector &wrappers); + + // Scans files and directories (recursively; every regular file, + // whatever its extension). Tag definitions are collected from all of + // them first, so a wrapper defined in one file is found in all. + // Prints one line per finding and a summary. Returns 0 when nothing is found, 1 when something is, + // 2 on an error (unreadable path). + int lintSql(const std::vector &paths, bool strict, std::ostream &out, std::ostream &err); + +} + +#endif diff --git a/jetc/build-compiled.sh b/jetc/build-compiled.sh index 525f71e..b055cbb 100755 --- a/jetc/build-compiled.sh +++ b/jetc/build-compiled.sh @@ -21,7 +21,7 @@ OUTNAME="$2" if [ ! -x "$HERE/jetc" ]; then echo "Building jetc..." - g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" + g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" "$HERE/JetLint.cpp" fi echo "Compiling $INPUT -> $OUTNAME.compiled.cpp" diff --git a/jetc/build-shared.sh b/jetc/build-shared.sh index 67a4340..30e4753 100755 --- a/jetc/build-shared.sh +++ b/jetc/build-shared.sh @@ -30,7 +30,7 @@ STEM="$(basename "$OUTPUT" .jtc)" if [ ! -x "$HERE/jetc" ]; then echo "Building jetc..." - g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" + g++ -g -std=c++23 -o "$HERE/jetc" "$HERE/jetc.cpp" "$HERE/JetParser.cpp" "$HERE/JetCodegen.cpp" "$HERE/JetLint.cpp" fi echo "Compiling $INPUT -> $STEM.compiled.cpp (shared mode)" diff --git a/jetc/jetc.cpp b/jetc/jetc.cpp index 218c05e..48c99de 100644 --- a/jetc/jetc.cpp +++ b/jetc/jetc.cpp @@ -1,5 +1,6 @@ #include "JetParser.h" #include "JetCodegen.h" +#include "JetLint.h" #include #include #include @@ -26,17 +27,27 @@ namespace { int main(int argc, char **argv) { bool shared = false; + bool lintSql = false; + bool strict = false; std::vector pos; for (int i = 1; i < argc; ++i) { std::string a = argv[i]; if (a == "--shared") shared = true; + else if (a == "--lint-sql") lintSql = true; + else if (a == "--strict") strict = true; else pos.push_back(a); } - if (pos.size() < 2) { + if (lintSql && !pos.empty()) + return jetc::lintSql(pos, strict, std::cout, std::cerr); + if (pos.size() < 2 || lintSql || strict) { std::cerr << "usage: jetc [--shared] " << std::endl; - std::cerr << " --shared emit a dlsym()-able entry point (see build-shared.sh)" << std::endl; - std::cerr << " instead of a standalone main()." << std::endl; - return 1; + std::cerr << " jetc --lint-sql [--strict] ..." << std::endl; + std::cerr << " --shared emit a dlsym()-able entry point (see build-shared.sh)" << std::endl; + std::cerr << " instead of a standalone main()." << std::endl; + std::cerr << " --lint-sql list every that pastes a variable into the statement" << std::endl; + std::cerr << " instead of using (exit 1 if any, 0 if none)." << std::endl; + std::cerr << " --strict with --lint-sql, also report row values ($[.column] etc.)." << std::endl; + return lintSql ? 2 : 1; } const std::string &inputPath = pos[0]; const std::string &outputPath = pos[1]; diff --git a/tests/lintsql/clean.expected b/tests/lintsql/clean.expected new file mode 100644 index 0000000..6829a99 --- /dev/null +++ b/tests/lintsql/clean.expected @@ -0,0 +1 @@ +lint-sql: 0 of 1 blocks paste values into the statement, in 0 of 1 files diff --git a/tests/lintsql/clean/ok.page b/tests/lintsql/clean/ok.page new file mode 100644 index 0000000..94b522f --- /dev/null +++ b/tests/lintsql/clean/ok.page @@ -0,0 +1 @@ +select * from t where id=? diff --git a/tests/lintsql/missing.expected b/tests/lintsql/missing.expected new file mode 100644 index 0000000..302f38a --- /dev/null +++ b/tests/lintsql/missing.expected @@ -0,0 +1,2 @@ +jetc: cannot open no-such-dir +lint-sql: 0 of 1 blocks paste values into the statement, in 0 of 1 files diff --git a/tests/lintsql/site.expected b/tests/lintsql/site.expected new file mode 100644 index 0000000..26c2ecc --- /dev/null +++ b/tests/lintsql/site.expected @@ -0,0 +1,6 @@ +site/kinds.page:1: pastes $[plain] (global), $[:form] (cgi), $[@sess] (session) +site/kinds.page:2: pastes $[&HTTP_HOST] (env), $[%kw] (keyword), $[!g] (global), #[loc] (local) +site/kinds.page:3: pastes $[late] (global) +site/kinds.page:6: pastes $[name$[ix]] (global) +site/uses.page:1: (wraps sql) pastes $[_login_id] (global) +lint-sql: 5 of 11 blocks (and uses of tags that wrap ) paste values into the statement, in 2 of 4 files (cgi 1, env 1, global 5, keyword 1, local 1, session 1) diff --git a/tests/lintsql/site/.hidden/x.page b/tests/lintsql/site/.hidden/x.page new file mode 100644 index 0000000..adeb359 --- /dev/null +++ b/tests/lintsql/site/.hidden/x.page @@ -0,0 +1 @@ +select $[hidden] diff --git a/tests/lintsql/site/kinds.page b/tests/lintsql/site/kinds.page new file mode 100644 index 0000000..de3cb29 --- /dev/null +++ b/tests/lintsql/site/kinds.page @@ -0,0 +1,6 @@ +select * from t where a='$[plain]' and b=$[:form] and c='$[@sess]' +select '$[&HTTP_HOST]', '$[%kw]', '$[!g]', '#[loc]' +select 1 + from t + where x=$[late] and y=$[late] +select * from a where k=$[name$[ix]] diff --git a/tests/lintsql/site/notes.txt b/tests/lintsql/site/notes.txt new file mode 100644 index 0000000..9a79689 --- /dev/null +++ b/tests/lintsql/site/notes.txt @@ -0,0 +1 @@ +no queries here $[x] diff --git a/tests/lintsql/site/safe.page b/tests/lintsql/site/safe.page new file mode 100644 index 0000000..9f3a9c7 --- /dev/null +++ b/tests/lintsql/site/safe.page @@ -0,0 +1,11 @@ +select $[.col], $[projman.col], $[?1], $[#], $[?#] from t +select * from t where a=? and b=? + + + + uses $[nothing] + +select '|>$[not_parsed]<|' + +$[not_sql] +$[outside_sql] diff --git a/tests/lintsql/site/tags.tag b/tests/lintsql/site/tags.tag new file mode 100644 index 0000000..a015382 --- /dev/null +++ b/tests/lintsql/site/tags.tag @@ -0,0 +1,4 @@ + + + + diff --git a/tests/lintsql/site/uses.page b/tests/lintsql/site/uses.page new file mode 100644 index 0000000..3dbaea0 --- /dev/null +++ b/tests/lintsql/site/uses.page @@ -0,0 +1,3 @@ +select * from t where u=$[_login_id] +select * from t where u=$[.id] +$[not_sql] diff --git a/tests/lintsql/strict.expected b/tests/lintsql/strict.expected new file mode 100644 index 0000000..6047b9d --- /dev/null +++ b/tests/lintsql/strict.expected @@ -0,0 +1,8 @@ +site/kinds.page:1: pastes $[plain] (global), $[:form] (cgi), $[@sess] (session) +site/kinds.page:2: pastes $[&HTTP_HOST] (env), $[%kw] (keyword), $[!g] (global), #[loc] (local) +site/kinds.page:3: pastes $[late] (global) +site/kinds.page:6: pastes $[name$[ix]] (global) +site/safe.page:1: pastes $[.col] (row), $[projman.col] (row), $[?1] (row), $[#] (row), $[?#] (row) +site/uses.page:1: (wraps sql) pastes $[_login_id] (global) +site/uses.page:2: (wraps sql) pastes $[.id] (row) +lint-sql: 7 of 11 blocks (and uses of tags that wrap ) paste values into the statement, in 3 of 4 files (cgi 1, env 1, global 5, keyword 1, local 1, row 6, session 1) diff --git a/tests/runbindtests.sh b/tests/runbindtests.sh index a03574b..c5c8a4a 100755 --- a/tests/runbindtests.sh +++ b/tests/runbindtests.sh @@ -36,7 +36,7 @@ else fi echo "== jetc ==" -g++ -g -std=c++23 -o "$WORK/jetc" "$JC/jetc/jetc.cpp" "$JC/jetc/JetParser.cpp" "$JC/jetc/JetCodegen.cpp" || exit 1 +g++ -g -std=c++23 -o "$WORK/jetc" "$JC/jetc/jetc.cpp" "$JC/jetc/JetParser.cpp" "$JC/jetc/JetCodegen.cpp" "$JC/jetc/JetLint.cpp" || exit 1 # A bare must not swallow the content after it looking for . printf 'Aselect ?BC\n' > "$WORK/void.jet" "$WORK/jetc" "$WORK/void.jet" "$WORK/void.cpp" >/dev/null || fail=1 diff --git a/tests/runlinttests.sh b/tests/runlinttests.sh new file mode 100755 index 0000000..d19b985 --- /dev/null +++ b/tests/runlinttests.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Tests for jetc --lint-sql (docs/design-sql-bound-parameters.md, phase 2). +# Runs the lint over tests/lintsql/ and compares with the .expected files. +# Needs no database. +set -u +T="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +JC="$(cd "$T/.." && pwd)" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT +fail=0 + +g++ -g -std=c++23 -o "$WORK/jetc" "$JC/jetc/jetc.cpp" "$JC/jetc/JetParser.cpp" "$JC/jetc/JetCodegen.cpp" "$JC/jetc/JetLint.cpp" || exit 1 + +# check NAME EXPECTED_EXIT ARGS... - output is compared with lintsql/NAME.expected +check() { + local name="$1" want="$2"; shift 2 + (cd "$T/lintsql" && "$WORK/jetc" --lint-sql "$@") > "$WORK/$name.out" 2>&1 + local got=$? + if [ "$got" != "$want" ]; then + echo "FAIL $name: exit $got, expected $want"; fail=1 + elif ! diff -u "$T/lintsql/$name.expected" "$WORK/$name.out"; then + echo "FAIL $name: output differs"; fail=1 + else + echo "PASS $name" + fi +} + +check site 1 site +check strict 1 --strict site +check clean 0 clean +check missing 2 clean no-such-dir + +[ $fail = 0 ] && echo "ALL PASSED" || echo "FAILURES" +exit $fail