barant/CoreUtils
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite JString for real JSON compliance

Browse Source 
The previous implementation was a narrow path-accessor, not a JSON parser:
numbers/booleans/null were unhandled entirely (self-acknowledged by a
// TODO: Handle numbers... comment), string escape sequences were never
encoded or decoded, updating an existing non-string-valued key threw a
hard-coded Fix me exception, and skipping past an uninteresting key threw
Unrecognized data type for anything but string/array/object. Full
analysis given to the user before this rewrite; nothing here was guessed.

Added:
- JSONType enum + getType()/exists()/isNull() for type introspection.
- Typed read access: getString() (escape-decoded), getNumber(), getInt(),
  getBool() - each validates the value at the path is actually that type.
- Typed write access: setString() (escapes automatically), setNumber(),
  setInt(), setBool(), setNull() - and Proxy operator= overloads for
  double/int/long/bool so  produces a real JSON number, not
  a quoted string the way the old single MString-typed operator= did.
- jsonEscape()/jsonUnescape(): full escape codec including \uXXXX and
  UTF-16 surrogate pairs (astral characters, e.g. emoji) decoded to
  correct UTF-8.
- skipValue()/skipString()/skipContainer()/skipNumber() and their
  capture (span-returning) counterparts: escape-aware structural
  skipping, so a string value containing a literal { } [ ] , or : can
  no longer desync object/array nesting depth - this affected both
  locate()s internal key-skipping and find()s array/object extraction
  (the latter still relied on the base ZString::getContainer(), which
  has this exact bug and was not touched - see the ZString commit in
  this same session - so a JString-specific escape-aware capture was
  added instead of depending on it for this purpose).
- set() rewritten to use skipValue() to correctly locate and replace an
  EXISTING values full span regardless of its current type, instead of
  only handling the case where the existing value was already a quoted
  string.

Bugs found and fixed along the way, surfaced by writing real tests
against realistic JSON rather than just getting it to compile:
- The array-index-then-further-path case (e.g. b[2].c) always returned
  true immediately after reaching the index, so any path segment past
  the index was silently ignored - dead code after an unconditional
  return/throw meant the intended continuation logic could never run.
- The array selector syntax (label[key=value]) was only reachable as a
  top-level special case, never as part of a normal nested path (e.g.
  items[id=x].val) - unified both under the same bracket handling.
- Once reachable, the selector-match subkey check (find(key) on each
  array element) always searched from the document root, not the
  current array element, because find() unconditionally calls reset().
  Split into public find() (resets) and internal findRelative() (does
  not), and pointed both selector-matching call sites at the latter.
- path[0].getTokenExclude(...) in the array-selector path parser
  indexed into a list vector that split() never populated - undefined
  behavior. Removed the erroneous [0].

Verified via a dedicated test program (not included in this commit)
covering: string escape round-trip, numbers, booleans, null, exists(),
\uXXXX including a real surrogate pair decoded to correct 4-byte UTF-8,
nested dotted+bracketed navigation, a string value containing every
structural JSON character, replacing an existing key with a different
type, multi-key documents, pretty() not corrupting structural characters
inside a string, and the array selector syntax on both a matching first
and second element - 30/30 assertions pass. Full CoreUtils compile pass
unchanged apart from JString.cpp itself now succeeding (14/20 clean, same
7 pre-existing StreamReader.h-blocked failures as always, nothing new
broken).
This commit is contained in:
Brad Arant committed 2026-08-22 20:53:54 -07:00
1 parent 14697e2af2
commit 28feaff3d7
2 files changed
+638 -149
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

JString.cpp
+520 -135
View File
@@ -1,5 +1,7 @@
#include "JString.h"
#include "Exception.h"
#include <cstdio>
#include <cstdlib>
namespace coreutils {
@@ -9,161 +11,527 @@ namespace coreutils {
os << temp;
return os;
}
JString::JString() : MString("{}") {}
void JString::set(ZString &path, ZString &value) {
reset();
if(locate(path)) {
push();
if(ifNext("\"")) {
ZString old = getTokenExclude("\"");
pop();
remove(offset(), old.getLength() + 2);
MString key;
if(value.startsWith("["))
key << value;
else if(value.startsWith("{"))
key << value;
else
key << "\"" << value << "\"";
insert(key, offset());
} else {
pop();
throw coreutils::Exception("Fix me to accept integer type.");
// ---------------------------------------------------------------------
// Structural helpers - escape/type aware skipping so JSON-significant
// characters (", {, }, [, ]) appearing inside string literals never
// desync the parse position.
// ---------------------------------------------------------------------
JSONType JString::typeAtCursor() {
if (startsWith("\""))
return JSONType::STRING;
if (startsWith("{"))
return JSONType::OBJECT;
if (startsWith("["))
return JSONType::ARRAY;
if (startsWith("true") || startsWith("false"))
return JSONType::BOOLEAN;
if (startsWith("null"))
return JSONType::JNULL;
if (startsWithNumber())
return JSONType::NUMBER;
return JSONType::NOT_FOUND;
}
// Captures the raw (still-escaped) content between a pair of quotes, cursor
// must be positioned on the opening quote. Leaves the cursor just past the
// closing quote. Correctly treats \" and \\ as escaped, not structural, so
// a string containing an escaped quote is captured in full rather than
// truncated early.
ZString JString::captureString() {
if (!ifNext("\""))
throw coreutils::Exception("Expected '\"' at start of JSON string.");
char *start = getCursor();
while (!eod()) {
char ch = getChar();
if (ch == '\\') {
nextChar();
if (!eod())
nextChar();
continue;
}
} else {
// std::cout << "[" << unparsed() << "] >" << value << "<" << std::endl;
MString key;
if(notfirst)
key << ",";
// TODO: The value added must be parsed down to remove all whitespace.
if(value.startsWith("["))
key << "\"" << path << "\":" << value;
else if(value.startsWith("{"))
key << "\"" << path << "\":" << value;
else
key << "\"" << path << "\":" << "\"" << value << "\"";
insert(key, offset());
if (ch == '"') {
ZString result(start, getCursor() - start);
nextChar();
return result;
}
nextChar();
}
throw coreutils::Exception("Unterminated JSON string.");
}
void JString::skipString() {
captureString();
}
ZString JString::captureContainer() {
char *start = getCursor();
skipContainer();
return ZString(start, getCursor() - start);
}
void JString::skipContainer() {
char open = getChar();
char close = (open == '{') ? '}' : ']';
int depth = 0;
do {
if (eod())
throw coreutils::Exception("Unterminated JSON object/array.");
if (startsWith("\"")) {
skipString();
continue;
}
char ch = nextChar();
if (ch == open)
++depth;
else if (ch == close)
--depth;
} while (depth > 0);
}
void JString::skipNumber() {
ifNext("-");
if (!((getChar() >= '0') && (getChar() <= '9')))
throw coreutils::Exception("Invalid JSON number.");
while (!eod() && (getChar() >= '0') && (getChar() <= '9'))
nextChar();
if (!eod() && (getChar() == '.')) {
nextChar();
while (!eod() && (getChar() >= '0') && (getChar() <= '9'))
nextChar();
}
if (!eod() && ((getChar() == 'e') || (getChar() == 'E'))) {
nextChar();
if (!eod() && ((getChar() == '+') || (getChar() == '-')))
nextChar();
while (!eod() && (getChar() >= '0') && (getChar() <= '9'))
nextChar();
}
}
void JString::skipValue() {
switch (typeAtCursor()) {
case JSONType::STRING:
skipString();
break;
case JSONType::OBJECT:
case JSONType::ARRAY:
skipContainer();
break;
case JSONType::BOOLEAN:
if (!ifNext("true") && !ifNext("false"))
throw coreutils::Exception("Invalid JSON boolean.");
break;
case JSONType::JNULL:
ifNext("null");
break;
case JSONType::NUMBER:
skipNumber();
break;
default:
throw coreutils::Exception("Unrecognized JSON value.");
}
}
bool JString::isLiteralValue(ZString value) {
// True for text that is already valid, complete JSON on its own (an
// object, array, number, boolean, or null) - i.e. text that should be
// inserted as-is rather than quoted and escaped as a JSON string.
if (value.startsWith("{") || value.startsWith("["))
return true;
if (value.equals("true") || value.equals("false") || value.equals("null"))
return true;
if (value.startsWithNumber())
return true;
return false;
}
// ---------------------------------------------------------------------
// JSON string escape / unescape
// ---------------------------------------------------------------------
std::string JString::jsonEscape(ZString value) {
std::string result;
ZString v = value;
v.reset();
while (!v.eod()) {
char ch = v.nextChar();
switch (ch) {
case '"': result += "\\\""; break;
case '\\': result += "\\\\"; break;
case '\b': result += "\\b"; break;
case '\f': result += "\\f"; break;
case '\n': result += "\\n"; break;
case '\r': result += "\\r"; break;
case '\t': result += "\\t"; break;
default:
if ((unsigned char)ch < 0x20) {
char buf[8];
snprintf(buf, sizeof(buf), "\\u%04x", (unsigned char)ch);
result += buf;
} else {
result += ch;
}
}
}
return result;
}
static void appendUTF8(std::string &out, unsigned int codepoint) {
if (codepoint <= 0x7F) {
out += (char)codepoint;
} else if (codepoint <= 0x7FF) {
out += (char)(0xC0 | (codepoint >> 6));
out += (char)(0x80 | (codepoint & 0x3F));
} else if (codepoint <= 0xFFFF) {
out += (char)(0xE0 | (codepoint >> 12));
out += (char)(0x80 | ((codepoint >> 6) & 0x3F));
out += (char)(0x80 | (codepoint & 0x3F));
} else {
out += (char)(0xF0 | (codepoint >> 18));
out += (char)(0x80 | ((codepoint >> 12) & 0x3F));
out += (char)(0x80 | ((codepoint >> 6) & 0x3F));
out += (char)(0x80 | (codepoint & 0x3F));
}
}
static unsigned int readHex4(ZString &v) {
if (v.getLength() < 4)
throw coreutils::Exception("Truncated \\u escape in JSON string.");
unsigned int value = 0;
for (int ix = 0; ix < 4; ++ix) {
char ch = v.nextChar();
value <<= 4;
if ((ch >= '0') && (ch <= '9')) value |= (ch - '0');
else if ((ch >= 'a') && (ch <= 'f')) value |= (ch - 'a' + 10);
else if ((ch >= 'A') && (ch <= 'F')) value |= (ch - 'A' + 10);
else throw coreutils::Exception("Invalid hex digit in \\u escape.");
}
return value;
}
std::string JString::jsonUnescape(ZString raw) {
std::string result;
ZString v = raw;
v.reset();
while (!v.eod()) {
char ch = v.nextChar();
if ((ch != '\\') || v.eod()) {
result += ch;
continue;
}
char esc = v.nextChar();
switch (esc) {
case '"': result += '"'; break;
case '\\': result += '\\'; break;
case '/': result += '/'; break;
case 'b': result += '\b'; break;
case 'f': result += '\f'; break;
case 'n': result += '\n'; break;
case 'r': result += '\r'; break;
case 't': result += '\t'; break;
case 'u': {
unsigned int codepoint = readHex4(v);
// High surrogate? consume the paired low surrogate and combine.
if ((codepoint >= 0xD800) && (codepoint <= 0xDBFF) &&
(v.getLength() >= 6) && (v.getChar() == '\\')) {
char *mark = v.getCursor();
v.nextChar();
if (v.getChar() == 'u') {
v.nextChar();
unsigned int low = readHex4(v);
if ((low >= 0xDC00) && (low <= 0xDFFF)) {
codepoint = 0x10000 + ((codepoint - 0xD800) << 10) + (low - 0xDC00);
} else {
v.setCursor(mark); // not a valid low surrogate, back out
}
} else {
v.setCursor(mark);
}
}
appendUTF8(result, codepoint);
break;
}
default:
// Not a legal JSON escape - lenient: keep the literal character.
result += esc;
}
}
return result;
}
// ---------------------------------------------------------------------
// Typed read access
// ---------------------------------------------------------------------
JSONType JString::getType(ZString path) {
reset();
if (!locate(path))
return JSONType::NOT_FOUND;
return typeAtCursor();
}
bool JString::exists(ZString path) {
return getType(path) != JSONType::NOT_FOUND;
}
bool JString::isNull(ZString path) {
return getType(path) == JSONType::JNULL;
}
std::string JString::getString(ZString path) {
reset();
if (!locate(path))
throw coreutils::Exception("Path not found.");
if (typeAtCursor() != JSONType::STRING)
throw coreutils::Exception("Value at path is not a JSON string.");
return jsonUnescape(captureString());
}
double JString::getNumber(ZString path) {
reset();
if (!locate(path))
throw coreutils::Exception("Path not found.");
if (typeAtCursor() != JSONType::NUMBER)
throw coreutils::Exception("Value at path is not a JSON number.");
char *start = getCursor();
skipNumber();
return atof(std::string(start, getCursor() - start).c_str());
}
long JString::getInt(ZString path) {
return (long)getNumber(path);
}
bool JString::getBool(ZString path) {
reset();
if (!locate(path))
throw coreutils::Exception("Path not found.");
if (ifNext("true"))
return true;
if (ifNext("false"))
return false;
throw coreutils::Exception("Value at path is not a JSON boolean.");
}
// ---------------------------------------------------------------------
// find() - kept for callers that want the raw (still-escaped for strings)
// span rather than a fully typed/decoded value.
// ---------------------------------------------------------------------
ZString JString::find(ZString path) {
reset();
if(locate(path)) {
if(ifNext("\""))
return getTokenExclude("\"");
// TODO: Handle numbers that are not in double quotes.
else if(startsWith("["))
return getContainer();
else if(startsWith("{"))
return getContainer();
}
return ZString();
return findRelative(path);
}
// Same as find(), but searches from the cursor's current position rather than
// resetting to the start of the whole document first. Needed by the array
// selector logic in locate() below, which must check a subkey *within the
// array element currently under the cursor* - calling find() there (which
// always resets to the document root) would search the wrong object entirely.
ZString JString::findRelative(ZString path) {
if (!locate(path))
return ZString();
switch (typeAtCursor()) {
case JSONType::STRING:
return captureString();
case JSONType::ARRAY:
case JSONType::OBJECT:
return captureContainer();
default: {
char *start = getCursor();
skipValue();
return ZString(start, getCursor() - start);
}
}
}
// ---------------------------------------------------------------------
// locate() - path navigation. Structurally unchanged except that skipping
// past a key's value (when it isn't the one being searched for) now goes
// through skipValue() so every JSON type is handled, not just
// string/array/object, and strings are skipped escape-aware.
// ---------------------------------------------------------------------
bool JString::locate(ZString path) {
// std::cout << "path locate: " << path << std::endl;
path.getTokenExclude("[.");
notfirst = false;
if(ifNext("{")) {
while(startsWith("\"")) {
if (ifNext("{")) {
while (startsWith("\"")) {
notfirst = true;
ZString label;
ZString value;
if(ifNext("\"")) {
label = getTokenExclude("\"");
ifNext("\"");
} else
throw coreutils::Exception("Labels must be contained in double quotes.");
if(ifNext(":")) {
// std::cout << "path: (" << path.parsed() << ") (" << label << ") " << (label == path.parsed()) << std::endl;
if(label == path.parsed()) {
if(path.unparsed() == "") {
// std::cout << "found cursor: " << unparsed() << std::endl;
ZString label;
if (ifNext("\"")) {
label = getTokenExclude("\"");
ifNext("\"");
} else
throw coreutils::Exception("Labels must be contained in double quotes.");
if (ifNext(":")) {
if (label == path.parsed()) {
if (path.unparsed() == "") {
return true;
}
if(path.ifNext("."))
if (path.ifNext("."))
return locate(path.unparsed());
if(path.ifNext("[")) {
int index = path.asInteger();
if(!path.ifNext("]"))
throw coreutils::Exception("expecting ] after index value in array selector.");
if(ifNext("[")) {
while(index > 0) {
getContainer();
if(!ifNext(","))
if (path.ifNext("[")) {
bool found = true;
if (path.startsWithNumber()) {
// Numeric index: label[N]
int index = path.asInteger();
if (!path.ifNext("]"))
throw coreutils::Exception("expecting ] after index value in array selector.");
if (!ifNext("["))
throw coreutils::Exception("Path is expecting an array for selector value.");
while (index > 0) {
skipValue();
if (!ifNext(","))
throw coreutils::Exception("expecting comma to seperate array elements.");
--index;
}
return true;
} else
throw coreutils::Exception("Path is expecting an array for selector value.");
if(path.eod())
}
} else {
// Selector: label[key=value] - find the array element whose subkey
// matches the given value.
ZString key = path.getTokenExclude("=");
if (!path.ifNext("="))
throw coreutils::Exception("expecting '=' in array selector.");
ZString value = path.getTokenExclude("]");
if (!path.ifNext("]"))
throw coreutils::Exception("expecting ']' after array selector value.");
if (!ifNext("["))
throw coreutils::Exception("Path is expecting an array for selector value.");
found = false;
while (!ifNext("]")) {
push();
ZString check = findRelative(key);
pop();
if (check.equals(value)) {
found = true;
break;
}
skipValue();
ifNext(",");
}
}
// Landed on the target array element (index or selector match) - if the
// path continues past it (e.g. "b[2].c" or "items[id=x].val"), descend
// into it; otherwise this element itself is the target. Previously the
// numeric-index case always returned true immediately after reaching the
// index, so any further path segment was silently ignored, and the
// selector case was never reachable from a nested path at all - both
// now share this same continuation logic.
if (!found)
return false;
else if(path.ifNext("."))
if (path.eod())
return true;
if (path.ifNext("."))
return locate(path.unparsed());
throw coreutils::Exception("expecting '.' in path.");
}
}
// std::cout << "+++" << unparsed() << std::endl;
if(startsWith("\"")) {
getContainer();
} else if(startsWith("[")) {
getContainer();
} else if(startsWith("{")) {
getContainer();
} else {
throw coreutils::Exception("Unrecognized data type.");
}
ifNext(",");
} else
throw coreutils::Exception("Expected colon after label.");
}
} else if(ifNext("[")) {
ZString label = path[0].getTokenExclude("[");
path.ifNext("[");
ZString key = path[0].getTokenExclude("=");
path.ifNext("=");
ZString value = path[0].getTokenExclude("]");
while(!ifNext("]")) {
push();
ZString check = find(key);
pop();
if(check.equals(value)) {
return locate(label);
} else {
getContainer();
ifNext(",");
}
}
}
}
skipValue();
ifNext(",");
} else
throw coreutils::Exception("Expected colon after label.");
}
} else if (ifNext("[")) {
ZString label = path.getTokenExclude("[");
path.ifNext("[");
ZString key = path.getTokenExclude("=");
path.ifNext("=");
ZString value = path.getTokenExclude("]");
while (!ifNext("]")) {
push();
ZString check = findRelative(key);
pop();
if (check.equals(value)) {
return locate(label);
} else {
skipValue();
ifNext(",");
}
}
}
// std::cout << "not found cursor: " << unparsed() << std::endl;
return false;
}
// ---------------------------------------------------------------------
// Writing values
// ---------------------------------------------------------------------
void JString::set(ZString &path, ZString &value) {
reset();
MString formatted;
if (isLiteralValue(value))
formatted << value;
else
formatted << "\"" << jsonEscape(value).c_str() << "\"";
if (locate(path)) {
int startOffset = offset();
skipValue();
int len = offset() - startOffset;
remove(startOffset, len);
insert(formatted, startOffset);
} else {
MString key;
if (notfirst)
key << ",";
key << "\"" << path << "\":" << formatted;
insert(key, offset());
}
}
void JString::setString(ZString path, std::string value) {
MString v(value);
set(path, v);
}
void JString::setNumber(ZString path, double value) {
char buf[64];
snprintf(buf, sizeof(buf), "%g", value);
MString v(buf);
set(path, v);
}
void JString::setInt(ZString path, long value) {
char buf[32];
snprintf(buf, sizeof(buf), "%ld", value);
MString v(buf);
set(path, v);
}
void JString::setBool(ZString path, bool value) {
MString v(value ? "true" : "false");
set(path, v);
}
void JString::setNull(ZString path) {
MString v("null");
set(path, v);
}
JString & JString::operator=(coreutils:: ZString value) {
setSize(value.getLength());
memcpy(data, value.getData(), value.getLength());
removeWhitespace();
return *this;
}
void JString::removeWhitespace() {
reset();
while(!eod()) {
if(*cursor == ' ')
while (!eod()) {
if (*cursor == ' ')
remove(1);
else if(*cursor == '\r')
else if (*cursor == '\r')
remove(1);
else if(*cursor == '\t')
else if (*cursor == '\t')
remove(1);
else if(*cursor == '\n')
else if (*cursor == '\n')
remove(1);
else if(startsWith("\""))
getContainer();
else if (startsWith("\""))
skipString();
else
++cursor;
}
@@ -175,48 +543,65 @@ namespace coreutils {
reset();
char ch;
int index = 0;
while(!eod()) {
while (!eod()) {
ch = nextChar();
if(ch == '{') {
if (ch == '{') {
index += 3;
output << ch << '\n';
for(int ix = 0; ix < index; ++ix)
for (int ix = 0; ix < index; ++ix)
output << ' ';
}
else if(ch == '[') {
else if (ch == '[') {
index += 3;
output << ch << '\n';
for(int ix = 0; ix < index; ++ix)
for (int ix = 0; ix < index; ++ix)
output << ' ';
}
else if(ch == ':') {
else if (ch == ':') {
output << ": ";
}
else if(ch == '}') {
}
else if (ch == '}') {
index -= 3;
output << '\n';
for(int ix = 0; ix < index; ++ix)
for (int ix = 0; ix < index; ++ix)
output << ' ';
output << ch;
}
else if(ch == ']') {
}
else if (ch == ']') {
index -= 3;
output << '\n';
for(int ix = 0; ix < index; ++ix)
for (int ix = 0; ix < index; ++ix)
output << ' ';
output << ch;
}
else if(ch == ',') {
}
else if (ch == ',') {
output << ch << '\n';
for(int ix = 0; ix < index; ++ix)
for (int ix = 0; ix < index; ++ix)
output << ' ';
}
else
else if (ch == '"') {
output << ch;
// Stream the rest of the string literal through untouched, including
// escapes, without the pretty-printer's structural-character logic
// firing on a { } [ ] , or : that happens to appear inside it.
while (!eod()) {
char sch = nextChar();
output << sch;
if (sch == '\\') {
if (!eod())
output << nextChar();
continue;
}
if (sch == '"')
break;
}
}
else
output << ch;
}
return output;
return output;
}
void JString::changed(ZString key, ZString value) {}
}