Stdio sidecar protocol¶
pydiggs serve --stdio runs pydiggs as a sidecar: a child process that any host
application can start and talk to over its standard input and output. It suits desktop
and native applications in any language (Rust, C#, Java, Swift, C++, TypeScript on Node or
Electron, and so on) that cannot embed CPython. This page is the normative specification
of protocol version 1.3 (1.0 plus the read method in 1.1, convert and
ags4.check in 1.2, and diff in 1.3). Host guidance covers supervision,
packaging and example clients.
The sidecar is part of the core package: it needs no extra and uses only the Python
standard library on top of pydiggs itself. A machine-readable description of the methods
ships with the package as pydiggs/sidecar/openrpc.json (OpenRPC 1.3) and is published as
schemas/openrpc.json.
Findings are untrusted text
Every string field in a findings document (messages, paths, element names, values) can carry text taken from the validated file. Escape it before rendering it in HTML, a rich-text control, a terminal or a log.
Launching¶
pydiggs serve --stdio [--max-bytes N] [--max-findings N] [--timeout S]
[--allow-path DIR]... [--log-level L] [--log-format text|json]
| option | default | meaning |
|---|---|---|
--max-bytes |
26,214,400 (25 MiB) | largest decoded document sent as content, and by path when given |
--max-findings |
10,000 | findings kept per document (a truncation finding marks the cut) |
--timeout |
60 | default and largest per-request time budget, in seconds |
--allow-path DIR |
none | enables by-path requests for files under DIR (repeatable) |
--log-level |
warning |
debug, info, warning, error, critical |
--log-format |
text |
json writes one object per line with ts, level, msg, request_id |
By-path documents may be up to 100 MiB, or --max-bytes when it is given.
python -m pydiggs serve --stdio is equivalent. The sidecar only ever uses pydiggs's
bundled schemas, dictionaries and Schematron rules; it never reaches the network and never
reads local dictionaries next to a document.
Exit status¶
| status | when |
|---|---|
| 0 | exit after shutdown; stdin closed (EOF); stdout closed (broken pipe); SIGTERM, SIGINT or Ctrl+Break |
| 1 | exit without a prior shutdown, or an unexpected internal failure |
| 2 | bad launch arguments, including a refused --allow-path root or unusable stdio |
| 3 | the bundled reference data could not be loaded during initialize |
Framing¶
- One JSON-RPC 2.0 message per line, UTF-8, terminated by
\n(LF). Input lines may end in\r\n; blank lines are skipped. - Split on byte
0x0Aonly. Responses never contain a raw LF or CR, and U+0085, U+2028 and U+2029 are always written as\uescapes, so line splitters that also break on those characters still see one message per line. - Input is decoded as strict UTF-8 whatever the locale. A byte-order mark is an error.
- A line may be at most
max_line_byteslong (reported byinitialize): by defaultceil(4/3 × max_bytes) + 65,536, about 34 MiB. A longer line is discarded up to the next LF and answered with-32600and"id": null. - Responses can be several megabytes on one line (up to
max_findingsfindings). Read them with a streaming line reader that has no fixed buffer size. - Batches (JSON arrays) are not supported (
-32600). Parameters are always an object (by name). - stdout carries only frames. At start the sidecar keeps a private copy of descriptor 1 for frames and points descriptor 1 at stderr, so stray output from Python, C libraries or child processes lands on stderr.
stderr¶
Logs go to stderr. The host must drain stderr or redirect it to a file or the null device: a full stderr pipe (often 64 KiB) blocks the sidecar. Logs never contain document content or absolute paths, and client strings are escaped.
Lifecycle¶
- The host starts the process and sends
initialize. Until its response arrives, the sidecar reads nothing more, so requests may be pipelined behind it. Requests sent beforeinitialize(on a fresh process) are answerednot-initialized; notifications are ignored. initializecompiles the validators for the requested DIGGS versions (all by default; about a second cold), then answers. If this fails the answer isresource-errorand the process exits with status 3.- The host sends requests. Each response carries the request's
id; responses can arrive in a different order from the requests (see Ordering). - To stop: send
shutdown, wait for itsnullresult, then send theexitnotification. Closing stdin also stops the sidecar (status 0) and abandons any running validation.
Version negotiation¶
initialize.params.protocol_version is "MAJOR.MINOR". The major must equal the
sidecar's (1), otherwise the answer is unsupported-protocol-version with
data.supported, e.g. ["1.0", "1.1", "1.2", "1.3"]. The sidecar answers the highest minor it supports that
does not exceed the requested one. Minor versions only add: hosts must ignore unknown
result fields; new methods are advertised in capabilities. The sidecar refuses unknown
parameters with -32602 (except unknown keys inside capabilities, which it ignores).
Ordering¶
ping,versions,rules.listandrules.explainare answered immediately, even while a validation runs.- Validations run one at a time. At most one more may wait behind the running one;
a further
validateis answeredbusyat once. A host that needs throughput runs a pool of sidecar processes. - Request ids must be unique among in-flight validations; reusing one is
-32600.
Cancellation and time limits¶
$/cancelRequest {id}: a waiting validation is removed and answeredcancelledat once. The running one endscancelledat its next step boundary. Unknown or finished ids are ignored.time_budget_s(default and maximum:--timeout) is checked at the same step boundaries. Running out is not an error: the result is a normal findings document with astructure.timeoutERROR finding and the unfinished steps markedincomplete.- Neither can interrupt a single schema or Schematron pass already running. For a hard limit, the host must kill the process (see host guidance) and show the file as "not checked", never as passed.
Shutdown and exit¶
shutdown lets the running validation finish, answers shutting-down to the waiting
one and to every later request, and then answers null (after the running validation's
own response). exit then ends the process with status 0; exit without shutdown
ends it with status 1. exit is honoured at any time and abandons any running work.
Methods¶
initialize¶
| param | type | notes |
|---|---|---|
protocol_version |
string, required | "1.3" (or "1.2": no 1.3 members, and diff is method-not-found; or "1.1": no 1.2 members either, and convert and ags4.check are method-not-found too; or "1.0": no 1.1 members either, and read is method-not-found too) |
client_info |
object | {name ≤ 128 chars, version ≤ 64 chars}, logged at info |
capabilities |
object | {progress: bool} asks for $/progress notifications |
diggs_versions |
string[] | DIGGS versions to warm up (default: every released bundled version, not the experimental 3.1-dev) |
Result:
{
"protocol_version": "1.3",
"pydiggs_version": "1.13.0",
"findings_schema_version": "1.2",
"findings_schema_url": "https://xinp-hub.github.io/pydiggs/schemas/diggs-findings-1.2.schema.json",
"model_schema_version": "1.0",
"model_schema_url": "https://xinp-hub.github.io/pydiggs/schemas/diggs-model-1.0.schema.json",
"diff_schema_version": "1.0",
"diff_schema_url": "https://xinp-hub.github.io/pydiggs/schemas/diggs-diff-1.0.schema.json",
"diggs_versions": ["2.5.a", "2.6", "3.0.0", "3.1-dev"],
"reference_data": {"def": "…", "schema-dev": "…", "schemas": "…"},
"capabilities": {
"validate": {"content": true, "path": false},
"read": {"content": true, "path": false, "select": true},
"convert": {"content": true, "path": false, "to": ["diggs", "ags4"], "ags_versions": ["4.2", "4.1.1"], "residual": true},
"ags4_check": {"content": true, "path": false, "dictionary": true},
"diff": {"content": true, "path": false, "match": ["auto", "gml-id", "natural", "gml_id"]},
"cancel": true, "progress": false, "rules": true, "ping": true
},
"limits": {
"max_bytes": 26214400, "max_path_bytes": 104857600, "max_line_bytes": 35018070,
"max_findings": 10000, "max_response_bytes_estimate": 20545536,
"max_result_bytes": 268435456, "time_budget_s": 60.0
}
}
capabilities.validate.path and capabilities.read.path are true only when the sidecar
was started with --allow-path. max_response_bytes_estimate is a rough upper bound for
sizing buffers for validate, not a guarantee; max_result_bytes is the hard limit for a
read result. A host that negotiates "1.0" gets the 1.0 answer: no model_schema_*,
no capabilities.read and no max_result_bytes; one that negotiates "1.1" gets no
capabilities.convert and no capabilities.ags4_check; one that negotiates "1.2" gets
no diff_schema_* and no capabilities.diff.
validate¶
| param | type | notes |
|---|---|---|
content_base64 |
string | the document, standard base64 with padding and no whitespace |
path |
string | an absolute path under an --allow-path root (instead of content) |
name |
string ≤ 4096 | the name to report; reduced to a basename (default input.xml, or the path's basename) |
checks |
string[] | steps to run (default: all), as for pydiggs validate --checks |
diggs_version |
string | force a DIGGS version instead of detecting it |
time_budget_s |
number | 0 < t ≤ --timeout |
Exactly one of content_base64 and path is required. The encoded length is checked
against max_bytes before decoding.
The result is a complete one-file diggs-findings 1.2 document, the same as
pydiggs validate --format json gives for that file apart from files[].path (always the
basename here) and with no absolute_path. Validate it against the document's own
$schema and tolerate unknown fields. A malformed or invalid document is a normal result
with findings, not an error.
read¶
Protocol 1.1. Reads one document into a diggs-model 1.0 document: what is in the file,
as pydiggs.read() returns it.
| param | type | notes |
|---|---|---|
content_base64 |
string | as for validate |
path |
string | as for validate |
name |
string ≤ 4096 | as for validate |
diggs_version |
string | force a DIGGS version instead of detecting it |
time_budget_s |
number | 0 < t ≤ --timeout |
select |
object | {"results": "all" \| "none" \| ["key", …]}: which result tables keep their values (default all; at most 1,000 keys) |
The result is the document pydiggs.read_bytes(data, name=name).to_json() gives for the
same bytes. Validate it against its own $schema and tolerate unknown fields.
select: a table that is left out keeps its columns androw_countwith emptyvalues. List a file with{"results": "none"}, then ask for the measurements you need bykey; the sidecar keeps the last few models it read (by content hash), so the second call does not parse the file again. It is a cache, not state: there are no handles.- A schema-invalid file still reads; run
validatefor a verdict. - When
time_budget_sruns out the result is still a document, withcomplete: falseand the members read so far. - A
readshares the queue withvalidate(one running, one waiting) and can be cancelled the same way: cancellation and the time budget are checked between top-level members and every so often inside a large one. - The reader's work limits (Reading files) apply; reaching one
is a result with
complete: false, not an error. - A
selectkey that matches no measurement is ignored. - Bytes that are not a DIGGS document are the error
invalid-document; a result overlimits.max_result_bytesisresult-too-large(ask for fewer tables withselect).
convert¶
Protocol 1.2. Converts one AGS4 file to DIGGS 3.0.0 (to: "diggs", the default),
or one DIGGS document to an AGS4 file (to: "ags4"), by the stable
pydiggs.convert and mapping table 1.2 (see AGS4 and DIGGS conversion),
with every loss reported. A host detects to: "ags4" and ags_version from
capabilities.convert (its to and ags_versions lists); a 1.10 sidecar also reports
protocol 1.2 and answers both with invalid-params.
| param | type | notes |
|---|---|---|
content_base64 |
string | the AGS4 file (or the DIGGS document for to: "ags4"), as for validate |
path |
string | as for validate |
name |
string ≤ 4096 | the name to report; reduced to a basename (default input.ags, input.xml for to: "ags4", or the path's basename) |
to |
string | the format to convert to: "diggs" (the default; the content is AGS4) or "ags4" (the content is DIGGS XML) |
diggs_version |
string | to: "diggs" only: the DIGGS version to write, "3.0.0" (the default) |
ags_version |
string | to: "ags4" only: the AGS4 version to write, "4.2" (the default) or "4.1.1"; as of November 2026 the BGS AGS validator and python-ags4 1.2.0 accept AGS4 only up to 4.1.1 |
time_budget_s |
number | 0 < t ≤ --timeout |
The result is the conversion result that POST /v1/convert answers, byte for byte: the
output bytes as output_base64 with media_type, output_name and output_sha256, the
report about the source and the validation of the output (both diggs-findings 1.2;
the output's is pydiggs.ags4.check for AGS4), diggs_version, ags_version,
mapping_version, complete and lossless, and residual_base64: the residual AGS4
file of what an AGS4 to DIGGS conversion did not convert, or null (the fields are in
AGS4 and DIGGS conversion). capabilities.convert lists the
targets, the AGS4 versions written and residual: true.
- Losses, an output that fails validation or the AGS4 check, and an exhausted
time_budget_s(complete: false) are normal results. - Bytes that are not AGS4 (for
to: "diggs") are the errorinvalid-documentwithkindnot-ags4(orunsupported-versionfor AGS 3); bytes that are not DIGGS (forto: "ags4") areinvalid-documentwith thekindofread. A parameter given for the other direction isinvalid-params; a result overlimits.max_result_bytesisresult-too-large. - A
convertshares the queue withvalidateandread. A queued one can be cancelled; a running one is checked for cancellation only when it ends.
ags4.check¶
Protocol 1.2. Checks one AGS4 file against the AGS4 rules (Rules 1 to 20) and answers
with the diggs-findings 1.2 document ags4.check gives for the same bytes, line
numbers included (see AGS4 files).
| param | type | notes |
|---|---|---|
content_base64, path, name, time_budget_s |
as for convert |
|
dictionary_base64 |
string | an AGS4 data dictionary (pyDIGGS bundles none), standard base64 |
dictionary_path |
string | the dictionary by path, under an --allow-path root (vetted like path) |
At most one of dictionary_base64 and dictionary_path. Each file is held to the limit
of the way it came: max_bytes inline (base64), max_path_bytes by path. Inline, both
travel in one request line, so they must fit in max_line_bytes together. A dictionary
that cannot be used is invalid-params naming dictionary. time_budget_s covers
reading the dictionary and checking together: a dictionary read it stops is
invalid-params naming dictionary ("could not be read within the time budget"). Rule breaks and a file that is not AGS4 at all are normal results with
findings. The sidecar writes the file to a private temporary folder (mode 0700) for the
checker and removes it at once.
diff¶
Protocol 1.3. Compares two DIGGS files with the stable
pydiggs.diff.compare() and answers with its diggs-diff 1.0 document:
what was added, removed or changed. A session that negotiated 1.2 or older gets
method-not-found.
| param | type | notes |
|---|---|---|
old_base64, new_base64 |
string | the earlier and the later file, standard base64 as for validate |
old_path, new_path |
string | a file by path, under an --allow-path root (vetted like path), instead of its content |
old_name, new_name |
string ≤ 4096 | the names to report; reduced to a basename (default input.xml, or the path's basename) |
match |
string | how records are paired: "auto" (the default), "gml-id" (also "gml_id") or "natural" |
rel_tol |
number | two numbers are equal within this fraction of the larger one: 0 ≤ r < 1, default 1e-9 |
abs_tol |
number | or within this much, in the value's own unit: ≥ 0, default 0 |
max_changes |
integer | most changes listed, ≥ 0 (default 10000); the counts stay exact |
show_reformatted, show_renamed |
boolean | also list values only written differently, and records whose gml:id changed (default false) |
time_budget_s |
number | 0 < t ≤ --timeout; covers reading both files and comparing them |
Exactly one of old_base64 and old_path, and one of new_base64 and new_path; the
two may come different ways. Each file is held to the limit of the way it came:
max_bytes inline, max_path_bytes by path. Inline, both travel in one request line, so
they must fit in max_line_bytes together: give paths for two large files. There is no
new limit.
The result is the document pydiggs.diff.compare(old, new, ...).to_json() gives for the
same bytes and options, each file under its basename, and the same bytes POST /v1/diff
answers. Validate it against its own $schema and tolerate unknown fields.
- Identical or not, the answer is a result: read
identicalandcomplete. - When
time_budget_sruns out, the result is still a document, withcomplete: falseand adiff.timeoutnote. - A file that is not a DIGGS document is the error
invalid-documentwith thekindofreadanddata.errorsnaming it ({"param": "old", "problem": "is not a DIGGS document"}); a file over its limit isinput-too-large. A file given by path that is refused names its parameter, never the path:{"param": "new_path", "problem": "is not permitted"}(path-not-permitted),"is larger than allowed"(input-too-large) or"is compressed"(compressed-input). A bad option isinvalid-paramsnaming it. A result overlimits.max_result_bytesisresult-too-large: list fewer changes withmax_changes. - A
diffshares the queue with the other document methods (one running, one waiting). A queued one can be cancelled; a running one is checked for cancellation only when it ends.$/progressreportsstep: "diff".
ping, versions¶
ping takes no parameters and returns {}. versions returns the same fields as
pydiggs.version_info().to_dict().
rules.list, rules.explain¶
rules.list {step?} returns every rule pydiggs can emit (RuleInfo objects: id,
pattern, step, description, severities, help_uri, known, guidance), optionally for one
step; open families appear as patterns. rules.explain {rule_id} (at most 256 characters)
describes one id; a well-formed unknown id returns known: false, a malformed one is
-32602.
shutdown, exit¶
See Shutdown and exit. shutdown takes no parameters and returns
null; exit is a notification.
Notifications¶
| method | direction | params |
|---|---|---|
exit |
host → sidecar | none |
$/cancelRequest |
host → sidecar | {id} |
$/progress |
sidecar → host, only if negotiated | {request_id, step, status} |
Other notifications, $/-prefixed or not, are ignored. $/progress
reports step: "queue", status: "waiting" when a validation or a read waits behind
another, step: "validate", status: "started" when a validation starts and
status: "running" at each step boundary (step: "read" for a read, step: "diff"
for a comparison). Treat it as a liveness signal; hosts must ignore step and status values they do
not know (later minors may name the individual checks).
Errors¶
Every error has error.code, a fixed error.message and error.data.code, a stable
slug. Switch on data.code. The -32000..-32099 band is pydiggs-defined: do not read
it with LSP or MCP meanings.
| code | data.code |
when | extra data |
|---|---|---|---|
| -32700 | parse-error |
invalid UTF-8, byte-order mark, invalid JSON (NaN/Infinity included), nesting deeper than 64 levels, integer too long | |
| -32600 | invalid-request |
not a request object, batch, missing/null/invalid id, line too long, duplicate in-flight id, second initialize |
max_line_bytes (line too long) |
| -32601 | method-not-found |
unknown method | |
| -32602 | invalid-params |
missing, unknown or malformed parameters | errors: [{param, problem}] |
| -32603 | internal |
unexpected failure; details go to stderr only | correlation_id |
| -32000 | busy |
one validation, read, conversion or check running and one waiting | |
| -32001 | not-initialized |
request before initialize completed |
|
| -32002 | shutting-down |
request after shutdown, or dropped by it |
|
| -32003 | input-too-large |
document over max_bytes / max_path_bytes |
max_bytes (content mode) |
| -32004 | compressed-input |
gzip-compressed document | |
| -32005 | path-not-permitted |
path mode off, refused path form, outside every root, missing, or not a regular file (all identical) | |
| -32006 | input-unreadable |
the permitted file could not be read | |
| -32007 | resource-error |
the bundled reference data could not be loaded | correlation_id |
| -32008 | cancelled |
cancelled by $/cancelRequest |
|
| -32009 | unsupported-protocol-version |
major version mismatch | supported |
| -32010 | invalid-document |
read, diff and convert to AGS4: the bytes are not a DIGGS document; convert to DIGGS: not AGS4 |
kind: malformed, not-diggs, unsafe-entity or namespace-too-long; for convert to DIGGS not-ags4 or unsupported-version; for diff, errors names the file (old or new) |
| -32011 | result-too-large |
read, convert, diff: the result is over max_result_bytes (for read, ask for fewer tables with select; for diff, list fewer changes with max_changes) |
max_result_bytes |
Errors for unparseable lines carry "id": null. A broken notification gets no answer.
Messages never echo request values; at most they name the reported basename.
Path access¶
Sending content is the default and works everywhere. By-path requests avoid base64 for
large local files but are off unless the host starts the sidecar with one or more
--allow-path DIR roots.
- Roots are resolved once at launch. A filesystem root (
/, a drive root, a UNC share root), the home folder itself or any of its ancestors is refused (exit status 2). - A request
pathmust be a plain absolute path. Refused: any URI (file:included), relative paths,..components, UNC paths,\\?\and\\.\prefixes, and Windows device names (CON,NUL,COM1, …) in any component, on every OS. - A path that, as spelled, is not under a root is refused without being opened. A
root may be spelled as given at launch or in its canonical form, so with
--allow-path /tmpon macOS both/tmp/a.xmland/private/tmp/a.xmlare accepted. - The sidecar then opens, then verifies: it opens the file without following a final
symlink (and without blocking on a FIFO), checks that the descriptor is a regular file,
asks the operating system for the descriptor's real path (
/proc/self/fdon Linux,F_GETPATHon macOS,GetFinalPathNameByHandleWon Windows) and accepts it only if that path lies inside a root. A symlink, or a file swapped between requests, cannot lead outside the roots. - Every refusal is the same
-32005, so a caller cannot probe which files exist.
Hard links are not contained
A hard link is not a pointer to another path: it is the file, under a second name.
A hard link inside a root to a file elsewhere therefore has a real path inside the
root, and the sidecar reads it. Anyone who can create files in a root can link any
file they can reach on the same volume into it. Do not give untrusted parties write
access to a root; point --allow-path at a folder only the host writes to.
Memory and size¶
With the default 25 MiB content limit a request line is about 34 MiB and the sidecar peaks at roughly 0.5 GiB. By-path documents of up to 100 MiB can peak at about 1.5 GiB on dense files. Plan for about 2 GiB per sidecar process and cap it from the host (see host guidance).
Example session¶
→ {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocol_version":"1.3","client_info":{"name":"example-host","version":"0.1"}}}
← {"jsonrpc":"2.0","id":1,"result":{"protocol_version":"1.3","pydiggs_version":"1.13.0",…}}
→ {"jsonrpc":"2.0","id":2,"method":"validate","params":{"content_base64":"PERpZ2dz…","name":"BH-01.xml"}}
→ {"jsonrpc":"2.0","id":3,"method":"ping"}
← {"jsonrpc":"2.0","id":3,"result":{}}
← {"jsonrpc":"2.0","id":2,"result":{"$schema":"https://xinp-hub.github.io/pydiggs/schemas/diggs-findings-1.2.schema.json","format":"diggs-findings",…}}
→ {"jsonrpc":"2.0","id":4,"method":"validate","params":{"path":"relative.xml"}}
← {"jsonrpc":"2.0","id":4,"error":{"code":-32005,"message":"The path is not permitted.","data":{"code":"path-not-permitted"}}}
→ {"jsonrpc":"2.0","id":5,"method":"read","params":{"content_base64":"PERpZ2dz…","name":"BH-01.xml","select":{"results":"none"}}}
← {"jsonrpc":"2.0","id":5,"result":{"$schema":"https://xinp-hub.github.io/pydiggs/schemas/diggs-model-1.0.schema.json","schema_version":"1.0",…}}
→ {"jsonrpc":"2.0","id":6,"method":"shutdown"}
← {"jsonrpc":"2.0","id":6,"result":null}
→ {"jsonrpc":"2.0","method":"exit"}