Skip to content

Stdio sidecar protocol

pydiggs serve --stdio runs pydiggs as a sidecar: a child process that any host application can start and talk to over its standard input and output. It suits desktop and native applications in any language (Rust, C#, Java, Swift, C++, TypeScript on Node or Electron, and so on) that cannot embed CPython. This page is the normative specification of protocol version 1.3 (1.0 plus the read method in 1.1, convert and ags4.check in 1.2, and diff in 1.3). Host guidance covers supervision, packaging and example clients.

The sidecar is part of the core package: it needs no extra and uses only the Python standard library on top of pydiggs itself. A machine-readable description of the methods ships with the package as pydiggs/sidecar/openrpc.json (OpenRPC 1.3) and is published as schemas/openrpc.json.

Findings are untrusted text

Every string field in a findings document (messages, paths, element names, values) can carry text taken from the validated file. Escape it before rendering it in HTML, a rich-text control, a terminal or a log.

Launching

pydiggs serve --stdio [--max-bytes N] [--max-findings N] [--timeout S]
                      [--allow-path DIR]... [--log-level L] [--log-format text|json]
option default meaning
--max-bytes 26,214,400 (25 MiB) largest decoded document sent as content, and by path when given
--max-findings 10,000 findings kept per document (a truncation finding marks the cut)
--timeout 60 default and largest per-request time budget, in seconds
--allow-path DIR none enables by-path requests for files under DIR (repeatable)
--log-level warning debug, info, warning, error, critical
--log-format text json writes one object per line with ts, level, msg, request_id

By-path documents may be up to 100 MiB, or --max-bytes when it is given. python -m pydiggs serve --stdio is equivalent. The sidecar only ever uses pydiggs's bundled schemas, dictionaries and Schematron rules; it never reaches the network and never reads local dictionaries next to a document.

Exit status

status when
0 exit after shutdown; stdin closed (EOF); stdout closed (broken pipe); SIGTERM, SIGINT or Ctrl+Break
1 exit without a prior shutdown, or an unexpected internal failure
2 bad launch arguments, including a refused --allow-path root or unusable stdio
3 the bundled reference data could not be loaded during initialize

Framing

  • One JSON-RPC 2.0 message per line, UTF-8, terminated by \n (LF). Input lines may end in \r\n; blank lines are skipped.
  • Split on byte 0x0A only. Responses never contain a raw LF or CR, and U+0085, U+2028 and U+2029 are always written as \u escapes, so line splitters that also break on those characters still see one message per line.
  • Input is decoded as strict UTF-8 whatever the locale. A byte-order mark is an error.
  • A line may be at most max_line_bytes long (reported by initialize): by default ceil(4/3 × max_bytes) + 65,536, about 34 MiB. A longer line is discarded up to the next LF and answered with -32600 and "id": null.
  • Responses can be several megabytes on one line (up to max_findings findings). Read them with a streaming line reader that has no fixed buffer size.
  • Batches (JSON arrays) are not supported (-32600). Parameters are always an object (by name).
  • stdout carries only frames. At start the sidecar keeps a private copy of descriptor 1 for frames and points descriptor 1 at stderr, so stray output from Python, C libraries or child processes lands on stderr.

stderr

Logs go to stderr. The host must drain stderr or redirect it to a file or the null device: a full stderr pipe (often 64 KiB) blocks the sidecar. Logs never contain document content or absolute paths, and client strings are escaped.

Lifecycle

  1. The host starts the process and sends initialize. Until its response arrives, the sidecar reads nothing more, so requests may be pipelined behind it. Requests sent before initialize (on a fresh process) are answered not-initialized; notifications are ignored.
  2. initialize compiles the validators for the requested DIGGS versions (all by default; about a second cold), then answers. If this fails the answer is resource-error and the process exits with status 3.
  3. The host sends requests. Each response carries the request's id; responses can arrive in a different order from the requests (see Ordering).
  4. To stop: send shutdown, wait for its null result, then send the exit notification. Closing stdin also stops the sidecar (status 0) and abandons any running validation.

Version negotiation

initialize.params.protocol_version is "MAJOR.MINOR". The major must equal the sidecar's (1), otherwise the answer is unsupported-protocol-version with data.supported, e.g. ["1.0", "1.1", "1.2", "1.3"]. The sidecar answers the highest minor it supports that does not exceed the requested one. Minor versions only add: hosts must ignore unknown result fields; new methods are advertised in capabilities. The sidecar refuses unknown parameters with -32602 (except unknown keys inside capabilities, which it ignores).

Ordering

  • ping, versions, rules.list and rules.explain are answered immediately, even while a validation runs.
  • Validations run one at a time. At most one more may wait behind the running one; a further validate is answered busy at once. A host that needs throughput runs a pool of sidecar processes.
  • Request ids must be unique among in-flight validations; reusing one is -32600.

Cancellation and time limits

  • $/cancelRequest {id}: a waiting validation is removed and answered cancelled at once. The running one ends cancelled at its next step boundary. Unknown or finished ids are ignored.
  • time_budget_s (default and maximum: --timeout) is checked at the same step boundaries. Running out is not an error: the result is a normal findings document with a structure.timeout ERROR finding and the unfinished steps marked incomplete.
  • Neither can interrupt a single schema or Schematron pass already running. For a hard limit, the host must kill the process (see host guidance) and show the file as "not checked", never as passed.

Shutdown and exit

shutdown lets the running validation finish, answers shutting-down to the waiting one and to every later request, and then answers null (after the running validation's own response). exit then ends the process with status 0; exit without shutdown ends it with status 1. exit is honoured at any time and abandons any running work.

Methods

initialize

param type notes
protocol_version string, required "1.3" (or "1.2": no 1.3 members, and diff is method-not-found; or "1.1": no 1.2 members either, and convert and ags4.check are method-not-found too; or "1.0": no 1.1 members either, and read is method-not-found too)
client_info object {name ≤ 128 chars, version ≤ 64 chars}, logged at info
capabilities object {progress: bool} asks for $/progress notifications
diggs_versions string[] DIGGS versions to warm up (default: every released bundled version, not the experimental 3.1-dev)

Result:

{
  "protocol_version": "1.3",
  "pydiggs_version": "1.13.0",
  "findings_schema_version": "1.2",
  "findings_schema_url": "https://xinp-hub.github.io/pydiggs/schemas/diggs-findings-1.2.schema.json",
  "model_schema_version": "1.0",
  "model_schema_url": "https://xinp-hub.github.io/pydiggs/schemas/diggs-model-1.0.schema.json",
  "diff_schema_version": "1.0",
  "diff_schema_url": "https://xinp-hub.github.io/pydiggs/schemas/diggs-diff-1.0.schema.json",
  "diggs_versions": ["2.5.a", "2.6", "3.0.0", "3.1-dev"],
  "reference_data": {"def": "…", "schema-dev": "…", "schemas": "…"},
  "capabilities": {
    "validate": {"content": true, "path": false},
    "read": {"content": true, "path": false, "select": true},
    "convert": {"content": true, "path": false, "to": ["diggs", "ags4"], "ags_versions": ["4.2", "4.1.1"], "residual": true},
    "ags4_check": {"content": true, "path": false, "dictionary": true},
    "diff": {"content": true, "path": false, "match": ["auto", "gml-id", "natural", "gml_id"]},
    "cancel": true, "progress": false, "rules": true, "ping": true
  },
  "limits": {
    "max_bytes": 26214400, "max_path_bytes": 104857600, "max_line_bytes": 35018070,
    "max_findings": 10000, "max_response_bytes_estimate": 20545536,
    "max_result_bytes": 268435456, "time_budget_s": 60.0
  }
}

capabilities.validate.path and capabilities.read.path are true only when the sidecar was started with --allow-path. max_response_bytes_estimate is a rough upper bound for sizing buffers for validate, not a guarantee; max_result_bytes is the hard limit for a read result. A host that negotiates "1.0" gets the 1.0 answer: no model_schema_*, no capabilities.read and no max_result_bytes; one that negotiates "1.1" gets no capabilities.convert and no capabilities.ags4_check; one that negotiates "1.2" gets no diff_schema_* and no capabilities.diff.

validate

param type notes
content_base64 string the document, standard base64 with padding and no whitespace
path string an absolute path under an --allow-path root (instead of content)
name string ≤ 4096 the name to report; reduced to a basename (default input.xml, or the path's basename)
checks string[] steps to run (default: all), as for pydiggs validate --checks
diggs_version string force a DIGGS version instead of detecting it
time_budget_s number 0 < t ≤ --timeout

Exactly one of content_base64 and path is required. The encoded length is checked against max_bytes before decoding.

The result is a complete one-file diggs-findings 1.2 document, the same as pydiggs validate --format json gives for that file apart from files[].path (always the basename here) and with no absolute_path. Validate it against the document's own $schema and tolerate unknown fields. A malformed or invalid document is a normal result with findings, not an error.

read

Protocol 1.1. Reads one document into a diggs-model 1.0 document: what is in the file, as pydiggs.read() returns it.

param type notes
content_base64 string as for validate
path string as for validate
name string ≤ 4096 as for validate
diggs_version string force a DIGGS version instead of detecting it
time_budget_s number 0 < t ≤ --timeout
select object {"results": "all" \| "none" \| ["key", …]}: which result tables keep their values (default all; at most 1,000 keys)

The result is the document pydiggs.read_bytes(data, name=name).to_json() gives for the same bytes. Validate it against its own $schema and tolerate unknown fields.

  • select: a table that is left out keeps its columns and row_count with empty values. List a file with {"results": "none"}, then ask for the measurements you need by key; the sidecar keeps the last few models it read (by content hash), so the second call does not parse the file again. It is a cache, not state: there are no handles.
  • A schema-invalid file still reads; run validate for a verdict.
  • When time_budget_s runs out the result is still a document, with complete: false and the members read so far.
  • A read shares the queue with validate (one running, one waiting) and can be cancelled the same way: cancellation and the time budget are checked between top-level members and every so often inside a large one.
  • The reader's work limits (Reading files) apply; reaching one is a result with complete: false, not an error.
  • A select key that matches no measurement is ignored.
  • Bytes that are not a DIGGS document are the error invalid-document; a result over limits.max_result_bytes is result-too-large (ask for fewer tables with select).

convert

Protocol 1.2. Converts one AGS4 file to DIGGS 3.0.0 (to: "diggs", the default), or one DIGGS document to an AGS4 file (to: "ags4"), by the stable pydiggs.convert and mapping table 1.2 (see AGS4 and DIGGS conversion), with every loss reported. A host detects to: "ags4" and ags_version from capabilities.convert (its to and ags_versions lists); a 1.10 sidecar also reports protocol 1.2 and answers both with invalid-params.

param type notes
content_base64 string the AGS4 file (or the DIGGS document for to: "ags4"), as for validate
path string as for validate
name string ≤ 4096 the name to report; reduced to a basename (default input.ags, input.xml for to: "ags4", or the path's basename)
to string the format to convert to: "diggs" (the default; the content is AGS4) or "ags4" (the content is DIGGS XML)
diggs_version string to: "diggs" only: the DIGGS version to write, "3.0.0" (the default)
ags_version string to: "ags4" only: the AGS4 version to write, "4.2" (the default) or "4.1.1"; as of November 2026 the BGS AGS validator and python-ags4 1.2.0 accept AGS4 only up to 4.1.1
time_budget_s number 0 < t ≤ --timeout

The result is the conversion result that POST /v1/convert answers, byte for byte: the output bytes as output_base64 with media_type, output_name and output_sha256, the report about the source and the validation of the output (both diggs-findings 1.2; the output's is pydiggs.ags4.check for AGS4), diggs_version, ags_version, mapping_version, complete and lossless, and residual_base64: the residual AGS4 file of what an AGS4 to DIGGS conversion did not convert, or null (the fields are in AGS4 and DIGGS conversion). capabilities.convert lists the targets, the AGS4 versions written and residual: true.

  • Losses, an output that fails validation or the AGS4 check, and an exhausted time_budget_s (complete: false) are normal results.
  • Bytes that are not AGS4 (for to: "diggs") are the error invalid-document with kind not-ags4 (or unsupported-version for AGS 3); bytes that are not DIGGS (for to: "ags4") are invalid-document with the kind of read. A parameter given for the other direction is invalid-params; a result over limits.max_result_bytes is result-too-large.
  • A convert shares the queue with validate and read. A queued one can be cancelled; a running one is checked for cancellation only when it ends.

ags4.check

Protocol 1.2. Checks one AGS4 file against the AGS4 rules (Rules 1 to 20) and answers with the diggs-findings 1.2 document ags4.check gives for the same bytes, line numbers included (see AGS4 files).

param type notes
content_base64, path, name, time_budget_s as for convert
dictionary_base64 string an AGS4 data dictionary (pyDIGGS bundles none), standard base64
dictionary_path string the dictionary by path, under an --allow-path root (vetted like path)

At most one of dictionary_base64 and dictionary_path. Each file is held to the limit of the way it came: max_bytes inline (base64), max_path_bytes by path. Inline, both travel in one request line, so they must fit in max_line_bytes together. A dictionary that cannot be used is invalid-params naming dictionary. time_budget_s covers reading the dictionary and checking together: a dictionary read it stops is invalid-params naming dictionary ("could not be read within the time budget"). Rule breaks and a file that is not AGS4 at all are normal results with findings. The sidecar writes the file to a private temporary folder (mode 0700) for the checker and removes it at once.

diff

Protocol 1.3. Compares two DIGGS files with the stable pydiggs.diff.compare() and answers with its diggs-diff 1.0 document: what was added, removed or changed. A session that negotiated 1.2 or older gets method-not-found.

param type notes
old_base64, new_base64 string the earlier and the later file, standard base64 as for validate
old_path, new_path string a file by path, under an --allow-path root (vetted like path), instead of its content
old_name, new_name string ≤ 4096 the names to report; reduced to a basename (default input.xml, or the path's basename)
match string how records are paired: "auto" (the default), "gml-id" (also "gml_id") or "natural"
rel_tol number two numbers are equal within this fraction of the larger one: 0 ≤ r < 1, default 1e-9
abs_tol number or within this much, in the value's own unit: ≥ 0, default 0
max_changes integer most changes listed, ≥ 0 (default 10000); the counts stay exact
show_reformatted, show_renamed boolean also list values only written differently, and records whose gml:id changed (default false)
time_budget_s number 0 < t ≤ --timeout; covers reading both files and comparing them

Exactly one of old_base64 and old_path, and one of new_base64 and new_path; the two may come different ways. Each file is held to the limit of the way it came: max_bytes inline, max_path_bytes by path. Inline, both travel in one request line, so they must fit in max_line_bytes together: give paths for two large files. There is no new limit.

The result is the document pydiggs.diff.compare(old, new, ...).to_json() gives for the same bytes and options, each file under its basename, and the same bytes POST /v1/diff answers. Validate it against its own $schema and tolerate unknown fields.

  • Identical or not, the answer is a result: read identical and complete.
  • When time_budget_s runs out, the result is still a document, with complete: false and a diff.timeout note.
  • A file that is not a DIGGS document is the error invalid-document with the kind of read and data.errors naming it ({"param": "old", "problem": "is not a DIGGS document"}); a file over its limit is input-too-large. A file given by path that is refused names its parameter, never the path: {"param": "new_path", "problem": "is not permitted"} (path-not-permitted), "is larger than allowed" (input-too-large) or "is compressed" (compressed-input). A bad option is invalid-params naming it. A result over limits.max_result_bytes is result-too-large: list fewer changes with max_changes.
  • A diff shares the queue with the other document methods (one running, one waiting). A queued one can be cancelled; a running one is checked for cancellation only when it ends. $/progress reports step: "diff".

ping, versions

ping takes no parameters and returns {}. versions returns the same fields as pydiggs.version_info().to_dict().

rules.list, rules.explain

rules.list {step?} returns every rule pydiggs can emit (RuleInfo objects: id, pattern, step, description, severities, help_uri, known, guidance), optionally for one step; open families appear as patterns. rules.explain {rule_id} (at most 256 characters) describes one id; a well-formed unknown id returns known: false, a malformed one is -32602.

shutdown, exit

See Shutdown and exit. shutdown takes no parameters and returns null; exit is a notification.

Notifications

method direction params
exit host → sidecar none
$/cancelRequest host → sidecar {id}
$/progress sidecar → host, only if negotiated {request_id, step, status}

Other notifications, $/-prefixed or not, are ignored. $/progress reports step: "queue", status: "waiting" when a validation or a read waits behind another, step: "validate", status: "started" when a validation starts and status: "running" at each step boundary (step: "read" for a read, step: "diff" for a comparison). Treat it as a liveness signal; hosts must ignore step and status values they do not know (later minors may name the individual checks).

Errors

Every error has error.code, a fixed error.message and error.data.code, a stable slug. Switch on data.code. The -32000..-32099 band is pydiggs-defined: do not read it with LSP or MCP meanings.

code data.code when extra data
-32700 parse-error invalid UTF-8, byte-order mark, invalid JSON (NaN/Infinity included), nesting deeper than 64 levels, integer too long
-32600 invalid-request not a request object, batch, missing/null/invalid id, line too long, duplicate in-flight id, second initialize max_line_bytes (line too long)
-32601 method-not-found unknown method
-32602 invalid-params missing, unknown or malformed parameters errors: [{param, problem}]
-32603 internal unexpected failure; details go to stderr only correlation_id
-32000 busy one validation, read, conversion or check running and one waiting
-32001 not-initialized request before initialize completed
-32002 shutting-down request after shutdown, or dropped by it
-32003 input-too-large document over max_bytes / max_path_bytes max_bytes (content mode)
-32004 compressed-input gzip-compressed document
-32005 path-not-permitted path mode off, refused path form, outside every root, missing, or not a regular file (all identical)
-32006 input-unreadable the permitted file could not be read
-32007 resource-error the bundled reference data could not be loaded correlation_id
-32008 cancelled cancelled by $/cancelRequest
-32009 unsupported-protocol-version major version mismatch supported
-32010 invalid-document read, diff and convert to AGS4: the bytes are not a DIGGS document; convert to DIGGS: not AGS4 kind: malformed, not-diggs, unsafe-entity or namespace-too-long; for convert to DIGGS not-ags4 or unsupported-version; for diff, errors names the file (old or new)
-32011 result-too-large read, convert, diff: the result is over max_result_bytes (for read, ask for fewer tables with select; for diff, list fewer changes with max_changes) max_result_bytes

Errors for unparseable lines carry "id": null. A broken notification gets no answer. Messages never echo request values; at most they name the reported basename.

Path access

Sending content is the default and works everywhere. By-path requests avoid base64 for large local files but are off unless the host starts the sidecar with one or more --allow-path DIR roots.

  • Roots are resolved once at launch. A filesystem root (/, a drive root, a UNC share root), the home folder itself or any of its ancestors is refused (exit status 2).
  • A request path must be a plain absolute path. Refused: any URI (file: included), relative paths, .. components, UNC paths, \\?\ and \\.\ prefixes, and Windows device names (CON, NUL, COM1, …) in any component, on every OS.
  • A path that, as spelled, is not under a root is refused without being opened. A root may be spelled as given at launch or in its canonical form, so with --allow-path /tmp on macOS both /tmp/a.xml and /private/tmp/a.xml are accepted.
  • The sidecar then opens, then verifies: it opens the file without following a final symlink (and without blocking on a FIFO), checks that the descriptor is a regular file, asks the operating system for the descriptor's real path (/proc/self/fd on Linux, F_GETPATH on macOS, GetFinalPathNameByHandleW on Windows) and accepts it only if that path lies inside a root. A symlink, or a file swapped between requests, cannot lead outside the roots.
  • Every refusal is the same -32005, so a caller cannot probe which files exist.

Hard links are not contained

A hard link is not a pointer to another path: it is the file, under a second name. A hard link inside a root to a file elsewhere therefore has a real path inside the root, and the sidecar reads it. Anyone who can create files in a root can link any file they can reach on the same volume into it. Do not give untrusted parties write access to a root; point --allow-path at a folder only the host writes to.

Memory and size

With the default 25 MiB content limit a request line is about 34 MiB and the sidecar peaks at roughly 0.5 GiB. By-path documents of up to 100 MiB can peak at about 1.5 GiB on dense files. Plan for about 2 GiB per sidecar process and cap it from the host (see host guidance).

Example session

→ {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocol_version":"1.3","client_info":{"name":"example-host","version":"0.1"}}}
← {"jsonrpc":"2.0","id":1,"result":{"protocol_version":"1.3","pydiggs_version":"1.13.0",…}}
→ {"jsonrpc":"2.0","id":2,"method":"validate","params":{"content_base64":"PERpZ2dz…","name":"BH-01.xml"}}
→ {"jsonrpc":"2.0","id":3,"method":"ping"}
← {"jsonrpc":"2.0","id":3,"result":{}}
← {"jsonrpc":"2.0","id":2,"result":{"$schema":"https://xinp-hub.github.io/pydiggs/schemas/diggs-findings-1.2.schema.json","format":"diggs-findings",…}}
→ {"jsonrpc":"2.0","id":4,"method":"validate","params":{"path":"relative.xml"}}
← {"jsonrpc":"2.0","id":4,"error":{"code":-32005,"message":"The path is not permitted.","data":{"code":"path-not-permitted"}}}
→ {"jsonrpc":"2.0","id":5,"method":"read","params":{"content_base64":"PERpZ2dz…","name":"BH-01.xml","select":{"results":"none"}}}
← {"jsonrpc":"2.0","id":5,"result":{"$schema":"https://xinp-hub.github.io/pydiggs/schemas/diggs-model-1.0.schema.json","schema_version":"1.0",…}}
→ {"jsonrpc":"2.0","id":6,"method":"shutdown"}
← {"jsonrpc":"2.0","id":6,"result":null}
→ {"jsonrpc":"2.0","method":"exit"}