HTTP API¶
pydiggs[server] serves pydiggs over HTTP: upload a DIGGS document, get a
diggs-findings 1.2 document (or a SARIF 2.1.0 log) back. It is a FastAPI
application you can run as is, or a router you mount in your own FastAPI app.
pip install 'pydiggs[server]'
pydiggs serve --http # http://127.0.0.1:8000
curl -s -H 'Content-Type: application/xml' --data-binary @BH-01.xml \
'http://127.0.0.1:8000/v1/validate?name=BH-01.xml'
Every string in a findings document is untrusted
Finding messages, paths and identifiers come from the uploaded file. Escape every string field before you render it in HTML or a terminal.
pydiggs adds no authentication, rate limiting per client or TLS. Put it behind your own app, a reverse proxy or an API gateway for that.
Behind a reverse proxy when not on loopback
Any deployment reachable from other machines must sit behind a reverse proxy (nginx,
Caddy, HAProxy, a cloud load balancer) with client header and body timeouts.
uvicorn has no timeout for reading request headers: a client that opens connections
and sends partial headers holds them, and enough of them exhaust uvicorn's
limit_concurrency (the admission slots plus 4), so the server stops accepting
work. pydiggs bounds only what happens after the headers: the body must arrive
within body_timeout_s (408), and validation within timeout_s (504).
Running it¶
--max-bytes, --max-findings, --timeout and --workers override the
PYDIGGS_* variables below. Bound to a non-loopback address, the server refuses to
start until PYDIGGS_ALLOWED_HOSTS lists the host names clients use.
from fastapi import FastAPI
from pydiggs.server import ServerSettings, create_router
app = FastAPI()
app.include_router(create_router(ServerSettings(isolation="process")))
The router enforces its own limits (body size, admission, multipart, cross-site
refusal), so they hold without create_app. It adds no Host check and no CORS
headers: those are your app's job. Its default isolation is "thread"; pass
isolation="process" for untrusted uploads.
Endpoints¶
| method and path | answer |
|---|---|
POST /v1/validate |
200 with a diggs-findings 1.2 document, or SARIF 2.1.0 for Accept: application/sarif+json |
POST /v1/read |
200 with a diggs-model 1.0 document: what is in the file (Reading files) |
POST /v1/convert |
200 with a conversion result: an AGS4 file as DIGGS 3.0.0, or a DIGGS document as AGS4 (base64), and every loss (AGS4 and DIGGS conversion) |
POST /v1/ags4/check |
200 with a diggs-findings 1.2 document of the AGS4 rules, or SARIF 2.1.0 (AGS4 files) |
POST /v1/diff |
200 with a diggs-diff 1.0 document: what was added, removed or changed between two DIGGS files (Comparing two files) |
GET /v1/versions |
pydiggs, findings schema and DIGGS versions, reference-data commits, api_version: "1" (off with versions_enabled=False) |
GET /v1/capabilities |
a diggs-capabilities 1.0 document: what this server's pydiggs can do (off with versions_enabled=False) |
GET /v1/rules?step= |
every rule id pydiggs can emit (open families as patterns) |
GET /v1/rules/{rule_id} |
one rule; a valid id pydiggs does not emit is 200 with known: false |
GET /v1/readyz |
200 once the workers are warm, else 503 not-ready |
GET /livez |
200 while the server runs (app only; use it for container health checks) |
GET /readyz |
as /v1/readyz (app only) |
POST /v1/validate¶
Send the document either way:
- Raw:
Content-Type: application/xmlortext/xml, the document as the body, and the query parametersname,checks,diggs_version. - Multipart:
Content-Type: multipart/form-datawith one part namedfileand the optional fieldsname,checks,diggs_version. A field wins over the same query parameter, but both must be valid. An overridden query value is checked in the request the fields complete, and only for itself: on/v1/convert,?to=ags4&ags_version=4.2with the fieldags_version=4.1.1, and?to=diggswith the fieldsto=ags4andags_version=4.2, are accepted;?to=boguswith the fieldto=diggsis refused (422).namewins over the part's file name. Other fields are refused.
| parameter | meaning |
|---|---|
name |
File name to report; reduced to a basename, input.xml when empty |
checks |
Comma-separated steps (repeatable): structure, schema, dictionary, schematron, geometry, deprecation, context; default all |
diggs_version |
2.5.a, 2.6 or 3.0.0; default detected. 3.1-dev selects the unreleased DIGGS 3.1 draft (experimental) |
curl -s -F file=@BH-01.xml -F checks=structure,schema \
-H 'Accept: application/sarif+json' http://127.0.0.1:8000/v1/validate
Malformed XML, findings past max_findings (the structure.findings-truncated finding)
and an exhausted time budget (structure.timeout) are all 200 with findings, as with
pydiggs validate --format json. Only a request that produced no document is an error.
Accept decides the format: application/json (the default, also for */* and no
header) or application/sarif+json. When both are listed with the same quality, JSON
wins. An Accept that allows neither is 406.
POST /v1/read¶
Reads the document into a diggs-model 1.0 document: boreholes, samples, tests and
their result tables, as pydiggs.read() returns them. The body is exactly
pydiggs.read_bytes(data, name=name).to_json(), so every surface gives the same bytes
for the same file. Send the document as for /v1/validate (raw XML or one multipart
file part); the body limit, admission, isolation and time limit are the same.
| parameter | meaning |
|---|---|
name |
File name to report; reduced to a basename, input.xml when empty |
diggs_version |
As for /v1/validate; default detected |
select |
JSON text: {"results": "all"} (the default), {"results": "none"} or {"results": ["key", …]} (at most 1,000 node keys) |
selectsays which result tables keep their values. A table that is left out keeps its columns androw_countwith emptyvalues. List a file first with{"results": "none"}, then ask for the measurements you need bykey. Each worker keeps the last few models it read (by content hash), so the second call does not parse the file again. That is a cache only: nothing is stored between requests that a caller could address.- Reading is not validating. A schema-invalid file still reads with 200; run
/v1/validatefor a verdict. What reading had to work around is in the document'snotes. - Not a DIGGS document is 422
invalid-document(title "Invalid document"), withkindmalformed,not-diggs,unsafe-entityornamespace-too-long. (/v1/validatereports the same cases as findings in a 200.) - Size. The reader's own work limits (Reading files)
apply, and a document larger than four times
max_body_bytesis 422result-too-large; ask for fewer tables withselect. - Time limit: with thread isolation, running out of time is a 200 with
complete: falseand the members read so far; with process isolation it is 504timeout. - The answer is always
application/json. On the conformance corpus a whole model is 0.6 to 2 times the size of its XML (1.1 times overall); compressing responses is the host's job.
POST /v1/convert¶
Converts one AGS4 file to DIGGS 3.0.0 (to=diggs, the default), or one DIGGS
document to an AGS4 file (to=ags4), by the stable pydiggs.convert and mapping
table 1.2 (see AGS4 and DIGGS conversion). The body limit, admission,
isolation and time limit are those of /v1/read.
- Raw: the file as the body and the parameters in the query. For
to=diggsthe body is AGS4,Content-Type: application/octet-stream,text/plainortext/csv(AGS4 has no registered media type); forto=ags4(given in the query) it is DIGGS XML,application/xmlortext/xml. Another media type is 415. - Multipart: one part named
fileand the optional fields below; as for/v1/validate, a field wins over the same query parameter and both must be valid.
| parameter | meaning |
|---|---|
name |
File name to report; reduced to a basename, input.ags (input.xml for to=ags4) when empty |
to |
The format to convert to: diggs (the default; the body is AGS4) or ags4 (the body is DIGGS XML) |
diggs_version |
to=diggs only: the DIGGS version to write, 3.0.0 (the default) |
ags_version |
to=ags4 only: the AGS4 version to write, 4.2 (the default) or 4.1.1. As of November 2026 the BGS AGS validator and python-ags4 1.2.0 accept AGS4 only up to 4.1.1 |
A parameter given for the other direction is 422 invalid-parameter.
curl -s -F file=@site.ags http://127.0.0.1:8000/v1/convert > result.json
jq -r .output_base64 result.json | base64 --decode > site.xml
jq -r '.residual_base64 // empty' result.json | base64 --decode > left.ags
jq '.report.files[0].findings[] | select(.step == "convert")' result.json
curl -s 'http://127.0.0.1:8000/v1/convert?to=ags4&ags_version=4.1.1' \
-H 'Content-Type: application/xml' --data-binary @site.xml \
| jq -r .output_base64 | base64 --decode > site.ags
The answer is one JSON object (application/json, component ConversionResult): the
output bytes as output_base64 with media_type (application/xml for DIGGS,
text/plain; charset=utf-8 for AGS4), output_name and output_sha256; the report
about the source (the reader's notes and convert findings, diggs-findings 1.2); the
validation of the output (pydiggs.validate for DIGGS, pydiggs.ags4.check for
AGS4); diggs_version, ags_version (null for to=diggs), mapping_version,
complete and lossless; and residual_base64, the residual AGS4 file of what an AGS4
to DIGGS conversion did not convert (null when nothing was left over, and for
to=ags4). The fields are described in AGS4 and DIGGS conversion
and the residual file in The residual file.
- Why one JSON object. A client gets the document and its findings in one call, with
one parser, and can check
output_sha256against the bytes it decodes. OpenAPI 3.1 describes base64 inside JSON directly (contentEncoding: base64withcontentMediaType), so generated clients type it; multipart responses are poorly supported by HTTP clients and code generators, and putting the report in a header would hit header size limits. The cost is a third more bytes for the output; compress responses at the host if that matters. - Losses are not errors. Unmapped content, an output that fails validation or the
AGS4 check, and an exhausted time budget (
complete: false, with thread isolation) are 200s; the findings say what happened. - Not AGS4 (
to=diggs) is 422invalid-documentwithkindnot-ags4, orunsupported-versionfor an AGS 3 file; not DIGGS (to=ags4) is 422invalid-documentwith thekindof/v1/read. Compressed input (gzip, zip) is 415. - Size. A result larger than four times
max_body_bytesis 422result-too-large.
POST /v1/ags4/check¶
Checks one AGS4 file against the AGS4 rules (Rules 1 to 20) and answers with a
diggs-findings 1.2 document of step ags4 (or SARIF 2.1.0 for
Accept: application/sarif+json), the report ags4.check gives for
the same bytes. Send the file raw (as for /v1/convert) or as multipart with a file
part and, optionally, a dictionary part: an AGS4 data dictionary, which pyDIGGS does
not bundle (see The data dictionary). The only parameter is
name.
curl -s -F file=@site.ags -F dictionary=@Standard_dictionary_v4_1_1.ags \
http://127.0.0.1:8000/v1/ags4/check
Rule breaks, a file that is not AGS4 at all, truncation and an exhausted time budget are
200s with findings, as for /v1/validate. A dictionary that cannot be used is 422
invalid-parameter naming dictionary. The file and the dictionary share the body
limit. The server writes the file to a private temporary folder (mode 0700) for the
checker and removes it at once; with the container's read-only root filesystem that is
the /tmp tmpfs.
POST /v1/diff¶
Compares two DIGGS files with the stable pydiggs.diff.compare() and answers
with its diggs-diff 1.0 document. The body is exactly what
compare(old, new, ...).to_json() gives for the same bytes and options, with each file
reported under its basename, so every surface gives the same bytes for the same pair.
From pyDIGGS 1.13.
Send both files as multipart (Content-Type: multipart/form-data): one part named
old (the earlier file) and one named new (the later), each exactly once, and the
optional fields below. A raw body is 415: there are two files. As for /v1/validate,
a field wins over the same query parameter, and both must be valid.
| parameter | meaning |
|---|---|
old_name, new_name |
File names to report; reduced to a basename, the part's file name when not given, input.xml when empty |
match |
How records are paired: auto (the default; by gml:id, then by natural key), gml-id (also written gml_id) or natural |
rel_tol |
Two numbers are equal within this fraction of the larger one: from 0 to less than 1, default 1e-9 |
abs_tol |
Or within this much, in the value's own unit: 0 or more, default 0 |
max_changes |
Most changes listed, 0 or more (default 10000); the counts stay exact, and a cut list is complete: false |
show_reformatted |
true to also list values only written differently (default false) |
show_renamed |
true to also list records whose gml:id changed (default false) |
curl -s -F old=@BH-01-v1.xml -F new=@BH-01-v2.xml -F max_changes=100 \
http://127.0.0.1:8000/v1/diff | jq '{identical, summary, value_changes}'
- A difference is not an error. Identical or not, the answer is 200; read
identicalandcomplete. - Limits are those of
/v1/read, per file. Each file may be up tomax_body_bytes, so the body may carry twice that (plus the multipart allowance); a larger file is 413. Both files are read and compared in one job in one worker, under the onetimeout_sand the one admission slot: no new limit or setting. A result larger than four timesmax_body_bytesis 422result-too-large; list fewer changes withmax_changes. - Not a DIGGS document is 422
invalid-documentwith thekindof/v1/read, anderrorsnaming the file:[{"parameter": "new", "problem": "is not a DIGGS document"}]. (The CLI exits 2 for the same input.) - Time limit: with thread isolation, running out of time is a 200 with
complete: falseand adiff.timeoutnote, andoptions.time_budgetis the server'stimeout_s; with process isolation it is 504timeoutandoptions.time_budgetis null. - The two files are compared in memory; nothing is written to disk. Every string in the document (names, keys, values) comes from the files: escape it before you render it.
Limits and admission¶
Checks run in this order, before the body is read where possible:
- Cross-site refusal (403).
Accept(406), then a query parameter the endpoint does not take (422invalid-parameter, see Unknown query parameters), then media type (415),Content-Encodingother thanidentity(415).Content-Lengthabove the limit: 413 at once, body not read.- Admission: a slot among
workers + queue_depthis taken without waiting; when none is free the answer is 503busywithRetry-After: 1, body not read. - The body is streamed with a running byte count and abandoned at the limit (413), so
chunked bodies are bounded too. Multipart allows one file (two for
/v1/ags4/checkand/v1/diff) and at most 8 fields of 64 KiB; each/v1/difffile counts againstmax_body_byteson its own, any other file against it together; spooled parts are deleted. - The whole body must arrive within
body_timeout_s(30 s) of admission; otherwise the answer is 408request-timeoutwithConnection: close, and the slot is freed.
Gzip-compressed documents are refused (415 compressed-input).
These limits live in the route, so they also hold when only the router is mounted.
Isolation¶
isolation |
how a request runs | time limit |
|---|---|---|
"process" (default for create_app and serve --http) |
in one of workers spawned processes |
hard: past timeout_s the worker is killed and replaced, 504 timeout |
"thread" (default for create_router) |
in one of workers threads |
cooperative only: a long libxml2 call holds its thread; unsafe for untrusted input |
Process workers warm every bundled DIGGS version before they report ready. On Linux each
worker runs under an address-space cap (worker_memory_mb, 2 GiB) and a CPU-time
backstop; macOS enforces only the CPU backstop and Windows neither, so use a container or
host limits there. Workers send results back as JSON bytes, never pickles. create_app
logs a warning when it is given isolation="thread".
Size the host for ASGI server processes × pydiggs workers × about 2 GiB.
Host, cross-site and CORS¶
create_apprefuses anyHostnot inallowed_hostswith 400invalid-host(DNS-rebinding protection). Unset, it allows127.0.0.1,localhostand[::1].- Every route refuses a request with
Sec-Fetch-Site: cross-site, or with anOriginwhose host differs fromHost(nullincluded), unless the origin is incors_origins(403cross-site; turn off withreject_cross_site=False). create_appsends CORS headers only to the origins incors_origins;"*"is refused./docs,/redocand/openapi.jsonare off unlessdocs_enabled(Swagger UI loads JavaScript from a CDN).- uvicorn runs with HTTP/1.1 only, no access log (query strings carry file names), no
Serverheader, a 16 KiB header limit, a 5 s keep-alive and a connection limit of the admission slots plus 4. On SIGTERM or SIGINT it stops accepting connections and waits at mosttimeout_splus 5 s for requests in flight, then cancels them and exits. /v1/versions,/v1/capabilitiesand the probes reveal versions to unauthenticated callers.
Settings¶
ServerSettings.from_env() reads these variables; unset ones keep their defaults.
| field | variable | default |
|---|---|---|
max_body_bytes |
PYDIGGS_MAX_BODY_BYTES |
26214400 (25 MiB) |
max_findings |
PYDIGGS_MAX_FINDINGS |
10000 |
timeout_s |
PYDIGGS_TIMEOUT_S |
60 |
body_timeout_s |
PYDIGGS_BODY_TIMEOUT_S |
30 |
workers |
PYDIGGS_WORKERS |
CPUs − 1, from 1 to 4 |
queue_depth |
PYDIGGS_QUEUE_DEPTH |
2 × workers |
isolation |
PYDIGGS_ISOLATION |
per factory (see above) |
worker_memory_mb |
PYDIGGS_WORKER_MEMORY_MB |
2048 (Linux only) |
cors_origins |
PYDIGGS_CORS_ORIGINS (comma-separated) |
none |
allowed_hosts |
PYDIGGS_ALLOWED_HOSTS (comma-separated) |
loopback names |
reject_cross_site |
PYDIGGS_REJECT_CROSS_SITE |
true |
docs_enabled |
PYDIGGS_DOCS_ENABLED |
false |
versions_enabled |
PYDIGGS_VERSIONS_ENABLED |
true |
unknown_query |
PYDIGGS_UNKNOWN_QUERY (refuse or ignore) |
refuse |
Unknown query parameters¶
From pyDIGGS 1.14, /v1/validate, /v1/read, /v1/convert, /v1/ags4/check,
/v1/diff and /v1/rules refuse a query parameter they do not take, so a misspelt
option (?max_change=5) is an error rather than a silent default:
{"type": "https://xinp-hub.github.io/pydiggs/http-api/#problem-invalid-parameter",
"title": "Invalid parameter", "status": 422, "detail": "invalid parameters",
"code": "invalid-parameter",
"errors": [{"parameter": "max_change", "problem": "unknown parameter"}]}
Each name is listed once, in the order of the URL, at most 20; values are never echoed. A
name with no value (?foo) counts. The check comes before the body is read, and an
unknown name is reported alone, not with value errors. @pydiggs/client's
createPydiggsHandler answers the same. Multipart fields are not query parameters: an
unknown field is 400 invalid-multipart, as before. /v1/versions, /v1/capabilities
and the probes ignore the query. The GET endpoints (/v1/rules, /v1/versions,
/v1/capabilities) always answer JSON and do not negotiate Accept, so on /v1/rules
an unknown parameter is 422 whatever Accept says.
Upgrading from 1.13 or earlier. Remove the parameter the answer names. If you cannot
change a client yet, unknown_query="ignore" (PYDIGGS_UNKNOWN_QUERY=ignore) restores
the old behaviour and logs a warning per request; it is deprecated and will be removed in
pyDIGGS 2.0.
Errors¶
Every error is RFC 9457 application/problem+json with type, title, status,
detail and code. Switch on code. Texts are fixed: they never echo a value you sent
or a server path. invalid-parameter adds errors (at most 20 {parameter, problem}
pairs); 500 answers add a correlation_id that matches the server's log line.
{"type": "https://xinp-hub.github.io/pydiggs/http-api/#problem-busy", "title": "Server busy",
"status": 503, "detail": "Too many requests are in progress; try again shortly.", "code": "busy"}
| status | code | when |
|---|---|---|
| 400 | missing-input |
no file part (/v1/diff: no old or new part), or the body stopped arriving |
| 400 | invalid-multipart |
unparseable multipart, a second file (other than /v1/ags4/check's dictionary and /v1/diff's pair), too many, unknown or oversized fields |
| 400 | invalid-host |
Host not in allowed_hosts (app only) |
| 403 | cross-site |
cross-site request |
| 404 | not-found |
unknown path (app only) |
| 405 | method-not-allowed |
wrong method; see Allow (app only) |
| 406 | not-acceptable |
Accept allows neither JSON nor SARIF (/v1/read, /v1/convert, /v1/diff: not JSON) |
| 408 | request-timeout |
the body did not arrive within body_timeout_s |
| 413 | input-too-large |
body or file above max_body_bytes |
| 415 | compressed-input |
gzip document, or Content-Encoding other than identity |
| 415 | unsupported-media-type |
neither XML nor multipart (/v1/diff: not multipart; to=ags4 takes application/xml or text/xml; AGS4 operations: neither application/octet-stream, text/csv, text/plain nor multipart) |
| 422 | invalid-parameter |
bad ags_version, checks, diggs_version, name, select, step, to, rule id, a /v1/diff option, an unusable AGS4 dictionary, or a query parameter the endpoint does not take |
| 422 | invalid-document |
Titled "Invalid document". /v1/read, /v1/convert?to=ags4 and /v1/diff: the bytes are not a DIGGS document (/v1/diff names the file in errors); /v1/convert to DIGGS: not AGS4 (not-ags4) or AGS 3 (unsupported-version); kind says why |
| 422 | result-too-large |
/v1/read, /v1/convert, /v1/diff: the result is larger than four times max_body_bytes (for /v1/read, ask for fewer tables with select; for /v1/diff, list fewer changes with max_changes) |
| 422 | input-unreadable |
the document could not be read |
| 500 | resource-error |
bundled reference data could not be loaded |
| 500 | internal |
anything else |
| 503 | busy |
admission full; see Retry-After |
| 503 | not-ready |
workers still warming (probes) |
| 504 | timeout |
hard timeout (process isolation only) |
OpenAPI and versioning¶
The OpenAPI 3.1 document is committed as pydiggs/server/openapi.json, tested
against the code and published as
schemas/openapi.json. Its
DiggsFindings*, DiggsModel* and DiggsDiff* components are derived from the
packaged JSON Schemas. A
breaking wire change would be served as /v2 beside /v1; within /v1, validate
documents against their own $schema and ignore unknown fields. One tightening was made
within /v1, in 1.14: a query parameter an endpoint does not take is refused, with the
unknown_query="ignore" setting to keep the old behaviour until 2.0 (see
Unknown query parameters).