Skip to content

HTTP API

pydiggs[server] serves pydiggs over HTTP: upload a DIGGS document, get a diggs-findings 1.2 document (or a SARIF 2.1.0 log) back. It is a FastAPI application you can run as is, or a router you mount in your own FastAPI app.

pip install 'pydiggs[server]'
pydiggs serve --http                       # http://127.0.0.1:8000
curl -s -H 'Content-Type: application/xml' --data-binary @BH-01.xml \
  'http://127.0.0.1:8000/v1/validate?name=BH-01.xml'

Every string in a findings document is untrusted

Finding messages, paths and identifiers come from the uploaded file. Escape every string field before you render it in HTML or a terminal.

pydiggs adds no authentication, rate limiting per client or TLS. Put it behind your own app, a reverse proxy or an API gateway for that.

Behind a reverse proxy when not on loopback

Any deployment reachable from other machines must sit behind a reverse proxy (nginx, Caddy, HAProxy, a cloud load balancer) with client header and body timeouts. uvicorn has no timeout for reading request headers: a client that opens connections and sends partial headers holds them, and enough of them exhaust uvicorn's limit_concurrency (the admission slots plus 4), so the server stops accepting work. pydiggs bounds only what happens after the headers: the body must arrive within body_timeout_s (408), and validation within timeout_s (504).

Running it

pydiggs serve --http --host 127.0.0.1 --port 8000 --workers 2 --timeout 60

--max-bytes, --max-findings, --timeout and --workers override the PYDIGGS_* variables below. Bound to a non-loopback address, the server refuses to start until PYDIGGS_ALLOWED_HOSTS lists the host names clients use.

from pydiggs.server import ServerSettings, create_app

app = create_app(ServerSettings.from_env())
from fastapi import FastAPI
from pydiggs.server import ServerSettings, create_router

app = FastAPI()
app.include_router(create_router(ServerSettings(isolation="process")))

The router enforces its own limits (body size, admission, multipart, cross-site refusal), so they hold without create_app. It adds no Host check and no CORS headers: those are your app's job. Its default isolation is "thread"; pass isolation="process" for untrusted uploads.

Endpoints

method and path answer
POST /v1/validate 200 with a diggs-findings 1.2 document, or SARIF 2.1.0 for Accept: application/sarif+json
POST /v1/read 200 with a diggs-model 1.0 document: what is in the file (Reading files)
POST /v1/convert 200 with a conversion result: an AGS4 file as DIGGS 3.0.0, or a DIGGS document as AGS4 (base64), and every loss (AGS4 and DIGGS conversion)
POST /v1/ags4/check 200 with a diggs-findings 1.2 document of the AGS4 rules, or SARIF 2.1.0 (AGS4 files)
POST /v1/diff 200 with a diggs-diff 1.0 document: what was added, removed or changed between two DIGGS files (Comparing two files)
GET /v1/versions pydiggs, findings schema and DIGGS versions, reference-data commits, api_version: "1" (off with versions_enabled=False)
GET /v1/capabilities a diggs-capabilities 1.0 document: what this server's pydiggs can do (off with versions_enabled=False)
GET /v1/rules?step= every rule id pydiggs can emit (open families as patterns)
GET /v1/rules/{rule_id} one rule; a valid id pydiggs does not emit is 200 with known: false
GET /v1/readyz 200 once the workers are warm, else 503 not-ready
GET /livez 200 while the server runs (app only; use it for container health checks)
GET /readyz as /v1/readyz (app only)

POST /v1/validate

Send the document either way:

  • Raw: Content-Type: application/xml or text/xml, the document as the body, and the query parameters name, checks, diggs_version.
  • Multipart: Content-Type: multipart/form-data with one part named file and the optional fields name, checks, diggs_version. A field wins over the same query parameter, but both must be valid. An overridden query value is checked in the request the fields complete, and only for itself: on /v1/convert, ?to=ags4&ags_version=4.2 with the field ags_version=4.1.1, and ?to=diggs with the fields to=ags4 and ags_version=4.2, are accepted; ?to=bogus with the field to=diggs is refused (422). name wins over the part's file name. Other fields are refused.
parameter meaning
name File name to report; reduced to a basename, input.xml when empty
checks Comma-separated steps (repeatable): structure, schema, dictionary, schematron, geometry, deprecation, context; default all
diggs_version 2.5.a, 2.6 or 3.0.0; default detected. 3.1-dev selects the unreleased DIGGS 3.1 draft (experimental)
curl -s -F file=@BH-01.xml -F checks=structure,schema \
  -H 'Accept: application/sarif+json' http://127.0.0.1:8000/v1/validate

Malformed XML, findings past max_findings (the structure.findings-truncated finding) and an exhausted time budget (structure.timeout) are all 200 with findings, as with pydiggs validate --format json. Only a request that produced no document is an error.

Accept decides the format: application/json (the default, also for */* and no header) or application/sarif+json. When both are listed with the same quality, JSON wins. An Accept that allows neither is 406.

POST /v1/read

Reads the document into a diggs-model 1.0 document: boreholes, samples, tests and their result tables, as pydiggs.read() returns them. The body is exactly pydiggs.read_bytes(data, name=name).to_json(), so every surface gives the same bytes for the same file. Send the document as for /v1/validate (raw XML or one multipart file part); the body limit, admission, isolation and time limit are the same.

parameter meaning
name File name to report; reduced to a basename, input.xml when empty
diggs_version As for /v1/validate; default detected
select JSON text: {"results": "all"} (the default), {"results": "none"} or {"results": ["key", …]} (at most 1,000 node keys)
curl -s -F file=@BH-01.xml -F 'select={"results": "none"}' http://127.0.0.1:8000/v1/read
  • select says which result tables keep their values. A table that is left out keeps its columns and row_count with empty values. List a file first with {"results": "none"}, then ask for the measurements you need by key. Each worker keeps the last few models it read (by content hash), so the second call does not parse the file again. That is a cache only: nothing is stored between requests that a caller could address.
  • Reading is not validating. A schema-invalid file still reads with 200; run /v1/validate for a verdict. What reading had to work around is in the document's notes.
  • Not a DIGGS document is 422 invalid-document (title "Invalid document"), with kind malformed, not-diggs, unsafe-entity or namespace-too-long. (/v1/validate reports the same cases as findings in a 200.)
  • Size. The reader's own work limits (Reading files) apply, and a document larger than four times max_body_bytes is 422 result-too-large; ask for fewer tables with select.
  • Time limit: with thread isolation, running out of time is a 200 with complete: false and the members read so far; with process isolation it is 504 timeout.
  • The answer is always application/json. On the conformance corpus a whole model is 0.6 to 2 times the size of its XML (1.1 times overall); compressing responses is the host's job.

POST /v1/convert

Converts one AGS4 file to DIGGS 3.0.0 (to=diggs, the default), or one DIGGS document to an AGS4 file (to=ags4), by the stable pydiggs.convert and mapping table 1.2 (see AGS4 and DIGGS conversion). The body limit, admission, isolation and time limit are those of /v1/read.

  • Raw: the file as the body and the parameters in the query. For to=diggs the body is AGS4, Content-Type: application/octet-stream, text/plain or text/csv (AGS4 has no registered media type); for to=ags4 (given in the query) it is DIGGS XML, application/xml or text/xml. Another media type is 415.
  • Multipart: one part named file and the optional fields below; as for /v1/validate, a field wins over the same query parameter and both must be valid.
parameter meaning
name File name to report; reduced to a basename, input.ags (input.xml for to=ags4) when empty
to The format to convert to: diggs (the default; the body is AGS4) or ags4 (the body is DIGGS XML)
diggs_version to=diggs only: the DIGGS version to write, 3.0.0 (the default)
ags_version to=ags4 only: the AGS4 version to write, 4.2 (the default) or 4.1.1. As of November 2026 the BGS AGS validator and python-ags4 1.2.0 accept AGS4 only up to 4.1.1

A parameter given for the other direction is 422 invalid-parameter.

curl -s -F file=@site.ags http://127.0.0.1:8000/v1/convert > result.json
jq -r .output_base64 result.json | base64 --decode > site.xml
jq -r '.residual_base64 // empty' result.json | base64 --decode > left.ags
jq '.report.files[0].findings[] | select(.step == "convert")' result.json

curl -s 'http://127.0.0.1:8000/v1/convert?to=ags4&ags_version=4.1.1' \
  -H 'Content-Type: application/xml' --data-binary @site.xml \
  | jq -r .output_base64 | base64 --decode > site.ags

The answer is one JSON object (application/json, component ConversionResult): the output bytes as output_base64 with media_type (application/xml for DIGGS, text/plain; charset=utf-8 for AGS4), output_name and output_sha256; the report about the source (the reader's notes and convert findings, diggs-findings 1.2); the validation of the output (pydiggs.validate for DIGGS, pydiggs.ags4.check for AGS4); diggs_version, ags_version (null for to=diggs), mapping_version, complete and lossless; and residual_base64, the residual AGS4 file of what an AGS4 to DIGGS conversion did not convert (null when nothing was left over, and for to=ags4). The fields are described in AGS4 and DIGGS conversion and the residual file in The residual file.

  • Why one JSON object. A client gets the document and its findings in one call, with one parser, and can check output_sha256 against the bytes it decodes. OpenAPI 3.1 describes base64 inside JSON directly (contentEncoding: base64 with contentMediaType), so generated clients type it; multipart responses are poorly supported by HTTP clients and code generators, and putting the report in a header would hit header size limits. The cost is a third more bytes for the output; compress responses at the host if that matters.
  • Losses are not errors. Unmapped content, an output that fails validation or the AGS4 check, and an exhausted time budget (complete: false, with thread isolation) are 200s; the findings say what happened.
  • Not AGS4 (to=diggs) is 422 invalid-document with kind not-ags4, or unsupported-version for an AGS 3 file; not DIGGS (to=ags4) is 422 invalid-document with the kind of /v1/read. Compressed input (gzip, zip) is 415.
  • Size. A result larger than four times max_body_bytes is 422 result-too-large.

POST /v1/ags4/check

Checks one AGS4 file against the AGS4 rules (Rules 1 to 20) and answers with a diggs-findings 1.2 document of step ags4 (or SARIF 2.1.0 for Accept: application/sarif+json), the report ags4.check gives for the same bytes. Send the file raw (as for /v1/convert) or as multipart with a file part and, optionally, a dictionary part: an AGS4 data dictionary, which pyDIGGS does not bundle (see The data dictionary). The only parameter is name.

curl -s -F file=@site.ags -F dictionary=@Standard_dictionary_v4_1_1.ags \
  http://127.0.0.1:8000/v1/ags4/check

Rule breaks, a file that is not AGS4 at all, truncation and an exhausted time budget are 200s with findings, as for /v1/validate. A dictionary that cannot be used is 422 invalid-parameter naming dictionary. The file and the dictionary share the body limit. The server writes the file to a private temporary folder (mode 0700) for the checker and removes it at once; with the container's read-only root filesystem that is the /tmp tmpfs.

POST /v1/diff

Compares two DIGGS files with the stable pydiggs.diff.compare() and answers with its diggs-diff 1.0 document. The body is exactly what compare(old, new, ...).to_json() gives for the same bytes and options, with each file reported under its basename, so every surface gives the same bytes for the same pair. From pyDIGGS 1.13.

Send both files as multipart (Content-Type: multipart/form-data): one part named old (the earlier file) and one named new (the later), each exactly once, and the optional fields below. A raw body is 415: there are two files. As for /v1/validate, a field wins over the same query parameter, and both must be valid.

parameter meaning
old_name, new_name File names to report; reduced to a basename, the part's file name when not given, input.xml when empty
match How records are paired: auto (the default; by gml:id, then by natural key), gml-id (also written gml_id) or natural
rel_tol Two numbers are equal within this fraction of the larger one: from 0 to less than 1, default 1e-9
abs_tol Or within this much, in the value's own unit: 0 or more, default 0
max_changes Most changes listed, 0 or more (default 10000); the counts stay exact, and a cut list is complete: false
show_reformatted true to also list values only written differently (default false)
show_renamed true to also list records whose gml:id changed (default false)
curl -s -F old=@BH-01-v1.xml -F new=@BH-01-v2.xml -F max_changes=100 \
  http://127.0.0.1:8000/v1/diff | jq '{identical, summary, value_changes}'
  • A difference is not an error. Identical or not, the answer is 200; read identical and complete.
  • Limits are those of /v1/read, per file. Each file may be up to max_body_bytes, so the body may carry twice that (plus the multipart allowance); a larger file is 413. Both files are read and compared in one job in one worker, under the one timeout_s and the one admission slot: no new limit or setting. A result larger than four times max_body_bytes is 422 result-too-large; list fewer changes with max_changes.
  • Not a DIGGS document is 422 invalid-document with the kind of /v1/read, and errors naming the file: [{"parameter": "new", "problem": "is not a DIGGS document"}]. (The CLI exits 2 for the same input.)
  • Time limit: with thread isolation, running out of time is a 200 with complete: false and a diff.timeout note, and options.time_budget is the server's timeout_s; with process isolation it is 504 timeout and options.time_budget is null.
  • The two files are compared in memory; nothing is written to disk. Every string in the document (names, keys, values) comes from the files: escape it before you render it.

Limits and admission

Checks run in this order, before the body is read where possible:

  1. Cross-site refusal (403).
  2. Accept (406), then a query parameter the endpoint does not take (422 invalid-parameter, see Unknown query parameters), then media type (415), Content-Encoding other than identity (415).
  3. Content-Length above the limit: 413 at once, body not read.
  4. Admission: a slot among workers + queue_depth is taken without waiting; when none is free the answer is 503 busy with Retry-After: 1, body not read.
  5. The body is streamed with a running byte count and abandoned at the limit (413), so chunked bodies are bounded too. Multipart allows one file (two for /v1/ags4/check and /v1/diff) and at most 8 fields of 64 KiB; each /v1/diff file counts against max_body_bytes on its own, any other file against it together; spooled parts are deleted.
  6. The whole body must arrive within body_timeout_s (30 s) of admission; otherwise the answer is 408 request-timeout with Connection: close, and the slot is freed.

Gzip-compressed documents are refused (415 compressed-input).

These limits live in the route, so they also hold when only the router is mounted.

Isolation

isolation how a request runs time limit
"process" (default for create_app and serve --http) in one of workers spawned processes hard: past timeout_s the worker is killed and replaced, 504 timeout
"thread" (default for create_router) in one of workers threads cooperative only: a long libxml2 call holds its thread; unsafe for untrusted input

Process workers warm every bundled DIGGS version before they report ready. On Linux each worker runs under an address-space cap (worker_memory_mb, 2 GiB) and a CPU-time backstop; macOS enforces only the CPU backstop and Windows neither, so use a container or host limits there. Workers send results back as JSON bytes, never pickles. create_app logs a warning when it is given isolation="thread".

Size the host for ASGI server processes × pydiggs workers × about 2 GiB.

Host, cross-site and CORS

  • create_app refuses any Host not in allowed_hosts with 400 invalid-host (DNS-rebinding protection). Unset, it allows 127.0.0.1, localhost and [::1].
  • Every route refuses a request with Sec-Fetch-Site: cross-site, or with an Origin whose host differs from Host (null included), unless the origin is in cors_origins (403 cross-site; turn off with reject_cross_site=False).
  • create_app sends CORS headers only to the origins in cors_origins; "*" is refused.
  • /docs, /redoc and /openapi.json are off unless docs_enabled (Swagger UI loads JavaScript from a CDN).
  • uvicorn runs with HTTP/1.1 only, no access log (query strings carry file names), no Server header, a 16 KiB header limit, a 5 s keep-alive and a connection limit of the admission slots plus 4. On SIGTERM or SIGINT it stops accepting connections and waits at most timeout_s plus 5 s for requests in flight, then cancels them and exits.
  • /v1/versions, /v1/capabilities and the probes reveal versions to unauthenticated callers.

Settings

ServerSettings.from_env() reads these variables; unset ones keep their defaults.

field variable default
max_body_bytes PYDIGGS_MAX_BODY_BYTES 26214400 (25 MiB)
max_findings PYDIGGS_MAX_FINDINGS 10000
timeout_s PYDIGGS_TIMEOUT_S 60
body_timeout_s PYDIGGS_BODY_TIMEOUT_S 30
workers PYDIGGS_WORKERS CPUs − 1, from 1 to 4
queue_depth PYDIGGS_QUEUE_DEPTH 2 × workers
isolation PYDIGGS_ISOLATION per factory (see above)
worker_memory_mb PYDIGGS_WORKER_MEMORY_MB 2048 (Linux only)
cors_origins PYDIGGS_CORS_ORIGINS (comma-separated) none
allowed_hosts PYDIGGS_ALLOWED_HOSTS (comma-separated) loopback names
reject_cross_site PYDIGGS_REJECT_CROSS_SITE true
docs_enabled PYDIGGS_DOCS_ENABLED false
versions_enabled PYDIGGS_VERSIONS_ENABLED true
unknown_query PYDIGGS_UNKNOWN_QUERY (refuse or ignore) refuse

Unknown query parameters

From pyDIGGS 1.14, /v1/validate, /v1/read, /v1/convert, /v1/ags4/check, /v1/diff and /v1/rules refuse a query parameter they do not take, so a misspelt option (?max_change=5) is an error rather than a silent default:

{"type": "https://xinp-hub.github.io/pydiggs/http-api/#problem-invalid-parameter",
 "title": "Invalid parameter", "status": 422, "detail": "invalid parameters",
 "code": "invalid-parameter",
 "errors": [{"parameter": "max_change", "problem": "unknown parameter"}]}

Each name is listed once, in the order of the URL, at most 20; values are never echoed. A name with no value (?foo) counts. The check comes before the body is read, and an unknown name is reported alone, not with value errors. @pydiggs/client's createPydiggsHandler answers the same. Multipart fields are not query parameters: an unknown field is 400 invalid-multipart, as before. /v1/versions, /v1/capabilities and the probes ignore the query. The GET endpoints (/v1/rules, /v1/versions, /v1/capabilities) always answer JSON and do not negotiate Accept, so on /v1/rules an unknown parameter is 422 whatever Accept says.

Upgrading from 1.13 or earlier. Remove the parameter the answer names. If you cannot change a client yet, unknown_query="ignore" (PYDIGGS_UNKNOWN_QUERY=ignore) restores the old behaviour and logs a warning per request; it is deprecated and will be removed in pyDIGGS 2.0.

Errors

Every error is RFC 9457 application/problem+json with type, title, status, detail and code. Switch on code. Texts are fixed: they never echo a value you sent or a server path. invalid-parameter adds errors (at most 20 {parameter, problem} pairs); 500 answers add a correlation_id that matches the server's log line.

{"type": "https://xinp-hub.github.io/pydiggs/http-api/#problem-busy", "title": "Server busy",
 "status": 503, "detail": "Too many requests are in progress; try again shortly.", "code": "busy"}
status code when
400 missing-input no file part (/v1/diff: no old or new part), or the body stopped arriving
400 invalid-multipart unparseable multipart, a second file (other than /v1/ags4/check's dictionary and /v1/diff's pair), too many, unknown or oversized fields
400 invalid-host Host not in allowed_hosts (app only)
403 cross-site cross-site request
404 not-found unknown path (app only)
405 method-not-allowed wrong method; see Allow (app only)
406 not-acceptable Accept allows neither JSON nor SARIF (/v1/read, /v1/convert, /v1/diff: not JSON)
408 request-timeout the body did not arrive within body_timeout_s
413 input-too-large body or file above max_body_bytes
415 compressed-input gzip document, or Content-Encoding other than identity
415 unsupported-media-type neither XML nor multipart (/v1/diff: not multipart; to=ags4 takes application/xml or text/xml; AGS4 operations: neither application/octet-stream, text/csv, text/plain nor multipart)
422 invalid-parameter bad ags_version, checks, diggs_version, name, select, step, to, rule id, a /v1/diff option, an unusable AGS4 dictionary, or a query parameter the endpoint does not take
422 invalid-document Titled "Invalid document". /v1/read, /v1/convert?to=ags4 and /v1/diff: the bytes are not a DIGGS document (/v1/diff names the file in errors); /v1/convert to DIGGS: not AGS4 (not-ags4) or AGS 3 (unsupported-version); kind says why
422 result-too-large /v1/read, /v1/convert, /v1/diff: the result is larger than four times max_body_bytes (for /v1/read, ask for fewer tables with select; for /v1/diff, list fewer changes with max_changes)
422 input-unreadable the document could not be read
500 resource-error bundled reference data could not be loaded
500 internal anything else
503 busy admission full; see Retry-After
503 not-ready workers still warming (probes)
504 timeout hard timeout (process isolation only)

OpenAPI and versioning

The OpenAPI 3.1 document is committed as pydiggs/server/openapi.json, tested against the code and published as schemas/openapi.json. Its DiggsFindings*, DiggsModel* and DiggsDiff* components are derived from the packaged JSON Schemas. A breaking wire change would be served as /v2 beside /v1; within /v1, validate documents against their own $schema and ignore unknown fields. One tightening was made within /v1, in 1.14: a query parameter an endpoint does not take is refused, with the unknown_query="ignore" setting to keep the old behaviour until 2.0 (see Unknown query parameters).