Skip to content

Findings: JSON and SARIF output

pydiggs validate and the validate() function report every problem in a DIGGS file as a finding, with a stable rule id, a severity and a location (line, XPath, gml:id). They print nothing and write no log files. The legacy subcommands (schema_check, dictionary_check, ...) and validator are unchanged.

pydiggs validate site/BH-01.xml --format json
pydiggs validate site/ --format sarif --output pydiggs.sarif
from pydiggs import validate

report = validate("site/BH-01.xml")
report.ok  # False if any finding is an error
report.diggs_version  # "2.5.a", "2.6" or "3.0.0" ("3.1-dev" only when asked for)
report.counts  # {"error": 2, "warning": 31, "info": 20}
report.steps  # {Step.SCHEMA: "failed", Step.DICTIONARY: "passed", ...}
for f in report.findings:
    print(f.step, f.rule_id, f.severity, f.line, f.xpath, f.gml_id, f.message)
report.to_dict()  # this file's entry of the JSON document
report.to_document()  # a whole diggs-findings document, as a dict
report.to_json(indent=2)  # the same document as text
report.to_sarif()  # a SARIF 2.1.0 log, as a dict

A document you received (an upload, a database blob) is checked with validate_bytes(), which takes bytes, a bytearray, a memoryview or a binary stream. It applies every limit validate() applies, never touches the filesystem for the instance, never reads local dictionaries, and uses only the bundled schema, dictionaries and Schematron rules. name is reported as path, reduced to its last component ("input.xml" when missing); absolute_path is None.

from pydiggs import findings_document, findings_json, validate_bytes

report = validate_bytes(upload, name="BH-01.xml", max_bytes=25 * 2**20, time_budget=30)
findings_document([report, other])  # one document for several reports, sorted by path
findings_json([report, other], indent=None)  # the same, on one line (safe for NDJSON)

validate() and validate_bytes() are safe to call from several threads at once; compiled validators are kept per thread, so call warm_up() in each new thread (a thread pool's initialiser) to keep the first file fast. time_budget is a soft limit, checked between steps (see Fail closed); a hard limit needs a separate process, like pydiggs validate --timeout.

validate() runs every native step by default, even after an earlier one fails, and records what happened to each in report.steps: passed, partial (no error, but part of the step could not be checked: the dictionary step when a code list was not fetched because network access is off, dictionary.offline), failed (at least one error), skipped (not asked for, the file is not well-formed XML, or it is not a DIGGS file) or incomplete (the step could not finish, or lost findings to max_findings; its findings are partial). A step with an error is failed or incomplete, never partial.

A file whose root element is not Diggs (any namespace) is not a DIGGS file: it gets one structure.diggs-root error naming the root it has, report.status is not-diggs, diggs_version is None, and every other step is skipped, since no DIGGS check means anything for it. In the test corpus, tests/fixtures/official/known_invalid/2.6_Temporal_Spatial_CRS_Example.xml (a gml:Dictionary, not a DIGGS instance) is reported this way.

report.verdict sums a file up in one word, derived only from its status, steps and findings (in this order): not-diggs; not-valid (any error); partly-checked (no error, but a step is partial or incomplete: a step stopped by a safety limit always has an error, so it is not-valid, but one whose warnings were cut by max_findings may have none); valid-with-warnings; valid. An unreadable input's verdict is unreadable. The text output starts each file's line with it. Text output lists errors and warnings, not info findings; for a partly-checked file it adds a not fully checked: line naming the partial or incomplete steps and lists its dictionary.offline findings, so the reason is always shown.

Well-formedness is always checked, because nothing else can run without it: a file that is not well-formed reports structure as failed even when structure was not asked for.

Fail closed

A check that cannot run to the end is never a pass. Whenever pydiggs stops a step for safety, or libxml2/libxslt stop it, the step is incomplete and an error finding says why, so report.ok is False and the CLI exits 1:

  • <step>.incomplete: an XPath error from libxml2 (its node-set limit is 10,000,000 nodes), an XSLT error while applying the bundled Schematron, a deprecation-registry XPath that cannot be evaluated, or libxml2 failing inside schema validation. Any other unexpected error inside a step (a bug) is reported the same way, with the exception's type and a short text, so it can neither escape validate() nor leave the step passed.
  • structure.markup-entity: the file's internal DTD declares an entity whose value contains markup (<, also when written &#60;), such as <!ENTITY e "<b/><b/>">. One such entity can multiply the number of elements without the file growing, so validate() reports this one error and runs no other check. The prolog is read first, with entities left unexpanded; if that parser cannot read the file (lxml's push parser rejects UTF-32 with a BOM, for example), the whole file is parsed without expanding entities, and the DTD the main parser saw is checked again after the full parse. Entities whose value is plain text (<!ENTITY org "Acme Ltd">) still work.
  • structure.namespace-too-long: a namespace URI longer than 1,024 characters is declared, in the file or as an xmlns default from the internal DTD's ATTLIST. Every element's tag carries its namespace, so one long URI multiplies the cost of every step; real DIGGS URIs are under 100 characters. One C-level pass over the namespace declarations decides this; validate() reports this one error and runs no other check.
  • structure.entity-expansion: the file has an internal DTD and, once parsed, its text (element text, tails, attribute values including ATTLIST defaults, and namespace URIs) is more than twice its raw size, both counted in UTF-8 bytes. Plain-text entities that stay under that still work (an &deg; here and there costs nothing). The text is measured in one pass, only when there is an internal DTD (with one, the file is parsed a second time with the internal subset's ATTLIST defaults filled in, to measure them); validate() then reports this one error and runs no other check. Only the internal DTD subset is ever used: an external subset (a SYSTEM id) is never opened by validate(), so a local file, a FIFO or a network share named there contributes nothing and is not even touched. Every size limit (max_bytes, the 10 MB the security review measures against) refers to the raw file, so this keeps the parsed size within a small factor of it.
  • A file with more than 9,900,000 elements does not run the XPath-based steps (dictionary, Schematron, deprecation, geometry, context, structure root checks): each gets a <step>.incomplete error. Schema validation still runs. The elements are counted on the parsed tree (about 0.1 s per million), so no encoding or entity can hide them.
  • schematron.incomplete when an element has more than 50,000 child elements (MAX_SCHEMATRON_SIBLINGS): the Schematron rules (bundled or your own) are not run, because libxslt can take minutes on such a file. pydiggs's own casing diameter rule still runs, so its findings can appear next to the error. Schema validation and the other steps are not affected; schematron_check() is unchanged.
  • schematron.incomplete when the rules would fire on more than 100,000 elements (MAX_SCHEMATRON_FIRED), counted before they run: libxslt's time grows with the square of that number (about 3 s at the limit, minutes at 1 million). Bundled and user rules alike; a context written *[local-name()='X']... is counted with lxml's C-level tag filter, any other with XPath count(), and a context XPath cannot evaluate on its own (a $variable) is not counted. The casing rule still runs.
  • dictionary.incomplete when a local dictionary was not read because of a safety limit: more than 16 different local dictionaries, more than their byte budget (see Limits), an internal DTD that declares a markup entity, or a libxml2 resource limit (entity amplification, nesting depth, a huge text node, an overlong name, a comment, processing instruction or CDATA section over libxml2's 10 MB node limit). The codeSpace also gets its dictionary.3 warning. Limits are recognised by libxml2's error code or its own limit wording; on a libxml2 build that reports a limit with neither, it is indistinguishable from a syntax error and gives only the warning.
  • dictionary.incomplete when the dictionary checks would cost too much: a Definition that lists more than 1,000 source or conditional paths or names, has more than 10,000 nodes, or uses a path of more than 64 steps; more than 250,000,000 Check 13 work units (one per dataValues character read, separators included, and a weight per element, attribute, coded element, index, group, reference, token, field and merged value, calibrated by scripts/calibrate_check13.py so that a unit costs at most about 30 ns whatever the loop (25 ns measured, the median of three runs, on an Apple M3 Max): the whole budget is about 6 s of CPU there, and 13 to 26 s on hardware two to four times slower, such as a CI runner or an older laptop. Share of the budget used by the script's data-shaped files: a CPT-like sounding (6 numeric columns, 230,000 rows, 9.5 MB) 21 %; one- character cells (10 columns, 500,000 rows, 10 MB) 23 %; one group of 10 to 200 coded columns cited by 1 to 10 ResultSets of 1,000 to 50,000 rows, 3 to 9 % at 2 to 6 MB; and the densest shape tried, 4.8 million cells in 10 MB, 147 million or 59 %); more than 10,000,000 values kept in Check 13's per-key caches (merged values, column unions, sorted columns, invalid values: a few hundred MB at most); or more than 5,000,000 dictionary work steps in one instance (path matching, sibling reads). Real DIGGS definitions list at most 9 paths, and the largest corpus file needs 39,000 steps; files shaped like real ones reach the budget at about 30 to 40 MB (an AtterbergExample measurement repeated 3,000 times, 26.5 MB, needs 3.7 million), so a very large file can be incomplete for this reason alone.
  • context.incomplete when dataBlocks inside nested measurements would need more than 5,000,000 comparisons.
  • structure.findings-truncated takes the severity of the worst finding it stands for.
  • structure.timeout: pydiggs validate --timeout SECONDS validates each file in its own process and stops it after SECONDS (start-up included); that file's requested steps are incomplete and the batch goes on. A worker that dies (killed, out of memory) is reported the same way. The time_budget of validate() and validate_bytes() gives the same shape without a process: it is checked between steps, the steps that had not started are incomplete, the steps that finished keep their findings, and one structure.timeout error names the steps that did not run. It cannot stop a step that is running.
  • structure.internal-error: pydiggs validate caught an unexpected exception for one file (a bug, or MemoryError); that file's requested steps are incomplete and the batch goes on. Only a broken resource you passed (ResourceError) stops the batch.

Limits

Parameter CLI Default What happens
max_findings --max-findings N 10,000 (None / 0: no limit) Findings past the limit are only counted. One extra structure.findings-truncated finding, as severe as the worst one left out (so report.ok and --fail-on stay right), says how many and from which steps; those steps are incomplete.
max_bytes --max-bytes N 100 MiB A larger file raises InstanceReadError (an unreadable entry in the CLI). The limit is on the raw bytes; entities may make the parsed text at most twice that (structure.entity-expansion). The legacy validator keeps its 1 GiB limit.
(fixed) 20 MiB of local dictionaries per instance All local dictionaries of one instance share this budget (or what max_bytes leaves after the instance, if less). A parsed dictionary takes up to about 35 times its file size in memory (about 800 MB peak for 20 MiB of <b/> elements); dictionaries with markup entities are refused, so entities cannot multiply that. A dictionary past the budget is not read: dictionary.3 plus a dictionary.incomplete error. Real DIGGS dictionaries are under 1 MB.
(fixed) 50,000 child elements under one element Above it the Schematron step is skipped with a schematron.incomplete error (the casing rule still runs).
(fixed) 100,000 elements the Schematron rules fire on Counted before the rules run; above it, as above.
(fixed) 1,000 paths or names and 10,000 nodes per dictionary Definition; 64 steps per path; 5,000,000 dictionary work steps per instance Above either, the dictionary step is incomplete with a dictionary.incomplete error. Check 6 and Check 8 messages list at most 20 paths, then "... and K more".
--timeout SECONDS none Positive, at most 86,400 (one day).
(fixed) 16 MiB per report Findings count towards a budget of roughly their JSON size; reaching it truncates exactly like max_findings.
(fixed) 4,096 characters A longer message is cut and ends in "…". A longer xpath or gml_id is reported as null (a cut path could name another element); the line is kept.

The structure.findings-truncated marker describes the report, not the file: it is not part of any step's status (the steps that lost findings are incomplete), it is left out of conformance comparisons, it is shown by --format text even when it is info, and counts include it.

Only regular files are read: a FIFO, socket or device fails at once with "not a regular file" instead of blocking.

It raises only InstanceReadError (the file cannot be read; also an OSError; for validate_bytes(), the data is too large or compressed or the stream failed), ResourceError (a schema, dictionary or Schematron file you passed, or pydiggs's bundled data, is missing or broken), ValueError (an unknown diggs_version or check name) and, for validate_bytes(), TypeError (text instead of bytes, a non-contiguous view or a non-string name). XML that is not well-formed is a finding, structure.well-formed, not an exception.

Command line

pydiggs validate PATH... [--format text|json|sarif] [--output FILE]
                 [--checks STEP[,STEP...]] [--diggs-version 2.5.a|2.6|3.0.0|3.1-dev]
                 [--schema-path P] [--dictionary-path P] [--schematron-path P]
                 [--fail-on error|warning] [--max-findings N] [--max-bytes N]
                 [--timeout SECONDS] [--snippets] [--absolute-paths] [--base-dir DIR]
                 [--local-dictionaries instance-dir|off] [--allow-network]
  • A directory expands to every regular *.xml file under it, recursively, in sorted order; symbolic links and special files are skipped.
  • A file that cannot be read becomes an entry with "status": "unreadable" and the rest of the batch still runs.
  • --checks picks steps from structure, schema, dictionary, schematron, geometry, deprecation, context (default: all).
  • --output is never written through a symbolic link. A new file is created with mode 0600; an existing file is overwritten and keeps its mode.
  • Text output escapes control and format characters (Unicode Cc and Cf, except tab) in paths and messages, so a file cannot send terminal escape sequences or bidi overrides to your terminal.

Exit codes

Code Meaning
0 No finding at or above --fail-on (default error)
1 At least one finding at or above --fail-on. A file that is not well-formed XML counts here.
2 Usage error, no input files found, a broken schema/dictionary/Schematron input, or any input that could not be read. 2 outranks 1.

The JSON document

The format is diggs-findings 1.2, described by a JSON Schema (draft 2020-12) that ships inside the package (pydiggs/findings/diggs-findings-1.2.schema.json) and is published with this site at schemas/diggs-findings-1.2.schema.json.

Version 1.2 (pyDIGGS 1.12.0) only adds to 1.1: the step diff, for the notes of a model comparison (Comparing two files). Every 1.1 document is a valid 1.2 document. The 1.1 schema stays packaged and published, unchanged, like 1.0; a consumer that checks documents against an earlier schema must move to the 1.2 schema, because each pins $schema to its own URL.

Version 1.1 (pyDIGGS 1.9.0) only adds to 1.0: the steps ags4 and convert, the optional location.path and the optional files[].format. Every 1.0 document is a valid 1.1 document, and the two new fields are written only when they say something, so a document about DIGGS files differs from its 1.0 form only in $schema and schema_version. The 1.0 schema stays packaged and published, unchanged. A consumer that checks documents against the 1.0 schema must move to the 1.1 schema, because the 1.0 schema pins $schema to its own URL.

{
  "$schema": "https://xinp-hub.github.io/pydiggs/schemas/diggs-findings-1.2.schema.json",
  "format": "diggs-findings",
  "schema_version": "1.2",
  "tool": {
    "name": "pydiggs",
    "version": "1.2.0",
    "reference_data": {"def": "2d76ba6c...", "schemas": "15ce2784..."}
  },
  "files": [
    {
      "path": "site/BH-01.xml",
      "sha256": "9f2c...",
      "diggs_version": "2.6",
      "status": "checked",
      "verdict": "not-valid",
      "steps": [{"step": "structure", "status": "passed"}, {"step": "schema", "status": "failed"}],
      "counts": {"error": 1, "warning": 0, "info": 0},
      "findings": [
        {
          "layer": "native",
          "step": "schema",
          "rule_id": "schema.cvc-complex-type.2.4",
          "severity": "error",
          "message": "Element '{http://diggsml.org/schemas/2.6}businessAssociate': This element is not expected. ...",
          "location": {
            "line": 12,
            "column": null,
            "xpath": "/Diggs[1]/documentInformation[1]/DocumentInformation[1]/author[1]/businessAssociate[1]",
            "gml_id": "BA_LADOTD"
          }
        }
      ]
    }
  ]
}
Field Meaning
tool.reference_data Upstream commit of each vendored DIGGS source (schemas, def), from the bundled PROVENANCE.toml
files[].path See Paths; absolute only with --absolute-paths / absolute_paths=True
files[].sha256 Digest of the file's bytes (null when unreadable)
files[].format Absent for a DIGGS file; ags4 for an AGS4 file, whose diggs_version is null (new in 1.1)
files[].status checked, not-diggs (the root is not Diggs; diggs_version is null and only structure ran) or unreadable (the entry also has error)
files[].verdict valid, valid-with-warnings, not-valid, partly-checked, not-diggs or unreadable, derived as described under validate(); SARIF carries status and verdict in each artifact's properties
findings[].layer native (pydiggs's own checks); official is reserved for the DIGGSml/validation layer
findings[].step One of structure, schema, codeType, dictionary, schematron, geometry, crs, coordinate, deprecation, context, and from 1.1 ags4 (reading and checking an AGS4 file) convert (what a conversion lost) and, from 1.2, diff (what a model comparison could not settle). validate() never runs the last three
location.line, location.column 1-based; the column is known only for XML syntax errors
location.xpath The official diggs:get-path form: every element as name()[n]
location.gml_id gml:id of the nearest element at or above the problem
location.path Only for a place in a source that is not XML (new in 1.1). For AGS4: GROUP, GROUP/row or GROUP/row/HEADING, with DATA rows counted from 1, for example GEOL/12/GEOL_LEG
source_snippet A shallow copy of the element: its attributes and text plus at most eight child elements (their attributes and text only; deeper content is "..."), a comment counting any further children, and only the namespace declarations those use; at most 1 KiB of UTF-8. Only with --snippets / include_snippets=True

There is no timestamp, keys come in a fixed order and findings are sorted (by step, line, column, severity, rule id, xpath, message), so the same input gives the same bytes. Compatibility: a new optional field or step is a new minor version (1.1 was the first, 1.2 the second); renaming or removing a field is 2.0. Messages are for people: compare rule_id, step, severity and location, never the message text. Unlike the legacy logs, a message has no Check N: prefix; the rule id carries the number.

Strings in the JSON document are the file's data as found: JSON escapes C0 control characters, but C1 controls (U+0080 to U+009F) and Unicode bidi or other format characters (for example U+202E) are left as they are. A consumer that shows paths or messages to people should escape those itself, as --format text does.

Paths

Situation Reported path
base_dir / --base-dir DIR given, file under it relative to DIR
base_dir given, file outside it the bare file name
no base_dir, working directory is / or an ancestor of your home directory the bare file name
no base_dir, file under the working directory relative to the working directory
otherwise the bare file name

A file name that is not valid UTF-8 is shown with its stray bytes as \xNN escapes.

Rule ids

Every rule id is <step>.<name> and matches ^<step>\.[A-Za-z0-9][A-Za-z0-9._-]*$. Rule ids are part of the stable contract.

Rule id Severity What it means
structure.well-formed error Not well-formed XML (line and column from libxml2)
structure.diggs-root error Root is not Diggs (then the file is not-diggs and nothing else runs), or documentInformation is missing or holds more than one DocumentInformation
schema.cvc-<rule> error / warning An XML Schema validation rule, e.g. schema.cvc-complex-type.2.4, schema.cvc-datatype-valid.1.2.1, schema.cvc-minInclusive-valid, named from libxml2's error type
schema.libxml2.<TYPE> error / warning A libxml2 error type that names no XSD rule, e.g. schema.libxml2.SCHEMAV_DOCUMENT_ELEMENT_MISSING
schema.unreleased-profile warning The file was checked against an unreleased DIGGS schema, because the experimental 3.1-dev profile was asked for by name; passing it is not conformance to a released DIGGS version (see Experimental: DIGGS 3.1-dev)
<step>.incomplete error The step could not finish: libxml2 could not finish schema validation, a libxml2/libxslt limit stopped an XPath or the bundled Schematron, the file has more than 9,900,000 elements, an element has more than 50,000 children or the rules would fire on more than 100,000 elements (Schematron), a local dictionary was refused by a safety limit, or dictionary path matching passed its limits (see Fail closed); the step is incomplete
structure.markup-entity error The internal DTD declares an entity whose value contains markup; no other check ran (see Fail closed)
structure.entity-expansion error The internal DTD's entities or ATTLIST defaults make the parsed text more than twice the file's raw size; no other check ran
structure.namespace-too-long error A namespace URI is longer than 1,024 characters; no other check ran
structure.read-limit error Reading only: the read stopped at a work limit (more objects or untyped elements than a file of its size may have, or untyped elements nested more than 64 deep); the model is incomplete
structure.duplicate-gml-id warning Reading only: two or more elements share one gml:id. Each copy is keyed by its xpath, and a link to that id resolves to the first copy
structure.value-unparsed info Reading only (pydiggs.read): a result value or quantity that should be a number is not one; the model keeps the text and reads the value as null
structure.timeout error Validation did not finish within its time limit: pydiggs validate --timeout stopped the file or its worker died, or the time_budget ran out; the unfinished steps are incomplete
structure.internal-error error pydiggs validate caught an unexpected error for this file; the requested steps are incomplete
dictionary.1 ... dictionary.12 see below The legacy dictionary checks, numbered as in their messages (1 uncontrolled codeSpace, 2 not whitelisted, 3 not accessible, 4 not a dictionary, 5 code not found, 6 wrong context, 7 name mismatch, 8 wrong measurement, 9 data type, 10 unitless with uom, 11 missing uom, 12 invalid uom)
dictionary.13 error / warning Result values (in the property's dataValues column) are not terms of the code list its definition names in valueCodeSpace
dictionary.offline info A whitelisted remote (http/https) code list that pydiggs does not bundle was not fetched because network access is off. This replaces the dictionary.3 (or dictionary.13) "could not be accessed" warning in validate() when allow_network=False; with network access on, a failed fetch is still dictionary.3 at warning.
structure.findings-truncated as the worst dropped finding max_findings or the 16 MiB report budget was reached; the rest were counted, not reported
schematron.<Context>.<property> error A bundled Schematron rule, e.g. schematron.Borehole.totalMeasuredDepth, schematron.Casing.outside-gt-inside
schematron.Casing.outside-gt-inside-converted, schematron.Casing.diameters-numeric-converted error pydiggs's unit-converting casing diameter rule
schematron.<id> / schematron.user-<hash> error A rule from your own Schematron file: its @id (up to 4,096 characters), else a hash of its context and test
geometry.srs-attributes warning A geometry object (one of the nine the official geometry module checks, with or without coordinates) has no srsName and srsDimension on itself or any ancestor: one warning per object, saying either what is missing or that they sit on a child (gml:pos/gml:posList, allowed by GML 3.2) when DIGGS best practice is the object itself
deprecation.deprecated warning Uses something the DIGGS schema deprecates
deprecation.removed error Uses something the DIGGS schema has removed
context.xlink-href-resolves warning A local xlink:href="#id" matches no gml:id
context.datablock-arity error A dataBlock tuple has a different number of values than Property declarations
geometry.reported-depth-mismatch warning A feature's totalMeasuredDepth and the length of its centerLine differ by more than 2 % (and more than 5 cm), the DIGGS File Inspector's tolerance. Checked only when the file states the axis units itself (uomLabels, or the axes of a LocalCartesianCRS it defines); otherwise skipped
geometry.crs-axis-order-suspect warning Coordinates in a geographic CRS (EPSG:4326, 4269, 4979 and other common ones) whose first value cannot be a latitude (absolute value over 90): the axes look swapped to longitude, latitude. A negative first value alone is not flagged, so southern- and western-hemisphere sites are fine. Reported once per srsName, at the first such element, with a count
geometry.crs-unresolvable warning An srsName that is neither a recognised EPSG or DIGGS CRS reference (the forms the Inspector recognises: urn:ogc:def:crs:EPSG::4326, compound URNs and crs-compound URLs, opengis.net/def/crs/EPSG/0/…, epsg.xml#…, #4326, urn:diggs:def:crs:DIGGS:0.1:…, diggsml.org/def/crs/…) nor the gml:id of an element in the file. Once per distinct value, with its number of uses
ags4.rule-1, ags4.rule-2, ags4.rule-2a, ags4.rule-2b, ags4.rule-3, ags4.rule-4, ags4.rule-5 error AGS4 reading only (pydiggs.ags4.read): the file breaks the AGS4 rule of that number, and the note says what the reader did about it. Rule 1: a byte order mark (dropped). Rule 2: a row before the first GROUP row (not read), or no GROUP row at all (the file is refused). Rule 2a: lines that do not end with CR LF. Rule 2b: a HEADING, UNIT or TYPE row that is missing (a missing UNIT or TYPE row is an empty units or types, not a row of empty values), repeated (only the first is read) or out of order. Rule 3: a row that starts with no known descriptor (not read). Rule 4: a row with more or fewer values than the HEADING row (cut or padded with empty values), or a GROUP row without exactly one name. Rule 5: a field not enclosed in double quotes
ags4.duplicate-group, ags4.unsupported-version, ags4.limit, ags4.timeout error AGS4 reading only. duplicate-group: a group name has more than one GROUP row, and only the first group is read. unsupported-version: an AGS 3 or older file, which is refused. limit and timeout: reading stopped at a work limit or at its time_budget, and the file is incomplete
ags4.encoding-assumed, ags4.version-unknown warning AGS4 reading only. encoding-assumed: the bytes are not UTF-8 and were read as Windows-1252. version-unknown: TRAN_AGS names a version pyDIGGS does not know, and the file is read as the newest known one
ags4.rule-6, ags4.rule-7, ags4.rule-8, ags4.rule-9, ags4.rule-10a, ags4.rule-10b, ags4.rule-10c, ags4.rule-11, ags4.rule-11a, ags4.rule-11b, ags4.rule-11c, ags4.rule-13, ags4.rule-14, ags4.rule-15, ags4.rule-16, ags4.rule-16a, ags4.rule-17, ags4.rule-18, ags4.rule-18a, ags4.rule-19, ags4.rule-19a, ags4.rule-19b, ags4.rule-20 error pydiggs.ags4.check and pydiggs ags4 check (which also report the reader's notes above, and Rules 1 and 2b where the tables show them: a value that is not ASCII, a group with no DATA rows, a heading named twice). Numbered as section 4.1.1 of the AGS4 specification, Edition 4.2. Rule 6: a line break inside a value. 7: headings out of dictionary order. 8: a value not written as its TYPE (nDP, nSF, nSCI, U, DT in the format its UNIT gives, T, DMS, YN). 9: a name in neither the dictionary nor the DICT group, or a user-defined heading before a standard one. 10a: a KEY heading missing, or two rows with the same KEY values. 10b: a REQUIRED heading missing or null. 10c: a row with no matching row in its parent group, or the parent group missing. 11: a Record Link that is not a GROUP name and KEY values; 11a, 11b: TRAN_DLIM, TRAN_RCON not one character; 11c: a link to a row or group not in the file. 13, 14: PROJ or TRAN missing or with more than one DATA row. 15, 16, 17: a unit, abbreviation or data type used but not listed in UNIT, ABBR or TYPE, or the group missing; 16a: a part of a combined abbreviation not listed. 18: a DICT row that does not say what it defines, or names outside the dictionary with no DICT group; 18a: user-defined headings out of DICT order. 19, 19a: a group or heading name of the wrong form; 19b: a heading that does not start with its group's name, or names another group that has no such heading. 20: a FILE_FSET the FILE group does not list, or a FILE_NAME that is a path (the FILE folder is never opened). Rules 7, 9, 10a, 10b, 10c and 18, and parts of 11c, 13, 14 and 19b, need the AGS4 data dictionary: they run for what the file's own DICT group defines, and for standard groups only with dictionary= (--dictionary). Messages name groups and headings, never a value from the file
ags4.formula-text warning AGS4 checking: a text value that starts with =, +, - (not a number) or @, which a spreadsheet would read as a formula. The value is kept as written; the writer never changes it
ags4.dictionary-not-checked info AGS4 checking without a data dictionary (pyDIGGS bundles none): the rules that need one were checked only for the groups and headings the file's own DICT group defines. Once per report
convert.unmapped-group, convert.unmapped-heading, convert.user-defined, convert.unmapped-code, convert.unmapped-unit, convert.value-unparsed, convert.assumed warning Conversion (pydiggs.convert; the AGS4 to DIGGS direction unless the text says otherwise): something the source holds did not reach the output, or the output holds a documented default. unmapped-group: a group with no mapping. unmapped-heading: a heading with no DIGGS target, or values that were not used. user-defined: a group or heading the file's own DICT defines, which is reported and never interpreted. unmapped-code: a code with no DIGGS code, left out or kept as text as the message says. unmapped-unit: a unit with no entry in the unit table. value-unparsed: a value that is not the number or date DIGGS needs. assumed: DIGGS requires a value the source lacks, and the default the message names was written. One finding per group, heading or reason, with a count; location.path is the first place, GROUP/row/HEADING (see AGS4 to DIGGS mapping)
convert.orphan-row, convert.incomplete error AGS4 to DIGGS conversion only. orphan-row: rows that name a parent record (a location, a sample, a test) that was not converted. incomplete: rows not converted because a limit or the time budget stopped the conversion, or because DIGGS requires something of the record that the source does not give
convert.unmapped-element warning DIGGS to AGS4 conversion only (pydiggs.convert.diggs_to_ags4, model_to_ags4): a value of the DIGGS model (a name or remark, a params entry, a typed field, an extra element, a link, or a result-table cell) has no AGS4 target in the mapping table, or cannot be written there as it is (another unit than its heading's, a line break). One finding per element and field, with a count; location.xpath is the first node's
diff.ambiguous-match info Model comparison only (pydiggs.diff.compare): several nodes of one collection share a natural key (for example two boreholes with the same name and no matching gml:id), so none of them was matched; they are listed as removed and added. At most 50 are written, then one note counts the rest
diff.limit, diff.timeout warning Model comparison only: the comparison stopped at max_changes (the list is cut; the counts stay exact), at its time_budget (the counts are partial), because a file was read in part, or because a model was read without the values of its result tables (results=False), so their cells were not compared. The result is incomplete and cannot claim the files are identical
convert.value-reformatted info AGS4 to DIGGS conversion only: the same value was written in another form (a date in ISO 8601 form, a number without trailing zeros); not a loss
context.xlink-remote-unresolved info An xlink:href into another document. pyDIGGS never fetches it, so what it points at was not checked. Once per document; links to DIGGS code lists (diggsml.org/def/codes/) are left to the dictionary step

rule_catalogue() returns this table as data (RuleInfo objects: fixed ids, then the open families with a pattern), and explain_rule(rule_id) describes one id; a well-formed id pydiggs never emits gets known=False. A test keeps the severities here and in the catalogue the same.

Plain-language guidance

Every rule in rule_catalogue(), and every id explain_rule() knows, also carries guidance: a short title, what the rule means for the data, and how to fix it, written for geotechnical and geoenvironmental engineers rather than DIGGS specialists. It is a Guidance object (title, meaning, fix, locale), and None only for an unknown id. explain_rule() gives an id its own guidance when it has some (the common schema.cvc-* rules, such as schema.cvc-complex-type.2.4), else its family's.

from pydiggs import explain_rule

g = explain_rule("dictionary.5").guidance
g.title  # 'Unknown code'
g.meaning  # 'A value uses a code that is not in the DIGGS code list it names, ...'
g.fix  # 'Pick a code from the list, checking spelling and capital letters, ...'

The findings document does not carry guidance, so diggs-findings stays at 1.2: an app shows a finding's guidance by looking up its rule_id with explain_rule() (or GET /v1/rules/{rule_id}, rules.explain, or the MCP explain_rule tool, which return the same guidance object). The text lives in pydiggs/findings/guidance/en.toml; only English ships today, and a test fails if a rule has no guidance or an entry matches no rule.

Dictionary check severities

Rule id Severity
dictionary.1 info
dictionary.2 warning
dictionary.3 warning
dictionary.4 error
dictionary.5 error
dictionary.6 error
dictionary.7 info
dictionary.8 error
dictionary.9 error
dictionary.10 error
dictionary.11 error / warning (warning for a dimensionless quantity)
dictionary.12 error / warning

Differences from the legacy checks

validate() differs from the legacy validator in these ways. Some of them (Check 13, Check 8, deprecation) mean a file can pass dictionary_check() and still have errors in validate():

  • Dictionary Check 13 (dictionary.13), a port of the official dictionary-validation.xsl check: for a property whose dictionary definition names a valueCodeSpace, every value in that property's column of the result set's dataValues (and of every result set citing the same parameters by xlink:href) must be a term (gml:id) of that code list. Empty values and the property's nullValue are allowed; Property/@index is read as an xs:integer (one longer than 64 characters, which no real column needs, is not a column). The message lists the first five invalid values and at most 20 valid terms (the official module lists all of them). dictionary_check() does not run it.
  • Dictionary Check 8 reads // in a definition's conditional paths as "any descendant", as XPath does. dictionary_check() reads ancestor::diggs:measurement//diggs:procedure as "procedure is a direct child of measurement", which is never true in real files (the path is measurement/Test/procedure), so it skips Check 8 for properties under a measurement. In the test corpus this adds two dictionary.8 errors in validate() for each of the two AtterbergExample files ("uscs group symbol" and "natural water content" are not allowed in that type of measurement); dictionary_check() still passes them (1.1 behaviour, kept byte for byte). The same reading applies to Check 6 source paths of the form ancestor::…//… (none of the bundled dictionaries has one). Predicates in definition paths ([text()="…"]) are ignored, as before, so matching is lenient; in validate() a / inside a predicate no longer splits the path into steps (dictionary_check() keeps splitting it).
  • Geometry SRS. validate() follows the official geometry module (modules/geometry-validation.xsl, DIGGSml/validation commit 5f7a365). It checks the same nine objects: LinearExtent, MultiPointLocation, MultiCurve, Solid, PointLocation, PlanarObservationRepresentation, PlanarSurface, MultiPlanarSurface and RectifiedGrid, with or without coordinates (so an empty <LinearExtent gml:id="l"/> is checked; gml:Point, gml:LineString and other GML primitives are not). An object passes when it or an ancestor has both srsName and srsDimension (present, even if empty); when only a child (gml:pos, gml:posList) has them, as GML 3.2 allows, it gets one best-practice warning; otherwise one warning names what is missing. The official module reports that last case, an object with no SRS on itself, any ancestor or any child, as an error; pydiggs keeps it a warning, for coordinate-less objects too. The legacy context_check() keeps its own list (Point, LineString, LinearExtent, Polygon, Curve, Surface, MultiPoint, MultiCurve, MultiSurface, PointLocation, MultiPointLocation), skips objects with no coordinates and no srsName, and gives one warning per missing attribute on the object itself, so DocumentationExample.xml gets 62 there and 31 in validate(). Across the 73 corpus files (the pydiggs test files and the desktop app's test data) the new list changes one verdict: the 3.x Aeromag.xml goes from valid to valid-with-warnings (its RectifiedGrid has SRS only on a child).
  • Local dictionaries. validate(local_dictionaries="instance-dir") (the default; CLI --local-dictionaries) reads a dictionary named by a relative or file: codeSpace only if it is a file inside the instance's folder: there is no fallback to the working directory, and .. or a symlink cannot leave the folder. Anything else, including a path the operating system rejects, is reported as dictionary.3 (or dictionary.13 for a Check 13 value code list; warning, "could not be accessed") with a note that says why; the check is made before the file is opened, so the finding is the same whether or not the file exists. Each file is read once per instance however its path is spelled, under max_bytes and only if it is a regular file, and at most 16 different local dictionaries are read per instance. All local dictionaries of an instance share one byte budget: 20 MiB, or what max_bytes leaves after the instance if that is less. A dictionary left unread by the cap or the budget is dictionary.3 with a note and makes the step incomplete with a dictionary.incomplete error. A local dictionary that is simply not well-formed XML (no safety limit involved) is a dictionary.3 warning, "could not be accessed", by design: that is what the official validator reports, and whoever controls the dictionary could as well write a permissive one. "off" reads no local dictionary. dictionary_check() keeps the 1.1 behaviour: next to the instance, then the working directory, and any file: URI.
  • dictionary_path. In validate() the dictionary you pass serves both properties.xml URLs (https and http), the legacy DIGGSTestPropertyDefinitions.xml URLs (which are looked up in it first) and the fallback for any other …/properties.xml. dictionary_check() keeps 1.1 behaviour: the bundled codes/properties.xml is registered after it under the same URLs and replaces it.
  • Offline dictionaries. validate() reports an unbundled remote dictionary as dictionary.offline (info) when network access is off; dictionary_check() still reports Check 3 as a warning.
  • Deprecation (deprecation.*), from the bundled DIGGS deprecation registry (deprecation/DeprecationRegistry.xml): deprecated items are warnings and removed items errors, with the official message text. An entry applies only from the release it names (since, or removedIn for removed items), so the registry's 3.1.0 entries never fire on 3.0.0, 2.6 or 2.5.a files; the official module does not make this distinction.

SARIF

--format sarif / report.to_sarif() writes SARIF 2.1.0 (validated in the test suite against the official OASIS errata01 schema):

SARIF From
one run per layer layer (run.properties.layer)
tool.driver.rules[] every rule id in the run, with shortDescription, helpUri and defaultConfiguration.level (the most severe level seen)
artifacts[] each file, uriBaseId: SRCROOT, with its sha-256; a file: URI only with absolute paths
results[].level error → error, warning → warning, info → note
results[].locations[].physicalLocation.region only when the line is known (startLine, and startColumn for syntax errors)
results[].locations[].logicalLocations[] {fullyQualifiedName: xpath, kind: "element"} ("attribute" for an attribute path); for a finding with no xpath but a location.path (AGS4), {fullyQualifiedName: path, kind: "element"}
results[].partialFingerprints["diggsConformanceKey/v1"] the conformance key, step|severity|location|rule_id
invocations[].toolExecutionNotifications one error per unreadable input

Conformance keys

Finding.conformance_key() returns (step, severity, location, rule_id), where location is "xpath:" plus the xpath with namespace prefixes dropped and every position made explicit; else, for a finding in a source that is not XML (diggs-findings 1.1), "path:" plus its path, followed by ";line:<n>" when the line is known; else "line:<n>", else "file". Two layers agree when their keys match as multisets over the steps both ran. The message is never compared.

Security notes

  • Schematron runs with generate-paths=false: lxml builds no @location strings (they copied the instance's namespace URIs, cost memory per failed assert and time per sibling). pydiggs locates each failed assert from the trusted schema instead, read from the same included and expanded schema lxml compiles, with the active phase applied: the n-th svrl:fired-rule of a pattern belongs to the n-th element, in document order, matched by one of that pattern's rule contexts (the first matching rule wins). The pairing is checked: the active-pattern id, the number of fired-rule elements and each one's context must agree with the elements found, or that pattern's failures have no location (never a wrong one). Contexts that match attributes, text or the root are never located. Nothing from the instance is evaluated as XPath. libxml2 error-log paths are checked against their /name[n] grammar and resolved by walking the tree.
  • A broken user Schematron file (a test that does not compile, or an unknown function) is a ResourceError (CLI exit 2), not a finding. That includes an XSLT error that only some instance triggers while the rules run: your rules are trusted input, so such an error is reported as a broken resource and stops the batch rather than being blamed on the file. Test your rules on representative files before using them on uploads.
  • Messages about an unreadable file do not contain its path; the report shows the path as described in Paths.

Residual risks

pydiggs's own code is bounded. On the security reviews' test files (up to 10 MB raw, default settings) no pydiggs-owned path took more than about 11 s of CPU (1,000 conditional paths tried against 180,000 elements, stopped by the work budget) or about 350 MB beyond parsing (ancestor-name sets under a 250-deep chain with 260,000 parents). The geometry rule is one pass over the tree (250 nested MultiCurve around 170,000 PointLocation, 9 MB: 1.4 s, where walking each object's ancestors and subtree took 99 s). Check 13 reads each ResultSet once for all the Properties of the PropertyParameters it holds (its nearest ResultSet, at any depth) or cites by xlink:href (400 of them over 2,000 rows of one 2 KB field: 1.4 s, where it took 44 s), and a column's values over a group's ResultSets are merged once for all the group's properties. The security review's 10 MB shapes are linear or stop at the budget: 55,000 ResultSets citing a 30,000-Property group, 1.4 s; 500 ResultSets citing one group up to 500 times each, 0.6 s; 20,000 PropertyParameters outside parameters over 7 million separators, 1.0 s (the first and last of these shapes ran for more than 460 s before, the last growing to 3 GB); 50,000 coded Properties read from 50,000 citing ResultSets, 2.0 s; 200 ResultSets each citing the same 400 groups of 400 columns, incomplete after about 5 s. Each index belongs to its nearest PropertyParameters only, so 250 nested PropertyParameters around 480,000 index holders (9.4 MB) take 1.5 s and about 240 MB, where re-reading every holder for each enclosing group took 102 s and 3.7 GB. Check 13 keeps no rows, only the distinct values of the columns its properties ask for, and one column set per group of PropertyParameters (2,200 ResultSets of 2,200 cells citing one group, 9.9 MB: about 205 MB beyond parsing, where keying the sets by ResultSet used 718 MB; one 9 MB row of 9 million empty fields takes 0.2 s and 113 MB, where reading every column took 2.1 GB). Groups read by the same ResultSets share one merge (300 groups each cited by the same 300 ResultSets, 2,700 distinct values each: 1.3 s and 171 MB); when every group is read by a different set of ResultSets, the merges are counted against the kept-values cap instead and the step stops incomplete (1.5 s, about 250 MB).

The costs below belong to lxml and libxml2/libxslt and are not bounded by pydiggs. Measured on an Apple M3 Max MacBook Pro in CPU seconds (one process, default settings, lxml 6.1.3, libxml2 2.14.6, libxslt 1.1.43; wall time on a busy machine is higher):

Cost Input CPU
Schematron (libxslt; grows with the square of the number of elements the rules fire on, whatever their parents) 1.8 MB, 100,000 fired elements, all failing (the most validate() runs) 3.4 to 5.5 s
1.7 MB, 100,000 fired elements, all passing 3.6 s
9 MB, about 1 million fired elements (21 parents × 50,000) 174 s and 1.7 GB before the cap; now refused (schematron.incomplete) in 1.7 s
9.9 MB, 100,000 fired elements among 1.1 million same-named elements the rules do not fire on (under both caps) 35 to 43 s and about 500 MB; 22 to 29 s with Casing
9 to 10 MB with no element a rule fires on 5 to 6 s
1 MB, 18,000 sibling features (no failures) 0.1 s (31 s with the pre-1.2 rule contexts)
Schema validation (lxml builds an XPath for every libxml2 error by counting siblings) 2 MB of sibling schema errors (37,000 siblings) 14 s
4 MB of the same 107 s
8 MB of the same over 400 s
Parsing (the lxml tree) 42 MB, 10.5 million elements 1.4 GiB, 1.4 s
Local dictionaries (parsed trees) 20 MiB of <b/> elements, the most the budget allows about 800 MB

The bundled rules' contexts were rewritten for 1.2 (*[local-name()='C'][parent::*[local-name()='P']] instead of //*[local-name()='P']/*[local-name()='C']; same elements, same verdicts). The old form made libxslt quadratic in the number of siblings of any element (over 400 s on several 8 to 11 MB files). No form tried is linear in both that and the number of elements a rule fires on, so validate() caps both: more than 50,000 children under one element, or more than 100,000 fired elements, skips Schematron with a schematron.incomplete error.

Entities are bounded relative to the raw file. An internal DTD entity with markup is refused (structure.markup-entity) before any check runs, in every encoding the parser reads: usually decided from the prolog alone, and for UTF-32 with a BOM, which lxml's push parser cannot read, after one parse that expands no entity. A file whose plain-text entities make its text more than twice its raw size is refused after the parse (structure.entity-expansion); the parse itself is libxml2's, which limits entity amplification to a factor of about 5 and any single text, comment, PI, CDATA or attribute node to 10 MB (without huge_tree, which pydiggs never sets), so the largest parse pydiggs can be asked to do is bounded by the raw size before its own 2x check runs. A 9.6 MB file that would expand to 12 million elements is reported in 0.1 s and 54 MB, where 1.2 development builds used 1.7 GB; dataBlocks of entity-expanded text that took 9 s are refused in under 0.1 s. Local dictionaries with markup entities are refused the same way (dictionary.incomplete); one 20 MiB dictionary that expanded to 26 million elements used 3.4 GB before.

Services that validate files from untrusted sources should therefore validate each upload in its own folder, in a separate process with a time and memory limit. pydiggs validate --timeout SECONDS gives a time limit (a file that runs out of time, or whose worker dies, is a structure.timeout error) and --max-bytes keeps inputs small. Also:

  • --timeout has a floor. Each file gets a fresh interpreter (the spawn start method, on every OS), which imports pydiggs and lxml and compiles the bundled schemas before validating: about 0.3 s of CPU for a small file, and several seconds of wall time on a loaded machine. The limit includes that start-up, so values under about 10 s can stop files that would pass; use 30 s or more on shared machines.
  • spawn re-imports your main module. A script that calls pydiggs.cli.main(["validate", "--timeout", ...]) must do it under if __name__ == "__main__":, or each worker runs the script again. The pydiggs command needs nothing.
  • No memory limit. --timeout limits time only. Workers have no memory or CPU rlimit, run with your privileges and inherit your environment; set memory limits with the operating system or the container (for example ulimit -v, a cgroup, or a Kubernetes memory limit).
  • instance-dir reads any file in the instance's folder as a dictionary when a codeSpace names it: in a shared upload folder that includes other users' uploads, and a hard link to a file elsewhere cannot be told apart from a real file.
  • The legacy entry points (validator, pydiggs schema_check and the other subcommands, detect_diggs_version) keep their 1.1.1 behaviour by design: local dictionaries next to the instance, then in the working directory, and any file: URI; a 1 GiB size limit; internal DTD entities expanded; Schematron with lxml's location strings; element paths that cost time in proportion to depth times the number of findings; a local dictionary parsed once per spelling of its path; and a dictionary_path that the bundled codes/properties.xml replaces. They use the same rewritten bundled Schematron rules, so their verdicts and output are unchanged. Use validate() for untrusted input.

Privacy

Findings can travel further than the file: into CI logs, issue trackers and code-scanning dashboards. By default pydiggs keeps them free of the file's content and your file system layout:

  • No snippets by default. source_snippet copies the element's XML, which is client data; it appears only with --snippets / include_snippets=True, capped at 1 KiB.
  • No absolute paths by default. Paths follow Paths in JSON, SARIF and text alike, so running from / or above your home directory never spells out your directory layout; messages never contain the absolute path of the instance. --absolute-paths opts in.
  • No timestamps, host names or user names in either format.
  • No network unless --allow-network / allow_network=True, and then only to whitelisted dictionary URLs.

Messages can still quote values from the file (a code that is not in a dictionary, for example). Treat a findings file as you would treat the DIGGS file itself.