Findings: JSON and SARIF output¶
pydiggs validate and the validate() function report every problem in a DIGGS file as a
finding, with a stable rule id, a severity and a location (line, XPath, gml:id).
They print nothing and write no log files. The legacy subcommands (schema_check,
dictionary_check, ...) and validator are unchanged.
pydiggs validate site/BH-01.xml --format json
pydiggs validate site/ --format sarif --output pydiggs.sarif
from pydiggs import validate
report = validate("site/BH-01.xml")
report.ok # False if any finding is an error
report.diggs_version # "2.5.a", "2.6" or "3.0.0" ("3.1-dev" only when asked for)
report.counts # {"error": 2, "warning": 31, "info": 20}
report.steps # {Step.SCHEMA: "failed", Step.DICTIONARY: "passed", ...}
for f in report.findings:
print(f.step, f.rule_id, f.severity, f.line, f.xpath, f.gml_id, f.message)
report.to_dict() # this file's entry of the JSON document
report.to_document() # a whole diggs-findings document, as a dict
report.to_json(indent=2) # the same document as text
report.to_sarif() # a SARIF 2.1.0 log, as a dict
A document you received (an upload, a database blob) is checked with
validate_bytes(), which takes bytes, a bytearray, a memoryview or a binary
stream. It applies every limit validate() applies, never touches the filesystem for the
instance, never reads local dictionaries, and uses only the bundled schema, dictionaries
and Schematron rules. name is reported as path, reduced to its last component
("input.xml" when missing); absolute_path is None.
from pydiggs import findings_document, findings_json, validate_bytes
report = validate_bytes(upload, name="BH-01.xml", max_bytes=25 * 2**20, time_budget=30)
findings_document([report, other]) # one document for several reports, sorted by path
findings_json([report, other], indent=None) # the same, on one line (safe for NDJSON)
validate() and validate_bytes() are safe to call from several threads at once;
compiled validators are kept per thread, so call warm_up() in each new thread (a
thread pool's initialiser) to keep the first file fast. time_budget is a soft limit,
checked between steps (see Fail closed); a hard limit needs a separate
process, like pydiggs validate --timeout.
validate() runs every native step by default, even after an earlier one fails, and
records what happened to each in report.steps: passed, partial (no error, but part
of the step could not be checked: the dictionary step when a code list was not fetched
because network access is off, dictionary.offline), failed (at least one error),
skipped (not asked for, the file is not well-formed XML, or it is not a DIGGS file) or
incomplete (the step could not finish, or lost findings to max_findings; its
findings are partial). A step with an error is failed or incomplete, never
partial.
A file whose root element is not Diggs (any namespace) is not a DIGGS file: it
gets one structure.diggs-root error naming the root it has, report.status is
not-diggs, diggs_version is None, and every other step is skipped, since no
DIGGS check means anything for it. In the test corpus,
tests/fixtures/official/known_invalid/2.6_Temporal_Spatial_CRS_Example.xml (a
gml:Dictionary, not a DIGGS instance) is reported this way.
report.verdict sums a file up in one word, derived only from its status, steps and
findings (in this order): not-diggs; not-valid (any error); partly-checked (no
error, but a step is partial or incomplete: a step stopped by a safety limit always
has an error, so it is not-valid, but one whose warnings were cut by max_findings
may have none); valid-with-warnings; valid. An unreadable input's verdict is
unreadable. The text output starts each file's line with it. Text output lists
errors and warnings, not info findings; for a partly-checked file it adds a
not fully checked: line naming the partial or incomplete steps and lists its
dictionary.offline findings, so the reason is always shown.
Well-formedness is always checked, because nothing else can run without it: a file
that is not well-formed reports structure as failed even when structure was not
asked for.
Fail closed¶
A check that cannot run to the end is never a pass. Whenever pydiggs stops a step for
safety, or libxml2/libxslt stop it, the step is incomplete and an error finding
says why, so report.ok is False and the CLI exits 1:
<step>.incomplete: an XPath error from libxml2 (its node-set limit is 10,000,000 nodes), an XSLT error while applying the bundled Schematron, a deprecation-registry XPath that cannot be evaluated, or libxml2 failing inside schema validation. Any other unexpected error inside a step (a bug) is reported the same way, with the exception's type and a short text, so it can neither escapevalidate()nor leave the steppassed.structure.markup-entity: the file's internal DTD declares an entity whose value contains markup (<, also when written<), such as<!ENTITY e "<b/><b/>">. One such entity can multiply the number of elements without the file growing, sovalidate()reports this one error and runs no other check. The prolog is read first, with entities left unexpanded; if that parser cannot read the file (lxml's push parser rejects UTF-32 with a BOM, for example), the whole file is parsed without expanding entities, and the DTD the main parser saw is checked again after the full parse. Entities whose value is plain text (<!ENTITY org "Acme Ltd">) still work.structure.namespace-too-long: a namespace URI longer than 1,024 characters is declared, in the file or as anxmlnsdefault from the internal DTD's ATTLIST. Every element's tag carries its namespace, so one long URI multiplies the cost of every step; real DIGGS URIs are under 100 characters. One C-level pass over the namespace declarations decides this;validate()reports this one error and runs no other check.structure.entity-expansion: the file has an internal DTD and, once parsed, its text (element text, tails, attribute values including ATTLIST defaults, and namespace URIs) is more than twice its raw size, both counted in UTF-8 bytes. Plain-text entities that stay under that still work (an°here and there costs nothing). The text is measured in one pass, only when there is an internal DTD (with one, the file is parsed a second time with the internal subset's ATTLIST defaults filled in, to measure them);validate()then reports this one error and runs no other check. Only the internal DTD subset is ever used: an external subset (aSYSTEMid) is never opened byvalidate(), so a local file, a FIFO or a network share named there contributes nothing and is not even touched. Every size limit (max_bytes, the 10 MB the security review measures against) refers to the raw file, so this keeps the parsed size within a small factor of it.- A file with more than 9,900,000 elements does not run the XPath-based steps
(dictionary, Schematron, deprecation, geometry, context, structure root checks): each
gets a
<step>.incompleteerror. Schema validation still runs. The elements are counted on the parsed tree (about 0.1 s per million), so no encoding or entity can hide them. schematron.incompletewhen an element has more than 50,000 child elements (MAX_SCHEMATRON_SIBLINGS): the Schematron rules (bundled or your own) are not run, because libxslt can take minutes on such a file. pydiggs's own casing diameter rule still runs, so its findings can appear next to the error. Schema validation and the other steps are not affected;schematron_check()is unchanged.schematron.incompletewhen the rules would fire on more than 100,000 elements (MAX_SCHEMATRON_FIRED), counted before they run: libxslt's time grows with the square of that number (about 3 s at the limit, minutes at 1 million). Bundled and user rules alike; a context written*[local-name()='X']...is counted with lxml's C-level tag filter, any other with XPathcount(), and a context XPath cannot evaluate on its own (a$variable) is not counted. The casing rule still runs.dictionary.incompletewhen a local dictionary was not read because of a safety limit: more than 16 different local dictionaries, more than their byte budget (see Limits), an internal DTD that declares a markup entity, or a libxml2 resource limit (entity amplification, nesting depth, a huge text node, an overlong name, a comment, processing instruction or CDATA section over libxml2's 10 MB node limit). The codeSpace also gets itsdictionary.3warning. Limits are recognised by libxml2's error code or its own limit wording; on a libxml2 build that reports a limit with neither, it is indistinguishable from a syntax error and gives only the warning.dictionary.incompletewhen the dictionary checks would cost too much: a Definition that lists more than 1,000 source or conditional paths or names, has more than 10,000 nodes, or uses a path of more than 64 steps; more than 250,000,000 Check 13 work units (one perdataValuescharacter read, separators included, and a weight per element, attribute, coded element,index, group, reference, token, field and merged value, calibrated byscripts/calibrate_check13.pyso that a unit costs at most about 30 ns whatever the loop (25 ns measured, the median of three runs, on an Apple M3 Max): the whole budget is about 6 s of CPU there, and 13 to 26 s on hardware two to four times slower, such as a CI runner or an older laptop. Share of the budget used by the script's data-shaped files: a CPT-like sounding (6 numeric columns, 230,000 rows, 9.5 MB) 21 %; one- character cells (10 columns, 500,000 rows, 10 MB) 23 %; one group of 10 to 200 coded columns cited by 1 to 10 ResultSets of 1,000 to 50,000 rows, 3 to 9 % at 2 to 6 MB; and the densest shape tried, 4.8 million cells in 10 MB, 147 million or 59 %); more than 10,000,000 values kept in Check 13's per-key caches (merged values, column unions, sorted columns, invalid values: a few hundred MB at most); or more than 5,000,000 dictionary work steps in one instance (path matching, sibling reads). Real DIGGS definitions list at most 9 paths, and the largest corpus file needs 39,000 steps; files shaped like real ones reach the budget at about 30 to 40 MB (an AtterbergExample measurement repeated 3,000 times, 26.5 MB, needs 3.7 million), so a very large file can beincompletefor this reason alone.context.incompletewhendataBlocks inside nestedmeasurements would need more than 5,000,000 comparisons.structure.findings-truncatedtakes the severity of the worst finding it stands for.structure.timeout:pydiggs validate --timeout SECONDSvalidates each file in its own process and stops it after SECONDS (start-up included); that file's requested steps areincompleteand the batch goes on. A worker that dies (killed, out of memory) is reported the same way. Thetime_budgetofvalidate()andvalidate_bytes()gives the same shape without a process: it is checked between steps, the steps that had not started areincomplete, the steps that finished keep their findings, and onestructure.timeouterror names the steps that did not run. It cannot stop a step that is running.structure.internal-error:pydiggs validatecaught an unexpected exception for one file (a bug, orMemoryError); that file's requested steps areincompleteand the batch goes on. Only a broken resource you passed (ResourceError) stops the batch.
Limits¶
| Parameter | CLI | Default | What happens |
|---|---|---|---|
max_findings |
--max-findings N |
10,000 (None / 0: no limit) |
Findings past the limit are only counted. One extra structure.findings-truncated finding, as severe as the worst one left out (so report.ok and --fail-on stay right), says how many and from which steps; those steps are incomplete. |
max_bytes |
--max-bytes N |
100 MiB | A larger file raises InstanceReadError (an unreadable entry in the CLI). The limit is on the raw bytes; entities may make the parsed text at most twice that (structure.entity-expansion). The legacy validator keeps its 1 GiB limit. |
| (fixed) | 20 MiB of local dictionaries per instance | All local dictionaries of one instance share this budget (or what max_bytes leaves after the instance, if less). A parsed dictionary takes up to about 35 times its file size in memory (about 800 MB peak for 20 MiB of <b/> elements); dictionaries with markup entities are refused, so entities cannot multiply that. A dictionary past the budget is not read: dictionary.3 plus a dictionary.incomplete error. Real DIGGS dictionaries are under 1 MB. |
|
| (fixed) | 50,000 child elements under one element | Above it the Schematron step is skipped with a schematron.incomplete error (the casing rule still runs). |
|
| (fixed) | 100,000 elements the Schematron rules fire on | Counted before the rules run; above it, as above. | |
| (fixed) | 1,000 paths or names and 10,000 nodes per dictionary Definition; 64 steps per path; 5,000,000 dictionary work steps per instance | Above either, the dictionary step is incomplete with a dictionary.incomplete error. Check 6 and Check 8 messages list at most 20 paths, then "... and K more". |
|
--timeout SECONDS |
none | Positive, at most 86,400 (one day). | |
| (fixed) | 16 MiB per report | Findings count towards a budget of roughly their JSON size; reaching it truncates exactly like max_findings. |
|
| (fixed) | 4,096 characters | A longer message is cut and ends in "…". A longer xpath or gml_id is reported as null (a cut path could name another element); the line is kept. |
The structure.findings-truncated marker describes the report, not the file: it is not
part of any step's status (the steps that lost findings are incomplete), it is left
out of conformance comparisons, it is shown by --format text even when it is info,
and counts include it.
Only regular files are read: a FIFO, socket or device fails at once with "not a regular file" instead of blocking.
It raises only InstanceReadError (the file cannot be read; also an OSError;
for validate_bytes(), the data is too large or compressed or the stream failed),
ResourceError (a schema, dictionary or Schematron file you passed, or pydiggs's bundled
data, is missing or broken), ValueError (an unknown diggs_version or check name) and,
for validate_bytes(), TypeError (text instead of bytes, a non-contiguous view or a
non-string name).
XML that is not well-formed is a finding, structure.well-formed, not an exception.
Command line¶
pydiggs validate PATH... [--format text|json|sarif] [--output FILE]
[--checks STEP[,STEP...]] [--diggs-version 2.5.a|2.6|3.0.0|3.1-dev]
[--schema-path P] [--dictionary-path P] [--schematron-path P]
[--fail-on error|warning] [--max-findings N] [--max-bytes N]
[--timeout SECONDS] [--snippets] [--absolute-paths] [--base-dir DIR]
[--local-dictionaries instance-dir|off] [--allow-network]
- A directory expands to every regular
*.xmlfile under it, recursively, in sorted order; symbolic links and special files are skipped. - A file that cannot be read becomes an entry with
"status": "unreadable"and the rest of the batch still runs. --checkspicks steps fromstructure,schema,dictionary,schematron,geometry,deprecation,context(default: all).--outputis never written through a symbolic link. A new file is created with mode0600; an existing file is overwritten and keeps its mode.- Text output escapes control and format characters (Unicode Cc and Cf, except tab) in paths and messages, so a file cannot send terminal escape sequences or bidi overrides to your terminal.
Exit codes¶
| Code | Meaning |
|---|---|
| 0 | No finding at or above --fail-on (default error) |
| 1 | At least one finding at or above --fail-on. A file that is not well-formed XML counts here. |
| 2 | Usage error, no input files found, a broken schema/dictionary/Schematron input, or any input that could not be read. 2 outranks 1. |
The JSON document¶
The format is diggs-findings 1.2, described by a JSON Schema (draft 2020-12) that
ships inside the package (pydiggs/findings/diggs-findings-1.2.schema.json) and is
published with this site at
schemas/diggs-findings-1.2.schema.json.
Version 1.2 (pyDIGGS 1.12.0) only adds to 1.1: the step diff, for the notes of a model
comparison (Comparing two files).
Every 1.1 document is a valid 1.2 document. The
1.1 schema stays packaged and published, unchanged, like 1.0; a consumer that checks
documents against an earlier schema must move to the 1.2 schema, because each pins $schema to its own URL.
Version 1.1 (pyDIGGS 1.9.0) only adds to 1.0: the steps ags4 and convert, the optional
location.path and the optional files[].format. Every 1.0 document is a valid 1.1
document, and the two new fields are written only when they say something, so a document
about DIGGS files differs from its 1.0 form only in $schema and schema_version. The
1.0 schema stays packaged and published, unchanged. A consumer that
checks documents against the 1.0 schema must move to the 1.1 schema, because the 1.0
schema pins $schema to its own URL.
{
"$schema": "https://xinp-hub.github.io/pydiggs/schemas/diggs-findings-1.2.schema.json",
"format": "diggs-findings",
"schema_version": "1.2",
"tool": {
"name": "pydiggs",
"version": "1.2.0",
"reference_data": {"def": "2d76ba6c...", "schemas": "15ce2784..."}
},
"files": [
{
"path": "site/BH-01.xml",
"sha256": "9f2c...",
"diggs_version": "2.6",
"status": "checked",
"verdict": "not-valid",
"steps": [{"step": "structure", "status": "passed"}, {"step": "schema", "status": "failed"}],
"counts": {"error": 1, "warning": 0, "info": 0},
"findings": [
{
"layer": "native",
"step": "schema",
"rule_id": "schema.cvc-complex-type.2.4",
"severity": "error",
"message": "Element '{http://diggsml.org/schemas/2.6}businessAssociate': This element is not expected. ...",
"location": {
"line": 12,
"column": null,
"xpath": "/Diggs[1]/documentInformation[1]/DocumentInformation[1]/author[1]/businessAssociate[1]",
"gml_id": "BA_LADOTD"
}
}
]
}
]
}
| Field | Meaning |
|---|---|
tool.reference_data |
Upstream commit of each vendored DIGGS source (schemas, def), from the bundled PROVENANCE.toml |
files[].path |
See Paths; absolute only with --absolute-paths / absolute_paths=True |
files[].sha256 |
Digest of the file's bytes (null when unreadable) |
files[].format |
Absent for a DIGGS file; ags4 for an AGS4 file, whose diggs_version is null (new in 1.1) |
files[].status |
checked, not-diggs (the root is not Diggs; diggs_version is null and only structure ran) or unreadable (the entry also has error) |
files[].verdict |
valid, valid-with-warnings, not-valid, partly-checked, not-diggs or unreadable, derived as described under validate(); SARIF carries status and verdict in each artifact's properties |
findings[].layer |
native (pydiggs's own checks); official is reserved for the DIGGSml/validation layer |
findings[].step |
One of structure, schema, codeType, dictionary, schematron, geometry, crs, coordinate, deprecation, context, and from 1.1 ags4 (reading and checking an AGS4 file) convert (what a conversion lost) and, from 1.2, diff (what a model comparison could not settle). validate() never runs the last three |
location.line, location.column |
1-based; the column is known only for XML syntax errors |
location.xpath |
The official diggs:get-path form: every element as name()[n] |
location.gml_id |
gml:id of the nearest element at or above the problem |
location.path |
Only for a place in a source that is not XML (new in 1.1). For AGS4: GROUP, GROUP/row or GROUP/row/HEADING, with DATA rows counted from 1, for example GEOL/12/GEOL_LEG |
source_snippet |
A shallow copy of the element: its attributes and text plus at most eight child elements (their attributes and text only; deeper content is "..."), a comment counting any further children, and only the namespace declarations those use; at most 1 KiB of UTF-8. Only with --snippets / include_snippets=True |
There is no timestamp, keys come in a fixed order and findings are sorted (by step, line,
column, severity, rule id, xpath, message), so the same input gives the same bytes.
Compatibility: a new optional field or step is a new minor version (1.1 was the first, 1.2 the second); renaming or removing a field is 2.0.
Messages are for people: compare rule_id, step, severity and location, never
the message text. Unlike the legacy logs, a message has no Check N: prefix; the rule id
carries the number.
Strings in the JSON document are the file's data as found: JSON escapes C0 control
characters, but C1 controls (U+0080 to U+009F) and Unicode bidi or other format
characters (for example U+202E) are left as they are. A consumer that shows paths or
messages to people should escape those itself, as --format text does.
Paths¶
| Situation | Reported path |
|---|---|
base_dir / --base-dir DIR given, file under it |
relative to DIR |
base_dir given, file outside it |
the bare file name |
no base_dir, working directory is / or an ancestor of your home directory |
the bare file name |
no base_dir, file under the working directory |
relative to the working directory |
| otherwise | the bare file name |
A file name that is not valid UTF-8 is shown with its stray bytes as \xNN escapes.
Rule ids¶
Every rule id is <step>.<name> and matches ^<step>\.[A-Za-z0-9][A-Za-z0-9._-]*$.
Rule ids are part of the stable contract.
| Rule id | Severity | What it means |
|---|---|---|
structure.well-formed |
error | Not well-formed XML (line and column from libxml2) |
structure.diggs-root |
error | Root is not Diggs (then the file is not-diggs and nothing else runs), or documentInformation is missing or holds more than one DocumentInformation |
schema.cvc-<rule> |
error / warning | An XML Schema validation rule, e.g. schema.cvc-complex-type.2.4, schema.cvc-datatype-valid.1.2.1, schema.cvc-minInclusive-valid, named from libxml2's error type |
schema.libxml2.<TYPE> |
error / warning | A libxml2 error type that names no XSD rule, e.g. schema.libxml2.SCHEMAV_DOCUMENT_ELEMENT_MISSING |
schema.unreleased-profile |
warning | The file was checked against an unreleased DIGGS schema, because the experimental 3.1-dev profile was asked for by name; passing it is not conformance to a released DIGGS version (see Experimental: DIGGS 3.1-dev) |
<step>.incomplete |
error | The step could not finish: libxml2 could not finish schema validation, a libxml2/libxslt limit stopped an XPath or the bundled Schematron, the file has more than 9,900,000 elements, an element has more than 50,000 children or the rules would fire on more than 100,000 elements (Schematron), a local dictionary was refused by a safety limit, or dictionary path matching passed its limits (see Fail closed); the step is incomplete |
structure.markup-entity |
error | The internal DTD declares an entity whose value contains markup; no other check ran (see Fail closed) |
structure.entity-expansion |
error | The internal DTD's entities or ATTLIST defaults make the parsed text more than twice the file's raw size; no other check ran |
structure.namespace-too-long |
error | A namespace URI is longer than 1,024 characters; no other check ran |
structure.read-limit |
error | Reading only: the read stopped at a work limit (more objects or untyped elements than a file of its size may have, or untyped elements nested more than 64 deep); the model is incomplete |
structure.duplicate-gml-id |
warning | Reading only: two or more elements share one gml:id. Each copy is keyed by its xpath, and a link to that id resolves to the first copy |
structure.value-unparsed |
info | Reading only (pydiggs.read): a result value or quantity that should be a number is not one; the model keeps the text and reads the value as null |
structure.timeout |
error | Validation did not finish within its time limit: pydiggs validate --timeout stopped the file or its worker died, or the time_budget ran out; the unfinished steps are incomplete |
structure.internal-error |
error | pydiggs validate caught an unexpected error for this file; the requested steps are incomplete |
dictionary.1 ... dictionary.12 |
see below | The legacy dictionary checks, numbered as in their messages (1 uncontrolled codeSpace, 2 not whitelisted, 3 not accessible, 4 not a dictionary, 5 code not found, 6 wrong context, 7 name mismatch, 8 wrong measurement, 9 data type, 10 unitless with uom, 11 missing uom, 12 invalid uom) |
dictionary.13 |
error / warning | Result values (in the property's dataValues column) are not terms of the code list its definition names in valueCodeSpace |
dictionary.offline |
info | A whitelisted remote (http/https) code list that pydiggs does not bundle was not fetched because network access is off. This replaces the dictionary.3 (or dictionary.13) "could not be accessed" warning in validate() when allow_network=False; with network access on, a failed fetch is still dictionary.3 at warning. |
structure.findings-truncated |
as the worst dropped finding | max_findings or the 16 MiB report budget was reached; the rest were counted, not reported |
schematron.<Context>.<property> |
error | A bundled Schematron rule, e.g. schematron.Borehole.totalMeasuredDepth, schematron.Casing.outside-gt-inside |
schematron.Casing.outside-gt-inside-converted, schematron.Casing.diameters-numeric-converted |
error | pydiggs's unit-converting casing diameter rule |
schematron.<id> / schematron.user-<hash> |
error | A rule from your own Schematron file: its @id (up to 4,096 characters), else a hash of its context and test |
geometry.srs-attributes |
warning | A geometry object (one of the nine the official geometry module checks, with or without coordinates) has no srsName and srsDimension on itself or any ancestor: one warning per object, saying either what is missing or that they sit on a child (gml:pos/gml:posList, allowed by GML 3.2) when DIGGS best practice is the object itself |
deprecation.deprecated |
warning | Uses something the DIGGS schema deprecates |
deprecation.removed |
error | Uses something the DIGGS schema has removed |
context.xlink-href-resolves |
warning | A local xlink:href="#id" matches no gml:id |
context.datablock-arity |
error | A dataBlock tuple has a different number of values than Property declarations |
geometry.reported-depth-mismatch |
warning | A feature's totalMeasuredDepth and the length of its centerLine differ by more than 2 % (and more than 5 cm), the DIGGS File Inspector's tolerance. Checked only when the file states the axis units itself (uomLabels, or the axes of a LocalCartesianCRS it defines); otherwise skipped |
geometry.crs-axis-order-suspect |
warning | Coordinates in a geographic CRS (EPSG:4326, 4269, 4979 and other common ones) whose first value cannot be a latitude (absolute value over 90): the axes look swapped to longitude, latitude. A negative first value alone is not flagged, so southern- and western-hemisphere sites are fine. Reported once per srsName, at the first such element, with a count |
geometry.crs-unresolvable |
warning | An srsName that is neither a recognised EPSG or DIGGS CRS reference (the forms the Inspector recognises: urn:ogc:def:crs:EPSG::4326, compound URNs and crs-compound URLs, opengis.net/def/crs/EPSG/0/…, epsg.xml#…, #4326, urn:diggs:def:crs:DIGGS:0.1:…, diggsml.org/def/crs/…) nor the gml:id of an element in the file. Once per distinct value, with its number of uses |
ags4.rule-1, ags4.rule-2, ags4.rule-2a, ags4.rule-2b, ags4.rule-3, ags4.rule-4, ags4.rule-5 |
error | AGS4 reading only (pydiggs.ags4.read): the file breaks the AGS4 rule of that number, and the note says what the reader did about it. Rule 1: a byte order mark (dropped). Rule 2: a row before the first GROUP row (not read), or no GROUP row at all (the file is refused). Rule 2a: lines that do not end with CR LF. Rule 2b: a HEADING, UNIT or TYPE row that is missing (a missing UNIT or TYPE row is an empty units or types, not a row of empty values), repeated (only the first is read) or out of order. Rule 3: a row that starts with no known descriptor (not read). Rule 4: a row with more or fewer values than the HEADING row (cut or padded with empty values), or a GROUP row without exactly one name. Rule 5: a field not enclosed in double quotes |
ags4.duplicate-group, ags4.unsupported-version, ags4.limit, ags4.timeout |
error | AGS4 reading only. duplicate-group: a group name has more than one GROUP row, and only the first group is read. unsupported-version: an AGS 3 or older file, which is refused. limit and timeout: reading stopped at a work limit or at its time_budget, and the file is incomplete |
ags4.encoding-assumed, ags4.version-unknown |
warning | AGS4 reading only. encoding-assumed: the bytes are not UTF-8 and were read as Windows-1252. version-unknown: TRAN_AGS names a version pyDIGGS does not know, and the file is read as the newest known one |
ags4.rule-6, ags4.rule-7, ags4.rule-8, ags4.rule-9, ags4.rule-10a, ags4.rule-10b, ags4.rule-10c, ags4.rule-11, ags4.rule-11a, ags4.rule-11b, ags4.rule-11c, ags4.rule-13, ags4.rule-14, ags4.rule-15, ags4.rule-16, ags4.rule-16a, ags4.rule-17, ags4.rule-18, ags4.rule-18a, ags4.rule-19, ags4.rule-19a, ags4.rule-19b, ags4.rule-20 |
error | pydiggs.ags4.check and pydiggs ags4 check (which also report the reader's notes above, and Rules 1 and 2b where the tables show them: a value that is not ASCII, a group with no DATA rows, a heading named twice). Numbered as section 4.1.1 of the AGS4 specification, Edition 4.2. Rule 6: a line break inside a value. 7: headings out of dictionary order. 8: a value not written as its TYPE (nDP, nSF, nSCI, U, DT in the format its UNIT gives, T, DMS, YN). 9: a name in neither the dictionary nor the DICT group, or a user-defined heading before a standard one. 10a: a KEY heading missing, or two rows with the same KEY values. 10b: a REQUIRED heading missing or null. 10c: a row with no matching row in its parent group, or the parent group missing. 11: a Record Link that is not a GROUP name and KEY values; 11a, 11b: TRAN_DLIM, TRAN_RCON not one character; 11c: a link to a row or group not in the file. 13, 14: PROJ or TRAN missing or with more than one DATA row. 15, 16, 17: a unit, abbreviation or data type used but not listed in UNIT, ABBR or TYPE, or the group missing; 16a: a part of a combined abbreviation not listed. 18: a DICT row that does not say what it defines, or names outside the dictionary with no DICT group; 18a: user-defined headings out of DICT order. 19, 19a: a group or heading name of the wrong form; 19b: a heading that does not start with its group's name, or names another group that has no such heading. 20: a FILE_FSET the FILE group does not list, or a FILE_NAME that is a path (the FILE folder is never opened). Rules 7, 9, 10a, 10b, 10c and 18, and parts of 11c, 13, 14 and 19b, need the AGS4 data dictionary: they run for what the file's own DICT group defines, and for standard groups only with dictionary= (--dictionary). Messages name groups and headings, never a value from the file |
ags4.formula-text |
warning | AGS4 checking: a text value that starts with =, +, - (not a number) or @, which a spreadsheet would read as a formula. The value is kept as written; the writer never changes it |
ags4.dictionary-not-checked |
info | AGS4 checking without a data dictionary (pyDIGGS bundles none): the rules that need one were checked only for the groups and headings the file's own DICT group defines. Once per report |
convert.unmapped-group, convert.unmapped-heading, convert.user-defined, convert.unmapped-code, convert.unmapped-unit, convert.value-unparsed, convert.assumed |
warning | Conversion (pydiggs.convert; the AGS4 to DIGGS direction unless the text says otherwise): something the source holds did not reach the output, or the output holds a documented default. unmapped-group: a group with no mapping. unmapped-heading: a heading with no DIGGS target, or values that were not used. user-defined: a group or heading the file's own DICT defines, which is reported and never interpreted. unmapped-code: a code with no DIGGS code, left out or kept as text as the message says. unmapped-unit: a unit with no entry in the unit table. value-unparsed: a value that is not the number or date DIGGS needs. assumed: DIGGS requires a value the source lacks, and the default the message names was written. One finding per group, heading or reason, with a count; location.path is the first place, GROUP/row/HEADING (see AGS4 to DIGGS mapping) |
convert.orphan-row, convert.incomplete |
error | AGS4 to DIGGS conversion only. orphan-row: rows that name a parent record (a location, a sample, a test) that was not converted. incomplete: rows not converted because a limit or the time budget stopped the conversion, or because DIGGS requires something of the record that the source does not give |
convert.unmapped-element |
warning | DIGGS to AGS4 conversion only (pydiggs.convert.diggs_to_ags4, model_to_ags4): a value of the DIGGS model (a name or remark, a params entry, a typed field, an extra element, a link, or a result-table cell) has no AGS4 target in the mapping table, or cannot be written there as it is (another unit than its heading's, a line break). One finding per element and field, with a count; location.xpath is the first node's |
diff.ambiguous-match |
info | Model comparison only (pydiggs.diff.compare): several nodes of one collection share a natural key (for example two boreholes with the same name and no matching gml:id), so none of them was matched; they are listed as removed and added. At most 50 are written, then one note counts the rest |
diff.limit, diff.timeout |
warning | Model comparison only: the comparison stopped at max_changes (the list is cut; the counts stay exact), at its time_budget (the counts are partial), because a file was read in part, or because a model was read without the values of its result tables (results=False), so their cells were not compared. The result is incomplete and cannot claim the files are identical |
convert.value-reformatted |
info | AGS4 to DIGGS conversion only: the same value was written in another form (a date in ISO 8601 form, a number without trailing zeros); not a loss |
context.xlink-remote-unresolved |
info | An xlink:href into another document. pyDIGGS never fetches it, so what it points at was not checked. Once per document; links to DIGGS code lists (diggsml.org/def/codes/) are left to the dictionary step |
rule_catalogue() returns this table as data (RuleInfo objects: fixed ids, then the
open families with a pattern), and explain_rule(rule_id) describes one id; a
well-formed id pydiggs never emits gets known=False. A test keeps the severities here
and in the catalogue the same.
Plain-language guidance¶
Every rule in rule_catalogue(), and every id explain_rule() knows, also carries
guidance: a short title, what the rule means for the data, and how to fix it, written
for geotechnical and geoenvironmental engineers rather than DIGGS specialists. It is a
Guidance object (title, meaning, fix, locale), and None only for an unknown
id. explain_rule() gives an id its own guidance when it has some (the common
schema.cvc-* rules, such as schema.cvc-complex-type.2.4), else its family's.
from pydiggs import explain_rule
g = explain_rule("dictionary.5").guidance
g.title # 'Unknown code'
g.meaning # 'A value uses a code that is not in the DIGGS code list it names, ...'
g.fix # 'Pick a code from the list, checking spelling and capital letters, ...'
The findings document does not carry guidance, so diggs-findings stays at 1.2: an app
shows a finding's guidance by looking up its rule_id with explain_rule() (or
GET /v1/rules/{rule_id}, rules.explain, or the MCP explain_rule tool, which return
the same guidance object). The text lives in pydiggs/findings/guidance/en.toml; only
English ships today, and a test fails if a rule has no guidance or an entry matches no
rule.
Dictionary check severities¶
| Rule id | Severity |
|---|---|
dictionary.1 |
info |
dictionary.2 |
warning |
dictionary.3 |
warning |
dictionary.4 |
error |
dictionary.5 |
error |
dictionary.6 |
error |
dictionary.7 |
info |
dictionary.8 |
error |
dictionary.9 |
error |
dictionary.10 |
error |
dictionary.11 |
error / warning (warning for a dimensionless quantity) |
dictionary.12 |
error / warning |
Differences from the legacy checks¶
validate() differs from the legacy validator in these ways. Some of them (Check 13,
Check 8, deprecation) mean a file can pass dictionary_check() and still have errors
in validate():
- Dictionary Check 13 (
dictionary.13), a port of the officialdictionary-validation.xslcheck: for a property whose dictionary definition names avalueCodeSpace, every value in that property's column of the result set'sdataValues(and of every result set citing the same parameters byxlink:href) must be a term (gml:id) of that code list. Empty values and the property'snullValueare allowed;Property/@indexis read as anxs:integer(one longer than 64 characters, which no real column needs, is not a column). The message lists the first five invalid values and at most 20 valid terms (the official module lists all of them).dictionary_check()does not run it. - Dictionary Check 8 reads
//in a definition's conditional paths as "any descendant", as XPath does.dictionary_check()readsancestor::diggs:measurement//diggs:procedureas "procedureis a direct child ofmeasurement", which is never true in real files (the path ismeasurement/Test/procedure), so it skips Check 8 for properties under a measurement. In the test corpus this adds twodictionary.8errors invalidate()for each of the two AtterbergExample files ("uscs group symbol" and "natural water content" are not allowed in that type of measurement);dictionary_check()still passes them (1.1 behaviour, kept byte for byte). The same reading applies to Check 6 source paths of the formancestor::…//…(none of the bundled dictionaries has one). Predicates in definition paths ([text()="…"]) are ignored, as before, so matching is lenient; invalidate()a/inside a predicate no longer splits the path into steps (dictionary_check()keeps splitting it). - Geometry SRS.
validate()follows the official geometry module (modules/geometry-validation.xsl, DIGGSml/validation commit5f7a365). It checks the same nine objects:LinearExtent,MultiPointLocation,MultiCurve,Solid,PointLocation,PlanarObservationRepresentation,PlanarSurface,MultiPlanarSurfaceandRectifiedGrid, with or without coordinates (so an empty<LinearExtent gml:id="l"/>is checked;gml:Point,gml:LineStringand other GML primitives are not). An object passes when it or an ancestor has bothsrsNameandsrsDimension(present, even if empty); when only a child (gml:pos,gml:posList) has them, as GML 3.2 allows, it gets one best-practice warning; otherwise one warning names what is missing. The official module reports that last case, an object with no SRS on itself, any ancestor or any child, as an error; pydiggs keeps it a warning, for coordinate-less objects too. The legacycontext_check()keeps its own list (Point,LineString,LinearExtent,Polygon,Curve,Surface,MultiPoint,MultiCurve,MultiSurface,PointLocation,MultiPointLocation), skips objects with no coordinates and nosrsName, and gives one warning per missing attribute on the object itself, soDocumentationExample.xmlgets 62 there and 31 invalidate(). Across the 73 corpus files (the pydiggs test files and the desktop app's test data) the new list changes one verdict: the 3.xAeromag.xmlgoes from valid to valid-with-warnings (itsRectifiedGridhas SRS only on a child). - Local dictionaries.
validate(local_dictionaries="instance-dir")(the default; CLI--local-dictionaries) reads a dictionary named by a relative orfile:codeSpace only if it is a file inside the instance's folder: there is no fallback to the working directory, and..or a symlink cannot leave the folder. Anything else, including a path the operating system rejects, is reported asdictionary.3(ordictionary.13for a Check 13 value code list; warning, "could not be accessed") with a note that says why; the check is made before the file is opened, so the finding is the same whether or not the file exists. Each file is read once per instance however its path is spelled, undermax_bytesand only if it is a regular file, and at most 16 different local dictionaries are read per instance. All local dictionaries of an instance share one byte budget: 20 MiB, or whatmax_bytesleaves after the instance if that is less. A dictionary left unread by the cap or the budget isdictionary.3with a note and makes the stepincompletewith adictionary.incompleteerror. A local dictionary that is simply not well-formed XML (no safety limit involved) is adictionary.3warning, "could not be accessed", by design: that is what the official validator reports, and whoever controls the dictionary could as well write a permissive one."off"reads no local dictionary.dictionary_check()keeps the 1.1 behaviour: next to the instance, then the working directory, and anyfile:URI. dictionary_path. Invalidate()the dictionary you pass serves bothproperties.xmlURLs (https and http), the legacyDIGGSTestPropertyDefinitions.xmlURLs (which are looked up in it first) and the fallback for any other…/properties.xml.dictionary_check()keeps 1.1 behaviour: the bundledcodes/properties.xmlis registered after it under the same URLs and replaces it.- Offline dictionaries.
validate()reports an unbundled remote dictionary asdictionary.offline(info) when network access is off;dictionary_check()still reports Check 3 as a warning. - Deprecation (
deprecation.*), from the bundled DIGGS deprecation registry (deprecation/DeprecationRegistry.xml): deprecated items are warnings and removed items errors, with the official message text. An entry applies only from the release it names (since, orremovedInfor removed items), so the registry's 3.1.0 entries never fire on 3.0.0, 2.6 or 2.5.a files; the official module does not make this distinction.
SARIF¶
--format sarif / report.to_sarif() writes SARIF 2.1.0 (validated in the test suite
against the official OASIS errata01 schema):
| SARIF | From |
|---|---|
one run per layer |
layer (run.properties.layer) |
tool.driver.rules[] |
every rule id in the run, with shortDescription, helpUri and defaultConfiguration.level (the most severe level seen) |
artifacts[] |
each file, uriBaseId: SRCROOT, with its sha-256; a file: URI only with absolute paths |
results[].level |
error → error, warning → warning, info → note |
results[].locations[].physicalLocation.region |
only when the line is known (startLine, and startColumn for syntax errors) |
results[].locations[].logicalLocations[] |
{fullyQualifiedName: xpath, kind: "element"} ("attribute" for an attribute path); for a finding with no xpath but a location.path (AGS4), {fullyQualifiedName: path, kind: "element"} |
results[].partialFingerprints["diggsConformanceKey/v1"] |
the conformance key, step|severity|location|rule_id |
invocations[].toolExecutionNotifications |
one error per unreadable input |
Conformance keys¶
Finding.conformance_key() returns (step, severity, location, rule_id), where location
is "xpath:" plus the xpath with namespace prefixes dropped and every position made
explicit; else, for a finding in a source that is not XML (diggs-findings 1.1), "path:"
plus its path, followed by ";line:<n>" when the line is known; else "line:<n>", else
"file". Two layers agree when their keys match as multisets over the steps both ran.
The message is never compared.
Security notes¶
- Schematron runs with
generate-paths=false: lxml builds no@locationstrings (they copied the instance's namespace URIs, cost memory per failed assert and time per sibling). pydiggs locates each failed assert from the trusted schema instead, read from the same included and expanded schema lxml compiles, with the active phase applied: the n-thsvrl:fired-ruleof a pattern belongs to the n-th element, in document order, matched by one of that pattern's rule contexts (the first matching rule wins). The pairing is checked: theactive-patternid, the number offired-ruleelements and each one's context must agree with the elements found, or that pattern's failures have no location (never a wrong one). Contexts that match attributes, text or the root are never located. Nothing from the instance is evaluated as XPath. libxml2 error-log paths are checked against their/name[n]grammar and resolved by walking the tree. - A broken user Schematron file (a test that does not compile, or an unknown function)
is a
ResourceError(CLI exit 2), not a finding. That includes an XSLT error that only some instance triggers while the rules run: your rules are trusted input, so such an error is reported as a broken resource and stops the batch rather than being blamed on the file. Test your rules on representative files before using them on uploads. - Messages about an unreadable file do not contain its path; the report shows the path as described in Paths.
Residual risks¶
pydiggs's own code is bounded. On the security reviews' test files (up to 10 MB raw,
default settings) no pydiggs-owned path took more than about 11 s of CPU (1,000
conditional paths tried against 180,000 elements, stopped by the work budget) or about
350 MB beyond parsing (ancestor-name sets under a 250-deep chain with 260,000 parents).
The geometry rule is one pass over the tree (250 nested MultiCurve around 170,000
PointLocation, 9 MB: 1.4 s, where walking each object's ancestors and subtree took
99 s). Check 13 reads each ResultSet once for all the Properties of the
PropertyParameters it holds (its nearest ResultSet, at any depth) or cites by
xlink:href (400 of them over 2,000 rows of one 2 KB field: 1.4 s, where it took
44 s), and a column's values over a group's ResultSets are merged once for all the
group's properties. The security review's 10 MB shapes are linear or stop at the
budget: 55,000 ResultSets citing a 30,000-Property group, 1.4 s; 500 ResultSets citing
one group up to 500 times each, 0.6 s; 20,000 PropertyParameters outside parameters
over 7 million separators, 1.0 s (the first and last of these shapes ran for more than
460 s before, the last growing to 3 GB); 50,000 coded Properties read from 50,000
citing ResultSets, 2.0 s; 200 ResultSets each citing the same 400 groups of 400
columns, incomplete after about 5 s. Each index belongs to its nearest
PropertyParameters only, so 250 nested PropertyParameters around 480,000 index holders
(9.4 MB) take 1.5 s and about 240 MB, where re-reading every holder for each enclosing
group took 102 s and 3.7 GB. Check 13 keeps no rows, only the distinct values of
the columns its properties ask for, and one column set per group of
PropertyParameters (2,200 ResultSets of 2,200 cells citing one group, 9.9 MB: about
205 MB beyond parsing, where keying the sets by ResultSet used 718 MB; one 9 MB row
of 9 million empty fields takes 0.2 s and 113 MB, where reading every column took
2.1 GB). Groups read by the same ResultSets share one merge (300 groups each cited by
the same 300 ResultSets, 2,700 distinct values each: 1.3 s and 171 MB); when every
group is read by a different set of ResultSets, the merges are counted against the
kept-values cap instead and the step stops incomplete (1.5 s, about 250 MB).
The costs below belong to lxml and libxml2/libxslt and are not bounded by pydiggs. Measured on an Apple M3 Max MacBook Pro in CPU seconds (one process, default settings, lxml 6.1.3, libxml2 2.14.6, libxslt 1.1.43; wall time on a busy machine is higher):
| Cost | Input | CPU |
|---|---|---|
| Schematron (libxslt; grows with the square of the number of elements the rules fire on, whatever their parents) | 1.8 MB, 100,000 fired elements, all failing (the most validate() runs) |
3.4 to 5.5 s |
| 1.7 MB, 100,000 fired elements, all passing | 3.6 s | |
| 9 MB, about 1 million fired elements (21 parents × 50,000) | 174 s and 1.7 GB before the cap; now refused (schematron.incomplete) in 1.7 s |
|
| 9.9 MB, 100,000 fired elements among 1.1 million same-named elements the rules do not fire on (under both caps) | 35 to 43 s and about 500 MB; 22 to 29 s with Casing |
|
| 9 to 10 MB with no element a rule fires on | 5 to 6 s | |
| 1 MB, 18,000 sibling features (no failures) | 0.1 s (31 s with the pre-1.2 rule contexts) | |
| Schema validation (lxml builds an XPath for every libxml2 error by counting siblings) | 2 MB of sibling schema errors (37,000 siblings) | 14 s |
| 4 MB of the same | 107 s | |
| 8 MB of the same | over 400 s | |
| Parsing (the lxml tree) | 42 MB, 10.5 million elements | 1.4 GiB, 1.4 s |
| Local dictionaries (parsed trees) | 20 MiB of <b/> elements, the most the budget allows |
about 800 MB |
The bundled rules' contexts were rewritten for 1.2 (*[local-name()='C'][parent::*[local-name()='P']]
instead of //*[local-name()='P']/*[local-name()='C']; same elements, same verdicts).
The old form made libxslt quadratic in the number of siblings of any element (over
400 s on several 8 to 11 MB files). No form tried is linear in both that and the
number of elements a rule fires on, so validate() caps both: more than 50,000
children under one element, or more than 100,000 fired elements, skips Schematron
with a schematron.incomplete error.
Entities are bounded relative to the raw file. An internal DTD entity with markup is
refused (structure.markup-entity) before any check runs, in every encoding the parser
reads: usually decided from the prolog alone, and for UTF-32 with a BOM, which lxml's
push parser cannot read, after one parse that expands no entity. A file whose
plain-text entities make its text more than twice its raw size is refused after the
parse (structure.entity-expansion); the parse itself is libxml2's, which limits
entity amplification to a factor of about 5 and any single text, comment, PI, CDATA or
attribute node to 10 MB (without huge_tree, which pydiggs never sets), so the
largest parse pydiggs can be asked to do is bounded by the raw size before its own 2x
check runs. A 9.6 MB file that would expand to 12 million
elements is reported in 0.1 s and 54 MB, where 1.2 development builds used 1.7 GB;
dataBlocks of entity-expanded text that took 9 s are refused in under 0.1 s. Local
dictionaries with markup entities are refused the same way (dictionary.incomplete);
one 20 MiB dictionary that expanded to 26 million elements used 3.4 GB before.
Services that validate files from untrusted sources should therefore validate each
upload in its own folder, in a separate process with a time and memory limit.
pydiggs validate --timeout SECONDS gives a time limit (a file that runs out of time,
or whose worker dies, is a structure.timeout error) and --max-bytes keeps inputs
small. Also:
--timeouthas a floor. Each file gets a fresh interpreter (thespawnstart method, on every OS), which imports pydiggs and lxml and compiles the bundled schemas before validating: about 0.3 s of CPU for a small file, and several seconds of wall time on a loaded machine. The limit includes that start-up, so values under about 10 s can stop files that would pass; use 30 s or more on shared machines.spawnre-imports your main module. A script that callspydiggs.cli.main(["validate", "--timeout", ...])must do it underif __name__ == "__main__":, or each worker runs the script again. Thepydiggscommand needs nothing.- No memory limit.
--timeoutlimits time only. Workers have no memory or CPU rlimit, run with your privileges and inherit your environment; set memory limits with the operating system or the container (for exampleulimit -v, a cgroup, or a Kubernetes memory limit). instance-dirreads any file in the instance's folder as a dictionary when a codeSpace names it: in a shared upload folder that includes other users' uploads, and a hard link to a file elsewhere cannot be told apart from a real file.- The legacy entry points (
validator,pydiggs schema_checkand the other subcommands,detect_diggs_version) keep their 1.1.1 behaviour by design: local dictionaries next to the instance, then in the working directory, and anyfile:URI; a 1 GiB size limit; internal DTD entities expanded; Schematron with lxml's location strings; element paths that cost time in proportion to depth times the number of findings; a local dictionary parsed once per spelling of its path; and adictionary_paththat the bundledcodes/properties.xmlreplaces. They use the same rewritten bundled Schematron rules, so their verdicts and output are unchanged. Usevalidate()for untrusted input.
Privacy¶
Findings can travel further than the file: into CI logs, issue trackers and code-scanning dashboards. By default pydiggs keeps them free of the file's content and your file system layout:
- No snippets by default.
source_snippetcopies the element's XML, which is client data; it appears only with--snippets/include_snippets=True, capped at 1 KiB. - No absolute paths by default. Paths follow Paths in JSON, SARIF and
text alike, so running from
/or above your home directory never spells out your directory layout; messages never contain the absolute path of the instance.--absolute-pathsopts in. - No timestamps, host names or user names in either format.
- No network unless
--allow-network/allow_network=True, and then only to whitelisted dictionary URLs.
Messages can still quote values from the file (a code that is not in a dictionary, for example). Treat a findings file as you would treat the DIGGS file itself.