Reading DIGGS files¶
Stable from pyDIGGS 1.7.0
pydiggs.read() returns diggs-model 1.0, which changes only as described under
Versioning. pyDIGGS 1.6 had the same API as a preview in
pydiggs.preview; that module still works, with a DeprecationWarning, until 2.0.
The same document is available over HTTP (POST /v1/read),
from the stdio sidecar (read) and in the browser
(Pyodide), and to AI assistants through the MCP server's
read tools (1.8.0).
Comments are welcome as GitHub issues.
pydiggs.validate() tells you whether a file is correct. The read API gives you what is
in the file: boreholes, samples, tests and their result tables, in the same shape for
DIGGS 2.5.a, 2.6 and 3.x, as Python objects and as JSON.
- JSON Schema:
schemas/diggs-model-1.0.schema.json(alsopydiggs.model_schema()). The betas pyDIGGS 1.6 wrote,1.0-beta.1and1.0-beta.2, stay published. - Example document (from the draft):
schemas/preview/diggs-model-1.0-draft.example.json
Transcript
The script. Here is a short Python script. It imports pydiggs, validates site dot X M L, and prints the verdict and the counts. Then it reads the same file into a model and prints the kind, name and total depth of each sampling feature.
Run it. Running the script prints valid, with zero errors and zero warnings. Below that are the three sampling features in the file: two boreholes and a trial pit, each with its depth exactly as the file writes it, and its unit.
Try it yourself
The files this video starts from, exactly as it used them: python-api-files.zip (3.7 KB, SHA-256 182aeba596fa2f2d1bed5c2c32af45c953bf6b1d4a177aec80db6eb9f3298758), or one by one: site.xml, check_site.py. They are synthetic examples, Apache-2.0.
Set up a folder with uv and the version the video shows, put the files in it, and run each command there (on Windows, activate with .venv\Scripts\activate):
Reading a file¶
import pydiggs
from pydiggs.model import Measurement
m = pydiggs.read("site.xml") # or pydiggs.read_bytes(data, name="site.xml")
for f in m.features:
print(f.key, f.kind, f.name, f.total_depth) # B-01 borehole B-01 Quantity(text='160.00', …)
bh = m.get("Location_B-01")
print(bh.params["boreholeType"]) # Code(value='Borehole', code_space='https://…#BH')
for c in bh.construction: # construction methods, casing, backfill …
print(c.role, c.interval)
for test in m.related(bh, relation="samplingFeature", direction="in"):
if not isinstance(test, Measurement): # sampling activities refer to the hole too
continue
print(test.procedure.type, dict(test.procedure.params)) # tipArea, hammerEfficiency …
for table in getattr(test, "results", ()):
depth = table.domain.index # e.g. the CPT depth of every row, in metres
n = table.column("n_value") # find columns by dictionary key
if n is not None:
print(n.values)
text = m.to_json() # a diggs-model document; the same model always gives the same bytes
again = pydiggs.Model.from_json(text) # == m
read(path, *, diggs_version=None, results=True, provenance="object", max_bytes=100 MiB,
time_budget=None):
results=Falsekeeps every table's columns androw_countbut no values: a quick inventory of a large file.provenance="field"also records where each value came from: every node'sprovenance.fieldsmaps the element a value was read from (its local name,name[2]for a second one) to its xpath relative to the node and its line. It makes the JSON larger, so it is off by default; every node always has its ownxpathandline.time_budget(seconds) returns what was read so far, withcomplete=False, when time runs out.- Work limits. Reading also stops, with
complete=Falseand the limit's name inlimits_hit, when a file has more objects (nodes), or more elements the model has no typed field for (extra_elements), than 100,000 plus one for every 200 bytes of the file, or nests such elements more than 64 deep (extra_depth). Real files have a few thousand of each, over 1,000 bytes apiece and about 10 deep; the limits keep a small hostile file from producing a huge model. The member being read at that point and the ones after it are left out. - Errors:
InstanceReadErrorwhen the file cannot be read (as forvalidate);DocumentError(aValueError) when the bytes are not a DIGGS document at all, withkindmalformed,not-diggs,unsafe-entityornamespace-too-longand the matchingstructure.*finding. Everything else becomes a note. - Nothing is fetched from the network, and the file is parsed one top-level member at a time, so memory stays close to the file's size.
Data frames and GIS¶
Three extras turn a model into the tables and layers other tools use. Nothing in them is
imported by import pydiggs.
| Extra | Gives | Installs |
|---|---|---|
pydiggs[frames] |
pydiggs.frames.to_pandas() |
pandas |
pydiggs[polars] |
pydiggs.frames.to_polars() |
polars |
pydiggs[gis] |
pydiggs.gis.transform(), to_geodataframe(), write() |
geopandas, pyproj, shapely, pyogrio, pyarrow |
import pydiggs
from pydiggs import frames, gis
m = pydiggs.read("site.xml")
frames.to_pandas(m, "features") # one row per borehole, sounding, well …
cpt = frames.to_pandas(m, "results", procedure="StaticConePenetrationTest")
cpt[["position", "tip_resistance"]] # columns are named by dictionary key
boreholes = gis.to_geodataframe(m, kind="borehole") # points in WGS 84
gis.write(m, "site.gpkg") # or .parquet (GeoParquet), .geojson
Views. to_pandas(model, view) and to_polars(model, view) take the same five views:
| View | One row per | Filters |
|---|---|---|
features |
sampling feature, with its location as written and lon/lat when known |
kind |
samples |
sample, with its depth interval, type, method and recovery | feature |
layers |
lithology layer, with its interval, description, classification and colour | feature |
measurements |
test, monitor or survey, with its procedure and how many result rows it has | feature, kind, procedure |
results (default) |
row of every result table: where it is, then one column per result | feature, procedure, measurement, si |
In results, tables with different columns share one frame and a cell a table does not
have is null, so filter by procedure (or measurement) for one tidy table. si=True
gives values in SI units where the unit converts. Units are not in the frame: they are in
the model (table.column("tip_resistance").uom). The columns position, top, base,
time and elapsed_time say where a row is; a result column with one of those names
(a dissipation test's measured elapsed_time, say) is always <name>_value.
Coordinates. pydiggs.read() never transforms a coordinate. gis.transform(model,
to="EPSG:4326") returns a copy in which every position whose srsName names an EPSG or
OGC CRS has geographic (longitude, latitude) in the CRS you ask for, and
transformed_by says which pyproj did it. coordinates and crs stay as written. A
position in a CRS defined in the file, or in one pyDIGGS cannot resolve, is left alone,
and such a feature has an empty geometry in to_geodataframe() and write(). pyproj
chooses the transformation; without its datum grids, a shift between datums (NAD27 to
WGS 84, say) can be off by metres.
gis.write() saves the features as points: GeoPackage (.gpkg, layer features),
GeoParquet (.parquet) or GeoJSON (.geojson). It replaces an existing file.
Principles¶
- Reading is not validating.
readworks on invalid files and never says "valid". Runvalidate()for a verdict. What reading had to work around (a link that goes nowhere, a number that is not a number) is recorded innotes, as findings. - What has no typed field is kept. Every element the reader has no typed field for
is kept, in document order, in the node's
extralist. Coverage grows in minor releases by moving content fromextrainto typed fields. An element a typed field reads from is not kept inextra, and the model holds what the field reads and not the rest of that element: a location'suomLabelsandsrsDimension, the geometry of a lithology layer, a grid or a data package under a result'soutcome, a column's declared type, a value that is not a number in a numeric column (read as null, with a note). A model is therefore not a copy of the file. To compare two files in full, givepydiggs.diffthe files, not their models. - Values are kept as written. A quantity keeps its
textexactly as reported, its parsedvalueanduom, and ansivalue only when the unit converts. Coordinates stay in the file's CRS and axis order. Nothing is converted or transformed silently. - Nothing is fetched. Links to other files stay unresolved, with a note.
- One key per node.
keyis thegml:idwhen it is present and unique, otherwise"@"plus the node's XPath.provenance.xpathuses the same form as a finding's location, so a finding fromvalidate()can be traced to the node that contains it.
The document¶
A diggs-model document is one JSON object:
| Field | What it holds |
|---|---|
schema_version |
1.0 (pyDIGGS 1.7 and later; 1.6.0 wrote 1.0-beta.2) |
tool, source |
pyDIGGS version; file name, sha256, size, DIGGS version and namespace |
complete, limits_hit |
false and the limit's name (time_budget, nodes, extra_elements, extra_depth; an open list) when a limit stopped reading |
document, projects, associates |
Document information, projects, business associates |
features |
Sampling features: boreholes, soundings, wells and every other kind |
sampling_activities, samples |
Sampling activities and samples |
observations |
Observation systems, with lithology layers typed |
measurements |
Tests, monitors and surveys, each with a procedure and result tables |
others |
Top-level members with no typed mapper yet; a top-level reference is kind reference, and a property of the file itself (gml:name, gml:description) is kind property with its value in params |
crs |
Every coordinate reference system used, keyed by its srsName |
links |
Every link between nodes: source, relation, target, href, resolved |
notes |
Findings about reading, in the diggs-findings shape |
Every node starts with the same header: key, gml_id, element (the DIGGS element
name), kind (for example borehole), name, names, description, remarks,
params, provenance and extra.
Simple values: params¶
Most of what DIGGS says about an object is simple values: a status, a purpose, a hammer
efficiency, a tip area. Every child element that has no child elements of its own,
appears once, and carries no attribute but uom or codeSpace is kept in params,
by its DIGGS element name, as a Quantity (it had a uom, so it has a parsed value and,
when the unit converts, an SI value), a Code (it had a codeSpace) or text. This
applies to every node and every procedure, so every test has typed parameters, not only
the ones pyDIGGS knows.
What is not a simple value stays in extra: repeated elements (all copies, in order),
elements with other attributes (howDetermined, refAzimuth …), empty elements and
structured ones. A plain reference (<projectRef xlink:href="#p1"/>) is one of the
model's links and is not repeated in extra.
Nested records: components¶
Records nested inside a feature, a lithology or a procedure are Component nodes: a
casing, a backfill, a water strike and its readings, a well opening, an SPT drive set,
the cone of a CPT, a sieve analysis, a constituent. Each has the full header (with its
own key, so links into it resolve to it), a role (the DIGGS property that holds it:
casing, waterStrike, driveSet …), the interval, location and time it applies
to when it says, its params, and its own nested components. Model.nodes() and
Model.get() include them.
Result tables¶
Every DIGGS result set has the same structure in 2.5.a, 2.6 and 3.x: property
definitions and a block of values. The model turns each into a table with a domain
(point, interval, temporal, grid), columns and a row_count. A column carries its
name, its dictionary key (for example liquid_limit), uom, type and values, and
the factor to SI where the unit has one. Tables are generic for every procedure, so a test
with no typed mapper still gives you its numbers.
Where each row is lies outside the values, in the result's location or time domain. The
model puts it in domain.index, columns of the same length as the table:
| The result gives | domain.index |
Example |
|---|---|---|
| a multi-point location, one position per row | position (position_1 … for more axes) |
CPT depths |
| an extent with a top and base for every row | top, base |
stepped tests |
| a list of times | time (datetime text) |
water levels |
| a list of elapsed times | elapsed_time, in the list's unit |
pore-pressure dissipation |
When the whole table was measured at one place instead, it is location (a point) or
extent (an interval, such as an SPT's test interval). domain.crs names the system
the index is in.
Notes¶
Notes use rule ids that validate() already reports, so one explanation covers both
(pydiggs explain <rule>):
| Situation | Rule id |
|---|---|
| A link to an id that is not in the file | context.xlink-href-resolves |
An srsName that is not a known CRS or an id in the file |
geometry.crs-unresolvable |
| Coordinates that look like longitude first | geometry.crs-axis-order-suspect |
| A link into another file | context.xlink-remote-unresolved |
| A value that is not a number where one is expected | structure.value-unparsed |
Two or more elements with the same gml:id |
structure.duplicate-gml-id |
| A result row longer or shorter than its columns | context.datablock-arity |
| A time limit stopped reading | structure.timeout |
| A work limit stopped reading | structure.read-limit |
Coordinate reference systems¶
m.crs describes every srsName the file uses: its form (epsg, diggs, ogc,
local or unknown), name, horizontal and vertical components, axis_order,
unit, and known (whether pyDIGGS found what it names). The same facts are available
directly:
from pydiggs import crs
projected = crs.epsg(2272)
projected.name # 'NAD83 / Pennsylvania South (ftUS)'
projected.unit # 'US survey foot'
projected.axis_order # 'east,north'
crs.parse("urn:ogc:def:crs:EPSG::5498").vertical.name # 'NAVD88 height'
- EPSG facts (name, type, axis order, unit, datum; no WKT, no transformation
parameters) come from a subset of the EPSG Dataset packaged with pyDIGGS: every
geographic and vertical CRS, every CRS used in the United States and its territories,
the WGS 84 and ETRS89 UTM zones, and every code in the conformance corpus.
crs.epsg_source()gives the dataset version. A code outside the subset still reads; it just has no name or unit (knownisfalse). - Positions keep their coordinates as written. For a geographic CRS,
geographicgives latitude and longitude in the CRS's axis order and its datum; a value beyond 90° where latitude should be is read the other way round, with a note. - Depths. A borehole's depths are written as bare numbers along its linear
referencing system (
srsName="#lsr-…"). When that system states itsunitsin the file, or refers to a method in the DIGGS registry (lrm.xml#md_ft,#md_m,#core_top_down_cm…, packaged with pyDIGGS), intervals and index columns get that unit (and an SI value). - Nothing is transformed or fetched; use pyproj for that.
The EPSG Dataset is owned by the International Association of Oil & Gas Producers (IOGP). pyDIGGS includes facts from it, unmodified, under the EPSG terms of use; the full dataset is at epsg.org.
Versioning¶
schema_version is separate from the pyDIGGS version.
- Minor (1.x): new optional fields, new definitions, new values of the open
vocabularies (
kind,procedure.type,relation,domain.shape,column.type), and content moving fromextrainto typed fields. - Major: anything else. Both majors are then produced side by side for at least one pyDIGGS minor release plus 12 months.
- The published schema is open. Objects accept unknown fields and vocabularies are
listed as
examples, notenum, so a consumer written for 1.0 accepts any 1.x document. pyDIGGS's own tests check its output against a closed variant. - The 1.0 schema is frozen and will never be edited; fixes become 1.1. The betas
remain under
schemas/preview/.
Coverage in 1.0¶
Every top-level member is read. Typed means named fields with tests; generic
means header, links, location and result tables, with the rest in extra.
| Area | Level in 1.0 |
|---|---|
| Document information, projects, business associates | Typed |
| Boreholes, soundings, wells (total depth, location, centreline, construction) | Typed |
| Other sampling features (trial pits, stations, piles, geophysical features) | Generic |
| Lithology observations and layers | Typed |
| Other observation systems | Generic |
| Sampling activities and samples | Typed |
| SPT, CPT, Atterberg limits, particle size, water level | Typed parameters and tables |
| Every other test | Typed params and components, generic tables |
What 1.0 reads¶
- Every top-level member, with its header,
params, links andextra; nothing is dropped. - Features:
location,centerline,total_depthandlrs;when_constructed;construction(method, casing, backfill, hole diameter, flush, advancement, events; a well's casing, openings and installation events) andwater_strikesas components; a well's or installation'sparenthole. - Sampling activities:
interval,method,recoveryanddate. Samples:type(their classification), and theinterval,methodandrecoveryof the activity that produced them. - Lithology: one layer per
LithologyObservation, withinterval,geologic_unit, its primary and component lithologies as components, anddescription,classificationandcolourfrom the primary lithology. - Measurements:
result_time; the procedure's type,paramsand nestedcomponents(SPT drive sets, CPT cone, sieve analyses …); every result table, for every procedure, with typed columns, dictionary keys, SI factors and index columns. Construction activities keep their result tables too (inothers). - Links, resolved to the node that holds the referenced element (a component when
the id is inside one), and the
feature,samples,sampling_activity,specimenandparentshortcuts built from them. - CRS: names, types, units, axis orders and datums from the EPSG subset; units of linear referencing systems, stated in the file or taken from the DIGGS registry.
- Provenance: every node's
xpathand line, and withprovenance="field"where each of its values was read.
Checking a reader¶
The conformance corpus
records, for each of its DIGGS files, what a reader must produce: members, procedures,
links by relation, result tables, rows and cells. The counts come from a plain parse of
each file, not from the pyDIGGS reader, so any reader can be checked against
conformance/model.toml: