Reproducible screening geoprocessing with ArcPy
Representative Stage 0 component- Question
- How does a screening step stay reproducible, auditable, and safe to hand to the next stage?
- Method
- A declared processing environment, in-memory intermediates, an explicit join cardinality, and a quality gate that checks coordinate reference system, geometry validity, join cardinality, and unmatched features before promotion.
- Data
- Parcel polygons, facility points, statutory radius.
- Implementation
- ArcPy with
arcpy.EnvManager; NAD83 / UTM Zone 18N for the Washington, DC pilot area. - Result
- A screened parcel set whose lineage record states exactly what was checked.
- Validation
- The gate fails loudly; nothing enters Stage 1 without passing it.
- Code / Demo
- Below
import arcpy
def screen_parcels(parcels, facilities, radius_m, out_fc):
# Stage 0 statutory screen in a fixed projected CRS: NAD83 / UTM 18N for the DC pilot.
with arcpy.EnvManager(outputCoordinateSystem=arcpy.SpatialReference(26918),
overwriteOutput=True):
buf = arcpy.analysis.Buffer(facilities, "memory/buf", f"{radius_m} Meters",
dissolve_option="ALL")
arcpy.analysis.SpatialJoin(target_features=parcels, join_features=buf,
out_feature_class=out_fc, join_operation="JOIN_ONE_TO_ONE",
match_option="INTERSECT")
return out_fc
def qa_gate(fc, expected_epsg=26918):
# Every stage boundary carries the same checks; a failed gate stops the pipeline.
sr = arcpy.Describe(fc).spatialReference
assert sr.factoryCode == expected_epsg, f"CRS {sr.factoryCode} != {expected_epsg}"
bad = [r[0] for r in arcpy.da.SearchCursor(fc, ["OID@", "SHAPE@"]) if r[1] is None or not r[1].area]
assert not bad, f"{len(bad)} null or degenerate geometries"
unmatched = int(arcpy.management.GetCount(
arcpy.management.MakeFeatureLayer(fc, "lyr", "Join_Count = 0"))[0])
arcpy.AddMessage(f"{fc}: CRS ok · geometry ok · {unmatched} parcels outside the screen")
- Purpose
- Normalize spatial relationships inside a reproducible processing environment: one declared coordinate system, in-memory intermediates, and a join whose cardinality is explicit.
- Validation
- Verify the coordinate reference system, geometry validity, join cardinality, and unmatched features before the output enters the next stage; the message line becomes the lineage record.
Ingestion workflows with provenance
CBO–TGRA prototypes- Question
- How do 42+ sources with different schemas and cadences enter one canonical model without losing where each fact came from?
- Method
- Per-source KNIME ingestion workflows normalize to the canonical representation and stamp provenance, version, and record time; a parameter registry governs every threshold.
- Data
- Federal, regulatory, commercial, open-source, and geospatial sources.
- Implementation
- KNIME workflows with Python and ArcPy nodes for proximity, spatial joins, network accessibility, terrain analysis, visibility, change detection, scoring, and QA.
- Result
- Any assessment reconstructs backward through fusion logic, stage score, modality output, parameters, evidence, and source.
- Validation
- ISO 19157 quality gates on positional accuracy, schema conformity, attribute completeness, duplication, provenance completeness, and stage completeness.