Skip to content

Authoritative source release runbook

Use this runbook for a receipt-backed alternative such as usgs-3dep, cop30, cop90, nasadem, aw3d30, or aster-gdem-v3. The canonical USGS SRTM state workflow follows the same gates but uses acquire-state and the STATE.yml manifest.

An alternative source is always a separate dataset and hostname. It must never silently fill from, replace, or be presented as the canonical release. A baseline_policy of none is enforced in the release and tested at the public edge.

Refresh a canonical USGS SRTM state

USGS M2M sessions for one account are serialized. Do not run state acquisitions or access checks concurrently. A transient login or catalog failure may be retried from a new process; authentication POSTs are never retried automatically.

Preserve the active release and prior audit before starting. Acquire into a timestamped directory so an interrupted attempt cannot replace another source:

export STATE=NC
export RUN_ID=$(date -u +%Y%m%dT%H%M%SZ)
export SOURCE_RUN=/srv/dted-sources/authoritative/usgs-srtm/$RUN_ID/$STATE
export DATA_ROOT=/srv/dted-data
export RUN_UID=$(stat -c %u "$DATA_ROOT")
export RUN_GID=$(stat -c %g "$DATA_ROOT")
export IMAGE_ID=$(docker image inspect dted-portal --format '{{.Id}}')
export TOOL_REVISION=$(git -C /home/gba/Agents/DTED.org rev-parse HEAD)
sudo install -d -o "$RUN_UID" -g "$RUN_GID" -m 0750 "$SOURCE_RUN"
sudo install -o "$RUN_UID" -g "$RUN_GID" -m 0400 \
  /etc/dted/usgs.env /run/dted-usgs.env

Run the approved access check, then acquire the native SRTM DTED products:

docker run --rm --init --read-only \
  --user "$RUN_UID:$RUN_GID" \
  --cap-drop ALL --security-opt no-new-privileges \
  --tmpfs /tmp:rw,nosuid,nodev,noexec,size=1g,mode=1777 \
  -e DTED_TOOL_REVISION="$TOOL_REVISION" \
  -e DTED_BOUNDARY_CACHE_DIR=/sources/boundaries \
  -v /run/dted-usgs.env:/run/secrets/usgs.env:ro \
  -v "$SOURCE_RUN:/sources" \
  "$IMAGE_ID" dtedctl acquire-state "$STATE" \
  --output /sources \
  --manifest-dir /app/datasets \
  --credential-file /run/secrets/usgs.env
sudo rm -f /run/dted-usgs.env

The receipt must identify usgs_m2m, srtm_v3, nonempty scene and product identifiers, the exact archive hash, boundary hashes, download hashes, and the tool revision. Scan it without printing values:

python3 /srv/dted/tools/scan_secret_metadata.py \
  "$SOURCE_RUN/$STATE-DTED.receipt.json"
sha256sum "$SOURCE_RUN/$STATE-DTED.zip"

Audit first into a timestamped run directory under /srv/dted-data/quality/source-audit-runs. When it finishes, archive the prior canonical report under quality/source-audit-history and atomically install the new report and summary under source-quality/$STATE. A passing report may be built and staged using the normal candidate workflow. A failed report remains visible and must not be built with an override merely to call the refresh successful.

If authoritative SRTM retains upstream voids, keep the canonical live release unchanged and acquire usgs-3dep as a separate source-specific dataset. Use STATE-3DEP.yml, baseline_policy: none, and the independent preview-<state>-3dep.dted.org hostname. Never substitute 3DEP silently at the canonical state hostname.

Prepare the host

Run from /srv/dted on the production host. Use the owner of the data root for all generated files:

export REGION=CA
export SOURCE_ID=usgs-3dep
export DATASET_ID=ca-3dep
export DATASET_FILE=CA-3DEP
export DATA_ROOT=/srv/dted-data
export SOURCE_ROOT=/srv/dted-sources
export MANIFEST_ROOT=/srv/dted-data/source-manifests
export GRID_ROOT=/srv/dted-grids
export RUN_UID=$(stat -c %u "$DATA_ROOT")
export RUN_GID=$(stat -c %g "$DATA_ROOT")
sudo install -d -o "$RUN_UID" -g "$RUN_GID" -m 0750 \
  "$SOURCE_ROOT" "$MANIFEST_ROOT" \
  "$DATA_ROOT/source-quality/$DATASET_ID"

Confirm the image and pinned grids before acquisition:

docker image inspect dted-portal --format '{{.Id}}'
sha256sum "$GRID_ROOT/manifest.json"
docker run --rm dted-portal dtedctl list-sources

USGS 3DEP needs no provider credential. For a credentialed adapter, leave the root-owned source file at mode 0600, make a temporary owner-readable copy on the host's volatile filesystem, bind it read-only at /run/secrets/provider.env, and remove the copy immediately after acquisition. Never put credential values in an environment shown by docker inspect, a command argument, a receipt, or an operator note.

sudo install -o "$RUN_UID" -g "$RUN_GID" -m 0400 \
  /etc/dted/provider.env /run/dted-provider.env
# Add: -v /run/dted-provider.env:/run/secrets/provider.env:ro
# Add: --credential-file /run/secrets/provider.env
sudo rm -f /run/dted-provider.env

Acquire and resume

The following command writes the archive, credential-free receipt, generated dataset manifest, and content-addressed source cache:

docker run --rm --init --read-only \
  --user "$RUN_UID:$RUN_GID" \
  --cap-drop ALL --security-opt no-new-privileges \
  --tmpfs /tmp:rw,nosuid,nodev,noexec,size=2g,mode=1777 \
  -e PROJ_NETWORK=OFF \
  -v "$SOURCE_ROOT:/sources" \
  -v "$MANIFEST_ROOT:/manifests" \
  -v "$GRID_ROOT:/grids:ro" \
  dted-portal dtedctl acquire-region "$REGION" \
  --source "$SOURCE_ID" \
  --output /sources \
  --manifest-dir /manifests \
  --grid-manifest /grids/manifest.json

Add --credential-file /run/secrets/provider.env and its read-only bind mount only for a source that requires it.

An interrupted run can be repeated. The adapter reuses only assets whose cached content still matches the recorded SHA-256 and redownloads missing, changed, or damaged assets. Do not delete the raw cache merely to restart a job. Do not edit the output ZIP or receipt.

Verify the result without printing upstream links:

sha256sum "$SOURCE_ROOT/$DATASET_FILE-DTED.zip"
python3 /srv/dted/tools/scan_secret_metadata.py \
  "$SOURCE_ROOT/$DATASET_FILE-DTED.receipt.json"
python3 - "$SOURCE_ROOT/$DATASET_FILE-DTED.receipt.json" <<'PY'
import json
import sys
receipt = json.load(open(sys.argv[1], encoding="utf-8"))
print({
    "schema": receipt["schema"],
    "dataset_id": receipt["dataset_id"],
    "source_id": receipt["source"]["id"],
    "source_version": receipt["source"]["version"],
    "asset_count": len(receipt["assets"]),
    "archive_sha256": receipt["archive"]["sha256"],
    "processing_profile_id": receipt["processing"]["processing_profile_id"],
    "proj_network": receipt["processing"]["proj_network"],
})
PY

Expected invariants are schema 3, the requested source ID, a frozen upstream version and asset list, an archive digest matching sha256sum, a pinned processing profile, and proj_network: off.

Audit

Audit in the same image used for conversion:

docker run --rm --init --read-only \
  --user "$RUN_UID:$RUN_GID" \
  --cap-drop ALL --security-opt no-new-privileges \
  --tmpfs /tmp:rw,nosuid,nodev,noexec,size=1g,mode=1777 \
  -v "$SOURCE_ROOT:/sources:ro" \
  -v "$MANIFEST_ROOT:/manifests:ro" \
  -v "$DATA_ROOT:/data" \
  dted-portal dtedctl audit-dataset \
  "/manifests/$DATASET_FILE.yml" \
  --source "/sources/$DATASET_FILE-DTED.zip" \
  --receipt "/sources/$DATASET_FILE-DTED.receipt.json" \
  --report "/data/source-quality/$DATASET_ID/report.json" \
  --summary "/data/source-quality/$DATASET_ID/summary.json"

Stop if the summary does not say result: pass. Never alter a report or use an override to describe failed data as passing. Review, at minimum:

python3 - "$DATA_ROOT/source-quality/$DATASET_ID/summary.json" <<'PY'
import json
import sys
summary = json.load(open(sys.argv[1], encoding="utf-8"))
print({
    "result": summary["result"],
    "checks": summary.get("checks"),
    "failures": summary.get("failures"),
    "cells": summary.get("cells"),
})
PY

The full gate validates archive and receipt hashes, DTED headers and matrices, expected land coverage, zero land voids, exact shared edges, elevation bounds, and representative control points.

Build and stage

Build creates an immutable release but does not activate it:

docker run --rm --init --read-only \
  --user "$RUN_UID:$RUN_GID" \
  --cap-drop ALL --security-opt no-new-privileges \
  --tmpfs /tmp:rw,nosuid,nodev,noexec,size=1g,mode=1777 \
  -v "$SOURCE_ROOT:/sources:ro" \
  -v "$MANIFEST_ROOT:/manifests:ro" \
  -v "$DATA_ROOT:/data" \
  dted-portal dtedctl build "/manifests/$DATASET_FILE.yml" \
  --source "/sources/$DATASET_FILE-DTED.zip" \
  --quality-summary "/data/source-quality/$DATASET_ID/summary.json" \
  --receipt "/sources/$DATASET_FILE-DTED.receipt.json" \
  --output /data

Record the release ID printed by the build, then stage exactly that release:

export RELEASE=release-id-from-build
docker compose run --rm worker dtedctl preview-state \
  "$DATASET_ID" "$RELEASE" --output /data

Preview hosts are public by default. They carry X-Robots-Tag: noindex, nofollow, noarchive, but that is not an access-control boundary: stage only public-information candidates. An installation that deliberately replaces /etc/dted/qa_allowlist.conf with trusted QA CIDRs may use the bounded preview-window helper to expose one candidate temporarily.

curl -fsS -o /dev/null -w '%{http_code}\n' \
  "https://preview-$DATASET_ID.dted.org/"
python3 tools/smoke_test.py \
  "https://preview-$DATASET_ID.dted.org" --require-quality

The preview landing page, QR, preference, client JSON, streaming endpoints, and offline ZIP must all name the preview host. A source-specific smoke test also probes an uncovered cell and requires 404, proving that canonical data was not mixed in.

For a public, passing, source-specific preview, the regional source catalog automatically adds an evaluation entry:

curl -fsS \
  "https://dted.org/api/v1/regions/$REGION_ID/terrain-sources"

Confirm the corresponding https://dted.org/states/REGION/ page labels it Evaluation preview and links to the preview hostname with no production claim. Canonical refresh previews and failed candidates are intentionally not added. Promotion replaces the evaluation entry with the production source without rebuilding another state's release.

TAK validation and attestation

Validate the exact candidate on representative ATAK and WinTAK hardware:

  1. Import the QR or preference and verify it names the preview host.
  2. Confirm ATAK's native elevation tool can retrieve and display covered DTED.
  3. Test Simple DTED Streamer when installed; it is optional.
  4. Download and import the state's offline DTED data package.
  5. Check representative control points and an uncovered source-isolation point.
  6. Record client, plugin, OS, and hardware versions.
  7. Save a screenshot or redacted log. Do not include user IPs, tokens, or other operationally sensitive data.

Both root and /elevation/DTED/ prefixed per-cell routes are supported. HTTP delivery alone is not client evidence.

Record the two attestations:

docker compose run --rm worker dtedctl attest "$DATASET_ID" "$RELEASE" \
  --output /data --client atak --client-version VERSION \
  --plugin-version VERSION-OR-NATIVE --platform DEVICE \
  --os-version VERSION --evidence /data/operator-evidence/atak.png \
  --confirm-all-checks --operator NAME --notes "locations and paths tested"

docker compose run --rm worker dtedctl attest "$DATASET_ID" "$RELEASE" \
  --output /data --client wintak --client-version VERSION \
  --plugin-version VERSION-OR-NATIVE --platform WORKSTATION \
  --os-version VERSION --evidence /data/operator-evidence/wintak.log \
  --confirm-all-checks --operator NAME --notes "locations and paths tested"

Promote, verify, or roll back

Promotion is an explicit, atomic change:

docker compose run --rm worker dtedctl promote-state \
  "$DATASET_ID" "$RELEASE" --output /data --operator NAME
python3 tools/smoke_test.py \
  "https://$DATASET_ID.dted.org" --require-quality

Verify the live release ID, headers, range support, public quality summary, offline package, native routes, and source-isolation 404. Confirm the preview host is again restricted.

Roll back only to a release with a prior promotion record:

docker compose run --rm worker dtedctl rollback-state \
  "$DATASET_ID" certified-release-id --output /data --operator NAME
python3 tools/smoke_test.py \
  "https://$DATASET_ID.dted.org" --require-quality

Never edit an immutable release or repoint an active symlink by hand. Keep the active release and two prior releases, then prune through the lifecycle command:

docker run --rm --init --read-only \
  --user "$RUN_UID:$RUN_GID" \
  --cap-drop ALL --security-opt no-new-privileges \
  --tmpfs /tmp:rw,nosuid,nodev,noexec,size=256m,mode=1777 \
  -v "$SOURCE_ROOT:/sources" \
  -v "$DATA_ROOT:/data" \
  dted-portal dtedctl prune \
  --output /data --source-root /sources --keep-previous 2

Current California 3DEP checkpoint

The staged California 3DEP candidate is 67a19f908494fd1e. It was acquired from 100 TNM assets at source version 2026-06-10; archive SHA-256 is 068bdd8274a32297c0834a0d34fd26853327c8f102f451bb1dbf3243cc5fdc75 and processing profile is 411a17ef58bddd0e. Automated QA passed all 69 cells with zero state-land voids and exact seams. It remains a protected preview at preview-ca-3dep.dted.org; ca-3dep.dted.org must remain inactive until the exact candidate receives TAK evidence and an explicit promotion.