Operations runbook

This runbook covers the current single-host DTED.org deployment. Run production commands from a trusted operator shell and never paste credential values into issues, commits, command arguments, or logs.

Locations and boundaries

Purpose	Location
Development checkout	/home/gba/Agents/DTED.org
Installed application	/srv/dted
Releases, active links, QA, and database	/srv/dted-data
Installed nginx configuration	/etc/nginx/sites-available/DTED.org
Wildcard certificate links	/etc/dted/tls/
Portal authentication	AndKit OAuth2 proxy on auth.andkit.io; edge includes in /etc/andkit-auth/
USGS credential file	/etc/dted/usgs.env
Name.com credential file	/etc/dted/namecom.env

Secret files must be owned by root and mode 0600 where applicable. The portal listens only on 127.0.0.1:8780; nginx and the central AndKit OAuth2 proxy are the public authentication and data boundary. DTED has no separate operator password.

Routine health check

cd /srv/dted
docker compose ps
curl -fsS http://127.0.0.1:8780/healthz
sudo nginx -t
python3 tools/smoke_test.py https://ca.dted.org
python3 tools/smoke_test.py https://tn.dted.org --require-quality
python3 tools/smoke_test.py https://va.dted.org --require-quality
python3 tools/smoke_test.py https://nc.dted.org
python3 tools/smoke_test.py https://sc.dted.org

Do not use --require-quality for a known failed or unverified release merely to make the command green. Investigate any unexpected release ID, HTTP status, content type, range failure, or quality result.

Deploying application changes

From the development checkout:

make test
.venv/bin/mkdocs build --strict
git diff --check
git status --short
sudo tools/install_production.sh

The installer copies the checkout to /srv/dted, rebuilds the containers, validates nginx, checks portal health, and smoke-tests California. Afterward, repeat the broader health check above and verify the pushed commit's CI run.

Publishing state data

State data must follow acquisition, audit, immutable build, preview, client attestation, and promotion. The normal sequence is:

make check-usgs-access STATE=CA
make acquire-state STATE=CA
make audit-state STATE=CA
make build-state STATE=CA
make preview-state STATE=CA RELEASE=release-id
make attest-state STATE=CA RELEASE=release-id CLIENT=atak \
  VERSION=version PLUGIN_VERSION=version PLATFORM=device \
  OS_VERSION=version EVIDENCE=evidence.png OPERATOR=name
make attest-state STATE=CA RELEASE=release-id CLIENT=wintak \
  VERSION=version PLUGIN_VERSION=version PLATFORM=workstation \
  OS_VERSION=version EVIDENCE=evidence.log OPERATOR=name
make promote-state STATE=CA RELEASE=release-id OPERATOR=name
make smoke-state STATE=CA

Never turn a failed quality result into a passing result. An exceptional operator override must remain explicit, preserve the failed public summary, and must not be described as certification.

QA previews are public by default and carry a noindex response header. Never stage sensitive or restricted data there. If an installation has deliberately replaced the public QA policy with trusted CIDRs, also set DTED_PUBLIC_QA_DISCOVERY=0 in /srv/dted/.env and recreate the portal container. This prevents public state comparison pages from advertising restricted preview links. The production installer derives this setting from the installed QA allowlist unless the operator explicitly supplies an environment override.

On a restricted-QA installation, open one staged candidate for a bounded public test window and close it immediately after evidence capture:

sudo /srv/dted/tools/qa_preview_window.sh open CA 24
sudo /srv/dted/tools/qa_preview_window.sh status
sudo /srv/dted/tools/qa_preview_window.sh close

The helper schedules automatic closure and refuses to expose multiple staged candidates. On a restricted-QA installation, confirm that the preview returns 403 again before promotion.

Verifying state-source discovery

After staging or promoting a source-specific release, verify the state page, regional comparison, and production-only product boundary:

curl -fsS https://ca.dted.org/ | grep 'Choose a terrain source'
curl -fsS https://dted.org/states/ca/ | grep 'Technical comparison'
curl -fsS https://dted.org/api/v1/regions/CA/terrain-sources
curl -fsS https://dted.org/api/v1/products?region=CA

An eligible preview must be labeled evaluation only in the regional source API. It must not appear in /api/v1/products until promotion. If the portal is unavailable, Nginx serves the release's immutable landing page as the state-root fallback; terrain and client artifacts continue to be served statically.

Refreshing QR and client assets

Client files are updated with an immutable successor rather than editing an active release:

cd /srv/dted
docker compose run --rm worker dtedctl refresh-client-assets ca active-release \
  --output /data
docker compose run --rm worker dtedctl activate-client-assets ca successor-release \
  --output /data --operator operator-name

The activation guard requires the source release to still be active and compares terrain hashes, dataset metadata, source acquisition, and quality results. Verify the public client JSON has schema 2 and qr_action set to preference, decode the published QR, and confirm the ZIP and .pref alternatives still return 200.

ATAK compatibility test

For each hardware test:

1. Record client, plugin, Android, and device versions without recording a user IP in shared notes.
2. Scan the state's published QR and approve ATAK's external-configuration prompt.
3. Pan and zoom across known DTED2 coverage and a DTED0 fallback area.
4. Confirm these resources as appropriate for the installed client:

console curl -I https://nc.dted.org/index.zip curl -I https://nc.dted.org/w082/n32.dt0.zip curl -I https://nc.dted.org/elevation/DTED/w082/n32.dt0.zip curl -I https://nc.dted.org/elevation/DTED/dted_nw_hemi.zip

1. Inspect recent nginx requests and status codes. Redact client IPs before sharing output:

console sudo tail -n 20000 /var/log/nginx/access.log | \ grep -E 'index\.zip|elevation/DTED|/[ew][0-9]{3}/[ns][0-9]{2}\.dt[0-3]\.zip' sudo tail -n 1000 /var/log/nginx/error.log

1. Save screenshots or logs through the attestation workflow only after package, preferences, index, DTED2, control points, and fallback checks all pass.

Both root and /elevation/DTED/ prefixed per-cell URLs use the same release-first, permitted-baseline-fallback lookup. Verify both forms and record representative hardware evidence. HTTP delivery alone is not a native ATAK pass.

Logs and incident triage

Static dataset traffic terminates at nginx and normally does not appear in portal container logs. Start with /var/log/nginx/access.log and error.log, then use:

cd /srv/dted
docker compose logs --since 30m portal worker
sudo journalctl -u nginx --since '30 minutes ago'

A 404 with no nginx error usually indicates route or release content mismatch. A 5xx, TLS error, truncated byte count, or failed Range request requires server-side investigation. Nginx's default access timestamp is written when the request is logged; the response byte count proves bytes sent by nginx but not client import.

Rollback and recovery

Use the certified rollback command for a release with a promotion record:

make rollback-state STATE=CA RELEASE=certified-release-id OPERATOR=name

For an urgent application rollback, redeploy a known-good Git revision only after confirming it understands the current on-disk schema. Do not edit immutable release directories or repoint active symlinks by hand. Preserve the SQLite database, release store, active links, quality records, nginx configuration, certificates, and root-managed credential files in backups.

Repository privacy and secret scanning

The GitHub repository is private as of this handoff. Before making it public again:

1. Scan the current tree and complete Git history with a maintained secret scanner.
2. Review findings rather than suppressing them broadly.
3. Rotate and remove any exposed credential before rewriting history.
4. Confirm documentation contains credential variable names and paths only.
5. Remember that changing repository visibility does not remove existing clones, caches, releases, package artifacts, or published documentation.

Repeat the redacted scans with:

GITLEAKS='zricethezav/gitleaks@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f'
docker pull "$GITLEAKS"
docker run --rm -v "$PWD:/repo:ro" "$GITLEAKS" dir /repo \
  --redact=100 --no-banner --max-target-megabytes 5 --max-archive-depth 0
docker run --rm -v "$PWD:/repo:ro" "$GITLEAKS" git /repo \
  --redact=100 --no-banner

An administrator can restore public visibility only after review:

gh repo edit snstac/DTED.org --visibility public \
  --accept-visibility-change-consequences

Public discovery performance

Published datasets, exact coverage, region catalogs, and source registry are cached in-process for 15 seconds (sources for 300 seconds). Public JSON catalog, product, coverage, and AndKit asset responses use strong ETags and gzip for large bodies. After promotion, allow one TTL before declaring stale discovery; verify coverage still matches the promoted release manifest.