docs(api): write down the gkapi deploy procedure, including setcap

[sanity]

Problem

Deploying gkapi is undocumented, and one step in it is invisible until it bites.

The service runs as the unprivileged gkapi user and binds port 80 for the HTTP-01 ACME challenge server, which needs CAP_NET_BIND_SERVICE as a file capability on the binary. cp does not preserve file capabilities. Replace the binary without re-running setcap and it binds 443, logs a healthy-looking Listening on 0.0.0.0:443, then panics with PermissionDenied on the port 80 bind and systemd restart-loops until it gives up.

What makes it genuinely nasty is that the same trap catches the rollback: copying a backup back into place also produces a file with no capability, so the service keeps panicking and it reads as "the new build is broken" rather than "the copy dropped a capability". mv keeps the inode and therefore the capability; cp does not.

This bit while deploying #94 and took the API down for about 100 seconds (16:27:05 to 16:28:45 UTC, 2026-08-16). Diagnosis was only obvious once getcap was compared between the backup (which had cap_net_bind_service=ep, because it was created with mv) and the installed binary (which had nothing).

Also recorded

• Merging a change to rust/api ships nothing. deploy.yml builds the Hugo site and publishes to GitHub Pages; it never touches the API. rust-api-tests.yml only runs fmt/build/test. fix(api): stop concurrent requests minting several Ghost Keys from one donation #94 sat merged and undeployed for 18 days for exactly this reason, and nothing in the repo said so.
• There is no Rust toolchain on vega (~gkapi/.cargo exists but bin/ is empty), so the binary is built elsewhere and copied. vega and nova are both Ubuntu 24.04 on identical glibc, which is what makes that safe.
• A verification step that would have caught this immediately: curl http://gkapi.freenet.org/.well-known/acme-challenge/probe should return 404. Connection refused means the capability is missing, and certificate renewal will fail at the next attempt even though HTTPS looks perfectly healthy. That is the part that would have turned a 100-second outage into a surprise expiry weeks later.
• Tier verification, since a missing notary keypair is a separate failure mode from the binary.

Testing

Documentation only, no code. Every command in it was run against vega during the #94 deploy and the recovery.

Not addressed

The real fix is CI deployment for this crate so the manual procedure stops mattering. That is a bigger change and wants its own discussion, so this just makes the manual path safe and writes down that the gap exists.

[AI-assisted - Claude]

[sanity]

Adversarial review lens run. Two Medium findings, both real, both fixed. It also empirically reproduced the load-bearing claim rather than taking it on trust.

Fixed

1. A window where no binary exists. The procedure had mv (old aside) and cp (new in) as two separate ssh calls, leaving /home/gkapi/bin/ghostkey-api absent in between. Anything restarting the unit in that gap fails status=203/EXEC and stays down, which is strictly worse than never starting the deploy — until that point the old process was still serving fine from its already-open inode.

Restructured to stage → verify → swap: the new binary is copied to ghostkey-api.staged at its final location, chowned, chmodded, setcap'd and getcap-verified while the live binary is untouched, then swapped in with two adjacent renames. A side benefit is that the verification gate now costs nothing to fail.

2. getcap without sudo. It lives in /usr/sbin, which isn't on a normal user's PATH, so the doc's single verification gate could fail as "command not found" and be silently skipped rather than enforced. Now sudo getcap, matching the sudo setcap above it. (It happens to resolve for my login on vega, but the doc shouldn't depend on that.)

Also addressed

• /create-donation has no dry-run mode, so the tier loop creates real PaymentIntents in the live Stripe account. Uncharged and unattached, identical to a visitor clicking between the amount radios, but now stated, with a warning not to wrap it in a retry loop.
• Rollback now says to substitute the timestamp, and restructured to stage-and-rename so it can't reintroduce the capability bug it exists to fix.

Verified independently by the reviewer

• cp drops file capabilities, mv preserves them — reproduced on a matching Ubuntu 24.04 image with a scratch file, not taken from the incident narrative.
• setcap cap_net_bind_service+ep syntax and getcap output format, confirmed by the same test.
• deploy.yml genuinely never touches rust/api; rust-api-tests.yml is exactly fmt/build/test; no workflow anywhere references vega, ssh, scp or rsync.
• $(date ...) inside the single-quoted ssh block expands remotely, which is what's wanted; the \$$a escaping in the tier loop produces the intended literal output.
• The release binary is unstripped, so strings | grep payment_claim really is a valid sanity check (4 matches).
• The tier list matches stripe-donation-form.html exactly, and the cited path exists.
• No secrets added; vega and gkapi.freenet.org were already public in this file.

All four bash blocks pass bash -n.

Accepted as-is

The reviewer noted that info!("Listening on 0.0.0.0:443") is logged before the 443 socket is actually bound, so strictly the panic is the first bind rather than the second. Since that's exactly what an operator sees in the logs, the troubleshooting narrative is unchanged and I left the wording alone.

[AI-assisted - Claude]