Operations
monitoring, backups, updates.

01Monitoring

Do not paste the operator key into third-party monitoring services. Monitor /health and use the console for the rest.

02Backups and restore

SQLite holds everything: merchants, invoices, the fee-credit ledger, webhook log and the audit ledger. scripts/backup.mjs uses SQLite's online backup API (safe while the server runs), verifies each copy with PRAGMA integrity_check, writes it with mode 0600 and rotates old files. It also backs up the console's admin.sqlite.

pnpm run backup                        # uses apps/server/.env
node --env-file=/etc/mikodes-checkout/server.env scripts/backup.mjs --dir /var/backups/mikodes-checkout --keep 14

The systemd timer mikodes-backup.timer runs it daily at 03:17; the PM2 file includes a daily backup. Also keep: a copy off the machine, the env files (in a password manager, not next to the backups) and, with the console, ADMIN_SECRET_KEY together with admin.sqlite.

sudo systemctl stop mikodes-checkout
sudo -u mikodes cp /var/backups/mikodes-checkout/checkout-<stamp>.sqlite /var/lib/mikodes-checkout/checkout.sqlite
sudo rm -f /var/lib/mikodes-checkout/checkout.sqlite-wal /var/lib/mikodes-checkout/checkout.sqlite-shm
sudo systemctl start mikodes-checkout

After a restore, invoices paid after the backup are re-detected only while they are in their watch window (pending, or expired less than 24 hours ago). Reconcile older ones from the on-chain evidence. Practise a restore once before you need it.

03Rotating keys and secrets

04Sanctions list

Screening is disabled until SANCTIONS_LIST_PATH points at a list; the server warns at every start. With a list, the payer is screened when a payment is first seen; a match marks the invoice flagged; a screener error fails closed (the invoice does not advance).

1. Download the SDN XML yourselfFrom OFAC's Sanctions List Service (public domain). The converter never downloads anything.
2. Convert

   node packages/core/scripts/ofac-sdn-to-json.mjs <sdn.xml> <YYYY-MM-DD>

   Check its exact arguments in the script header first, and review the first converted file by hand.
3. Point the server at itSet SANCTIONS_LIST_PATH (readable by the mikodes user) and restart; the list is loaded at start.
4. Refresh on a scheduledeploy/sanctions/refresh.sh downloads from the URL you set in SDN_XML_URL, converts, refuses an empty result, swaps the file atomically and restarts the service. Run it from root's crontab.

05Updating to a new version

1. Read the changelogChangelog.
2. Back upnode --env-file=/etc/mikodes-checkout/server.env scripts/backup.mjs --dir /var/backups/mikodes-checkout
3. Replace the code and rebuildUnpack the new package over a copy of the old folder, keeping your env files, then:

   cd /opt/mikodes-checkout
   sudo -u mikodes -H pnpm install --frozen-lockfile
   sudo -u mikodes -H pnpm --filter "@mikodes-checkout/server..." build
   sudo -u mikodes -H pnpm --filter @mikodes-checkout/ops build
   sudo -u mikodes -H pnpm run build:web
   sudo systemctl restart mikodes-checkout
   curl -s http://127.0.0.1:8787/health

4. MigrationsRun automatically at start and are forward-only. To roll back, restore the backup from step 2 and the previous code, never just the code.

Docker: back up the server-data volume, then docker compose -f deploy/docker-compose.yml up -d --build.

06Incident checklist

RPC down or rate-limited

Invoices stay pending or confirming; wallet pay fails at "prepare" (/public/solana/prep/:id returns 502 rpc_error). Switch to a second provider and restart. Nothing is lost: payments land on-chain regardless, and the watcher finds them when the RPC is back, including invoices that expired less than 24 hours ago (recorded as late).

Webhook dead letters

Console → Webhook deliveries. Look at the recorded status or error (expired TLS certificate at the merchant, firewall, old secret). After the merchant fixes it, retry; receivers deduplicate on event.id.

Flagged payment

The funds are already at the merchant's address; the system is non-custodial and cannot reverse or freeze them. Tell the merchant not to deliver the order, keep the evidence (timeline, audit entry, explorer link) and follow your compliance procedure and legal advice.

Exchange pay missing for a merchant

Its fee credit is below the invoice fee. Top up (portal Fee credit, or Console → Merchants → merchant). Wallet pay keeps working meanwhile.

Audit chain broken

chainIntact: false means an audit row was changed outside the application. Copy the database as evidence, compare with the last intact backup, rotate the operator key and review host access.

Operator key leaked

Rotate it, restart, review the audit ledger and check every merchant's payout addresses and fees for changes you did not make.