Back up, restore and uninstall¶

For an installation from the self-hosting bundle. The database is the archive: everything your team's laptops pushed lives there and nowhere else, so a backup you have restored is the only copy that counts. Installing a later release, and going back, is in Upgrade and roll back.

Every server command here runs as the user who owns <install>/.env (the scripts read it, and only its owner can) and who can run docker. <install> is the folder the running release was unpacked into, the one holding its .env and scripts/: winner-install/winner-ai-server-<X.Y.Z> for the first release, winner-install-<X.Y.Z>/winner-ai-server-<X.Y.Z> for a later one. Write it as an absolute path.

Older release folders stay on disk after an upgrade, and every one of them drives the same running stack: the stack's name is fixed, not taken from the folder. Run these commands only from the running release's folder; docker compose ls shows the compose files the running stack was started from, and grep ^FLEETVIEW_VERSION= .env which release a folder holds.

What a backup holds, and what it does not¶

A backup is one gzipped pg_dumpall of the whole database: the organisation's sessions, the control plane's own tables (people, laptops, settings, the licence) and every database role, the laptops' own included, so a restore brings back the whole installation rather than its tables. Each backup is restored into a throwaway database and its row counts compared with the live one before it is given its name, so a file with that name has already restored once.

It does not hold .env, and a restore needs the .env the database ran with. The backup carries the database's own passwords, so a stack started with any other POSTGRES_PASSWORD cannot log in to the database it restored; and without the same settings key it cannot read the team model keys sealed with it. Keep a copy of .env somewhere safe, and not beside the backups, so that no single stolen copy holds both. Note which release each backup was taken under, and restore it into that release.

Certificates are taken again and the AI runner pulls its model again on a new server, so neither needs a backup.

Take a backup¶

# the user who owns <install>/.env
mkdir -p ~/winner-backups
<install>/scripts/backup-server.sh ~/winner-backups/<date>.sql.gz

It runs against the live stack, which keeps serving. It prints backup <file>: <bytes> bytes, restore test PASS (<n> tables match) when the backup is good; otherwise one line, backup: refused: ... or backup: FAIL: ..., says why, and the exit status is non-zero. A failed run keeps its dump as <file>.failed, for diagnosis, or as <file>.part when it could not rename it. A laptop push between the dump and the count can fail a run with differing counts: run it again under another name.

Each run needs free disk for the dump where you write it and, while it is proved, a scratch copy of the database in Docker's own storage. A nightly backup needs that room every night, beside the backups you keep.

The file is readable by its owner only and holds every transcript and every database login: keep every copy that way. Delete .failed and .part files once you have read why they failed, and old pre-restore-* files on the same schedule as old backups; never schedule the deletion of a pre-restore-*.failed file, which can be the only copy of the database as it was before a restore. The command refuses a path inside the installation folder, a name that exists already (or whose .part or .failed does), a name ending in .part or .failed, a folder it cannot write, and a database fenced for a restore: a dump taken mid-restore would pass its own test against a half-replayed database.

It also refuses a database that holds a CHECK constraint, generated column or domain check stock agentsview does not create (backup: refused: the database holds ... CHECK constraint(s) or generated column(s) ...), since a restore would run it as the database superuser. Clear it as Code in an org schema that agentsview did not create says, listing it with docker compose exec -T backend uv run --no-dev fleetview-provision audit in <install>, then back up again.

Schedule it and copy it off the server¶

Nothing on a bundle install schedules a backup or copies one anywhere. Both are yours. Run the command above by hand first and see the PASS line, then add a line with crontab -e as the same user, nightly:

# crontab -e, as the user who runs the install; <install> as an absolute path
15 3 * * * <install>/scripts/backup-server.sh "$HOME/winner-backups/winner-$(date -u +\%Y\%m\%d).sql.gz" >> "$HOME/winner-backups/backup.log" 2>&1

Cron finds docker only in /usr/bin or /bin: for any other install of Docker, put a PATH= line above it. <install> changes with every upgrade, since each release unpacks into a new folder: point the line at the new folder once the upgrade is done. A rollback (scripts/rollback-server.sh) goes the other way: point it back at the previous release's folder, which runs again. The newer release's winner-install-<X.Y.Z> folder is kept as winner-install-<X.Y.Z>.rolled-back-<UTC stamp> when the upgrade made it, and otherwise stays where it is, never to be run.

Nothing tells you when a night fails. Read the last line of backup.log, which is the restore test PASS line only when the latest night succeeded, or have your own monitoring alert when no good backup is newer than two days:

# the user who takes the backups; prints nothing when there is none
find ~/winner-backups -name 'winner-*.sql.gz' -mtime -2

Copy each good backup off the server, encrypted, with your own tool: a backup on the server it protects does not survive losing that server.

Restore¶

# the user who owns <install>/.env
<install>/scripts/restore-server.sh ~/winner-backups/<date>.sql.gz

A restore replaces the whole database with the backup. The command checks the file and proves it replays before it changes anything, then stops the stack, shuts every laptop out of the database, backs up the database as it was (pre-restore-<stamp>.sql.gz, beside the backup), replays the backup, proves the row counts match, and starts the stack again. The dashboard is down and laptops wait until it ends. It prints restore: restored <file>: <n> tables match the backup, then where the database as it was is kept, and last restore: a laptop enrolled after this backup was taken must run `winner login` again. With --keep-fence, as the rollback command runs it, a line after the first says the fence stays up and the stack stopped, as --keep-fence asked.

If it prints restore: refused: ..., nothing was stopped or changed: the reason names what to fix, such as the file, a path, or a stack it cannot find.

If it prints restore: FAIL: ..., the line after it says what state it left; no line after it means nothing was stopped:

nothing was changed; the stack was started again, or nothing was restored; the fence is lifted and the stack started again: the database is as it was and the stack keeps running. Read the reason, fix it, and restore again. nothing was changed, but starting the stack failed: the database is as it was; start the stack with docker compose up -d, then fix the reason and restore again.
nothing was restored, but lifting the fence or starting the stack failed: run the same command again, as the line says.
the database stays fenced and the stack stopped, or nothing more was restored; the database stays fenced from the earlier restore: a replay began in this run, or may have in an earlier one, so the database may not be the backup and must not take a push. Leave the stack stopped and do not start it by hand. Restore again, from the same backup or from the pre-restore- file the run that first raised the fence named, and keep that file: it can be the only copy of what was there.

A run that prints restore: restored ... can still have printed WARNING: could not start every stopped service above it: the database is restored, so start the stack with docker compose up -d in <install>.

While it runs, Postgres listens only inside its own container, so every laptop is refused a connection. A refused connection writes no Postgres log line, so no laptop is banned; its push retries as it would while the server is down, and catches up once the restore ends. The restore needs free disk for one more dump beside the backup and, while each is proved, a scratch copy of the database in Docker's own storage. Run one restore at a time: two are not locked against each other.

After a restore:

A laptop enrolled after the backup was taken cannot push. Its owner runs winner login again, which enrolls it afresh. Its refused pushes count toward the Postgres jail where you run one, so unban its address if the jail already has (The Postgres jail).
A laptop revoked after the backup was taken shows as enrolled again on Admin but cannot log in to the database; the command names each one. Revoke it again on Admin.

After losing the server, the restore is the rehearsal below with production's names: a new server, the release the backup was taken under, installed with the .env you kept, DNS pointed at the new server, then restore-server.sh. The new server's empty database had no laptop logins, so the command reads every laptop in the backup as revoked after it, takes each one's login away and names it: there that list is every laptop, not revocations to repeat. Each person whose laptop should push again runs winner login on it. Until they do, every laptop's pushes are refused, and where you run the Postgres jail on the new server those refusals count toward it: unban an address it has already banned, as The Postgres jail shows.

Rehearse a restore¶

Every backup is already restored once, into a throwaway database. The rehearsal proves the rest: that you can build a server from the bundle, the .env you kept and a backup, and that people can sign in to the result. Do it once the installation is running, and again after each upgrade.

Rehearse on a second server, never in a second folder on the production server: every folder drives the same stack, so a restore there replaces production's database.

Prepare a test server that meets the same requirements as production (Check the requirements), with its own DNS name (two, if production's PUBLIC_PG_HOST differs from PUBLIC_HOST). In the http certificate mode, keep ports 80 and 443 open to the internet, since the edge takes its certificate over port 80; in the dns mode, put the test names in the same Route 53 zone and give the test server an instance role that may write their challenge record. Either way, admit only your own address to 5432: once restored, it holds every transcript and every laptop's database login.
Install the same release on it from its bundle, as Installing the Winner AI server says, signature and checksum checks included, with a copy of the production .env in which you change only:
PUBLIC_HOST, PUBLIC_PG_HOST and PUBLIC_BASE_URL, to the test server's names;
FLEETVIEW_HEARTBEAT_URL, FLEETVIEW_SENTRY_DSN and FLEETVIEW_POSTHOG_KEY, to blank, so the test server neither answers for production's dead-man's switch nor reports as a second installation;
the FLEETVIEW_FEATURES line, removed, so no model pass runs there.

To sign in to it, add https://<test-host>/oauth2/callback to the sign-in application's allowed callback addresses at your identity provider, and its base URL to the allowed logout addresses where the provider has them.

Copy the latest backup to it over scp or rsync (never a shared drive or a chat), and restore it: <install>/scripts/restore-server.sh <backup>. It must print restored <file>: <n> tables match the backup. It also names every laptop as revoked after the backup, as on a new server after losing one: leave them; no laptop pushes to the test server.
Sign in as a Manager, and look without changing anything. Admin lists your people and laptops as they were at the backup, and the Dashboard shows the newest sessions you expect from before it. Do not add anyone there: the test server shares production's identity provider, and on Auth0 a person added there is added to production's tenant.
Take the test server down. On the test server only, in its <install>: docker compose down --volumes --rmi all. Leave the laptops, any crontab and the identity provider's applications alone: they are production's. Then delete the test server's copies of the backup and .env, the pre-restore-* file the restore wrote beside the backup, its DNS records, the callback and logout addresses you added, and in the dns mode the test server's instance role, or the test names you added to a policy.

A rehearsal that fails at step 3 means your backups would fail you too: send support the line the command printed. One that fails at step 4 after step 3 printed its restored line is first the test server's DNS, certificate or Auth0 addresses; once those are right, a step 4 that still fails means the same as a step 3 failure.

Uninstall¶

Removing an installation deletes its database, the only copy of every session. Take a last backup first, unless you are certain nobody will want the sessions again, and copy it and .env off the server, kept apart as above: a restore of that backup needs that .env.

Laptops first, so none keeps calling an address that stops answering. Each person removes the client from their own laptop as Uninstall in the laptop client guide says.

Then the server. Check you are on the right server and in the running release's folder (grep -E '^(PUBLIC_HOST|FLEETVIEW_VERSION)=' .env), then:

# the user who owns <install>/.env; in <install>
docker compose down --volumes --rmi all

That stops and removes every container, deletes the stack's volumes (the database, the dashboard's data, the certificates and the AI runner's models) and removes the images this release ran. It cannot be undone. It removes the stock images too (Postgres, Caddy, agentsview, oauth2-proxy, Ollama, and the certificate helper in the dns mode), so another stack on the same server that uses one pulls or loads it again. Images of earlier releases stay: docker image ls lists them. Remove the crontab line that ran backups (crontab -e), then delete the winner-install* folders, with the .env in each, and the backups you no longer need, wherever they were copied. Keep the off-server copy of .env for as long as you keep a backup it restores.

Last, what lives elsewhere: the DNS records, the firewall rules for 80, 443 and 5432, the Postgres jail and its filter where you set them up on the server, in the dns mode the instance role and its Route 53 policy, your dead-man's switch and any telemetry project, and the sign-in application at your identity provider (on Auth0, the applications and the organisation).