Upgrade and roll back

For an installation from the self-hosting bundle. An upgrade takes the server to the release in a newer bundle and keeps every person, Device and session; a rollback takes it back to the release the last upgrade replaced, or to an older one you name.

Every command here runs from the folder of the release that runs now, as the user who owns its .env (mode 0600, so only its owner can read it) and may run docker. Another user who may run docker is refused, because it cannot read the file: the upgrade says cannot read <folder>/.env, ..., and the rollback says <folder>/.env records no FLEETVIEW_VERSION. An upgrade backs the database up first, and a rollback restores that backup: Back up, restore and uninstall says how backups are taken, kept and restored.

Installing a later release

One command takes the server to the release in a newer bundle and keeps every person, Device and session. The uploaded licence, Settings and roles are kept too, so nothing is uploaded or set again. Before you upgrade, read the new release's Requires of you section in the Changelog. Download the new winner-ai-server-<X.Y.Z>.tar.gz anywhere outside the installed folder, then, from the installed release's folder (winner-install/winner-ai-server-<old X.Y.Z> for the first upgrade), as the user who owns its .env, giving the path you downloaded it to:

./scripts/upgrade-server.sh /path/to/winner-ai-server-<X.Y.Z>.tar.gz

There is nothing to verify by hand: the installed release checks the new bundle's signature against the release key it carries, then every file, and refuses anything else. ./scripts/upgrade-server.sh --help lists its steps in order. It unpacks the new release beside the installed one, in winner-install-<X.Y.Z>/winner-ai-server-<X.Y.Z>, carries .env across (the telemetry secret with it), runs the new release's host and .env checks and pulls its Docker Hub images, then backs up the database with scripts/backup-server.sh. Until that backup has passed its restore test, nothing that runs is touched. Only then does it load and start the new release, which migrates the database, and wait for it to answer with every migration applied:

upgrade-server: step 1 of 10: check the new bundle: ...
upgrade-server: the new bundle is v<X.Y.Z>, signed by the release key, every file as signed
upgrade-server: upgrading v<old X.Y.Z> in <old folder> to v<X.Y.Z> in <new folder>
...
upgrade-server: the database is backed up at <folder the archive is in>/pre-upgrade-v<old X.Y.Z>-<UTC stamp>.sql.gz
...
upgrade-server: every migration v<X.Y.Z> holds is applied
upgrade-server: the web server runs the new bundle's binary
upgrade-server: the edge serves ports 80 and 443
upgrade-server: the org schemas hold only what agentsview creates
upgrade-server: upgraded v<old X.Y.Z> -> v<X.Y.Z>; the previous release stays in <old folder> for a rollback, its images loaded

When usage counts are off, or the telemetry secret is not set, a line before the last says usage counts are off or the telemetry secret is not set; the upgrade was not reported.

By default the backup goes in the folder the new bundle's archive is in. --backup <file> writes the backup somewhere else; like any backup it must be outside both release folders. The new release's folder is where you run docker compose from now on: its .env names the compose files and the release. Keep the previous folder and its images, and never prune images: they are what a failed upgrade, and a rollback, start again. winner-install-<X.Y.Z>/winner-upgrade.record records the release replaced, its folder and the backup's path, for the rollback.

Laptops after an upgrade. Nothing runs on a laptop. Each one keeps pushing with the agentsview and winner it has until winner update brings it to the versions the new release runs. A Mac or Linux laptop whose winner login ran without --no-service runs it on its own each day (Running on its own), except a Linux laptop without systemd, which gets only the crontab lines winner login printed, to add by hand. On any other laptop its owner runs winner update.

On the server, point your nightly backup line at the new release's folder (Schedule it and copy it off the server), and rehearse a restore again (Rehearse a restore).

Auth0 after an upgrade or a rollback. The tenant's post-login Action is deployed by hand, so neither command changes it. Compare it with the copy in the release that now runs, as The post-login Action says, and deploy that copy if they differ.

A slow upgrade is not a failed one. A release that moves the collector version rebuilds every organisation's schema on its first start, which can take half an hour, and the control plane does not answer until it is done; the command says so about once a minute. If it is still migrating after thirty minutes (UPGRADE_WAIT_SECONDS), it puts nothing back, says still migrating after 1800s; this is slow, not failed, and exits 3. Watch it with docker compose logs -f backend in the new release's folder, and when it is done, run ./scripts/upgrade-server.sh --finish there.

When it refuses, before it starts the new release, it prints one upgrade-server: FAIL: line saying nothing was changed, removes the new folder, and the installed release runs as before. A refusal once the backup has passed, docker load failing included, also says the backup at <file> stays:

What it says	What to do
SHA256SUMS.sig is signed by another key (...), ... was made in the licence namespace, or SHA256SUMS does not match its signature in SHA256SUMS.sig: ...	The archive is not a release Winner AI signed for you, or it was changed. Download it again; if it still says so, ask your Winner AI contact
these files of the new bundle do not match SHA256SUMS: ..., the new bundle holds files SHA256SUMS does not name: ..., or ... holds a link or a special file	The archive changed after it was signed. Download it again
no .env in <folder>; ..., cannot read <folder>/.env, ... or <folder>/.env records no FLEETVIEW_VERSION; ...	Run it from the installed release's folder, as the user who owns its .env
<tool> is not installed, and the new bundle cannot be checked without it	Install the tool it names (ssh-keygen, sha256sum, tar or od), and run it again
the new bundle carries no scripts/<name>.sh to finish the upgrade with, or to roll it back	The archive is not a whole release. Download it again
this server already runs v<X.Y.Z>	Nothing to do
the new bundle is v<X.Y.Z>, older than v<old X.Y.Z> which this server runs; migrations do not reverse, ...	The upgrade command never installs an older bundle over a newer release. To go back to the release the last upgrade replaced, roll back (Going back to the previous release); to an older release, roll back with --bundle naming its folder and --backup the backup taken before the upgrade from it
winner-install-<X.Y.Z> already exists; ...	An earlier attempt left that folder, and it may hold a release that ran. Read winner-install-<X.Y.Z>/winner-upgrade.record first. state=migrating: the upgrade was slow, not failed; run ./scripts/upgrade-server.sh --finish in its release folder. state=started: it was interrupted while the new release started; do what that run printed, as When it fails after the start below says. state=done: that release is installed, so run from its folder. No record: nothing started from it; move it aside, and run it again
the backup: ...	Choose another --backup path; the reason says why this one was refused
<folder>/.env names COMPOSE_FILE=<files>, not the bundle's docker-compose.yml:docker-compose.prod.yml:docker-compose.bundle.yml, COMPOSE_FILE in <folder>/.env names <path>, which is not an absolute path to a file, or ... inside a release folder	The upgrade command moves a production install only, never a trial (A trial from the bundle), whose .env names no production overlay. On a production install, put COMPOSE_FILE in that .env back to the bundle's three files, with any file of your own after them as an absolute path outside both release folders, and run it again
<old folder> has no scripts/backup-server.sh to back the database up with	The installed release's folder is incomplete: run it from the folder the installer or the last upgrade left running
this host is missing what v<X.Y.Z> needs (above)	Fix each check-host: FAIL: line above it, and run it again
v<X.Y.Z>'s scripts/check-env.sh refused .env (above)	The new release needs a change to .env: make it in the installed release's .env, and run it again
<image> is neither in the new bundle nor a pinned Docker Hub image	A compose file of your own, named in COMPOSE_FILE after the bundle's three, adds an image that is not pinned by @sha256: on Docker Hub. Pin it by digest, and run it again
could not pull <image> from Docker Hub: ...	As at install (step 5): wait, and run it again
the running release does not answer as v<old X.Y.Z> ... or the database of the release in <folder> is not running	Start the installed release (docker compose up -d in its folder) and run it again
<image>, which v<old X.Y.Z> runs, is no longer loaded, ...	An image of the installed release was removed. One of ours: load its images.tar again (docker load -i images.tar in its folder). A Docker Hub image, named by @sha256: and not in images.tar: pull it again by that name (docker pull <image>)
the backup failed: ...	Fix what the backup's own line names (Take a backup)
the backup did not report a passed restore test: ...	The pre-upgrade backup was written but its restore test did not pass; the line quotes the backup's last line. Fix what it names, and run it again
docker load could not load the new images.tar: ... or docker load left no ...	The line gives Docker's own reason. Fix it, and run it again; the backup it took stays

When it fails after the start, it puts the previous release's containers back from its own folder and prints one line saying why, the command that restores the database as it was before the upgrade, and where it kept the attempt:

upgrade-server: FAIL: the upgrade to v<X.Y.Z> failed: the migration did not complete: ...; v<old X.Y.Z>'s containers are running again
upgrade-server: to restore the database as it was before the upgrade, in <old folder> run: scripts/restore-server.sh <backup>
upgrade-server: the new release and its record are kept in <parent>/winner-install-<X.Y.Z>.failed-<UTC stamp>

If the containers cannot be put back, the first line ends v<old X.Y.Z>'s containers could not be put back: <reason> instead: start the previous release yourself with docker compose up -d in its folder, then read on.

Ctrl-C, or a TERM signal, after the start puts nothing back. It prints interrupted while v<X.Y.Z> was starting; nothing was put back, the docker compose up -d that puts the previous release back, with its folder, and the same restore command. From step 2 until the start, an interrupt is a refusal like any other.

Migrations do not reverse, so a failed one may leave the database in a state the previous release does not expect: if it misbehaves, run the restore command it printed (Restore), which takes the database back to the backup, and send the failure line to your Winner AI contact. When the new release carried another collector version, a line before the restore command says so: the previous release has just run its older collector over organisations' schemas the new release may have rebuilt, so run the restore before your team uses it.

Why an older release is refused

Migrations only go forward. The control plane migrates the database to its own release when it starts, so an older release would start on tables it does not know. The upgrade command therefore refuses a bundle older than the release this server runs, before its long check of images.tar, and the installer (scripts/install-server.sh) refuses one over a newer release that runs. The only way back is the rollback, below: it restores the pre-upgrade backup first, and starts the previous release only once that restore has passed. The installer sees only a stack that runs: over a stopped one it does not know which release last ran, and an older bundle is not refused.

The upgrade command's refusal begins:

upgrade-server: FAIL: the new bundle is v<X.Y.Z>, older than v<old X.Y.Z> which this server runs; migrations do not reverse, ...

Going back to the previous release

Migrations do not reverse, so the release an upgrade replaced never starts on the database the newer one migrated. The rollback command puts both back: the database as it was before the upgrade, from the backup the upgrade took, then the previous release. From the folder of the release that runs now, as the user who owns its .env:

./scripts/rollback-server.sh

It reads winner-upgrade.record beside that folder for the previous release's folder and the pre-upgrade backup (--bundle <folder> and --backup <file> name others, and both are needed without a record). It checks the previous folder's signature and every file as the upgrade checks a new bundle, refuses a backup that is partial or marked failed, and changes nothing until you type the previous release back:

rollback-server: rolling back v<X.Y.Z> in <folder> to v<old X.Y.Z> in <old folder>
rollback-server: everything written since the upgrade at <time> will be replaced by <backup>: people, Devices, settings and the sessions laptops pushed
rollback-server: the database as it is now is backed up first, to <folder of the backup>/pre-rollback-v<X.Y.Z>-<UTC stamp>.sql.gz
rollback-server: a laptop enrolled since must run `winner login` again; a Device revoked since comes back unable to log in and must be revoked again
Type v<old X.Y.Z> to roll back to it:

--pre-rollback-backup <path> writes that backup somewhere else; like any backup it must be outside both release folders. --yes skips the question. It then restores the backup with scripts/restore-server.sh (Restore), which keeps laptops out while it replays, and starts the previous release only once the restore has passed. It is done when that release answers with every migration applied:

rollback-server: the newer release and its record are kept in <parent>/winner-install-<X.Y.Z>.rolled-back-<UTC stamp>; never run docker compose there
rollback-server: rolled back v<X.Y.Z> -> v<old X.Y.Z>; the database is as it was at <time>
rollback-server: the database as it was before the rollback is in <pre-rollback backup>
rollback-server: run docker compose from <old folder> from now on; point your nightly backup line at it
rollback-server: a laptop enrolled since the upgrade must run `winner login` again; a Device revoked since cannot log in and must be revoked again from the roster

With --backup naming another backup than the record's, or without a record, the two lines that give the upgrade's time name the backup instead: the first says everything written since <backup> was taken will be replaced by it, the last the database is as it was when <backup> was taken.

The newer folder is kept as winner-install-<X.Y.Z>.rolled-back-<UTC stamp> when the upgrade made it, and the first line says so; without a record it stays where it is, and the first line reads the newer release stays in <folder>; never run docker compose there. Never run docker compose there. A laptop enrolled since the upgrade lost its access in the restore and must run winner login again, and a Device revoked since comes back unable to log in: revoke it again from the roster.

Sessions laptops pushed between the upgrade and the rollback are gone from the archive, and laptops do not push them again: they are only in the pre-rollback backup. When the two releases pin different collector versions, the last line says so: a laptop that ran winner update against the newer release runs the newer collector, which fails every push until its next winner update takes it back down.

A refusal prints one rollback-server: FAIL: line saying nothing was changed. A restore that fails before it replays changes nothing either. One that fails once it has started replaying leaves laptops fenced out and the stack stopped, and prints the two commands to choose from: scripts/restore-server.sh <pre-rollback backup> in the newer release's folder, which puts the newer release back as it was, or scripts/rollback-server.sh there again, which restores the pre-upgrade backup once more. A failure after the restore has passed leaves the database fenced, so no laptop writes to a database half way between two releases, and says which step failed: fix it, then run ./scripts/rollback-server.sh --finish in the same folder. A rollback cut off during its restore (a dropped SSH session or a reboot) is taken up by running ./scripts/rollback-server.sh again, which restores once more under the fence it finds. An upgrade that failed, or was cut off before it finished, has nothing to roll back: from the folder of the release it started from, the command refuses even where an earlier upgrade's record says done, and names the restore that undoes the attempt from that attempt's own pre-upgrade backup. To roll back the upgrade before it anyway, name its previous folder with --bundle and its backup with --backup.