Installing the Winner AI server¶

This page installs Winner AI on a server you control, from the release's self-hosting bundle, winner-ai-server-<X.Y.Z>.tar.gz, in eight steps: check the requirements, verify the bundle's signature and check the server, load the images, fill in .env, start, check the result, upload the licence and switch the features on, and enrol the laptops and schedule backups. Start here: the install in order says what has to be ready before step 1, and who does it. <X.Y.Z> is the release you were given, and <control-plane-host> is the hostname your team will reach Winner AI at. Installing a later release over it, and going back, is Upgrade and roll back.

The bundle holds everything of ours the server runs, so the server needs no account with any registry of ours:

images.tar: our compiled control-plane image and the Dashboard's and sign-in gate's images, all for linux/amd64;
caddy-linux-amd64: the web server that brands the dashboard, as one binary;
the compose files, .env.example, the configuration the stack mounts (deploy/), the installer scripts/install-server.sh, the host check it runs, scripts/check-host.sh, the .env check it runs, scripts/check-env.sh, the DNS zone check, scripts/check-dns-zone.sh, the command that fills .env from AWS Secrets Manager, scripts/env-from-secrets.sh, the backup and restore commands, scripts/backup-server.sh and scripts/restore-server.sh with the scripts/backup-lib.sh they share, the diagnostics command, scripts/diagnostics-server.sh, the upgrade command, scripts/upgrade-server.sh, and the rollback command, scripts/rollback-server.sh;
the Auth0 post-login Action, auth0-post-login-claims.action.js in the backend folder, which the operator deploys to the tenant by hand (The post-login Action);
NOTICES.md, the third-party notices, and this documentation as a PDF under docs/;
SHA256SUMS, one line per file above, and SHA256SUMS.sig, its signature with the Winner AI release key.

Winner AI is delivered as a signed installer. The server's code is compiled, which makes it hard to read and to change; it does not make it secret. Page templates, database scripts and configuration files are readable. The licence agreement is what protects our code.

1. Check the requirements¶

The server. The first release supports the server on linux/amd64 (x86-64) only, on Ubuntu with Docker Engine 28.0 or later and the Docker Compose plugin 2.24.4 or later. arm64 is not supported. Nothing is built on the server: the web server is the bundle's binary, mounted into a stock Caddy image, and where that binary cannot run the install fails once the stack has started (below). SELinux in enforcing mode, rootless Docker, Docker Desktop and Podman have not been tried. The user who runs the install must be able to run docker.

Installing Docker. On a server without Docker, follow Install Docker Engine on Ubuntu, which covers the Ubuntu releases it lists. First remove Ubuntu's own packages, which conflict with Docker's and may be older than the minimums above:

sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc docker-buildx podman-docker containerd runc | cut -f1)

Then add Docker's apt repository as that page describes, and install Docker Engine and the Compose plugin, the docker compose command the install uses:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

On a server that already has Docker, docker version and docker compose version tell whether it meets the minimums; the host check refuses one that does not. Docker's post-installation steps say how to let a user run docker without sudo; membership of the docker group is root-equivalent on the server, so give it only to the operator who runs the install, never a shared or service account.

Nothing on this list needs checking by hand: ./scripts/check-host.sh, which step 2 runs once the bundle is verified, checks the server, Docker, Compose, free disk, the ports, Docker Hub, the clock and ssh-keygen, and names every fault in one pass, one check-host: FAIL: line each; the script says what it checks and why. The installer runs the same check again, against .env, and stops before anything is installed on any fault, after the check's own lines:

install-server: FAIL: this host is missing what the install needs (above); nothing was loaded, pulled or started

For the server and Docker, the check prints:

The server	`check-host: FAIL:`
is not Linux on x86-64	`Winner AI's server runs on linux/amd64 only; this host is <os>/<arch>. Nothing is built here to work around it.`
runs a Docker daemon for another platform	`Winner AI's server runs on linux/amd64 only; this host runs a Docker daemon on <os>/<arch>. Nothing is built here to work around it.`
has no Docker	`docker is not installed; install Docker Engine 28.0 or later with the Compose plugin 2.24.4 or later.`
has an older Docker Engine or Compose	`Docker Engine <version> is older than 28.0, ...` or `Docker Compose <version> is older than 2.24.4, ...`
reports a version the check cannot read	`cannot read the Docker Engine version ('<what it printed>'); ...` or `cannot read the Docker Compose version ('<what it printed>'); ...`
has Docker but not the Compose plugin	`the Docker Compose plugin is not installed (docker compose fails: <its reason>); install it, 2.24.4 or later.`
cannot reach its Docker daemon	`cannot reach the Docker daemon as <user>: <Docker's reason>; run this and the install as a user who may run docker.`

Where the server passes those checks but still cannot run the mounted web server, the stack has already started when that shows, so the installer says so after the start:

install-server: this host cannot run the mounted web server (<the log line>); install on a linux/amd64 Ubuntu server with Docker Engine 28.0 or later
install-server: FAIL: the stack started but the web server is not the bundle's; the install did not succeed
install-server: the stack may still be running; to stop it, in <bundle folder> run:
  FLEETVIEW_VERSION=v<X.Y.Z> docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.bundle.yml down

In the dns certificate mode the printed command starts with COMPOSE_PROFILES= and the profiles the installer ran with, cert-dns among them.

Tools. ssh-keygen from OpenSSH 8.1 or later, the first release with ssh-keygen -Y verify (ssh -V says which; Ubuntu 20.04 and later ship a new enough one, in the openssh-client package), sha256sum and tar. openssl for the secrets in step 4 and the certificate check in step 6, curl for the host check and step 6, ss (the iproute2 package) for the host check's ports, and jq for the diagnostics command, which the host check does not look for.

Disk and memory. Where Docker keeps its data, the host check asks for 5 GB free, for the images and a first database, and 15 GB with the AI runner, whose image and models take the other 10 GB. Where Docker keeps its images in containerd's image store on another filesystem, it asks for 3 GB there (10 GB with the AI runner) and 2 GB where Docker keeps its data (5 GB with the AI runner). Leave room beside that for the unpacked bundle. No memory figure has been measured for this page.

Network.

A way to a publicly trusted certificate for <control-plane-host>, in one of two certificate modes (Network says which to choose):
http, the default: a public DNS name pointing at the server, with ports 80 and 443 reachable from the internet. The edge takes its certificate from a public certificate authority, which checks the name on port 80 or 443; the server needs outbound https to that authority.
dns, for a server on a private network that nothing on the internet can reach: a Route 53 public zone holding the name, and an instance role on the server that may write the challenge record in it, with the policy in Private VPC: the certificate through DNS. The dns-certs helper takes the certificate through a TXT record in that zone, and the edge and Postgres present it. The server needs outbound https to the certificate authority and the Route 53 API, and no inbound port for the certificate.

Ports 80, 443 and 5432 must be free on the server; the host check says which is not. - Port 5432 reachable from your team's laptops. They push their sessions to Postgres over TLS and verify its certificate, the same one the edge holds, against their system trust store. - Only 80, 443 and 5432 public. The compose files publish nothing else on a public address, and nothing on the server checks that for you: the host check says only whether the ports are free, never who can reach them. A firewall on the server itself (ufw, say) does not stop a port Docker publishes, because Docker publishes past it; filter in front of the server or in Docker's DOCKER-USER chain. - None of the server's networks, routes or VPN ranges may use 172.16.86.0/24. The stack's own network is pinned to it, and Postgres trusts that subnet as the stack itself, so a client reaching Postgres from an address inside it would pass as the stack. - Outbound https to Docker Hub. Our images come from images.tar and are never pulled from any registry. The server pulls the public images itself, from Docker Hub by digest, with no login:

When	Image
Always	the database (`pgvector/pgvector`), the edge and the web server base (`caddy`)
`local-llm` in `COMPOSE_PROFILES`	the AI runner (`ollama/ollama`), which then downloads its models from the Ollama registry
`CERTIFICATE_MODE=dns`	the certificate helper (`goacme/lego`)
`trial` in `COMPOSE_PROFILES`	the trial's sign-in provider (`ghcr.io/dexidp/dex`, from GitHub's registry, by digest); a trial runs no edge

Step 4 has the command that lists the exact references, each Docker Hub image pinned by digest, for the .env you write. Docker Hub limits pulls without a login per outbound address (step 5 says what a refused pull looks like). - Outbound https to your identity provider's issuer. The sign-in gate does not start without it. - A clock within 5 seconds of the real time, kept by time sync (NTP); the host check compares the clock with Docker Hub's.

An identity provider. Auth0, or any OpenID Connect provider such as JumpCloud. On another provider you need the gate's application there, a group for the Managers and the answers in Connect your identity provider, and step 4 takes its values from that page instead of the Auth0 ones. On Auth0 you need a tenant with the applications .env names, an organisation for your team, the post-login Action with its two secrets, and at least one manager. Setting up the tenant is not on this page; what it must allow is in Who may sign in, the Action is The post-login Action, and adding a manager is A manager. Connect your identity provider lists every answer the install asks of the provider.

2. Verify the bundle's signature and check the server¶

The installer checks the bundle's signature itself, against the release key it carries, on its own copy of SHA256SUMS, which is the only copy it reads after that, and before it loads any image; it refuses a bundle the release key did not sign. That check is only as good as the installer's own bytes, and a check an installer makes of itself proves nothing, so check them here first, by hand, with the stock ssh-keygen and the release key written out here, never a key taken from the bundle. The signature covers SHA256SUMS alone, so sha256sum --check then holds every file, the installer included, to it.

First, the release key. Write it out and print its fingerprint:

keys=$(mktemp -d)   # outside what the archive unpacks into
printf '%s\n' 'winner-release namespaces="release@getwinner.com" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMMDttCgimgvdz/6/0q4s5TC8RLUg783EsuPF4AGF0nW' > "$keys/allowed_signers"
awk '{print $3, $4}' "$keys/allowed_signers" | ssh-keygen -lf -

It must print SHA256:EKI6sQilBqVmM1HZ/jNEamMFhDX7GcJWl0ykl69k2hs. This page travels with the docs, and the docs may arrive by the same channel as the bundle, so confirm that fingerprint with your Winner AI contact over a different channel before you go on; where it differs, stop. The same public key is the one the laptop installer, which your control plane will serve at /install.sh, checks every winner download against.

Then the bundle, in the folder that holds the archive:

mkdir winner-install && tar --no-same-owner --no-same-permissions -xzf winner-ai-server-<X.Y.Z>.tar.gz -C winner-install
cd winner-install/winner-ai-server-<X.Y.Z>
ssh-keygen -Y verify -f "$keys/allowed_signers" -I winner-release -n release@getwinner.com \
  -s SHA256SUMS.sig < SHA256SUMS &&
sha256sum --check --strict --quiet SHA256SUMS   # prints nothing when every file matches

ssh-keygen must print a line that starts Good "release@getwinner.com" signature for winner-release, and sha256sum must print nothing. Anything else, Could not verify signature. or a file sha256sum names, means the bundle is not the one we signed: stop, and load nothing from it. Ask your Winner AI contact for the bundle again.

Why the commands look like this:

Unpack into a new, empty folder, so an archive that carries files beside the bundle folder cannot write over your own (~/.ssh, ~/.bashrc).
The key file is written outside the unpacked folder, so a link the archive planted cannot redirect that write.
sha256sum --check reads only the files SHA256SUMS names, so the installer later refuses a bundle folder holding any other file (beyond SHA256SUMS.sig and .env), or any link.

Then check the server, in the same folder, once both checks pass:

./scripts/check-host.sh   # every fault on one pass; fix them and run it again

It must end check-host: this host has what the install needs. It runs only after the signature and sha256sum checks, because a script from a bundle nothing has vouched for could do anything with the rights of a user who can run docker; a missing or too-old ssh-keygen is named by the signature check itself. It only reads: it loads, pulls and starts nothing. With no .env yet it reads .env.example, which keeps the AI runner and the disk it needs; to go without it, leave COMPOSE_PROFILES blank in the .env you make in step 4. The installer runs the check again, against .env.

To start again from this step, as step 5 asks for some failures, first set the old folder aside, then repeat the step from the release key on:

cd <the folder that holds the archive>
(umask 077 && cp winner-install/winner-ai-server-<X.Y.Z>/.env winner-env.saved)   # only to keep the .env you filled in
mv winner-install "winner-install.old.$(date +%s)"

With the bundle verified again, bring a kept .env back with (umask 077 && cp --remove-destination ../../winner-env.saved .env) in place of step 4's first command, then delete winner-env.saved.

3. Load the images¶

Nothing to run here: the installer loads images.tar itself in step 5, and refuses to go on when it does not hold the control plane for the bundle's release. It loads our control plane, ghcr.io/getwinner-com/fleetview-backend:v<X.Y.Z>, and the stock ghcr.io/kenn-io/agentsview and quay.io/oauth2-proxy/oauth2-proxy at the versions docker-compose.yml names. These images are never pulled from any registry; the Docker Hub images in step 1 are pulled in step 5.

images.tar is about a gigabyte, so loading it takes a while. To see that it loads before you fill in .env, you may run docker load -i images.tar in the bundle folder now, which prints one Loaded image: line per image; the installer still loads it again in step 5.

4. Fill in `.env`¶

(umask 077 && cp --remove-destination .env.example .env)

--remove-destination replaces anything the archive put at .env instead of writing through it, and the umask makes the copy readable by you alone before any secret goes into it. Then open .env and set the keys below. A line that starts with # KEY= is a comment: remove the # to set it. The comments in .env.example say what each key means, and Configuration lists every one.

Always required:

Key	Value
`POSTGRES_PASSWORD`	any strong value
`AUTH0_DOMAIN`	your Auth0 tenant's domain
`AUTH0_AGENT_CLIENT_ID`	the native device-code application a `winner` older than the control plane's own laptop sign-in uses (Configuration)
`AUTH0_PROXY_CLIENT_ID`, `AUTH0_PROXY_CLIENT_SECRET`	the sign-in gate's application
`AUTH0_TEST_ORG_ID`	the key of the installation's one organisation, which the control plane provisions and laptops enroll into: on Auth0, the organisation's id (`org_...`); on another provider, a name you choose. It needs a letter or a digit and must make a schema of at most 56 bytes. Choose it once: the schema is named from it at the first start, and a different key later makes another organisation (One organisation)
`FLEETVIEW_SERVE_DB_PASSWORD`	any strong value
`PGSERVE_AUTH_TOKEN`	any strong value
`OAUTH2_PROXY_COOKIE_SECRET`	32 random bytes, URL-safe base64

One of these, for who is a Manager:

Key	Value
`AUTH0_MGMT_CLIENT_ID`, `AUTH0_MGMT_CLIENT_SECRET`	the control plane's Management API application, both set, and roles come from Auth0
`FLEETVIEW_MANAGER_GROUP`	with both Management keys blank, `manager`: the Auth0 role the post-login Action sends (Configuration)
or `FLEETVIEW_MANAGER_EMAILS`	with both Management keys blank, the Managers' e-mail addresses, comma-separated (`ada@example.com,bob@example.com`), with `OAUTH2_PROXY_ALLOW_UNVERIFIED_EMAIL=false` (Configuration)

scripts/check-env.sh refuses one Management key without the other, and a blank pair with neither manager setting. The post-login Action refuses a sign-in through the gate's application from any Auth0 Organization but this installation's, or from none, once its two secrets, AUTH0_PROXY_CLIENT_ID and AUTH0_TEST_ORG_ID, hold the same values as this .env. Set them with the commands in The post-login Action when you deploy it: an Action deployed without them refuses every sign-in. Enable only this installation's Organization on the gate's application as well, the first check of the two: with the pair blank another Organization's Managers would otherwise become Managers here (Who may sign in).

Required because the server serves https:

Key	Value
`PUBLIC_BASE_URL`	`https://<control-plane-host>`, lower case, no trailing slash
`PUBLIC_HOST`	`<control-plane-host>`
`ACME_EMAIL`	the contact on the certificate authority account
`OAUTH2_PROXY_COOKIE_SECURE`	`true`
`PUBLIC_PG_HOST`	the hostname laptops dial for Postgres: `<control-plane-host>`, unless a CDN sits in front of it (Behind a CDN)
`PUBLIC_PG_SSLMODE`	`verify-full`
`FLEETVIEW_RELOAD_DB_PASSWORD`	any strong value
`OAUTH2_PROXY_OIDC_ISSUER_URL`	`https://<AUTH0_DOMAIN>/`, or another provider's issuer
`OIDC_END_SESSION_URL`	`https://<AUTH0_DOMAIN>/v2/logout`, or another provider's end-session address

On a provider other than Auth0, such as JumpCloud, the gate's settings, the manager group and the placeholders for the Auth0 keys it does not use are in Connect your identity provider (for JumpCloud, its own section). Leave both Management keys blank: with another issuer scripts/check-env.sh refuses them, because with them a person is found by e-mail in Auth0.

Strong values, one per key:

openssl rand -hex 32      # a password or token
openssl rand -base64 32 | tr -- '+/' '-_'   # OAUTH2_PROXY_COOKIE_SECRET: the gate reads only URL-safe base64

POSTGRES_PORT can stay as it is. A server installed from the bundle publishes Postgres on 5432 and tells laptops 5432, whatever it says; the template's 5433 is for a development machine and for a trial from the bundle.

Choose:

ORG_DISPLAY_NAME: your organisation's name, which the page headers show. Set it before the first start: it is read once, when the organisation is first made, and a later change does not rename it. Left unset, it is Winner AI.
COMPOSE_PROFILES: local-llm runs the AI runner on the server's CPU. Leave it blank (COMPOSE_PROFILES=) to run without it, once every model endpoint is a hosted one, and ollama/ollama is then never pulled. Choosing a model is Model and telemetry. To evaluate the bundle before you have a host name or a provider, add trial (below, "A trial from the bundle").
FLEETVIEW_FEATURES: which of Prompt Coaching, Skills and Benchmarks run, as a JSON list. Commented out, as in .env.example, it is []: all three pages show Soon and none of their model passes runs. Leave it so for the install; step 7 switches them on, in order: the model the FLEETVIEW_LLM_* keys name, its check, then the list, up to FLEETVIEW_FEATURES=["coaching","skills","benchmarks"] for all three. Skills also needs each laptop to run Ollama with ollama pull nomic-embed-text.
CERTIFICATE_MODE: leave it unset, or http, for the edge to take its own certificate; dns for the dns-certs helper, with AWS_HOSTED_ZONE_ID, the public zone's id, and AWS_REGION. No AWS key goes in .env: the helper signs with the server's instance role. Before the first start in the dns mode, run scripts/check-dns-zone.sh in the bundle folder: it proves, with the same role, that the zone takes and serves the challenge record and that the certificate authority answers, since a first issue that fails costs one of five a week. The check also needs outbound port 53 to the zone's own name servers, which the helper does not; where only https may leave, its read-back fails although the issue could work.

Leave alone: FLEETVIEW_TELEMETRY_SECRET, which the installer writes (step 5); FLEETVIEW_VERSION and COMPOSE_FILE, which the installer writes once the install has proved itself; FLEETVIEW_BACKEND_DIGEST and FLEETVIEW_CADDY_DIGEST, which an install from the bundle leaves blank; and DB_CERTS_SOURCE and DB_CERTS_SOURCE_DIR, which the installer sets from CERTIFICATE_MODE. In the dns mode it also adds cert-dns to COMPOSE_PROFILES, the profile that starts the helper; it writes that profile and the certificate source into .env once the install has proved itself, so a later docker compose up -d keeps the mode.

Every other key, what it means and its default, is in Configuration.

Keep the secrets in AWS Secrets Manager¶

On AWS you can keep .env's secrets in AWS Secrets Manager instead of typing them into the file. The stack still reads only .env: scripts/env-from-secrets.sh reads one secret with the server's instance role and writes its keys into .env, leaving every other line as it is. Use it when your secrets must live in Secrets Manager, or when you rotate them there.

The server needs the AWS CLI v2 and jq, neither of which a stock Ubuntu image has, and an instance role allowed to read the secret. A secret's ARN ends in six random characters, so name it in full or with -??????. Drop the kms:Decrypt statement when the secret uses Secrets Manager's default key; it is needed only for a customer-managed KMS key:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:<region>:<account>:secret:<name>-??????"
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:<region>:<account>:key/<key-id>",
      "Condition": {"StringEquals": {"kms:ViaService": "secretsmanager.<region>.amazonaws.com"}}
    }
  ]
}

Create the secret once, as one JSON object of KEY: value pairs, from a file only you can read, and then delete the file. Do this from a machine whose AWS credentials may create secrets: the server's role may only read it.

(umask 077 && cat > server-env.json)    # paste {"POSTGRES_PASSWORD": "...", "PGSERVE_AUTH_TOKEN": "..."}, then Ctrl-D
aws secretsmanager create-secret --region <region> --name <name> --secret-string file://server-env.json
rm -f server-env.json

Any key from the tables above may go in it. Leave out the keys under Leave alone, which the installer writes, and COMPOSE_PROFILES, to which the installer adds cert-dns in the dns mode: a later run would put the secret's value back over the installer's. Keys must be upper-case .env names, and values one line each, with no single quote and no backslash at the end; the command refuses anything else and says which key, without printing a value.

Then, in the bundle folder, before step 5:

./scripts/env-from-secrets.sh <secret-ARN>                       # the region comes from the ARN
./scripts/env-from-secrets.sh <name> --region <region>

It prints env-from-secrets: wrote <N> keys from <secret> into .env. With no .env yet it starts from .env.example, so set the keys not in the secret as above, in the same file. .env stays readable by its owner alone. Run it as the user who installs, as for the other steps; run as root, it keeps the file's owner.

An upgrade carries .env across to the new release, so it needs no new run. After a rotation, run the command again in the folder of the release that runs now, then restart with docker compose up -d there. The database keeps the password it was created with, so leave POSTGRES_PASSWORD in the secret as it is: a new value there leaves the stack unable to log in to the database.

The images this .env runs. With .env filled in, this lists the exact references, each Docker Hub image pinned by digest, the way the installer lists them:

FLEETVIEW_VERSION=v<X.Y.Z> docker compose -f docker-compose.yml -f docker-compose.prod.yml \
  -f docker-compose.bundle.yml config --images | sort -u

In the dns mode the installer adds the cert-dns profile, so put it in front: COMPOSE_PROFILES= set to the profiles in your .env, then cert-dns. With .env.example's local-llm:

COMPOSE_PROFILES=local-llm,cert-dns FLEETVIEW_VERSION=v<X.Y.Z> docker compose -f docker-compose.yml \
  -f docker-compose.prod.yml -f docker-compose.bundle.yml config --images | sort -u

With COMPOSE_PROFILES blank in .env, it is COMPOSE_PROFILES=cert-dns. Without the profile the list leaves out goacme/lego.

The ghcr.io/getwinner-com, ghcr.io/kenn-io and quay.io lines are the images images.tar holds, which the installer never pulls; every other line is an image the server pulls in step 5, ollama/ollama only while COMPOSE_PROFILES names local-llm, and goacme/lego only in the dns mode. For a trial, leave -f docker-compose.prod.yml out, as a trial's .env does; its ghcr.io/dexidp/dex line is the one image pulled from GitHub's registry.

Close the editor before you go on. The installer refuses a bundle folder holding any file SHA256SUMS does not name, an editor's swap or backup file of .env included:

install-server: FAIL: the bundle folder holds files SHA256SUMS does not name: .env.swp Unpack the bundle again into an empty folder. Nothing was started.

For a swap or backup file of your own, deleting it is enough.

5. Start¶

From the bundle folder, as the user who may run docker:

./scripts/install-server.sh

The first start takes a while, and longer than any one wait: after it hashes every file and loads images.tar, it waits in turn for the web server, the control plane and the edge, and each of those waits may last up to thirty minutes. Do not interrupt it. It first checks the signature, then runs the host check again, once its bytes match their signed line, and its check-host: lines come between the first two lines below:

install-server: SHA256SUMS is signed by the release key (SHA256:EKI6sQilBqVmM1HZ/jNEamMFhDX7GcJWl0ykl69k2hs)
install-server: installing v<X.Y.Z> from <bundle folder>
install-server: checking every file against SHA256SUMS (images.tar takes a while)
install-server: generated FLEETVIEW_TELEMETRY_SECRET in .env (never shown)
install-server: loading our images from images.tar
install-server: pulling <image>@sha256:<digest> from Docker Hub
install-server: starting the stack
install-server: the web server runs the bundle's binary
install-server: the edge serves ports 80 and 443
install-server: the org schemas hold only what agentsview creates
install-server: installed v<X.Y.Z>

with one pulling line per Docker Hub image. In the dns mode it also prints, around those lines,

install-server: the dns certificate mode: the dns-certs helper takes the certificate through the public zone, and the edge and Postgres present it
install-server: the certificate helper, dns-certs, holds a certificate
install-server: db-certs reads the helper's certificate; Postgres presents it from db-certs' next pass, within five minutes

and pulls the helper's goacme/lego image, which the http mode never does. The first issue takes a minute or more, and the edge serves nothing until it has the certificate, so the installer waits for the helper first. Postgres takes the certificate on db-certs' next pass, up to five minutes later; until then it presents a self-signed one and no laptop can push. It prints install-server: caddy-linux-amd64 set to mode 0755 before the checking line when the unpacked binary lost its mode.

The telemetry secret. Once scripts/check-env.sh accepts .env, and before anything starts, the installer writes a random FLEETVIEW_TELEMETRY_SECRET into .env when none is set. It never prints the value and never changes one already there, since a new secret would cut the deployment's past telemetry off from its future; then it prints kept the FLEETVIEW_TELEMETRY_SECRET already in .env instead, with a WARNING: when the kept value is shorter than 32 characters. .env is the secret's one record: keep it there, never beside the backups. What it turns on is Model and telemetry.

Once it prints installed, .env names the compose files and the release, so a later docker compose command in this folder (docker compose ps, docker compose logs, docker compose up -d) runs the same stack and still pulls none of our images.

When it fails, it prints install-server: FAIL: and the reason, and exits non-zero. Up to the starting the stack line nothing is running; fix what it names and run it again.

What it says	What to do
`this host is missing what the install needs (above); nothing was loaded, pulled or started`	Fix each `check-host: FAIL:` line above it (step 1), and run it again
`no SHA256SUMS in <folder>; ...` or `no SHA256SUMS.sig in <folder>; ...`	Run the bundle's own `./scripts/install-server.sh`, from the bundle folder, not a copy of it elsewhere. If the folder is the bundle's and still has no `SHA256SUMS.sig`, nothing vouches for it: ask your Winner AI contact for the bundle again
`SHA256SUMS does not match its signature in SHA256SUMS.sig: ...`, `SHA256SUMS.sig is signed by another key (...)`, `SHA256SUMS.sig was made in the licence namespace ...`, or `SHA256SUMS ... is a link or not a regular file`	The bundle is not one Winner AI signed, or it changed after it was signed; nothing was loaded. Start again from step 2 with a new, empty folder, and if step 2's `ssh-keygen` line fails too, ask your Winner AI contact for the bundle again
`ssh-keygen is not installed, so the bundle's signature cannot be checked; ...` or `sha256sum is not installed, ...`	Install the OpenSSH client or coreutils (step 1's tools), and run it again
`SHA256SUMS does not name one release (no single docs/winner-ai-docs-X.Y.Z.pdf line)`, `SHA256SUMS has no line for caddy-linux-amd64`, `SHA256SUMS has no line for scripts/check-host.sh`, or `images.tar holds no ghcr.io/getwinner-com/fleetview-backend:v<X.Y.Z>; this bundle is not release v<X.Y.Z>`	The signed bundle itself is not a whole release. Ask your Winner AI contact for a new bundle
`the bundle folder holds files SHA256SUMS does not name: ...`	Delete a file of your own it names (step 4); for any other, start again from step 2 with a new, empty folder
`the bundle folder holds links or other non-files, which no bundle ships: ...`, `scripts/check-host.sh does not match its line in SHA256SUMS; ...`, `scripts/check-host.sh is missing or not a regular file`, `these files do not match SHA256SUMS: ...`, `caddy-linux-amd64 does not match its line in SHA256SUMS; its bytes changed`, `caddy-linux-amd64 is missing or not a regular file`, or `the copy of caddy-linux-amd64 changed while it was made`	The bundle folder changed after you verified it. Start again from step 2 with a new, empty folder
`caddy-linux-amd64 is built for <architecture>, not x86-64; this host cannot run it. Nothing was started.`, or another `caddy-linux-amd64 is ...; this host cannot run it` such as `is not an ELF executable`	Run step 2's `sha256sum --check` line again in the bundle folder. If it names `caddy-linux-amd64`, the folder changed: start again from step 2 with a new, empty folder. If it prints nothing, the signed binary itself is wrong: ask your Winner AI contact for a new bundle
`no .env in <folder>. Run: cp .env.example .env then fill it in`	Do step 4
`scripts/check-env.sh refused .env (above); nothing was started`, after the check's own message, such as `Missing values in .env:` and the keys	Set what the message names (step 4), and run it again
`.env sets FLEETVIEW_TELEMETRY_SECRET on <n> lines; keep one, ...`	Delete every line but the one holding the value the deployment already uses, and run it again
`FLEETVIEW_TELEMETRY_SECRET is set in this shell but not in .env; ...`	Put the value in `.env`, its one record, or `unset FLEETVIEW_TELEMETRY_SECRET` in the shell to have one generated, and run it again
`compose cannot read its files with this .env: <compose's last line>`	Compose's own words name the key; usually a required value is missing or a line is malformed. Fix it in `.env` (step 4)
`docker load could not load images.tar: <Docker's output>`	Docker's words give the reason, most often no space left for the images. Free the space, and run it again
`<image> is neither in the bundle nor a pinned Docker Hub image; nothing was started`	Something in `.env` changed which image a service runs, or the bundle is wrong. Check the keys step 4 says to leave alone, list the images with step 4's last command, and if the line still shows, ask your Winner AI contact
`.env's COMPOSE_FILE names docker-compose.prod.yml, so this folder holds a production install; ...`	A trial was asked for in a folder that holds a production install. Unpack the bundle into a folder of its own for the trial
`compose project <name> has containers of ... from a production install, ...`	A production install shares the trial's compose project: its edge would go on serving ports 80 and 443, and compose names the database volume by the project, so the trial would open the production database. Give the trial another `COMPOSE_PROJECT_NAME` in `.env`; stopping the production install does not help, as its volume stays
`compose project <name> has volumes ... which a trial does not define: ...`	A production install's volumes, its database among them, share the trial's compose project; `docker compose down` keeps them. Give the trial another `COMPOSE_PROJECT_NAME` in `.env`
`could not pull <image> from Docker Hub: <Docker's last line>. Nothing was started.`	Docker Hub refused the pull or could not be reached; the reason is in Docker's own words. A refusal for the rate limit means the server's outbound address used up Docker Hub's allowance for pulls without a login (100 per IPv4 address, or IPv6 /64, every 6 hours, at the time of writing), which a server sharing its address with other Docker users can reach. Wait, and run the installer again
`this bundle is v<X.Y.Z>, older than v<newer> which this server runs; migrations do not reverse, ...`	A newer release already runs on this server's database, and an older one would start on tables it does not know. `scripts/rollback-server.sh` in the folder of v goes back to the release the last upgrade to v replaced, if one did; for an older release, name its folder with `--bundle` and the backup taken before the upgrade from it with `--backup` (Going back to the previous release); after an upgrade that failed, run the restore command it printed instead. The installer sees only a stack that runs: over a stopped one it does not know which release last ran, and an older bundle is not refused

A failure the installer did not foresee reads unexpected error at line <n> (<command>); quote it to your Winner AI contact.

After the start, from the starting the stack line on, a failure may leave some or all of the stack running, so the installer prints the command that stops it (step 1 shows its shape, and in the dns mode it starts with COMPOSE_PROFILES=), because .env does not yet name the compose files or the release. A docker compose up failure can come before any service starts; the stop command is then harmless. Fix what it names and run the installer again, or stop the stack with that command. The failures there:

docker compose up failed (above), after compose's own output, which names the service and the reason.
could not remove dns-certs and edge-reload in the http certificate mode: an earlier dns-mode install left the certificate helper behind, and the installer could not remove it. Run the installer again; if the line stays, send it to your Winner AI contact.
the web server container was not created; the install did not succeed.
the stack started but the web server is not the bundle's; the install did not succeed, after a line saying which: the mount over /usr/bin/caddy is missing, the binary differs, it lacks the brand plugin, the web server keeps restarting, or the host cannot run it (step 1).
/health did not answer ok within 1800s (it said '...'); the install did not succeed.
/internal/version reports '...', not v<X.Y.Z>; the install did not succeed.
In a trial, the trial's sign-in provider, Dex, was not created; the install did not succeed, or the trial's sign-in provider, Dex, is not running (...), ending with Dex's last log line, which says why.
the edge container was not created; the install did not succeed.
the edge, which serves ports 80 and 443, is not running and healthy (...), ending with the edge's last log line. In the http mode, check that <control-plane-host> resolves to the server and that ports 80 and 443 reach it.
the certificate helper, dns-certs, has no certificate (...), ending with the helper's last log line, or the certificate helper, dns-certs, was not created: in the dns mode the helper could not get the certificate, and nothing serves until it does. docker compose logs dns-certs says why; scripts/check-dns-zone.sh checks the zone, the instance role and the certificate authority. The helper retries on its own an hour later; once it holds a certificate, run the installer again.
edge-reload ... is not running, edge-reload does not run in the edge's PID namespace, or db-certs does not read the helper's certificates: the dns mode's wiring did not come up as compose was told. Run the installer again; if the line stays, send it to your Winner AI contact.
an org schema holds code stock agentsview does not create, with the findings: Code in an org schema that agentsview did not create says how to clear it.
the org-schema code audit did not run (exit <n>):, with the audit's last lines, which say why.

6. Check the installation¶

Health, from any machine:

curl -sS https://<control-plane-host>/health      # {"status":"ok","database":"ok"}
curl -sSI http://<control-plane-host>/ | head -1  # 308, to https

Every service up, in the bundle folder; each with a health check reads healthy:

docker compose ps

Postgres presents a public certificate, which is what laptops verify:

echo | openssl s_client -connect <PUBLIC_PG_HOST>:5432 -starttls postgres \
  -verify_return_error -verify_hostname <PUBLIC_PG_HOST> 2>/dev/null | openssl x509 -noout -issuer -dates

<PUBLIC_PG_HOST> is the value of PUBLIC_PG_HOST in .env, the name laptops dial (step 4), usually <control-plane-host>. The certificate it prints must be issued by a public certificate authority and be within its validity dates. The check verifies that the certificate chains to an authority this machine trusts and was issued for that name, as a laptop does; where either fails, it prints Unable to load certificate instead and exits non-zero.

Sign in. From a browser, open https://<control-plane-host> and sign in as a manager of the organisation AUTH0_TEST_ORG_ID names (A manager). Reach Home and the Dashboard, then sign out; sign-out must end both the gate's session and Auth0's.

Enroll the first laptop. On a laptop of someone in the organisation, install the client from your server and sign in. A new laptop sends no session until its developer chooses which repos it sends (Choose which repos send): tick at least one in the list winner login shows, or the dashboard stays empty (a trial has lines of its own, in A trial from the bundle):

curl -fsSL --proto '=https' --proto-redir '=https' https://<control-plane-host>/install.sh | bash
winner login --control-url https://<control-plane-host>
winner push status

winner push status must report the background push running (Laptop client guide covers the laptop side). Then, in Admin (/admin), the laptop is on the roster; within the push interval it shows a session count, "· last push <time>", "· last change <time>" or "· push seen, time unknown", which is what proves a push arrived. "· awaiting first push" alone does not.

When the laptop's push fails to connect to Postgres, port 5432 must reach the server from the laptop's network (step 1).

Steps 7 and 8 finish the install. When something goes wrong from here on, Troubleshooting is the symptom index.

7. Upload the licence and switch the features on¶

Prompt Coaching, Skills and Benchmarks are all off by default, because each sends prompt text to the model unredacted (What each feature sends to the model). Switch them on in this order:

Upload the licence. A Manager signs in, opens Settings and uploads the licence file Winner AI sent (Upload it). Until then the deployment runs on 30 days of grace from its first start; a listed feature is live only while the licence is valid or in its grace, and once it lapses all three leave the sidebar.
Point the stack at the model. With local-llm the default FLEETVIEW_LLM_* values call the server's own AI runner, slowly on CPU, and there is nothing to set. For a hosted model or Amazon Bedrock, set the model keys in .env and recreate the services that read them, as Point the stack at a hosted model says (Amazon Bedrock for Bedrock). Set the Benchmarks Routes, FLEETVIEW_BENCHMARK_ROUTES, to model ids your provider serves; on Amazon Bedrock, to Bedrock model or inference-profile ids (Benchmarks Routes on Bedrock).
Check the model, in the install folder:
```
docker compose exec -T jobs uv run --no-dev fleetview-model-check
```
It ends with a verdict; Check the model before Prompt Coaching says how to read it, and how to give a slow local model more time.
List the features in .env: skills and benchmarks, as FLEETVIEW_FEATURES=["skills","benchmarks"], and coaching only once the model check passes, as FLEETVIEW_FEATURES=["coaching","skills","benchmarks"]. A team can switch a listed feature off on Settings, never on.
Recreate the services that read it, in the install folder:
```
docker compose up -d backend jobs
```
Check. The pages fill once laptops have pushed sessions (step 8) and a jobs sweep has run over them: the recreated jobs service sweeps as it starts, then every 15 minutes by default. Skills and Benchmarks show their first results after the next sweep that follows a push. Switch Prompt Coaching on says how to read a sweep's result.

8. Enrol the laptops and schedule backups¶

The laptops. Each developer installs the client from your server, signs in, and ticks the repos it may send in the list winner login shows:

curl -fsSL --proto '=https' --proto-redir '=https' https://<control-plane-host>/install.sh | bash
winner login --control-url https://<control-plane-host>

Hand them the Laptop client guide: it covers macOS and Linux, choosing repos and what to expect day to day. Skills also needs vectors from each laptop: each laptop needs Ollama, which Winner AI does not install, with ollama pull nomic-embed-text (Embeddings).

Backups. Nothing on a server installed from the bundle schedules a backup or copies one off the server. Schedule scripts/backup-server.sh and copy each backup to storage off the server, as Schedule it and copy it off the server shows, then rehearse a restore once (Rehearse a restore). Nothing watches the certificate's expiry either: Certificate renewals has the check to schedule.

A trial from the bundle¶

To evaluate the bundle before you have a host name or a provider, install it as a trial: put trial in COMPOSE_PROFILES, uncomment the trial block of .env.example (Connect your identity provider, "A trial with the bundled Dex"), and leave PUBLIC_BASE_URL blank or http://localhost:4180. The installer reads the profiles from .env, never from your shell, and then stacks docker-compose.bundle.yml on the base file alone, without the production overlay: no edge, nothing published beyond loopback, the gate at http://localhost:4180 on that machine only, and no edge check at the end. The host check, reading the same .env, then needs ports 4180 and POSTGRES_PORT (5433 unless set) free instead of 80, 443 and 5432. Step 2's run of it comes before there is a .env, so it reads .env.example and checks the production ports: once .env names the trial, run ./scripts/check-host.sh again, and take its answer on ports over step 2's. The installer runs it against .env either way. scripts/check-env.sh refuses a trial with an https base URL or with docker-compose.prod.yml in COMPOSE_FILE, and the installer refuses a trial in a folder that holds a production install (step 5's table). It removes nothing. Compose names the database volume by the project, not the folder, so a trial runs as its own project: when .env names none, the installer writes COMPOSE_PROJECT_NAME=fleetview-trial into it, and a production install made later on the same machine, under the default project fleetview, never shares the trial's database. The project comes from .env alone, never from your shell, for a trial and a production install alike. Unpack the trial into a folder of its own, and leave its COMPOSE_PROJECT_NAME as the installer wrote it. With local-llm kept beside trial, as .env.example suggests, a trial runs the AI runner too, so listing the three pages in FLEETVIEW_FEATURES (step 7) turns them on; they fill once laptops push sessions and a jobs sweep has run them on that model, and Skills also needs vectors: each laptop needs Ollama and ollama pull nomic-embed-text (Embeddings). Left out, they show Soon. A trial is for evaluation only; a production install starts from a fresh .env.

Enroll a laptop against a trial. The production line in step 6 refuses a trial: its --proto '=https' flags reject the gate's plain http://localhost:4180. A trial installs the client and signs in with:

curl -fsS --noproxy localhost http://localhost:4180/install.sh | bash
winner login --control-url http://localhost:4180

Dropping the flags is safe only because the address is loopback: the installer still checks winner against the release signature, and winner login accepts plain http to a loopback address and nothing else. Never drop them for a production host. --noproxy localhost keeps a laptop's http_proxy from carrying the script, which no signature covers; the installer's own downloads follow the shell's proxy settings, so a laptop that sets one also needs localhost in no_proxy. The laptop must be the trial machine, or reach it through a tunnel that forwards both the gate and the POSTGRES_PORT in the trial's .env (5433 unless set) at the same numbers, both free on the laptop, since Dex and the laptop's push know only localhost:

ssh -N -L 4180:localhost:4180 -L <POSTGRES_PORT>:localhost:<POSTGRES_PORT> <trial-machine>

Keep the tunnel up while the laptop is in use: its background push and update timer go through it, and with it closed their pushes fail quietly.

Open the approval page winner login prints in that laptop's browser and sign in as one of the trial's Dex accounts (Connect your identity provider, "A trial with the bundled Dex"). Leave PUBLIC_PG_HOST and PUBLIC_PG_SSLMODE blank in a trial's .env: laptops are then told localhost with no TLS, which loopback is exempt from.

Collecting a diagnostics file¶

A support case starts with one redacted file from the server. Run the command as root from the folder of the release that runs now: the one you ran ./scripts/install-server.sh in, or after an upgrade the folder Upgrade and roll back left running. Unlike the server's other commands (below), it runs with sudo and refuses to run without it. It needs jq on the server (apt install jq or dnf install jq):

sudo ./scripts/diagnostics-server.sh                  # 200 lines a service, into your home directory
sudo ./scripts/diagnostics-server.sh --lines 1000     # more lines

It prints the path of the 0600 file it wrote and sends nothing. It writes into your home directory, never the release folder: a file there that SHA256SUMS does not name stops the next install or a rollback from that folder. --out DIR writes elsewhere. Run through sudo from your own account, it takes a directory only you (and your own group) can write, or one with the sticky bit such as /tmp; from a root shell, only a directory only root can write, so not /tmp. It opens no connection and reads no organisation's data, so it works with the database down.

The file holds the build version and commit, each service's image and state, every setting on the Settings ledger with the layer it came from (a key only as set or not set, an endpoint without credentials or query), and each service's last log lines. Those lines lose organisation and user ids, device roles, client addresses, UUIDs, e-mail addresses, URL query strings, the row data and statement text a Postgres error can carry, and the value of every secret setting; a team's own model key is never read. Any other secret in a log line is caught only where a pattern names it: the masking errs toward too much, but it cannot prove a line holds nothing private, so read the file before you attach it.

Commands on the other pages¶

The other pages write the server's commands for a server installed from the bundle. Read them this way:

Run the server's commands in <install>, the running release's folder (Back up, restore and uninstall says which), as the user who owns <install>/.env and can run docker, without sudo. The installer leaves .env readable by that user alone, and compose reads the list of compose files from it, so compose run by another user who can run docker fails or runs without the bundle's files. Read .env there with plain grep.
Once the installer has printed installed, docker compose needs no list of compose files and no COMPOSE_FILE= in front of it: COMPOSE_FILE in .env already names the bundle's. A command you meet written for another layout, run in /srv/fleetview or with -f docker-compose.yml -f docker-compose.prod.yml, is plain docker compose in <install>: that pair would leave out docker-compose.bundle.yml, which keeps compose from pulling an image of ours.
There is no make. make check-env is ./scripts/check-env.sh, run in the bundle folder, before the install as well as after it: in an unpacked bundle it does not ask for the values the installer writes. scripts/install-server.sh runs the same check itself (step 5). make up is docker compose up -d. Every other make target, make eval included, needs a checkout of the release, not the server.
Nothing schedules a backup or watches the certificate's expiry on a bundle install unless you set it up (Back up, restore and uninstall, Certificate renewals). The bundle ships no Postgres jail: its commands apply once you set one up (The Postgres jail).

Later releases¶

To install a later release over this one, or go back to the one an upgrade replaced, see Upgrade and roll back.