Troubleshooting

A symptom index for whoever runs a deployment. Each row names the likely cause, the command that confirms it and the section that fixes it; the fix itself lives on that page, or in the row when it is short. Each command here says where it runs: the server, the person's laptop, a shell whose Auth0 CLI is signed in to the tenant (auth0 login), or a machine outside the server.

On an installation from the bundle, read the server's commands as Commands on the other pages says: in <install>, the folder of the release that runs now, as the user who owns <install>/.env, with plain docker compose and no sudo. Rows that use the Auth0 CLI need a shell signed in to your tenant.

You see	Likely cause	Confirm with	Fix
Prompt Coaching, Skills or Benchmarks shows Soon	It is not in FLEETVIEW_FEATURES, which is empty by default, or the team switched it off on Settings	grep ^FLEETVIEW_FEATURES= .env (server); the Features on row on Settings	Switch Prompt Coaching on
Signing in ends on Auth0's error page about the callback or redirect address	The sign-in application does not list <PUBLIC_BASE_URL>/oauth2/callback, or PUBLIC_BASE_URL is not the address it lists	curl -sS -o /dev/null -w '%{redirect_url}\n' https://<control-plane-host>/ (a machine outside the host) prints the address the gate sends browsers to, with its redirect_uri	On Auth0, Who may sign in, the callback and logout addresses; on another provider, Connect your identity provider
A laptop's pushes time out or are refused before any login, and winner push status names the failure	Port 5432 does not reach the server from where the laptop is: a firewall, or the database name resolving elsewhere; or a restore that failed left the database fenced; or, where you set up the Postgres jail, it banned the address (the jail row below)	nc -vz <database-host> 5432 (laptop): a timeout is the network, a refusal the server; docker compose exec db sh -c 'psql -U "$POSTGRES_USER" "$POSTGRES_DB" -c "SHOW listen_addresses"' (server) prints localhost when fenced. A fence writes no log line, and a failed restore leaves the backend stopped: the restore that left it printed restore: FAIL	Network; for a fence, Restore
After a restore, one laptop's pushes are refused while others push	It was enrolled after the backup was taken, so the restored database does not admit its role	winner push status (laptop)	Restore: its owner runs winner login again
On a bundle install, browsers warn about the certificate or laptops fail on it, after weeks of working	The edge could not renew: port 80 or 443 closed to the certificate authority, or a name's DNS record changed. Nothing watches expiry on a bundle install	curl -sSvo /dev/null https://<control-plane-host>/ (a machine outside the host) prints the certificate's expire date, or certificate has expired; docker compose logs --tail 200 edge (server)	Certificate renewals
On a bundle install, no new backup, or backup.log ends in backup: FAIL or backup: refused	The line says why: often counts that differed because a laptop pushed mid-run, a name that exists, a full disk, or a restore in progress	tail -n 20 ~/winner-backups/backup.log (server)	Take a backup
The AI runner's log shows its model pull failing	The server cannot reach the Ollama registry, or the storage host it redirects to	docker compose logs --tail 100 ollama (server)	Allow outbound https to the registry and its storage host, as What the server reaches lists them
A member sees "Manager role needed"	By design: the manager pages and the Dashboard are for managers; a member's Home shows only their own sessions, and they also have /devices, their own laptops' repos at /repos, their own findings at /coaching, their own candidates at /skills and their own patterns at /benchmarks	On Auth0, auth0 api get organizations/<org id>/members/<user id>/roles (Auth0 CLI); on another provider, the person's groups claim against the manager group and the e-mail list that Settings shows	Who is a Manager
Sign-in refused with "Your email address is not verified. Verify it, then sign in again."	The post-login Action: Auth0 has not verified the address, and the control plane did not create the account	auth0 api get "users-by-email?email=<address, @ as %40>" shows email_verified (Auth0 CLI)	The post-login Action
Sign-in refused with "This installation does not serve the organization you signed in with. Sign in through your own organization."	The post-login Action: the sign-in came through another Auth0 Organization or none, or the Action's AUTH0_TEST_ORG_ID secret holds another installation's value	auth0 api get actions/actions/<action id> (Auth0 CLI) lists the secrets' names, never their values, under deployed_version.secrets; Auth0 shows no value back, so run the update with AUTH0_PROXY_CLIENT_ID and AUTH0_TEST_ORG_ID from <install>/.env	The post-login Action
Sign-in refused with "Sign-in is not configured for this installation. Ask its operator to set the post-login Action's secrets."	The post-login Action was deployed without AUTH0_PROXY_CLIENT_ID or AUTH0_TEST_ORG_ID, or with one empty, and refuses every sign-in on the tenant	auth0 api get actions/actions/<action id> (Auth0 CLI) lists the secrets' names under deployed_version.secrets	The post-login Action: run the update with both values
winner login fails with 403 "user is not a member of the organization"	A winner from before the control plane's own login enrolled a person who is not a member of the bootstrap organisation, the one its Auth0 device flow resolves	auth0 orgs members list <org id> (Auth0 CLI)	Run the installer again on that laptop for a current winner, whose login goes through the control plane: Install on macOS, Install on Ubuntu or Debian. Why: Members and the first team
The hostname reporter's log says the control plane no longer accepts this laptop's pass	The laptop's device was revoked, its person was removed, or a later login replaced its pass	winner report (laptop) prints the same line and exits 1 after one request	Run winner login on that laptop: Sign in. If its sign-in page says the account was removed, ask a Manager: Leaving
winner login says "this client is older than the server's sign-in", or, from a client up to 0.12.0, that the control plane "returned an incomplete config"	The laptop's winner predates the sign-in the control plane serves	winner update --check (laptop) names the control plane's winner version; up to 0.12.0 it fails the same way	Run the update step the message names; up to 0.12.0, run the installer again: Install on macOS, Install on Ubuntu or Debian
winner login says "Could not list this laptop's repos (... listing repos needs git 2.36 or newer), so none are chosen and nothing is sent", winner repos fails with the same reason, or winner login says "Kept sending every repo: could not read every name agentsview holds (... needs git 2.36 or newer)"	The laptop's git is missing or older than 2.36, as stock git is on Ubuntu 20.04 and 22.04 and Debian 11; "could not ask git its version (...)" means git itself failed, which on a Mac is usually /usr/bin/git without the Command Line Tools	git --version (laptop)	Requirements says how to get git 2.36 or newer; then winner repos pick (laptop), or winner login where it kept sending every repo
winner login refuses because this laptop's agentsview is newer than the control plane's	The control plane's collector version is the ceiling a laptop may run	winner update --check (laptop) reports both versions on any platform; the install it previews runs only on macOS and Linux, amd64 or arm64	The collector version ceiling
The roster shows a laptop Stale, or its sessions stop arriving	The background push stopped, or its pushes fail	winner push status (laptop)	Day to day
Support asks what a laptop's push is doing	Support needs the laptop's versions, push state and recent push log	winner diagnostics (laptop) writes one file to attach	Collecting diagnostics
A laptop's pushes fail on a refused connection, on a server where you set up the Postgres jail (the bundle ships none)	The jail banned its address after repeated refused logins, usually a stale device password	sudo fail2ban-client status fleetview-postgres (server; fail2ban is the host's, not the stack's, so this one needs sudo), and fleetview-postgres-vpn where you added it	The Postgres jail
The jobs sweep's dead-man's switch (FLEETVIEW_HEARTBEAT_URL) pages	No ping arrived: the jobs service stopped or hangs. Or a ping reported a failure: a sweep that raised, a lost database included, or one in which an organisation's sweep raised. A single pass that fails is logged (pass <name> failed for <org>) and does not fail the ping	docker compose ps --all and docker compose logs --tail 500 jobs (server); a failed sweep logs jobs sweep failed	docker compose up -d (server) restarts a stopped service; a failed sweep names its reason in the log
The edge serves no valid certificate, so browsers warn and laptops cannot push with verify-full	The edge could not obtain a certificate over ACME: the hostname has no public DNS record, port 80 is not reachable from the internet, or a CAA record refuses the authority	docker compose logs --tail 200 edge (server) shows its certificate attempts; from outside the server, curl -sSI http://<control-plane-host>/ answers 308 when port 80 is reachable	Public deployment
No sign-in page in the dns certificate mode, and the edge log says edge: no certificate from the dns-certs helper yet	The dns-certs helper holds no certificate yet: a wrong AWS_HOSTED_ZONE_ID, an instance role without the zone, no outbound https to Let's Encrypt or the Route 53 API, or a CAA record that refuses Let's Encrypt	docker compose logs --tail 200 dns-certs (server) carries lego's error	Private VPC: the certificate through DNS, Certificate renewals
A laptop on the VPN cannot resolve the host, or its pushes fail with the name not found	Its VPN client did not apply the pushed resolver, or the split-tunnel routes do not carry the resolver	getent hosts <control-plane-host> on Linux, dscacheutil -q host -a name <control-plane-host> on a Mac (laptop, on the VPN), must print the private address	AWS Client VPN
The backend stops at start with Auth0 tenant <domain> is missing the role(s) ...	The tenant has no role named manager or member, which member creation resolves by name; with Management credentials the control plane checks both at start and refuses to start without them	docker compose logs --tail 200 backend (server)	Create the missing role in the Auth0 Dashboard under User Management, Roles, as the message says, then docker compose up -d backend (server)
The installer or an upgrade fails with an org schema holds code stock agentsview does not create, or the backend log names org schema … holds … owned by …	A laptop's device role planted a function, view, trigger or other code object in the organisation's schema, or one was left behind	The findings the line lists	Code in an org schema that agentsview did not create
Home's Team spent reads "unavailable"	The backend's spend read from pg serve is failing or its stored figure is stale, or the organisation has no pg serve; the tile's note says which	docker compose logs --tail 200 backend (server) shows the pg serve … summary: line with the reason	The read waits 120 seconds, so a timeout there is a pg serve or database problem, not a slow page; start with the pg-serve and db services' logs
Every Dashboard change (star, pin, rename, trash) fails with cross-site request refused by the edge	People browse an address that is not PUBLIC_BASE_URL, which the web server compares with each write's Origin exactly	grep ^PUBLIC_BASE_URL= .env (server), against the browser's address bar	Open the server at https://<PUBLIC_HOST>, the only PUBLIC_BASE_URL the .env check accepts on an https install; changing PUBLIC_BASE_URL would also move the sign-in callback
The jobs log's rows written this sweep line counts more than 0 passes had no model to call	No usable model: the host does not lend its model and the team set no key, the team's model is incomplete, the team key cannot be read with this host's FLEETVIEW_SETTINGS_KEYS, or, with no model lent by the host, the team endpoint is one of the installation's own services	The Model rows on Settings say why; docker compose logs --tail 500 jobs (server) names each pass that was skipped for or runs without a model for an organisation, with the reason	Switch Prompt Coaching on; Team settings
fleetview-model-check ends verdict: fail with refused: or the model call failed: on the line directly above	In that order: the endpoint is neither https nor a compose-internal host, so the key was never sent, or under FLEETVIEW_LLM_PROVIDER=bedrock the region is not an AWS region or the endpoint is not an https Bedrock runtime host; the endpoint refused the calls, with the status it answered, a call ran past FLEETVIEW_LLM_TIMEOUT_SECONDS, or under Bedrock no AWS credentials found neither a key nor an instance role	The command itself, in the jobs container (server)	Check the model before Prompt Coaching; Amazon Bedrock
Home says "No organization yet."	No organisation provisioned on this deployment holds the person: the bootstrap organisation is not set or names another organisation, startup provisioning failed, or the person has no row here and none of the Auth0 organisations they belong to is provisioned here	docker compose logs --tail 200 backend (server) shows startup provisioning failed or the installation's Org was not provisioned: ... when it did; auth0 api get "users/<user_id>/organizations" (Auth0 CLI) lists the person's organisations	Authentication, the AUTH0_TEST_ORG_ID row, which is the .env key behind the FLEETVIEW_BOOTSTRAP_ORG_ID the page names. The control plane cannot create an organisation: create it in your provider, add its people, and set its id there
The Dashboard answers 403 "this installation serves another organization"	The person manages an organisation other than the one AUTH0_TEST_ORG_ID names; an installation serves that one alone	docker compose logs --tail 200 backend (server)	By design: an organisation this installation served before keeps its rows, and its people still sign in, but it has no dashboard and no spend
The Dashboard answers 502 and pg-serve restarts with "names no schema for this installation's Org"	The backend has not provisioned the Org the key names: the key is unset, or startup provisioning failed	docker compose logs --tail 50 pg-serve and docker compose logs --tail 200 backend (server), whose log shows startup provisioning failed or the installation's Org was not provisioned: ... when it did	Authentication
The Dashboard answers 503 "No dashboard is provisioned for your organization."	The web server and pg serve were started for a different AUTH0_TEST_ORG_ID than the backend now holds: the key changed and only the backend was restarted. A pg serve still running for another organisation loses its database login at the backend's start, so until the whole stack is brought up again spend reads as unavailable too: the backend reads no spend while another organisation's serve role can still log in or is connected	Bring the whole stack up again: docker compose up -d (server)	Authentication, the AUTH0_TEST_ORG_ID row

Code in an org schema that agentsview did not create

An organisation's stock schema holds two functions, three triggers and its own CHECK constraints, and nothing else that runs code. Any other function, view, trigger, rule, operator, policy, CHECK constraint, generated column or CHECK on a domain an org column uses there was planted by a laptop's device role, and runs with the rights of whoever reads or writes past it, a restore included. The owner it reports is not proof either way. The installer and each upgrade refuse to finish on one, and the backend and jobs log each at ERROR on every start. A view, materialized view, foreign table or policy also stops that team's reads: Home and the roster's session counts show nothing, and the Coaching, Skills and Benchmarks pages fail, until it is cleared. Row-level security switched on with no policy is not a finding.

Nothing is dropped for you: whether the object is an attack or a leftover is a judgement. List and read them in <install>:

docker compose exec -T backend uv run --no-dev fleetview-provision audit   # one line per finding; changes nothing
docker compose exec db sh -c 'psql -U "$POSTGRES_USER" "$POSTGRES_DB"'      # then \df+, \do+ or \d+ <schema>.<name> to read it

1. Revoke the laptop. Where the owner is a device role (dev_…), revoke that laptop on Admin first, so it cannot plant the object again.
2. Drop each object you judge planted, with the DROP for the kind the audit printed. This step is only for a code object: a line that starts with a team's schema and names the object's kind second (a function, view, trigger and the like). The grant lines and the - lines below are never cleared with a DROP. Never drop or recreate the schema itself: it is your team's archive, and nothing restores it short of a backup. Where Postgres refuses and suggests dropping an extension instead, do not: that removes everything the extension holds. Write to support with the message.
3. Run again what refused: the installer, the upgrade, or the backup or restore command.

A rollback that refuses on this leaves the database fenced and the stack stopped, so the audit above cannot run. Write to support with the rollback's lines before you change anything; once the object is cleared, in the folder of the newer release, run scripts/rollback-server.sh --finish.

The same audit also prints grant, grant under another grantor, excess grant, default grant and foreign owner lines: one laptop gave another, or a role that is not the team's, rights over the team's tables. A default grant line is the same right on everything one of the team's roles creates later; the line says whose and what. The backend's start revokes a grant or default grant it can, so one still listed was made since or was refused at that start. Clearing these takes superuser statements specific to each line, so write to support with the audit's lines, and revoke the laptop the line names on Admin meanwhile.

A line whose first field is - is about the whole database server, not a team's schema:

• database postgres public=CONNECT or database template1 public=CONNECT: every role may connect to one of Postgres's own databases, which no laptop's role should.
• function pg_catalog.lo_...(...) public=EXECUTE: every role may run one of Postgres's own large-object functions, through which laptops of two organisations could pass data.

These are Postgres's own databases and functions, not something a laptop planted: never drop or alter them. The backend takes both rights away at every start, so a line here means a right came back since (a restore into a new server brings both back) or that start's revoke failed, which the backend log records at ERROR (could not revoke ..., ... stays executable by every role, or database ... stays open to every role). Restart the backend with docker compose restart backend, then run the audit again. If a - line stays, write to support with it and the backend's ERROR line.

When in doubt about any line, write to support before you drop anything.

Getting help

When a row here does not settle it, write to support as Where to write says. Add what the Confirm column above showed, except log output, and the exact line a failed backup-server.sh or restore-server.sh printed. Attach the server's redacted diagnostics file, as Collecting a diagnostics file says, after reading it yourself. Do not attach raw server logs, which can name people, laptops and your server, and never send .env, a backup, a key or a password.

What changed in a release

Each release's changes are on the changelog; its Requires of you part says what you must change and whether your people must reinstall the client or sign in again. The guides an installation serves at /guide/ are its own release's, and each bundle carries the same pages as docs/winner-ai-docs-<X.Y.Z>.pdf inside the unpacked folder, so a new bundle's PDF says what changes before you install it.