helm upgrade --wait can return success while the migration is still running or has failed.
Check the Job explicitly.
Prerequisites
- A verified backup — see back up and restore
- The version you are upgrading to, and the one you are on
- For Kubernetes:
helmandkubectlagainst the target namespace
Steps
Choose a tag to pin
Images are published to Docker Hub as
Chart version and application version are separate:
agentarea/agentarea-<component> for components agentarea-api,
agentarea-worker, agentarea-frontend, agentarea-mcp-manager,
agentarea-events, and agentarea-mcp-runner.The release workflow does not rebuild. It retags the existing
X.Y.Z-<short-sha> artifact that CI already pushed, so the bytes you test on a
release tag are the bytes CI built from main.Pin X.Y.Z for any deployment you care about. latest moves under you on
someone else’s schedule, and a merge to main alone does not move it — only a
tag push does, which makes latest both mutable and unpredictable in timing.The chart’s values.yaml defaults every image tag to latest. Override them:Chart.yaml carries
version (the chart) and appVersion (the platform). The release workflow bumps
Chart.yaml and opens a pull request; do not bump chart versions by hand.Back up before touching anything
Migrations are not reversible in practice. Alembic can generate a
downgrade,
but nothing in this repository tests one, and a downgrade that drops a column
destroys the data in it.Capture the database, the object store, and the encryption key. See
back up and restore.Read what changed
Check the GitHub release notes for the target version. Look specifically for
new required configuration — a variable added to
charts/agentarea/config.yaml
between your version and the target becomes a hard startup failure if the
platform validates it, which is the intended behaviour rather than a silent
default.Diff the chart values if you are jumping more than a patch release:Upgrade on Kubernetes
db-migration
Job, and the registryReconcile Job, then rolls the Deployments.Generated secrets are preserved. The chart looks each one up and reuses the
existing value, so the database password and the encryption key do not rotate
under a running deployment.--wait waits on the Deployments. It does not wait on the migration Job in a
way you should trust — verify it separately, in step 6.Upgrade on Docker Compose
VERSION in .env instead of on the command line to keep it consistent
across invocations — every image reference in docker-compose.yaml is
${VERSION:-latest}.Pull the repository as well as the images. The Compose file bind-mounts
./config/auth/kratos and ./agentarea-platform/temporal-config, so image and
configuration have to move together.app_migrations re-runs automatically, and app waits for it to complete
successfully before starting.Confirm the migration, not only the rollout
This is the step that distinguishes a completed upgrade from a running one.The Job must read
1/1, and the log must end in Migrations completed successfully. ttlSecondsAfterFinished: 300 removes the Job five minutes after
success, so a missing Job with healthy pods is fine; a Job at 0/1 is not.Roll back if it failed
Rolling back images is straightforward. Rolling back the schema is not.This reverts the chart, including image tags. It does not revert the
database — the new schema stays. If the new schema is backward-compatible with
the old code, the rollback works. If it is not, restore the database backup you
took in step 2, and accept the loss of everything written since.Under Compose, set
VERSION back and up -d again, with the same caveat.Verify
Confirm the running images are the ones you pinned:Troubleshooting
helm upgrade reports success and the platform behaves like the old version
helm upgrade reports success and the platform behaves like the old version
The Deployments rolled but the migration Job failed, or a component’s image
tag was not updated. Check every tag in
values.yaml — backend , worker
, frontend , mcpManager , and eventService are set independently, and
moving four of the five is a common slip.The migration Job fails with relation already exists
The migration Job fails with relation already exists
A partially applied migration from an interrupted upgrade. See
database and migrations for the stamp
procedure, and take a backup before stamping.
The API crash-loops after upgrade on a missing environment variable
The API crash-loops after upgrade on a missing environment variable
The new version requires configuration the old one did not. The platform
fails loudly rather than defaulting. Add the value; do not work around it by
reverting only the API.
Tasks fail after upgrade with workflow errors, while the API is fine
Tasks fail after upgrade with workflow errors, while the API is fine
The worker is running an older image against a newer schema, or Temporal is
replaying history with changed workflow code. Confirm the worker image
matches, and check the Temporal UI for the failing workflow’s history.
helm upgrade rotated the database password
helm upgrade rotated the database password
The Secret was deleted between operations, so
lookup found nothing and
generated a new value while the volume kept the old one. Restore the Secret
from backup. The helm.sh/resource-policy: keep annotation protects against
helm uninstall , not against kubectl delete secret .Compose comes up with the old images
Compose comes up with the old images
VERSION was not exported to the pull , or the pull was skipped. Every
image is ${VERSION:-latest} ; without it you get latest .Related
Back up and restore
Identify everything AgentArea stores, back each store up
Run database migrations
Apply, inspect, and recover Alembic migrations for the AgentArea schema, on
Kubernetes and on
Deploy on Kubernetes with Helm
Install the agentarea Helm chart, decide which bundled dependencies to keep
Troubleshoot a self-hosted deployment
Diagnose a deployment that will not start