Upgrade process

This page describes how a new Immich release becomes a new Immich Distribution release. The process is heavily automated: GitHub Actions detect the release, prepare the version bump, and run a full test suite against a real installed instance. A human reviews the result and merges.

This page is for developers

This page does not describe the update process done as a user. That's automatic! 🥳 For update and user focused instructions see the install page.

The upgrade process: upstream release, tracking issue, bump PR with CI, review and merge, channel promotion

Release detection

A scheduled GitHub Action monitors Immich releases. For every new minor release (1.23, not 1.23.4) it opens a tracking issue, tagged new-version. The issue tracks everything related to that release and is a good place for users and developers to ask questions and report findings.

The same automation then creates a pull request on a bump/x.y.z branch that updates the VERSION file and snap/snapcraft.yaml. The PR body contains links to the upstream release notes and diffs of monitored upstream files (Dockerfiles, startup scripts, env and compose files) generated by update.sh. These are the files most likely to require changes on the snap side. Note that update.sh used to carry a lot more logic, but I have slowly moved it out of the script and into more modern solutions like the LLM/AI based review described below.

Only one bump PR is open at a time, and the oldest open tracking issue is processed first. This keeps the PRs sequential and avoids merge conflicts.

CI tests

Every push to a bump/* branch builds the snap package from scratch and installs it on a CI runner, where the test suite runs against the live instance:

  • Fresh install web checks: drives the web UI with a real browser (Playwright) on a newly installed instance. Videos of the browser sessions are recorded.
  • Populated instance web checks: populates the instance with assets first, then verifies the web UI against real data.
  • Asset processing tests: uploads test assets over the API and verifies the results: asset counts, EXIF and location extraction, face detection, and OCR.
  • HAProxy test: queries the HAProxy stats endpoint and fails if any backend is DOWN. HAProxy health-checks the services, so this verifies that everything started correctly.
  • Metrics and certificate tests: verifies the metrics endpoint and the certificate tooling.

Run the tests locally

Install the snap locally, then run the snap tests from the repository root:

make tests

Or call pytest directly with uv run --project tests pytest -s -v --run-snap-tests tests. The tests expect a freshly installed instance on localhost and will create a user foo@example.com with the password secret. Log in with that if you do manual testing afterwards.

If you like to contribute test assets, like more file types with an appropriate license, open an issue! The upstream immich test assets repository is a good resource, some of the tests already use files from it. If the file you have in mind is already there, we can use that instead.

AI-assisted review

A version bump that passes CI is not automatically safe to merge, someone still needs to look at what actually changed upstream. I let an AI agent do the heavy lifting, driven by a verification prompt checked into the repository. It is a strictly read-only verification that fans out subagents to:

  • Read the release notes for every release between the old and new version
  • Sweep all upstream commits between the two tags for changes that may affect the snap: schema changes, dependency bumps, new environment variables, moved paths
  • Compare upstream base image dependencies against the custom built package dependencies (nsg-* packages), which usually need bumping when upstream bumps theirs
  • Check that the patches carried against upstream still apply
  • Verify that paths and version strings referenced by the snap build still exist in the upstream tree
  • Investigate any failed CI jobs

The output is a report with action items: what must happen before merge, whether the era-based upgrade protection should be bumped, and whether a news entry should be written. The known gotchas documented in the repository serve as the review checklist.

Why not in CI?

I intend to run everything I can in CI, but due to the economics some of the AI calls are done on my laptop as part of my subscription to save money. At the moment I am the only one doing this kind of work anyway, so there is no point in me paying API prices. If someone else likes to help me in the future I may consider moving some of the AI checks into the current CI jobs.

Local build

For a simple version bump, passing CI is usually enough. For changes to the packaging itself I build and test locally with snapcraft (I use LXD as the build provider):

  • make: builds and installs a real (unsigned) snap package, equivalent to a real release.
  • snapcraft: use it directly if you prefer control, like rebuilding a specific part or stage.

To tail the logs of all services on an installed instance:

journalctl -fu snap.immich-distribution.*

Release

When the PR is merged, a GitHub Action builds the snap again, runs the tests, signs the package, and publishes it to the edge channel automatically.

From there I promote it manually. My own server runs the beta channel, so the release gets real production use for a few days. If everything works it moves to candidate, and a few days later to stable. The tracking issue stays open for a few more days so users can see, read, and comment.