Disaster recovery: rebuilding Nezal from nothing

This is the sequence to rehearse and the one to follow when the cluster is gone. It assumes the worst realistic case: the cluster no longer exists and all you have is a backup file and your git server.

It is deliberately a runbook rather than a command. A restore that can be run against a live installation is a way to destroy one. A disaster recovery plan a security team can sign off is a sequence its own people can rehearse rather than a button.

Nezal's own release checks perform exactly these steps on a fresh cluster before every release so the procedure cannot rot.

What you need before you start

The keys are the thing to check now rather than later. They live in the nezal-api-keys Secret and they are deliberately not in the backup: a stolen volume snapshot should not yield both the data and the key that decrypts it. The backup command says so every time it runs. Back that Secret up somewhere your database backup tooling does not reach.

The Secret holds three keys:

• backup decrypts the backups.
• session signs sign-in tokens and encrypts stored second-factor secrets.
• audit-export signs evidence packages.

Losing session costs everyone a re-enrolment. Losing backup costs you the backups and nothing can be done about it afterwards.

The sequence

1. Install onto the new cluster

Follow the install guide as for a first install with one difference: do not enter the licence key or configure anything yet. You are creating an empty installation to restore into.

helm install nezal oci://registry.nezal.dev/charts/nezal --version <the version you were running> \
  --namespace nezal-system --create-namespace \
  --set ingress.domain=dev.acme.internal \
  --set api.persistence.enabled=true

Install the same version you were running. A backup restores into its own schema. A newer version would migrate it forward on first start. That is a second change happening during an incident and one change at a time is the rule.

2. Put the keys back before the database

The API's signing keys live in the Secret nezal-api-keys and not in the backup. Restore that Secret from wherever you keep it:

kubectl apply -f ./nezal-api-keys.yaml       # the Secret as you backed it up
kubectl rollout restart deploy/nezal-api -n nezal-system

Without this Secret the backup in step 3 cannot be decrypted at all. That is why it comes first.

Keys first because the API mints new ones on first start if it finds none. A restored database whose stored TOTP secrets were encrypted under the old session key would then be undecryptable and every user with a second factor would be locked out.

If the keys are gone: the database still restores and the product works. Everyone signs in again. Users with a second factor must re-enrol (Users → Reset second factor). Evidence packages exported before the loss can no longer be verified against this installation's public key. They remain verifiable against the old public key inside those packages so keep the packages themselves.

3. Restore the database

Follow Backup and restore in the admin guide from step 1. In short: scale the API to zero and decrypt the backup. Copy it over nezal.db in the data volume from a pod mounting the same claim and scale back to one.

Decrypt it with the backup key from the Secret you restored in step 2:

kubectl -n nezal-system get secret nezal-api-keys -o jsonpath='{.data.backup}' | base64 -d > ./backup.key
nezal-api verify-backup  --in ./nezal-2026-09-12.db.enc --key-file ./backup.key
nezal-api decrypt-backup --in ./nezal-2026-09-12.db.enc --out ./nezal.db --key-file ./backup.key

verify-backup proves the key is right and the file opens before you overwrite anything. decrypt-backup then writes the plain database to copy into place. Remove that plain copy and backup.key once the restore is done.

4. Check what came back before going further

kubectl -n nezal-system port-forward svc/nezal-api 8080:8080 &
curl -s -H "Authorization: Bearer $TOKEN" localhost:8080/api/v1/audit/verify | jq

intact: true and a headSeq matching what you had is the signal that this is the same installation and not a new one wearing its name. That database holds the users and workspaces together with the repository definitions and settings and the licence. All of them should be present. Applications and policies are not — they are Kubernetes objects and step 6 deals with them.

5. Re-enter repository credentials

Credentials are in Kubernetes Secrets and not in the database so they did not come back with it. Under Repositories re-enter the credential for each repository. Everything else about the repository — its URL and ref and the application built from it — is restored already.

6. Restore the Kubernetes objects

Applications (Baselines) and Environments and Policies live in the cluster and not in the database. They are Custom Resources and they went with the cluster. This is the step most likely to go wrong and the one to rehearse.

If you have a CRD dump — from Velero or an etcd snapshot or a scheduled kubectl get baselines,policies -A -o yaml — restore Baselines and Policies from it:

kubectl apply -f ./nezal-baselines.yaml
kubectl apply -f ./nezal-policies.yaml

Take that dump on a schedule. It is two objects per team and it is the difference between a ten-minute recovery and re-entering every team's quotas and TTL limits and terminal settings by hand:

kubectl get baselines,policies -A -o yaml > nezal-crds-$(date +%F).yaml

If you do not have one then re-create them through the dashboard. Connect each application again under Applications (its repository and credentials are restored so this is filling in the same form). Set each team's Policy again under Quotas & cost. Write down what your policies were before you need this.

Environments are not worth restoring: a personal environment is a copy of a branch and a developer re-creates theirs with nezal up in a minute. Per-PR environments reappear on the next webhook event from an open pull request.

7. Confirm end to end

Create one environment and open its preview URL. Check that the audit log shows the whole sequence. The installation is recovered when a developer can do their ordinary work.

What you get back and what you do not

Terminal history survives. A shell that was open when the cluster went is gone with it but the audit entries for every session are in the database and come back with it. Who opened each session and which pod it entered and when and why it ended is all still there for an investigator.

If you have no backup

The audit log is gone and nothing recreates it: that is the point of a hash-chained log. What you can rebuild is the working product. Install fresh and enter the licence. Re-create workspaces and re-connect repositories. Users re-appear as they sign in through your identity provider.

Say so plainly in your incident report rather than quietly starting a new log. An auditor who finds a gap they were not told about draws a worse conclusion than one who reads that the cluster was lost on a known date and the log restarts there.

Rehearsing it

Rehearse it by hand once a year against a restored copy of your own backup. The parts that go wrong in practice are always the same three:

• the nezal-api-keys Secret nobody had backed up
• repository credentials nobody had written down
• a chart version that no longer matched the backup

All three are cheap to check today and expensive to discover during an incident.