Administrator guide

First run

1. Sign in with the bootstrap token (api.bootstrap.* in the chart). Once recording is on everything you do is audited under the actor bootstrap-admin. The token stops working once SSO is configured and the API restarts. That is the intended end of its life. It is for the first sign-in only. The dashboard and nezal login --token trade it once for a session, and a second sign-in with it ends the first like any other account. Create your own account next and use that.

2. Turn on audit recording under Audit log > Recording and retention. A fresh install records nothing until an administrator turns it on. Turning it on is itself the first entry and names who did it. Do it now on any install a regulator reads so that every later step is in the log. See Recording on or off.

3. Licence: paste the key from Nezal. It is verified locally with no call to us and so it works air-gapped. State valid shows the customer and the seat count as well as the expiry and the features. After expiry there are 30 days of grace (7 for a trial) during which everything still works and the dashboard says so. After grace no new environments or users are admitted. Nothing is ever deleted because of the licence and nothing existing stops running.

4. SMTP before you invite anyone. Under Security & email enter the host and port and the security mode (starttls or tls or none) and then the credentials and the From address. Send test proves it before an invitation depends on it. The password is write-only: it comes back redacted and the dashboard sends the redaction back untouched. Saving the form therefore cannot overwrite a working password with an empty one.

5. SSO or local accounts or both. OIDC (issuer and client id and secret and groups claim) or SAML (IdP metadata XML and SP key pair). Map IdP groups to the admin role and to workspaces as developer or team lead.

6. MFA policy at the top of Security & email. With enforced on every local account must enrol a TOTP authenticator before it can do anything. Any free authenticator app works and nothing outside the cluster is contacted. SSO accounts are unaffected because their second factor belongs to your identity provider.

7. Workspaces: one per team. A workspace maps to an application namespace. The Baselines and Environments and Policy of that team live there.

8. Policy per workspace. It sets:

   • environments per user and per team
   • CPU and memory quota
   • default and maximum TTL
   • idle sleep and the grace period after expiry
   • terminal on or off with the idle and maximum session length

   The operator's admission webhook enforces it on every write. It therefore holds for kubectl too and not only for the dashboard.

9. A registry per workspace under Registries. It is required: an application cannot be created in a workspace until one is connected. Connect it once with its credential. The first one you connect is the default.

   An application uses the default unless you choose another connected registry for it under Registry in its dialog. The dialog asks for the repository first and then the registry with the default already chosen. A registry an application chose cannot be disconnected until it chooses another.

   The default registry does two things. Builds push their images there because a node cannot run an image that exists only on another node's disk. And its credential is what the workspace's pods pull with.

   It does not rewrite image names. An image runs from exactly the registry your chart or manifests name. On an air-gapped cluster point those names at your own registry. The chart's image.repository values are the usual place.

   A registry that needs a password works for developers' copies too. Nezal writes the credential into a Kubernetes Secret and the operator copies it into each environment namespace, because a pod may only reference a Secret in its own namespace. The operator can read a Secret it was told about by name and cannot list them; see ADR 0012 for what that means for a security review.

10. Trusted proxies if the API sits behind an ingress controller or load balancer. Set api.trustedProxies to that proxy's pod network. Until you do X-Forwarded-For is ignored entirely and the audit log records the proxy's address rather than the user's. That default is deliberate: believing the header unconditionally would let any caller write an arbitrary address into the tamper-evident record an investigator relies on.

11. Check the install answered: helm test nezal -n nezal-system.

People: local accounts with invitations and second factors

Users come from your identity provider or are invited directly or both.

Inviting: Users & teams > Invite. Give an email and a workspace and a role and say whether they are an admin. The workspace is required and starts as the one you are in. An invited developer sees that workspace and its applications as soon as they sign in. The Users list shows each person's workspaces and roles. They receive a link to set a password. An invited account holds a seat from the moment it is created and not from first sign-in. Otherwise an administrator could invite an unlimited number of people and have them all working before the seat count noticed.

Second factor: TOTP for local accounts. Any free authenticator app works such as Google Authenticator or Microsoft Authenticator or FreeOTP. Nothing outside the cluster is contacted and no paid service is involved. Enrolment shows ten single-use recovery codes once. Each one signs in once in place of a code from the phone. With MFA enforced a new local account is walked through enrolment on first sign-in. Reset MFA on a user clears their enrolment and makes them set it up again at their next sign-in even when MFA is not enforced. It is the answer to a lost phone and it is audited. SSO accounts and the setup token have no password in Nezal and so no second factor here. Their Account page says so.

Rate limiting is on and needs no configuration: ten failed sign-ins for one account within fifteen minutes locks that account for fifteen minutes and sixty failures from one address in the same window blocks the address. An account that has been invited but has not yet set a password is rate-limited too.

Sessions and tokens: each person can see their own active sessions under Account with the device and the last time each was used. They can revoke any of them. "Sign out everywhere" spares the session doing the asking so nobody signs themselves out by accident. An administrator can revoke every session of another user. That takes effect on the user's next request rather than whenever their token happens to expire. CLI tokens are minted from the dashboard and appear in the same list.

Removing someone: deleting a user revokes their sessions and tokens and frees the seat. A user who still owns environments is refused with the count rather than deleted: removing them silently would leave environments nobody can reach and quota nobody is accountable for. Delete or hand over their environments first and then delete the account.

Roles

Terminal sessions and logs are always owner-only and never on the baseline. Nobody else reads an environment's logs, a team lead and an admin included, because a log carries whatever the application printed. An admin or the Policy can switch terminals off per team.

Terminal limits

A web shell is the highest-privilege thing a developer can do here and the only action that would otherwise leave no detailed trace: every other action is an audit entry naming what was done while a shell is one entry saying it was opened. That is why every session is written to the audit log with who opened it and which pod it entered and when and why it ended. The Policy bounds the session itself:

A session closed by a limit says so: the sessions list and the audit entry both carry the reason (idle or max or cap or user) because "the shell exited" and "we closed it after an hour idle" are different facts to whoever reads the log later.

A session's output is also capped. Reaching the cap ends the session and a runaway process reaches it in seconds. That is the case the cap exists for and the session ends with reason cap so the log says which limit closed it.

Repositories and applications

Connect the repository first under Repositories: its URL and the default ref and how to authenticate (an HTTPS token or an SSH key or none for a public mirror). Test clones it before anything depends on it. Credentials go into a Kubernetes Secret in the application namespace labelled nezal.dev/git-credentials=true and are never readable back through the API. The operator is granted access to that one Secret by name and holds no cluster-wide Secret permission at all.

Optionally set a webhook secret on the repository. That is what GitHub's X-Hub-Signature-256 or GitLab's X-Gitlab-Token is checked against on POST /webhooks/{workspace}/{repository}. That is how a pull request gets its own environment. Like the git credential it is write-only.

Then connect the application by pointing at that repository and a path (a Helm chart or Kustomize or plain manifests). The operator renders it and lists the services and reports in status.warnings everything it could not carry over. Read those warnings on the application page: they are where "this service has no health check" is said out loud. Mark databases and queues as shared so environments never copy them.

Promoting a release means changing the source ref. Every promotion is recorded in the application's release history with the commit and the time and who did it. Rolling back is promoting an older ref from that list. Environments keep sharing the baseline. Only their overridden services are their own.

Per-PR environments (licensed feature per-pr): set prEnvironments to all or to label with a label name. A pull request opening then creates pr-<number>-<application> and a push syncs it and closing the pull request deletes it. They are owned by webhook and not by a person so they do not consume anyone's personal environment quota.

Posting the preview URL back. Tick Post the preview URL to the pull request on the repository and Nezal sets a commit status (nezal/preview) and writes one comment. It edits that same comment on every push rather than repeating it — a dozen identical bot comments is why people mute these integrations.

Off by default because writing to your pull requests is not something to assume. It needs a token that can write to the repository: a read-only deploy key connects and clones perfectly well but cannot comment and Nezal then skips posting quietly rather than filling the log. GitHub (including Enterprise) and GitLab are recognised from the repository URL. Any other git server has no pull requests to post to and nothing is attempted.

Each post is in the audit log as forge.post. Nothing new is contacted: this is the same git server the repository already points at.

Preview URLs

Every environment publishes <environment>--<application>.<domain>. With previewAuth.enabled (the default) a visitor must have signed in to Nezal and have read access to that environment. The cookie is scoped to one environment and lasts eight hours.

Leave it on. With it off a preview host is reachable by anyone who can reach the ingress. On a company's internal network that is every employee and every contractor's laptop and every other workload in the cluster. A developer testing against production-shaped data would be publishing it to all of them without being told.

Audit and evidence

Every action of every user is one entry with a sha256 chained to the previous entry. Audit > Verify walks the chain. Export downloads a package of JSON lines plus a manifest signed with the installation's key (public key at /api/v1/audit/public-key). Forward entries live to your SIEM over syslog (TCP or UDP per RFC 5424) or HTTP JSON under Settings > SIEM. Terminal sessions are listed under Terminal sessions with who opened each one and which pod it entered and why it ended.

Recording on or off

Recording is off on a fresh install. Nothing is written to the audit log until an administrator turns it on under Audit log > Recording and retention. The Audit log page and the admin Overview both say so while it is off. An upgraded install that already holds entries keeps recording so an upgrade never stops a log that was running. Turning it on and turning it off are always written with who did it and from where. Every stretch in which the log was not recording is therefore visible to an auditor. Leave it on wherever a regulator reads the log.

Retention

The audit log lives in the API database on the data volume. To stop it filling the volume set a number of days in the Helm values. Entries older than that are removed once an hour.

helm upgrade nezal oci://registry-1.docker.io/nezal/nezal -n nezal-system --reuse-values \
  --set audit.retention.days=365

Removing old entries does not break verification:

• The chain still verifies. Nezal keeps the sequence number and hash of the last entry it removed as an anchor. Audit > Verify checks the entries left from that anchor.
• Every run is written. Each run adds an audit.prune entry with the range it removed and the anchor hash.
• Nothing else can delete. The database still refuses UPDATE on every entry. It refuses DELETE on every entry except inside a retention run.

Freed space is reused by new entries. The database file does not shrink on disk.

Archiving

Archiving exports everything older than a cut into a signed evidence package that you keep wherever you keep evidence. It also records that the range is accounted for. Archive before retention removes a range if you have to keep it longer than audit.retention.days.

curl -X POST -H "Authorization: Bearer $TOKEN" \
  "https://nezal.acme.internal/api/v1/audit/archive?olderThanDays=365&location=vault://audit/2025" \
  -o nezal-audit-archive.multipart

olderThanDays defaults to 365. location is recorded exactly as you give it and is never read back. An archive Nezal could reach is an archive Nezal could alter. Archiving a range that holds nothing is not an error and records nothing. A scheduled call can therefore run on a quiet week.

GET /api/v1/audit/archives lists what has been archived. It shows the range each archive covers with its head hash and the head hash of the archive before it.

Each archive carries a manifest with a sha256 of the entries and an ed25519 signature over the manifest. An auditor can check both years later without this installation being available. The manifest carries the public key it was signed with.

Lifecycle

Environments expire at their TTL: copies scale to zero and the preview URL shows an "expired" page. After the grace period the environment is deleted. Idle environments sleep. The first request wakes them. Each transition is a Kubernetes Event and an audit entry.

Licence and seats

Licence shows the customer and the term and the features. It also shows the seat count against active users. A seat is someone who has signed in or holds a pending invitation.

A seat is one person. An account can be used by one person at a time. A browser sign-in ends the other browser session of the same account and a nezal login ends the other CLI session. The person who was signed out is told why. An account that keeps being signed in from other places is marked Shared account under Users and listed on Overview and Licence. It is written to the audit log as auth.sharedAccount. Give each person their own account. Single sign-on does that on its own because every person has their own identity. Where someone is signed in is under Sessions on their row under Users and on their own Account page.

Seats are soft. Passing the count raises a warning and a 10% burst is tolerated outright. Beyond that new invitations are refused while everyone already working carries on. A company does not want its developers stopped mid-sprint by a counter. Over-deployment is a renewal conversation and not an outage.

Usage report downloads a signed report of monthly active seats over the term. That is what a renewal is negotiated against. It needs no network and reveals nothing outside the cluster.

Being warned before it expires

Nezal warns about expiry through four channels at once because in a regulated site at least one of them is usually unavailable. It starts 30 days ahead. It repeats at 14 days and again at 7 days and 3 days and 1 day before expiry and once more on the day itself. It then warns weekly through the grace period and once when the grace period ends. A trial warns at 7 and 3 and 1 days instead — its whole term is shorter than a paid licence's first notice.

Each notice is sent once. Restarting the API does not resend one and installing a renewed key starts the sequence again for the new term.

The email recipients are every enabled administrator plus whatever is set at PUT /settings/licence-notify. That list exists because the people who approve a renewal are usually in procurement and rarely hold an admin account. A disabled administrator is not notified.

If there is no SMTP relay the other four channels still work and the audit log records licence.notice with smtp: false — so "we were never told" has an answer either way.

Nothing about any of this leaves the cluster. The mail goes through the customer's own relay. Nezal does not contact its vendor at any point.

Monitoring

The API and the operator both serve Prometheus metrics on /metrics. They are unauthenticated on the pod network and not exposed through the ingress. Nothing is sent anywhere: your Prometheus scrapes them the same as anything else in the cluster.

No metric carries a user or a customer or an environment or a workspace name. Labels end up in dashboards and in long-term storage so a label is a place data goes and stays — and one series per environment would be unbounded cardinality besides.

Alert rules worth having

These are the five that matter. Everything else is a dashboard.

groups:
  - name: nezal
    rules:
      # The licence stops new work after its grace period. Thirty days is enough for a
      # purchase order to go through; the product also emails, but not every site has SMTP.
      - alert: NezalLicenceExpiringSoon
        expr: nezal_licence_seconds_until_expiry < 30 * 86400
        for: 1h
        annotations: {summary: "Nezal licence expires in under 30 days"}

      # The volume holds the database and the keys. When it fills, the database
      # becomes read-only and the audit log stops with it.
      - alert: NezalDataVolumeLow
        expr: nezal_data_volume_free_bytes / nezal_data_volume_total_bytes < 0.15
        for: 15m
        annotations: {summary: "Nezal data volume below 15% free"}

      - alert: NezalAPIErrorRate
        expr: |
          sum(rate(nezal_api_requests_total{status=~"5.."}[5m]))
            / sum(rate(nezal_api_requests_total[5m])) > 0.05
        for: 10m
        annotations: {summary: "Over 5% of Nezal API requests are failing"}

      # A stalled audit head is either a very quiet install or a broken one, and you need to
      # know which. Pair it with the error-rate alert: quiet plus errors is not quiet.
      - alert: NezalAuditLogStalled
        expr: increase(nezal_audit_head_seq[6h]) == 0
        for: 30m
        annotations: {summary: "No Nezal audit entries in six hours"}

      # Evidence that never reaches your object storage is evidence you do not have.
      - alert: NezalEvidenceUploadsPending
        expr: nezal_evidence_pending_uploads > 0
        for: 1h
        annotations: {summary: "Nezal evidence files are not reaching object storage"}

The operator publishes its own series (nezal_environments_by_phase and render counts and durations). A disagreement between the operator's view and the API's is itself a signal: they read the same objects by different paths.

Getting support

Nezal sends no telemetry so there is nothing for us to look at when something goes wrong. The support bundle is the substitute: one file with what a support engineer would otherwise spend a day asking for.

Admin → Installation → Download support bundle or from the command line when the dashboard itself is the problem:

kubectl exec -n nezal-system deploy/nezal-api -- \
  /nezal-api support-bundle > nezal-support.json

It holds:

• the component versions
• the licence state and plan and seat counts (never the key)
• every setting with credentials redacted
• the last known result of each git and registry and SMTP and evidence-store check
• pod status and recent warning Events in the release namespace
• the audit head and whether the chain verifies
• the last 2000 API log lines

Nothing is sent anywhere by producing it. The first line of the file says so. Read it and then attach it to your support request yourself — that is the whole point of the format being one readable JSON file rather than an archive.

Credentials and tokens and keys and Authorization headers are redacted wherever they appear. That includes log lines. They are redacted as they are recorded rather than as the bundle is written. A test plants realistic credentials through the API and fails the build if any of them reaches a bundle. Even so: read the file before you send it. It describes your installation in detail and only you can judge whether that detail is something you are happy to share.

The command-line version reads the database directly so it works when the API is not serving. It cannot include the running process's recent logs. The dashboard download can.

Backup and restore

The API's PVC holds the SQLite database and the session and audit-export keys. It carries helm.sh/resource-policy: keep so helm uninstall leaves it alone. Removing it is a deliberate kubectl delete pvc.

Back up with the command and not a file copy. Copying the database file while it is open can produce an unusable backup:

kubectl exec -n nezal-system deploy/nezal-api -- \
  /nezal-api backup --out - > "nezal-$(date +%F).db.enc"

The backup is encrypted. It holds every password hash and every stored second-factor secret and the whole audit log. Once it reaches object storage its custodian is not necessarily the person administering Nezal. AES-256-GCM with a 32-byte key in the nezal-api-keys Secret alongside the signing keys.

Lose that Secret and every backup becomes unreadable. There is no recovery path and there is deliberately no escrow. Back the Secret up separately and before you need it. Keep it somewhere your database backup tooling does not reach.

--plaintext writes an unencrypted backup for a customer whose own key management requires it. The command says plainly what that file contains.

--out - streams the backup to stdout. That is the retrieval path and not a convenience: kubectl cp shells out to tar inside the container and the API image is distroless — no shell and no tar — so kubectl cp cannot work against it at all. Everything the command says about itself goes to stderr so the redirected file holds nothing but the database.

Check it before trusting it. verify-backup opens the file with SQLite so it needs a real file rather than a pipe — run it wherever the backup now lives:

nezal-api verify-backup --in "nezal-$(date +%F).db.enc" --key-file ./backup.key

Inside the cluster the key is found automatically on the data volume so kubectl exec -n nezal-system deploy/nezal-api -- /nezal-api verify-backup --in … needs no --key-file. verify-backup names the key each file was made with so an unlabelled backup found later can be matched to the right one.

The API image is distroless — /nezal-api is the only executable in it — so the check runs outside the cluster on the copy you have just taken. The scheduled CronJob verifies each backup it writes. That is the path to prefer.

A backup nobody has opened is a hope rather than a backup.

backup uses SQLite's VACUUM INTO and that is safe while the database is being written to. The chart ships a CronJob that does all of this on a schedule and writes to its own claim. See backup in values.yaml. That is the path to prefer — the manual commands above are for a backup taken before a deliberate change.

Restoring is deliberately a documented sequence rather than a subcommand because a restore command that can be run against a live database is a way to destroy one:

1. Scale the API to zero: kubectl scale -n nezal-system deploy/nezal-api --replicas=0.

2. Copy the backup over nezal.db in the data directory from a pod mounting the same claim. With the API scaled to zero nothing else holds the claim so a short-lived pod mounting nezal-api-data is enough:

   kubectl cp ./nezal-2026-09-11.db nezal-system/<restore-pod>:/var/lib/nezal/nezal.db
   

   Remove any nezal.db-wal and nezal.db-shm beside it first: they belong to the database you are replacing and SQLite will otherwise try to replay them over the restored file.

3. Scale back to one and check the audit chain verifies in the dashboard.

Baselines and Environments and Policies are Kubernetes objects. Back them up with whatever your cluster already uses. They are also rebuilt from the repository so the database is the part that matters.