Nezalنزال
العربية ← Home الرئيسية

Nezal security whitepaper (v0.1 draft for the buyer's security team)

Deployment model

Nezal runs entirely inside the customer's Kubernetes cluster. Three components make it up:

No component calls the internet on its own. The endpoints it talks to are the ones an administrator configures:

The network policies open the internet in both directions by default so those endpoints may live anywhere. The cloud metadata address is excluded from every policy. An air-gapped install sets networkPolicy.internet and environments.internet to false and lists its private networks and the CNI then refuses everything else.

Isolation

Each developer environment is a Kubernetes namespace with a ResourceQuota and a LimitRange derived from the team Policy. Copies of changed services run there. Every other service is an ExternalName alias to the shared baseline. The router sits behind the baseline Service names and routes by the X-Nezal-Env header. At the cluster edge the header is derived only from the Host and any client-supplied routing header is discarded and counted. Unknown environments fall back to the baseline and are reported as Events.

In full mode every environment is a complete copy in the workspace namespace. The copy runs with no service account token and none of the chart's own roles. A NetworkPolicy per environment lets its pods talk to each other and to DNS and to the internet in both directions. It refuses every other environment and the Nezal system and other namespaces and the nodes and the cloud metadata address. Internal networks are reached only when the admin lists them in the Policy's egressCIDRs.

Identity and access

Authentication is delegated to the customer's IdP via OIDC (authorization code with PKCE) or SAML 2.0. Session tokens are EdDSA-signed. Roles are re-read from the store on every request. Three roles are enforced in the API and never only in the UI. A bootstrap token exists for the first run and is ignored once SSO is configured.

One person per identity at a time. Every sign-in is a recorded session that can be listed and ended. A browser sign-in ends the other browser session of the same identity and a nezal login ends the other CLI session. That holds for a password account and an SSO account and the setup token alike. The device that was signed out is told why. The setup token itself is never a bearer: it is traded once for a session that follows the same rule and that stops working the moment SSO is configured. Named API tokens for CI are created one by one under Account and are not limited. An identity that keeps being signed in from other places is reported to the administrator and written to the audit log. See ADR 0021.

Rate limiting. Every endpoint that accepts a secret from someone not signed in carries two budgets at once: ten wrong passwords locks one account and sixty failed attempts in fifteen minutes exhausts one client address across all of them. Counters live in SQLite. Restarting the API therefore cannot clear a lockout and an auditor can see what happened. A forwarded address is honoured only behind a configured trusted proxy — otherwise any caller could invent a new address per request and side-step the budget.

The per-PR webhook endpoint is unauthenticated by design (a forge posts to it with an HMAC signature) and carries its own budget. That budget is charged only on requests that fail to authenticate. A forge posting correctly signed events is never limited however busy the repository. An address guessing webhook secrets runs out. It has its own bucket so that traffic cannot exhaust the budget protecting sign-in.

Policy enforcement

A validating admission webhook rejects an Environment write that does any of the following:

The reconciler re-checks TTL caps and override validity. The webhook's certificate is issued and renewed by the operator. Using cert-manager is optional.

Secret access

The operator holds no cluster-wide Secret permission of any kind. Neither read nor write. Two independent limits enforce that. A mistake in either is not sufficient:

  1. Permission is granted per Secret by name. Connecting a repository makes the API create a namespaced Role that names exactly that Secret in resourceNames and is bound to the operator's service account. Disconnecting revokes it. The full list of what the operator may read is kubectl get rolebindings -A.
  2. The informer cache watches only labelled Secrets (nezal.dev/git-credentials=true). An unrelated application Secret is therefore never fetched and never held in memory even where permission would allow it.

There is deliberately no kubebuilder:rbac marker for Secrets in the operator's source. A unit test reads the generated ClusterRole and fails if one reappears: editing the YAML alone would be undone by the next make manifests.

The operator also cannot write Secrets outside its own namespace. For exactly this reason distributing the preview signing key to application namespaces goes through the API rather than the operator.

Credential values never appear in logs or status or audit entries. Error messages from git operations are scrubbed of credential material by value and by shape.

Audit

Every user action is appended to a hash-chained log in a database that refuses UPDATE and DELETE. The one exception is retention: when the chart sets audit.retention.days the oldest entries are removed once an hour. The chain left verifies from an anchor that holds the hash of the last entry removed and each run is itself written to the log. Recording is off on a fresh install until an administrator turns it on. An upgraded install that already holds entries keeps recording. Turning it on or off is always written with who did it. Evidence packages are signed with a per-installation ed25519 key. Entries can be forwarded to a SIEM. Terminal sessions are audited at open and close with who opened the session and which pod it entered and why it ended. No keystrokes or output are kept and a session cannot be replayed. Secrets never appear in log lines or audit details. Error messages from git operations are scrubbed of credential material.

Where evidence is held. By default everything Nezal keeps is on one volume inside the cluster. That includes the database and the audit log as well as backups. A customer may instead configure their own S3-compatible object storage. Signed audit archives and scheduled backups are then copied there as they are produced.

That endpoint is the customer's own: MinIO or Ceph on-premises or a storage appliance or a sovereign cloud's object storage in your own jurisdiction. It is off until an administrator enters it. Once entered it joins the short list of things the product talks to at all. Nothing about it assumes or reaches a public cloud. The credential lives in a Kubernetes Secret rather than the database because the database is what gets backed up and shipped around. A credential for the place the backups go should not travel inside the backups.

Archives and backups are never deleted by Nezal once uploaded: their lifecycle belongs to the customer and is enforced by bucket rules. Where the bucket has object lock enabled those archives cannot be altered or deleted by anyone — including a Nezal administrator — until their retention expires. That is the write-once control an auditor asks for. The settings screen detects and states whether the bucket has it because Nezal cannot turn it on.

An upload failure never fails the thing that produced the file. The file is written locally first exactly as it would be with no object storage configured. The upload is then retried. nezal_evidence_pending_uploads and the dashboard say plainly when the queue is not draining because evidence that never arrives is evidence the customer does not have.

Licensing

Licences are ed25519-signed payloads verified offline against vendor public keys built into the binaries. There is no call home. Verification therefore works in a sealed room. The development signing key is behind a build tag and is not compiled into release binaries.

One tier and what no key means. An install with no licence key is not refused. It runs free for three developers and one workspace with every feature. That is the administrator and two more people and the limit is the same for invitations and for first sign-ins through SSO. A licence removes the seat limit for any number of developers and workspaces and is issued for one named cluster. Every feature is in both. Licences issued before the single tier may still name a plan in their payload and every plan carries every feature.

Licence state What happens
missing (no key installed) Free: 3 developers and 1 workspace with every feature
valid The licence's seats with every feature
grace Everything continues. The dashboard states the expiry
expired (after grace) Only the left-hand column below is refused. Nothing stops
invalid (bad signature or a development key in a release build) Refused in the API and in the admission webhook

missing and invalid are deliberately different. The first is somebody evaluating the product. The second is somebody who installed something wrong. Quietly treating it as the free tier would hide that.

Where exactly it is enforced. The licence is a commercial mechanism and not a security boundary. It is worth being precise about what it can and cannot stop because a security team will ask.

Checked Not checked
Creating an environment Waking a sleeping environment
Extending an environment's TTL Opening a preview URL
Syncing an environment Reading or exporting the audit log
Creating a per-PR environment from a webhook Signing in and reading the dashboard
Building a service from source Anything already running
The optional features: terminal and siem and saml and per-pr and build and exec-reports Reading a build's status or logs

Everything in the right-hand column is deliberate. A developer opening their own preview must never meet a billing error. An auditor must never be told the evidence is unavailable for commercial reasons. Nothing is ever deleted or stopped because of licence state.

Expiry enters a grace period (30 days or 7 for a trial) in which everything continues to work and the dashboard says so. After grace only the left-hand column is refused. Administrators are warned 30 days before expiry and again at 14 and 7 and 3 days and 1 day before. They are warned on the expiry day itself and weekly through grace and once more at the freeze. Every warning goes out through each of these channels so that an air-gapped site with no mail relay is still told:

Installing a renewed key lifts the freeze on the next request.

Seat counts are soft: exceeding them warns and a 10% burst is tolerated. Beyond that new invitations are refused while everyone already working continues. Enforcement runs in the API. That is also where the audit record is written and so a licence refusal is auditable. The operator does not enforce licensing. An environment already reconciling is never interrupted by it.

Building from source

This is the one thing in Nezal that deliberately executes code the customer's developers wrote on the customer's cluster. It is worth being exact about what that costs and what contains it.

What runs. A Dockerfile from the repository together with whatever it RUNs and every dependency it fetches. All of it is treated as untrusted input.

Where. One namespace named nezal-build which holds builder Jobs and nothing else. It is the only namespace Nezal runs at the privileged Pod Security Standard. Rootless BuildKit maps its own user namespace. That needs an unconfined seccomp profile and the "baseline" standard forbids it outright. That is the price of building container images inside a cluster without a Docker daemon and it is not hidden in a values file.

What contains it:

Control Effect
No service account token (automountServiceAccountToken: false) A build cannot talk to the Kubernetes API at all.
Runs as uid 1000 and never root Nothing in the build is root either inside the container or on the node.
All capabilities dropped except SETUID and SETGID The two rootless BuildKit needs to map a user namespace. Both apply within the container's own namespace.
Neither a Docker socket nor a host path nor a privileged container A mounted Docker socket is root on the node. There is none.
NetworkPolicy allow-list A build reaches DNS and the git server and the registry and the team's declared dependency mirrors. Nothing else. A fetch from the public internet fails rather than quietly succeeding.
Link-local carved out of every allowed range Cloud instance metadata at 169.254.169.254 hands credentials to anything that can reach it.
activeDeadlineSeconds and CPU/memory limits set per team A build cannot hang forever or take the cluster with a make -j.
backoffLimit: 0 A failed build is reported and looked at. It is never silently retried under a new pod whose predecessor's logs are gone.
Separate checkout container The git credential is used by an init container and is never visible to the build container where repository code runs.
Build arguments screened at admission They are recorded in the image history and readable by anyone who can pull it. Anything named like a credential is therefore refused.
Context and Dockerfile paths screened at admission A path climbing out of the checkout would read the Job's own filesystem.

What it produces. An image pushed to the customer's own registry and tagged by commit and deployed by digest: a tag can be moved after the fact and a digest cannot. Every build is audited with the commit and the digest. A build is always of one resolved commit and never of a branch. Two builds of "main" therefore cannot produce two different images under one name.

Switching it off. build.enabled: false in the chart removes the namespace entirely. The build licence feature gates it commercially. A team's Policy can disable it independently: a team may be allowed personal environments without being allowed to execute arbitrary build steps.

No built-in image scanning. Nezal does not scan images and depends on no scanner. A built image lands in the customer's registry and the customer's own scanner can check it there.

Preview images are still not signed. The release pipeline signs what ships. A preview image is built from a developer's branch and used by one environment and deleted with it. A signature would attest to something nobody promotes. A customer whose policy requires every image in the cluster to be signed should build into their own registry and pipeline instead — registry per workspace exists for exactly that.

Supply chain

All binaries are static Go builds on distroless images. They run as a non-root user with a read-only root filesystem and all capabilities dropped. The third-party dependencies are:

Each release does three things:

Signing by digest rather than by tag matters: a tag can be moved to a different image after signing and a signature over the name would still verify.

The air-gap bundle carries those SBOMs and signatures alongside the images. A customer with no route to the internet can therefore still verify what they are installing.

Known gaps

Stated plainly because a whitepaper that lists none is not believed.

Closed since v0.1: