---
title: Kubernetes
---

`deploy/helm/loom` runs Loom as [N identical replicas](/docs/zh-CN/operations/replicas) on Kubernetes: one
Deployment of web pods over one ClickHouse service, each pod a full replica. It works on
any conformant cluster; `values-eks.yaml` sets it up on EKS, `values-kind.yaml` runs it on
a laptop, and `values-clickhouse-cloud.yaml`, over the EKS one, adds ClickHouse Managed
Postgres as the memory authority ([ClickHouse Cloud and Managed
Postgres](/docs/zh-CN/operations/clickhouse-cloud)).

| Object                                | What it does                                                                                                                                                  |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Deployment `<release>`                | The web pods. Rolls one pod at a time and takes none down before its replacement is ready.                                                                    |
| Service, optional Ingress             | Routes to ready pods only. The readiness probe is `/v1/health/ready`.                                                                                         |
| Secret                                | The keys the Loom pods share (the extractor's read only its token). Bring your own with `secrets.existingSecret`. The rest of the env is in the pod template. |
| ConfigMap `<release>-settings`        | The settings file (`settingsFile`), mounted at `/app/config/loom.yaml` on every pod; each hook Job carries its own copy.                                      |
| PodDisruptionBudget                   | One pod at a time on node drains and upgrades.                                                                                                                |
| Optional HPA                          | CPU-based. Rate-limit shares resize as pods come and go.                                                                                                      |
| Optional `<release>-workers`          | Background workers in their own pods (`workers.enabled`). Off by default: one web pod holds the lease and runs them.                                          |
| Optional `<release>-publish`          | A pre-upgrade Job for a change that cannot roll ([below](#changing-what-cannot-roll)).                                                                        |
| Optional `<release>-analytics-export` | A daily CronJob and its own ServiceAccount (`analyticsExport.enabled`, [below](#analytics-export)).                                                           |
| Optional `<release>-extractor`        | The document extractor's Deployment, Service and NetworkPolicy (`extractor.enabled`, [below](#document-extractor)).                                           |
| Optional `<release>-librechat`        | The assistant: LibreChat's Deployment, Service, ConfigMap and NetworkPolicy, and its MongoDB (`-librechat-mongo`) (`assistant.enabled`, [below](#assistant)). |
| NetworkPolicy `<release>`             | Full policy with `networkPolicy.enabled`; else all egress but [`egressExcept`](#instance-metadata-service), unless it is `[]` or `guardMetadata: false`.      |
| Optional `<release>-code-tiers`       | A PrometheusRule on the failing code-index tiers (`alerts.prometheusRule.enabled`, [below](#alert-rules)).                                                    |
| Optional dev ClickHouse               | `devClickhouse.enabled`: one ClickHouse pod with embedded Keeper. For test clusters only.                                                                     |

## Before you install

- **ClickHouse.** One service for every pod: ClickHouse Cloud, or a self-managed server
  with the `deploy/clickhouse-config/` files mounted and Keeper
  ([requirements](/docs/zh-CN/operations/deployment#clickhouse-requirements)). Create the database:
  `CREATE DATABASE loom`. On a self-managed replicated cluster, point every pod at one
  replica.
- **A root secret.** `LOOM_JWT_SECRET`, at least 32 random bytes
  (`openssl rand -hex 32`). Every pod shares it, and it seals the signing keys and stored
  credentials in ClickHouse, so keep a copy outside the cluster.
- **A public URL.** `publicBaseUrl`, what clients reach Loom on. Pods refuse to start
  without it.

## Install

```bash
helm install loom deploy/helm/loom -n loom --create-namespace \
  --set publicBaseUrl=https://loom.example.com \
  --set clickhouse.host=abc123.us-east-1.aws.clickhouse.cloud \
  --set secrets.jwtSecret=$(openssl rand -hex 32) \
  --set secrets.clickhousePassword=... \
  --set secrets.extra.OPENAI_API_KEY=...
kubectl -n loom rollout status deploy/loom
```

The first pod runs the migrations under the migrate lease and the others wait for it,
so the first install takes a few minutes. The startup probe allows 15 minutes
(`probes.startup`), which also covers cloning the git mirrors.

The embedding settings go through `embedding.*`, and the root secret and its previous
value through `secrets.jwtSecret` and `secrets.jwtSecretPrevious`; the chart guards these
(below). The root secret can instead live in the `secrets.existingSecret` Secret, whose
contents the chart cannot check. The chart refuses all of them in `env`, `extraEnv`,
`settingsFile` and `secrets.extra`, and an `envFrom` source must not carry them either. The
deployment probe port goes through `cellProbe.port` below. Every other
setting goes in `env` (plain values) or `secrets.extra` (keys), named as in
the [configuration reference](/docs/zh-CN/reference/configuration). `settingsFile` takes the same
settings as the YAML settings file holds them, lowercase and typed:

```yaml
settingsFile:
  memory_search_default_mode: rrf
  memory_ask_recency_weight: 0.3
```

The chart always renders the settings file (an empty `settings` map when `settingsFile`
is empty), mounts it at `/app/config/loom.yaml` on every pod and on the publish, reseal
and rollback-check Jobs (each from its own copy, rendered with the Job), and rolls the
pods when it changes; `env` wins over it. A pod of the previous ReplicaSet that starts
during a rollout still finds the file, and reads the new revision's copy: a pod of the
previous build logs a setting it does not have and ignores it, so a new setting can go
in the same upgrade, but it refuses to start on a value only the new build accepts (a
new choice of an existing setting, a widened bound). Set such a value in the upgrade
after the one that changes the image. A live setting can also change without a rollout
([runtime settings](/docs/zh-CN/operations/runtime-settings)). `values.yaml` documents the chart's own values.

### Preparing a cell

For [cell preparation](/docs/zh-CN/architecture/cells#preparing-a-cell), enable the private
deployment-identity listener in the release's values:

```yaml
cellProbe:
  port: 8777
```

Upgrade the release and wait for the web and, when enabled, workers Deployments to
finish rolling before running prepare. Both pod types run the listener; a separate
workers child does not. The listener requires Postgres and authentication; this
value does not configure those prerequisites.

The chart creates a ClusterIP Service named `<fullname>-cell-probe`, shortening the
fullname to keep the suffix within 63 characters. It publishes unready pod addresses
and selects only those two workload types. Its configured origin is
`http://<probe-service>.<namespace>.svc:<port>`. A later cell passes that origin as
`--probe-url`; the router appends `/v1/health/deployment`. For the first cell, put the
origin in the router's `LOOM_ROUTER_UPSTREAM_PROBE_URL` instead and omit `--probe-url`
from prepare. The full direct diagnostic endpoint is
`http://<probe-service>.<namespace>.svc:<port>/v1/health/deployment`.

The default port is zero, which disables the listener and Service. Enabled ports
must be whole numbers from 1 to 65535 except 7777. Set the port only through
`cellProbe.port`: the chart refuses it in `env`, `extraEnv`, `secrets.extra` and
`settingsFile`, and renders an explicit value, including zero, over `envFrom`.
The endpoint membership label `loom.clickhouse.com/cell-probe` is chart-owned when
enabled. Application traffic and health probes continue to use the HTTP port.

ClusterIP supplies routing, not authentication. The listener is unauthenticated and
returns nonsecret deployment identity. With the chart's ingress NetworkPolicy on,
add a separate ingress policy allowing the probe port from router pods and the CLI's
location; without ingress isolation, cluster peers can reach it. The chart does not
add that rule. A load-balanced response does not prove every process was sampled;
the cell preparation checks remain required.

### On a laptop

```bash
kind create cluster
helm install loom deploy/helm/loom -f deploy/helm/loom/values-kind.yaml \
  --set secrets.jwtSecret=$(openssl rand -hex 32)
kubectl port-forward svc/loom 7777:7777
```

This runs ClickHouse inside the release, the fake embedder and no repository-job
sandbox. Loom's pods meet the `restricted` Pod Security Standard; the dev ClickHouse
does not. Mint a token with
`kubectl exec deploy/loom -- loom mint-token --subject me --org dev --namespaces '*' --roles admin`.

### On EKS

`values-eks.yaml` assumes the AWS Load Balancer Controller, the EBS CSI driver with a
`gp3` StorageClass and ClickHouse Cloud. Replace its placeholders (hostname,
certificate, account, VPC range) and install with it:

```bash
helm install loom deploy/helm/loom -n loom -f deploy/helm/loom/values-eks.yaml
```

- **ALB to pod IPs.** `target-type: ip` sends to pods directly. Label the namespace
  before the install, so each new pod carries the controller's readiness gate and a
  rollout removes no old pod before the ALB sends to its replacement:

  ```bash
  kubectl label namespace loom elbv2.k8s.aws/pod-readiness-gate-inject=enabled
  ```

  The target group checks `/v1/health/ready` every 5 s. A stopping pod keeps serving
  for 20 s (`shutdown.readyDelaySeconds`) while the controller deregisters it, and the
  deregistration delay (50 s, `readyDelaySeconds` + `gracefulSeconds`) lets it finish
  what it holds. The 300 s idle timeout covers long calls, and Loom keeps idle
  connections open longer (`keepAliveSeconds`, 310 s), so the ALB never reuses one Loom
  has just closed. The image reads it as `LOOM_KEEP_ALIVE_S`.

- **`trustedProxies`.** Set it to the ALB's subnets, or every client counts as the ALB
  for the per-address sign-in limits. Not the whole VPC: any pod there could then set
  `X-Forwarded-For`.
- **Secrets.** The overlay names a Secret `loom-secrets`. With the External Secrets
  Operator and AWS Secrets Manager:

  ```yaml
  apiVersion: external-secrets.io/v1
  kind: ExternalSecret
  metadata: { name: loom-secrets, namespace: loom }
  spec:
    secretStoreRef: { name: aws-secrets-manager, kind: ClusterSecretStore }
    target: { name: loom-secrets }
    refreshInterval: "0" # sync on demand only (see below)
    dataFrom:
      - extract: { key: loom/production } # LOOM_JWT_SECRET, LOOM_CH_PASSWORD, OPENAI_API_KEY
  ```

- **Identity.** The service account carries an IRSA role annotation, for Bedrock, S3
  backups or the document bucket. With EKS Pod Identity, bind the role to the service
  account and drop it.
- **Zones.** Pods spread over zones, then nodes, and still schedule when a zone is short.
  The spread is kept at scheduling only: after a drain two pods can share a zone until
  the next rollout or the descheduler moves one.
- **Cache on gp3.** Each pod gets its own ephemeral volume for `/app/var` (git mirrors,
  the Go module cache), created with the pod and deleted with it. The StorageClass needs
  `volumeBindingMode: WaitForFirstConsumer`, so each volume is made in its pod's zone.
- **NetworkPolicy.** On, with ingress from the ALB's subnets only. Pods reach DNS,
  ClickHouse, the EKS Pod Identity agent and port 443 anywhere but the instance metadata
  service; the cluster needs a policy engine (the VPC CNI's network policy agent, Calico
  or Cilium). Also set the node groups' IMDS hop limit to 1. A ClickHouse inside the
  cluster needs its own rule, as engines differ on matching pod IPs by range:
  `networkPolicy.extraEgress: [{to: [{namespaceSelector: …, podSelector: …}], ports: [{port: 8123}]}]`.
- **ClickHouse Cloud's IP access list** must allow the cluster's NAT gateway addresses.
- **Image.** Pin `image.digest`, and `extractor.image.digest` with `extractor.enabled`;
  each release's run summary prints both, after each image passed a CVE scan. The
  assistant's LibreChat and MongoDB images are pinned by digest in `values.yaml`
  (`assistant.image.digest`, `assistant.mongo.image.digest`).

## Upgrades and scaling

`helm upgrade` with a new `image.tag` rolls the pods one at a time. A new pod starts
serving only after its startup probe passes, and the old one is removed after that. The
mixed-version window is safe: builds N-1 and N run side by side on N's schema
([rolling upgrades](/docs/zh-CN/architecture/rolling-upgrades)). Each pod template carries its
env, so a pod of the previous ReplicaSet that restarts mid-rollout keeps its own. The
Secret is one for every pod, so a change to it reaches each pod as it next starts; a
chart-managed Secret change rolls the pods through its checksum, and an `existingSecret`
change needs `kubectl rollout restart`. Keep a reloader off the Secret that holds
`LOOM_JWT_SECRET`: a root change has to go through a publish (below). For an upgrade whose
release notes ask for every pod at once (state that moves between builds), add
`--set strategy.rollingUpdate.maxSurge=100%`, and set it back on the next upgrade:
`--reuse-values` keeps it.

A stopping pod answers not-ready and keeps serving for `shutdown.readyDelaySeconds`
(15 s) while the Service and the balancer drop it. uvicorn then waits up to
`shutdown.gracefulSeconds` (30 s, the longest `task.pull` wait) for requests in flight,
and the lifespan drains and retires the pod. `terminationGracePeriodSeconds` is the sum
of the three `shutdown` values, 95 s by default. Behind a balancer with a slower check,
raise `readyDelaySeconds`. A pod that is killed (OOM, node loss) fails only the requests
it held; a client's retry lands on another pod.

Scaling is `replicaCount`, `kubectl scale` or the HPA. It changes no pod config, so it
rolls nothing. With `autoscaling.enabled` each revision records the Deployment's live
replica count (at least `autoscaling.minReplicas`), so turning the HPA on does not first
scale it down. A `helm rollback` writes the count its revision recorded, and the HPA
then corrects it: size `minReplicas` for the load a rollback must carry. The workers lease moves when its holder stops: within seconds on a
graceful stop, and within the 30 s lease TTL when a pod or node dies. Every pod is
labelled `deploy:k8s:<pod>`, so a new leader runs the workers whose last pass is
overdue rather than waiting out their interval again.

A pod that dies without stopping leaves its provider participant row. A missing
or replaced pod object, a Failed/Succeeded API phase and a completed rollout do
not prove that an old process on a partitioned node stopped. Such rows remain
blockers until guarded shutdown or verified exact physical death with settled
queries. Loom requests no pod-read permission or service-account token for
participant retirement. The explicit `serviceAccount.automountToken` value stays
available for workloads that need a token for another integration.

Use `loom participant retire --participant <id>` on the trusted host where its
exact PID/namespace/start identity is verifiable. If that evidence is unavailable,
use the [quiescent lifecycle recovery](/docs/zh-CN/architecture/cells#participant-lifecycle-baseline)
after positively terminating or fencing every relevant runtime. API object
cleanup alone cannot authorize retirement.

There is no migration Job. Every pod migrates at boot under a ClickHouse lease, so a
Job would only repeat that step, and a pre-upgrade hook would run the new schema while
every old pod still serves, outside the window the carry barrier protects.

## Changing what cannot roll

The embedding settings (`embedding.*`) and `LOOM_JWT_SECRET` must be the same on every
pod. A pod whose values differ from the deployment's published ones is not ready, so
these do not roll. The chart refuses an upgrade that changes them: it keeps a hash of
the embedding settings on the Deployment and compares the root secret with its Secret.

```text
Error: UPGRADE FAILED: ... this upgrade changes the embedding settings or LOOM_JWT_SECRET ...
```

To make the change on purpose, upgrade with `--set mustMatchChange.publish=<token>`, any
value the pods don't carry yet, such as `$(date +%s)`: the upgrade that brings a new token
publishes, and later ones that keep it (`--reuse-values`) don't. A pre-upgrade Job runs
`loom signing-key record-root` and `loom deployment publish --yes` with the new values.
From that moment the old pods are not ready and answer calls on the connections they
still hold with `503 CONFIG_DRIFT`, and the rollout replaces every pod at once
(`maxSurge: 100%`). The outage lasts until the first new pod is ready and registered with
the balancer: about 10 s with the image on the node, plus the image pull and the ALB's
registration otherwise. Change nothing else in that upgrade (no new image, no pending
migration), and pull the image onto the nodes first. The Deployments keep that surge
strategy until the next upgrade: follow the publish with a `helm upgrade --reuse-values`
(the same token, so no publish), or a later `kubectl rollout restart` surges every pod.
A `helm rollback` to the publish revision brings that strategy back too: follow it the
same way.
Then:

- **New root secret:** tokens the old root signed (kid `default`) stop verifying at once;
  mint new ones. Tokens signed with a keyring key (`loom signing-key rotate`) keep
  verifying. Set `secrets.jwtSecretPrevious` to the old root in the same upgrade, and what
  it sealed keeps opening: after the rollout a `post-upgrade` Job runs `loom secrets
reseal`, which seals the stored credentials, signing keys and rebuild recipes again under
  the new root (the publish Job checks first that it can, so a failing one publishes
  nothing). The previous root only opens sealed values; it never verifies a token. Remove
  it in a later upgrade once `kubectl exec deploy/loom -- loom secrets reseal --check`
  exits 0. Until then it also counts the SSO sign-ins in flight (a code, a device grant,
  an approval or link awaiting its click) whose identity-provider token only the old root
  opens: they are not rewritten, and expire within minutes; the check names when the last
  one does. A value it lists as opening with neither root was lost before this rotation:
  enter it again. Without it, the stored credentials stop opening: enter them again.
  If the old root leaked, also rotate the signing keys and retire the old kids
  ([tokens and grants](/docs/zh-CN/guides/tokens-and-grants#revoke-and-rotate)).
- **New embedding model:** run `loom reembed` in a pod
  ([deployment](/docs/zh-CN/operations/deployment)).

Run the upgrade with `--wait` and a `--timeout` above the startup budget (15 minutes), and
never with `--atomic`: a rollback after the Job has published leaves every pod drifted. If
the upgrade fails after the Job ran (the Job half-done, or the apply refused), every pod
is drifted: run the same upgrade again with the same token, which publishes again. If
the upgrade timed out after the rollout started, the pods already carry the token and the
retry does not publish; while `secrets.jwtSecretPrevious` is set, every upgrade's
`post-upgrade` Job reseals what is left, and `loom secrets reseal` does it by hand.

Every `helm rollback` first runs `loom deployment publish --check` with the target
revision's env, and refuses a rollback whose root secret or embedding settings are not the
published ones: its pods would never be ready, and the Secret it re-applies would drift
every pod that restarts. Helm then leaves the release `pending-rollback`; a
`helm rollback loom <current revision>` passes the check and clears it. To go back past a
publish, upgrade with the old values and a new token instead.

Installing onto a ClickHouse whose deployment already published another root secret or
embedding space (a move from another install, a restore) brings up only drifted pods, as
hooks do not publish on install: follow the install with an upgrade that sets a token.

With `secrets.existingSecret` the chart cannot see the Secret's contents, so it does not
guard a new root secret there. A pod that starts after the Secret changed and before the
publish is drifted and never ready, so change it only together with a publish: with the
External Secrets Operator, update Secrets Manager (the new `LOOM_JWT_SECRET`, and the old
one as `LOOM_JWT_SECRET_PREVIOUS`; `secrets.jwtSecretPrevious` applies to the chart's own
Secret only), force a sync
(`kubectl annotate externalsecret loom-secrets force-sync=$(date +%s) --overwrite`), wait
for the ExternalSecret to be `Ready`, then upgrade with a new token, which rolls the pods
so they read it. The guard and the live replica count read the release with `lookup`,
which `helm template` (and so Argo CD) does not run: there the guard is off, and every
render with a token set would publish again, so never set `mustMatchChange.publish` there.
A publish is then `loom secrets reseal --dry-run && loom signing-key record-root && loom
deployment publish --yes` in a pod with the new env (the dry run checks the previous root
while the old one is still the recorded root), then `loom secrets reseal` once the new pods
serve. A `loom login` code shown while the root changes stops matching (its key derives
from the root): run `loom login` again. So does an SSO sign-in started on a pod with the
other root: start it again.

## Cache and git mirrors

Each pod clones its own mirrors at startup and keeps them fetched. A pod is ready before
its clones finish: until a mirror is there, history reads (blame, commit checks, PR
history) answer `no_git_mirror` and the rest of the code plane serves from ClickHouse.
A clone takes as long as the repository: a bare clone of a large monorepo (14 GB) is
about 12 minutes, two at a time per pod. To hold traffic until the mirrors are cloned,
set `env.LOOM_READY_REQUIRES_MIRRORS: "true"` and raise `probes.startup`.

The mirrors live in the pod's cache at `/app/var`, with the Go module cache, and are
lost with the pod. The default `emptyDir` (`cache.sizeLimit`, 20 GiB) counts against the
node's storage, and the kubelet evicts a pod that outgrows it. `cache.volume: ephemeral`
gives each pod its own volume from a StorageClass instead (`cache.ephemeral`). Size it
for full clones of every indexed repository plus the module cache. `workers.cache` is
merged over `cache` for the workers pods, which also keep a checkout per compiler tier:
about 30 GB beside the mirror for a ClickHouse-sized C++ entry with its build tree.

`cache.ephemeral.dataSourceRef` starts each new pod's volume from a snapshot instead of
empty, e.g. a `VolumeSnapshot` of a warm workers volume (on EKS this needs the CSI
snapshot controller and a `VolumeSnapshotClass` for the EBS driver). The mirrors and
checkouts then fetch only what is new, and the C++ tier skips its CMake configure.
Refresh the snapshot now and then: an old one still works, it just leaves more to fetch.

## Compiler tiers

Every compiler tier runs in the workers process: the per-merge deltas, the weekly C++
full re-index (`code_scip_full`, which also seeds a new entry), the SQL ↔ C++ crossref
and the checkout repacks. The image carries every indexer; scip-clang is in the amd64
image only, because upstream builds no arm64 Linux binary, so the chart schedules on
`kubernetes.io/arch: amd64` nodes by default. On arm64 set `nodeSelector` to
`{kubernetes.io/arch: null}` and mount your own scip-clang
(`env.LOOM_CODE_SCIP_CLANG_PATH`), or the C++ tier skips with `scip-clang missing`.

A C++ full index is the workers pod's peak. scip-clang runs `workers.scipClangJobs` (4)
translation units at once at a few GB each, and the default `workers.resources` (32 GiB
limit) is sized for that. Without the workers Deployment the web pod that holds the
lease runs them with one job inside its own limits. The index and its delta are written
under `/app/var/scip-work` on the cache volume (`LOOM_CODE_SCIP_WORK_DIR`), not `/tmp`:
several GB for a full run of a large tree.

## Repository-job sandbox

Repository jobs run repository code ([repository-job sandbox](/docs/zh-CN/operations/code-index-operations#repository-job-sandbox)).
Loom runs them under bubblewrap, which needs unprivileged user namespaces. The default
pod security does not allow them, so by default (`sandbox.required: false`) the compiler
tiers of the repositories the operator lists in `LOOM_CODE_REPOS_JSON` run as plain
processes in the pod, and a repository added from the dashboard is refused. List only
repositories you trust. `LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY` and
`LOOM_CODE_SELF_SERVICE_GIT_UNSANDBOXED` in `env` let platform admins add repositories
with `code.repo.add` in such pods: their clone and fetches run as plain processes too,
and they get no compiler tier
([without the sandbox](/docs/zh-CN/operations/code-index-operations#repository-job-sandbox)). With
`sandbox.required: true` a pod is not ready until the sandbox works, so set it only once
the nodes are prepared:

1. Run Loom on nodes whose OS allows unprivileged user namespaces.
2. Install `dev/seccomp/loom-repo-sandbox.json` under the kubelet's seccomp directory
   (`/var/lib/kubelet/seccomp/`) and load `deploy/apparmor/loom-repo-sandbox` on those
   nodes, for example with a DaemonSet or the node image.
3. Point the container at them:

   ```yaml
   sandbox:
     required: true
     hostUsers: false
     securityContext:
       seccompProfile: { type: Localhost, localhostProfile: loom-repo-sandbox.json }
       appArmorProfile: { type: Localhost, localhostProfile: loom-repo-sandbox }
       procMount: Unmasked
   ```

`procMount: Unmasked` needs a user namespace (`sandbox.hostUsers: false`, Kubernetes 1.33
or later, where `ProcMountType` and `UserNamespacesSupport` are on), and the namespace's
Pod Security level must allow it (`baseline` and `restricted` do not). Run
`dev/sandbox_probe.sh` in a pod to check the result before setting `sandbox.required: true`.

## Instance metadata service

By default, and with `networkPolicy.enabled`, a NetworkPolicy (enforced only where the
cluster runs a policy engine) keeps the Loom pods off `networkPolicy.egressExcept` (the
instance metadata service), so repository code a pod runs cannot take the node's cloud
role. With the policy off, `networkPolicy.guardMetadata` renders an egress-only
NetworkPolicy whose one rule allows every pod and every address but those (none with
`egressExcept: []`). The policy's effects:

- A pod has no cloud credentials from the metadata service: not the node's instance
  role, and not GKE Workload Identity on Dataplane V2. IRSA, and EKS Pod Identity on an
  engine that matches node addresses by range, work. With the policy off, set
  `guardMetadata: false` to keep metadata access; with `networkPolicy.enabled`, open it
  through `extraEgress`.
- `egressExcept` applies with the policy off too: a range listed there (a VPC range,
  say) cuts the pods off every address in it but other pods, and an entry that is not a
  CIDR fails the upgrade.
- A metadata proxy that redirects 169.254.169.254 to another address is reached at
  that address, which the engine may check instead: add the address of one that passes
  node credentials through to `egressExcept` (outside the cluster's pods, which stay
  reachable).
- A service in the cluster that fetches URLs for its callers (a forward proxy, a
  ClickHouse whose `url()` reaches any host, such as the test-only dev ClickHouse) reaches
  `egressExcept` for them: deny those addresses in its own policy.
- NetworkPolicies add up: where your own policies limit the pods' egress, this one
  lets them reach everything else. Set `guardMetadata: false` and deny the metadata
  addresses in your own policy. Set it too when your own policy is named after the
  release: Helm refuses to take over an object it does not own, so the upgrade fails.
- An engine that keeps address ranges off node IPs (Cilium's default) blocks host-network
  endpoints and the nodes' own addresses: NodeLocal DNSCache (so DNS), the EKS Pod
  Identity agent, where the engine
  gives it an identity of its own. Turn on the engine's node matching, or set
  `guardMetadata: false` and allow them in your own policy.

A `helm upgrade --reuse-values` from a chart with a shorter `egressExcept` keeps that
list: set `networkPolicy.egressExcept` again, or upgrade with `--reset-then-reuse-values`
(Helm 3.14 or later).

## TLS on the Loom port

`tls.secretName` names a `kubernetes.io/tls` Secret in the release's namespace. Every web
and workers pod mounts its `tls.crt` and `tls.key` (mode 0440, group
`podSecurityContext.fsGroup`), serves them on port 7777 with HTTPS only and probes over
HTTPS; the Service marks the port `appProtocol: https`. `tls.minVersion` sets
`LOOM_TLS_MIN_VERSION`. The chart refuses the `LOOM_TLS_*` settings anywhere else: the
mount and the HTTPS probes come with `tls.secretName`.

A controller that renews the Secret in place needs no restart. The kubelet updates the
mount within its sync period, and each web process serves the new pair from its next
handshake ([TLS on the Loom port](/docs/zh-CN/operations/deployment#tls-on-the-loom-port)). The Secret must
exist before the pods start.

The Ingress or proxy in front then connects over HTTPS. On EKS, add
`alb.ingress.kubernetes.io/backend-protocol: HTTPS` to `ingress.annotations`: the chart
refuses an `alb` Ingress without it, or with another `backend-protocol` or
`healthcheck-protocol` in `ingress.annotations` or `service.annotations` (the controller
reads both, the Service's first). With ingress-nginx, set
`nginx.ingress.kubernetes.io/backend-protocol: HTTPS`; other controllers have their own
setting, which the chart does not check. Setting or
clearing `tls.secretName` on a running release breaks requests until the rollout ends:
the balancer and in-cluster clients switch scheme for every pod at once, while the pods
restart one at a time. Plan a window, or install a second release with TLS and move the
clients to it. A client in the cluster that calls the Service by URL, such as
`deploy/okf-sync/cronjob.yaml` (`PRIVATE_URL`), switches to `https://` and trusts the CA
that issued the certificate. For the `loom` CLI, `SSL_CERT_FILE` names that CA and
replaces the system roots, so set it on the command that calls the Service only, or point
it at a bundle of the system roots and the CA. The assistant calls the port
over plain HTTP, so the chart refuses `assistant.enabled` with `tls.secretName`. The hop
from Loom to the release's extractor (`extractor.enabled`) stays plain HTTP. Keep
`tls.minVersion` at `"1.2"` unless every client of the port, the balancer included,
speaks TLS 1.3. `helm test` calls the Service over HTTPS without verifying the
certificate.

## Separate workers

`workers.enabled: true` moves the background workers to their own Deployment
(`workers.replicaCount`, `workers.resources`). Its pods take no traffic and elect one
leader among themselves. The web pods run no workers. Use it when the workers need
more memory than a web pod, or to scale them apart from it. With `replicaCount: 2`
the second pod is a standby with warm mirrors, and a PodDisruptionBudget keeps one of
the two through a drain.

## Document extractor

`extractor.enabled: true` runs the document extractor service ([document
extraction](/docs/zh-CN/architecture/document-extraction)) as a Deployment and a Service of
its own, `extractor.replicaCount` pods (two by default), each running one extraction at
a time. The Loom pods get `LOOM_DOCUMENT_EXTRACTOR_URL` pointing at it, unless `env` or
`extraEnv` sets one, and both sides read the bearer token from the release Secret's
`LOOM_DOCUMENT_EXTRACTOR_TOKEN` key: `secrets.extractorToken` (16 characters or more;
`openssl rand -hex 24`), or that key in `secrets.existingSecret`. The extractor pods
read that one key and nothing else of the Secret, mount no service-account token, run as
uid 10001 on a read-only root filesystem with every capability dropped, and carry labels
of their own, so no policy or Service that selects the Loom pods selects them. Their
NetworkPolicy, rendered whatever `networkPolicy.enabled` says, admits the release's Loom
pods on 8790 and allows no egress; either form of the Loom pods' policy allows egress to
them. `extractor.env` takes the service's `LOOM_EXTRACTOR_*` limits
([extractor](https://github.com/ClickHouse/loom/blob/main/extractor/README.md)), and `extractor.resources` bounds each pod.
Each side holds one token, so changing `secrets.extractorToken` restarts the Loom
Deployments (web, and workers when enabled) and the extractor's, and while they roll, a
Loom pod and an extractor pod may hold different tokens: extraction answers unavailable
until every roll finishes.

## Assistant

`assistant.enabled: true` runs LibreChat headless beside Loom with a MongoDB of its own
([deployment](/docs/zh-CN/architecture/assistant#deployment)): a Deployment of one pod,
replaced rather than rolled and restarted when its configuration changes, its Service
and ConfigMap, and a MongoDB StatefulSet with `assistant.mongo.storage` of disk. It
needs `assistant.existingSecret`, a Secret with `CREDS_KEY`, `CREDS_IV`, `JWT_SECRET`,
`JWT_REFRESH_SECRET`, and `ANTHROPIC_API_KEY` while the assistant runs on Anthropic or
`LOCAL_LLM_BASE_URL` for `LOOM_ASSISTANT_PROVIDER=local`; the chart refuses to render
without it.
The Loom pods get `LOOM_ASSISTANT_LIBRECHAT_URL` pointing at it, unless `env` or
`extraEnv` sets one, and set the service user and agent up at boot. LibreChat's
NetworkPolicy admits only the release's Loom pods and lets it reach DNS, its MongoDB,
the Loom pods and TCP 443 for its model provider; MongoDB's admits only LibreChat and
reaches nothing, and Loom's own policy admits LibreChat when it names its peers. Both
run non-root on a read-only root filesystem. `assistant.nodeSelector`,
`assistant.tolerations` and `imagePullSecrets` apply to both pods.

For `LOOM_ASSISTANT_PROVIDER=local`, a model server you run instead of Anthropic, add
`LOCAL_LLM_BASE_URL` to the Secret and write a NetworkPolicy of your own that lets the
LibreChat pod reach the server; the
[local-model setup](/docs/zh-CN/getting-started/models-and-keys#a-model-server-on-your-machine-no-api-key)
lists the rest.

## Analytics export

`analyticsExport.enabled: true` adds a daily CronJob that writes the previous UTC day's
usage and a namespace snapshot to an S3 bucket as Parquet, under a ServiceAccount of its
own ([exporting usage to a data warehouse](/docs/zh-CN/operations/analytics-export)).

## Alert rules

`alerts.prometheusRule.enabled: true` adds a PrometheusRule (the Prometheus Operator's
CRD, which the chart does not install) for a deployment that exports Loom's metrics over
OTLP into Prometheus: `LoomCodeTierFailing` for a code-index tier the `code_tier_failing`
health problem reports, and `LoomCodeTierFailingFarBehind` for a tier the problem
grades `error`. Their thresholds are the `LOOM_CODE_TIER_FAILING_*` settings
([when a tier fails](/docs/zh-CN/operations/code-index-operations#when-a-tier-fails)).
`alerts.prometheusRule.labels` adds the labels your Prometheus selects rules by.

## Checks

```bash
helm test loom -n loom                       # live, ready and /v1/health through the Service
kubectl -n loom exec deploy/loom -- loom doctor
kubectl -n loom logs -l app.kubernetes.io/instance=loom --tail=-1 | grep -E "WorkerSupervisor: starting|supervisor lease: acquired"
```

## Sign-in

SSO is configured as on any host ([single sign-on](/docs/zh-CN/operations/sso)): the issuer and client id
in `env`, the client secret in the Secret.

```yaml
env:
  LOOM_OAUTH_LOGIN_PROVIDER: oidc
  LOOM_OAUTH_ISSUER: https://loom.example.com # the ingress host; the callback is built from it
  LOOM_OIDC_ISSUER: https://your-org.okta.com/oauth2/default
  LOOM_OIDC_CLIENT_ID: your-client-id
secrets:
  extra:
    LOOM_OIDC_CLIENT_SECRET: your-client-secret # or a key of secrets.existingSecret
```

Register `<LOOM_OAUTH_ISSUER>/oauth/callback` with the identity provider. Every
replica must run the same provider: set it in one place, and roll it out like any
other change; a sign-in whose steps land on two replicas that disagree is started
again. Run `loom sso adopt` (see the SSO page) from one pod, after the rollout:

```bash
kubectl -n loom exec deploy/loom -- loom sso adopt --domain example.com
```

## Model calls through an inference gateway

To send Loom's own model calls through a gateway that takes a workload token instead
of a provider key ([models and keys](/docs/zh-CN/getting-started/models-and-keys#through-an-inference-gateway)),
mount a projected service-account token and point Loom at it:

```yaml
env:
  LOOM_OPENAI_BASE_URL: https://gateway.example/v1
  LOOM_ANTHROPIC_BASE_URL: https://gateway.example
  LOOM_INFERENCE_TOKEN_URL: https://gateway.example/oauth/token
  LOOM_INFERENCE_WORKLOAD_TOKEN_FILE: /var/run/secrets/inference/token
extraVolumes:
  - name: inference-token
    projected:
      sources:
        - serviceAccountToken:
            audience: gateway.example # what the gateway expects
            expirationSeconds: 3600
            path: token
extraVolumeMounts:
  - name: inference-token
    mountPath: /var/run/secrets/inference
    readOnly: true
```

The kubelet renews the file before it expires and Loom rereads it on each exchange,
so nothing restarts. The gateway trusts the cluster's OIDC issuer for this service
account (`serviceAccount.name`); that trust is set up on the gateway's side. The
NetworkPolicy's egress to port 443 already covers the gateway and the identity
provider.

A gateway that verifies the projected token itself takes it as the bearer: drop the
token URL and set the bearer mode. Loom rereads the file at most once a minute, and at
once after a 401. To name the agent each call is for, add identity headers
([models and keys](/docs/zh-CN/getting-started/models-and-keys#naming-the-agent-on-every-call)):

```yaml
env:
  LOOM_INFERENCE_WORKLOAD_TOKEN_MODE: bearer
  LOOM_INFERENCE_WORKLOAD_TOKEN_FILE: /var/run/secrets/inference/token
  LOOM_INFERENCE_GATEWAY_HOSTS: gateway.example
  LOOM_INFERENCE_IDENTITY_HEADERS_JSON: '{"X-Agent": "{agent}"}'
```

## Git through a forward proxy

To send the code index's network git through a forward proxy
([git through a forward proxy](/docs/zh-CN/operations/code-index-operations#git-through-a-forward-proxy)),
give every Loom pod the proxy's address, a token file for its password and, for a proxy
that terminates TLS under its own authority, the CA bundle:

```yaml
env:
  LOOM_CODE_GIT_PROXY_URL: http://git-proxy.example:3128
  LOOM_CODE_GIT_PROXY_TOKEN_FILE: /var/run/secrets/git-proxy/token
  LOOM_CODE_GIT_PROXY_CA_FILE: /etc/git-proxy/ca.crt
  LOOM_CODE_GIT_PROXY_IDENTITY: loom
extraVolumes:
  - name: git-proxy-token
    projected:
      sources:
        - serviceAccountToken:
            audience: git-proxy.example # what the proxy expects
            expirationSeconds: 21600 # well above the longest git job
            path: token
  - name: git-proxy-ca
    configMap:
      name: git-proxy-ca
extraVolumeMounts:
  - name: git-proxy-token
    mountPath: /var/run/secrets/git-proxy
    readOnly: true
  - name: git-proxy-ca
    mountPath: /etc/git-proxy
    readOnly: true
```

`env` and the two mounts reach the web pods and the workers alike: both fetch mirrors.
The kubelet renews the token file and Loom reads it again for every git job, so nothing
restarts. A job presents the token it read at its start for as long as it runs (a fetch
up to `LOOM_CODE_GIT_FETCH_TIMEOUT_S`, an hour by default), so the proxy must accept a
token for that long after Loom read it. With `networkPolicy.enabled`, a proxy on a port
other than 443 needs a `networkPolicy.extraEgress` rule.

An `http://` proxy URL, as above, sends the identity and the token to the proxy in clear
on every job: keep that path inside a network you trust, or give the proxy an `https://`
URL. Git then verifies the proxy's certificate before it sends them, against its default
roots. For a certificate under an authority of your own, mount that authority's bundle
and name it:

```yaml
env:
  LOOM_CODE_GIT_PROXY_URL: https://git-proxy.example:8443
  LOOM_CODE_GIT_PROXY_TLS_CA_FILE: /etc/git-proxy/proxy-ca.crt
```

`proxy-ca.crt` is one more key of the `git-proxy-ca` config map here. It is not
`LOOM_CODE_GIT_PROXY_CA_FILE`, which verifies the remotes behind a proxy that terminates
their TLS: a proxy that does both under one authority gets the same file in both
settings. The proxy's certificate must name `git-proxy.example`, the host the URL
spells.

The proxy and the [repository-job sandbox](#repository-job-sandbox) are not used
together. `sandbox.required: false` alone does not keep the sandbox off: a pod whose
profiles let bubblewrap run uses it whatever that value says, and then every network git
job is refused. Where the proxy is set, leave `sandbox.securityContext` and
`sandbox.hostUsers` at their defaults as well. Repository jobs are then plain processes:
a compiler tier's build runs as Loom's user and can read the token file, so run compiler
tiers there only for repositories you trust. To add repositories with `code.repo.add` in
such pods, set `LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY` and
`LOOM_CODE_SELF_SERVICE_GIT_UNSANDBOXED` as well
([repository-job sandbox](#repository-job-sandbox)): their git goes through the proxy
under the identity the add names, else the default one.

## Related

- [Running N replicas](/docs/zh-CN/operations/replicas) and [deployment](/docs/zh-CN/operations/deployment#n-identical-replicas)
- [Upgrades](/docs/zh-CN/operations/upgrades), [monitoring](/docs/zh-CN/operations/monitoring)
