5 Commits

Author SHA1 Message Date
Hyeonsang Cho
54dc5024a6
fix(gateway): serialize the readiness database probe (#8277)
* fix(gateway): serialize the readiness database probe

/health/ready is public, and the module promises that connection-opening
probes run behind a per-kind gate. The ORM database probe never did: it
checked a connection out of the repositories' own engine pool on every
request, so concurrent probe traffic could hold the whole pool (15
connections by default) and leave request handlers waiting.

Run it behind _probe_gate("database") like the checkpointer, stream
bridge and provisioner probes. The gate is per kind, so one request's
probes still run concurrently.

* test(gateway): assert separate probe gates by rendezvous, not wall clock

Each probe now waits until the other has entered its gate, so a shared
gate fails through the readiness deadline instead of a timing margin.
2026-11-16 15:35:18 +08:00
Nan Gao
5e3c24fe94
fix(gateway): probe the Redis stream bridge and provisioner in /health/ready (#6447)
* fix(gateway): probe the Redis stream bridge and provisioner in /health/ready

GET /health/ready only probed the ORM engine and the LangGraph checkpointer.
The Redis stream bridge client connects lazily and nothing ever pinged it, so a
Gateway whose Redis was down started, answered 200 ready to the Kubernetes
readinessProbe and Compose healthcheck, and kept receiving traffic while every
run's first publish failed. Since #6328 the Redis bridge is mandatory for
multi-instance deployments, so such a replica is not serviceable at all.

- StreamBridge gains `async ping() -> bool`; the base returns True (no external
  backend), RedisStreamBridge round-trips PING and answers False on RedisError.
- /health/ready adds a `stream_bridge` probe (not_configured for the memory
  bridge; unreachable flips the endpoint to 503; a missing bridge fails closed)
  and a report-only `provisioner` probe of `{sandbox.provisioner_url}/health`
  that never changes the status code, since every replica shares one
  provisioner. The provisioner URL is snapshotted on app.state at startup.
- A probe that overruns the endpoint deadline is now the only one marked
  unreachable; finished verdicts are kept.
- The public probe gate is one lock per probe kind, so one request's probes
  still run concurrently while a thundering herd opens at most one probe
  connection per kind per process.
- Tests: bridge ping cases, stream-bridge/provisioner probe cases, verdict and
  deadline cases, and the first route-level /health/ready tests via TestClient.
- Docs: gateway AGENTS.md, CONFIGURATION.md multi-instance notes, Helm README
  and probe comment, Compose healthcheck comment.

* docs(changelog): note the readiness probe fix (#6447)

* fix(gateway): apply the sandbox proxy policy to the provisioner probe

Review follow-up for #6447.

- The provisioner readiness probe built its httpx client with the default
  trust_env=True. httpx 0.28 treats CIDR entries in NO_PROXY as host patterns
  rather than subnet bypasses, so with HTTP_PROXY set and NO_PROXY=10.0.0.0/8
  a private provisioner that the requests-based sandbox calls reach directly
  would have been probed through the proxy and reported unreachable on every
  check. The probe now reuses `sandbox_http_trust_env`: loopback, private,
  link-local and cluster-local addresses bypass the proxy, external hosts keep
  the environment's proxy settings.
- Pin the liveness/readiness separation with a route test: `GET /health`
  stays 200 while the stream bridge and database are unreachable and
  `/health/ready` is 503, so a Redis blip can never become a restart loop.
2026-10-08 06:56:33 +08:00
Hyeonsang Cho
b37f769b67
fix(persistence): reject SQLite file: URIs instead of opening them as filenames (#6069)
* fix(persistence): reject SQLite file: URIs instead of opening them as filenames

resolve_sqlite_conn_str passed a file: connection string through
unchanged, but LangGraph's SQLite checkpointer and Store factories
(SqliteSaver, SqliteStore, AsyncSqliteSaver, AsyncSqliteStore) connect
without uri=True, so SQLite treated the whole URI as a literal filename:
file:checkpoints.db?mode=rwc created a file with that exact name in the
working directory, file::memory:?cache=shared persisted to disk, and a
file:/// URI failed to open. The readiness probe did open URIs with
uri=True, so it checked a different file than the runtime used and
reported in-memory URIs as not_configured.

Reject file: URIs in resolve_sqlite_conn_str, the one helper all four
factories go through, with an error that names
checkpointer.connection_string. The probe drops its URI parsing and
reports a rejected connection string as unreachable.

* docs(changelog): reference #6069

* test(persistence): pin the case-sensitive file: URI check
2026-09-30 09:40:02 +08:00
NanPan
ce99b37a67
fix(gateway): drain readiness probe connection close (#5684)
* fix(gateway): drain readiness probe connection close

* docs(gateway): clarify readiness teardown deadlines
2026-09-23 10:24:56 +08:00
Michael
4791e94a73
feat(gateway): add /health/ready readiness probe backed by the database (#5166)
* feat(gateway): add /health/ready readiness probe backed by the database

## Why

GET /health only proves the process is up: it returns 200 even when the persistence engine cannot reach the database. Orchestrators already treat it as a readiness gate (docker-compose.yaml marks the gateway service healthy and nginx depends_on service_healthy), so a DB outage or a still-migrating Postgres leaves the stack 'healthy' while every request fails.

## What changed

- New GET /health/ready endpoint: bounded SELECT 1 against the existing persistence engine (deerflow.persistence.engine.get_engine) with a 2s timeout.

- Response is 200 {'status': 'ready', 'database': 'ok'} when reachable, 503 {'status': 'degraded', 'database': 'unreachable'} when the probe fails, and 200 ready with database=not_configured for backend=memory (nothing to probe).

- GET /health is unchanged (pure liveness), and /health/ready is public through the existing /health auth whitelist.

- docker-compose.yaml gateway healthcheck now polls /health/ready so service_healthy reflects database reachability.

- Documented both endpoints in backend/app/gateway/AGENTS.md.

## Surface area

- [x] Backend API - new GET /health/ready endpoint under backend/app/gateway

- [x] Sandbox / Docker - gateway healthcheck in docker/docker-compose.yaml now gates on readiness

- [ ] Frontend UI / Agents / Skills / Dependencies

- [x] Default behavior change - existing /health unchanged; the prod compose healthcheck is stricter (503 while the database is unreachable)

## Validation

- New unit tests in backend/tests/test_gateway_health.py cover ok / unreachable / not_configured probe results and the 200/503 payload mapping (6 passed).

- app.gateway.app imports cleanly and registers both /health and /health/ready.

- ruff check + ruff format clean.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** design, implementation, and unit tests produced with AI assistance; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

* fix(helm): point the gateway readiness probe at /health/ready

## Why

Review on #5166 (willem-bd, P1): the chart still probed /health for readiness,
so Kubernetes marked the pod ready and routed traffic while the database was
unreachable - exactly the failure mode /health/ready was added to catch.

## What changed

- deploy/helm/deer-flow/templates/gateway-deployment.yaml: readinessProbe
  httpGet.path now hits /health/ready (DB-backed, 503 while the database is
  unreachable). The liveness probe stays on /health.

## Verification

- One-line path change inside the existing readinessProbe block; git diff
  confirms only the readiness path changed (liveness untouched).

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** implemented the reviewer-requested probe path change; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

* fix(gateway): readiness probe also checks the effective checkpointer/Store backend

## Why

Follow-up review on #5166 (willem-bd, P1): get_engine() only represents the
ORM backend selected by `database:`. The legacy `checkpointer:` section takes
precedence for the LangGraph checkpointer and Store, so a split configuration
(a local SQLite/memory `database:` with `checkpointer.type: postgres`) could
report 200 while the PostgreSQL backend agent runs depend on was down.

## What changed

- GET /health/ready now probes both persistence halves: the ORM engine behind
  `database:` (unchanged) and the effective LangGraph checkpointer/Store
  backend resolved with the runtime's own rule (legacy `checkpointer:` config,
  otherwise derived from `database:`), for memory/sqlite/postgres backends.

- The payload gains a `checkpointer` field with the same
  ok / not_configured / unreachable vocabulary as `database`; 503 degraded is
  returned when either probe is unreachable.

- Probes are bounded by the existing 2s timeout: sqlite via aiosqlite SELECT 1
  on the resolved path, postgres via a bounded psycopg AsyncConnection SELECT 1
  on the DSN with the configured search_path. A missing driver for a configured
  backend degrades readiness (the runtime could not run either).

- Documented the two-probe semantics in the endpoint docstring and
  backend/app/gateway/AGENTS.md.

## Verification

- New tests: healthy ORM engine + unreachable legacy checkpointer backend ->
  503 degraded with database: ok / checkpointer: unreachable; checkpointer
  probe mapping for memory/sqlite(postgres missing-driver) backends; existing
  payload tests now pin the checkpointer field.

- cd backend && python -m pytest tests/test_gateway_health.py: 11 passed.

- app.gateway.app imports cleanly; ruff check + ruff format clean.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** design, implementation, and unit tests produced with AI assistance; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

* fix(gateway): bound /health/ready to one deadline and probe the startup checkpointer snapshot

## Why

Second round of review on #5166 (zhfeng P1/P2, willem-bd P1/P1/P2). Three
correctness issues remained in the readiness endpoint:

- The database and checkpointer probes ran sequentially, each allowed 2s, so a
  healthy response could take almost 4s - past Kubernetes' 1s default
  readinessProbe timeout and inside Docker's 3s client timeout. A slow but
  healthy backend could make every replica unready.

- The checkpointer probe re-resolved process-wide, hot-reloaded configuration
  per request, while app.state.checkpointer/store are built once from the
  startup_config snapshot in langgraph_runtime(). After a live config edit the
  endpoint could probe a backend the running gateway does not use, and a
  resolution failure was swallowed into None -> not_configured -> 200.

- The SQLite probe opened the path with aiosqlite.connect(), which creates the
  file when missing: a deleted checkpoint database was silently resurrected as
  an empty file and reported ok instead of surfacing the outage.

## What changed

- backend/app/gateway/health.py: the two probes now run concurrently beneath a
  single endpoint-wide deadline (_READINESS_DEADLINE_SECONDS=3.0) so a healthy
  response completes within one probe window (~2s), never the sum of both.
  A probe that overruns the deadline degrades the endpoint instead of hanging.

- langgraph_runtime() now records the checkpointer/Store config resolved from
  the same startup_config snapshot its checkpointer/store singletons are built
  from (app.state.checkpointer_config); /health/ready probes that snapshot and
  never re-resolves hot-reloaded config. resolve_checkpointer_config() returns
  None on resolution failure and the endpoint fails closed (503, checkpointer:
  unreachable) instead of reporting not_configured.

- The SQLite probe opens disk-backed paths with the non-creating mode=rw URI
  flag, so a missing database file stays missing and yields unreachable;
  in-memory forms (:memory:, file:...mode=memory) have nothing external to
  probe and report not_configured like the memory backend.

- Orchestrator timeouts now sit above the endpoint bound: Helm readinessProbe
  gains timeoutSeconds: 5 (Kubernetes default is 1s) and the docker-compose
  gateway healthcheck client timeout moves from 3s to 5s.

## Verification

- New regression tests: concurrent probes keep total elapsed time within one
  probe window; a probe ignoring its budget trips the endpoint deadline to 503;
  missing SQLite file stays absent and yields unreachable; in-memory SQLite
  forms map to not_configured; missing startup snapshot / config resolution
  failure fail closed to 503; resolve_checkpointer_config() raising is covered.

- cd backend && python -m pytest tests/test_gateway_health.py: 21 passed;
  tests/test_gateway_docs_toggle.py and lifespan/shutdown gateway suites pass.

- ruff check + ruff format clean on all changed files.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** implemented the reviewer-requested concurrency/deadline, startup-snapshot probing, fail-closed resolution, and non-creating SQLite probe; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

* fix(gateway): serialize connection-opening readiness probes behind a strict gate

## Why

Review on #5166 (willem-bd, P1): every request to /health/ready opened a new
PostgreSQL connection in _probe_postgres_backend, outside both the ORM pool and
the runtime checkpointer pool. The route is public through the /health auth
prefix and nginx proxies /health/*, so concurrent unauthenticated requests
could create an unbounded number of connections (each held for up to two
seconds), exhaust PostgreSQL max_connections, and take down both normal
traffic and the readiness probe itself.

## What changed

- backend/app/gateway/health.py: connection-opening checkpointer probes
  (sqlite connect, postgres AsyncConnection.connect) now run inside a strict
  per-process gate - an asyncio.Lock cached per running event loop - so at
  most one probe connection can be in flight per worker process. Requests
  that queue behind the gate are still shed by the existing endpoint-wide
  deadline, so a flood cannot pile up new connections or open files.

- Memory and unknown-backend decisions stay outside the gate; payload and
  probe semantics are unchanged. The serialization is documented in the
  module docstring and backend/app/gateway/AGENTS.md.

## Verification

- New regression test: 8 concurrent readiness_payload() requests against an
  instrumented sqlite probe assert the maximum number of in-flight probe
  connections is 1 while every request still returns 200.

- cd backend && python -m pytest tests/test_gateway_health.py: 22 passed;
  tests/test_gateway_docs_toggle.py and tests/test_gateway_lifespan_shutdown.py
  also pass on the merged main head.

- ruff check + ruff format clean on all changed files.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** implemented the reviewer-requested strict concurrency bound for the public readiness probe; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
2026-09-04 23:26:53 +08:00