5 Commits

Author	SHA1	Message	Date
Vanzeren	095092418c	fix(gateway):unify thread id validation (#4589) ... * fix(gateway): unify thread ID validation at the API boundary Thread ID entry points accepted arbitrary strings while downstream consumers (filesystem paths, Kubernetes Provisioner, JSONL event store) each enforced different character restrictions, so invalid IDs were persisted first and only failed later during sandbox/workspace init. Centralize validation in deerflow.utils.thread_id (pattern ^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest, scheduler dispatch, paths.py, JSONL store, embedded client, and align the Provisioner pattern (pinned by a parity test). UUIDs are still generated only when no ID is supplied; caller-supplied opaque IDs stay supported. Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy cleanup escape hatch (filesystem cleanup guarded), read-only client.get_thread stays unvalidated, and scheduler rows with legacy invalid IDs record a failed dispatch instead of raising out of the poll loop. * docs: document canonical thread ID contract README: caller-supplied thread IDs need not be UUIDs; the canonical pattern and per-endpoint behavior. AGENTS.md: the shared deerflow.utils.thread_id contract, its enforcement boundaries, and the legacy-ID escape hatches. * fix(gateway): close thread ID validation gaps at remaining entry points Follow-up to the canonical thread ID contract: a full audit found the uniform-422 coverage only reached about half of the thread_id surfaces. - routers: 18 routes still took a bare thread_id: str — 13 in thread_runs.py (including the five messages/events/workspace-changes reads that returned 500 on the JSONL event store vs 404/empty on the DB store), 4 read routes in threads.py, and the suggestions route flagged in review. DELETE /api/threads/{id} keeps str as the declared legacy-cleanup escape hatch. - client: upload_files/delete_upload/list_uploads/get_artifact now validate up front, fulfilling the RFC's 'all mutating entry points' clause (get_thread stays unvalidated as the declared legacy read path). - tui: the /resume literal-ref fallback validates against the canonical contract and reports a descriptive error instead of failing deep in the client. - scripts/support_bundle.py: replace the drifted dot-allowing pattern with a byte-identical copy of THREAD_ID_PATTERN (kept local so the script still runs with a broken venv). * test(gateway): guard the canonical thread ID contract against regressions - test_thread_id_route_contract.py: static AST sweep asserting every route handler with a thread_id parameter annotates ThreadId (whitelist: the DELETE escape hatch), plus a runtime sweep hitting all 44 thread_id routes with a non-canonical ID and asserting a 422 that names thread_id, plus a websocket upgrade-rejection case. - test_thread_id_validation.py: client entry-point validation, support_bundle pattern parity, and TUI literal-ref fallback tests. - Align two tests that encoded the old contract (dotted IDs).	2026-08-01 19:42:44 +08:00
ShitK	4e44938551	fix: align pnpm consumers with Corepack fallback (#4405) ... * fix: align pnpm consumers with Corepack fallback * fix: run pnpm helper from frontend workspace * fix: preserve Corepack resolution hint	2026-07-29 08:08:33 +08:00
Vanzeren	42baed8c8c	feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel (#4292) ... * feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel Add a restart-required database.checkpoint_channel_mode ("full" default, "delta") that stores the messages channel via LangGraph 1.2 DeltaChannel, cutting checkpoint storage from O(n^2) to O(n) for append-only history. Existing full checkpoints seed delta state transparently; no data migration. - config: mode schema + freeze-on-first-use with CheckpointModeReconfigurationError; mode marker persisted in checkpoint metadata; unsafe delta->full downgrade rejected fail-closed with CheckpointModeMismatchError (run-level error, failed state read) - state: delta message state schema; CheckpointStateAccessor centralizes materialized reads for all consumers (threads API, branches, regeneration, compaction, state updates, memory, goal workers) - runtime: raw writers (run durations, interrupted title, thread goal) parent their checkpoints to the checkpoint they derive from, preserving delta ancestry; rollback forks the pre-run lineage through a state mutation graph with Overwrite restores; InMemorySaver delta-history override delegates to the base walk (fixes dropped first write after migration, also present upstream) - tests: conformance suite over {memory, sqlite, postgres} covering migration replay, stable message IDs, storage shape and writer preservation; conftest fixture isolates the frozen mode between tests; stale config fakes refreshed - ci: backend unit tests gain a postgres service * fix(checkpoint): close materialization gaps in goal flow, guard public factory - Route goal-continuation message reads through CheckpointStateAccessor: raw channel_values reads see the delta sentinel in delta mode, which disabled goal continuation (stand_down=no_durable_end_of_turn) after durable assistant turns. Raw tuples remain for tuple-only metadata (checkpoint id, pending_writes). - Reject checkpoint_channel_mode='delta' + checkpointer in create_deerflow_agent at construction: factory-built persisted graphs bypass mode-marker injection and the fail-closed gate, reproducing silent mixed-mode state loss. Delta without persistence stays allowed. - Import the postgres saver lazily (pytest.importorskip in the fixture) so the documented default install collects the suite; add a CI job running pytest --collect-only on uv sync --group dev without extras. - Fix test_checkpointer fallback test to patch get_app_config at its use site (provider module), making it deterministic when a local config.yaml selects a persistent backend. * fix(gateway): preserve extension-owned channels in state mutations, bump config version - build_state_mutation_graph / build_checkpoint_state_mutation_accessor accept an explicit state_schema; branch and POST /state now compile the mutation graph from the thread's effective schema (graph_state_schema on the assistant graph). The base-ThreadState fallback silently discarded channels contributed by custom AgentMiddleware.state_schema on branch (data loss) and returned a false-success 200 on POST /state. - POST /state validates values keys against the mutation graph's channels and rejects unknown fields with 422 instead of ignoring them; reducer detection covers extension channels (BinaryOperatorAggregate or DeltaChannel) so Overwrite replace semantics work for middleware reducers in both modes. - Endpoint regression: custom AgentMiddleware.state_schema value survives branch, updates through POST /state, and an unknown field receives 422. - config_version 26 -> 27 for the new database.checkpoint_channel_mode (example, Helm chart values + README, support-bundle fixture), so existing installs get the outdated-config warning and make config-upgrade merges the field; covered by a test driving the real example file and the real config-upgrade script. * fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite GET /threads/{id}/state now resolves the thread's assistant_id through a single reusable boundary (thread metadata -> assistant_id -> effective graph), so channels contributed by a custom AgentMiddleware.state_schema are materialized instead of dropped by the default lead schema. POST /state uses the same boundary instead of resolving the schema ad hoc. Branch writes wrap every copied reducer channel in Overwrite (derived from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel), not just messages, so already-aggregated values are never re-merged. Regression tests use a real AgentMiddleware.state_schema with a non-identity reducer in both full and delta modes: GET /state returns the extension value, POST /state replaces it, branch preserves it byte-for-byte; the unknown-field 422 is a separate assertion. * refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests Address review round 4 on PR #4292: - Push ahistory/history limit through Pregel into checkpointer.alist (SQL LIMIT) instead of materializing all rows and breaking in Python - Fold the read-side mode-compat gate onto the returned snapshot's metadata; only writes keep the pre-write tuple fetch (fail-closed) - Cache factory-built accessor graphs per (assistant_id, mode) with factory-identity revalidation; state reads no longer build a lead agent per request - get_thread: one snapshot fetch + one raw pending_writes fetch on the resolved checkpoint (post-checkpoint __error__ writes never surface in snapshot.tasks; verified empirically) - DeerFlowClient.get_thread: single checkpointer.list walk collects pending_writes per checkpoint instead of N get_tuple calls - InMemorySaver delta-history patch: stand-down when the upstream override disappears, try/except guard, validated-version warning, guard tests - make_lead_agent mode precedence: first freeze is owned by app_config (client-supplied configurable key ignored); once frozen, injected key/app_config must match or fail closed - Rollback: lock in non-message channel restoration via fork inheritance with a dedicated reducer-channel test - Add tests/test_threads_checkpoint_mode.py and tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and the PR validation section: lifecycle parity (memory + sqlite), per-step blob-count storage guard, gateway endpoint parity Counted-saver tests pin checkpoint round-trips for aget/ahistory so these regressions cannot silently return. * fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience - threads router: map CheckpointModeMismatchError to 409 (with cause and thread id) and CheckpointModeReconfigurationError to 503 across all state endpoints instead of swallowing both into a generic 500 - gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary through the real route stack - rollback: compile the restore mutation graph with the thread's effective state schema per the build_state_mutation_graph contract - inheritance contract locks: rollback and manual compaction preserve middleware-contributed channels via checkpoint fork cloning - services: revalidate the accessor-graph cache against app_config identity so config.yaml hot-reloads never serve a stale compiled graph - services: degrade full-mode state reads to raw checkpointer reads when the agent factory is unavailable (delta gate still applies; delta mode has no fallback) - deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility) and bump the langchain lower bound to what the lockfile actually resolves * fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread - _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while pregel's get_state_history treats config.checkpoint_id as the inclusive start; fetch the anchor explicitly so both read paths paginate identically - extend the degraded-path gateway test: GET /thread returns raw values, and POST /history with before starts at the anchor checkpoint * fix(gateway): preserve degraded checkpoint timestamps * fix(gateway): harden degraded checkpoint access * fix(gateway): resolve assistants for checkpoint reads --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-22 08:33:29 +08:00
Daoyuan Li	283cea567e	fix(scripts): broaden support bundle secret-key redaction denylist (#4242) ... * fix(scripts): broaden support bundle secret-key redaction denylist SECRET_KEY_RE only matched a fixed keyword allowlist, so a secret stored under an unanticipated key name inside an open-ended config dict (e.g. guardrails.provider.config, an arbitrary provider-kwargs dict) was emitted verbatim into config-summary.json even though manifest.json claims redacted_secret_fields=true. This gap was flagged on PR #3886's review before merge but not fully addressed. Broaden the key-name match to mirror env_policy.py's wildcard denylist (*KEY*/*SECRET*/*TOKEN*/*PASS*/*CREDENTIAL*/*DSN*) already used for sandbox env-scrubbing, plus its no-flag credential exact names (GH_PAT/GITHUB_PAT/ REDIS_AUTH/REDISCLI_AUTH/PGSERVICEFILE). The new bare key/pass/dsn alternatives are boundary-guarded so they match only their own delimited token, not an unrelated word that starts with the same letters (routing "keywords", guardrails "passport"). * fix(scripts): stop the pass token boundary from missing passphrase/passcode SECRET_KEY_RE's bare "pass" alternative, (?<![a-zA-Z])pass(?![a-zA-Z]), excludes any key where "pass" is followed by another letter. That correctly keeps "passport" out of the redaction set, but it also excludes genuine secret-bearing key names like "passphrase" and "passcode" -- both of which env_policy.py's *PASS* substring denylist does catch, so a secret stored under either name in an open-ended config dict (e.g. guardrails.provider.config) would still leak into config-summary.json. Narrow the lookahead to only exclude a trailing "port" -- pass(?!port) -- so passphrase/passcode/pass/db_pass all match while passport stays excluded; compass/bypass stay excluded via the existing leading-letter lookbehind, independent of the lookahead. Added a regression test covering both the newly-caught names and the still-excluded ones in one place. Reverting to the old lookahead reproduces the exact leak (passphrase left unredacted); with the fix, tests/test_support_bundle.py (30 tests) is green, and ruff check/format are clean.	2026-07-19 07:39:52 +08:00
Ryker_Feng	cf02646489	feat(scripts): add redacted community support bundle generator (#3886) ... * feat(scripts): add redacted community support bundle generator Add `make support-bundle` (scripts/support_bundle.py) to help users file high-signal, privacy-safe GitHub issues for local setup/config/runtime problems. The command produces: - `*-issue-summary.md` to paste into the issue body - `*-issue-draft.md` scaffold for AI-assisted filing (REQUIRED placeholders, never invents repro/expected/summary facts) - an optional evidence zip under `.deer-flow/support-bundles/` containing a stable `triage.json` plus redacted environment/config/extensions/git/doctor evidence Privacy: secrets are redacted across config values, URL userinfo, query strings, CLI flags, custom headers, bearer/sk- tokens, and home paths. The bundle never includes `.env`, raw conversation messages, or user file contents; optional `--thread-id` adds file manifests only. `thread_id` input is validated against path traversal. Wire it into the Makefile, AGENTS.md, README/README_zh, CONTRIBUTING, and the bug-report issue template. Covered by backend/tests/test_support_bundle.py. * fix(scripts): redact MCP env values by default in support bundle Address PR #3886 review (willem-bd, P2): the key-name allowlist let literal secrets under non-standard env keys (e.g. SUPABASE_SERVICE_ROLE_KEY, R2_ACCESS_KEY, hardcoded AIza… keys) leak verbatim into the bundle that users are told is safe to share publicly. Mask all MCP `env` values by default, keeping only `$VAR`/`${VAR}` references visible, and broaden SECRET_KEY_RE (access_key, pwd, private_key). Add tests for non-keyword env secrets, broadened key names, and end-to-end zip redaction.	2026-07-01 22:30:55 +08:00