* docs: list the summarize event tag and scope the extension example's claims
RUN_EVENT_STREAM.md listed seven middleware tags, but the catalog and the
run event stream contract carry eight; `summarize` is emitted by
SummarizationMiddleware for every successful summary result. Add the tag,
describe the event, and pin the documented list to MIDDLEWARE_EVENT_TAGS so
the next tag cannot drift silently. summarization.md now lists all eight
`changes` fields of that event.
The extension example said it showed all five (README, pyproject, module
docstrings, test name) or every contribution kind, while ExtensionRegistry
has eight. Say it demonstrates five, name the three it does not, point to
the observers guide and the bookmarks plugin, and assert in the example's
test that the two observer kinds stay unregistered.
* docs(changelog): reference #6582
* docs: clarify extension example package layout scope
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* perf(summarization): skip the summarizer LLM call for an input already proven no-op
When the exact rendered summarizer prompt (plus candidate models) previously
produced output byte-identical to the current summary_text, reuse that summary
instead of calling the LLM. Compaction proceeds through the normal path, so
messages, hooks and state transitions are unchanged.
- Only proven no-op inputs are cached (bounded LRU, thread-safe); failed
attempts are never cached and any cache error falls back to the LLM call.
- The middleware:summarize event gains changes.llm_call_skipped.
- Applies to both compact_state and acompact_state.
Builds on the per-call telemetry in the P0 branch.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(summarization): keep fallback no-ops retryable
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(goal): show the active goal to the agent on every model call
Nothing put ThreadState.goal into model requests. The agent only knew the
goal from messages: the objective the Web UI and IM channels send, and the
hidden <goal_continuation> message. Goals set through PUT /goal, the
embedded client or the TUI never reached it, and compaction could remove
the messages that stated the others.
DurableContextMiddleware now renders the active objective first in its
hidden durable-context data message, escaped, bounded, PII-redacted and
treated as data like the summary. Only the objective is rendered, so the
block changes only when the goal is set or cleared.
* fix(goal): let the agent pursue the active goal at user priority
The durable-context contract told the model never to follow instructions in
its data message, and that message now carried the goal. While a goal is
rendered, the contract ends with a static exception for the <active_goal>
element that opens the data message: work toward it as a request in a user
message, with no system or developer authority, while every other field
value, and any other text that calls itself a goal, stays data.
The element replaces the heading the goal used, which a summary could
reproduce. Other field values in the data message are HTML-escaped, and
active_goal joins the sanitizer denylist, so user input and remote tool
results cannot carry a raw one either.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix: rebuild todo reminders after context compaction
* chore: remove implementation plan from PR
* refactor: share todo reminder message name
* docs: keep agent guidance within CI size budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(summarization): resolve fraction triggers from declared context_window, degrade instead of crashing the agent build
A fraction trigger/keep clause requires profile["max_input_tokens"], which any
third-party OpenAI-compatible model lacks, so SummarizationMiddleware
construction raised ValueError out of create_summarization_middleware and failed
the whole agent build (#3103).
- factory: translate a declared model context_window into the langchain
profile (metadata-only, never reaches the provider payload); explicit
caller/override profiles win
- summarization factory: drop unusable fraction trigger clauses (absolute
clauses survive), fall a fraction keep back to the messages default, and
disable compaction with an actionable warning only when no usable trigger
clause remains — the agent build never dies from summarization config
- docs: config.example.yaml, ModelConfig.context_window, summarization.md
* refactor(summarization): share the default keep constant with the fraction fallback
The fraction-keep degradation fallback hardcoded ("messages", 20),
duplicating SummarizationConfig.keep's default_factory literal. Move the
value to a shared DEFAULT_KEEP constant so the two cannot drift apart.
* fix(summarization): keep trigger-null + fraction-keep constructing after degradation
A trigger of None with a fraction keep hit the all-clauses-dropped branch
(has_usable_trigger=False) and disabled compaction, and the accompanying
warning claimed configured triggers were all fraction-based when none were
configured. Only report nothing-usable when trigger clauses actually
existed; trigger:null keeps constructing the never-firing middleware with
the degraded keep, matching its behavior outside the degradation path.
* fix(summarization): address review — keep manual compaction, validate ContextSize, pin wiring
Review follow-ups on #4901:
- When every configured trigger is a dropped fraction clause, keep
constructing the never-firing middleware (trigger=None) instead of
returning None: manual /compact runs with force=True and never consults
trigger clauses, so it must keep working for a profile-less model
rather than reporting 'compaction is disabled'. The warning now says
auto-compaction will not fire while manual compaction remains.
- ContextSize gains a config-load validator: fraction values must be in
(0,1] (a percent-style 80 instead of 0.8 previously produced a threshold
the context could never reach — a silently inert trigger), absolute
values must be positive.
- New un-monkeypatched integration test pins the shipped wiring
(context_window declared -> real factory attaches profile -> fraction
clause survives -> middleware constructs), which the stubbed
middleware-side tests and kwarg-capturing factory-side tests each
stopped short of.
- Docs (summarization.md + config.example.yaml) clarify that the fraction
resolves against the summary/anchor model's context_window
(summarization.model_name when set, else the run model), including the
mismatch caveat for a larger-window summary model.
* fix(summarization): reject non-finite ContextSize values at config load
YAML .nan / .inf pass pydantic's float parsing, and nan <= 0 is False,
so the positivity check alone let them through as dead thresholds
(count >= nan is always False) — the same silent-inert-trigger class the
range validator was added to close. Guard with math.isfinite first,
consistent with the existing non-finite guards on mem0 timeout_seconds
and poll_after_seconds.
* fix(summarization): merge context_window into inferred profile, require whole message counts
- construct the model first, then merge max_input_tokens into the
provider-inferred langchain profile: passing profile= to the
constructor replaced the whole inferred metadata (tool_calling,
structured_output, io capabilities, output limits) with the single
key. An explicitly configured profile is still never clobbered.
- reject non-integral ContextSize values for type=messages at config
load: langchain slices the message list with them, so a float index
raised TypeError mid-compaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(summarization): own the run model for compaction; bound failure
With summarization.model_name: null the summary model resolved to
config.models[0] while the executing model is selected per run; when they
differ and models[0]'s provider is broken (expired key, quota, outage)
compaction silently failed every triggered turn and context grew unbounded
until the main provider 400s the run (#3103's shape), even though the run's
own model was healthy.
Model ownership is now sourced from the builders, not re-derived at runtime:
- The lead, subagent, and manual /compact builders each pass the resolved run
model into create_summarization_middleware(run_model_name=...). The middleware
no longer reads runtime.context / get_config(), which do not carry a custom
agent's or a subagent's resolved model, so a custom-agent lead run and a
distinct-model subagent now summarize with their own model, not models[0] /
the parent's. Runtime re-resolution and the per-name model cache are removed.
- model_name: null summarizes with the run's own model; an explicitly configured
summary model generates and falls back to the run model on failure. The
fallback is built lazily after the primary fails and its construction is
guarded, so a broken fallback cannot skip a healthy primary or escape the
automatic failure boundary.
Failure is bounded and side-effect-safe:
- An empty or whitespace-only response is treated as a generation failure, not a
valid summary, so compaction never removes all history for an empty replacement.
- compact_state/acompact_state take raise_on_failure independent of force: the
manual /compact path always surfaces a generation failure (even force=false)
and routes it to the existing ContextCompactionFailed path (HTTP 500 ->
frontend error toast) instead of an unconsumed response reason. The automatic
path leaves compaction state unchanged.
- before_summarization hooks fire only after a replacement summary exists.
SummarizationConfig.model_name, config.example.yaml, and docs/summarization.md
document the final lead/subagent/manual ownership rules.
Part of RFC #4346 (section A). Evaluating fraction/triggers against the run
model's profile (profile ownership) is a separate follow-up.
* fix(summarization): manual /compact model ownership + fail-open construct/parse
Manual /compact carried only agent_name, so it derived the run model from the
custom-agent model or config.models[0] and missed the request-selected model the
run path uses (request -> custom-agent -> default). Carry model_name through
ThreadCompactRequest and the frontend compact call, resolve with the same
precedence, and move the custom-agent config read off the event loop (asyncio
.to_thread) with user_id so the strict blocking-IO gate is not bypassed by the
broad except.
Make one summary attempt own its full lifecycle so the fail-open boundary covers
construction and response parsing, not just invocation: build each candidate model
lazily and guarded (a raising constructor falls through to the healthy run model
instead of breaking agent construction), build the model_name:null primary from the
run model rather than config.models[0], and run response text extraction inside the
invocation try so a failing .text accessor falls back instead of escaping compaction.
Adds factory-level constructor-failure, response-extraction-failure (sync/async), and
route-path model-ownership tests.
* feat: emit structured runtime metadata
* fix: avoid subagent import cycle in replay gateway
* fix: preserve legacy subtask result parsing
* refactor: tighten runtime metadata contracts
* fix(middleware): keep recovery hint on task exception wrapper content
The structured-metadata stamp overwrote the wrapper text with the bare
task-failure message, dropping the model-facing 'Continue with available
context, or choose an alternative tool.' guidance that every other tool
exception keeps. Append the shared hint after the formatted message.
* fix(subagents): require lowercase hex for result_sha256 reader
Length-only validation accepted any 64-char string; a faulty serializer
or relaying wrapper could store a non-digest value in the delegation
ledger. Enforce the producer's hexdigest shape with a fullmatch.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor: extract shared utils to break harness→app cross-layer imports
Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: split backend/src into harness (deerflow.*) and app (app.*)
Physically split the monolithic backend/src/ package into two layers:
- **Harness** (`packages/harness/deerflow/`): publishable agent framework
package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
models, MCP, skills, config, and all core infrastructure.
- **App** (`app/`): unpublished application code with import prefix `app.*`.
Contains gateway (FastAPI REST API) and channels (IM integrations).
Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure
Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: add harness→app boundary check test and update docs
Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.
Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add config versioning with auto-upgrade on startup
When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.
- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix comments
* fix: update src.* import in test_sandbox_tools_security to deerflow.*
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle empty config and search parent dirs for config.example.yaml
Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
looking next to config.yaml, fixing detection in common setups
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: correct skills root path depth and config_version type coercion
- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
after harness split, file lives at packages/harness/deerflow/skills/
so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
_check_config_version() to prevent TypeError when YAML stores value
as string (e.g. config_version: "1")
- tests: add regression tests for both fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update test imports from src.* to deerflow.*/app.* after harness refactor
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
