* implement goal continuations
* fix(goal): address review findings for goal continuations
- goal: key the no-progress breaker on a signature of the latest visible
assistant evidence instead of the evaluator's volatile free-text, so it
actually fires on stalled turns; thread the signature through every
worker persist / no-progress call site
- goal: align _stand_down_reason default caps with should_continue_goal
(8 / 2) so the two gate functions agree on goals missing the fields
- runtime: offload the synchronous checkpointer fallback via
asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop
- frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types)
- frontend: extract pure composer helpers into input-box-helpers.ts with
unit tests (parseGoalCommand, readGoalResponseError, skill suggestions)
- tests: cover the evidence-based no-progress and default-cap behavior
- docs: align backend/AGENTS.md goal paragraph with actual behavior
- e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): hide goal continuation counter until the agent continues
The goal status bar rendered a raw "0/8" before any auto-continuation, which
read as a mysterious score. Now the counter is hidden until
continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining
the auto-continuation cap.
- Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests
- Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(goal): address review findings for goal continuations
Frontend correctness
- Fix the optimistic /goal result permanently shadowing server goal state:
the streamed continuation counter never surfaced for a goal set in-session.
Extract a shared useActiveGoal hook (used by both chat pages) that reconciles
the optimistic copy with server state via a goalReconciliationKey, de-duping
the copy-pasted goal block across the two pages.
- Stop /goal status|clear failures from escaping handleSubmit as unhandled
rejections (handleGoalCommand now returns success; the run only starts when a
goal was actually saved).
- Use a function replacer for the goal-status toast so an objective containing
$&/$1 isn't treated as a replacement pattern.
Backend cleanliness / correctness
- De-duplicate four byte-identical helpers (_call_checkpointer_method,
_message_type, _additional_kwargs, _is_visible_message) by importing them
from runtime.goal instead of re-defining them in the run worker.
- Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has
no tasks field) and document that pending_writes is the durability signal.
- Decompose the 176-line _prepare_goal_continuation_input: extract
_reread_goal_and_checkpoint and a _persist closure so the thread-unchanged
guard and stand-down persistence aren't open-coded three times. Document the
last-writer-wins write-window limitation as a follow-up.
- Add a shared parse_goal_command helper and use it from the TUI and IM-channel
/goal handlers (one place for the status/clear/set semantics).
Tests
- Restore the 11 command-registry tests dropped by the previous goal change
(filter_commands ranking/description, build_registry builtins/skills, resolve
cases) alongside the new goal tests.
- Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal
handler, parse_goal_command, and goalReconciliationKey.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix goal review feedback
* fix goal continuation checkpoint races
* prioritize goal commands while streaming
Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run.
Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check
* preserve goal status during clarification
Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint.
Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* style: format active goal hook
Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate.
Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* fix goal review followups
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(sandbox): per-call env injection + platform-secret scrubbing for skills
Add an env parameter to Sandbox.execute_command (abstract + local + AIO) so request-scoped secrets can be injected into skill subprocesses, and scrub platform credentials (*KEY*/*SECRET*/*TOKEN*/*PASSWORD*/*CREDENTIAL*) from the inherited environment by default so scoped injection is not security theatre. LocalSandbox always passes an explicit scrubbed env; AioSandbox routes env-bearing commands through bash.exec(env=) on a fresh session and leaves the legacy persistent-shell path unchanged. Part of #3861. BEHAVIOR CHANGE: execute_command no longer inherits the full os.environ; Windows encoding tests updated to assert the scrubbed dict.
* feat(skills): parse required-secrets frontmatter declaration
Add SecretRequirement and Skill.required_secrets, and parse the required-secrets SKILL.md frontmatter field (a string list or {name, optional} mappings), dropping malformed entries with a warning so one bad declaration does not invalidate the skill. The declared name is both the context.secrets key and the env var injected at activation. Part of #3861.
* feat(runtime): request-scoped secret carrier (context.secrets)
Add SECRETS_CONTEXT_KEY + extract_request_secrets, centralising the context.secrets carrier contract. The existing context passthrough (build_run_config -> _build_runtime_context) already carries the sub-key to runtime.context without mirroring it into configurable; characterization tests lock that behaviour. Part of #3861.
* feat(skills): inject declared secrets at slash-activation into bash env
Binding point A: when a skill is slash-activated, SkillActivationMiddleware resolves its declared required-secrets against the request's context.secrets and writes the per-run injection set to runtime.context. The bash tool forwards that set to execute_command(env=). A skill cannot harvest a host platform credential (is_host_platform_secret guard, cf. GHSA-rhgp-j443-p4rf), and injected values are redacted from bash output (mask_secret_values) so an echoed secret never re-enters the prompt/trace. Part of #3861.
* test(skills): lock the five secret leak surfaces + add trace redaction helper
Regression tests assert the secret value is absent from all five surfaces: prompt (activation message), checkpoint (graph state vs context separation), audit (journal records names only), trace (metadata builder never copies context; never mirrored to configurable), and stdout (mask_secret_values). Add redact_secret_context_keys as a defensive helper for any context serialization. Part of #3861.
* docs(backend): document request-scoped secrets for skills
Add Request-Scoped Secrets subsection (Skills) + env policy note (Sandbox) and the execute_command(env=) signature change, per the doc-sync policy. Part of #3861.
* fix(skills): close gaps found by end-to-end verification of request-scoped secrets
Real-gateway e2e + independent review of #3861 surfaced three defects, now fixed:
1. Slash activation never fired in the live chain. InputSanitizationMiddleware
wraps user input in BEGIN/END markers before SkillActivationMiddleware sees it,
and the original text was only preserved when an upload or IM channel set it.
For a plain text message the slash command became undetectable, so no secret
was ever resolved. Fix: the sanitizer now setdefaults the pre-wrap text into
ORIGINAL_USER_CONTENT_KEY (additive; sanitization behaviour unchanged), so
slash activation works for all messages. Pre-existing latent bug surfaced here.
2. The raw request config (with context.secrets) was persisted to runs.kwargs_json
and echoed by the run API (RunResponse.kwargs). Fix: redact_config_secrets()
strips secret-bearing context keys from the persisted/echoed copy in start_run;
the live config that drives the run keeps them. build_run_config now also sets
configurable.thread_id on the context path (the checkpointer requires it).
3. Connection-string credentials (DATABASE_URL, REDIS_URL, SENTRY_DSN, GH_PAT, ...)
were not scrubbed from the inherited sandbox env. Fix: env_policy adds a *DSN*
pattern plus an explicit connection-string denylist (no blanket *URL* — benign
service URLs stay readable).
Verified end-to-end via a real gateway run (real LLM + skill activation + bash):
the secret reaches the sandbox subprocess and appears in NONE of prompt, trace,
checkpoint, audit, stdout, runs.kwargs_json, or the run API. Part of #3861.
* docs(backend): document the env scrub, persistence redaction, and sanitizer interaction
Sync the Request-Scoped Secrets section with the verification-driven fixes: inherited-env scrub (incl. connection-string denylist), run-record/run-API redaction as the 6th sealed leak surface, and the sanitizer preserving original content so slash activation fires. Part of #3861.
* fix(skills): inject caller secret over scrubbed host value; drop redundant host-name guard
A real-world demo (a skill calling a third-party cloud API with a request-scoped
key) exposed that the is_host_platform_secret guard was both wrong and harmful:
it refused to inject a caller-supplied secret whenever a same-named variable
existed in the Gateway env — which is exactly the #3861 use case (a per-user key
overriding a shared platform key). The guard was also redundant: build_sandbox_env
already scrubs secret-looking names from the inherited env before injection, so a
skill can never read a host credential — it only ever receives the caller's value.
Remove the guard; the injected (caller) value simply wins over the scrubbed host
value. Verified end-to-end: the agent called the real cloud API successfully with
the caller's key, the host's same-named key was scrubbed and never used, and the
caller's key leaked to none of the surfaces. Part of #3861.
* fix(skills): address review on request-scoped secrets (#3861)
Review fixes from PR #3871:
- E2BSandbox.execute_command now accepts env/timeout and routes them to
commands.run(envs=, timeout=). The bash tool passes env= unconditionally,
so the prior signature (command only) raised TypeError on every e2b bash
call and broke e2b deployments entirely. env=None stays backward-compatible.
- SkillActivationMiddleware clears the active-secret set before resolving each
activation, so a later skill in the same run never inherits an earlier
skill's injection set (the #3861 contract: a skill only receives what the
caller supplied AND that skill declared).
- AioSandbox env path uses a dedicated _DEFAULT_HARD_TIMEOUT — bash.exec exposes
no idle/no-change timeout, so the prior reuse of the legacy idle constant
conflated wall-clock vs idle semantics. The env path also retries on the
ErrorObservation signature now, sharing the legacy persistent-shell recovery
contract.
- mask_secret_values skips values below a minimum length floor so a short
declared secret (e.g. "42") cannot shred unrelated bytes (exit codes,
timestamps, sizes) of tool output. The secret is still injected into the
subprocess; only the output mask skips it.
session_id reuse on the env path is intentionally NOT added: a shared session
could let request-scoped secrets ride the session env into later commands,
which the SDK does not contractually forbid. The fresh-session choice matches
the LocalSandbox model (each call is a fresh subprocess); the trade-off
(consecutive env-bearing calls do not share cwd/venv/exports) is documented on
_execute_with_env.
get_or_new_skill_storage() and reset_skill_storage() touch the
process-global skill storage singleton without a lock - the same
unsynchronized check-then-create that #3730 just fixed in
sandbox_provider.py, the module this file documents itself as mirroring.
Two callers racing a cold start can both see _default_skill_storage is
None and each build a SkillStorage, so the second overwrites the first;
a reset_skill_storage() racing a get can also null the global between
the None-check and the return.
Guard the build/return and the reset with a module-level threading.Lock
and a double check, mirroring get_memory_storage(). Construction stays
inside the lock (rather than sandbox_provider's build-outside-then-
discard-the-loser) because SkillStorage has no teardown hook, so a
losing racer's orphan could not be cleaned up.
Add backend/tests/test_skill_storage_lifecycle.py with concurrency
regression tests (8-thread cold-start race asserting a single instance;
reset racing gets asserting no None is returned).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(skills): keep skill archive installation off the event loop
ainstall_skill_from_archive — the async entry point awaited by the gateway
POST /skills/install route — ran its entire filesystem pipeline inline on
the event loop: zip extraction, frontmatter validation, rglob enumeration,
per-file read_text, shutil.copytree staging, and tempdir cleanup.
Restructure into offloaded phases: prepare (extract + validate) and commit
(stage + move) run via asyncio.to_thread, the tempdir lifecycle is
offloaded, and the security scanner's file enumeration and reads move off
the loop — only the per-file LLM scan (genuinely async) stays awaited.
Security decision logic and exception contract are unchanged.
Anchor: tests/blocking_io/test_skills_install.py drives the real install
pipeline (real .skill archive, real FS; only scan_skill_content stubbed)
under the strict Blockbuster gate. Verified red on pre-fix code
(BlockingError: os.stat), green with the fix.
* fix(skills): log temp-dir cleanup failures instead of swallowing them
Review follow-up on the install offload: rmtree(ignore_errors=True) kept
the primary install exception but silently leaked the extraction dir on
cleanup failure. Keep the never-mask behaviour, add a warning log.
* fix(skills): bound install tmp cleanup and pass skill_dir explicitly (review)
- Wrap the best-effort temp-dir cleanup in asyncio.wait_for (5s) so a
hung filesystem in the finally block cannot stall or mask the install
outcome; timeout is logged like the existing OSError path.
- Hoist _collect_scannable_files to module level with skill_dir as an
explicit argument instead of a closure capture.
* fix(skills): surface offending line and quoting hint on SKILL.md YAML errors
When a SKILL.md front-matter fails to parse, the existing log only
echoes PyYAML's raw message, leaving authors to grep the file for the
offending line. This is especially painful for the very common
LLM-authored mistake of an unquoted scalar containing ': '
(e.g. 'description: foo: bar'), which fails with
'mapping values are not allowed here' and silently drops the skill.
Enrich the error log with:
- the source line PyYAML pointed at via problem_mark
- a targeted, copy-pasteable quoting hint when (and only when) the
error is the well-known 'mapping values are not allowed' scanner
error on an unquoted value
The skill is still rejected (no semantics are guessed or rewritten);
only the diagnostic is improved.
Fixes#3333
* improve(skills): address CR feedback on SKILL.md YAML error diagnostics
Per review on #3335:
- Log the file line number (mark.line + 2) instead of the
front-matter-internal line number, so authors land on the right
row in their editor.
- Use exc.problem == "mapping values are not allowed here" for a
tighter match than substring-scanning str(exc).
- Preserve the offending key's leading whitespace in the quoting
hint so nested mappings stay nested when authors paste the fix
back.
- Rewrite the regression test to actually exercise the new
behaviour: PyYAML's own message already echoes the offending
line (and truncates it with "..."), so the old assertion
passed on main. New assertions pin (a) the file-line number,
(b) the full untruncated line, and (c) the copy-pasteable hint.
- Add a guard test for nested-key indentation so the
partition()/strip() shape cannot regress silently.
Refs #3333, #3335
* fix(skills): escape backslashes in YAML quoting hint
The hint emitted by _format_yaml_error previously escaped only double
quotes, so values containing backslashes (e.g. Windows paths like
C:\Temp or regex escapes like \d) produced a suggested scalar that
was either invalid YAML or silently re-interpreted by PyYAML's
double-quoted escape rules when pasted back. Escape order matters:
backslashes first, then double quotes.
Adds two regression tests covering Windows-path and regex-style
backslashes.
Address Copilot CR feedback on PR #3335.
The moderation model's response was silently falling through to a
conservative block when LLMs wrapped structured output in markdown
code fences, added prose around the JSON, returned case-variant
decisions (e.g. "Allow"), or included nested braces in the reason
field. The greedy `\{.*\}` regex also over-matched on nested braces.
- Rewrite _extract_json_object() with markdown fence stripping and
brace-balanced string-aware extraction
- Normalize decision field to lowercase for case-insensitive matching
- Distinguish "model unavailable" from "unparseable output" in fallback
- Strengthen system prompt to explicitly forbid code fences and prose
- Add 15 tests covering all reported scenarios
Fixes#2985
* fix(harness): resolve runtime paths from project root
* docs(config): update
* fix(config): address runtime path review feedback
* test(config): fix skills path e2e root
* test(config): cover legacy config fallback when project root lacks config files
Verifies that when DEER_FLOW_PROJECT_ROOT is unset and cwd has no
config.yaml/extensions_config.json, AppConfig and ExtensionsConfig fall back
to the legacy backend/repo-root candidates — the backward-compat path
requested in PR #2642 review.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(trace): Add `run_name` to the trace info for suggestions and memory.
before(in langsmith):
CodexChatModel
CodexChatModel
lead_agent
after:
suggest_agent
memory_agent
lead_agent
feat(trace): Add `run_name` to the trace info for suggestions and memory.
before(in langsmith):
CodexChatModel
CodexChatModel
lead_agent
after:
suggest_agent
memory_agent
lead_agent
* feat(trace): Add `run_name` to the trace info for system agents.
before(in langsmith):
CodexChatModel
CodexChatModel
CodexChatModel
CodexChatModel
lead_agent
after:
suggest_agent
title_agent
security_agent
memory_agent
lead_agent
* chore(code format):code format
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Refactor tests for SKILL.md parser
Updated tests for SKILL.md parser to handle quoted names and descriptions correctly. Added new tests for parsing plain and single-quoted names, and ensured multi-line descriptions are processed properly.
* Implement tool name validation and deduplication
Add tool name mismatch warning and deduplication logic
* Refactor skill file parsing and error handling
* Add tests for tool name deduplication
Added tests for tool name deduplication in get_available_tools(). Ensured that duplicates are not returned, the first occurrence is kept, and warnings are logged for skipped duplicates.
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update minimal config to include tools list
* Update test for nonexistent skill file
Ensure the test for nonexistent files checks for None.
* Refactor tool loading and add skill management support
Refactor tool loading logic to include skill management tools based on configuration and clean up comments.
* Enhance code comments for tool loading logic
Added comments to clarify the purpose of various code sections related to tool loading and configuration.
* Fix assertion for duplicate tool name warning
* Fix indentation issues in tools.py
* Fix the lint error of test_tool_deduplication
* Fix the lint error of tools.py
* Fix the lint error
* Fix the lint error
* make format
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Replace all bare print() calls with proper logging using Python's
standard logging module across the deerflow harness package.
Changes across 8 files (16 print statements replaced):
- agents/middlewares/clarification_middleware.py: use logger.info/debug
- agents/middlewares/memory_middleware.py: use logger.debug
- agents/middlewares/thread_data_middleware.py: use logger.debug
- agents/middlewares/view_image_middleware.py: use logger.debug
- agents/memory/queue.py: use logger.info/debug/warning/error
- agents/lead_agent/prompt.py: use logger.error
- skills/loader.py: use logger.warning
- skills/parser.py: use logger.error
Each file follows the established codebase convention:
import logging
logger = logging.getLogger(__name__)
Log levels chosen based on message semantics:
- debug: routine operational details (directory creation, timer resets)
- info: significant state changes (memory queued, updates processed)
- warning: recoverable issues (config load failures, skipped updates)
- error: unexpected failures (parsing errors, memory update errors)
Note: client.py is intentionally excluded as it uses print() for
CLI output, which is the correct behavior for a command-line client.
Co-authored-by: moose-lab <moose-lab@users.noreply.github.com>
* Fix Windows backend test compatibility
* Preserve ACP path style on Windows
* Fix installer import ordering
* Address review comments for Windows fixes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor: extract shared skill installer and upload manager to harness
Move duplicated business logic from Gateway routers and Client into
shared harness modules, eliminating code duplication.
New shared modules:
- deerflow.skills.installer: 6 functions (zip security, extraction, install)
- deerflow.uploads.manager: 7 functions (normalize, deduplicate, validate,
list, delete, get_uploads_dir, ensure_uploads_dir)
Key improvements:
- SkillAlreadyExistsError replaces stringly-typed 409 status routing
- normalize_filename rejects backslash-containing filenames
- Read paths (list/delete) no longer mkdir via get_uploads_dir
- Write paths use ensure_uploads_dir for explicit directory creation
- list_files_in_dir does stat inside scandir context (no re-stat)
- install_skill_from_archive uses single is_file() check (one syscall)
- Fix agent config key not reset on update_mcp_config/update_skill
Tests: 42 new (22 installer + 20 upload manager) + client hardening
* refactor: centralize upload URL construction and clean up installer
- Extract upload_virtual_path(), upload_artifact_url(), enrich_file_listing()
into shared manager.py, eliminating 6 duplicated URL constructions across
Gateway router and Client
- Derive all upload URLs from VIRTUAL_PATH_PREFIX constant instead of
hardcoded "mnt/user-data/uploads" strings
- Eliminate TOCTOU pre-checks and double file read in installer — single
ZipFile() open with exception handling replaces is_file() + is_zipfile()
+ ZipFile() sequence
- Add missing re-exports: ensure_uploads_dir in uploads/__init__.py,
SkillAlreadyExistsError in skills/__init__.py
- Remove redundant .lower() on already-lowercase CONVERTIBLE_EXTENSIONS
- Hoist sandbox_uploads_dir(thread_id) before loop in uploads router
* fix: add input validation for thread_id and filename length
- Reject thread_id containing unsafe filesystem characters (only allow
alphanumeric, hyphens, underscores, dots) — prevents 500 on inputs
like <script> or shell metacharacters
- Reject filenames longer than 255 bytes (OS limit) in normalize_filename
- Gateway upload router maps ValueError to 400 for invalid thread_id
* fix: address PR review — symlink safety, input validation coverage, error ordering
- list_files_in_dir: use follow_symlinks=False to prevent symlink metadata
leakage; check is_dir() instead of exists() for non-directory paths
- install_skill_from_archive: restore is_file() pre-check before extension
validation so error messages match the documented exception contract
- validate_thread_id: move from ensure_uploads_dir to get_uploads_dir so
all entry points (upload/list/delete) are protected
- delete_uploaded_file: catch ValueError from thread_id validation (was 500)
- requires_llm marker: also skip when OPENAI_API_KEY is unset
- e2e fixture: update TitleMiddleware exclusion comment (kept filtering —
middleware triggers extra LLM calls that add non-determinism to tests)
* chore: revert uv.lock to main — no dependency changes in this PR
* fix: use monkeypatch for global config in e2e fixture to prevent test pollution
The e2e_env fixture was calling set_title_config() and
set_summarization_config() directly, which mutated global singletons
without automatic cleanup. When pytest ran test_client_e2e.py before
test_title_middleware_core_logic.py, the leaked enabled=False caused
5 title tests to fail in CI.
Switched to monkeypatch.setattr on the module-level private variables
so pytest restores the originals after each test.
* fix: address code review — URL encoding, API consistency, test isolation
- upload_artifact_url: percent-encode filename to handle spaces/#/?
- deduplicate_filename: mutate seen set in place (caller no longer
needs manual .add() — less error-prone API)
- list_files_in_dir: document that size is int, enrich stringifies
- e2e fixture: monkeypatch _app_config instead of set_app_config()
to prevent global singleton pollution (same pattern as title/summarization fix)
- _make_e2e_config: read LLM connection details from env vars so
external contributors can override defaults
- Update tests to match new deduplicate_filename contract
* docs: rewrite RFC in English and add alternatives/breaking changes sections
* fix: address code review feedback on PR #1202
- Rename deduplicate_filename to claim_unique_filename to make
the in-place set mutation explicit in the function name
- Replace PermissionError with PathTraversalError(ValueError) for
path traversal detection — malformed input is 400, not 403
* fix: set _app_config_is_custom in e2e test fixture to prevent config.yaml lookup in CI
---------
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
os.walk() does not follow symbolic links by default. This means
custom skills installed as symlinks in skills/custom/ are discovered
as directories but never descended into, so their SKILL.md files
are never found and the skills silently fail to load.
Adding followlinks=True fixes this for users who symlink skill
directories from external projects into the custom skills folder.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor: extract shared utils to break harness→app cross-layer imports
Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: split backend/src into harness (deerflow.*) and app (app.*)
Physically split the monolithic backend/src/ package into two layers:
- **Harness** (`packages/harness/deerflow/`): publishable agent framework
package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
models, MCP, skills, config, and all core infrastructure.
- **App** (`app/`): unpublished application code with import prefix `app.*`.
Contains gateway (FastAPI REST API) and channels (IM integrations).
Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure
Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: add harness→app boundary check test and update docs
Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.
Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add config versioning with auto-upgrade on startup
When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.
- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix comments
* fix: update src.* import in test_sandbox_tools_security to deerflow.*
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle empty config and search parent dirs for config.example.yaml
Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
looking next to config.yaml, fixing detection in common setups
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: correct skills root path depth and config_version type coercion
- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
after harness split, file lives at packages/harness/deerflow/skills/
so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
_check_config_version() to prevent TypeError when YAML stores value
as string (e.g. config_version: "1")
- tests: add regression tests for both fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update test imports from src.* to deerflow.*/app.* after harness refactor
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
