mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-15 15:50:35 +00:00
Compare commits
2 Commits
main
...
v2.2.0-rc0
| Author | SHA1 | Date | |
|---|---|---|---|
|
573968a78d | ||
|
|
c133f5e0da |
@ -62,9 +62,6 @@ docker/
|
|||||||
skills/
|
skills/
|
||||||
frontend/.next
|
frontend/.next
|
||||||
frontend/node_modules
|
frontend/node_modules
|
||||||
# Frontend unit tests can import fixtures outside frontend/ (for example, examples/),
|
|
||||||
# so exclude them from the frontend-only Docker build context.
|
|
||||||
frontend/tests
|
|
||||||
backend/.venv
|
backend/.venv
|
||||||
backend/htmlcov
|
backend/htmlcov
|
||||||
backend/.coverage
|
backend/.coverage
|
||||||
@ -76,23 +73,6 @@ backend/.coverage
|
|||||||
# root that the build client cannot read.
|
# root that the build client cannot read.
|
||||||
**/.deer-flow/
|
**/.deer-flow/
|
||||||
backend/sandbox/
|
backend/sandbox/
|
||||||
|
|
||||||
# Local configuration holds model API keys, MCP credentials, frontend secrets and
|
|
||||||
# local HTTPS keys (frontend/*.pem, the frontend .gitignore's mkcert pair).
|
|
||||||
# Compose and Helm supply it at runtime. The config resolvers still read the legacy
|
|
||||||
# backend/ copies, and `COPY backend ./backend` / `COPY frontend ./frontend` would
|
|
||||||
# otherwise bake them into every image layer built from this checkout.
|
|
||||||
config.yaml
|
|
||||||
config.yaml.bak
|
|
||||||
extensions_config.json
|
|
||||||
mcp_config.json
|
|
||||||
backend/config.yaml
|
|
||||||
backend/config.yaml.bak
|
|
||||||
backend/extensions_config.json
|
|
||||||
backend/mcp_config.json
|
|
||||||
**/.env*.local
|
|
||||||
frontend/*.pem
|
|
||||||
|
|
||||||
*.md
|
*.md
|
||||||
!README.md
|
!README.md
|
||||||
!frontend/README.md
|
!frontend/README.md
|
||||||
@ -104,17 +84,3 @@ frontend/*.pem
|
|||||||
!backend/extensions/
|
!backend/extensions/
|
||||||
!backend/extensions/sources/
|
!backend/extensions/sources/
|
||||||
!backend/extensions/sources/**
|
!backend/extensions/sources/**
|
||||||
|
|
||||||
# The re-inclusion above wins over every earlier rule, so the files the extension
|
|
||||||
# manager refuses to snapshot (deerflow/extensions/manager.py) are excluded again
|
|
||||||
# here. A snapshot the manager wrote never contains them; one edited by hand
|
|
||||||
# must not carry credentials into the image either.
|
|
||||||
backend/extensions/sources/**/.env
|
|
||||||
backend/extensions/sources/**/.env.*
|
|
||||||
backend/extensions/sources/**/.npmrc
|
|
||||||
backend/extensions/sources/**/.pypirc
|
|
||||||
backend/extensions/sources/**/credentials.json
|
|
||||||
backend/extensions/sources/**/*.key
|
|
||||||
backend/extensions/sources/**/*.pem
|
|
||||||
backend/extensions/sources/**/*.p12
|
|
||||||
backend/extensions/sources/**/*.pfx
|
|
||||||
|
|||||||
25
.env.example
25
.env.example
@ -1,40 +1,37 @@
|
|||||||
# Serper API Key (Google Search) - https://serper.dev
|
# Serper API Key (Google Search) - https://serper.dev
|
||||||
# SERPER_API_KEY=your-serper-api-key
|
SERPER_API_KEY=your-serper-api-key
|
||||||
|
|
||||||
# Optional operator-controlled Serper-compatible API base URL, without /search or /images.
|
# Optional operator-controlled Serper-compatible API base URL, without /search or /images.
|
||||||
# The configured API key is sent in X-API-KEY to this host; use a trusted provider.
|
# The configured API key is sent in X-API-KEY to this host; use a trusted provider.
|
||||||
SERPER_BASE_URL=https://google.serper.dev
|
SERPER_BASE_URL=https://google.serper.dev
|
||||||
|
|
||||||
# Exa API Key - https://exa.ai
|
# Exa API Key - https://exa.ai
|
||||||
# EXA_API_KEY=your-exa-api-key
|
EXA_API_KEY=your-exa-api-key
|
||||||
|
|
||||||
# Optional operator-controlled Exa-compatible API base URL for the Exa web_search and web_fetch tools.
|
# Optional operator-controlled Exa-compatible API base URL for the Exa web_search and web_fetch tools.
|
||||||
# The configured API key is sent in x-api-key to this host; use a trusted provider.
|
# The configured API key is sent in x-api-key to this host; use a trusted provider.
|
||||||
EXA_BASE_URL=https://api.exa.ai
|
EXA_BASE_URL=https://api.exa.ai
|
||||||
|
|
||||||
# Serply API Key (Google Search, News and Scholar) - https://serply.io
|
# Serply API Key (Google Search, News and Scholar) - https://serply.io
|
||||||
# SERPLY_API_KEY=your-serply-api-key
|
SERPLY_API_KEY=your-serply-api-key
|
||||||
|
|
||||||
# TAVILY API Key
|
# TAVILY API Key
|
||||||
# TAVILY_API_KEY=your-tavily-api-key
|
TAVILY_API_KEY=your-tavily-api-key
|
||||||
|
|
||||||
# Optional Webz.io News Search key (select Webz in make setup)
|
# Optional Webz.io News Search key (select Webz in make setup)
|
||||||
# WEBZ_API_KEY=your-webz-api-key
|
# WEBZ_API_KEY=your-webz-api-key
|
||||||
|
|
||||||
# Jina API Key (optional; the Jina Reader web_fetch works without one at a lower rate limit)
|
# Jina API Key
|
||||||
# JINA_API_KEY=your-jina-api-key
|
JINA_API_KEY=your-jina-api-key
|
||||||
|
|
||||||
# InfoQuest API Key
|
# InfoQuest API Key
|
||||||
# INFOQUEST_API_KEY=your-infoquest-api-key
|
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||||
|
|
||||||
# Sofya API Key (web search and fetch) - https://sofya.co
|
# Sofya API Key (web search and fetch) - https://sofya.co
|
||||||
# SOFYA_API_KEY=your-sofya-api-key
|
SOFYA_API_KEY=your-sofya-api-key
|
||||||
|
|
||||||
# SpecStream API Key (web search and fetch) - https://specstream.ai
|
|
||||||
# SPECSTREAM_API_KEY=your-specstream-api-key
|
|
||||||
|
|
||||||
# Unbrowse API Key (web fetch) - https://unbrowse.ai/app
|
# Unbrowse API Key (web fetch) - https://unbrowse.ai/app
|
||||||
# UNBROWSE_API_KEY=your-unbrowse-api-key
|
UNBROWSE_API_KEY=your-unbrowse-api-key
|
||||||
|
|
||||||
# Browser CORS allowlist for split-origin or port-forwarded deployments (comma-separated exact origins).
|
# Browser CORS allowlist for split-origin or port-forwarded deployments (comma-separated exact origins).
|
||||||
# Leave unset when using the unified nginx endpoint, e.g. http://localhost:2026.
|
# Leave unset when using the unified nginx endpoint, e.g. http://localhost:2026.
|
||||||
@ -75,10 +72,6 @@ EXA_BASE_URL=https://api.exa.ai
|
|||||||
# DEEPSEEK_API_KEY=your-deepseek-api-key
|
# DEEPSEEK_API_KEY=your-deepseek-api-key
|
||||||
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
|
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
|
||||||
# MINIMAX_API_KEY=your-minimax-api-key # OpenAI-compatible, see https://platform.minimax.io
|
# MINIMAX_API_KEY=your-minimax-api-key # OpenAI-compatible, see https://platform.minimax.io
|
||||||
# MiniMax API host (choose by region):
|
|
||||||
# China: https://api.minimaxi.com (default)
|
|
||||||
# International: https://api.minimax.io
|
|
||||||
# MINIMAX_API_HOST=https://api.minimax.io
|
|
||||||
# OPPER_API_KEY=your-opper-api-key # OpenAI-compatible, see https://opper.ai
|
# OPPER_API_KEY=your-opper-api-key # OpenAI-compatible, see https://opper.ai
|
||||||
# OpenAI-compatible Images API (used by the image-generation skill)
|
# OpenAI-compatible Images API (used by the image-generation skill)
|
||||||
# IMAGE_GENERATION_PROVIDER=openai # openai-compatible is also accepted
|
# IMAGE_GENERATION_PROVIDER=openai # openai-compatible is also accepted
|
||||||
|
|||||||
6
.github/workflows/backend-unit-tests.yml
vendored
6
.github/workflows/backend-unit-tests.yml
vendored
@ -57,7 +57,7 @@ jobs:
|
|||||||
# its peers.
|
# its peers.
|
||||||
fail-fast: false
|
fail-fast: false
|
||||||
matrix:
|
matrix:
|
||||||
shard: [1, 2, 3, 4, 5, 6]
|
shard: [1, 2, 3, 4]
|
||||||
|
|
||||||
services:
|
services:
|
||||||
postgres:
|
postgres:
|
||||||
@ -105,7 +105,7 @@ jobs:
|
|||||||
working-directory: backend
|
working-directory: backend
|
||||||
run: uv sync --group dev --extra postgres
|
run: uv sync --group dev --extra postgres
|
||||||
|
|
||||||
- name: Run unit tests of backend (shard ${{ matrix.shard }} of ${{ strategy.job-total }})
|
- name: Run unit tests of backend (shard ${{ matrix.shard }} of 4)
|
||||||
working-directory: backend
|
working-directory: backend
|
||||||
env:
|
env:
|
||||||
# Expose the job's Postgres service to the cross-pod dedupe integration
|
# Expose the job's Postgres service to the cross-pod dedupe integration
|
||||||
@ -114,4 +114,4 @@ jobs:
|
|||||||
# exercised in CI.
|
# exercised in CI.
|
||||||
DEERFLOW_TEST_WORKFLOW_POSTGRES: ${{ env.TEST_POSTGRES_URI }}
|
DEERFLOW_TEST_WORKFLOW_POSTGRES: ${{ env.TEST_POSTGRES_URI }}
|
||||||
DEDUPE_TEST_POSTGRES_URL: ${{ env.TEST_POSTGRES_URI }}
|
DEDUPE_TEST_POSTGRES_URL: ${{ env.TEST_POSTGRES_URI }}
|
||||||
run: make test-shard SPLITS=${{ strategy.job-total }} GROUP=${{ matrix.shard }}
|
run: make test-shard SPLITS=4 GROUP=${{ matrix.shard }}
|
||||||
|
|||||||
104
.github/workflows/bundled-extension-tests.yml
vendored
104
.github/workflows/bundled-extension-tests.yml
vendored
@ -1,104 +0,0 @@
|
|||||||
name: Bundled Extension Tests
|
|
||||||
|
|
||||||
# Bundled extension examples own their behavior suites, which carry the
|
|
||||||
# `bundled_extension` marker. The core backend shards exclude them so an
|
|
||||||
# optional example cannot fail an unrelated core PR; this workflow runs them
|
|
||||||
# whenever an example, the extension contract, the host they load into or the
|
|
||||||
# bundled tests change.
|
|
||||||
#
|
|
||||||
# The filter is deliberately bounded: the marked modules import `deerflow.*`
|
|
||||||
# (packages) and `app.gateway.*`, so only the Gateway subtree is listed rather
|
|
||||||
# than all of `backend/app`. Test files are listed one by one instead of
|
|
||||||
# `backend/tests/**`, and `test_bundled_extension_ci.py` fails when a module
|
|
||||||
# carries the marker without appearing here, so this list cannot drift.
|
|
||||||
|
|
||||||
on:
|
|
||||||
pull_request:
|
|
||||||
types: [opened, synchronize, reopened, ready_for_review]
|
|
||||||
paths:
|
|
||||||
- 'examples/deerflow-extension-*/**'
|
|
||||||
- 'backend/packages/**'
|
|
||||||
- 'backend/app/gateway/**'
|
|
||||||
- 'backend/tests/conftest.py'
|
|
||||||
- 'backend/tests/support/**'
|
|
||||||
- 'backend/tests/test_agent_teams_plugin.py'
|
|
||||||
- 'backend/tests/test_bookmark_plugin.py'
|
|
||||||
- 'backend/tests/test_bundled_extension_ci.py'
|
|
||||||
- 'backend/tests/test_jev_classify_extension.py'
|
|
||||||
- 'backend/tests/test_jev_context_extension.py'
|
|
||||||
- 'backend/tests/test_jev_result_screening_extension.py'
|
|
||||||
- 'backend/tests/test_jev_screening_pipeline.py'
|
|
||||||
- 'backend/tests/test_jev_screening_policy.py'
|
|
||||||
- 'backend/tests/test_knowledge_center_extension.py'
|
|
||||||
- 'backend/tests/test_knowledge_management_extension.py'
|
|
||||||
- 'backend/tests/test_knowledge_platform_extension.py'
|
|
||||||
- 'backend/tests/test_lightrag_relations_extension.py'
|
|
||||||
- 'backend/tests/test_report_sections_extension.py'
|
|
||||||
- 'backend/tests/test_retrieval_plugin_artifacts.py'
|
|
||||||
- 'backend/tests/test_weknora_extension.py'
|
|
||||||
- 'backend/tests/test_workflow_plugin.py'
|
|
||||||
- 'backend/pyproject.toml'
|
|
||||||
- 'backend/uv.lock'
|
|
||||||
- 'backend/Makefile'
|
|
||||||
- '.github/workflows/bundled-extension-tests.yml'
|
|
||||||
push:
|
|
||||||
branches: [main, "*-dev"]
|
|
||||||
paths:
|
|
||||||
- 'examples/deerflow-extension-*/**'
|
|
||||||
- 'backend/packages/**'
|
|
||||||
- 'backend/app/gateway/**'
|
|
||||||
- 'backend/tests/conftest.py'
|
|
||||||
- 'backend/tests/support/**'
|
|
||||||
- 'backend/tests/test_agent_teams_plugin.py'
|
|
||||||
- 'backend/tests/test_bookmark_plugin.py'
|
|
||||||
- 'backend/tests/test_bundled_extension_ci.py'
|
|
||||||
- 'backend/tests/test_jev_classify_extension.py'
|
|
||||||
- 'backend/tests/test_jev_context_extension.py'
|
|
||||||
- 'backend/tests/test_jev_result_screening_extension.py'
|
|
||||||
- 'backend/tests/test_jev_screening_pipeline.py'
|
|
||||||
- 'backend/tests/test_jev_screening_policy.py'
|
|
||||||
- 'backend/tests/test_knowledge_center_extension.py'
|
|
||||||
- 'backend/tests/test_knowledge_management_extension.py'
|
|
||||||
- 'backend/tests/test_knowledge_platform_extension.py'
|
|
||||||
- 'backend/tests/test_lightrag_relations_extension.py'
|
|
||||||
- 'backend/tests/test_report_sections_extension.py'
|
|
||||||
- 'backend/tests/test_retrieval_plugin_artifacts.py'
|
|
||||||
- 'backend/tests/test_weknora_extension.py'
|
|
||||||
- 'backend/tests/test_workflow_plugin.py'
|
|
||||||
- 'backend/pyproject.toml'
|
|
||||||
- 'backend/uv.lock'
|
|
||||||
- 'backend/Makefile'
|
|
||||||
- '.github/workflows/bundled-extension-tests.yml'
|
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
group: bundled-extension-tests-${{ github.event.pull_request.number || github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
bundled-extension-tests:
|
|
||||||
if: github.event.pull_request.draft == false
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 10
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v6
|
|
||||||
|
|
||||||
- uses: actions/setup-python@v6
|
|
||||||
with:
|
|
||||||
python-version: '3.12'
|
|
||||||
|
|
||||||
- uses: astral-sh/setup-uv@v7
|
|
||||||
with:
|
|
||||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
|
||||||
version: "0.11.1"
|
|
||||||
|
|
||||||
- name: Install backend dependencies
|
|
||||||
working-directory: backend
|
|
||||||
run: uv sync --group dev
|
|
||||||
|
|
||||||
- name: Run bundled extension behavior tests
|
|
||||||
working-directory: backend
|
|
||||||
run: make test-bundled-extensions
|
|
||||||
41
.github/workflows/e2e-tests.yml
vendored
41
.github/workflows/e2e-tests.yml
vendored
@ -14,7 +14,6 @@ on:
|
|||||||
- 'backend/pyproject.toml'
|
- 'backend/pyproject.toml'
|
||||||
- 'backend/uv.lock'
|
- 'backend/uv.lock'
|
||||||
- 'examples/deerflow-extension-bookmarks/**'
|
- 'examples/deerflow-extension-bookmarks/**'
|
||||||
- 'examples/deerflow-extension-office-preview/**'
|
|
||||||
- '.github/workflows/e2e-tests.yml'
|
- '.github/workflows/e2e-tests.yml'
|
||||||
pull_request:
|
pull_request:
|
||||||
types: [opened, synchronize, reopened, ready_for_review]
|
types: [opened, synchronize, reopened, ready_for_review]
|
||||||
@ -29,7 +28,6 @@ on:
|
|||||||
- 'backend/pyproject.toml'
|
- 'backend/pyproject.toml'
|
||||||
- 'backend/uv.lock'
|
- 'backend/uv.lock'
|
||||||
- 'examples/deerflow-extension-bookmarks/**'
|
- 'examples/deerflow-extension-bookmarks/**'
|
||||||
- 'examples/deerflow-extension-office-preview/**'
|
|
||||||
- '.github/workflows/e2e-tests.yml'
|
- '.github/workflows/e2e-tests.yml'
|
||||||
|
|
||||||
concurrency:
|
concurrency:
|
||||||
@ -43,15 +41,6 @@ jobs:
|
|||||||
e2e-tests:
|
e2e-tests:
|
||||||
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
|
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
# Keep the image version aligned with frontend/pnpm-lock.yaml. Browsers and
|
|
||||||
# OS libraries are installed together, without per-shard CDN downloads.
|
|
||||||
container:
|
|
||||||
image: mcr.microsoft.com/playwright:v1.62.1-noble
|
|
||||||
options: --ipc=host
|
|
||||||
env:
|
|
||||||
# The host tool cache can contain Python built for a newer OpenSSL than
|
|
||||||
# this image. uv's standalone Python carries compatible runtime libraries.
|
|
||||||
UV_PYTHON_PREFERENCE: only-managed
|
|
||||||
timeout-minutes: 25
|
timeout-minutes: 25
|
||||||
strategy:
|
strategy:
|
||||||
fail-fast: false
|
fail-fast: false
|
||||||
@ -62,20 +51,21 @@ jobs:
|
|||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: actions/checkout@v6
|
uses: actions/checkout@v6
|
||||||
|
|
||||||
|
# The bookmark browser test starts the real Python action/router fixture.
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@v6
|
||||||
|
with:
|
||||||
|
python-version: '3.12'
|
||||||
|
|
||||||
- name: Install uv
|
- name: Install uv
|
||||||
uses: astral-sh/setup-uv@v7
|
uses: astral-sh/setup-uv@v7
|
||||||
with:
|
with:
|
||||||
# Keep aligned with backend/Dockerfile and other backend workflows.
|
# Keep aligned with backend/Dockerfile and other backend workflows.
|
||||||
version: "0.11.1"
|
version: "0.11.1"
|
||||||
|
|
||||||
- name: Install container-compatible Python
|
|
||||||
run: uv python install 3.12
|
|
||||||
|
|
||||||
- name: Install backend dependencies (bookmark plugin gateway)
|
- name: Install backend dependencies (bookmark plugin gateway)
|
||||||
working-directory: backend
|
working-directory: backend
|
||||||
run: |
|
run: uv sync --locked
|
||||||
uv sync --locked
|
|
||||||
uv run --no-sync python -c 'import ssl, sqlite3'
|
|
||||||
|
|
||||||
- name: Setup Node.js
|
- name: Setup Node.js
|
||||||
uses: actions/setup-node@v4
|
uses: actions/setup-node@v4
|
||||||
@ -92,20 +82,9 @@ jobs:
|
|||||||
working-directory: frontend
|
working-directory: frontend
|
||||||
run: pnpm install --frozen-lockfile
|
run: pnpm install --frozen-lockfile
|
||||||
|
|
||||||
# Playwright collects the resource suite before selecting a shard. Its
|
- name: Install Playwright Chromium
|
||||||
# synthetic Office fixtures use JSZip from the extension's locked install.
|
|
||||||
- name: Install Office preview test dependencies
|
|
||||||
working-directory: examples/deerflow-extension-office-preview
|
|
||||||
run: npm ci --ignore-scripts --no-audit --no-fund
|
|
||||||
|
|
||||||
- name: Verify Playwright Chromium
|
|
||||||
working-directory: frontend
|
working-directory: frontend
|
||||||
id: chromium
|
run: npx playwright install chromium --with-deps
|
||||||
run: >-
|
|
||||||
node --input-type=module -e
|
|
||||||
'import { chromium } from "@playwright/test";
|
|
||||||
const browser = await chromium.launch();
|
|
||||||
await browser.close();'
|
|
||||||
|
|
||||||
- name: Run E2E tests
|
- name: Run E2E tests
|
||||||
working-directory: frontend
|
working-directory: frontend
|
||||||
@ -121,7 +100,7 @@ jobs:
|
|||||||
|
|
||||||
- name: Run auth recovery E2E tests
|
- name: Run auth recovery E2E tests
|
||||||
# Keep auth coverage unsharded and run it even if the first shard fails.
|
# Keep auth coverage unsharded and run it even if the first shard fails.
|
||||||
if: ${{ !cancelled() && matrix.shard == 1 && steps.chromium.outcome == 'success' }}
|
if: ${{ !cancelled() && matrix.shard == 1 }}
|
||||||
working-directory: frontend
|
working-directory: frontend
|
||||||
run: >-
|
run: >-
|
||||||
pnpm exec playwright test -c playwright.auth.config.ts
|
pnpm exec playwright test -c playwright.auth.config.ts
|
||||||
|
|||||||
24
.github/workflows/replay-e2e.yml
vendored
24
.github/workflows/replay-e2e.yml
vendored
@ -71,27 +71,21 @@ jobs:
|
|||||||
name: Layer 2 — full-stack render (no API key)
|
name: Layer 2 — full-stack render (no API key)
|
||||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
# Match the locked Playwright runtime so the image supplies its browsers.
|
|
||||||
container:
|
|
||||||
image: mcr.microsoft.com/playwright:v1.62.1-noble
|
|
||||||
options: --ipc=host
|
|
||||||
env:
|
|
||||||
UV_PYTHON_PREFERENCE: only-managed
|
|
||||||
timeout-minutes: 25
|
timeout-minutes: 25
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v6
|
- uses: actions/checkout@v6
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@v6
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
- name: Install uv
|
- name: Install uv
|
||||||
uses: astral-sh/setup-uv@v7
|
uses: astral-sh/setup-uv@v7
|
||||||
with:
|
with:
|
||||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||||
version: "0.11.1"
|
version: "0.11.1"
|
||||||
- name: Install container-compatible Python
|
|
||||||
run: uv python install 3.12
|
|
||||||
- name: Install backend dependencies (replay gateway)
|
- name: Install backend dependencies (replay gateway)
|
||||||
working-directory: backend
|
working-directory: backend
|
||||||
run: |
|
run: uv sync --group dev
|
||||||
uv sync --group dev
|
|
||||||
uv run --no-sync python -c 'import ssl, sqlite3'
|
|
||||||
- name: Setup Node.js
|
- name: Setup Node.js
|
||||||
uses: actions/setup-node@v4
|
uses: actions/setup-node@v4
|
||||||
with:
|
with:
|
||||||
@ -103,13 +97,9 @@ jobs:
|
|||||||
- name: Install frontend dependencies
|
- name: Install frontend dependencies
|
||||||
working-directory: frontend
|
working-directory: frontend
|
||||||
run: pnpm install --frozen-lockfile
|
run: pnpm install --frozen-lockfile
|
||||||
- name: Verify Playwright Chromium
|
- name: Install Playwright Chromium
|
||||||
working-directory: frontend
|
working-directory: frontend
|
||||||
run: >-
|
run: npx playwright install chromium --with-deps
|
||||||
node --input-type=module -e
|
|
||||||
'import { chromium } from "@playwright/test";
|
|
||||||
const browser = await chromium.launch();
|
|
||||||
await browser.close();'
|
|
||||||
- name: Full-stack replay render (DOM assertions are the gate)
|
- name: Full-stack replay render (DOM assertions are the gate)
|
||||||
working-directory: frontend
|
working-directory: frontend
|
||||||
run: pnpm exec playwright test -c playwright.real-backend.config.ts
|
run: pnpm exec playwright test -c playwright.real-backend.config.ts
|
||||||
|
|||||||
12
.github/workflows/sandbox-image-smoke.yml
vendored
12
.github/workflows/sandbox-image-smoke.yml
vendored
@ -29,7 +29,7 @@ permissions:
|
|||||||
jobs:
|
jobs:
|
||||||
sandbox-image-smoke:
|
sandbox-image-smoke:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
timeout-minutes: 45
|
timeout-minutes: 25
|
||||||
env:
|
env:
|
||||||
SANDBOX_SMOKE_IMAGE_REF: ${{ vars.SANDBOX_SMOKE_IMAGE || 'enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest' }}
|
SANDBOX_SMOKE_IMAGE_REF: ${{ vars.SANDBOX_SMOKE_IMAGE || 'enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest' }}
|
||||||
FOWNER_SMOKE_IMAGE_REF: ${{ vars.SANDBOX_FOWNER_SMOKE_IMAGE || 'enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:1.11.0' }}
|
FOWNER_SMOKE_IMAGE_REF: ${{ vars.SANDBOX_FOWNER_SMOKE_IMAGE || 'enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:1.11.0' }}
|
||||||
@ -38,16 +38,7 @@ jobs:
|
|||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: actions/checkout@v6
|
uses: actions/checkout@v6
|
||||||
|
|
||||||
- name: Free disk space for sandbox images
|
|
||||||
# This job only needs Python and Docker, not the runner's mobile/.NET/Haskell SDKs.
|
|
||||||
run: |
|
|
||||||
df -h /
|
|
||||||
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /usr/local/.ghcup /opt/ghc
|
|
||||||
docker image prune --all --force
|
|
||||||
df -h /
|
|
||||||
|
|
||||||
- name: Resolve immutable image references for this run
|
- name: Resolve immutable image references for this run
|
||||||
timeout-minutes: 30
|
|
||||||
# Pull on the runner's platform, then pass only immutable repo@sha256
|
# Pull on the runner's platform, then pass only immutable repo@sha256
|
||||||
# references to pytest. This keeps both the baseline and the 1.11.0
|
# references to pytest. This keeps both the baseline and the 1.11.0
|
||||||
# regression run reproducible even if either source tag moves later.
|
# regression run reproducible even if either source tag moves later.
|
||||||
@ -85,6 +76,5 @@ jobs:
|
|||||||
run: uv sync --group dev
|
run: uv sync --group dev
|
||||||
|
|
||||||
- name: Run the live real-image smoke tests
|
- name: Run the live real-image smoke tests
|
||||||
timeout-minutes: 10
|
|
||||||
working-directory: backend
|
working-directory: backend
|
||||||
run: uv run pytest -m live tests/test_aio_sandbox_local_backend.py -v
|
run: uv run pytest -m live tests/test_aio_sandbox_local_backend.py -v
|
||||||
|
|||||||
2
.github/workflows/skill-review-ci.yml
vendored
2
.github/workflows/skill-review-ci.yml
vendored
@ -67,7 +67,7 @@ jobs:
|
|||||||
working-directory: backend
|
working-directory: backend
|
||||||
run: |
|
run: |
|
||||||
# Fail instead of silently skipping data-analysis tests on missing extras.
|
# Fail instead of silently skipping data-analysis tests on missing extras.
|
||||||
uv run --locked python -c "import duckdb, openpyxl, pandas, requests, xlrd"
|
uv run --locked python -c "import duckdb, openpyxl, requests"
|
||||||
uv run --locked python -m pytest ../tests/skills -q -rs
|
uv run --locked python -m pytest ../tests/skills -q -rs
|
||||||
|
|
||||||
skill-review:
|
skill-review:
|
||||||
|
|||||||
8
.github/workflows/triage.yml
vendored
8
.github/workflows/triage.yml
vendored
@ -55,13 +55,7 @@ jobs:
|
|||||||
// ---- area: replaces .github/labeler.yml (path -> area) ----
|
// ---- area: replaces .github/labeler.yml (path -> area) ----
|
||||||
const AREA_RULES = [
|
const AREA_RULES = [
|
||||||
['area:frontend', [/^frontend\//]],
|
['area:frontend', [/^frontend\//]],
|
||||||
// Cover both backend packages. Keep the harness exclusions aligned
|
['area:backend', [/^backend\/app\//, /^backend\/packages\/harness\/deerflow\/(runtime|persistence|config|tools|guardrails|tracing|models|utils|uploads)\//]],
|
||||||
// with the dedicated tree rules below; the offline replay checks this.
|
|
||||||
['area:backend', [
|
|
||||||
/^backend\/app\//,
|
|
||||||
/^backend\/packages\/harness\/(?!deerflow\/(?:agents|subagents|reflection|sandbox|skills|mcp)\/)/,
|
|
||||||
/^backend\/packages\/extension-api\//,
|
|
||||||
]],
|
|
||||||
['area:agents', [/^backend\/packages\/harness\/deerflow\/(agents|subagents|reflection)\//, /(^|\/)langgraph\.json$/, /^backend\/.*\/prompts\//]],
|
['area:agents', [/^backend\/packages\/harness\/deerflow\/(agents|subagents|reflection)\//, /(^|\/)langgraph\.json$/, /^backend\/.*\/prompts\//]],
|
||||||
['area:sandbox', [/^docker\//, /^backend\/packages\/harness\/deerflow\/sandbox\//, /(^|\/)Dockerfile$/]],
|
['area:sandbox', [/^docker\//, /^backend\/packages\/harness\/deerflow\/sandbox\//, /(^|\/)Dockerfile$/]],
|
||||||
['area:skills', [/^skills\//, /^backend\/packages\/harness\/deerflow\/skills\//, /^frontend\/src\/core\/skills\//]],
|
['area:skills', [/^skills\//, /^backend\/packages\/harness\/deerflow\/skills\//, /^frontend\/src\/core\/skills\//]],
|
||||||
|
|||||||
325
AGENTS.md
325
AGENTS.md
@ -1,122 +1,251 @@
|
|||||||
# DeerFlow
|
# AGENTS.md
|
||||||
|
|
||||||
DeerFlow is an agent harness: a LangGraph-based backend with thread-isolated
|
Guidance source; `CLAUDE.md` imports `@AGENTS.md`.
|
||||||
sandboxes, memory, subagents, skills and MCP tools, and a Next.js frontend. IM
|
Module guides:
|
||||||
channels connect through the Gateway. `CLAUDE.md` imports this file; edit
|
|
||||||
guidance here, not in the shim.
|
|
||||||
|
|
||||||
## How guidance is organized
|
- **[backend/AGENTS.md](backend/AGENTS.md)** — backend depth: harness/app split, agent &
|
||||||
|
middleware chain, sandbox, MCP, skills, memory, IM channels, persistence/migrations,
|
||||||
|
config system, test layout.
|
||||||
|
- **[frontend/AGENTS.md](frontend/AGENTS.md)** — frontend depth: Next.js App Router layout,
|
||||||
|
thread/streaming data flow, code style, commands.
|
||||||
|
|
||||||
Rules live in the `AGENTS.md` nearest to the code. Before you edit a directory,
|
## What is DeerFlow
|
||||||
read every `AGENTS.md` from that directory up to this file. Some coding agents do
|
|
||||||
not load the nested files on their own.
|
|
||||||
|
|
||||||
| Area | Guide |
|
DeerFlow combines LangGraph and Next.js with thread-isolated sandbox execution,
|
||||||
| --- | --- |
|
memory, subagents and extensible tools. IM channels connect through the Gateway.
|
||||||
| Backend: harness, Gateway, channels, tests | [backend/AGENTS.md](backend/AGENTS.md) |
|
|
||||||
| Frontend | [frontend/AGENTS.md](frontend/AGENTS.md) |
|
|
||||||
| nginx, Compose, provisioner | [docker/AGENTS.md](docker/AGENTS.md) |
|
|
||||||
| Orchestration scripts, `scripts/pnpm.py` | [scripts/AGENTS.md](scripts/AGENTS.md) |
|
|
||||||
| Skill review, skill tests | [skills/AGENTS.md](skills/AGENTS.md) |
|
|
||||||
| Extension examples | `examples/*/AGENTS.md` |
|
|
||||||
|
|
||||||
Humans start at [README.md](README.md), [CONTRIBUTING.md](CONTRIBUTING.md) and
|
## Service Topology
|
||||||
[docs/guide/](docs/guide/).
|
|
||||||
|
|
||||||
## Service topology
|
A single `make dev` / Docker stack runs four cooperating services:
|
||||||
|
|
||||||
| Service | Port | Role |
|
| Service | Port | Role |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Nginx | 2026 | The single entry point; open this in the browser |
|
| **Nginx** | `2026` | Unified reverse-proxy entry point — open this in the browser |
|
||||||
| Gateway | 8001 | FastAPI REST API and the embedded agent runtime |
|
| **Gateway API** | `8001` | FastAPI REST API + embedded LangGraph-compatible agent runtime |
|
||||||
| Frontend | 3000 | Next.js web interface |
|
| **Frontend** | `3000` | Next.js web interface |
|
||||||
| Provisioner | 8002 | Optional; only for provisioner or Kubernetes sandboxes |
|
| **Provisioner** | `8002` | Optional — only when sandbox is configured for provisioner/K8s mode |
|
||||||
|
|
||||||
Nginx serves the frontend, proxies `/api/*` to the Gateway and rewrites
|
Nginx is the single public entry: it proxies `/api/*` to the Gateway, rewriting
|
||||||
`/api/langgraph/*` to the Gateway's native routes.
|
`/api/langgraph/*` onto the Gateway's native routes, and serves the frontend — see
|
||||||
|
[backend/AGENTS.md](backend/AGENTS.md) for the runtime and router detail. It compresses
|
||||||
|
HTML and configured textual assets, deliberately leaving SSE, fonts, images, audio, and
|
||||||
|
video uncompressed at the proxy layer.
|
||||||
|
|
||||||
Before changing ingress, port bindings or provisioner mounts, read
|
Keep the dedicated `/api/threads/{thread_id}/browser/stream` WebSocket regex before
|
||||||
[service topology details](docs/agent-service-topology.md).
|
the generic threads regex in Docker, local and Helm nginx configs. Upgrade headers
|
||||||
|
and the 600s read timeout are pinned by `backend/tests/test_nginx_browser_websocket.py`.
|
||||||
|
Helm uses nginx's default connect/send timeouts for the in-cluster Gateway.
|
||||||
|
|
||||||
## Repository map
|
Both Compose files publish nginx at `"${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026"`
|
||||||
|
(README's **loopback default**); bare `"${PORT}:2026"` binds `0.0.0.0`. Root `PORT`
|
||||||
|
controls Docker ingress only; local Next.js stays on `3000` so `.env` cannot
|
||||||
|
redirect `make dev`'s port wait. Nginx's IPv4+IPv6 `default_server` and Gateway's
|
||||||
|
`0.0.0.0:8001` bind are container-internal; only published nginx is external.
|
||||||
|
Require explicit bind addresses for new published ports;
|
||||||
|
`backend/tests/test_compose_default_bind_host.py` pins every service in both files.
|
||||||
|
|
||||||
| Path | Content |
|
Provisioner PVC mode declares `USERDATA_PVC_NAME` once as `user-data`; extra and
|
||||||
| --- | --- |
|
Lark broker credential mounts reuse it with their existing subPaths and read-only
|
||||||
| `backend/packages/harness/` | `deerflow-harness`, import `deerflow.*`: the agent framework |
|
flags. Keep credential mounts broker-only and keep distinct volumes in hostPath mode.
|
||||||
| `backend/packages/extension-api/` | `deerflow-extension-api`: the public extension contract |
|
|
||||||
| `backend/app/` | The Gateway and the IM channels, import `app.*` |
|
|
||||||
| `frontend/` | The Next.js app (pnpm) |
|
|
||||||
| `docker/`, `deploy/helm/` | Compose files, nginx config, provisioner, Helm chart |
|
|
||||||
| `skills/public/`, `skills/custom/` | Committed skills and local, gitignored skills |
|
|
||||||
| `contracts/` | JSON contracts shared across components |
|
|
||||||
| `examples/` | Installable extension examples |
|
|
||||||
| `scripts/`, `tests/` | Root orchestration scripts, and tests for public skills |
|
|
||||||
|
|
||||||
`config.yaml` and `extensions_config.json` are copied from the example files at
|
## Repository Map
|
||||||
the repository root. Both are gitignored and the Gateway can write them. Do not
|
|
||||||
commit them.
|
|
||||||
|
|
||||||
## Commands
|
```
|
||||||
|
deer-flow/
|
||||||
`make help` lists the root targets. First setup is `make config`, then
|
├── Makefile # Root orchestration: drives the full stack (dev/start/stop, docker, setup)
|
||||||
`make install`, then `make dev`, which serves the stack on `localhost:2026` and
|
├── config.example.yaml # Template → copy to config.yaml (gitignored) at repo root
|
||||||
writes `logs/gateway.log` and `logs/frontend.log`.
|
├── extensions_config.example.json # Template → copy to extensions_config.json (gitignored): MCP servers + skills
|
||||||
|
├── backend/ # Python backend — see backend/AGENTS.md
|
||||||
```bash
|
│ ├── Makefile # Per-module backend commands (dev, gateway, test, lint, migrate-rev)
|
||||||
cd backend && make test && make test-blocking-io # offline suites
|
│ ├── extensions/sources/ # Deployable snapshots of locally installed Python extensions
|
||||||
cd backend && make format # CI runs ruff format --check
|
│ ├── packages/extension-api/ # deerflow-extension-api package (import: deerflow_extension_api.*) — public extension contract
|
||||||
cd backend && python -m pytest tests/path/test_x.py::test_name -q
|
│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*) — agent framework
|
||||||
cd frontend && pnpm check && pnpm test
|
│ └── app/ # FastAPI Gateway + IM channels (import: app.*)
|
||||||
cd frontend && pnpm rstest run <pattern>
|
├── frontend/ # Next.js frontend (pnpm) — see frontend/AGENTS.md
|
||||||
make check-agent-guidance # size budget of these guides
|
├── docker/ # docker-compose files, nginx config, provisioner
|
||||||
|
├── skills/ # Agent skills: public/ (committed), custom/ (gitignored)
|
||||||
|
│ # Managed integration skill packs are global at .deer-flow/integrations/skills/{provider}/
|
||||||
|
│ # Integration credentials and enabled state remain per-user
|
||||||
|
├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review)
|
||||||
|
├── examples/ # Extension examples: deerflow-extension-{example,bookmarks,agent-teams}
|
||||||
|
├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
|
||||||
|
├── tests/ # Root-level tests (currently tests/skills/ — public skill tests)
|
||||||
|
└── docs/ # Cross-cutting docs, plans, and design notes
|
||||||
```
|
```
|
||||||
|
|
||||||
## Rules for every change
|
Third-party extensions load from operator-controlled `config.yaml -> plugins:`.
|
||||||
|
This imports code, so the list stays outside API-writable `extensions_config.json`.
|
||||||
|
Extensions contribute middleware, lifecycle observers, Gateway services, FastAPI
|
||||||
|
routers, and experimental full-stack plugins. Use
|
||||||
|
`deerflow extensions install/upgrade/list/enable/disable/remove` or root
|
||||||
|
`make extension-*` wrappers. Restart Gateway after every mutation. Build hooks
|
||||||
|
and extension code run with Gateway privileges; use only trusted operator sources.
|
||||||
|
Transactions, accepted source forms, locking, and contribution contracts:
|
||||||
|
[extensions guide](backend/packages/harness/deerflow/extensions/AGENTS.md).
|
||||||
|
User manual: `frontend/src/content/{en,zh}/harness/extensions/`.
|
||||||
|
|
||||||
- **Evaluate an extension first.** Before you add a feature to core code, check
|
**Extension-first evaluation** — before implementing a feature by editing core code,
|
||||||
whether it can ship as a packaged extension. Core is the place for bug fixes
|
evaluate whether it can ship as a packaged extension through the contribution points
|
||||||
and for changes to the agent loop, memory, context compaction and
|
above. Stay in core for bug fixes and for changes to the agent loop, memory, context
|
||||||
authentication. When the existing hooks cannot express a feature, add a
|
compaction, or authentication. If existing hooks cannot express the feature, add a
|
||||||
generic hook to the extension contract and an extension that uses it; business
|
generic hook to the extension contract plus an extension that uses it — do not
|
||||||
logic is not hard-coded into core. Contract and examples:
|
hard-code business logic into core. Working examples:
|
||||||
[extensions guide](backend/packages/harness/deerflow/extensions/AGENTS.md),
|
`examples/deerflow-extension-{example,bookmarks,agent-teams}`.
|
||||||
`examples/`.
|
[Workflow guide](examples/deerflow-extension-workflows/AGENTS.md): public
|
||||||
- **Extensions are operator-controlled code.** They load from `plugins:` in
|
AgentRuns; POSIX, one process/store; approval and session-authorized recovery; uncertain admissions need reconciliation;
|
||||||
`config.yaml`. The list stays out of the API-writable `extensions_config.json`,
|
no multi-Gateway failover or hard currency budget.
|
||||||
because loading an extension imports code with Gateway privileges. Install
|
|
||||||
from trusted sources only, and restart the Gateway after every extension
|
|
||||||
change.
|
|
||||||
- **Ship tests with every feature and bug fix.**
|
|
||||||
- **Document behavior next to its topic, in the same change**: user-facing
|
|
||||||
behavior in `docs/guide/` or `backend/docs/`. `README.md` is an index; change
|
|
||||||
it only to add or re-point a link.
|
|
||||||
- **Keep the version fields identical**: `backend/pyproject.toml`,
|
|
||||||
`backend/packages/harness/pyproject.toml`, `frontend/package.json`, and
|
|
||||||
`version` and `appVersion` in `deploy/helm/deer-flow/Chart.yaml`.
|
|
||||||
`scripts/bump_version.sh <ver>` updates them and `backend/uv.lock`, and
|
|
||||||
`scripts/verify_versions.sh <ver>` checks the result. Tag CI blocks publishing
|
|
||||||
on drift. See [RELEASING.md](RELEASING.md).
|
|
||||||
|
|
||||||
## What belongs in an `AGENTS.md`
|
Knowledge management/space policy: `examples/deerflow-extension-knowledge-{management,platform}/`
|
||||||
|
(scoped `AGENTS.md`, `CONTRACT.md`). Use existing plugin/router/principal contracts;
|
||||||
|
retrieval stays independent. Space policy intersects administrator/resource/operation
|
||||||
|
and upstream grants, never grants chat access. SQLite holds plugin-owned space policy.
|
||||||
|
Provider stages are not completion evidence; reconcile uncertain writes, never retry.
|
||||||
|
|
||||||
An agent reads these files on every task, so each line costs attention. A guide
|
Root runtime configs: copy `config.example.yaml` → `config.yaml` and
|
||||||
holds four kinds of content:
|
`extensions_config.example.json` → `extensions_config.json` (MCP/skills).
|
||||||
|
Both are gitignored and Gateway-editable; schema/resolution:
|
||||||
|
[backend/AGENTS.md](backend/AGENTS.md).
|
||||||
|
|
||||||
1. Scope: what the directory owns, in one or two sentences.
|
Skill quality review note:
|
||||||
2. Verification: the exact commands.
|
- `skills/public/skill-reviewer/` is the built-in read-only reviewer (harness tool
|
||||||
3. Traps: where the code misleads, or where a plausible change breaks something
|
`review_skill_package`, contracts in `contracts/skill_review/`). Model-visible review
|
||||||
that a local test run does not show.
|
data is tag-neutralized; raw payloads stay in tool artifacts. See
|
||||||
4. Decisions and their reasons that the code cannot state, and links to docs.
|
[backend/AGENTS.md](backend/AGENTS.md) for the ownership boundaries.
|
||||||
|
- CI waivers live in `.github/skill-review-waivers.v1.json`, enforced by
|
||||||
|
`scripts/review_changed_public_skills.py`. Only the manifest from the trusted base
|
||||||
|
revision can suppress a run, and blocker findings can never be waived.
|
||||||
|
|
||||||
Values, defaults, field lists, function inventories, test names, issue numbers
|
Scheduled-task note:
|
||||||
and history stay in the code, the tests and git. Write full sentences in the
|
- `scheduler.tool_enabled` (default off) offers owner-bound schedule tools only
|
||||||
positive form and give the reason. `make check-agent-guidance` enforces the size
|
through Gateway capability admission. Interactive turns create/manage their
|
||||||
budget.
|
schedules; scheduled turns can request stopping only their own schedule.
|
||||||
|
Per-occurrence goal success does not end a recurring schedule.
|
||||||
|
- The scheduled-task MVP adds a workspace page at `/workspace/scheduled-tasks` plus a
|
||||||
|
background scheduler gated by `config.yaml -> scheduler.enabled`.
|
||||||
|
- Scheduled runs are non-interactive: `ask_clarification` is excluded. `non_interactive`,
|
||||||
|
`disable_clarification`, and `github_token` are honored only for internally-authenticated
|
||||||
|
callers; client-supplied copies are dropped.
|
||||||
|
- Busy occurrences persist as `queued`; `scheduler.queue_timeout_seconds` bounds the wait.
|
||||||
|
Do not reintroduce skip-on-overlap.
|
||||||
|
|
||||||
### Office preview example
|
## Commands: Root vs. Module
|
||||||
|
|
||||||
`examples/deerflow-extension-office-preview/` packages Office readers through the
|
**Root stack commands** (from repo root):
|
||||||
optional `artifactPreviews` browser contract. Keep parser dependencies and resources
|
|
||||||
in the extension; rebuild its manifest and static module graph after UI changes.
|
```bash
|
||||||
Test packaged assets with the Office E2E fixtures; rendering must preserve original
|
make setup # Interactive setup wizard (recommended for new users)
|
||||||
downloads and state its search limits.
|
make doctor # Check configuration and system requirements
|
||||||
|
make support-bundle # Generate redacted troubleshooting summary, AI issue draft, and optional zip
|
||||||
|
make config # Generate local config files from the examples
|
||||||
|
make check # Check that required tools are installed
|
||||||
|
make install # Install all dependencies (frontend + backend + pre-commit hooks)
|
||||||
|
make extension-install SOURCE=... # Install and enable a trusted Python extension
|
||||||
|
make extension-upgrade SOURCE=... # Replace an installed extension and keep its config
|
||||||
|
make extension-list # List configured Python extensions
|
||||||
|
make extension-enable NAME=... # Enable an installed extension (restart required)
|
||||||
|
make extension-disable NAME=... # Disable without uninstalling (restart required)
|
||||||
|
make extension-remove NAME=... # Remove package and config entry (restart required)
|
||||||
|
make dev # Start all services with hot-reload (Gateway + Frontend + Nginx)
|
||||||
|
make start # Start all services in production mode (local, optimized); SKIP_FRONTEND_BUILD=1 reuses the last frontend build
|
||||||
|
make stop # Stop all running services
|
||||||
|
make up / down # Build/stop the production Docker stack (browser at localhost:2026)
|
||||||
|
make docker-start / docker-stop / docker-logs # Docker development environment
|
||||||
|
```
|
||||||
|
|
||||||
|
Production uses image Python (`uv run --no-sync`) and `exec` for SIGTERM;
|
||||||
|
Compose sets `init: true` (`backend/tests/test_gateway_sigterm_delivery.py`).
|
||||||
|
`make up` waits for Gateway `/health`; failures show Compose status and Gateway logs.
|
||||||
|
|
||||||
|
Docker logs/restarts resolve `DEER_FLOW_ROOT` from this checkout before Compose,
|
||||||
|
as start/stop do.
|
||||||
|
|
||||||
|
Full list: `make help`.
|
||||||
|
|
||||||
|
**Module commands** (from each module):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backend (see backend/AGENTS.md for the full set)
|
||||||
|
cd backend && make dev # Gateway API with reload (port 8001)
|
||||||
|
cd backend && make test # Four parallel shards; excludes live and blocking-I/O tests
|
||||||
|
cd backend && make test-blocking-io # Strict blocking-I/O suite
|
||||||
|
cd backend && make lint # ruff check
|
||||||
|
cd backend && make format # ruff format
|
||||||
|
|
||||||
|
# Frontend (see frontend/AGENTS.md for the full set)
|
||||||
|
cd frontend && pnpm dev # Dev server: Webpack by default (override with DEER_FLOW_DEV_BUNDLER=turbo)
|
||||||
|
cd frontend && pnpm check # Lint + type check (run before committing)
|
||||||
|
cd frontend && pnpm test # Unit tests
|
||||||
|
```
|
||||||
|
|
||||||
|
Host pnpm calls use `scripts/pnpm.py`: native Windows tries `pnpm.cmd` before
|
||||||
|
`pnpm`; POSIX reverses the order. Its Corepack fallback applies the same ordering
|
||||||
|
to `corepack.cmd` and `corepack`. The runner operates from `frontend/` so
|
||||||
|
Corepack honors its pinned package-manager version.
|
||||||
|
|
||||||
|
### Prerequisites before `make dev`
|
||||||
|
|
||||||
|
Run `make config`, then `make install`, then `make dev` on first setup.
|
||||||
|
`make dev` requires existing config; both runtime configs are API-editable and
|
||||||
|
gitignored. Never commit them.
|
||||||
|
|
||||||
|
### Run a single test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backend (pytest); run one file or one test function
|
||||||
|
cd backend && python -m pytest tests/test_compose_default_bind_host.py -q
|
||||||
|
cd backend && python -m pytest tests/path/to/test.py::test_func -q
|
||||||
|
|
||||||
|
# Frontend (rstest)
|
||||||
|
cd frontend && pnpm rstest run <pattern> # e.g. pnpm rstest run my-component
|
||||||
|
```
|
||||||
|
|
||||||
|
### Logs
|
||||||
|
|
||||||
|
- Docker stack: `make docker-logs` (or `docker compose -f docker/... logs -f <svc>`).
|
||||||
|
- Local `make dev`: Gateway and frontend output goes to
|
||||||
|
`logs/gateway.log` and `logs/frontend.log` in the repository root.
|
||||||
|
Run `tail -f logs/gateway.log logs/frontend.log` there to follow both.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- Backend work → **[backend/AGENTS.md](backend/AGENTS.md)**
|
||||||
|
- Frontend work → **[frontend/AGENTS.md](frontend/AGENTS.md)**
|
||||||
|
- Setup & install → **[Install.md](Install.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**
|
||||||
|
- Project overview & usage → **[README.md](README.md)** (translations: `README_zh.md`,
|
||||||
|
`README_ja.md`, `README_fr.md`, `README_ru.md`, `README_pt.md`, `README_ko.md`)
|
||||||
|
- Security policy → **[SECURITY.md](SECURITY.md)**
|
||||||
|
- Changes → **[CHANGELOG.md](CHANGELOG.md)**
|
||||||
|
- Cutting a release → **[RELEASING.md](RELEASING.md)**
|
||||||
|
|
||||||
|
## Cross-Cutting Conventions
|
||||||
|
|
||||||
|
Repo-wide conventions:
|
||||||
|
|
||||||
|
- Update `README.md` for user-facing changes and the relevant `AGENTS.md` for
|
||||||
|
development/architecture changes in the same change set.
|
||||||
|
- **Test-driven development** — features and bug fixes ship with tests. Backend tests live
|
||||||
|
in `backend/tests/` (TDD is mandatory there; see [backend/AGENTS.md](backend/AGENTS.md));
|
||||||
|
frontend tests live in `frontend/tests/`.
|
||||||
|
Skill-creator tests must preserve YAML values/body, restore bytecode settings,
|
||||||
|
and check invalid metadata through offline CLI subprocesses.
|
||||||
|
Public skill script regressions live in `tests/skills/` and run in Skill Review CI
|
||||||
|
independently of package review. Exercise dependency fallbacks with the optional
|
||||||
|
import unavailable and mock HTTP transport to keep tests offline.
|
||||||
|
Video, podcast, and chart-visualization CLI failures exit non-zero;
|
||||||
|
diagnostics on stderr, success URLs on stdout; test offline.
|
||||||
|
- **Before pushing** — backend `make format`, frontend `pnpm check`.
|
||||||
|
Backend CI enforces `ruff format --check`.
|
||||||
|
- **Skill text encoding** — use explicit UTF-8 for textual skill resources.
|
||||||
|
Read `SKILL.md` with `encoding="utf-8-sig"` to strip a leading BOM; write with
|
||||||
|
`encoding="utf-8"`. Never rely on the platform locale.
|
||||||
|
- **Version sources must stay in lockstep** — a release version must match identically in
|
||||||
|
`backend/pyproject.toml`, `backend/packages/harness/pyproject.toml`, `frontend/package.json`,
|
||||||
|
and `deploy/helm/deer-flow/Chart.yaml` (`version` + `appVersion`), and `backend/uv.lock` must
|
||||||
|
record the same version for the root `deer-flow` and `deerflow-harness` workspace members
|
||||||
|
(uv stores its PEP 440 form, e.g. `2.1.0rc0`). Pushing a `v*` git tag triggers CI
|
||||||
|
that runs `scripts/verify_versions.sh` and **blocks all publishing** if any source drifts.
|
||||||
|
Before bumping a version, run `scripts/bump_version.sh <ver>` (aligns the five fields and
|
||||||
|
refreshes `backend/uv.lock` at once — it needs `uv` on `PATH`) and
|
||||||
|
`scripts/verify_versions.sh <ver>` to catch drift early. See [RELEASING.md](RELEASING.md).
|
||||||
|
- **Don't edit `CLAUDE.md`** — it only contains `@AGENTS.md`. All agent guidance changes
|
||||||
|
belong here in `AGENTS.md`; `CLAUDE.md` is a thin import shim.
|
||||||
|
|||||||
561
CHANGELOG.md
561
CHANGELOG.md
@ -7,491 +7,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
This section accumulates work toward the **2.3.0** milestone
|
|
||||||
([2.3.0](https://github.com/bytedance/deer-flow/milestone/4)).
|
|
||||||
|
|
||||||
### Added
|
|
||||||
- **summarization:** The built-in compaction prompt guides the summarizer to
|
|
||||||
preserve active user requirements, literal approval phrases, action conditions,
|
|
||||||
and explicit revisions. Explicit `summarization.summary_prompt` configurations
|
|
||||||
continue to take precedence. ([#8114](https://github.com/bytedance/deer-flow/pull/8114))
|
|
||||||
|
|
||||||
- **sandbox:** Short ranged `read_file` calls on uploaded and workspace files
|
|
||||||
display source line numbers when the result contains 1–40 lines and the
|
|
||||||
labeled view fits the output budget. ([#8109](https://github.com/bytedance/deer-flow/pull/8109))
|
|
||||||
|
|
||||||
- **projects:** Owner-scoped project memory with automatic-extraction, delete and
|
|
||||||
move-to-instructions controls. Project facts share DeerMem's extraction call
|
|
||||||
while remaining isolated from user facts. New shelf documents receive
|
|
||||||
best-effort summaries by default. ([#8117])
|
|
||||||
|
|
||||||
- **hitl:** Optional Ask, Auto and Full tool-permission modes, independent of
|
|
||||||
reasoning mode. Auto uses a separate tool-free review call with an existing
|
|
||||||
model profile; uncertain or high-risk operations require a human. Cards show
|
|
||||||
complete arguments and support approve, reject, edit and exact-operation chat
|
|
||||||
grants with revocation. Root approvals survive restart through checkpoints;
|
|
||||||
interactive subagents wait through an owner-scoped channel with cancellation
|
|
||||||
and expiry. Mandatory deployment rules and account permissions apply in every
|
|
||||||
mode. Personal/platform MCP connections, policy and arguments bind approvals.
|
|
||||||
Disabled by default; see the configuration guide. `config_version` 66.
|
|
||||||
|
|
||||||
- **knowledge:** The WeKnora retrieval extension offers opt-in shared read-only
|
|
||||||
access for authenticated users, without maintaining a per-user allowlist.
|
|
||||||
Exact-user allowlists remain the default; shared access requires matching
|
|
||||||
host-verified identity and grants no management permissions. ([#7944](https://github.com/bytedance/deer-flow/pull/7944))
|
|
||||||
|
|
||||||
- **community:** New SpecStream `web_search` and `web_fetch` provider. Search returns
|
|
||||||
ranked results with a direct answer when the sources agree on one, and fetch
|
|
||||||
returns a page or document as markdown. ([#7968](https://github.com/bytedance/deer-flow/pull/7968))
|
|
||||||
|
|
||||||
- **subagents:** Durable batch items can wait for explicitly named prerequisites.
|
|
||||||
An optional, once-only lead-Agent follow-up completes a previously requested
|
|
||||||
summary or delivery after the batch ends, with persisted admission, cancellation
|
|
||||||
and authorization checks. Active Goals retain continuation ownership. ([#7839](https://github.com/bytedance/deer-flow/pull/7839))
|
|
||||||
|
|
||||||
- **chat:** Running Web chats offer Queue and Steer in the composer. Queue
|
|
||||||
waits for the next turn; Steer lets the main Agent handle updates while
|
|
||||||
native subagents pause and resume. Unconfirmed messages can be retried
|
|
||||||
after a Gateway restart. ([#7729])
|
|
||||||
|
|
||||||
- **models:** Administrators can select StepFun in **Settings → Models** and
|
|
||||||
connect an ordinary StepFun API account with an API key and model ID.
|
|
||||||
([#7988](https://github.com/bytedance/deer-flow/pull/7988))
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- **checkpoint:** DeltaChannel deployments now require `langgraph-checkpoint`
|
|
||||||
4.3.0, SQLite saver 3.1.2, and Postgres saver 3.1.3, excluding versions that
|
|
||||||
could reorder parallel writes during replay or lose paged ancestor history.
|
|
||||||
The ordering and Postgres pagination contracts now fail on regression instead
|
|
||||||
of skipping. ([#8139])
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- **frontend:** Cron presets in the task form now require minute 0-59, hour
|
|
||||||
0-23 and day of month 1-31, so an out-of-range expression such as
|
|
||||||
`0 24 * * *` stays a custom schedule and keeps its original text. Previously,
|
|
||||||
it was parsed as a preset and could be clamped to a different time when
|
|
||||||
serialized for submission. Creating or duplicating a task with an invalid
|
|
||||||
expression, or updating its schedule configuration while the expression is
|
|
||||||
invalid, returns the Gateway's `422`; edits to other fields do not revalidate
|
|
||||||
an unchanged schedule. ([#8020])
|
|
||||||
|
|
||||||
- **gateway:** Creating a thread with checkpoint-owned keys such as `step`,
|
|
||||||
`parents` or `counters_since_delta_snapshot` in its metadata no longer
|
|
||||||
leaves the thread broken. Those keys used to land on the thread's first
|
|
||||||
checkpoint, so a bad `parents` made the thread's history fail with 500 for
|
|
||||||
good, and a bad `step` or delta counter made every later turn fail. A
|
|
||||||
`deerflow_agent_name` key was also copied into the next state update as if
|
|
||||||
the server had set it. The first checkpoint now takes none of these keys
|
|
||||||
from the request. ([#8019])
|
|
||||||
|
|
||||||
- **mcp:** Two enabled MCP servers could bind the same tool name (server `a`
|
|
||||||
exposing `b_x` and server `a_b` exposing `x` both bind `a_b_x`). Both
|
|
||||||
declarations reached the model and the provider silently kept only one.
|
|
||||||
Duplicate bound names are now dropped with a warning naming the surviving
|
|
||||||
server, in enabled-server declaration order. Note the surviving server is
|
|
||||||
stable only while its discovery keeps succeeding: if the winner's discovery
|
|
||||||
fails transiently, the later duplicate becomes served again, so the
|
|
||||||
implementation behind a bound name can alternate across cache generations
|
|
||||||
with server health.
|
|
||||||
([#8135](https://github.com/bytedance/deer-flow/pull/8135))
|
|
||||||
|
|
||||||
- **channels:** WeCom no longer drops images between 2 MB and 10 MB. The
|
|
||||||
adapter capped image uploads at 2 MB — the voice limit in the same protocol —
|
|
||||||
so a larger chart was skipped before upload and the turn arrived without it.
|
|
||||||
The cap now matches the platform's 10 MB for images; the 20 MB file limit is
|
|
||||||
unchanged. ([#8174])
|
|
||||||
|
|
||||||
- **threads:** Deleting a conversation on Windows no longer fails with
|
|
||||||
`Failed to delete local thread data.` when the workspace holds a file whose
|
|
||||||
path exceeds `MAX_PATH`. The thread directory is now resolved through the
|
|
||||||
Windows extended-length namespace before removal.
|
|
||||||
([#8144](https://github.com/bytedance/deer-flow/pull/8144))
|
|
||||||
|
|
||||||
- **channels:** A WeCom group command that follows the bot mention now runs as a
|
|
||||||
command. WeCom keeps the mention inside the message text (`@RobotA /new`), and
|
|
||||||
the classifier required a bare leading `/`, so `/new` and `/status` sent in a
|
|
||||||
group became chat turns. Leading mentions are skipped for the command lookup
|
|
||||||
only; ordinary chat keeps them and `/connect` is unchanged. Mirrors the Feishu
|
|
||||||
and DingTalk handling. ([#8118])
|
|
||||||
|
|
||||||
- **subagents:** Apply tool allowlists and denylists to middleware-declared
|
|
||||||
tools even when global authorization is disabled, including the default
|
|
||||||
denial of `task`. Restricted subagents reject non-Sequence declarations
|
|
||||||
explicitly, and invalid declarations report middleware-specific configuration
|
|
||||||
guidance. Final graph assembly checks both explicit and middleware tools;
|
|
||||||
framework-generated discovery helpers retain their filtered catalogs.
|
|
||||||
([#8136](https://github.com/bytedance/deer-flow/pull/8136))
|
|
||||||
|
|
||||||
- **frontend:** **Share** in a sidebar chat's menu now copies a link to the
|
|
||||||
deployment you are using. On `localhost` and `127.0.0.1`, the default
|
|
||||||
`make dev` and Docker addresses, it copied a link to the hosted demo
|
|
||||||
(`deer-flow-v2.vercel.app`) instead. That link never opened the
|
|
||||||
conversation, and opening it sent the thread id to an outside host.
|
|
||||||
([#8126])
|
|
||||||
|
|
||||||
- **runtime:** In multi-worker mode (`run_ownership.heartbeat_enabled`), a
|
|
||||||
worker that had lost its run to a peer could put the row back. Its assembly
|
|
||||||
evidence write was an unconditional full-row upsert, so after a peer
|
|
||||||
reclaimed the run it reset the row to `running` and erased the takeover
|
|
||||||
error. The run could then finish as `success` after the peer had already
|
|
||||||
reported it failed, or be reclaimed a second time. The evidence is now merged
|
|
||||||
into the row's metadata only while the worker still holds a live lease on an
|
|
||||||
active run (`RunStore.merge_owned_metadata()`); otherwise the worker stops
|
|
||||||
the run before executing it. Custom `RunStore` implementations used with
|
|
||||||
heartbeats must implement that method. ([#8046])
|
|
||||||
|
|
||||||
- **subagents:** Concurrent creates for the same managed-subagent name in shared
|
|
||||||
file storage now preserve the winning definition and return HTTP 409 to the
|
|
||||||
losing request instead of reporting two successes and overwriting one definition.
|
|
||||||
Gateway startup also probes hard-link support and warns if the storage volume
|
|
||||||
cannot support managed subagent creation. ([#8132](https://github.com/bytedance/deer-flow/pull/8132))
|
|
||||||
|
|
||||||
- **agents:** A high-risk tool call without a usable id no longer wedges the
|
|
||||||
thread when `hitl_approval` is enabled. A provider that omits a tool-call id
|
|
||||||
(`None`) or repeats one within a response made the middleware pause with an
|
|
||||||
approval payload the Gateway can never validate: every resume answered 422
|
|
||||||
and every new run stayed 409-blocked, all-reject included. An empty id was
|
|
||||||
technically resumable but is denied as well, since strict providers reject it
|
|
||||||
when the conversation is replayed. Such calls are now removed from the model
|
|
||||||
response with a visible rejection note instead of pausing, and deny paths no
|
|
||||||
longer build an invalid `ToolMessage` for a null id. A call whose id an
|
|
||||||
ordinary call also carries is stripped before it can be answered by result,
|
|
||||||
so the next request cannot advertise the id twice with two results.
|
|
||||||
A pause stored before
|
|
||||||
this change stays unresolvable; in-place recovery is tracked in [#8071].
|
|
||||||
([#8069])
|
|
||||||
|
|
||||||
- **runtime:** A HITL rejection, including the all-reject escape after approval is
|
|
||||||
disabled or the gated tools change during a pause, now reaches the thread's
|
|
||||||
message history under the resume run instead of existing only in checkpoint
|
|
||||||
state. ([#8073](https://github.com/bytedance/deer-flow/pull/8073))
|
|
||||||
|
|
||||||
- **summarization:** vLLM compaction counts reasoning tokens replayed to the
|
|
||||||
model when checking the trigger and estimating how much history to keep.
|
|
||||||
([#8077](https://github.com/bytedance/deer-flow/pull/8077))
|
|
||||||
|
|
||||||
- **gateway:** Posting feedback twice for the same run now returns a 409
|
|
||||||
instead of a 500. `POST /api/threads/{id}/runs/{run_id}/feedback` allows one
|
|
||||||
feedback row per user and run; a retry, or feedback on another message of the
|
|
||||||
same run, hit that unique key and the Gateway failed with an unhandled
|
|
||||||
database error. The 409 points callers at `PUT` on the same route, which
|
|
||||||
updates the existing feedback. ([#8064])
|
|
||||||
|
|
||||||
- **frontend:** Reopening a conversation restores successfully handed-off shared
|
|
||||||
artifacts in the Artifacts sidebar. ([#8080](https://github.com/bytedance/deer-flow/pull/8080))
|
|
||||||
|
|
||||||
- **sandbox:** AIO commands from independent subagents now run concurrently when
|
|
||||||
they carry per-call environment variables, including Lark CLI commands.
|
|
||||||
Commands within one subagent remain ordered and client cleanup drains them.
|
|
||||||
|
|
||||||
- **sandbox:** BoxLite VMs recovered at startup now run health checks, commands,
|
|
||||||
and cleanup on the provider's event loop, avoiding sync/async SDK mismatches
|
|
||||||
that made healthy recovered VMs fail their first command. ([#8072])
|
|
||||||
|
|
||||||
- **uploads:** Re-uploading a file whose name collides with a compound archive
|
|
||||||
suffix no longer inserts the dedupe counter inside the suffix. A second
|
|
||||||
`report.tar.gz` is now named `report_1.tar.gz` (matching `report_1.tar` and
|
|
||||||
`report_1.txt`), instead of `report.tar_1.gz`, so the extension stays intact
|
|
||||||
for file-type detection. ([#8042](https://github.com/bytedance/deer-flow/pull/8042))
|
|
||||||
|
|
||||||
- **gateway:** Deleting a thread no longer reports success while its
|
|
||||||
conversation stays readable. When the checkpoint store failed during
|
|
||||||
`DELETE /api/threads/{id}`, the route logged it at DEBUG, removed the
|
|
||||||
thread's metadata row and returned success; any signed-in user could then
|
|
||||||
read the leftover title and messages through `GET /threads/{id}`, `/state`,
|
|
||||||
`/history` and `/goal`, and the owner could not retry. The delete now
|
|
||||||
returns a 500 and keeps the thread, so the owner can delete it again.
|
|
||||||
([#7992])
|
|
||||||
|
|
||||||
- **gateway:** Creating a thread with a caller-supplied `thread_id` that
|
|
||||||
already belongs to another user now returns a non-disclosing 404
|
|
||||||
(`Thread not found`) instead of a 500. The duplicate-key recovery path reads
|
|
||||||
the surviving row back bypassing the owner scope so the cross-owner case
|
|
||||||
mirrors `ThreadOwnershipConflictError`; a duplicate key with no surviving row
|
|
||||||
is still surfaced as a 500. ([#8057])
|
|
||||||
|
|
||||||
- **models:** Cancelling a Codex-backed run now closes its upstream HTTP/SSE
|
|
||||||
connection, including while waiting for response headers, and retry waits
|
|
||||||
are cancellable. Local resources are released; the remote service may still
|
|
||||||
continue generation. ([#8053](https://github.com/bytedance/deer-flow/pull/8053))
|
|
||||||
|
|
||||||
- **middleware:** With PII redaction enabled, a long run of text without
|
|
||||||
spaces (such as a pasted hex dump) no longer stalls the Gateway. The email
|
|
||||||
detector was quadratic in the length of such a run, and redaction re-scans
|
|
||||||
every user message on every model call, so one 100 KB paste blocked every
|
|
||||||
thread for about 6 s per model call. The scan is now linear, and it redacts
|
|
||||||
exactly the same addresses as before. ([#8006])
|
|
||||||
|
|
||||||
- **agents:** Token budgets count separate accepted model calls even when a
|
|
||||||
provider reuses message IDs, so repeated IDs no longer bypass the configured
|
|
||||||
cap. Hard stops preserve already-completed structured-output tool results.
|
|
||||||
([#8050](https://github.com/bytedance/deer-flow/pull/8050))
|
|
||||||
|
|
||||||
- **subagents:** Managed subagent creation now rejects Windows reserved device
|
|
||||||
names (`CON`, `PRN`, `AUX`, `NUL`, `COM1-9`, `LPT1-9`) at the shared
|
|
||||||
`normalize_managed_subagent_name(..., for_creation=True)` validator and
|
|
||||||
Gateway `/api/subagents` boundary before attempting filesystem writes,
|
|
||||||
preventing `OSError` failures on Windows hosts while keeping existing
|
|
||||||
read and delete operations permissive.
|
|
||||||
|
|
||||||
- **scheduler:** A stored `schedule_spec` that is JSON null, a list or a string
|
|
||||||
no longer aborts the scheduler poll. `stored_next_run_at` caught the value
|
|
||||||
errors from a dropped zone or a newer cron parser, but a non-object spec
|
|
||||||
fails on `.get`; the launch-failure handler recomputed it again, so the
|
|
||||||
`AttributeError` escaped `run_once`'s claimed-task loop. Such a task is now
|
|
||||||
paused with the reason, like the other unusable schedules. The poll loop also
|
|
||||||
isolates one task's dispatch failure from the rest of its claimed batch, so a
|
|
||||||
future per-record raise costs one tick instead of every other claimed task.
|
|
||||||
([#7998])
|
|
||||||
|
|
||||||
- **scheduler:** Exceptions escaping the queue-timeout sweep or the
|
|
||||||
queued-occurrence drain are now contained to that occurrence. The
|
|
||||||
scheduler logs the failed row and continues with the remaining queue
|
|
||||||
and the tasks claimed in the same poll. Expired rows are excluded before
|
|
||||||
owner ranking and the batch limit, and a launch claim enforces the same
|
|
||||||
cutoff, so a row left queued by a failed sweep neither starts nor crowds
|
|
||||||
healthy work out of the queue while timeout recovery retries it. ([#7998])
|
|
||||||
|
|
||||||
- **scheduler:** One scheduled task whose saved schedule can no longer be
|
|
||||||
computed no longer stops every other task. When a stored timezone or cron
|
|
||||||
expression stopped working (a zone missing from the time zone database
|
|
||||||
after an upgrade, a cron expression a newer croniter rejects), recomputing
|
|
||||||
its next run raised inside the scheduler poll; once its launch lease
|
|
||||||
expired, every later poll failed before admitting any due task. That task
|
|
||||||
is now paused with the reason as its last error, and resuming it reports
|
|
||||||
the problem until the schedule is fixed. ([#7993])
|
|
||||||
|
|
||||||
- **models:** A Claude model using a Claude Code OAuth token no longer fails
|
|
||||||
every request once the request carries `betas`. The OAuth betas
|
|
||||||
(`oauth-2025-04-20`, …) were only the client's default `anthropic-beta`
|
|
||||||
header, and the SDK sends a request's `betas` as that header instead, so a
|
|
||||||
configured `betas` list (even `[]`), per-call `betas`, or a server tool for
|
|
||||||
which langchain-anthropic adds a beta left them out and Anthropic rejected
|
|
||||||
the token. The OAuth betas now lead any request `betas` and any
|
|
||||||
`anthropic-beta` in `extra_headers`.
|
|
||||||
([#8061](https://github.com/bytedance/deer-flow/pull/8061))
|
|
||||||
|
|
||||||
- **community:** A search or fetch API key that ends in a newline no longer
|
|
||||||
leaks into the conversation. GroundRoute (`GROUNDROUTE_API_KEY`), InfoQuest,
|
|
||||||
Tavily, Exa, Jina, Firecrawl, fastCRW and Crawl4AI put the key into an HTTP
|
|
||||||
header as read, so a key from a file (a Kubernetes Secret created with
|
|
||||||
`--from-file`) made the transport fail with `Illegal header value b'Bearer
|
|
||||||
<key>\n'`. That message, key included, became the tool result the model and
|
|
||||||
the conversation history kept, and four of them also logged it. Keys now go
|
|
||||||
through `provider_credentials.resolve_api_key` like the other providers', and
|
|
||||||
a blank configured key falls back to the environment for Tavily, Exa,
|
|
||||||
Firecrawl and fastCRW. fastCRW without a key of its own no longer lets the
|
|
||||||
Firecrawl SDK send `FIRECRAWL_API_KEY` to the fastCRW endpoint, and a blank
|
|
||||||
`TAVILY_API_KEY`, `EXA_API_KEY` or `FIRECRAWL_API_KEY` now counts as missing
|
|
||||||
instead of reaching the SDK as is. ([#7981])
|
|
||||||
|
|
||||||
- **community:** Browserless is the last search/fetch provider that read its
|
|
||||||
token raw. A `BROWSERLESS_TOKEN` or configured `token` from a file kept its
|
|
||||||
trailing newline and became the auth query parameter, so every `web_fetch`
|
|
||||||
and `web_capture` request failed authentication while the key was correct.
|
|
||||||
Both now go through `provider_credentials.resolve_api_key`, and only an
|
|
||||||
omitted `token` key falls back to the environment: an explicit `""` or
|
|
||||||
`null` keeps opting a self-hosted capture endpoint out of authentication.
|
|
||||||
([#8043])
|
|
||||||
|
|
||||||
- **docker:** Images built from a checkout no longer contain its local
|
|
||||||
configuration. `.dockerignore` did not exclude `config.yaml`,
|
|
||||||
`config.yaml.bak`, `extensions_config.json` or `mcp_config.json`, so a
|
|
||||||
checkout that keeps them in the legacy `backend/` location baked its model
|
|
||||||
API keys and MCP credentials into every gateway image layer through
|
|
||||||
`COPY backend ./backend`. `frontend/.env.local` reached the frontend image the
|
|
||||||
same way. They are now excluded at the root and in `backend/`, along with
|
|
||||||
`.env*.local` at any depth, `frontend/*.pem`, and the credential files the
|
|
||||||
extension manager refuses inside extension snapshots. A locally built
|
|
||||||
frontend image no longer inlines `NEXT_PUBLIC_*` values from
|
|
||||||
`frontend/.env.local`; the Docker stack's single nginx origin needs none of
|
|
||||||
them. Published CI images were not affected; rebuild local images that were
|
|
||||||
pushed or shared. ([#7958])
|
|
||||||
|
|
||||||
- **agents:** Unfinished todos no longer trigger additional model calls or
|
|
||||||
tool execution after a loop-detection, run token-budget, provider-safety, or
|
|
||||||
model output-length stop. Todo state is preserved so users can continue in a
|
|
||||||
later run. ([#7953])
|
|
||||||
|
|
||||||
- **runtime:** A thread can accept new runs and mutations after a transient
|
|
||||||
database failure leaves a completed operation's reservation in place. The
|
|
||||||
next request for the thread on the same Gateway worker retries the release,
|
|
||||||
including in single-worker deployments with lease heartbeats disabled.
|
|
||||||
([#8068](https://github.com/bytedance/deer-flow/pull/8068))
|
|
||||||
|
|
||||||
- **channels:** Long Feishu/Lark answers are no longer lost. Card text was
|
|
||||||
unbounded and ASCII-escaped, so an answer of about 4,400 CJK characters made
|
|
||||||
the card request exceed Feishu's 30 KB limit (`230025`); the final patch and
|
|
||||||
its reply fallback were both rejected, and the running card stayed on its
|
|
||||||
last streamed text or a bare `▉`. Cards now send CJK unescaped and are
|
|
||||||
budgeted by the bytes they add to the request. A streamed update shows what
|
|
||||||
fits, and a longer final answer continues in up to 10 cards replied to the
|
|
||||||
same message, split at line breaks; a retry resumes after the cards already
|
|
||||||
delivered. ([#7798])
|
|
||||||
|
|
||||||
- **channels:** WeCom group messages now have their own conversation. Every
|
|
||||||
message used the sender's userid as both chat and topic ID, so a member's
|
|
||||||
private chat and every group they wrote in shared one DeerFlow thread
|
|
||||||
wherever no group binding kept them apart: in auth-disabled mode, with
|
|
||||||
`require_bound_identity: false`, and under legacy group bindings. A group
|
|
||||||
reply could then draw on the member's private chat or another group, and a
|
|
||||||
reply sent without its frame went to the private chat. Groups are now keyed
|
|
||||||
by `chatid` with one thread per member, and private chats keep their
|
|
||||||
threads. A group bound with its own `/connect` keeps its thread; a legacy
|
|
||||||
group binding, which covers every group of the bot, starts a new one in
|
|
||||||
each group. ([#7794])
|
|
||||||
|
|
||||||
- **memory:** DeerMem rejects backups with duplicate fact IDs before replacing
|
|
||||||
saved memory, regardless of entry order. ([#8151])
|
|
||||||
|
|
||||||
- **memory:** Importing a backup without a `facts` array now returns HTTP 422
|
|
||||||
and preserves saved memory. An explicit empty array still clears the selected
|
|
||||||
fact bucket. ([#8131])
|
|
||||||
|
|
||||||
- **memory:** An agent-scoped memory write whose `agent_name` is a Windows
|
|
||||||
reserved device name (`NUL`, `CON`, `COM1`, ...) is now rejected with a `422`
|
|
||||||
on the two routes that create that scope, `POST /memory/facts` and
|
|
||||||
`POST /memory/import`. DeerMem resolves the scope to
|
|
||||||
`{storage}/users/{uid}/agents/{name}/facts` and creates it on the first write;
|
|
||||||
the character-class check admitted such names, so the write raised
|
|
||||||
`FileNotFoundError` and the endpoint answered an opaque `500`. The check reuses
|
|
||||||
`windows_incompatible_segment()` and applies on every platform. Read and delete
|
|
||||||
routes stay permissive, so a scope with a reserved-name identifier that an
|
|
||||||
older release stored on a Linux host can still be exported and cleared.
|
|
||||||
([#7941])
|
|
||||||
|
|
||||||
- **skills:** A custom skill name that is a Windows reserved device name
|
|
||||||
(`nul`, `con`, `com1`, ...) is now rejected when the skill is created —
|
|
||||||
`skill_manage` `create` and `.skill` archive install — on every platform. The
|
|
||||||
hyphen-case grammar admitted such names, and the skill name becomes a
|
|
||||||
directory under `custom/`, so creating one "succeeded" on Windows without
|
|
||||||
producing a directory and the first write inside it failed with
|
|
||||||
`FileNotFoundError`; on Linux the install succeeded but could never be
|
|
||||||
copied, mounted or exported to a Windows host. The archive path also refuses
|
|
||||||
member path segments that cannot be created on every host, matching support
|
|
||||||
file writes. Reading and deleting an existing skill stay permissive, so a
|
|
||||||
skill an older release created on a Linux host can still be inspected and
|
|
||||||
removed. ([#7945])
|
|
||||||
|
|
||||||
- **sandbox:** Fresh lazy AIO runs with an explicit authorized skill view no
|
|
||||||
longer create or acquire a sandbox before the first sandbox-backed tool
|
|
||||||
call. Existing sandbox bindings and providers that require skill uploads
|
|
||||||
retain their eager synchronization behavior. ([#7782])
|
|
||||||
|
|
||||||
- **parsing:** Follow-up suggestions and the `/goal` evaluator keep a literal
|
|
||||||
`<think>` tag that appears inside a JSON string value, such as an answer
|
|
||||||
quoting the tag, while reasoning around the JSON is still removed. An
|
|
||||||
unclosed literal tag inside a string no longer truncates the document and
|
|
||||||
fails parsing, and reasoning outside the payload still strips and truncates
|
|
||||||
as before. ([#7907])
|
|
||||||
|
|
||||||
- **gateway:** A caller-supplied thread ID that is a Windows reserved device
|
|
||||||
name (`NUL`, `CON`, `COM1`, ...) is now rejected at the shared
|
|
||||||
`validate_thread_id()` boundary. Such IDs previously passed validation but
|
|
||||||
failed workspace creation on Windows hosts with `FileNotFoundError`. The
|
|
||||||
check reuses `windows_incompatible_segment()` and applies on every platform,
|
|
||||||
so threads with reserved-name IDs created on Linux hosts will fail
|
|
||||||
validation on read/resume/upload routes after upgrade (deletion still
|
|
||||||
works). ([#7892])
|
|
||||||
|
|
||||||
- **extensions:** Context-compaction observers now learn which task a
|
|
||||||
compaction happened in. Inside a lead run or subagent they receive that
|
|
||||||
task's store instead of a detached one, and `CompactionEvent` gains
|
|
||||||
`task_id`, `run_id` and `thread_id` (default `None`), copied from the
|
|
||||||
`TaskInfo` the host now seeds into every task store even when no lifecycle
|
|
||||||
contributor is registered. A manual `/compact` still gets a detached store
|
|
||||||
and `None` ids. `deerflow-extension-api` is now 0.2.10; existing
|
|
||||||
extensions keep working. ([#7831])
|
|
||||||
|
|
||||||
- **channels:** A DingTalk reply in AI card mode (`card_template_id`) is no
|
|
||||||
longer lost when a card request fails, for example without card permission
|
|
||||||
or during a DingTalk outage. dingtalk-stream logs a failed card request and
|
|
||||||
returns as if it succeeded, so neither the text running reply nor the
|
|
||||||
`sampleMarkdown` fallback for the final answer ever ran. The channel now
|
|
||||||
sends card requests itself and falls back when one fails. ([#7781])
|
|
||||||
|
|
||||||
- **harness:** Retrying a failed item no longer reopens a cancelled durable
|
|
||||||
subagent batch. `retry_item` kept only a paused batch as it was, so a failed
|
|
||||||
item of a cancelled batch put the batch back to `queued`, the worker claimed
|
|
||||||
and ran the item, and the batch ended `completed`. The retry route now
|
|
||||||
answers 409 for a cancelled batch, including when the cancel commits while
|
|
||||||
the retry is in flight on SQLite, and the workspace no longer offers Retry
|
|
||||||
there. Retrying in a completed, failed, running or paused batch is
|
|
||||||
unchanged. ([#7792])
|
|
||||||
|
|
||||||
- **gateway:** A run that names a missing or invalid custom agent now says so.
|
|
||||||
Messaging a chat whose custom agent was deleted, or passing a malformed
|
|
||||||
`agent_name`, returned the 422 detail "knowledge_scope assistant configuration
|
|
||||||
could not be resolved" even when no knowledge scope was involved. The 422 now
|
|
||||||
reports `Agent '<name>' not found`, an invalid `agent_name`, or an agent
|
|
||||||
config that could not be loaded. ([#8087])
|
|
||||||
|
|
||||||
- **agents:** `PUT /api/agents/{name}` with `"description": null` now resets the
|
|
||||||
description to empty. It used to store the null, which the agent config
|
|
||||||
rejects, so the agent then failed every load: the update, GET and repair
|
|
||||||
updates returned 500, the file backend dropped it from the agent list, the
|
|
||||||
database backend failed the owner's whole list, and chats with it failed.
|
|
||||||
Agents already stored with a null description load again. ([#7638])
|
|
||||||
|
|
||||||
- **sandbox:** Cancelled native Runs now stop their owned AIO and POSIX local
|
|
||||||
bash processes, including ordinary `nohup` children, before releasing the
|
|
||||||
execution lease. Concurrent work and background services from successful
|
|
||||||
earlier turns remain running. ([#7772])
|
|
||||||
|
|
||||||
- **agents:** A clarification no longer overwrites an earlier question when the
|
|
||||||
provider reuses a tool-call ID across turns (for example `call_0` on every
|
|
||||||
response). Its message ID, which is also the human-input `request_id`, named
|
|
||||||
only the call ID, so the later question replaced the earlier one in place and
|
|
||||||
the later call was left without a result. The ID now also names the
|
|
||||||
AIMessage that issued the call; a retry of the same call keeps its ID.
|
|
||||||
([#7762])
|
|
||||||
|
|
||||||
- **projects:** The trash retention sweep no longer deletes live document
|
|
||||||
originals when the data directory, `users/`, a user's directory or a
|
|
||||||
user's `projects/` is a symlink. It protected each document by its resolved
|
|
||||||
path but walked the unresolved tree, so no document matched and every
|
|
||||||
original older than 24 hours was removed as an orphan, at Gateway startup
|
|
||||||
and from `GET /api/trash/documents`. ([#7732])
|
|
||||||
|
|
||||||
- **chat:** Cancelling a queued input after dispatch starts no longer
|
|
||||||
prevents subsequent queued inputs from executing. ([#7746])
|
|
||||||
|
|
||||||
- **dev:** `scripts/dev_multi_instance.sh logs all` now prints the nginx logs
|
|
||||||
as well as both Gateway logs; it printed only the Gateway logs, the same as
|
|
||||||
`logs` with no argument. `logs nginx` and `logs all` also include `nginx.log`,
|
|
||||||
the nginx start command's own output, which no option printed before.
|
|
||||||
([#7723])
|
|
||||||
|
|
||||||
- **memory:** `GET /api/memory/config` and `/api/memory/status` now mask
|
|
||||||
credential values in `backend_config` (`api_key`, tokens, passwords,
|
|
||||||
`Authorization` headers, at any depth) as `***`. They returned the config as
|
|
||||||
loaded, with `$VAR` references already resolved, so a DeerMem extraction
|
|
||||||
model key or a Honcho `api_key` reached every caller with `memory:read` in
|
|
||||||
plaintext, which with authorization disabled (the default) is every
|
|
||||||
signed-in user. Non-secret knobs, `api_key_env` names and unset values are
|
|
||||||
unchanged. ([#7665])
|
|
||||||
- **gateway:** `GET /threads/{thread_id}/token-usage` now resolves the live
|
|
||||||
config in a worker thread. `build_context_usage` called the accessor inline,
|
|
||||||
and `get_app_config()` revalidates on every call — it digests the whole
|
|
||||||
`config.yaml`, then `extensions_config.json` — so a config edit landing just
|
|
||||||
before a thread view stalled the Gateway event loop that runs, streams and
|
|
||||||
every other thread share. The response is unchanged. ([#7727])
|
|
||||||
|
|
||||||
- **community:** The shared SSRF guard now returns its documented `Error: ...`
|
|
||||||
rejection for malformed URLs instead of raising `ValueError`.
|
|
||||||
`validate_public_http_url` and `validate_delegated_backend_url` called
|
|
||||||
`urlparse()` unguarded, so an authority it rejects, such as the unmatched
|
|
||||||
IPv6 bracket in `http://[::1/api`, escaped to the caller. That surfaced as an
|
|
||||||
unhandled 500 from the personal MCP router and as generic exception text in
|
|
||||||
the crawl4ai and fastCRW `web_fetch` tools. Both validators now reject any
|
|
||||||
malformed authority before the scheme check and the private-address escape
|
|
||||||
hatches, including a raw bracket in userinfo, which RFC 3986 forbids and
|
|
||||||
Python versions disagree on. This is a fail-closed contract fix plus
|
|
||||||
version-drift hardening: no host reachable through a well-formed URL becomes
|
|
||||||
blocked. ([#8079])
|
|
||||||
|
|
||||||
## [2.2.0] — 2026-10-29
|
## [2.2.0] — 2026-10-29
|
||||||
|
|
||||||
This section accumulates work toward the **2.2.0** milestone
|
This section accumulates work toward the **2.2.0** milestone
|
||||||
@ -14323,7 +13838,8 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
- **ci:** Consolidate PR/issue labeling and fix the reviewing-job crash and
|
- **ci:** Consolidate PR/issue labeling and fix the reviewing-job crash and
|
||||||
label thrash. ([#3455])
|
label thrash. ([#3455])
|
||||||
|
|
||||||
[Unreleased]: https://github.com/bytedance/deer-flow/compare/v2.1.0...HEAD
|
[Unreleased]: https://github.com/bytedance/deer-flow/compare/v2.2.0...HEAD
|
||||||
|
[2.2.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.2.0
|
||||||
[2.1.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.1.0
|
[2.1.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.1.0
|
||||||
[2.0.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.0.0
|
[2.0.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.0.0
|
||||||
|
|
||||||
@ -15165,7 +14681,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5217]: https://github.com/bytedance/deer-flow/pull/5217
|
[#5217]: https://github.com/bytedance/deer-flow/pull/5217
|
||||||
[#5219]: https://github.com/bytedance/deer-flow/pull/5219
|
[#5219]: https://github.com/bytedance/deer-flow/pull/5219
|
||||||
[#5221]: https://github.com/bytedance/deer-flow/pull/5221
|
[#5221]: https://github.com/bytedance/deer-flow/pull/5221
|
||||||
[#5224]: https://github.com/bytedance/deer-flow/commit/572744975d369fda88168aaa7b481282f939affe
|
[#5224]: https://github.com/bytedance/deer-flow/pull/5224
|
||||||
[#5225]: https://github.com/bytedance/deer-flow/pull/5225
|
[#5225]: https://github.com/bytedance/deer-flow/pull/5225
|
||||||
[#5227]: https://github.com/bytedance/deer-flow/pull/5227
|
[#5227]: https://github.com/bytedance/deer-flow/pull/5227
|
||||||
[#5228]: https://github.com/bytedance/deer-flow/pull/5228
|
[#5228]: https://github.com/bytedance/deer-flow/pull/5228
|
||||||
@ -15174,9 +14690,9 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5236]: https://github.com/bytedance/deer-flow/pull/5236
|
[#5236]: https://github.com/bytedance/deer-flow/pull/5236
|
||||||
[#5238]: https://github.com/bytedance/deer-flow/pull/5238
|
[#5238]: https://github.com/bytedance/deer-flow/pull/5238
|
||||||
[#5239]: https://github.com/bytedance/deer-flow/pull/5239
|
[#5239]: https://github.com/bytedance/deer-flow/pull/5239
|
||||||
[#5244]: https://github.com/bytedance/deer-flow/commit/9ce6fdcb2252ea005b4fa98ca50bd3857e1f681a
|
[#5244]: https://github.com/bytedance/deer-flow/pull/5244
|
||||||
[#5245]: https://github.com/bytedance/deer-flow/pull/5245
|
[#5245]: https://github.com/bytedance/deer-flow/pull/5245
|
||||||
[#5247]: https://github.com/bytedance/deer-flow/commit/fa89a125267be4213009376ec0e232a0c2411ca8
|
[#5247]: https://github.com/bytedance/deer-flow/pull/5247
|
||||||
[#5249]: https://github.com/bytedance/deer-flow/pull/5249
|
[#5249]: https://github.com/bytedance/deer-flow/pull/5249
|
||||||
[#5251]: https://github.com/bytedance/deer-flow/pull/5251
|
[#5251]: https://github.com/bytedance/deer-flow/pull/5251
|
||||||
[#5254]: https://github.com/bytedance/deer-flow/pull/5254
|
[#5254]: https://github.com/bytedance/deer-flow/pull/5254
|
||||||
@ -15233,7 +14749,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5367]: https://github.com/bytedance/deer-flow/pull/5367
|
[#5367]: https://github.com/bytedance/deer-flow/pull/5367
|
||||||
[#5369]: https://github.com/bytedance/deer-flow/pull/5369
|
[#5369]: https://github.com/bytedance/deer-flow/pull/5369
|
||||||
[#5371]: https://github.com/bytedance/deer-flow/pull/5371
|
[#5371]: https://github.com/bytedance/deer-flow/pull/5371
|
||||||
[#5373]: https://github.com/bytedance/deer-flow/commit/d5ae3882b6708fa9af7289a0c3c0be3eee157203
|
[#5373]: https://github.com/bytedance/deer-flow/pull/5373
|
||||||
[#5374]: https://github.com/bytedance/deer-flow/pull/5374
|
[#5374]: https://github.com/bytedance/deer-flow/pull/5374
|
||||||
[#5375]: https://github.com/bytedance/deer-flow/pull/5375
|
[#5375]: https://github.com/bytedance/deer-flow/pull/5375
|
||||||
[#5377]: https://github.com/bytedance/deer-flow/pull/5377
|
[#5377]: https://github.com/bytedance/deer-flow/pull/5377
|
||||||
@ -15260,7 +14776,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5408]: https://github.com/bytedance/deer-flow/pull/5408
|
[#5408]: https://github.com/bytedance/deer-flow/pull/5408
|
||||||
[#5410]: https://github.com/bytedance/deer-flow/pull/5410
|
[#5410]: https://github.com/bytedance/deer-flow/pull/5410
|
||||||
[#5411]: https://github.com/bytedance/deer-flow/pull/5411
|
[#5411]: https://github.com/bytedance/deer-flow/pull/5411
|
||||||
[#5413]: https://github.com/bytedance/deer-flow/commit/daf32cd24632f2cc7465aeb62f2a1e1e768eceb2
|
[#5413]: https://github.com/bytedance/deer-flow/pull/5413
|
||||||
[#5415]: https://github.com/bytedance/deer-flow/pull/5415
|
[#5415]: https://github.com/bytedance/deer-flow/pull/5415
|
||||||
[#5416]: https://github.com/bytedance/deer-flow/pull/5416
|
[#5416]: https://github.com/bytedance/deer-flow/pull/5416
|
||||||
[#5418]: https://github.com/bytedance/deer-flow/pull/5418
|
[#5418]: https://github.com/bytedance/deer-flow/pull/5418
|
||||||
@ -15443,8 +14959,8 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5736]: https://github.com/bytedance/deer-flow/pull/5736
|
[#5736]: https://github.com/bytedance/deer-flow/pull/5736
|
||||||
[#5738]: https://github.com/bytedance/deer-flow/pull/5738
|
[#5738]: https://github.com/bytedance/deer-flow/pull/5738
|
||||||
[#5739]: https://github.com/bytedance/deer-flow/pull/5739
|
[#5739]: https://github.com/bytedance/deer-flow/pull/5739
|
||||||
[#5740]: https://github.com/bytedance/deer-flow/commit/e7128dc7d6b4d49153b10f7276f010384ec4e6af
|
[#5740]: https://github.com/bytedance/deer-flow/pull/5740
|
||||||
[#5741]: https://github.com/bytedance/deer-flow/commit/363b0fc1b4a5bd6c142d1f1df9fb9743cca550a3
|
[#5741]: https://github.com/bytedance/deer-flow/pull/5741
|
||||||
[#5745]: https://github.com/bytedance/deer-flow/pull/5745
|
[#5745]: https://github.com/bytedance/deer-flow/pull/5745
|
||||||
[#5748]: https://github.com/bytedance/deer-flow/pull/5748
|
[#5748]: https://github.com/bytedance/deer-flow/pull/5748
|
||||||
[#5750]: https://github.com/bytedance/deer-flow/pull/5750
|
[#5750]: https://github.com/bytedance/deer-flow/pull/5750
|
||||||
@ -15457,7 +14973,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5760]: https://github.com/bytedance/deer-flow/pull/5760
|
[#5760]: https://github.com/bytedance/deer-flow/pull/5760
|
||||||
[#5761]: https://github.com/bytedance/deer-flow/pull/5761
|
[#5761]: https://github.com/bytedance/deer-flow/pull/5761
|
||||||
[#5762]: https://github.com/bytedance/deer-flow/pull/5762
|
[#5762]: https://github.com/bytedance/deer-flow/pull/5762
|
||||||
[#5763]: https://github.com/bytedance/deer-flow/commit/39351315bf5d096fc84ad8307d3702b6aa5b3e24
|
[#5763]: https://github.com/bytedance/deer-flow/pull/5763
|
||||||
[#5767]: https://github.com/bytedance/deer-flow/pull/5767
|
[#5767]: https://github.com/bytedance/deer-flow/pull/5767
|
||||||
[#5769]: https://github.com/bytedance/deer-flow/pull/5769
|
[#5769]: https://github.com/bytedance/deer-flow/pull/5769
|
||||||
[#5775]: https://github.com/bytedance/deer-flow/pull/5775
|
[#5775]: https://github.com/bytedance/deer-flow/pull/5775
|
||||||
@ -15525,7 +15041,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5880]: https://github.com/bytedance/deer-flow/pull/5880
|
[#5880]: https://github.com/bytedance/deer-flow/pull/5880
|
||||||
[#5881]: https://github.com/bytedance/deer-flow/pull/5881
|
[#5881]: https://github.com/bytedance/deer-flow/pull/5881
|
||||||
[#5883]: https://github.com/bytedance/deer-flow/pull/5883
|
[#5883]: https://github.com/bytedance/deer-flow/pull/5883
|
||||||
[#5884]: https://github.com/bytedance/deer-flow/commit/e8cb5e9c4f5f274401eeb555aff62e713035cbf4
|
[#5884]: https://github.com/bytedance/deer-flow/pull/5884
|
||||||
[#5888]: https://github.com/bytedance/deer-flow/pull/5888
|
[#5888]: https://github.com/bytedance/deer-flow/pull/5888
|
||||||
[#5890]: https://github.com/bytedance/deer-flow/pull/5890
|
[#5890]: https://github.com/bytedance/deer-flow/pull/5890
|
||||||
[#5892]: https://github.com/bytedance/deer-flow/pull/5892
|
[#5892]: https://github.com/bytedance/deer-flow/pull/5892
|
||||||
@ -15587,9 +15103,9 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#5999]: https://github.com/bytedance/deer-flow/pull/5999
|
[#5999]: https://github.com/bytedance/deer-flow/pull/5999
|
||||||
[#6000]: https://github.com/bytedance/deer-flow/pull/6000
|
[#6000]: https://github.com/bytedance/deer-flow/pull/6000
|
||||||
[#6008]: https://github.com/bytedance/deer-flow/pull/6008
|
[#6008]: https://github.com/bytedance/deer-flow/pull/6008
|
||||||
[#6009]: https://github.com/bytedance/deer-flow/commit/b87ebd940227465d611084e5bf49582aaeb4e9bd
|
[#6009]: https://github.com/bytedance/deer-flow/pull/6009
|
||||||
[#6013]: https://github.com/bytedance/deer-flow/pull/6013
|
[#6013]: https://github.com/bytedance/deer-flow/pull/6013
|
||||||
[#6015]: https://github.com/bytedance/deer-flow/commit/f87a04e581829ff35afd4fa228240071b486192f
|
[#6015]: https://github.com/bytedance/deer-flow/pull/6015
|
||||||
[#6017]: https://github.com/bytedance/deer-flow/pull/6017
|
[#6017]: https://github.com/bytedance/deer-flow/pull/6017
|
||||||
[#6018]: https://github.com/bytedance/deer-flow/pull/6018
|
[#6018]: https://github.com/bytedance/deer-flow/pull/6018
|
||||||
[#6020]: https://github.com/bytedance/deer-flow/pull/6020
|
[#6020]: https://github.com/bytedance/deer-flow/pull/6020
|
||||||
@ -15613,7 +15129,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#6060]: https://github.com/bytedance/deer-flow/pull/6060
|
[#6060]: https://github.com/bytedance/deer-flow/pull/6060
|
||||||
[#6062]: https://github.com/bytedance/deer-flow/pull/6062
|
[#6062]: https://github.com/bytedance/deer-flow/pull/6062
|
||||||
[#6063]: https://github.com/bytedance/deer-flow/pull/6063
|
[#6063]: https://github.com/bytedance/deer-flow/pull/6063
|
||||||
[#6066]: https://github.com/bytedance/deer-flow/commit/73fdbcfdf4b5f4037b62bc2ce9d8b60ec04e69f8
|
[#6066]: https://github.com/bytedance/deer-flow/pull/6066
|
||||||
[#6067]: https://github.com/bytedance/deer-flow/pull/6067
|
[#6067]: https://github.com/bytedance/deer-flow/pull/6067
|
||||||
[#6068]: https://github.com/bytedance/deer-flow/pull/6068
|
[#6068]: https://github.com/bytedance/deer-flow/pull/6068
|
||||||
[#6069]: https://github.com/bytedance/deer-flow/pull/6069
|
[#6069]: https://github.com/bytedance/deer-flow/pull/6069
|
||||||
@ -15639,7 +15155,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#6105]: https://github.com/bytedance/deer-flow/pull/6105
|
[#6105]: https://github.com/bytedance/deer-flow/pull/6105
|
||||||
[#6107]: https://github.com/bytedance/deer-flow/pull/6107
|
[#6107]: https://github.com/bytedance/deer-flow/pull/6107
|
||||||
[#6109]: https://github.com/bytedance/deer-flow/pull/6109
|
[#6109]: https://github.com/bytedance/deer-flow/pull/6109
|
||||||
[#6112]: https://github.com/bytedance/deer-flow/commit/52b21d13ac6e674a19cb76013d097e73f108c994
|
[#6112]: https://github.com/bytedance/deer-flow/pull/6112
|
||||||
[#6113]: https://github.com/bytedance/deer-flow/pull/6113
|
[#6113]: https://github.com/bytedance/deer-flow/pull/6113
|
||||||
[#6115]: https://github.com/bytedance/deer-flow/pull/6115
|
[#6115]: https://github.com/bytedance/deer-flow/pull/6115
|
||||||
[#6117]: https://github.com/bytedance/deer-flow/pull/6117
|
[#6117]: https://github.com/bytedance/deer-flow/pull/6117
|
||||||
@ -15740,7 +15256,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#6298]: https://github.com/bytedance/deer-flow/pull/6298
|
[#6298]: https://github.com/bytedance/deer-flow/pull/6298
|
||||||
[#6299]: https://github.com/bytedance/deer-flow/pull/6299
|
[#6299]: https://github.com/bytedance/deer-flow/pull/6299
|
||||||
[#6302]: https://github.com/bytedance/deer-flow/pull/6302
|
[#6302]: https://github.com/bytedance/deer-flow/pull/6302
|
||||||
[#6304]: https://github.com/bytedance/deer-flow/commit/b85e4ae74bb407e1e2396243a8a6e8f5b2420e3f
|
[#6304]: https://github.com/bytedance/deer-flow/pull/6304
|
||||||
[#6305]: https://github.com/bytedance/deer-flow/pull/6305
|
[#6305]: https://github.com/bytedance/deer-flow/pull/6305
|
||||||
[#6306]: https://github.com/bytedance/deer-flow/pull/6306
|
[#6306]: https://github.com/bytedance/deer-flow/pull/6306
|
||||||
[#6307]: https://github.com/bytedance/deer-flow/pull/6307
|
[#6307]: https://github.com/bytedance/deer-flow/pull/6307
|
||||||
@ -16124,7 +15640,6 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#7017]: https://github.com/bytedance/deer-flow/pull/7017
|
[#7017]: https://github.com/bytedance/deer-flow/pull/7017
|
||||||
[#7018]: https://github.com/bytedance/deer-flow/pull/7018
|
[#7018]: https://github.com/bytedance/deer-flow/pull/7018
|
||||||
[#7020]: https://github.com/bytedance/deer-flow/pull/7020
|
[#7020]: https://github.com/bytedance/deer-flow/pull/7020
|
||||||
[#7022]: https://github.com/bytedance/deer-flow/issues/7022
|
|
||||||
[#7023]: https://github.com/bytedance/deer-flow/pull/7023
|
[#7023]: https://github.com/bytedance/deer-flow/pull/7023
|
||||||
[#7024]: https://github.com/bytedance/deer-flow/pull/7024
|
[#7024]: https://github.com/bytedance/deer-flow/pull/7024
|
||||||
[#7031]: https://github.com/bytedance/deer-flow/pull/7031
|
[#7031]: https://github.com/bytedance/deer-flow/pull/7031
|
||||||
@ -16397,47 +15912,3 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
|||||||
[#7643]: https://github.com/bytedance/deer-flow/pull/7643
|
[#7643]: https://github.com/bytedance/deer-flow/pull/7643
|
||||||
[#7651]: https://github.com/bytedance/deer-flow/pull/7651
|
[#7651]: https://github.com/bytedance/deer-flow/pull/7651
|
||||||
[#7659]: https://github.com/bytedance/deer-flow/pull/7659
|
[#7659]: https://github.com/bytedance/deer-flow/pull/7659
|
||||||
[#7638]: https://github.com/bytedance/deer-flow/pull/7638
|
|
||||||
[#7665]: https://github.com/bytedance/deer-flow/pull/7665
|
|
||||||
[#7723]: https://github.com/bytedance/deer-flow/pull/7723
|
|
||||||
[#7727]: https://github.com/bytedance/deer-flow/pull/7727
|
|
||||||
[#7729]: https://github.com/bytedance/deer-flow/pull/7729
|
|
||||||
[#7732]: https://github.com/bytedance/deer-flow/pull/7732
|
|
||||||
[#7746]: https://github.com/bytedance/deer-flow/pull/7746
|
|
||||||
[#7762]: https://github.com/bytedance/deer-flow/pull/7762
|
|
||||||
[#7772]: https://github.com/bytedance/deer-flow/pull/7772
|
|
||||||
[#7781]: https://github.com/bytedance/deer-flow/pull/7781
|
|
||||||
[#7782]: https://github.com/bytedance/deer-flow/pull/7782
|
|
||||||
[#7792]: https://github.com/bytedance/deer-flow/pull/7792
|
|
||||||
[#7794]: https://github.com/bytedance/deer-flow/pull/7794
|
|
||||||
[#7798]: https://github.com/bytedance/deer-flow/pull/7798
|
|
||||||
[#7831]: https://github.com/bytedance/deer-flow/pull/7831
|
|
||||||
[#7892]: https://github.com/bytedance/deer-flow/pull/7892
|
|
||||||
[#7907]: https://github.com/bytedance/deer-flow/pull/7907
|
|
||||||
[#7941]: https://github.com/bytedance/deer-flow/pull/7941
|
|
||||||
[#7945]: https://github.com/bytedance/deer-flow/issues/7945
|
|
||||||
[#7953]: https://github.com/bytedance/deer-flow/issues/7953
|
|
||||||
[#7958]: https://github.com/bytedance/deer-flow/pull/7958
|
|
||||||
[#7981]: https://github.com/bytedance/deer-flow/pull/7981
|
|
||||||
[#7992]: https://github.com/bytedance/deer-flow/pull/7992
|
|
||||||
[#7993]: https://github.com/bytedance/deer-flow/pull/7993
|
|
||||||
[#7998]: https://github.com/bytedance/deer-flow/issues/7998
|
|
||||||
[#8006]: https://github.com/bytedance/deer-flow/pull/8006
|
|
||||||
[#8019]: https://github.com/bytedance/deer-flow/pull/8019
|
|
||||||
[#8020]: https://github.com/bytedance/deer-flow/pull/8020
|
|
||||||
[#8043]: https://github.com/bytedance/deer-flow/pull/8043
|
|
||||||
[#8046]: https://github.com/bytedance/deer-flow/pull/8046
|
|
||||||
[#8057]: https://github.com/bytedance/deer-flow/pull/8057
|
|
||||||
[#8064]: https://github.com/bytedance/deer-flow/pull/8064
|
|
||||||
[#8069]: https://github.com/bytedance/deer-flow/issues/8069
|
|
||||||
[#8071]: https://github.com/bytedance/deer-flow/issues/8071
|
|
||||||
[#8072]: https://github.com/bytedance/deer-flow/pull/8072
|
|
||||||
[#8079]: https://github.com/bytedance/deer-flow/pull/8079
|
|
||||||
[#8087]: https://github.com/bytedance/deer-flow/pull/8087
|
|
||||||
[#8117]: https://github.com/bytedance/deer-flow/pull/8117
|
|
||||||
[#8118]: https://github.com/bytedance/deer-flow/pull/8118
|
|
||||||
[#8126]: https://github.com/bytedance/deer-flow/pull/8126
|
|
||||||
[#8131]: https://github.com/bytedance/deer-flow/pull/8131
|
|
||||||
[#8139]: https://github.com/bytedance/deer-flow/pull/8139
|
|
||||||
[#8151]: https://github.com/bytedance/deer-flow/pull/8151
|
|
||||||
[#8174]: https://github.com/bytedance/deer-flow/pull/8174
|
|
||||||
|
|||||||
267
CHANGELOG_zh.md
267
CHANGELOG_zh.md
@ -9,203 +9,6 @@
|
|||||||
|
|
||||||
## [未发布]
|
## [未发布]
|
||||||
|
|
||||||
本节累积面向 **2.3.0** 里程碑（[2.3.0](https://github.com/bytedance/deer-flow/milestone/4)）的工作。
|
|
||||||
|
|
||||||
|
|
||||||
### 新增
|
|
||||||
|
|
||||||
- **子智能体：** 后台批次可声明条目前置关系，并可登记一次主 Agent 接续，在批次结束后完成
|
|
||||||
用户已要求的汇总或交付。接续保留持久化启动记录、取消和授权检查；启用 Goal 时仍由 Goal 负责续跑。([#7839](https://github.com/bytedance/deer-flow/pull/7839))
|
|
||||||
|
|
||||||
- **对话：** 网页端在任务运行时，可在输入框选择 Queue（排队执行）或 Steer（调整当前任务）。
|
|
||||||
Queue 等当前任务结束后再执行下一条消息；Steer 让主 Agent 先处理新要求，原生子 Agent 可暂停后恢复。
|
|
||||||
Gateway 重启后，尚未确认生效的消息可以手动重试。([#7729])
|
|
||||||
|
|
||||||
- **模型：** 管理员可在 **设置 → 模型** 中选择 StepFun（阶跃星辰），
|
|
||||||
填写 API Key 和模型 ID 即可接入普通 API。([#7988](https://github.com/bytedance/deer-flow/pull/7988))
|
|
||||||
|
|
||||||
### 变更
|
|
||||||
|
|
||||||
- **Checkpoint：** DeltaChannel 部署现在要求 `langgraph-checkpoint` 4.3.0、
|
|
||||||
SQLite saver 3.1.2 和 Postgres saver 3.1.3，排除会在重放时重排并行写入或
|
|
||||||
丢失分页祖先历史的版本。顺序和 Postgres 分页契约在回归时现在会直接失败，
|
|
||||||
不再跳过。([#8139])
|
|
||||||
|
|
||||||
### 修复
|
|
||||||
|
|
||||||
- **前端：** 任务表单的 Cron 预设现在要求分钟 0-59、小时 0-23、日期 1-31，
|
|
||||||
超出范围的表达式（例如 `0 24 * * *`）会保持为自定义并保留原始文本。此前它会被识别为预设，
|
|
||||||
在序列化以供提交时可能被限制到合法范围内的另一个时间。使用非法表达式创建或复制任务，
|
|
||||||
或在表达式仍非法时更新调度配置，会由 Gateway 返回 `422`；仅修改其他字段不会重新校验
|
|
||||||
未变化的调度配置。([#8020])
|
|
||||||
|
|
||||||
- **网关：** 创建对话时在元数据中带上 `step`、`parents`、`counters_since_delta_snapshot`
|
|
||||||
等由检查点自身维护的键，不再让该对话损坏。此前这些键会写入对话的首个检查点，错误的 `parents`
|
|
||||||
会让该对话的历史记录永久返回 500，错误的 `step` 或增量计数器会让之后每一轮对话都失败；
|
|
||||||
`deerflow_agent_name` 还会被当作服务端写入的值复制到下一次状态更新中。现在首个检查点不再从
|
|
||||||
请求中接收这些键。([#8019])
|
|
||||||
|
|
||||||
- **通道：** 企业微信不再丢弃 2–10 MB 的图片。此前适配器把图片上限定为 2 MB（同一协议中语音的限制），
|
|
||||||
超出即在上传前被跳过，消息到达时缺少图片。现在图片上限与平台的 10 MB 对齐；普通文件 20 MB 的限制不变。([#8174])
|
|
||||||
|
|
||||||
- **通道：** WeCom 群聊中 `@机器人 /new` 这类带提及的命令现在按命令执行。WeCom 会把提及保留在
|
|
||||||
正文里，而命令识别要求裸的前导 `/`，于是群里的 `/new`、`/status` 被当成聊天轮次。现在仅在命令
|
|
||||||
识别时跳过前导提及；普通聊天保留提及，`/connect` 行为不变。与飞书、钉钉的处理一致。([#8118])
|
|
||||||
|
|
||||||
- **前端：** 侧边栏对话菜单中的 **分享** 现在复制指向当前部署的链接。此前在 `localhost` 和
|
|
||||||
`127.0.0.1`（`make dev` 与 Docker 的默认地址）上，复制的是托管演示站
|
|
||||||
（`deer-flow-v2.vercel.app`）的链接：该链接无法打开对话，打开时还会把线程 ID 发送到外部主机。
|
|
||||||
([#8126])
|
|
||||||
|
|
||||||
- **运行时：** 多 Worker 模式（`run_ownership.heartbeat_enabled`）下，已被其他 Worker
|
|
||||||
接管的运行不再被原 Worker 恢复。此前装配证据通过无条件的整行 upsert 写入，在对端接管后会把
|
|
||||||
行改回 `running` 并清除接管错误；运行可能在对端已报告失败后以 `success` 结束，或被再次接管。
|
|
||||||
现在仅当该 Worker 仍持有活跃运行的有效租约时，才将证据合并到行元数据
|
|
||||||
（`RunStore.merge_owned_metadata()`），否则在执行前停止该运行。启用心跳的自定义
|
|
||||||
`RunStore` 实现必须实现该方法。([#8046])
|
|
||||||
|
|
||||||
- **网关：** 同一用户对同一次运行重复提交反馈时，现在返回 409 而不是 500。
|
|
||||||
`POST /api/threads/{id}/runs/{run_id}/feedback` 每个用户每次运行只允许一条反馈；此前重试，
|
|
||||||
或对同一次运行的另一条消息提交反馈，会触发该唯一约束，Gateway 因未处理的数据库错误而失败。
|
|
||||||
409 会提示调用方改用同一路由的 `PUT` 更新已有反馈。([#8064])
|
|
||||||
|
|
||||||
- **网关：** 删除线程时不再在对话仍可读取的情况下返回成功。此前
|
|
||||||
`DELETE /api/threads/{id}` 在检查点存储失败时只记录 DEBUG 日志，仍删除线程元数据行并
|
|
||||||
返回成功；之后任何已登录用户都能通过 `GET /threads/{id}`、`/state`、`/history` 和
|
|
||||||
`/goal` 读取残留的标题和消息，所有者也无法重试。现在删除会返回 500 并保留线程，所有者
|
|
||||||
可以再次删除。([#7992])
|
|
||||||
|
|
||||||
- **模型：** 取消使用 Codex 的任务时，会关闭上游 HTTP/SSE 连接，包括等待响应头期间；
|
|
||||||
重试等待也可取消。本地资源会释放，远端服务仍可能继续生成。([#8053](https://github.com/bytedance/deer-flow/pull/8053))
|
|
||||||
|
|
||||||
- **中间件：** 启用 PII 脱敏时，不含空格的长文本（例如粘贴的十六进制转储）不再让 Gateway
|
|
||||||
卡住。此前邮箱检测器的耗时与这类文本长度呈平方关系，而脱敏会在每次模型调用时重新扫描全部
|
|
||||||
用户消息，因此一次 100 KB 的粘贴会让每次模型调用阻塞所有线程约 6 秒。现在扫描为线性耗时，
|
|
||||||
且脱敏的地址与此前完全相同。([#8006])
|
|
||||||
|
|
||||||
- **调度器：** 队列超时扫描或队列排空中的单条记录出现异常时，调度器会记录该失败并继续处理剩余队列，以及同一轮启动的到期任务。过期记录会在所有者排名和批次限额之前被排除，启动领取还会再次检查同一截止时间，因此超时清理失败时遗留的排队记录既不会启动，也不会挤占正常排队工作的候选位置，只会继续等待超时恢复流程处理。([#7998])
|
|
||||||
- **调度器：** 一个已保存的计划无法再计算的定时任务，不再让其他所有任务停止运行。此前
|
|
||||||
已保存的时区或 cron 表达式失效时（例如升级后时区数据库中缺少该时区，或新版 croniter
|
|
||||||
不再接受该表达式），在调度器轮询中计算其下一次运行时间会抛出异常；其启动租约过期后，
|
|
||||||
之后每次轮询都会在接纳到期任务之前失败。现在该任务会被暂停，并把原因记录为最近错误；
|
|
||||||
在修复计划之前，恢复该任务会提示具体问题。([#7993])
|
|
||||||
|
|
||||||
- **模型：** 使用 Claude Code OAuth token 的 Claude 模型，不再在请求带有 `betas` 时每次
|
|
||||||
都失败。此前 OAuth 所需的 beta（`oauth-2025-04-20` 等）只放在客户端默认的
|
|
||||||
`anthropic-beta` 请求头里，而 SDK 会用请求自身的 `betas` 替换这个请求头；因此只要配置了
|
|
||||||
`betas`（即使是 `[]`）、调用时传入 `betas`，或使用了 langchain-anthropic 会为其追加 beta
|
|
||||||
的服务端工具，OAuth beta 就会丢失，Anthropic 随即拒绝该 token。现在 OAuth beta 会排在
|
|
||||||
请求 `betas` 以及 `extra_headers` 中 `anthropic-beta` 的最前面。
|
|
||||||
([#8061](https://github.com/bytedance/deer-flow/pull/8061))
|
|
||||||
|
|
||||||
- **社区工具：** 以换行结尾的搜索或抓取 API 密钥不再泄露到对话中。GroundRoute
|
|
||||||
（`GROUNDROUTE_API_KEY`）、InfoQuest、Tavily、Exa、Jina、Firecrawl、fastCRW 和
|
|
||||||
Crawl4AI 此前把读取到的密钥原样写入 HTTP 请求头；从文件读取的密钥（例如用
|
|
||||||
`--from-file` 创建的 Kubernetes Secret）会让传输层报出 `Illegal header value
|
|
||||||
b'Bearer <key>\n'`，这条包含完整密钥的消息成为模型和对话历史保存的工具结果，
|
|
||||||
其中四个提供方还会把它写入日志。现在这些密钥与其他提供方一样经过
|
|
||||||
`provider_credentials.resolve_api_key` 处理；Tavily、Exa、Firecrawl 和 fastCRW
|
|
||||||
的配置密钥为空时会回退到环境变量。fastCRW 没有自己的密钥时，不再让 Firecrawl SDK
|
|
||||||
把 `FIRECRAWL_API_KEY` 发送到 fastCRW 端点；`TAVILY_API_KEY`、`EXA_API_KEY` 或
|
|
||||||
`FIRECRAWL_API_KEY` 只含空白时按未设置处理，不再原样交给 SDK。([#7981])
|
|
||||||
|
|
||||||
- **社区工具：** Browserless 是最后一个直接读取原始 token 的搜索/抓取提供方。从文件读取的
|
|
||||||
`BROWSERLESS_TOKEN` 或配置 `token` 会保留尾部换行，并被原样用作鉴权查询参数，导致
|
|
||||||
`web_fetch` 和 `web_capture` 在密钥本身正确时也全部鉴权失败。现在两者都经过
|
|
||||||
`provider_credentials.resolve_api_key` 处理，且只有省略 `token` 键时才回退到环境变量：
|
|
||||||
显式配置 `""` 或 `null` 仍可让自建 capture 端点跳过鉴权。([#8043])
|
|
||||||
|
|
||||||
- **docker：** 从本地 checkout 构建的镜像不再包含本地配置。`.dockerignore`
|
|
||||||
此前没有排除 `config.yaml`、`config.yaml.bak`、`extensions_config.json` 和
|
|
||||||
`mcp_config.json`，因此配置放在旧位置 `backend/` 时，`COPY backend ./backend`
|
|
||||||
会把模型 API 密钥和 MCP 凭据写进每一层 gateway 镜像；`frontend/.env.local`
|
|
||||||
也以同样方式进入前端镜像。现在根目录和 `backend/` 下的这些文件、任意层级的
|
|
||||||
`.env*.local`、`frontend/*.pem`，以及扩展管理器拒绝写入扩展快照的凭据文件都会
|
|
||||||
被排除。本地构建的前端镜像不再内联 `frontend/.env.local` 中的 `NEXT_PUBLIC_*`
|
|
||||||
值；Docker 栈通过同一个 nginx 源提供前端和 Gateway，不需要这些变量。CI 发布的
|
|
||||||
镜像不受影响；已推送或分享过的本地镜像请重新构建。([#7958])
|
|
||||||
|
|
||||||
- **智能体：** 循环检测、运行 token 预算、模型服务安全限制或输出长度限制触发停止后，
|
|
||||||
未完成的 Todo 不再触发额外的模型调用或工具执行。Todo 状态仍会保留，
|
|
||||||
用户可在后续运行中继续处理。([#7953])
|
|
||||||
|
|
||||||
- **渠道：** 飞书/Lark 的长回答不再丢失。此前卡片文本没有长度上限，并以 ASCII 转义
|
|
||||||
序列化，约 4,400 个中日韩字符就会让卡片请求超过飞书 30 KB 的限制（`230025`）：
|
|
||||||
最终更新及其回复兜底都会被拒绝，运行中卡片停留在最后一次流式内容或只剩 `▉`。
|
|
||||||
现在卡片不再转义中日韩字符，并按其在请求中占用的字节计算预算。流式更新只显示能
|
|
||||||
放下的部分；更长的最终回答会在同一条消息下以最多 10 张卡片按行续发，重试会从
|
|
||||||
已送达的卡片之后继续。([#7798])
|
|
||||||
|
|
||||||
- **渠道：** 企业微信群消息现在拥有独立的会话。此前每条消息都把发送者的 userid
|
|
||||||
同时用作会话 ID 和话题 ID，因此在没有群绑定加以区分的情况下（免认证模式、
|
|
||||||
`require_bound_identity: false`，以及旧版群绑定），成员的私聊和其发过言的所有群
|
|
||||||
共用同一个 DeerFlow 线程：群内回复可能用到该成员私聊或其他群的内容，失去原始
|
|
||||||
frame 的回复也会被发到私聊。现在群以 `chatid` 为键、每位成员一个线程，私聊线程
|
|
||||||
保持不变。通过各自 `/connect` 绑定的群保留原有线程；覆盖该机器人所有群的旧版
|
|
||||||
群绑定会在每个群中开始新线程。([#7794])
|
|
||||||
|
|
||||||
- **沙箱：** 使用显式授权技能视图的新 AIO 运行，在启用惰性初始化时，不再于首次调用
|
|
||||||
沙箱工具前创建或获取沙箱。已有沙箱绑定和需要上传技能的提供方继续保留原有的即时同步行为。
|
|
||||||
([#7782])
|
|
||||||
|
|
||||||
- **扩展：** 上下文压缩观察者现在能知道压缩发生在哪个任务里。在 Lead 运行或子 Agent
|
|
||||||
内部，观察者收到的是该任务的存储，而不再是分离存储；`CompactionEvent` 新增
|
|
||||||
`task_id`、`run_id`、`thread_id`（默认 `None`），取自宿主放进每个任务存储的
|
|
||||||
`TaskInfo`，即使没有注册生命周期贡献者也会放入。手动 `/compact` 仍收到分离存储，
|
|
||||||
id 为 `None`。`deerflow-extension-api` 升级到 0.2.10，已有扩展不受影响。([#7831])
|
|
||||||
|
|
||||||
- **智能体：** 即使模型提供方复用消息 ID，Token 预算也会分别统计已接受的模型调用，
|
|
||||||
避免重复 ID 绕过配置的上限；触发硬停止时保留已完成的结构化输出工具结果。
|
|
||||||
([#8050](https://github.com/bytedance/deer-flow/pull/8050))
|
|
||||||
|
|
||||||
- **渠道：** 钉钉 AI 卡片模式（`card_template_id`）下，卡片请求失败（例如缺少卡片权限或钉钉服务异常）时
|
|
||||||
不再丢失回复。dingtalk-stream 会记录失败的卡片请求后按成功返回，因此文本形式的运行中提示和最终答案的
|
|
||||||
`sampleMarkdown` 兜底都不会执行。现在渠道自行发送卡片请求，并在失败时回退。([#7781])
|
|
||||||
|
|
||||||
- **harness：** 重试失败条目不再重新打开已取消的持久子智能体批次。`retry_item`
|
|
||||||
此前只保留已暂停批次的状态，因此重试已取消批次中的失败条目会把批次改回
|
|
||||||
`queued`，worker 随即认领并执行该条目，批次最终变为 `completed`。现在对已取消
|
|
||||||
批次的重试路由返回 409，包括 SQLite 上取消在重试进行中提交的情况；工作区也不再
|
|
||||||
为这些条目显示重试按钮。在已完成、失败、运行中或已暂停批次中重试的行为不变。
|
|
||||||
([#7792])
|
|
||||||
|
|
||||||
- **网关：** 运行指定的自定义智能体不存在或名称无效时，现在会明确说明原因。此前向一个
|
|
||||||
自定义智能体已被删除的对话发消息，或传入格式错误的 `agent_name`，即使与知识库范围无关，
|
|
||||||
也会返回 422 和 "knowledge_scope assistant configuration could not be resolved"。现在
|
|
||||||
422 会提示 `Agent '<name>' not found`、`agent_name` 无效，或智能体配置无法加载。([#8087])
|
|
||||||
|
|
||||||
- **智能体：** `PUT /api/agents/{name}` 传入 `"description": null` 时现在会把描述重置为
|
|
||||||
空。此前会把 null 原样保存，而智能体配置不接受该值，导致之后每次加载都失败：更新本身、
|
|
||||||
GET 和修复性更新都返回 500，文件后端会把它从智能体列表中漏掉，数据库后端会让该用户的
|
|
||||||
整个列表失败，与它的对话也会失败。已以 null 描述保存的智能体现在可以重新加载。([#7638])
|
|
||||||
|
|
||||||
- **智能体：** 当模型提供方在不同轮次复用同一个工具调用 ID（例如每次响应都是 `call_0`）时，
|
|
||||||
澄清提问不再覆盖之前的问题。此前其消息 ID（同时也是人工输入的 `request_id`）只包含调用 ID，
|
|
||||||
后一个问题会原地替换前一个问题，后一次调用也因此缺少结果。现在 ID 还会包含发起该调用的
|
|
||||||
AIMessage；同一调用的重试仍保持原 ID。([#7762])
|
|
||||||
|
|
||||||
- **项目：** 当数据目录、`users/`、某个用户目录或用户的 `projects/` 是符号链接时，
|
|
||||||
回收站保留期清扫不再删除仍在使用的文档原件。此前它按解析后的路径保护每个文档，
|
|
||||||
却遍历未解析的目录树，导致没有任何文档能匹配上，所有超过 24 小时的原件都被当作
|
|
||||||
孤立文件删除，Gateway 启动时和 `GET /api/trash/documents` 都会触发。([#7732])
|
|
||||||
|
|
||||||
- **对话：** 队列分发开始后，取消某条排队输入不再阻止后续排队输入继续执行。([#7746])
|
|
||||||
|
|
||||||
- **开发：** `scripts/dev_multi_instance.sh logs all` 现在除两个 Gateway 日志外也会输出
|
|
||||||
nginx 日志；此前它只输出 Gateway 日志，与不带参数的 `logs` 相同。`logs nginx` 与
|
|
||||||
`logs all` 还会包含 `nginx.log`（nginx 启动命令自身的输出），此前没有任何选项会输出它。
|
|
||||||
([#7723])
|
|
||||||
|
|
||||||
- **记忆：** `GET /api/memory/config` 与 `/api/memory/status` 现在会把 `backend_config`
|
|
||||||
中的凭据值（`api_key`、token、密码、`Authorization` 请求头，任意层级）遮蔽为 `***`。
|
|
||||||
此前它们按加载结果原样返回配置，而 `$VAR` 引用此时已被解析，因此 DeerMem 提取模型的密钥
|
|
||||||
或 Honcho 的 `api_key` 会以明文返回给所有拥有 `memory:read` 的调用方；在默认关闭授权时，
|
|
||||||
这就是所有已登录用户。非机密配置项、`api_key_env` 变量名和未设置的值保持不变。([#7665])
|
|
||||||
- **网关：** `GET /threads/{thread_id}/token-usage` 现在在工作线程里解析实时配置。
|
|
||||||
此前 `build_context_usage` 直接内联调用该访问器，而 `get_app_config()` 每次调用都会重新校验
|
|
||||||
——它先对整个 `config.yaml` 取摘要，再对 `extensions_config.json` 做同样的事——所以在有人打开
|
|
||||||
会话视图前刚好改了配置，就会让运行、消息流与其他线程共用的网关事件循环整段停顿。响应内容不变。([#7727])
|
|
||||||
|
|
||||||
## [2.2.0] — 2026-10-29
|
## [2.2.0] — 2026-10-29
|
||||||
|
|
||||||
本节累积面向 **2.2.0** 里程碑（[2.2.0](https://github.com/bytedance/deer-flow/milestone/3)）的工作。
|
本节累积面向 **2.2.0** 里程碑（[2.2.0](https://github.com/bytedance/deer-flow/milestone/3)）的工作。
|
||||||
@ -420,12 +223,6 @@
|
|||||||
选择器缺失、是目录或不可读时会在调用 Docker 之前直接失败，而文件删除后
|
选择器缺失、是目录或不可读时会在调用 Docker 之前直接失败，而文件删除后
|
||||||
`make down` 仍能停止整个栈；不设置或为空的选择器保持可选的仓库根 `.env`
|
`make down` 仍能停止整个栈；不设置或为空的选择器保持可选的仓库根 `.env`
|
||||||
默认值。([#6522])
|
默认值。([#6522])
|
||||||
- **HITL：** 新增独立于推理模式的 Ask、Auto、Full 工具权限模式。Auto 通过单独的
|
|
||||||
无工具模型调用审核具体操作，默认复用现有模型配置；不确定或高风险操作转人工审批。
|
|
||||||
卡片展示完整参数，支持批准、拒绝、修改参数及可撤销的会话精确授权。主代理审批通过
|
|
||||||
checkpoint 支持重启恢复；交互式子代理通过属主限定的通道等待，并处理取消和过期。
|
|
||||||
所有模式均保留部署规则和账户权限，个人/平台 MCP 连接、参数和策略均参与授权绑定。
|
|
||||||
默认关闭；`config_version` 66。
|
|
||||||
|
|
||||||
- **MCP：** 按未变更的服务器缓存完成包装的工具，仅在绑定对账后重新发现缺失或配置
|
- **MCP：** 按未变更的服务器缓存完成包装的工具，仅在绑定对账后重新发现缺失或配置
|
||||||
发生变化的服务器。将成功但未发现工具的结果与发现失败分开保存，并保留代际、共享
|
发生变化的服务器。将成功但未发现工具的结果与发现失败分开保存，并保留代际、共享
|
||||||
@ -10812,7 +10609,8 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
- **CI：** 统一 PR / issue 打标签逻辑，修复 reviewing 任务的崩溃与标签抖动。
|
- **CI：** 统一 PR / issue 打标签逻辑，修复 reviewing 任务的崩溃与标签抖动。
|
||||||
([#3455])
|
([#3455])
|
||||||
|
|
||||||
[未发布]: https://github.com/bytedance/deer-flow/compare/v2.1.0...HEAD
|
[未发布]: https://github.com/bytedance/deer-flow/compare/v2.2.0...HEAD
|
||||||
|
[2.2.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.2.0
|
||||||
[2.1.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.1.0
|
[2.1.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.1.0
|
||||||
[2.0.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.0.0
|
[2.0.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.0.0
|
||||||
|
|
||||||
@ -11654,7 +11452,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5217]: https://github.com/bytedance/deer-flow/pull/5217
|
[#5217]: https://github.com/bytedance/deer-flow/pull/5217
|
||||||
[#5219]: https://github.com/bytedance/deer-flow/pull/5219
|
[#5219]: https://github.com/bytedance/deer-flow/pull/5219
|
||||||
[#5221]: https://github.com/bytedance/deer-flow/pull/5221
|
[#5221]: https://github.com/bytedance/deer-flow/pull/5221
|
||||||
[#5224]: https://github.com/bytedance/deer-flow/commit/572744975d369fda88168aaa7b481282f939affe
|
[#5224]: https://github.com/bytedance/deer-flow/pull/5224
|
||||||
[#5225]: https://github.com/bytedance/deer-flow/pull/5225
|
[#5225]: https://github.com/bytedance/deer-flow/pull/5225
|
||||||
[#5227]: https://github.com/bytedance/deer-flow/pull/5227
|
[#5227]: https://github.com/bytedance/deer-flow/pull/5227
|
||||||
[#5228]: https://github.com/bytedance/deer-flow/pull/5228
|
[#5228]: https://github.com/bytedance/deer-flow/pull/5228
|
||||||
@ -11663,9 +11461,9 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5236]: https://github.com/bytedance/deer-flow/pull/5236
|
[#5236]: https://github.com/bytedance/deer-flow/pull/5236
|
||||||
[#5238]: https://github.com/bytedance/deer-flow/pull/5238
|
[#5238]: https://github.com/bytedance/deer-flow/pull/5238
|
||||||
[#5239]: https://github.com/bytedance/deer-flow/pull/5239
|
[#5239]: https://github.com/bytedance/deer-flow/pull/5239
|
||||||
[#5244]: https://github.com/bytedance/deer-flow/commit/9ce6fdcb2252ea005b4fa98ca50bd3857e1f681a
|
[#5244]: https://github.com/bytedance/deer-flow/pull/5244
|
||||||
[#5245]: https://github.com/bytedance/deer-flow/pull/5245
|
[#5245]: https://github.com/bytedance/deer-flow/pull/5245
|
||||||
[#5247]: https://github.com/bytedance/deer-flow/commit/fa89a125267be4213009376ec0e232a0c2411ca8
|
[#5247]: https://github.com/bytedance/deer-flow/pull/5247
|
||||||
[#5249]: https://github.com/bytedance/deer-flow/pull/5249
|
[#5249]: https://github.com/bytedance/deer-flow/pull/5249
|
||||||
[#5251]: https://github.com/bytedance/deer-flow/pull/5251
|
[#5251]: https://github.com/bytedance/deer-flow/pull/5251
|
||||||
[#5254]: https://github.com/bytedance/deer-flow/pull/5254
|
[#5254]: https://github.com/bytedance/deer-flow/pull/5254
|
||||||
@ -11722,7 +11520,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5367]: https://github.com/bytedance/deer-flow/pull/5367
|
[#5367]: https://github.com/bytedance/deer-flow/pull/5367
|
||||||
[#5369]: https://github.com/bytedance/deer-flow/pull/5369
|
[#5369]: https://github.com/bytedance/deer-flow/pull/5369
|
||||||
[#5371]: https://github.com/bytedance/deer-flow/pull/5371
|
[#5371]: https://github.com/bytedance/deer-flow/pull/5371
|
||||||
[#5373]: https://github.com/bytedance/deer-flow/commit/d5ae3882b6708fa9af7289a0c3c0be3eee157203
|
[#5373]: https://github.com/bytedance/deer-flow/pull/5373
|
||||||
[#5374]: https://github.com/bytedance/deer-flow/pull/5374
|
[#5374]: https://github.com/bytedance/deer-flow/pull/5374
|
||||||
[#5375]: https://github.com/bytedance/deer-flow/pull/5375
|
[#5375]: https://github.com/bytedance/deer-flow/pull/5375
|
||||||
[#5377]: https://github.com/bytedance/deer-flow/pull/5377
|
[#5377]: https://github.com/bytedance/deer-flow/pull/5377
|
||||||
@ -11749,7 +11547,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5408]: https://github.com/bytedance/deer-flow/pull/5408
|
[#5408]: https://github.com/bytedance/deer-flow/pull/5408
|
||||||
[#5410]: https://github.com/bytedance/deer-flow/pull/5410
|
[#5410]: https://github.com/bytedance/deer-flow/pull/5410
|
||||||
[#5411]: https://github.com/bytedance/deer-flow/pull/5411
|
[#5411]: https://github.com/bytedance/deer-flow/pull/5411
|
||||||
[#5413]: https://github.com/bytedance/deer-flow/commit/daf32cd24632f2cc7465aeb62f2a1e1e768eceb2
|
[#5413]: https://github.com/bytedance/deer-flow/pull/5413
|
||||||
[#5415]: https://github.com/bytedance/deer-flow/pull/5415
|
[#5415]: https://github.com/bytedance/deer-flow/pull/5415
|
||||||
[#5416]: https://github.com/bytedance/deer-flow/pull/5416
|
[#5416]: https://github.com/bytedance/deer-flow/pull/5416
|
||||||
[#5418]: https://github.com/bytedance/deer-flow/pull/5418
|
[#5418]: https://github.com/bytedance/deer-flow/pull/5418
|
||||||
@ -11932,8 +11730,8 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5736]: https://github.com/bytedance/deer-flow/pull/5736
|
[#5736]: https://github.com/bytedance/deer-flow/pull/5736
|
||||||
[#5738]: https://github.com/bytedance/deer-flow/pull/5738
|
[#5738]: https://github.com/bytedance/deer-flow/pull/5738
|
||||||
[#5739]: https://github.com/bytedance/deer-flow/pull/5739
|
[#5739]: https://github.com/bytedance/deer-flow/pull/5739
|
||||||
[#5740]: https://github.com/bytedance/deer-flow/commit/e7128dc7d6b4d49153b10f7276f010384ec4e6af
|
[#5740]: https://github.com/bytedance/deer-flow/pull/5740
|
||||||
[#5741]: https://github.com/bytedance/deer-flow/commit/363b0fc1b4a5bd6c142d1f1df9fb9743cca550a3
|
[#5741]: https://github.com/bytedance/deer-flow/pull/5741
|
||||||
[#5745]: https://github.com/bytedance/deer-flow/pull/5745
|
[#5745]: https://github.com/bytedance/deer-flow/pull/5745
|
||||||
[#5748]: https://github.com/bytedance/deer-flow/pull/5748
|
[#5748]: https://github.com/bytedance/deer-flow/pull/5748
|
||||||
[#5750]: https://github.com/bytedance/deer-flow/pull/5750
|
[#5750]: https://github.com/bytedance/deer-flow/pull/5750
|
||||||
@ -11946,7 +11744,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5760]: https://github.com/bytedance/deer-flow/pull/5760
|
[#5760]: https://github.com/bytedance/deer-flow/pull/5760
|
||||||
[#5761]: https://github.com/bytedance/deer-flow/pull/5761
|
[#5761]: https://github.com/bytedance/deer-flow/pull/5761
|
||||||
[#5762]: https://github.com/bytedance/deer-flow/pull/5762
|
[#5762]: https://github.com/bytedance/deer-flow/pull/5762
|
||||||
[#5763]: https://github.com/bytedance/deer-flow/commit/39351315bf5d096fc84ad8307d3702b6aa5b3e24
|
[#5763]: https://github.com/bytedance/deer-flow/pull/5763
|
||||||
[#5767]: https://github.com/bytedance/deer-flow/pull/5767
|
[#5767]: https://github.com/bytedance/deer-flow/pull/5767
|
||||||
[#5769]: https://github.com/bytedance/deer-flow/pull/5769
|
[#5769]: https://github.com/bytedance/deer-flow/pull/5769
|
||||||
[#5775]: https://github.com/bytedance/deer-flow/pull/5775
|
[#5775]: https://github.com/bytedance/deer-flow/pull/5775
|
||||||
@ -12014,7 +11812,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5880]: https://github.com/bytedance/deer-flow/pull/5880
|
[#5880]: https://github.com/bytedance/deer-flow/pull/5880
|
||||||
[#5881]: https://github.com/bytedance/deer-flow/pull/5881
|
[#5881]: https://github.com/bytedance/deer-flow/pull/5881
|
||||||
[#5883]: https://github.com/bytedance/deer-flow/pull/5883
|
[#5883]: https://github.com/bytedance/deer-flow/pull/5883
|
||||||
[#5884]: https://github.com/bytedance/deer-flow/commit/e8cb5e9c4f5f274401eeb555aff62e713035cbf4
|
[#5884]: https://github.com/bytedance/deer-flow/pull/5884
|
||||||
[#5888]: https://github.com/bytedance/deer-flow/pull/5888
|
[#5888]: https://github.com/bytedance/deer-flow/pull/5888
|
||||||
[#5890]: https://github.com/bytedance/deer-flow/pull/5890
|
[#5890]: https://github.com/bytedance/deer-flow/pull/5890
|
||||||
[#5892]: https://github.com/bytedance/deer-flow/pull/5892
|
[#5892]: https://github.com/bytedance/deer-flow/pull/5892
|
||||||
@ -12076,9 +11874,9 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#5999]: https://github.com/bytedance/deer-flow/pull/5999
|
[#5999]: https://github.com/bytedance/deer-flow/pull/5999
|
||||||
[#6000]: https://github.com/bytedance/deer-flow/pull/6000
|
[#6000]: https://github.com/bytedance/deer-flow/pull/6000
|
||||||
[#6008]: https://github.com/bytedance/deer-flow/pull/6008
|
[#6008]: https://github.com/bytedance/deer-flow/pull/6008
|
||||||
[#6009]: https://github.com/bytedance/deer-flow/commit/b87ebd940227465d611084e5bf49582aaeb4e9bd
|
[#6009]: https://github.com/bytedance/deer-flow/pull/6009
|
||||||
[#6013]: https://github.com/bytedance/deer-flow/pull/6013
|
[#6013]: https://github.com/bytedance/deer-flow/pull/6013
|
||||||
[#6015]: https://github.com/bytedance/deer-flow/commit/f87a04e581829ff35afd4fa228240071b486192f
|
[#6015]: https://github.com/bytedance/deer-flow/pull/6015
|
||||||
[#6017]: https://github.com/bytedance/deer-flow/pull/6017
|
[#6017]: https://github.com/bytedance/deer-flow/pull/6017
|
||||||
[#6018]: https://github.com/bytedance/deer-flow/pull/6018
|
[#6018]: https://github.com/bytedance/deer-flow/pull/6018
|
||||||
[#6020]: https://github.com/bytedance/deer-flow/pull/6020
|
[#6020]: https://github.com/bytedance/deer-flow/pull/6020
|
||||||
@ -12102,7 +11900,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#6060]: https://github.com/bytedance/deer-flow/pull/6060
|
[#6060]: https://github.com/bytedance/deer-flow/pull/6060
|
||||||
[#6062]: https://github.com/bytedance/deer-flow/pull/6062
|
[#6062]: https://github.com/bytedance/deer-flow/pull/6062
|
||||||
[#6063]: https://github.com/bytedance/deer-flow/pull/6063
|
[#6063]: https://github.com/bytedance/deer-flow/pull/6063
|
||||||
[#6066]: https://github.com/bytedance/deer-flow/commit/73fdbcfdf4b5f4037b62bc2ce9d8b60ec04e69f8
|
[#6066]: https://github.com/bytedance/deer-flow/pull/6066
|
||||||
[#6067]: https://github.com/bytedance/deer-flow/pull/6067
|
[#6067]: https://github.com/bytedance/deer-flow/pull/6067
|
||||||
[#6068]: https://github.com/bytedance/deer-flow/pull/6068
|
[#6068]: https://github.com/bytedance/deer-flow/pull/6068
|
||||||
[#6069]: https://github.com/bytedance/deer-flow/pull/6069
|
[#6069]: https://github.com/bytedance/deer-flow/pull/6069
|
||||||
@ -12128,7 +11926,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#6105]: https://github.com/bytedance/deer-flow/pull/6105
|
[#6105]: https://github.com/bytedance/deer-flow/pull/6105
|
||||||
[#6107]: https://github.com/bytedance/deer-flow/pull/6107
|
[#6107]: https://github.com/bytedance/deer-flow/pull/6107
|
||||||
[#6109]: https://github.com/bytedance/deer-flow/pull/6109
|
[#6109]: https://github.com/bytedance/deer-flow/pull/6109
|
||||||
[#6112]: https://github.com/bytedance/deer-flow/commit/52b21d13ac6e674a19cb76013d097e73f108c994
|
[#6112]: https://github.com/bytedance/deer-flow/pull/6112
|
||||||
[#6113]: https://github.com/bytedance/deer-flow/pull/6113
|
[#6113]: https://github.com/bytedance/deer-flow/pull/6113
|
||||||
[#6115]: https://github.com/bytedance/deer-flow/pull/6115
|
[#6115]: https://github.com/bytedance/deer-flow/pull/6115
|
||||||
[#6117]: https://github.com/bytedance/deer-flow/pull/6117
|
[#6117]: https://github.com/bytedance/deer-flow/pull/6117
|
||||||
@ -12229,7 +12027,7 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#6298]: https://github.com/bytedance/deer-flow/pull/6298
|
[#6298]: https://github.com/bytedance/deer-flow/pull/6298
|
||||||
[#6299]: https://github.com/bytedance/deer-flow/pull/6299
|
[#6299]: https://github.com/bytedance/deer-flow/pull/6299
|
||||||
[#6302]: https://github.com/bytedance/deer-flow/pull/6302
|
[#6302]: https://github.com/bytedance/deer-flow/pull/6302
|
||||||
[#6304]: https://github.com/bytedance/deer-flow/commit/b85e4ae74bb407e1e2396243a8a6e8f5b2420e3f
|
[#6304]: https://github.com/bytedance/deer-flow/pull/6304
|
||||||
[#6305]: https://github.com/bytedance/deer-flow/pull/6305
|
[#6305]: https://github.com/bytedance/deer-flow/pull/6305
|
||||||
[#6306]: https://github.com/bytedance/deer-flow/pull/6306
|
[#6306]: https://github.com/bytedance/deer-flow/pull/6306
|
||||||
[#6307]: https://github.com/bytedance/deer-flow/pull/6307
|
[#6307]: https://github.com/bytedance/deer-flow/pull/6307
|
||||||
@ -12613,7 +12411,6 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#7017]: https://github.com/bytedance/deer-flow/pull/7017
|
[#7017]: https://github.com/bytedance/deer-flow/pull/7017
|
||||||
[#7018]: https://github.com/bytedance/deer-flow/pull/7018
|
[#7018]: https://github.com/bytedance/deer-flow/pull/7018
|
||||||
[#7020]: https://github.com/bytedance/deer-flow/pull/7020
|
[#7020]: https://github.com/bytedance/deer-flow/pull/7020
|
||||||
[#7022]: https://github.com/bytedance/deer-flow/issues/7022
|
|
||||||
[#7023]: https://github.com/bytedance/deer-flow/pull/7023
|
[#7023]: https://github.com/bytedance/deer-flow/pull/7023
|
||||||
[#7024]: https://github.com/bytedance/deer-flow/pull/7024
|
[#7024]: https://github.com/bytedance/deer-flow/pull/7024
|
||||||
[#7031]: https://github.com/bytedance/deer-flow/pull/7031
|
[#7031]: https://github.com/bytedance/deer-flow/pull/7031
|
||||||
@ -12886,35 +12683,3 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
|||||||
[#7643]: https://github.com/bytedance/deer-flow/pull/7643
|
[#7643]: https://github.com/bytedance/deer-flow/pull/7643
|
||||||
[#7651]: https://github.com/bytedance/deer-flow/pull/7651
|
[#7651]: https://github.com/bytedance/deer-flow/pull/7651
|
||||||
[#7659]: https://github.com/bytedance/deer-flow/pull/7659
|
[#7659]: https://github.com/bytedance/deer-flow/pull/7659
|
||||||
[#7638]: https://github.com/bytedance/deer-flow/pull/7638
|
|
||||||
[#7665]: https://github.com/bytedance/deer-flow/pull/7665
|
|
||||||
[#7723]: https://github.com/bytedance/deer-flow/pull/7723
|
|
||||||
[#7727]: https://github.com/bytedance/deer-flow/pull/7727
|
|
||||||
[#7729]: https://github.com/bytedance/deer-flow/pull/7729
|
|
||||||
[#7732]: https://github.com/bytedance/deer-flow/pull/7732
|
|
||||||
[#7746]: https://github.com/bytedance/deer-flow/pull/7746
|
|
||||||
[#7762]: https://github.com/bytedance/deer-flow/pull/7762
|
|
||||||
[#7772]: https://github.com/bytedance/deer-flow/pull/7772
|
|
||||||
[#7781]: https://github.com/bytedance/deer-flow/pull/7781
|
|
||||||
[#7792]: https://github.com/bytedance/deer-flow/pull/7792
|
|
||||||
[#7794]: https://github.com/bytedance/deer-flow/pull/7794
|
|
||||||
[#7798]: https://github.com/bytedance/deer-flow/pull/7798
|
|
||||||
[#7831]: https://github.com/bytedance/deer-flow/pull/7831
|
|
||||||
[#7782]: https://github.com/bytedance/deer-flow/pull/7782
|
|
||||||
[#7953]: https://github.com/bytedance/deer-flow/issues/7953
|
|
||||||
[#7958]: https://github.com/bytedance/deer-flow/pull/7958
|
|
||||||
[#7981]: https://github.com/bytedance/deer-flow/pull/7981
|
|
||||||
[#7992]: https://github.com/bytedance/deer-flow/pull/7992
|
|
||||||
[#7993]: https://github.com/bytedance/deer-flow/pull/7993
|
|
||||||
[#7998]: https://github.com/bytedance/deer-flow/issues/7998
|
|
||||||
[#8006]: https://github.com/bytedance/deer-flow/pull/8006
|
|
||||||
[#8019]: https://github.com/bytedance/deer-flow/pull/8019
|
|
||||||
[#8020]: https://github.com/bytedance/deer-flow/pull/8020
|
|
||||||
[#8043]: https://github.com/bytedance/deer-flow/pull/8043
|
|
||||||
[#8046]: https://github.com/bytedance/deer-flow/pull/8046
|
|
||||||
[#8064]: https://github.com/bytedance/deer-flow/pull/8064
|
|
||||||
[#8087]: https://github.com/bytedance/deer-flow/pull/8087
|
|
||||||
[#8118]: https://github.com/bytedance/deer-flow/pull/8118
|
|
||||||
[#8126]: https://github.com/bytedance/deer-flow/pull/8126
|
|
||||||
[#8139]: https://github.com/bytedance/deer-flow/pull/8139
|
|
||||||
[#8174]: https://github.com/bytedance/deer-flow/pull/8174
|
|
||||||
|
|||||||
@ -320,14 +320,6 @@ Nginx (port 2026) ← Unified entry point
|
|||||||
git push origin feature/your-feature-name
|
git push origin feature/your-feature-name
|
||||||
```
|
```
|
||||||
|
|
||||||
Please don't include changes to `CHANGELOG.md` or `CHANGELOG_zh.md` in your
|
|
||||||
pull request. Maintainers add the changelog entry at merge time: this keeps
|
|
||||||
the Unreleased section's structure and entry ordering consistent and its
|
|
||||||
English and Chinese files in sync, and it avoids conflicts with other
|
|
||||||
in-flight merges, where the Unreleased section changes frequently. If your
|
|
||||||
PR touches the changelog, a reviewer will ask you to drop that hunk — the
|
|
||||||
rest of the PR is unaffected.
|
|
||||||
|
|
||||||
## AI assistance disclosure
|
## AI assistance disclosure
|
||||||
|
|
||||||
DeerFlow is an AI project and we welcome AI-assisted contributions. To help
|
DeerFlow is an AI project and we welcome AI-assisted contributions. To help
|
||||||
@ -350,7 +342,7 @@ before review.
|
|||||||
python -m pip install -r tests/skills/requirements.txt
|
python -m pip install -r tests/skills/requirements.txt
|
||||||
python -m pytest tests/skills -q
|
python -m pytest tests/skills -q
|
||||||
|
|
||||||
# Default backend tests (four parallel shards; excludes live, blocking-I/O and bundled-extension tests)
|
# Default backend tests (four parallel shards; excludes live and blocking-I/O tests)
|
||||||
cd backend
|
cd backend
|
||||||
make test
|
make test
|
||||||
|
|
||||||
@ -360,9 +352,6 @@ make test TEST_JOBS=1
|
|||||||
# Strict blocking-I/O tests
|
# Strict blocking-I/O tests
|
||||||
make test-blocking-io
|
make test-blocking-io
|
||||||
|
|
||||||
# Behavior suites of the bundled extension examples
|
|
||||||
make test-bundled-extensions
|
|
||||||
|
|
||||||
# Live DeerFlowClient integration tests (explicit opt-in)
|
# Live DeerFlowClient integration tests (explicit opt-in)
|
||||||
# Requires a valid root config.yaml and API credentials.
|
# Requires a valid root config.yaml and API credentials.
|
||||||
make test-live
|
make test-live
|
||||||
@ -391,57 +380,9 @@ must also set `DEER_FLOW_RUN_LIVE_TESTS=1`.
|
|||||||
Every pull request triggers the following CI workflows:
|
Every pull request triggers the following CI workflows:
|
||||||
|
|
||||||
- **Backend unit tests** — [.github/workflows/backend-unit-tests.yml](.github/workflows/backend-unit-tests.yml)
|
- **Backend unit tests** — [.github/workflows/backend-unit-tests.yml](.github/workflows/backend-unit-tests.yml)
|
||||||
- **Bundled extension tests** — [.github/workflows/bundled-extension-tests.yml](.github/workflows/bundled-extension-tests.yml) (triggered by example packages, `backend/packages/**`, `backend/app/gateway/**` and the marked test modules)
|
|
||||||
- **Frontend unit tests** — [.github/workflows/frontend-unit-tests.yml](.github/workflows/frontend-unit-tests.yml)
|
- **Frontend unit tests** — [.github/workflows/frontend-unit-tests.yml](.github/workflows/frontend-unit-tests.yml)
|
||||||
- **Frontend E2E tests** — [.github/workflows/e2e-tests.yml](.github/workflows/e2e-tests.yml) (triggered only when `frontend/` files change)
|
- **Frontend E2E tests** — [.github/workflows/e2e-tests.yml](.github/workflows/e2e-tests.yml) (triggered only when `frontend/` files change)
|
||||||
|
|
||||||
The browser E2E and full-stack replay jobs use the official Playwright image,
|
|
||||||
which includes Chromium and its system libraries. When upgrading the locked
|
|
||||||
Playwright version, update the image tags in both browser workflows to match.
|
|
||||||
Each job launches Chromium before running tests; auth recovery tests run only
|
|
||||||
after that browser readiness check succeeds.
|
|
||||||
These containers use uv-managed Python instead of the host's Python tool cache,
|
|
||||||
so the interpreter's OpenSSL libraries match its runtime. Backend dependency
|
|
||||||
setup also verifies SSL and SQLite imports before launching fixture gateways.
|
|
||||||
|
|
||||||
### Test and diagnostics notes
|
|
||||||
|
|
||||||
Backend `make test` runs four duration-balanced shards concurrently and reports
|
|
||||||
failure if any shard fails. Use `make test TEST_JOBS=1` to run the shards sequentially.
|
|
||||||
It excludes live external-API, blocking-I/O and bundled-extension coverage.
|
|
||||||
Run `cd backend && make test-blocking-io` for strict blocking-I/O checks.
|
|
||||||
Maintainers can run the real `DeerFlowClient` suite with `cd backend && make test-live`.
|
|
||||||
This command requires a valid root `config.yaml` and API credentials.
|
|
||||||
It can incur API costs and create local sandboxes, artifacts, or files.
|
|
||||||
Direct pytest runs additionally require `DEER_FLOW_RUN_LIVE_TESTS=1`.
|
|
||||||
|
|
||||||
Bundled extension examples keep their behavior suites in `backend/tests/` under
|
|
||||||
the `bundled_extension` pytest marker. The core shards exclude them so an
|
|
||||||
optional example cannot fail an unrelated core PR, and a separate CI workflow
|
|
||||||
runs them when an example, the public extension contract or the bundled tests
|
|
||||||
change. Run `cd backend && make test-bundled-extensions` to exercise them
|
|
||||||
locally. Tests that validate a host mechanism — loading, registration, the
|
|
||||||
public extension API and authorization — stay in the core shards even when an
|
|
||||||
example package is the payload. The host/extension compatibility checks (an
|
|
||||||
extension requiring a newer API than the host provides, authorization of
|
|
||||||
contributed routes) live in `backend/tests/test_extension_host_contracts.py` so
|
|
||||||
a broken compatibility guarantee fails core CI.
|
|
||||||
|
|
||||||
Regression coverage includes Docker sandbox mode detection and provisioner kubeconfig-path handling tests in `backend/tests/`.
|
|
||||||
Backend blocking-IO diagnostics are available from the repository root with
|
|
||||||
`make detect-blocking-io`: it statically scans backend business code for
|
|
||||||
blocking IO that may run on the backend event loop, prints a concise summary,
|
|
||||||
and writes complete JSON findings to `.deer-flow/blocking-io-findings.json`.
|
|
||||||
The JSON includes compact review records with `priority`, `location`,
|
|
||||||
`blocking_call`, `event_loop_exposure`, `reason`, and `code`.
|
|
||||||
Gateway artifact serving now forces active web content types (`text/html` and XML documents such as `.xml`, `.xhtml`, and `.svg`) to download as attachments instead of inline rendering, reducing XSS risk for generated artifacts.
|
|
||||||
|
|
||||||
Frontend route asset budgets can be checked with `cd frontend && pnpm
|
|
||||||
perf:check`. The command measures `/login` from a normal production build, then
|
|
||||||
performs a production static-demo build for the fixture-backed workspace routes.
|
|
||||||
It measures the unique JavaScript and CSS referenced by representative routes
|
|
||||||
and writes the detailed result to `.next/performance-results.json`.
|
|
||||||
|
|
||||||
## Code Style
|
## Code Style
|
||||||
|
|
||||||
- **Backend (Python)**: We use `ruff` for linting and formatting. Run `make format` before committing.
|
- **Backend (Python)**: We use `ruff` for linting and formatting. Run `make format` before committing.
|
||||||
|
|||||||
@ -1195,7 +1195,7 @@ Agent settings에서 **Tool access**는 설정된 모든 도구 그룹을 상속
|
|||||||
비전을 지원하는 에이전트의 경우 `view_image`와 그 뒤에 이어지는 모델 컨텍스트 이미지 읽기에도 `sandbox:execute`가 필요합니다. 도구 이름만 허용해서는 이미지 파일 접근 권한이 부여되지 않으며, 샌드박스 실행이 거부된 역할은 권한이 변경된 후 이전에 기록된 이미지 메타데이터를 다시 읽을 수 없습니다. 외부 실행 입력과 스레드 상태 업데이트로는 `viewed_images`나 `thread_data`를 설정할 수 없습니다. 저장된 이미지를 호스트 사본에서 읽을 때는 해당 경로가 현재 사용자의 스레드와 기록된 가상 이미지 경로로 해석되어야 합니다.
|
비전을 지원하는 에이전트의 경우 `view_image`와 그 뒤에 이어지는 모델 컨텍스트 이미지 읽기에도 `sandbox:execute`가 필요합니다. 도구 이름만 허용해서는 이미지 파일 접근 권한이 부여되지 않으며, 샌드박스 실행이 거부된 역할은 권한이 변경된 후 이전에 기록된 이미지 메타데이터를 다시 읽을 수 없습니다. 외부 실행 입력과 스레드 상태 업데이트로는 `viewed_images`나 `thread_data`를 설정할 수 없습니다. 저장된 이미지를 호스트 사본에서 읽을 때는 해당 경로가 현재 사용자의 스레드와 기록된 가상 이미지 경로로 해석되어야 합니다.
|
||||||
각 모델 요청에는 최근에 본 이미지가 최대 4개, base64 이미지 데이터가 최대 28 MiB까지 포함됩니다. 이 제한으로 제외된 이미지는 생략된 것으로 보고되며, 에이전트는 `view_image`를 다시 호출해 이미지를 컨텍스트로 다시 가져올 수 있습니다.
|
각 모델 요청에는 최근에 본 이미지가 최대 4개, base64 이미지 데이터가 최대 28 MiB까지 포함됩니다. 이 제한으로 제외된 이미지는 생략된 것으로 보고되며, 에이전트는 `view_image`를 다시 호출해 이미지를 컨텍스트로 다시 가져올 수 있습니다.
|
||||||
|
|
||||||
고급 배포에서는 `config.yaml` 또는 `extensions_config.json`의 `extensions.middlewares` 아래에 `AgentMiddleware` 클래스를 선언하여 에이전트 런타임 자체를 확장할 수도 있습니다. 각 항목은 `module.path:ClassName` 문자열(인자 없는 생성자) 또는 객체 `{class, kwargs, apply_to}`이며, 객체의 `kwargs`는 생성자에 전달됩니다. `kwargs` 값은 JSON 타입(객체, 배열, 문자열, 숫자, 불리언, null)이어야 하며, YAML 날짜와 타임스탬프는 JSON과 일치하도록 ISO 문자열로 변환됩니다. DeerFlow는 각 체인에 대해 선택된 설정 항목을 리드 에이전트와 서브에이전트 파이프라인에 불러오며, 그 위치는 기본 제공 런타임 미들웨어와 루프/토큰 가드 뒤, 그리고 terminal-response/safety/clarification 꼬리 앞입니다. 따라서 엔터프라이즈 포크는 기본 제공 미들웨어 빌더를 패치하지 않고도 도메인 가드레일, 도구 호출 거버넌스, 관측성 훅을 추가할 수 있습니다. 패키지 누락, 잘못된 클래스, 손상된 모듈, 생성자 오류는 에이전트 생성 시점에 명확한 오류로 실패합니다. `config.yaml`과 `extensions_config.json`은 운영자가 제어하는 신뢰된 파일로 취급하세요. 커스텀 도구, 모델, 샌드박스, 가드레일, MCP 서버, MCP 인터셉터 선언과 마찬가지로 미들웨어 경로는 곧 코드 실행입니다. Gateway의 스킬/MCP 토글 엔드포인트는 이 필드를 보존하지만 `extensions.middlewares`에 대한 API 쓰기 경로는 제공하지 않습니다. 객체의 선택적 `apply_to`는 `both`(기본값), `lead`, `subagent`를 지원하며, 문자열 항목과 이 필드를 생략한 객체는 계속 두 파이프라인 모두에 적용됩니다. 해당 체인에서 제외된 항목은 임포트하거나 인스턴스화하지 않습니다.
|
고급 배포에서는 `config.yaml` 또는 `extensions_config.json`의 `extensions.middlewares` 아래에 `AgentMiddleware` 클래스를 선언하여 에이전트 런타임 자체를 확장할 수도 있습니다. 각 항목은 `module.path:ClassName` 문자열(인자 없는 생성자) 또는 객체 `{class, kwargs}`이며, 객체의 `kwargs`는 생성자에 전달됩니다. `kwargs` 값은 JSON 타입(객체, 배열, 문자열, 숫자, 불리언, null)이어야 하며, YAML 날짜와 타임스탬프는 JSON과 일치하도록 ISO 문자열로 변환됩니다. DeerFlow는 설정된 같은 목록을 리드 에이전트와 서브에이전트 파이프라인에 불러오며, 그 위치는 기본 제공 런타임 미들웨어와 루프/토큰 가드 뒤, 그리고 terminal-response/safety/clarification 꼬리 앞입니다. 따라서 엔터프라이즈 포크는 기본 제공 미들웨어 빌더를 패치하지 않고도 도메인 가드레일, 도구 호출 거버넌스, 관측성 훅을 추가할 수 있습니다. 패키지 누락, 잘못된 클래스, 손상된 모듈, 생성자 오류는 에이전트 생성 시점에 명확한 오류로 실패합니다. `config.yaml`과 `extensions_config.json`은 운영자가 제어하는 신뢰된 파일로 취급하세요. 커스텀 도구, 모델, 샌드박스, 가드레일, MCP 서버, MCP 인터셉터 선언과 마찬가지로 미들웨어 경로는 곧 코드 실행입니다. Gateway의 스킬/MCP 토글 엔드포인트는 이 필드를 보존하지만 `extensions.middlewares`에 대한 API 쓰기 경로는 제공하지 않습니다. 리드 에이전트 전용/서브에이전트 전용 미들웨어 목록을 따로 지정하는 기능은 아직 지원되지 않습니다.
|
||||||
|
|
||||||
패키지화되고 설정 가능한 런타임 통합에는 DeerFlow의 확장 관리자를 사용하세요. 확장 관리자는 Python 패키지 요구 사항, 공개 HTTPS Git URL, 또는 로컬 디렉터리를 받아 백엔드 전용 `extensions` 의존성 그룹에 패키지를 설치하고, `backend/uv.lock`을 업데이트하며, 시작 시에만 읽는 `config.yaml`의 최상위 `plugins:` 목록에 활성화된 항목을 추가합니다.
|
패키지화되고 설정 가능한 런타임 통합에는 DeerFlow의 확장 관리자를 사용하세요. 확장 관리자는 Python 패키지 요구 사항, 공개 HTTPS Git URL, 또는 로컬 디렉터리를 받아 백엔드 전용 `extensions` 의존성 그룹에 패키지를 설치하고, `backend/uv.lock`을 업데이트하며, 시작 시에만 읽는 `config.yaml`의 최상위 `plugins:` 목록에 활성화된 항목을 추가합니다.
|
||||||
|
|
||||||
@ -1842,7 +1842,7 @@ WeChat은 바인딩 후 현재 대화 컨텍스트를 안전하게 저장하기
|
|||||||
|
|
||||||
양식이 그룹 알림 지원 여부를 확인하는 동안에는 로딩 안내를 표시하고 저장된 선택을 유지합니다.
|
양식이 그룹 알림 지원 여부를 확인하는 동안에는 로딩 안내를 표시하고 저장된 선택을 유지합니다.
|
||||||
|
|
||||||
그룹은 자동으로 선택되지 않습니다. 선택을 해제하면 새 그룹 알림이 중지되고, 선택을 변경하면 이후 알림에 반영되며, 이미 큐에 들어간 알림은 원래 대상을 유지합니다. 연결이 해제된 바인딩은 대기 중인 알림을 절대 받지 않습니다. 첫 버전에서는 작업 양식 또는 REST API로 설정하며, 대화용 일정 도구의 입력은 기존과 같습니다. [그룹 전달 가이드](backend/docs/IM_CHANNEL_CONNECTIONS.md#group-notifications)를 참고하세요.
|
그룹은 자동으로 선택되지 않습니다. 선택을 해제하면 새 그룹 알림이 중지되고, 선택을 변경하면 이후 알림에 반영되며, 이미 큐에 들어간 알림은 원래 대상을 유지합니다. 연결이 해제된 바인딩은 대기 중인 알림을 절대 받지 않습니다. 첫 버전에서는 작업 양식 또는 REST API로 설정하며, 대화용 일정 도구의 입력은 기존과 같습니다. [그룹 전달 가이드](backend/docs/IM_CHANNEL_CONNECTIONS.md#feishu-group-notifications)를 참고하세요.
|
||||||
|
|
||||||
### 수명 주기, 안전 상한, 중지 조건
|
### 수명 주기, 안전 상한, 중지 조건
|
||||||
|
|
||||||
|
|||||||
22
README_zh.md
22
README_zh.md
@ -109,7 +109,7 @@ DeerFlow 新近集成了 BytePlus 自研的智能搜索与抓取工具集——[
|
|||||||
|
|
||||||
管理员可在 **设置 → 模型 → 添加模型** 中选择厂商，仅填写 **API Key** 和
|
管理员可在 **设置 → 模型 → 添加模型** 中选择厂商，仅填写 **API Key** 和
|
||||||
**模型 ID**。后端预置 OpenAI、Anthropic、DeepSeek、百炼/通义千问、火山/豆包、
|
**模型 ID**。后端预置 OpenAI、Anthropic、DeepSeek、百炼/通义千问、火山/豆包、
|
||||||
Moonshot/Kimi、智谱/Z.AI、MiniMax 和阶跃星辰（StepFun）的端点、认证与适配器，地域和 Coding Plan
|
Moonshot/Kimi、智谱/Z.AI 和 MiniMax 的端点、认证与适配器，地域和 Coding Plan
|
||||||
使用明确区分的选项。模型 ID 不受内置枚举限制；内部名称自动生成且保持稳定。
|
使用明确区分的选项。模型 ID 不受内置枚举限制；内部名称自动生成且保持稳定。
|
||||||
新模型默认启用，保存后自动刷新聊天及 Agent 的模型选择器，无需重启 Gateway。
|
新模型默认启用，保存后自动刷新聊天及 Agent 的模型选择器，无需重启 Gateway。
|
||||||
|
|
||||||
@ -712,18 +712,6 @@ DeerFlow 2.0 不再是一个需要你自己拼装的 framework。它是一个开
|
|||||||
|
|
||||||
## 核心特性
|
## 核心特性
|
||||||
|
|
||||||
### Token 用量统计
|
|
||||||
|
|
||||||
点击侧边栏的 **Token 用量**（`/workspace/usage`），查看近 7/30 天的用量、
|
|
||||||
每日输入/输出 Token、各模型的缓存命中输入 Token，以及支持失败筛选和分页的
|
|
||||||
运行记录。运行记录包含所有日期，不受图表所选周期限制。
|
|
||||||
|
|
||||||
此功能需要 `runs:read` 权限和 SQLite 或 PostgreSQL 数据库，静态演示不提供统计。
|
|
||||||
数据来自已持久化的服务商用量；运行过程中或服务商未返回用量时可能不完整，不能
|
|
||||||
作为服务商账单。缓存命中 Token 已包含在输入 Token 中。每日分桶使用浏览器当前
|
|
||||||
UTC 偏移，并按运行的创建日期归属。页面复用现有 `/api/console` 接口，保留服务端
|
|
||||||
价格配置及可选的费用、币种字段。
|
|
||||||
|
|
||||||
### Skills 与 Tools
|
### Skills 与 Tools
|
||||||
|
|
||||||
Skills 是 DeerFlow 能做“几乎任何事”的关键。
|
Skills 是 DeerFlow 能做“几乎任何事”的关键。
|
||||||
@ -830,11 +818,8 @@ Web UI 输入框支持浏览器侧语音听写。浏览器提供 Web Speech API
|
|||||||
/goal finish the implementation and make all tests pass
|
/goal finish the implementation and make all tests pass
|
||||||
/goal # 查看当前激活的 goal
|
/goal # 查看当前激活的 goal
|
||||||
/goal clear # 清除它
|
/goal clear # 清除它
|
||||||
/goal from-chat # 仅 IM：根据当前对话生成 goal
|
|
||||||
```
|
```
|
||||||
|
|
||||||
在受支持的 IM 渠道里，`/goal from-chat [提示]` 会从当前对话中归纳 goal，而不是把文字当成目标本身。它会保留对话中已经约定的约束和停止条件，展示保存后的目标，并只启动一次 run。命令后的可选提示用于补充约束，不需要重述完整目标。如果对话不足以形成清楚的目标，则不会保存或启动任何内容，回复会说明原因；如果助手还在等你回答，回复会先请你回复。
|
|
||||||
|
|
||||||
每次 Gateway 驱动的 run 结束后，DeerFlow 会用一个 non-thinking 的评估模型，把可见的对话内容（包括助手的工具调用和截短的工具结果）拿去和激活的 goal 比对。工具调用成功本身不代表 goal 已完成；如果助手不得不猜测缺失或含糊的信息，评估模型会报告 `needs_user_input`。评估模型必须返回一个带类型的 blocker（`missing_evidence`、`needs_user_input`、`run_failed`、`external_wait` 或 `goal_not_met_yet`），并附上可见证据。只有在最近一轮 assistant 回复已被持久化 checkpoint、blocker 是 `goal_not_met_yet`、评估期间 thread 没有变化、且无进展熔断器没有触发时，DeerFlow 才会注入一次 hidden continuation。安全上限默认是 8 次 hidden continuation；连续两次相同的无进展评估后就会停止。`/goal clear` 以及任何用户手动输入的新内容，优先级都高于排队中的 continuation。当 goal 被满足时，DeerFlow 会自动清除它，并发布更新后的 thread 状态。同一个 checkpoint 会把这次达成记为 `goal_outcome`，保留到下一次修改 goal。
|
每次 Gateway 驱动的 run 结束后，DeerFlow 会用一个 non-thinking 的评估模型，把可见的对话内容（包括助手的工具调用和截短的工具结果）拿去和激活的 goal 比对。工具调用成功本身不代表 goal 已完成；如果助手不得不猜测缺失或含糊的信息，评估模型会报告 `needs_user_input`。评估模型必须返回一个带类型的 blocker（`missing_evidence`、`needs_user_input`、`run_failed`、`external_wait` 或 `goal_not_met_yet`），并附上可见证据。只有在最近一轮 assistant 回复已被持久化 checkpoint、blocker 是 `goal_not_met_yet`、评估期间 thread 没有变化、且无进展熔断器没有触发时，DeerFlow 才会注入一次 hidden continuation。安全上限默认是 8 次 hidden continuation；连续两次相同的无进展评估后就会停止。`/goal clear` 以及任何用户手动输入的新内容，优先级都高于排队中的 continuation。当 goal 被满足时，DeerFlow 会自动清除它，并发布更新后的 thread 状态。同一个 checkpoint 会把这次达成记为 `goal_outcome`，保留到下一次修改 goal。
|
||||||
|
|
||||||
重新生成此前的回答时，会沿用当前对话的 goal 和已用续跑预算，不会恢复已清除或已达成的旧 goal。重放历史 checkpoint 时会移除旧回答的达成记录，避免把它当作新回答已经达成的证明。
|
重新生成此前的回答时，会沿用当前对话的 goal 和已用续跑预算，不会恢复已清除或已达成的旧 goal。重放历史 checkpoint 时会移除旧回答的达成记录，避免把它当作新回答已经达成的证明。
|
||||||
@ -857,11 +842,6 @@ Web UI 会在输入框上方展示当前激活的 goal，说明自动续跑为
|
|||||||
|
|
||||||
### Sub-Agents
|
### Sub-Agents
|
||||||
|
|
||||||
显式后台 `batch_task` 可用 `depends_on` 声明同批条目的前置关系，前置工作完成后即可继续，
|
|
||||||
无需等待无关任务。用户已要求后续汇总或交付时，可用可选的 `continuation` 登记一次主 Agent 接续；
|
|
||||||
启用 Goal 时仍由 Goal 负责续跑。执行、取消、授权和恢复范围见
|
|
||||||
[批次依赖与完成接续说明](backend/docs/BATCH_CONTINUATION.md)。
|
|
||||||
|
|
||||||
Sub-agent 是一种执行优化，而不是遇到复杂任务时的默认选择。
|
Sub-agent 是一种执行优化，而不是遇到复杂任务时的默认选择。
|
||||||
|
|
||||||
lead agent 只会在委派具有明确净收益时动态拉起 sub-agents，例如真正缩短耗时的并行工作、专业能力收益或上下文隔离收益。存在跨 Agent 依赖或重叠副作用的工作不会并行分派；当专业能力或上下文隔离收益明显占优时，一条有界的顺序任务链仍可交给一个 sub-agent 完成。lead agent 会使用能取得收益的最少 sub-agents，并在每一批完成后重新评估，而不会仅仅因为任务规模大或步骤多就继续拆分。每个 sub-agent 都有自己独立的上下文、工具和终止条件，返回结构化结果后由 lead agent 验证并汇总成完整输出。
|
lead agent 只会在委派具有明确净收益时动态拉起 sub-agents，例如真正缩短耗时的并行工作、专业能力收益或上下文隔离收益。存在跨 Agent 依赖或重叠副作用的工作不会并行分派；当专业能力或上下文隔离收益明显占优时，一条有界的顺序任务链仍可交给一个 sub-agent 完成。lead agent 会使用能取得收益的最少 sub-agents，并在每一批完成后重新评估，而不会仅仅因为任务规模大或步骤多就继续拆分。每个 sub-agent 都有自己独立的上下文、工具和终止条件，返回结构化结果后由 lead agent 验证并汇总成完整输出。
|
||||||
|
|||||||
74392
backend/.test_durations
74392
backend/.test_durations
File diff suppressed because one or more lines are too long
@ -1,63 +1,327 @@
|
|||||||
# Backend (`backend/`)
|
# AGENTS.md
|
||||||
|
|
||||||
The Python backend: the publishable harness in `packages/harness/deerflow/`, the
|
## Project Overview
|
||||||
public extension contract in `packages/extension-api/`, and the unpublished
|
|
||||||
Gateway and IM channels in `app/`. In every deployment the agent runtime runs
|
|
||||||
inside the Gateway. Design docs are in [docs/](docs/); start with
|
|
||||||
[ARCHITECTURE.md](docs/ARCHITECTURE.md) and [API.md](docs/API.md).
|
|
||||||
|
|
||||||
## Guides by area
|
The LangGraph backend provides sandbox execution, persistent memory, subagent delegation, and extensible tools, isolated per thread.
|
||||||
|
|
||||||
Read the guide for an area before you change it.
|
Service roles and ports: [root service topology](../AGENTS.md#service-topology).
|
||||||
|
|
||||||
| Area | Guide |
|
**Runtime**:
|
||||||
| --- | --- |
|
- `make dev`, Docker dev, and production all run the agent runtime in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Nginx exposes that runtime at `/api/langgraph/*` and rewrites it to Gateway's native `/api/*` routers.
|
||||||
| Run streaming, checkpoints, run ownership | [runtime](packages/harness/deerflow/runtime/AGENTS.md) |
|
- Gateway streams `write_file` and `str_replace` argument deltas in bounded batches for multi-mode `messages-tuple` consumers; single-mode message consumers retain the original per-chunk contract. Non-message frames flush pending batches, and `values` remains an optional complete-state snapshot rather than a prerequisite for batching.
|
||||||
| Repositories, leases, scheduled tasks, schema changes | [persistence](packages/harness/deerflow/persistence/AGENTS.md), [migrations](packages/harness/deerflow/persistence/migrations/AGENTS.md) |
|
- `stream_subgraphs` preserves subgraph SSE namespaces (`values|<ns>`, LangGraph Platform style); bare `values` would overwrite the SDK thread view (#4399). Delegated subagents inherit the parent's checkpoint namespace. Root-only consumers (file-tool chunk batcher, subagent event persistence, LLM error-fallback detection) ignore namespaced frames. The web frontend uses root `task_*` events without subgraph streaming.
|
||||||
| Scheduler failure isolation | [design conventions](app/scheduler/guide/failure-isolation.md) |
|
- Background subagent identity is deliberately split: the provider `tool_call_id` remains the correlation key for `ToolMessage`, `task_*` SSE events, persisted lifecycle events, frontend cards, and the public `ExtensionData.scope_id` contract (stored as `SubagentResult.external_task_id`), while `SubagentExecutor.execute_async()` generates a full server-side `execution_id` for `SubagentResult.task_id`, the process-wide registry, polling, cancellation, timeout handling, and cleanup. Provider IDs are not globally unique across parent runs, so they must never become registry ownership keys; scheduler closures retain their own `SubagentResult` rather than resolving ownership again through the mutable registry. Terminal subagent token usage travels in the current run's `ToolMessage.additional_kwargs` and is attributed from message state, never through a process-global provider-ID cache.
|
||||||
| `config.yaml`, `extensions_config.json` | [config](packages/harness/deerflow/config/AGENTS.md) |
|
- Scheduled runs use Gateway's normal run path. `launch_scheduled_thread_run` reads `get_app_config()` at dispatch: `scheduler.recursion_limit` defaults to 1000 (web UI), capped by `max_recursion_limit`; YAML changes apply next run without restart. IM enqueue checks recipients before locking preferences in the outcome transaction.
|
||||||
| Lead agent, prompt, thread state, middleware chain | [agents](packages/harness/deerflow/agents/AGENTS.md), [middlewares](packages/harness/deerflow/agents/middlewares/AGENTS.md) |
|
- Run-history `status` filters are occurrence states, not task states. `ScheduledTaskRunStatus` in `persistence/scheduled_tasks/model.py` is the shared API/repository vocabulary and must match the active and terminal occurrence-status sets. Keep owner lookup before reading history, and apply SQL task/status predicates before pagination; omitted status preserves the existing response.
|
||||||
| Memory, subagents, skills, tools | [memory](packages/harness/deerflow/agents/memory/AGENTS.md), [subagents](packages/harness/deerflow/subagents/AGENTS.md), [skills](packages/harness/deerflow/skills/AGENTS.md), [tools](packages/harness/deerflow/tools/AGENTS.md) |
|
- The background scheduler is single-instance by default. `scheduler.multi_instance=true` opts into lease-aware recovery across Gateway instances and requires shared Postgres, `run_ownership.heartbeat_enabled=true`, and `run_events.backend=db`; otherwise startup rejects the configuration. Live scheduled runs are preserved when a peer starts; expired launch claims return to the durable queue, expired run leases are atomically taken over, stale launch writes are fenced by lease ownership, and the Postgres advisory-locked budget makes `max_concurrent_runs` a shared global cap for `launching`/`running` rows.
|
||||||
| Sandbox, uploads, MCP, models, extensions | [sandbox](packages/harness/deerflow/sandbox/AGENTS.md), [uploads](packages/harness/deerflow/uploads/AGENTS.md), [mcp](packages/harness/deerflow/mcp/AGENTS.md), [models](packages/harness/deerflow/models/AGENTS.md), [extensions](packages/harness/deerflow/extensions/AGENTS.md) |
|
- The multi-process startup gate also fires on `deployment.multi_instance: true`; contract in `docs/CONFIGURATION.md`.
|
||||||
| Trace ID, embedded client, other root modules | [harness package](packages/harness/deerflow/AGENTS.md) |
|
- Long-running MCP work uses a separate durable task runtime (`McpTaskService` + `mcp_tasks`, lease-based recovery) rather than keeping remote task IDs or status polling inside the Agent loop; only submit remains Agent-visible, the database is the source of truth, and `ThreadState` receives only a bounded current-thread projection. Full contract — including notification retries, dead-lettering, and the cancel endpoint's worker-stopped 503 (leases, cancellation fencing, delivery idempotency, management-tool exposure): [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md).
|
||||||
| Search, fetch and sandbox providers | [community](packages/harness/deerflow/community/AGENTS.md), then the provider's own guide |
|
- Scheduled-task dispatch permits one active occurrence per task via `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). Durable `queued` rows survive restarts; only lease-fenced `launching` may call Gateway launch; `running` references the durable run. Stable admission idempotency keys reuse that run after recovery. Reused-thread `ConflictError` returns `launching` to `queued`; other launch errors become `failed`. Atomic queue claims enforce `max_concurrent_runs` and the per-owner cap, excluding waiting rows; the budget count and its UPDATE are separate statements, so writers must serialize before the count (Postgres advisory lock; SQLite `BEGIN IMMEDIATE`, whose deferred transaction otherwise reserves the writer only at the UPDATE) or claims on distinct rows overshoot the cap. Repeated triggers coalesce; same-thread FIFO blocks behind older active rows. Queue admission, PATCH/resume, pause and delete lock the parent before the occurrence, freezing active task definitions. Pause/delete atomically cancel `queued` work but reject `launching`/`running`; PATCH/resume reject all active states. Only queued conflicts offer pause cancellation. Manual triggers may queue/run while paused. Manual-trial run origins reuse the admitted `scheduled_for`, so an idempotent retry submits the same request instead of colliding with the first attempt.
|
||||||
| Gateway, routers, IM channels | [gateway](app/gateway/AGENTS.md), [routers](app/gateway/routers/AGENTS.md), [channels](app/channels/AGENTS.md) |
|
- Scheduled-task lifecycle, recovery and queue timeout: [persistence guide](packages/harness/deerflow/persistence/AGENTS.md#scheduled-task-lifecycle).
|
||||||
| Tests, benchmarks | [tests](tests/AGENTS.md), [benchmarks](scripts/benchmark/AGENTS.md) |
|
- `POST /api/scheduled-tasks/preview-cron` requires authenticated `threads:read`. Bounded cron previews call the shared scheduler calculator in `asyncio.to_thread`, preserving its DST semantics. Capture the optional aware reference once; return UTC and offset-bearing local occurrences without acquiring task/thread/run stores or dispatching work. This advisory API does not reserve execution.
|
||||||
|
- Gateway MCP/skill updates write `extensions_config.json` at runtime. Production Compose mounts it read-write (`config.yaml` stays `:ro`); Helm seeds a writable home-volume copy. Hold both `extensions_config_write_lock` and the sidecar advisory `extensions_config_file_lock` during read-modify-write to prevent lost updates across workers. `atomic_write_extensions_config` uses temp-file replacement; only mount-point `EBUSY` permits a non-atomic in-place overwrite (crashes can truncate). Warn about this fallback once per target; propagate other errors. `get_extensions_config()` revalidates reads by file signature; see [config caching](packages/harness/deerflow/config/AGENTS.md). Tests: `tests/test_compose_extensions_config_writable.py`, `tests/test_extensions_config_atomic_write.py`, `tests/test_helm_extensions_config_writable.py`.
|
||||||
|
- MCP cache reset scope and discovery fencing: [MCP guide](packages/harness/deerflow/mcp/AGENTS.md).
|
||||||
|
|
||||||
## Verify
|
**Project structure:** The [root repository map](../AGENTS.md#repository-map)
|
||||||
|
indexes the stack. Backend code lives in `app/` (Gateway and channels),
|
||||||
|
`packages/harness/deerflow/` (agent runtime), and `packages/extension-api/`
|
||||||
|
(public extension contracts). Tests, detailed docs, and standalone benchmarks
|
||||||
|
live in `tests/`, `docs/`, and `scripts/benchmark/` respectively. Module-local
|
||||||
|
`AGENTS.md` files describe their implementation boundaries.
|
||||||
|
|
||||||
|
## Important Development Guidelines
|
||||||
|
|
||||||
|
### Documentation Update Policy
|
||||||
|
Every code change must keep docs accurate and current: update `README.md` for
|
||||||
|
user-facing behavior and the relevant `AGENTS.md` for development changes.
|
||||||
|
`CLAUDE.md` imports `AGENTS.md`; do not edit the shim.
|
||||||
|
|
||||||
|
Image result URL policy: [image search guide](packages/harness/deerflow/community/image_search/AGENTS.md).
|
||||||
|
|
||||||
|
### Backend Benchmarks
|
||||||
|
|
||||||
|
`scripts/benchmark/context_snapshot/`: explicit `run-live` needs provider env
|
||||||
|
vars; `summarize` and pytest are offline. See its README for the protocol.
|
||||||
|
|
||||||
|
Benchmarks in `scripts/benchmark/` must be standalone and reproducible. Import
|
||||||
|
production functions; never duplicate them or introduce an alternative runtime.
|
||||||
|
|
||||||
|
Sandbox benchmarks reuse production lifecycle. Reject negative/NaN timing values
|
||||||
|
before provider creation, export no secrets, and fail on final cleanup errors.
|
||||||
|
See the sandbox benchmark README for supported options and tests.
|
||||||
|
|
||||||
|
- Pin every external dataset by immutable revision and SHA-256. Callers provide
|
||||||
|
the local dataset path; evaluation commands must not silently download data.
|
||||||
|
- Never commit upstream dataset text, credentials, complete provider requests,
|
||||||
|
or response headers. Committed manifests may contain stable IDs and source
|
||||||
|
locators. Synthetic cases must identify themselves as synthetic.
|
||||||
|
- Read provider credentials and endpoints from named environment variables.
|
||||||
|
Version model IDs, inference parameters, prompts, retry rules, clocks, and
|
||||||
|
random seeds in the evaluation config.
|
||||||
|
- Public raw results may contain case IDs, policy decisions, model hypotheses,
|
||||||
|
grades, and non-secret response metadata. Keep dataset questions, reference
|
||||||
|
answers, memory content, and full provider payloads in ignored local run
|
||||||
|
directories.
|
||||||
|
- Use fixed clocks and deterministic ordering for offline selection. Results
|
||||||
|
must record the config, manifest, prompt, dataset, and git revisions used.
|
||||||
|
|
||||||
|
`scripts/benchmark/deermem_eviction/` evaluates the production
|
||||||
|
`select_facts_for_capacity()` implementation used by DeerMem. It compares only
|
||||||
|
the historical `confidence` policy and PR #4789's opt-in `hybrid-v1`; do not add
|
||||||
|
another eviction strategy to this evaluation. Run its offline checks from
|
||||||
|
`backend/`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
make test # offline suite in parallel shards
|
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate-contracts
|
||||||
make test-blocking-io # strict blocking-I/O suite
|
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate --dataset "$LONGMEMEVAL_ORACLE_PATH"
|
||||||
make test-bundled-extensions # behavior suites of bundled examples (not in make test)
|
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction run-policy \
|
||||||
make lint
|
--dataset "$LONGMEMEVAL_ORACLE_PATH" \
|
||||||
make format
|
--output-dir /tmp/deermem-eviction-policy-run
|
||||||
make test-live # calls real APIs; needs DEER_FLOW_RUN_LIVE_TESTS=1
|
PYTHONPATH=. uv run pytest tests/test_bench_deermem_eviction_*.py -q
|
||||||
```
|
```
|
||||||
|
|
||||||
Run `make test` and `make test-blocking-io` before and after a change. Every
|
The offline test suite must not require network access, provider credentials,
|
||||||
feature and bug fix ships with a test in `tests/`. Live tests stay opt-in, so the
|
or the LongMemEval dataset. Small LongMemEval-shaped fixtures must be synthetic
|
||||||
default suite needs no network and no API key.
|
and generated by tests.
|
||||||
|
|
||||||
## Rules
|
`scripts/benchmark/concurrency/` measures multi-process contention on the
|
||||||
|
`users` table (N separate OS processes, not asyncio tasks) for SQLite vs
|
||||||
|
Postgres -- the scenario `CONFIGURATION.md` requires Postgres for. `worker.py`
|
||||||
|
connects directly via SQLAlchemy (skipping the ~8.5s Alembic bootstrap the
|
||||||
|
orchestrator already ran once) and mirrors the app's per-connection SQLite
|
||||||
|
PRAGMAs; `run_concurrency_bench.py` seeds a disposable per-run Postgres schema,
|
||||||
|
synchronises workers on a READY/GO barrier before timing, and exits non-zero on
|
||||||
|
any crash, short op count, or `errors > 0`. Postgres runs need a throwaway
|
||||||
|
database via `--pg-url`; nothing here touches `public`. Run from `backend/`:
|
||||||
|
|
||||||
- **The harness never imports `app`.** `app.*` imports `deerflow.*`, and not the
|
```bash
|
||||||
reverse, because the harness is published on its own.
|
uv run python scripts/benchmark/concurrency/run_concurrency_bench.py \
|
||||||
`tests/test_harness_boundary.py` enforces this in CI.
|
--backend sqlite --workers 2,4,8,16 --ops-per-worker 50 --read-ratio 0.7
|
||||||
- **Start the Gateway with `make dev` or `make gateway`.** `make dev` excludes
|
uv run pytest tests/test_bench_concurrency.py tests/test_bench_worker.py -q
|
||||||
`DEER_FLOW_HOME` and `backend/sandbox` from the reload watcher. A bare
|
```
|
||||||
`uvicorn --reload` restarts the Gateway during a run, because agent tasks write
|
|
||||||
Python files below `DEER_FLOW_HOME`.
|
## Commands
|
||||||
- **From the repository root, run backend tools with
|
|
||||||
`uv run --no-sync --project backend`.** `--project` keeps the caller's working
|
Root orchestration and extension commands live in the
|
||||||
directory, so relative config paths still resolve. `--no-sync` keeps installed
|
[root guide](../AGENTS.md#commands-root-vs-module); run them from the repo root.
|
||||||
extras such as the PostgreSQL drivers.
|
`make detect-thread-boundaries` inventories backend executor, thread and event-loop
|
||||||
- **Run Python through `uv run` inside `backend/`**, without `--no-sync`, so the
|
boundaries.
|
||||||
environment matches the lock.
|
|
||||||
- **Guard POSIX-only test markers with `os.name`.** Windows collects the same
|
**Backend directory** (for backend development only):
|
||||||
files.
|
```bash
|
||||||
- **Tool approvals need addressable calls.** Strip missing, empty and duplicate
|
make install # Install backend dependencies
|
||||||
tool-call IDs before creating a pause or a tool result. See the
|
make dev # Gateway API, reload (port 8001)
|
||||||
[run interaction policy](docs/RUN_INTERACTION_POLICY.md#hitl-tool-approval).
|
make gateway # Gateway API only (port 8001)
|
||||||
- **Add a schema change with `make migrate-rev MSG="..."`** and follow the
|
make test # four offline workers; TEST_JOBS=1 for serial
|
||||||
migrations guide.
|
make test-live # live tests (real APIs)
|
||||||
|
make test-blocking-io # strict Blockbuster gate on tests/blocking_io/
|
||||||
|
make test-shard SPLITS=4 GROUP=2 # one duration-aware shard
|
||||||
|
make test-shard-durations # refresh baseline
|
||||||
|
make lint # ruff lint
|
||||||
|
make format # ruff format
|
||||||
|
make migrate-rev MSG="..." # Autogenerate a new alembic revision (see Schema Migrations section)
|
||||||
|
```
|
||||||
|
|
||||||
|
The backend `make dev` target pre-creates and excludes `DEER_FLOW_HOME`
|
||||||
|
(default: `backend/.deer-flow`) and `backend/sandbox` from Uvicorn's reload
|
||||||
|
watcher. Do not replace it with a bare `uvicorn --reload`: agent tasks write
|
||||||
|
Python and other runtime files below `DEER_FLOW_HOME`, which would otherwise
|
||||||
|
restart the Gateway during an active run.
|
||||||
|
|
||||||
|
Configuration checks and operator migration scripts reuse the installed backend
|
||||||
|
environment with `uv run --no-sync --project backend` from the repository root.
|
||||||
|
`--project` preserves the caller's working directory and relative runtime/config
|
||||||
|
selectors; `--no-sync` retains installed extras such as PostgreSQL drivers.
|
||||||
|
|
||||||
|
Read the nearest scoped `AGENTS.md` for subsystem ownership.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
Read-only knowledge: [contract](docs/KNOWLEDGE.md) covers independent retrieval
|
||||||
|
tools, scope intersection and extension-owned providers.
|
||||||
|
|
||||||
|
### Harness / App Split
|
||||||
|
|
||||||
|
The backend is split into two layers with a strict dependency direction:
|
||||||
|
|
||||||
|
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
|
||||||
|
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk). Feishu option clarifications can render as interactive button cards behind `channels.feishu.card_interaction_enabled` (default off); see `app/channels/AGENTS.md`.
|
||||||
|
|
||||||
|
Channel credential staging (`app/channels/credential_staging.py`) is swept on worker-thread state loads, including SQL legacy import. Only owned regular files older than 24h are removed; leave anonymous temporary files alone and keep WeChat constructors IO-free.
|
||||||
|
|
||||||
|
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||||||
|
|
||||||
|
**Import conventions**:
|
||||||
|
```python
|
||||||
|
# Harness internal
|
||||||
|
from deerflow.agents import make_lead_agent
|
||||||
|
from deerflow.models import create_chat_model
|
||||||
|
|
||||||
|
# App internal
|
||||||
|
from app.gateway.app import app
|
||||||
|
from app.channels.service import start_channel_service
|
||||||
|
|
||||||
|
# App → Harness (allowed)
|
||||||
|
from deerflow.config import get_app_config
|
||||||
|
|
||||||
|
# Harness → App (FORBIDDEN — enforced by test_harness_boundary.py)
|
||||||
|
# from app.gateway.routers.uploads import ... # ← will fail CI
|
||||||
|
```
|
||||||
|
|
||||||
|
Package import hygiene: the `deerflow.agents` and `deerflow.subagents` package
|
||||||
|
roots expose heavyweight graph/executor entrypoints lazily. The
|
||||||
|
`deerflow.agents:make_lead_agent` LangGraph Server entrypoint is a concrete thin
|
||||||
|
module-level function because the server resolves graph factories directly from
|
||||||
|
the module dictionary; the wrapper keeps the lead-agent and skill-cache imports
|
||||||
|
inside the function so importing the package remains lightweight. Internal
|
||||||
|
modules that only need lightweight types, config, or registries should import
|
||||||
|
the concrete submodule instead of adding eager package-root imports that pull in
|
||||||
|
the tool graph or subagent executor during state/schema imports.
|
||||||
|
|
||||||
|
`ThreadMetaStore.search()` keeps JSON filter semantics identical across memory,
|
||||||
|
SQLite, and PostgreSQL: missing differs from null, bool differs from int, and
|
||||||
|
float filters accept integer or real JSON numbers through `json_value_matches`.
|
||||||
|
|
||||||
|
### Gateway Run-Context Trust Boundary
|
||||||
|
|
||||||
|
Gate server-owned run context on both client feeds (`body.context` and
|
||||||
|
`body.config`): `merge_run_context_overrides` admits it only for `internal=True`,
|
||||||
|
while `strip_internal_context_keys` scrubs both destinations. A validated
|
||||||
|
`context.locale` (`en-US` / `zh-CN`) is a runtime-only language preference for
|
||||||
|
replies and memory writes; it never persists into checkpoint `configurable`,
|
||||||
|
and an explicit invalid/null top-level value clears a legacy preference. Treat
|
||||||
|
`disable_clarification` like `non_interactive`. Before run/state writes,
|
||||||
|
`_normalize_input_messages` rejects canonical external system/developer roles;
|
||||||
|
only `AUTH_SOURCE_INTERNAL` run input may retain them.
|
||||||
|
|
||||||
|
`contracts/ui_locales.v1.json` pins the UI/Gateway locale allowlists in tests;
|
||||||
|
DeerMem's portable map stays separate. Scheduled/MCP notification runs omit
|
||||||
|
locale; persisted owner locale is used for scheduled IM notices only.
|
||||||
|
|
||||||
|
`start_run` assigns IDs to ID-less messages before recording input or invoking
|
||||||
|
the graph; keyed retries keep those IDs stable, while the pre-stamp input
|
||||||
|
remains the idempotency identity. The record and checkpoint share IDs so
|
||||||
|
frontend reconnect can deduplicate the user's message.
|
||||||
|
|
||||||
|
## Development Workflow
|
||||||
|
|
||||||
|
### Test-Driven Development (TDD) — MANDATORY
|
||||||
|
|
||||||
|
**Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.**
|
||||||
|
|
||||||
|
- Write tests in `backend/tests/` following the existing naming convention `test_<feature>.py`
|
||||||
|
- Run both offline targets before and after your change: `make test` and `make test-blocking-io`
|
||||||
|
- Tests must pass before a feature is considered complete
|
||||||
|
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
|
||||||
|
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
|
||||||
|
|
||||||
|
Keep live tests opt-in via `DEER_FLOW_RUN_LIVE_TESTS=1`; guard POSIX-only
|
||||||
|
markers with `os.name` for Windows collection.
|
||||||
|
|
||||||
|
Jina logging tests use dummy keys; the Jina redirect regression runs offline via `httpx.MockTransport` and keeps POST across temporary redirects; InfoQuest connect/read timeout is 30s, separate from crawl timeouts (`tests/test_jina_client.py`, `tests/test_infoquest_http_timeout.py`).
|
||||||
|
|
||||||
|
### Running the Full Application
|
||||||
|
|
||||||
|
Run `make dev` from the repo root to start all services at `http://localhost:2026`.
|
||||||
|
|
||||||
|
**All startup modes:**
|
||||||
|
|
||||||
|
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||||
|
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||||
|
|
||||||
|
| Action | Local | Docker Dev | Docker Prod |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||||
|
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||||
|
|
||||||
|
### Running Backend Services Separately
|
||||||
|
|
||||||
|
From the **backend** directory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Gateway API
|
||||||
|
make gateway
|
||||||
|
```
|
||||||
|
|
||||||
|
Direct access (without nginx):
|
||||||
|
- Gateway: `http://localhost:8001`
|
||||||
|
|
||||||
|
### Frontend Configuration
|
||||||
|
|
||||||
|
The frontend uses environment variables to connect to backend services:
|
||||||
|
- `NEXT_PUBLIC_LANGGRAPH_BASE_URL` - Defaults to `/api/langgraph` (through nginx)
|
||||||
|
- `NEXT_PUBLIC_BACKEND_BASE_URL` - Defaults to empty string (through nginx)
|
||||||
|
|
||||||
|
When using `make dev` from root, the frontend automatically connects through nginx.
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
### Web Search Recency
|
||||||
|
|
||||||
|
Optional `web_search` `time_range=day|week|month|year` mappings:
|
||||||
|
DDG → `d|w|m|y`; Brave → `pd|pw|pm|py`; Tavily/SearXNG → unchanged;
|
||||||
|
Sofya → unchanged in `freshness`; Serper → `qdr:d|w|m|y` in `tbs`.
|
||||||
|
Omission preserves request shape; Serper null also omits `tbs`. Image search is unchanged.
|
||||||
|
DDGS 9.14.1 recency uses only enabled Brave/DuckDuckGo/Yahoo engines honoring
|
||||||
|
`timelimit`: `auto`/`all` selects this set; drop incompatible engines and fall back
|
||||||
|
to this set if empty. Re-check on DDGS upgrades.
|
||||||
|
|
||||||
|
### Tavily Fetch
|
||||||
|
|
||||||
|
Title fallback: result URL, then request URL.
|
||||||
|
|
||||||
|
### File Upload
|
||||||
|
|
||||||
|
Outlines use ATX syntax (1–6 hashes, space/tab separator, ≤3 leading spaces), strip closing hashes and skip fenced code; closing markers use a linear suffix scan — never unanchored whitespace regex on unbounded headings.
|
||||||
|
- Endpoint: `POST /api/threads/{thread_id}/uploads`; PDF/PPT/Excel/Word via `markitdown`, one conversion worker per request; directories rejected, keeping uploads all-or-nothing; duplicate filenames in one request get `_N` suffixes.
|
||||||
|
- Files live in thread-isolated `users/{user_id}/threads/{thread_id}/user-data/uploads` (IM channels thread the owner via the `user_id=` kwarg — see IM Channels → Owner-scoped file storage; HTTP/embedded callers resolve `get_effective_user_id()`); per-thread `upload-companions/` holds source mtime/ctime.
|
||||||
|
- Gateway HTTP uploads stage `.upload-*.part` files hidden from upload listings, agent context, and sandbox listings/searches; publication is atomic after size validation. Startup sweep keeps lone `.part` files under 24h (possibly in flight on another replica); `st_nlink > 1` means a published alias, removed at any age so the multi-link check can still replace the file.
|
||||||
|
- Upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io` (ContextVar-preserving); non-mounted sandbox uploads use `SandboxProvider.acquire_async()` and offload `read_bytes()` + `sandbox.update_file()` together. Mounted uploads skip acquire/sync; AIO remote/provisioner requires accurate `sandbox.thread_data_mounts: true`, omission keeps backend auto-detection.
|
||||||
|
- `UploadsMiddleware` caps outline titles at 200 characters and previews at 2000 including markers. Titles use `original_user_content`; attachment-only titles use a sanitized, bounded filename or count.
|
||||||
|
|
||||||
|
See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details.
|
||||||
|
|
||||||
|
### Plan Mode
|
||||||
|
|
||||||
|
`config.configurable.is_plan_mode=True` enables TodoList `write_todos` for
|
||||||
|
multi-step tasks: one `in_progress` task, real-time updates. See
|
||||||
|
[usage](docs/plan_mode_usage.md).
|
||||||
|
|
||||||
|
### Run Interaction Policy
|
||||||
|
|
||||||
|
Interaction-sensitive changes must follow [policy](docs/RUN_INTERACTION_POLICY.md).
|
||||||
|
|
||||||
|
### Context Summarization
|
||||||
|
|
||||||
|
Automatic conversation summarization when approaching token limits, configured
|
||||||
|
under the `summarization` key in `config.yaml` (triggers: tokens, messages, or
|
||||||
|
fraction of max input); recent messages are kept while older ones are summarized.
|
||||||
|
- Manual compaction uses `POST /api/threads/{id}/compact`, reuses the same `DeerFlowSummarizationMiddleware`, writes a new checkpoint with updated `messages`/`summary_text`, and bumps only those channel versions. The route uses the shared `reserve_checkpoint_write()` boundary (also used by manual state updates); its short-lived `checkpoint_write` operation shares the durable active-thread uniqueness constraint with run admission, preventing worker-local and cross-worker checkpoint-write races.
|
||||||
|
- Cache only first-candidate no-ops; never suppress primary retries with fallback results. See [reuse and telemetry](docs/summarization.md#reuse-and-telemetry) for cache and counter contracts.
|
||||||
|
|
||||||
|
See [docs/summarization.md](docs/summarization.md) for details.
|
||||||
|
|
||||||
|
### Vision Support
|
||||||
|
|
||||||
|
For models with `supports_vision: true`: `ViewImageMiddleware` processes images and adds `view_image_tool`. Images reach the model as base64 in a hidden message carrying a reserved ID prefix plus a server-owned metadata marker (Gateway strips the marker from untrusted input; the middleware requires both to recognize its own message). The injection happens inside `wrap_model_call`, so payloads never enter graph state — checkpoints keep only lightweight `viewed_images` metadata, and client-chosen IDs survive; the middleware also sweeps its own message out of every request before rebuilding it, so a payload stranded in an older checkpoint by an interrupted run stops being resent.
|
||||||
|
|
||||||
|
## Code Style
|
||||||
|
|
||||||
|
- Uses `ruff` for linting and formatting
|
||||||
|
- Line length: 240 characters
|
||||||
|
- Python 3.12+ with type hints
|
||||||
|
- Double quotes, space indentation
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
See `docs/` directory for detailed documentation:
|
||||||
|
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
|
||||||
|
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
|
||||||
|
- [API.md](docs/API.md) - API reference
|
||||||
|
- [SETUP.md](docs/SETUP.md) - Setup guide
|
||||||
|
- [FILE_UPLOAD.md](docs/FILE_UPLOAD.md) - File upload feature
|
||||||
|
- [PATH_EXAMPLES.md](docs/PATH_EXAMPLES.md) - Path types and usage
|
||||||
|
- [summarization.md](docs/summarization.md) - Context summarization
|
||||||
|
- [plan_mode_usage.md](docs/plan_mode_usage.md) - Plan mode with TodoList
|
||||||
|
|||||||
@ -22,7 +22,7 @@ gateway:
|
|||||||
# Run all four duration-balanced shards locally; wait for every shard on failure.
|
# Run all four duration-balanced shards locally; wait for every shard on failure.
|
||||||
TEST_JOBS ?= 4
|
TEST_JOBS ?= 4
|
||||||
TEST_SHARD_TARGETS := test-shard-1 test-shard-2 test-shard-3 test-shard-4
|
TEST_SHARD_TARGETS := test-shard-1 test-shard-2 test-shard-3 test-shard-4
|
||||||
.PHONY: test test-shard test-bundled-extensions $(TEST_SHARD_TARGETS)
|
.PHONY: test test-shard $(TEST_SHARD_TARGETS)
|
||||||
test:
|
test:
|
||||||
$(MAKE) --no-print-directory -k -j$(TEST_JOBS) $(TEST_SHARD_TARGETS)
|
$(MAKE) --no-print-directory -k -j$(TEST_JOBS) $(TEST_SHARD_TARGETS)
|
||||||
|
|
||||||
@ -49,21 +49,13 @@ DURATIONS_FILE ?= .test_durations
|
|||||||
test-shard:
|
test-shard:
|
||||||
@test -f "$(DURATIONS_FILE)" || \
|
@test -f "$(DURATIONS_FILE)" || \
|
||||||
(echo "error: $(DURATIONS_FILE) is missing; run 'make test-shard-durations' to regenerate the duration baseline" >&2; exit 1)
|
(echo "error: $(DURATIONS_FILE) is missing; run 'make test-shard-durations' to regenerate the duration baseline" >&2; exit 1)
|
||||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live and not bundled_extension" --ignore=tests/blocking_io --splits $(SPLITS) --group $(GROUP) --splitting-algorithm least_duration --durations-path=$(DURATIONS_FILE) tests/ -q
|
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live" --ignore=tests/blocking_io --splits $(SPLITS) --group $(GROUP) --splitting-algorithm least_duration --durations-path=$(DURATIONS_FILE) tests/ -q
|
||||||
|
|
||||||
# Regenerate .test_durations from the offline core suite (non-live,
|
# Regenerate .test_durations from the full offline (non-live, non-blocking-I/O)
|
||||||
# non-blocking-I/O, non-bundled-extension) so duration-aware sharding reflects
|
# suite so duration-aware sharding reflects real wall-clock cost. Run locally or
|
||||||
# real wall-clock cost. Run locally or in a periodic job, then commit the
|
# in a periodic job, then commit the refreshed file.
|
||||||
# refreshed file.
|
|
||||||
test-shard-durations:
|
test-shard-durations:
|
||||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live and not bundled_extension" --ignore=tests/blocking_io --store-durations --clean-durations --durations-path=$(DURATIONS_FILE) tests/ -q
|
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live" --ignore=tests/blocking_io --store-durations --clean-durations --durations-path=$(DURATIONS_FILE) tests/ -q
|
||||||
|
|
||||||
# Bundled extension examples own their behavior suites; the core shards exclude
|
|
||||||
# them so an optional example cannot fail an unrelated core PR. The extension CI
|
|
||||||
# workflow runs this target when an example, the extension contract or the
|
|
||||||
# bundled tests themselves change.
|
|
||||||
test-bundled-extensions:
|
|
||||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "bundled_extension and not live" tests/ -q
|
|
||||||
|
|
||||||
lint:
|
lint:
|
||||||
uv run ruff check .
|
uv run ruff check .
|
||||||
|
|||||||
@ -54,10 +54,6 @@ The single LangGraph agent (`lead_agent`) is the runtime entry point, created vi
|
|||||||
|
|
||||||
Middlewares execute in strict order, each handling a specific concern:
|
Middlewares execute in strict order, each handling a specific concern:
|
||||||
|
|
||||||
Starting a new graph invocation resets tool-progress tracking. Results and model
|
|
||||||
retries still finishing from an earlier invocation cannot block its tools or
|
|
||||||
restore obsolete progress hints, including same-run goal continuations.
|
|
||||||
|
|
||||||
| # | Middleware | Purpose |
|
| # | Middleware | Purpose |
|
||||||
|---|-----------|---------|
|
|---|-----------|---------|
|
||||||
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
|
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
|
||||||
@ -78,12 +74,6 @@ receive a transient hint on the next model request.
|
|||||||
|
|
||||||
### Sandbox System
|
### Sandbox System
|
||||||
|
|
||||||
With lazy initialization, a fresh AIO run prepares its authorized Agent skill
|
|
||||||
view without acquiring a container until a sandbox-backed tool needs one.
|
|
||||||
Existing sandbox state still requires eager policy rebinding; upload-based
|
|
||||||
providers still acquire and synchronize skills eagerly. Explicit sandbox
|
|
||||||
prewarming remains a separate opt-in behavior.
|
|
||||||
|
|
||||||
Per-thread isolated execution with virtual path translation:
|
Per-thread isolated execution with virtual path translation:
|
||||||
|
|
||||||
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
|
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||||
@ -115,10 +105,6 @@ the same item or undo a pause/cancellation observed after their initial read.
|
|||||||
|
|
||||||
### Memory System
|
### Memory System
|
||||||
|
|
||||||
If the local retrieval index is busy, read-only, or temporarily unavailable,
|
|
||||||
DeerMem falls back to substring search without deleting it. Automatic index
|
|
||||||
recreation is reserved for SQLite-confirmed corruption or a non-database file.
|
|
||||||
|
|
||||||
LLM-powered persistent context retention across conversations:
|
LLM-powered persistent context retention across conversations:
|
||||||
|
|
||||||
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
|
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
|
||||||
@ -170,8 +156,8 @@ to a boundary (SQLite) or causing the search to fail (PostgreSQL).
|
|||||||
| `POST /api/skills/install` | Install skill from `.skill` archive |
|
| `POST /api/skills/install` | Install skill from `.skill` archive |
|
||||||
| `GET /api/memory` | Retrieve memory data |
|
| `GET /api/memory` | Retrieve memory data |
|
||||||
| `POST /api/memory/reload` | Force memory reload |
|
| `POST /api/memory/reload` | Force memory reload |
|
||||||
| `GET /api/memory/config` | Memory configuration; credential values in `backend_config` are masked |
|
| `GET /api/memory/config` | Memory configuration |
|
||||||
| `GET /api/memory/status` | Combined config (masked the same way) + data |
|
| `GET /api/memory/status` | Combined config + data |
|
||||||
| `GET /api/threads/{id}/runs/{run_id}/events` | Debug/audit events for one run; filter `event_types=context:memory` for effective memory identity |
|
| `GET /api/threads/{id}/runs/{run_id}/events` | Debug/audit events for one run; filter `event_types=context:memory` for effective memory identity |
|
||||||
| `POST /api/threads/{id}/uploads` | Upload files (optional PDF/PPT/Excel/Word-to-Markdown conversion, rejects directory paths, auto-renames duplicate filenames in one request) |
|
| `POST /api/threads/{id}/uploads` | Upload files (optional PDF/PPT/Excel/Word-to-Markdown conversion, rejects directory paths, auto-renames duplicate filenames in one request) |
|
||||||
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
||||||
|
|||||||
@ -126,8 +126,8 @@ FastAPI 应用程序，为前端集成提供 REST 接口：
|
|||||||
| `POST /api/skills/install` | 从 `.skill` 归档文件安装技能 |
|
| `POST /api/skills/install` | 从 `.skill` 归档文件安装技能 |
|
||||||
| `GET /api/memory` | 获取记忆数据 |
|
| `GET /api/memory` | 获取记忆数据 |
|
||||||
| `POST /api/memory/reload` | 强制重新加载记忆 |
|
| `POST /api/memory/reload` | 强制重新加载记忆 |
|
||||||
| `GET /api/memory/config` | 获取记忆配置；`backend_config` 中的凭据值会被遮蔽 |
|
| `GET /api/memory/config` | 获取记忆配置 |
|
||||||
| `GET /api/memory/status` | 获取组合后的配置（同样遮蔽凭据）和数据状态 |
|
| `GET /api/memory/status` | 获取组合后的配置和数据状态 |
|
||||||
| `GET /api/threads/{id}/runs/{run_id}/events` | 获取某次运行的调试或审计事件；可使用 `event_types=context:memory` 筛选实际记忆标识 |
|
| `GET /api/threads/{id}/runs/{run_id}/events` | 获取某次运行的调试或审计事件；可使用 `event_types=context:memory` 筛选实际记忆标识 |
|
||||||
| `POST /api/threads/{id}/uploads` | 上传文件（可选地将 PDF、PPT、Excel、Word 转换为 Markdown；拒绝目录路径；自动重命名单次请求中的重复文件名） |
|
| `POST /api/threads/{id}/uploads` | 上传文件（可选地将 PDF、PPT、Excel、Word 转换为 Markdown；拒绝目录路径；自动重命名单次请求中的重复文件名） |
|
||||||
| `GET /api/threads/{id}/uploads/list` | 列出已上传的文件 |
|
| `GET /api/threads/{id}/uploads/list` | 列出已上传的文件 |
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@ -250,12 +250,7 @@ class Channel(ABC):
|
|||||||
log_prefix: str | None = None,
|
log_prefix: str | None = None,
|
||||||
operation_name: str = "send",
|
operation_name: str = "send",
|
||||||
) -> T:
|
) -> T:
|
||||||
"""Run an outbound send operation with the shared channel retry policy.
|
"""Run an outbound send operation with the shared channel retry policy."""
|
||||||
|
|
||||||
The wait between attempts is ``_outbound_retry_delay``. The default
|
|
||||||
is ``2**attempt``; a provider that is told how long to wait overrides
|
|
||||||
that without changing the attempt bound or the other channels.
|
|
||||||
"""
|
|
||||||
prefix = log_prefix or f"[{self.name}]"
|
prefix = log_prefix or f"[{self.name}]"
|
||||||
last_exc: Exception | None = None
|
last_exc: Exception | None = None
|
||||||
for attempt in range(max_retries):
|
for attempt in range(max_retries):
|
||||||
@ -264,9 +259,9 @@ class Channel(ABC):
|
|||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
last_exc = exc
|
last_exc = exc
|
||||||
if attempt < max_retries - 1:
|
if attempt < max_retries - 1:
|
||||||
delay = self._outbound_retry_delay(attempt, exc)
|
delay = 2**attempt
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"%s %s failed (attempt %d/%d), retrying in %ss: %s",
|
"%s %s failed (attempt %d/%d), retrying in %ds: %s",
|
||||||
prefix,
|
prefix,
|
||||||
operation_name,
|
operation_name,
|
||||||
attempt + 1,
|
attempt + 1,
|
||||||
@ -281,16 +276,6 @@ class Channel(ABC):
|
|||||||
raise RuntimeError(f"{self.name} {operation_name} failed without an exception from any attempt")
|
raise RuntimeError(f"{self.name} {operation_name} failed without an exception from any attempt")
|
||||||
raise last_exc
|
raise last_exc
|
||||||
|
|
||||||
def _outbound_retry_delay(self, attempt: int, exc: Exception) -> float:
|
|
||||||
"""Seconds to wait before the next attempt.
|
|
||||||
|
|
||||||
``exc`` is the failure just caught. The shared policy ignores it and
|
|
||||||
sleeps ``2**attempt`` (1s, then 2s, then 4s, ...). A provider that
|
|
||||||
receives a server wait hint overrides this and may return a longer
|
|
||||||
delay; it must not shorten the local backoff.
|
|
||||||
"""
|
|
||||||
return 2**attempt
|
|
||||||
|
|
||||||
def _log_future_error(self, fut: Any, name: str, msg_id: Any) -> None:
|
def _log_future_error(self, fut: Any, name: str, msg_id: Any) -> None:
|
||||||
"""Callback for concurrent futures scheduled from channel worker threads."""
|
"""Callback for concurrent futures scheduled from channel worker threads."""
|
||||||
try:
|
try:
|
||||||
|
|||||||
@ -10,7 +10,6 @@ import re
|
|||||||
import stat
|
import stat
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
import uuid
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
from urllib.parse import quote_plus
|
from urllib.parse import quote_plus
|
||||||
@ -203,6 +202,7 @@ class DingTalkChannel(Channel):
|
|||||||
self._stream_lock = threading.Lock()
|
self._stream_lock = threading.Lock()
|
||||||
self._incoming_messages: dict[str, Any] = {}
|
self._incoming_messages: dict[str, Any] = {}
|
||||||
self._incoming_messages_lock = threading.Lock()
|
self._incoming_messages_lock = threading.Lock()
|
||||||
|
self._card_repliers: dict[str, Any] = {}
|
||||||
# Serialize inbound-file writes into the uploads directory to avoid
|
# Serialize inbound-file writes into the uploads directory to avoid
|
||||||
# racing writers clobbering one another (mirrors FeishuChannel).
|
# racing writers clobbering one another (mirrors FeishuChannel).
|
||||||
self._file_write_lock = threading.Lock()
|
self._file_write_lock = threading.Lock()
|
||||||
@ -263,6 +263,7 @@ class DingTalkChannel(Channel):
|
|||||||
self._dingtalk_client = None
|
self._dingtalk_client = None
|
||||||
with self._incoming_messages_lock:
|
with self._incoming_messages_lock:
|
||||||
self._incoming_messages.clear()
|
self._incoming_messages.clear()
|
||||||
|
self._card_repliers.clear()
|
||||||
self._card_track_ids.clear()
|
self._card_track_ids.clear()
|
||||||
if self._thread:
|
if self._thread:
|
||||||
# The thread exits once its loop has stopped; keep the join off the
|
# The thread exits once its loop has stopped; keep the join off the
|
||||||
@ -350,10 +351,12 @@ class DingTalkChannel(Channel):
|
|||||||
logger.warning("[DingTalk] card stream failed, falling back to sampleMarkdown")
|
logger.warning("[DingTalk] card stream failed, falling back to sampleMarkdown")
|
||||||
if msg.is_final:
|
if msg.is_final:
|
||||||
self._card_track_ids.pop(source_key, None)
|
self._card_track_ids.pop(source_key, None)
|
||||||
|
self._card_repliers.pop(out_track_id, None)
|
||||||
await self._send_markdown_fallback(robot_code, conversation_type, sender_staff_id, conversation_id, msg.text)
|
await self._send_markdown_fallback(robot_code, conversation_type, sender_staff_id, conversation_id, msg.text)
|
||||||
return
|
return
|
||||||
if msg.is_final:
|
if msg.is_final:
|
||||||
self._card_track_ids.pop(source_key, None)
|
self._card_track_ids.pop(source_key, None)
|
||||||
|
self._card_repliers.pop(out_track_id, None)
|
||||||
return
|
return
|
||||||
|
|
||||||
async def send_markdown() -> None:
|
async def send_markdown() -> None:
|
||||||
@ -438,24 +441,6 @@ class DingTalkChannel(Channel):
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
data = response.json()
|
|
||||||
process_key = data.get("processQueryKey") if isinstance(data, dict) else None
|
|
||||||
if not isinstance(process_key, str) or not process_key.strip():
|
|
||||||
logger.warning(
|
|
||||||
"[DingTalk] file send response missing processQueryKey for %s; treating as not delivered",
|
|
||||||
attachment.filename,
|
|
||||||
)
|
|
||||||
return False
|
|
||||||
if conversation_type != _CONVERSATION_TYPE_GROUP:
|
|
||||||
target = sender_staff_id
|
|
||||||
for field in ("invalidStaffIdList", "flowControlledStaffIdList"):
|
|
||||||
failed = data.get(field, [])
|
|
||||||
if not isinstance(failed, list):
|
|
||||||
logger.warning("[DingTalk] unexpected %s in file send response for %s", field, attachment.filename)
|
|
||||||
return False
|
|
||||||
if target in failed:
|
|
||||||
logger.warning("[DingTalk] file not delivered for %s (%s)", attachment.filename, field)
|
|
||||||
return False
|
|
||||||
|
|
||||||
logger.info("[DingTalk] file sent: %s", attachment.filename)
|
logger.info("[DingTalk] file sent: %s", attachment.filename)
|
||||||
return True
|
return True
|
||||||
@ -1259,42 +1244,30 @@ class DingTalkChannel(Channel):
|
|||||||
logger.warning("[DingTalk] SDK client or chatbot_message unavailable, skipping AI card")
|
logger.warning("[DingTalk] SDK client or chatbot_message unavailable, skipping AI card")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
out_track_id = uuid.uuid4().hex
|
|
||||||
try:
|
try:
|
||||||
await self._card_request("POST", "/v1.0/card/instances/createAndDeliver", self._card_create_body(out_track_id, chatbot_message, initial_text))
|
from dingtalk_stream.card_replier import AICardReplier
|
||||||
|
except ImportError:
|
||||||
|
logger.warning("[DingTalk] dingtalk-stream card_replier not available")
|
||||||
|
return None
|
||||||
|
|
||||||
|
try:
|
||||||
|
token_client = _CardTokenClient(self._dingtalk_client.credential)
|
||||||
|
token_client.access_token = await self._get_access_token()
|
||||||
|
replier = AICardReplier(token_client, chatbot_message)
|
||||||
|
card_instance_id = await replier.async_create_and_deliver_card(
|
||||||
|
card_template_id=self._card_template_id,
|
||||||
|
card_data={"content": initial_text},
|
||||||
|
)
|
||||||
|
if not card_instance_id:
|
||||||
|
return None
|
||||||
|
|
||||||
|
self._card_repliers[card_instance_id] = replier
|
||||||
|
logger.info("[DingTalk] AI card created: outTrackId=%s", card_instance_id)
|
||||||
|
return card_instance_id
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("[DingTalk] failed to create AI card")
|
logger.exception("[DingTalk] failed to create AI card")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
logger.info("[DingTalk] AI card created: outTrackId=%s", out_track_id)
|
|
||||||
return out_track_id
|
|
||||||
|
|
||||||
def _card_create_body(self, out_track_id: str, chatbot_message: Any, initial_text: str) -> dict[str, Any]:
|
|
||||||
"""The ``createAndDeliver`` body dingtalk-stream's ``AICardReplier`` sends."""
|
|
||||||
body: dict[str, Any] = {
|
|
||||||
"cardTemplateId": self._card_template_id,
|
|
||||||
"outTrackId": out_track_id,
|
|
||||||
"cardData": {"cardParamMap": {"content": initial_text}},
|
|
||||||
"callbackType": "STREAM",
|
|
||||||
"imGroupOpenSpaceModel": {"supportForward": True},
|
|
||||||
"imRobotOpenSpaceModel": {"supportForward": True},
|
|
||||||
}
|
|
||||||
hosting_context = getattr(chatbot_message, "hosting_context", None)
|
|
||||||
conversation_type = str(chatbot_message.conversation_type)
|
|
||||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
|
||||||
body["openSpaceId"] = f"dtv1.card//IM_GROUP.{chatbot_message.conversation_id}"
|
|
||||||
deliver_model: dict[str, Any] = {"robotCode": self._client_id}
|
|
||||||
body["imGroupOpenDeliverModel"] = deliver_model
|
|
||||||
elif conversation_type == _CONVERSATION_TYPE_P2P:
|
|
||||||
body["openSpaceId"] = f"dtv1.card//IM_ROBOT.{chatbot_message.sender_staff_id}"
|
|
||||||
deliver_model = {"spaceType": "IM_ROBOT"}
|
|
||||||
body["imRobotOpenDeliverModel"] = deliver_model
|
|
||||||
else:
|
|
||||||
return body
|
|
||||||
if hosting_context is not None:
|
|
||||||
deliver_model["extension"] = {"hostingRepliedContext": json.dumps({"userId": hosting_context.user_id})}
|
|
||||||
return body
|
|
||||||
|
|
||||||
async def _stream_update_card(
|
async def _stream_update_card(
|
||||||
self,
|
self,
|
||||||
out_track_id: str,
|
out_track_id: str,
|
||||||
@ -1303,35 +1276,20 @@ class DingTalkChannel(Channel):
|
|||||||
is_finalize: bool = False,
|
is_finalize: bool = False,
|
||||||
is_error: bool = False,
|
is_error: bool = False,
|
||||||
) -> None:
|
) -> None:
|
||||||
await self._card_request(
|
replier = self._card_repliers.get(out_track_id)
|
||||||
"PUT",
|
if not replier:
|
||||||
"/v1.0/card/streaming",
|
raise RuntimeError(f"No AICardReplier found for track ID {out_track_id}")
|
||||||
{
|
|
||||||
"outTrackId": out_track_id,
|
replier.dingtalk_client.access_token = await self._get_access_token()
|
||||||
"guid": str(uuid.uuid4()),
|
await replier.async_streaming(
|
||||||
"key": "content",
|
card_instance_id=out_track_id,
|
||||||
"content": content,
|
content_key="content",
|
||||||
"isFull": True,
|
content_value=content,
|
||||||
"isFinalize": is_finalize,
|
append=False,
|
||||||
"isError": is_error,
|
finished=is_finalize,
|
||||||
},
|
failed=is_error,
|
||||||
)
|
)
|
||||||
|
|
||||||
async def _card_request(self, method: str, path: str, body: dict[str, Any]) -> None:
|
|
||||||
"""Send an AI card request with the channel token; raise if it fails.
|
|
||||||
|
|
||||||
dingtalk-stream's ``AICardReplier`` logs a failed card request and
|
|
||||||
returns as if it succeeded; its create call even returns the card ID.
|
|
||||||
Neither the running reply's text fallback nor the sampleMarkdown
|
|
||||||
fallback in ``send`` could then run, so the reply was lost.
|
|
||||||
"""
|
|
||||||
token = await self._get_access_token()
|
|
||||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
|
||||||
response = await client.request(method, f"{DINGTALK_API_BASE}{path}", headers=self._api_headers(token), json=body)
|
|
||||||
if response.is_error:
|
|
||||||
logger.warning("[DingTalk] card request %s %s returned %s: %s", method, path, response.status_code, response.text[:500])
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
# -- media upload --------------------------------------------------------
|
# -- media upload --------------------------------------------------------
|
||||||
|
|
||||||
async def _upload_media(self, file_path: str | Path, media_type: str) -> str | None:
|
async def _upload_media(self, file_path: str | Path, media_type: str) -> str | None:
|
||||||
@ -1360,6 +1318,26 @@ class DingTalkChannel(Channel):
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class _CardTokenClient:
|
||||||
|
"""The client ``AICardReplier`` sees: it serves the channel's access token.
|
||||||
|
|
||||||
|
The replier's async methods call ``dingtalk_client.get_access_token()``
|
||||||
|
inline, and dingtalk-stream's version refreshes through ``requests.post``
|
||||||
|
with no timeout, so on the Gateway loop every refresh (about every two
|
||||||
|
hours) blocked the loop for the whole request, indefinitely if DingTalk
|
||||||
|
stalled. The channel sets ``access_token`` from its own bounded
|
||||||
|
``_get_access_token()`` before every replier call, so the replier never
|
||||||
|
fetches a token itself.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, credential: Any) -> None:
|
||||||
|
self.credential = credential
|
||||||
|
self.access_token = ""
|
||||||
|
|
||||||
|
def get_access_token(self) -> str:
|
||||||
|
return self.access_token
|
||||||
|
|
||||||
|
|
||||||
class _DingTalkMessageHandler:
|
class _DingTalkMessageHandler:
|
||||||
"""Callback handler registered with dingtalk-stream."""
|
"""Callback handler registered with dingtalk-stream."""
|
||||||
|
|
||||||
|
|||||||
@ -28,7 +28,6 @@ from app.channels.message_bus import (
|
|||||||
OutboundMessage,
|
OutboundMessage,
|
||||||
ResolvedAttachment,
|
ResolvedAttachment,
|
||||||
)
|
)
|
||||||
from deerflow.utils.atomic_write import publish_bytes_atomically
|
|
||||||
from deerflow.utils.file_io import await_drained
|
from deerflow.utils.file_io import await_drained
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@ -276,10 +275,7 @@ class DiscordChannel(Channel):
|
|||||||
try:
|
try:
|
||||||
snapshot = dict(self._active_threads)
|
snapshot = dict(self._active_threads)
|
||||||
self._thread_store_path.parent.mkdir(parents=True, exist_ok=True)
|
self._thread_store_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
# Publish through the shared helper: it fsyncs the staged bytes
|
self._thread_store_path.write_text(json.dumps(snapshot, indent=2), encoding="utf-8", newline="")
|
||||||
# before the rename (so a machine crash cannot leave an empty
|
|
||||||
# mapping behind) and re-applies an existing file's mode.
|
|
||||||
publish_bytes_atomically(self._thread_store_path, json.dumps(snapshot, indent=2).encode("utf-8"))
|
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("[Discord] failed to persist thread mappings")
|
logger.exception("[Discord] failed to persist thread mappings")
|
||||||
|
|
||||||
|
|||||||
@ -55,9 +55,6 @@ PENDING_CLARIFICATION_TTL_SECONDS = 30 * 60
|
|||||||
FEISHU_STOP_TIMEOUT_SECONDS = 5
|
FEISHU_STOP_TIMEOUT_SECONDS = 5
|
||||||
FEISHU_WS_CLOSE_TIMEOUT_SECONDS = 2
|
FEISHU_WS_CLOSE_TIMEOUT_SECONDS = 2
|
||||||
_PENDING_CARD_METADATA_KEY = "_feishu_pending_card_id"
|
_PENDING_CARD_METADATA_KEY = "_feishu_pending_card_id"
|
||||||
# Number of attachments from a post's top-level ``files`` array.
|
|
||||||
_POST_FILES_COUNT_METADATA_KEY = "_feishu_post_files_count"
|
|
||||||
_DEFER_POST_FILE_PLACEHOLDERS_METADATA_KEY = "_feishu_defer_post_file_placeholders"
|
|
||||||
# Channel-store deadlines. ``_on_message`` runs on lark-oapi's private event loop,
|
# Channel-store deadlines. ``_on_message`` runs on lark-oapi's private event loop,
|
||||||
# which also carries the SDK's websocket reads, pongs and its 120s ping loop:
|
# which also carries the SDK's websocket reads, pongs and its 120s ping loop:
|
||||||
# every second the bridge blocks that loop is a second the connection cannot
|
# every second the bridge blocks that loop is a second the connection cannot
|
||||||
@ -110,7 +107,6 @@ class _ParsedMessage:
|
|||||||
chat_type: str | None
|
chat_type: str | None
|
||||||
text: str
|
text: str
|
||||||
files: list[dict[str, Any]]
|
files: list[dict[str, Any]]
|
||||||
post_file_count: int = 0
|
|
||||||
msg_type: InboundMessageType = InboundMessageType.CHAT
|
msg_type: InboundMessageType = InboundMessageType.CHAT
|
||||||
|
|
||||||
|
|
||||||
@ -148,110 +144,12 @@ class _RoutingAttempt:
|
|||||||
FEISHU_INBOUND_BATCH_WINDOW_SECONDS = 0.75
|
FEISHU_INBOUND_BATCH_WINDOW_SECONDS = 0.75
|
||||||
FEISHU_MAX_INBOUND_FILE_BYTES = 20_000_000
|
FEISHU_MAX_INBOUND_FILE_BYTES = 20_000_000
|
||||||
SOURCE_PREVIEW_METADATA_KEY = "feishu_source_preview"
|
SOURCE_PREVIEW_METADATA_KEY = "feishu_source_preview"
|
||||||
# Feishu rejects a card request body over 30 KB (230025), and style tags make
|
|
||||||
# the stored message larger than the body sent. Card text is budgeted by the
|
|
||||||
# bytes it adds to that body, leaving room for the other fields and that growth.
|
|
||||||
FEISHU_CARD_MAX_BYTES = 24 * 1024
|
|
||||||
# A long final answer continues in further cards, up to this many per reply.
|
|
||||||
FEISHU_MAX_CARDS_PER_REPLY = 10
|
|
||||||
_CARD_TRUNCATION_MARKER = "\n\n... (truncated)"
|
|
||||||
_CARD_CONTINUES_MARKER = "\n\n…"
|
|
||||||
_LONE_SURROGATE = re.compile("[\ud800-\udfff]")
|
|
||||||
# Characters JSON escapes, measured after the second escape: ``"`` is ``\\\"``.
|
|
||||||
_ESCAPED_CARD_CHAR_BYTES = {'"': 4, "\\": 4, "\b": 3, "\f": 3, "\n": 3, "\r": 3, "\t": 3}
|
|
||||||
|
|
||||||
|
|
||||||
def _card_char_bytes(char: str) -> int:
|
|
||||||
escaped = _ESCAPED_CARD_CHAR_BYTES.get(char)
|
|
||||||
if escaped is not None:
|
|
||||||
return escaped
|
|
||||||
code = ord(char)
|
|
||||||
if code < 0x20:
|
|
||||||
return 7 # \u00XX, whose backslash is escaped again
|
|
||||||
if code < 0x80:
|
|
||||||
return 1
|
|
||||||
if code < 0x800:
|
|
||||||
return 2
|
|
||||||
if code < 0x10000:
|
|
||||||
return 3 # a lone surrogate is sent as U+FFFD
|
|
||||||
return 4
|
|
||||||
|
|
||||||
|
|
||||||
def _card_text_bytes(text: str) -> int:
|
|
||||||
"""Bytes *text* adds to a card request body.
|
|
||||||
|
|
||||||
The text is escaped twice: once inside the card JSON, and again as the
|
|
||||||
request body's ``content`` string. JSON escapes each character on its own,
|
|
||||||
so the cost of a text is the sum of its characters' costs.
|
|
||||||
"""
|
|
||||||
return sum(map(_card_char_bytes, text))
|
|
||||||
|
|
||||||
|
|
||||||
def _card_text_budget() -> int:
|
|
||||||
return FEISHU_CARD_MAX_BYTES - _card_text_bytes(FeishuChannel._build_card_content(""))
|
|
||||||
|
|
||||||
|
|
||||||
def _fitting_prefix_end(text: str, start: int, budget: int) -> tuple[int, int]:
|
|
||||||
"""Return the end of the longest prefix of ``text[start:]`` within *budget*, and its last newline."""
|
|
||||||
used = 0
|
|
||||||
end = start
|
|
||||||
last_newline = -1
|
|
||||||
while end < len(text):
|
|
||||||
cost = _card_char_bytes(text[end])
|
|
||||||
if used + cost > budget:
|
|
||||||
break
|
|
||||||
used += cost
|
|
||||||
if text[end] == "\n":
|
|
||||||
last_newline = end
|
|
||||||
end += 1
|
|
||||||
return end, last_newline
|
|
||||||
|
|
||||||
|
|
||||||
def _clip_card_text(text: str, marker: str = _CARD_TRUNCATION_MARKER) -> str:
|
|
||||||
"""Clip *text* to one card, ending it with *marker* when anything was cut."""
|
|
||||||
budget = _card_text_budget()
|
|
||||||
if _card_text_bytes(text) <= budget:
|
|
||||||
return text
|
|
||||||
end, _ = _fitting_prefix_end(text, 0, budget - _card_text_bytes(marker))
|
|
||||||
return text[:end] + marker
|
|
||||||
|
|
||||||
|
|
||||||
def _split_card_text(text: str) -> list[str]:
|
|
||||||
"""Split *text* into cards, preferring line boundaries so markdown survives.
|
|
||||||
|
|
||||||
Each newline stays on the tail of its card, so the cards join back to
|
|
||||||
the original text. Past ``FEISHU_MAX_CARDS_PER_REPLY`` the last card is
|
|
||||||
clipped instead.
|
|
||||||
"""
|
|
||||||
budget = _card_text_budget()
|
|
||||||
if _card_text_bytes(text) <= budget:
|
|
||||||
return [text]
|
|
||||||
cards: list[str] = []
|
|
||||||
start = 0
|
|
||||||
while start < len(text):
|
|
||||||
if len(cards) == FEISHU_MAX_CARDS_PER_REPLY - 1:
|
|
||||||
logger.warning("[Feishu] reply of %d characters exceeds %d cards, clipping the last one", len(text), FEISHU_MAX_CARDS_PER_REPLY)
|
|
||||||
cards.append(_clip_card_text(text[start:]))
|
|
||||||
break
|
|
||||||
end, last_newline = _fitting_prefix_end(text, start, budget)
|
|
||||||
if end < len(text) and last_newline >= start:
|
|
||||||
end = last_newline + 1
|
|
||||||
end = max(end, start + 1) # always advance, even past an oversized character
|
|
||||||
cards.append(text[start:end])
|
|
||||||
start = end
|
|
||||||
return cards
|
|
||||||
|
|
||||||
|
|
||||||
def _is_feishu_command(text: str) -> bool:
|
def _is_feishu_command(text: str) -> bool:
|
||||||
return is_known_channel_command(text)
|
return is_known_channel_command(text)
|
||||||
|
|
||||||
|
|
||||||
def _append_post_file_placeholders(text: str, count: int) -> str:
|
|
||||||
paragraphs = [text] if text else []
|
|
||||||
paragraphs.extend(["[file]"] * count)
|
|
||||||
return "\n\n".join(paragraphs)
|
|
||||||
|
|
||||||
|
|
||||||
class FeishuChannel(Channel):
|
class FeishuChannel(Channel):
|
||||||
"""Feishu/Lark IM channel using the ``lark-oapi`` WebSocket client.
|
"""Feishu/Lark IM channel using the ``lark-oapi`` WebSocket client.
|
||||||
|
|
||||||
@ -651,10 +549,8 @@ class FeishuChannel(Channel):
|
|||||||
len(msg.text),
|
len(msg.text),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Cards already delivered by an earlier attempt are not sent again.
|
|
||||||
delivered: list[str | None] = []
|
|
||||||
await self._send_with_retry(
|
await self._send_with_retry(
|
||||||
lambda: self._send_card_message(msg, delivered=delivered),
|
lambda: self._send_card_message(msg),
|
||||||
max_retries=_max_retries,
|
max_retries=_max_retries,
|
||||||
log_prefix="[Feishu]",
|
log_prefix="[Feishu]",
|
||||||
)
|
)
|
||||||
@ -772,7 +668,9 @@ class FeishuChannel(Channel):
|
|||||||
async def _upload_image(self, path) -> str:
|
async def _upload_image(self, path) -> str:
|
||||||
"""Upload an image to Feishu and return the image_key."""
|
"""Upload an image to Feishu and return the image_key."""
|
||||||
response = await asyncio.to_thread(self._upload_image_sync, path)
|
response = await asyncio.to_thread(self._upload_image_sync, path)
|
||||||
return self._uploaded_key(response, "image")
|
if not response.success():
|
||||||
|
raise RuntimeError(f"Feishu image upload failed: code={response.code}, msg={response.msg}")
|
||||||
|
return response.data.image_key
|
||||||
|
|
||||||
def _upload_file_sync(self, path, filename: str, file_type: str):
|
def _upload_file_sync(self, path, filename: str, file_type: str):
|
||||||
with open(str(path), "rb") as f:
|
with open(str(path), "rb") as f:
|
||||||
@ -794,17 +692,9 @@ class FeishuChannel(Channel):
|
|||||||
file_type = "stream"
|
file_type = "stream"
|
||||||
|
|
||||||
response = await asyncio.to_thread(self._upload_file_sync, path, filename, file_type)
|
response = await asyncio.to_thread(self._upload_file_sync, path, filename, file_type)
|
||||||
return self._uploaded_key(response, "file")
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _uploaded_key(response, kind: str) -> str:
|
|
||||||
"""Return the ``image_key``/``file_key`` of an upload response or raise with its log_id."""
|
|
||||||
if not response.success():
|
if not response.success():
|
||||||
raise RuntimeError(f"Feishu {kind} upload failed: code={response.code}, msg={response.msg}, log_id={response.get_log_id()}")
|
raise RuntimeError(f"Feishu file upload failed: code={response.code}, msg={response.msg}")
|
||||||
key = getattr(getattr(response, "data", None), f"{kind}_key", None)
|
return response.data.file_key
|
||||||
if not isinstance(key, str) or not key:
|
|
||||||
raise RuntimeError(f"Feishu {kind} upload returned no {kind}_key: log_id={response.get_log_id()}")
|
|
||||||
return key
|
|
||||||
|
|
||||||
async def receive_file(self, msg: InboundMessage, thread_id: str, *, user_id: str | None = None) -> InboundMessage:
|
async def receive_file(self, msg: InboundMessage, thread_id: str, *, user_id: str | None = None) -> InboundMessage:
|
||||||
"""Download a Feishu file into the thread uploads directory.
|
"""Download a Feishu file into the thread uploads directory.
|
||||||
@ -819,10 +709,6 @@ class FeishuChannel(Channel):
|
|||||||
logger.warning("[Feishu] received message with no files: %s", msg)
|
logger.warning("[Feishu] received message with no files: %s", msg)
|
||||||
return msg
|
return msg
|
||||||
text = msg.text
|
text = msg.text
|
||||||
post_file_count = msg.metadata.pop(_POST_FILES_COUNT_METADATA_KEY, 0)
|
|
||||||
defer_post_file_placeholders = msg.metadata.pop(_DEFER_POST_FILE_PLACEHOLDERS_METADATA_KEY, False)
|
|
||||||
if not isinstance(post_file_count, int) or post_file_count < 0:
|
|
||||||
post_file_count = 0
|
|
||||||
search_from = 0
|
search_from = 0
|
||||||
|
|
||||||
def replace_next_placeholder(placeholder: str, replacement: str) -> None:
|
def replace_next_placeholder(placeholder: str, replacement: str) -> None:
|
||||||
@ -833,40 +719,14 @@ class FeishuChannel(Channel):
|
|||||||
text = f"{text[:idx]}{replacement}{text[idx + len(placeholder) :]}"
|
text = f"{text[:idx]}{replacement}{text[idx + len(placeholder) :]}"
|
||||||
search_from = idx + len(replacement)
|
search_from = idx + len(replacement)
|
||||||
|
|
||||||
downloaded: list[tuple[str, str]] = []
|
|
||||||
for file in files:
|
for file in files:
|
||||||
if file.get("image_key"):
|
if file.get("image_key"):
|
||||||
virtual_path = await self._receive_single_file(msg.thread_ts, file["image_key"], "image", thread_id, user_id=user_id)
|
virtual_path = await self._receive_single_file(msg.thread_ts, file["image_key"], "image", thread_id, user_id=user_id)
|
||||||
downloaded.append(("[image]", virtual_path))
|
replace_next_placeholder("[image]", virtual_path)
|
||||||
elif file.get("file_key"):
|
elif file.get("file_key"):
|
||||||
virtual_path = await self._receive_single_file(msg.thread_ts, file["file_key"], "file", thread_id, user_id=user_id)
|
virtual_path = await self._receive_single_file(msg.thread_ts, file["file_key"], "file", thread_id, user_id=user_id)
|
||||||
downloaded.append(("[file]", virtual_path))
|
replace_next_placeholder("[file]", virtual_path)
|
||||||
|
|
||||||
# Top-level post attachments append their ``[file]`` placeholders
|
|
||||||
# after the parsed body, so their slots are the final
|
|
||||||
# ``post_file_count`` ones. Fill them from the end first: a literal
|
|
||||||
# ``[file]`` typed by the user must not consume a top-level
|
|
||||||
# attachment's path. Inline rich-text attachments keep the historical
|
|
||||||
# left-to-right matching below; that path is deliberately unchanged.
|
|
||||||
post_file_count = min(post_file_count, len(downloaded))
|
|
||||||
if defer_post_file_placeholders is True:
|
|
||||||
# A command such as /goal has now entered the chat path. Add the
|
|
||||||
# generated slots to the chat text after its arguments were parsed.
|
|
||||||
text = _append_post_file_placeholders(text, post_file_count)
|
|
||||||
suffix_end = len(text)
|
|
||||||
for placeholder, virtual_path in reversed(downloaded[len(downloaded) - post_file_count :] if post_file_count else []):
|
|
||||||
idx = text.rfind(placeholder, 0, suffix_end)
|
|
||||||
if idx < 0:
|
|
||||||
continue
|
|
||||||
text = f"{text[:idx]}{virtual_path}{text[idx + len(placeholder) :]}"
|
|
||||||
suffix_end = idx
|
|
||||||
|
|
||||||
for placeholder, virtual_path in downloaded[: len(downloaded) - post_file_count]:
|
|
||||||
replace_next_placeholder(placeholder, virtual_path)
|
|
||||||
msg.text = text
|
msg.text = text
|
||||||
# Clear the descriptors so the generic URL-based _ingest_inbound_files
|
|
||||||
# pass does not try to re-fetch files that were downloaded by key.
|
|
||||||
msg.files = []
|
|
||||||
return msg
|
return msg
|
||||||
|
|
||||||
async def _receive_single_file(
|
async def _receive_single_file(
|
||||||
@ -998,11 +858,9 @@ class FeishuChannel(Channel):
|
|||||||
"""
|
"""
|
||||||
card = {
|
card = {
|
||||||
"config": {"wide_screen_mode": True, "update_multi": True},
|
"config": {"wide_screen_mode": True, "update_multi": True},
|
||||||
# Unpaired surrogates cannot be encoded as UTF-8.
|
"elements": [{"tag": "markdown", "content": text}],
|
||||||
"elements": [{"tag": "markdown", "content": _LONE_SURROGATE.sub("\ufffd", text)}],
|
|
||||||
}
|
}
|
||||||
# ASCII escapes would make each CJK character 6 bytes instead of 3.
|
return json.dumps(card)
|
||||||
return json.dumps(card, ensure_ascii=False)
|
|
||||||
|
|
||||||
# -- reaction helpers --------------------------------------------------
|
# -- reaction helpers --------------------------------------------------
|
||||||
|
|
||||||
@ -1082,7 +940,7 @@ class FeishuChannel(Channel):
|
|||||||
metadata: dict[str, Any] | None = None,
|
metadata: dict[str, Any] | None = None,
|
||||||
) -> str | None:
|
) -> str | None:
|
||||||
"""Create the running card and cache its message ID when available."""
|
"""Create the running card and cache its message ID when available."""
|
||||||
running_card_id = await self._reply_card(source_message_id, _clip_card_text(self._compose_card_text(text, metadata), _CARD_CONTINUES_MARKER))
|
running_card_id = await self._reply_card(source_message_id, self._compose_card_text(text, metadata))
|
||||||
if running_card_id:
|
if running_card_id:
|
||||||
self._running_card_ids[source_message_id] = running_card_id
|
self._running_card_ids[source_message_id] = running_card_id
|
||||||
logger.info("[Feishu] running card created: source=%s card=%s", source_message_id, running_card_id)
|
logger.info("[Feishu] running card created: source=%s card=%s", source_message_id, running_card_id)
|
||||||
@ -1196,14 +1054,8 @@ class FeishuChannel(Channel):
|
|||||||
self._remember_pending_clarification(msg, card_message_id, topic_id=topic_id)
|
self._remember_pending_clarification(msg, card_message_id, topic_id=topic_id)
|
||||||
await self._remember_thread_mapping(msg, msg.thread_ts, card_message_id)
|
await self._remember_thread_mapping(msg, msg.thread_ts, card_message_id)
|
||||||
|
|
||||||
async def _send_card_message(self, msg: OutboundMessage, *, delivered: list[str | None] | None = None) -> None:
|
async def _send_card_message(self, msg: OutboundMessage) -> None:
|
||||||
"""Send or update the Feishu card tied to the current request.
|
"""Send or update the Feishu card tied to the current request."""
|
||||||
|
|
||||||
A final answer over one card's budget continues in further cards replied
|
|
||||||
to the same source message; a streamed update shows only what fits.
|
|
||||||
*delivered* collects the IDs of the final answer's cards across retries.
|
|
||||||
"""
|
|
||||||
delivered = [] if delivered is None else delivered
|
|
||||||
source_message_id = msg.thread_ts
|
source_message_id = msg.thread_ts
|
||||||
# Option clarifications render as an interactive button card on the
|
# Option clarifications render as an interactive button card on the
|
||||||
# final outbound; everything else keeps the plain markdown card.
|
# final outbound; everything else keeps the plain markdown card.
|
||||||
@ -1211,45 +1063,11 @@ class FeishuChannel(Channel):
|
|||||||
card_topic_id = await self._bind_card_thread(msg) if card_request else None
|
card_topic_id = await self._bind_card_thread(msg) if card_request else None
|
||||||
if card_topic_id is None:
|
if card_topic_id is None:
|
||||||
card_request = None
|
card_request = None
|
||||||
|
card_text = build_button_card(card_request) if card_request else self._compose_card_text(msg.text, msg.metadata)
|
||||||
# Button cards arrive as serialized card JSON and bypass the markdown
|
# Button cards arrive as serialized card JSON and bypass the markdown
|
||||||
# wrapper; the plain path keeps its exact historical call signature.
|
# wrapper; the plain path keeps its exact historical call signature.
|
||||||
raw_card_kw: dict[str, bool] = {"raw_card_content": True} if card_request else {}
|
raw_card_kw: dict[str, bool] = {"raw_card_content": True} if card_request else {}
|
||||||
if source_message_id:
|
if source_message_id:
|
||||||
if card_request:
|
|
||||||
cards = [build_button_card(card_request)]
|
|
||||||
elif msg.is_final:
|
|
||||||
cards = _split_card_text(self._compose_card_text(msg.text, msg.metadata))
|
|
||||||
else:
|
|
||||||
cards = [_clip_card_text(self._compose_card_text(msg.text, msg.metadata), _CARD_CONTINUES_MARKER)]
|
|
||||||
if not delivered:
|
|
||||||
await self._send_first_card(msg, cards[0], card_request=card_request, card_topic_id=card_topic_id, delivered=delivered, raw_card_kw=raw_card_kw)
|
|
||||||
if msg.is_final:
|
|
||||||
for card_text in cards[len(delivered) :]:
|
|
||||||
delivered.append(await self._reply_card(source_message_id, card_text))
|
|
||||||
if len(delivered) > 1:
|
|
||||||
# A reply to any of the cards continues this thread.
|
|
||||||
await self._remember_thread_mapping(msg, *delivered[1:])
|
|
||||||
self._running_card_ids.pop(source_message_id, None)
|
|
||||||
self._remembered_thread_aliases.pop(source_message_id, None)
|
|
||||||
await self._add_reaction(source_message_id, "DONE")
|
|
||||||
return
|
|
||||||
|
|
||||||
for card_text in _split_card_text(msg.text)[len(delivered) :]:
|
|
||||||
await self._create_card(msg.chat_id, card_text)
|
|
||||||
delivered.append(None)
|
|
||||||
|
|
||||||
async def _send_first_card(
|
|
||||||
self,
|
|
||||||
msg: OutboundMessage,
|
|
||||||
card_text: str,
|
|
||||||
*,
|
|
||||||
card_request: ClarificationCardRequest | None,
|
|
||||||
card_topic_id: str | None,
|
|
||||||
delivered: list[str | None],
|
|
||||||
raw_card_kw: dict[str, bool],
|
|
||||||
) -> None:
|
|
||||||
"""Put *card_text* on the running card, or reply with it; record it in *delivered* when final."""
|
|
||||||
source_message_id = msg.thread_ts
|
|
||||||
running_card_id = self._running_card_ids.get(source_message_id)
|
running_card_id = self._running_card_ids.get(source_message_id)
|
||||||
awaited_running_card_task = False
|
awaited_running_card_task = False
|
||||||
|
|
||||||
@ -1270,11 +1088,8 @@ class FeishuChannel(Channel):
|
|||||||
running_card_id,
|
running_card_id,
|
||||||
)
|
)
|
||||||
fallback_card_id = await self._reply_card(source_message_id, card_text, **raw_card_kw)
|
fallback_card_id = await self._reply_card(source_message_id, card_text, **raw_card_kw)
|
||||||
delivered.append(fallback_card_id)
|
|
||||||
await self._remember_card_delivery(msg, card_request, fallback_card_id, topic_id=card_topic_id)
|
await self._remember_card_delivery(msg, card_request, fallback_card_id, topic_id=card_topic_id)
|
||||||
else:
|
else:
|
||||||
if msg.is_final:
|
|
||||||
delivered.append(running_card_id)
|
|
||||||
await self._remember_card_delivery(msg, card_request, running_card_id, topic_id=card_topic_id)
|
await self._remember_card_delivery(msg, card_request, running_card_id, topic_id=card_topic_id)
|
||||||
logger.info("[Feishu] running card updated: source=%s card=%s", source_message_id, running_card_id)
|
logger.info("[Feishu] running card updated: source=%s card=%s", source_message_id, running_card_id)
|
||||||
elif msg.is_final:
|
elif msg.is_final:
|
||||||
@ -1283,7 +1098,6 @@ class FeishuChannel(Channel):
|
|||||||
card_text,
|
card_text,
|
||||||
**raw_card_kw,
|
**raw_card_kw,
|
||||||
)
|
)
|
||||||
delivered.append(final_card_id)
|
|
||||||
await self._remember_card_delivery(msg, card_request, final_card_id, topic_id=card_topic_id)
|
await self._remember_card_delivery(msg, card_request, final_card_id, topic_id=card_topic_id)
|
||||||
elif awaited_running_card_task:
|
elif awaited_running_card_task:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
@ -1298,6 +1112,14 @@ class FeishuChannel(Channel):
|
|||||||
)
|
)
|
||||||
await self._remember_thread_mapping(msg, source_message_id, created_card_id)
|
await self._remember_thread_mapping(msg, source_message_id, created_card_id)
|
||||||
|
|
||||||
|
if msg.is_final:
|
||||||
|
self._running_card_ids.pop(source_message_id, None)
|
||||||
|
self._remembered_thread_aliases.pop(source_message_id, None)
|
||||||
|
await self._add_reaction(source_message_id, "DONE")
|
||||||
|
return
|
||||||
|
|
||||||
|
await self._create_card(msg.chat_id, msg.text)
|
||||||
|
|
||||||
# -- internal ----------------------------------------------------------
|
# -- internal ----------------------------------------------------------
|
||||||
|
|
||||||
async def _remember_thread_mapping(self, msg: OutboundMessage, *topic_ids: str | None) -> None:
|
async def _remember_thread_mapping(self, msg: OutboundMessage, *topic_ids: str | None) -> None:
|
||||||
@ -1787,10 +1609,6 @@ class FeishuChannel(Channel):
|
|||||||
}
|
}
|
||||||
if source_preview:
|
if source_preview:
|
||||||
metadata[SOURCE_PREVIEW_METADATA_KEY] = source_preview
|
metadata[SOURCE_PREVIEW_METADATA_KEY] = source_preview
|
||||||
if parsed.post_file_count:
|
|
||||||
metadata[_POST_FILES_COUNT_METADATA_KEY] = parsed.post_file_count
|
|
||||||
if parsed.msg_type == InboundMessageType.COMMAND:
|
|
||||||
metadata[_DEFER_POST_FILE_PLACEHOLDERS_METADATA_KEY] = True
|
|
||||||
if routing.clarification_card_id:
|
if routing.clarification_card_id:
|
||||||
metadata[_PENDING_CARD_METADATA_KEY] = routing.clarification_card_id
|
metadata[_PENDING_CARD_METADATA_KEY] = routing.clarification_card_id
|
||||||
|
|
||||||
@ -2129,26 +1947,6 @@ class FeishuChannel(Channel):
|
|||||||
text = ""
|
text = ""
|
||||||
text = text.strip()
|
text = text.strip()
|
||||||
|
|
||||||
# Feishu desktop attaches files to a message as a post whose rich-text
|
|
||||||
# body holds the caption and whose top-level ``files`` array holds the
|
|
||||||
# attachments. The body may be empty. Keep these attachments last so
|
|
||||||
# ``receive_file`` can match their placeholders from the end. Skip
|
|
||||||
# folders and attachments already represented by a rich-text element.
|
|
||||||
attachments = content.get("files") if isinstance(content, dict) else None
|
|
||||||
if not isinstance(attachments, list):
|
|
||||||
attachments = []
|
|
||||||
post_file_count = 0
|
|
||||||
for attachment in attachments:
|
|
||||||
if not isinstance(attachment, dict) or attachment.get("is_folder"):
|
|
||||||
continue
|
|
||||||
file_key = attachment.get("file_key")
|
|
||||||
if not isinstance(file_key, str) or not file_key:
|
|
||||||
continue
|
|
||||||
if any(existing.get("file_key") == file_key for existing in files_list):
|
|
||||||
continue
|
|
||||||
files_list.append({"file_key": file_key})
|
|
||||||
post_file_count += 1
|
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"[Feishu] parsed message: chat_id=%s, msg_id=%s, root_id=%s, parent_id=%s, thread_id=%s, chat_type=%s, sender=%s, text_len=%d",
|
"[Feishu] parsed message: chat_id=%s, msg_id=%s, root_id=%s, parent_id=%s, thread_id=%s, chat_type=%s, sender=%s, text_len=%d",
|
||||||
chat_id,
|
chat_id,
|
||||||
@ -2174,7 +1972,6 @@ class FeishuChannel(Channel):
|
|||||||
chat_type=chat_type,
|
chat_type=chat_type,
|
||||||
text=text,
|
text=text,
|
||||||
files=files_list,
|
files=files_list,
|
||||||
post_file_count=post_file_count,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
def _on_message(self, event) -> None:
|
def _on_message(self, event) -> None:
|
||||||
@ -2217,8 +2014,6 @@ class FeishuChannel(Channel):
|
|||||||
if _is_feishu_command(command_text):
|
if _is_feishu_command(command_text):
|
||||||
parsed.msg_type = InboundMessageType.COMMAND
|
parsed.msg_type = InboundMessageType.COMMAND
|
||||||
parsed.text = command_text
|
parsed.text = command_text
|
||||||
elif parsed.post_file_count:
|
|
||||||
parsed.text = _append_post_file_placeholders(text, parsed.post_file_count)
|
|
||||||
|
|
||||||
# topic_id determines which LangGraph thread the message maps to.
|
# topic_id determines which LangGraph thread the message maps to.
|
||||||
# P2P chats: topic_id=None so all messages share one thread (like Telegram DMs).
|
# P2P chats: topic_id=None so all messages share one thread (like Telegram DMs).
|
||||||
|
|||||||
@ -16,7 +16,6 @@ from __future__ import annotations
|
|||||||
import math
|
import math
|
||||||
from collections.abc import Mapping
|
from collections.abc import Mapping
|
||||||
|
|
||||||
from app.channels.reply_locale import supported_locale
|
|
||||||
from app.scheduler.notification_text import goal_reason_text
|
from app.scheduler.notification_text import goal_reason_text
|
||||||
from deerflow.persistence.scheduled_task_runs.finalization import CHECK_FAILURE_CODES
|
from deerflow.persistence.scheduled_task_runs.finalization import CHECK_FAILURE_CODES
|
||||||
from deerflow.runtime.goal import is_active_goal
|
from deerflow.runtime.goal import is_active_goal
|
||||||
@ -39,18 +38,6 @@ _TEXT: dict[str, dict[str, str]] = {
|
|||||||
"fetch_failed": "Failed to fetch goal information.",
|
"fetch_failed": "Failed to fetch goal information.",
|
||||||
"clear_failed": "Failed to clear goal.",
|
"clear_failed": "Failed to clear goal.",
|
||||||
"set_failed": "Failed to set goal.",
|
"set_failed": "Failed to set goal.",
|
||||||
"busy": "A task is still running. Wait for it to finish before changing the goal.",
|
|
||||||
"from_chat_saved": "Goal set from the conversation: {objective}",
|
|
||||||
"from_chat_saved_but_not_started": "Goal saved from the conversation, but the task did not start. Send any message to pick it up, or check /goal.",
|
|
||||||
"from_chat_run_status_unknown": "Goal saved from the conversation. I couldn't confirm the task's status, and it may still be running. Check the conversation in DeerFlow before sending it again.",
|
|
||||||
"from_chat_unclear": "I couldn't form a clear goal from this conversation. Try adding a hint, e.g. /goal from-chat focus on the vendor totals.",
|
|
||||||
"from_chat_no_context": "There is no conversation content to build a goal from yet. Discuss the task first, or set it directly with /goal <objective>.",
|
|
||||||
"from_chat_pending_clarification": "The assistant is still waiting for your answer in this conversation. Reply to it first, then retry /goal from-chat.",
|
|
||||||
"from_chat_no_thread": "No conversation yet. Send a message first, then ask /goal from-chat.",
|
|
||||||
"from_chat_changed": "The conversation changed while the goal was being composed, so nothing was saved. Retry /goal from-chat, or set it directly with /goal <objective>.",
|
|
||||||
"from_chat_conversation_switched": "This chat switched to a new conversation while the goal was being composed, so the task did not start. The goal remains saved in the previous conversation: {objective}",
|
|
||||||
"from_chat_timeout": "Timed out waiting for the goal to be composed. The save may still finish — check with /goal, and retry if there is no goal.",
|
|
||||||
"from_chat_failed": "Failed to compose the goal from the conversation. Try again, or set it directly with /goal <objective>.",
|
|
||||||
},
|
},
|
||||||
"zh-CN": {
|
"zh-CN": {
|
||||||
"goal": "目标：{objective}",
|
"goal": "目标：{objective}",
|
||||||
@ -69,18 +56,6 @@ _TEXT: dict[str, dict[str, str]] = {
|
|||||||
"fetch_failed": "无法获取目标信息。",
|
"fetch_failed": "无法获取目标信息。",
|
||||||
"clear_failed": "无法清除目标。",
|
"clear_failed": "无法清除目标。",
|
||||||
"set_failed": "无法设置目标。",
|
"set_failed": "无法设置目标。",
|
||||||
"busy": "当前任务仍在运行，请等待结束后再修改目标。",
|
|
||||||
"from_chat_saved": "已从对话设置目标：{objective}",
|
|
||||||
"from_chat_saved_but_not_started": "目标已从对话保存，但任务未能启动。发送任意消息即可继续，或用 /goal 查看状态。",
|
|
||||||
"from_chat_run_status_unknown": "目标已从对话保存，但暂时无法确认任务状态，任务可能仍在运行。请先在 DeerFlow 中查看会话，再决定是否重试。",
|
|
||||||
"from_chat_unclear": "无法从当前对话中形成清楚的目标。可以补充提示，例如 /goal from-chat 重点核对供应商合计。",
|
|
||||||
"from_chat_no_context": "当前对话还没有可用于构建目标的内容。先讨论任务，或用 /goal <目标> 直接设置。",
|
|
||||||
"from_chat_pending_clarification": "助手还在等你回答对话里的问题。先回复它，再重试 /goal from-chat。",
|
|
||||||
"from_chat_no_thread": "还没有会话。先发一条消息，再用 /goal from-chat。",
|
|
||||||
"from_chat_changed": "生成目标期间对话内容发生了变化，本次未保存。请重试 /goal from-chat，或用 /goal <目标> 直接设置。",
|
|
||||||
"from_chat_conversation_switched": "生成目标期间当前聊天切换到了新会话，任务没有启动。目标仍保存在之前的会话里：{objective}",
|
|
||||||
"from_chat_timeout": "从对话生成目标超时。保存可能仍会完成——先用 /goal 查看，如果没有再重试。",
|
|
||||||
"from_chat_failed": "从对话生成目标失败。请重试，或用 /goal <目标> 直接设置。",
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -90,7 +65,6 @@ _WAITING_KEYS = {"needs_user_input": "waiting_for_you", "external_wait": "waitin
|
|||||||
|
|
||||||
def goal_reply_text(key: str, locale: str) -> str:
|
def goal_reply_text(key: str, locale: str) -> str:
|
||||||
"""A fixed ``/goal`` reply (``none``, ``cleared``, ``*_failed``) in ``locale``."""
|
"""A fixed ``/goal`` reply (``none``, ``cleared``, ``*_failed``) in ``locale``."""
|
||||||
locale = supported_locale(locale)
|
|
||||||
return _TEXT[locale][key]
|
return _TEXT[locale][key]
|
||||||
|
|
||||||
|
|
||||||
@ -128,7 +102,6 @@ def goal_status_text(goal: object, outcome: object, *, running: bool, locale: st
|
|||||||
|
|
||||||
``running`` says whether the thread has a run pending or in progress.
|
``running`` says whether the thread has a run pending or in progress.
|
||||||
"""
|
"""
|
||||||
locale = supported_locale(locale)
|
|
||||||
text = _TEXT[locale]
|
text = _TEXT[locale]
|
||||||
if is_active_goal_state(goal):
|
if is_active_goal_state(goal):
|
||||||
lines = [text["goal"].format(objective=goal["objective"])]
|
lines = [text["goal"].format(objective=goal["objective"])]
|
||||||
|
|||||||
@ -24,7 +24,7 @@ from urllib.parse import quote, urlparse
|
|||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from fastapi import HTTPException
|
from fastapi import HTTPException
|
||||||
from langgraph_sdk.errors import APIStatusError, ConflictError
|
from langgraph_sdk.errors import ConflictError
|
||||||
|
|
||||||
from app.channels import buzz_run_policy as _buzz_run_policy # noqa: F401
|
from app.channels import buzz_run_policy as _buzz_run_policy # noqa: F401
|
||||||
from app.channels import feishu_run_policy as _feishu_run_policy # noqa: F401
|
from app.channels import feishu_run_policy as _feishu_run_policy # noqa: F401
|
||||||
@ -69,7 +69,6 @@ from deerflow.skills.storage.skill_storage import SkillStorage
|
|||||||
from deerflow.trace_context import ensure_trace_context
|
from deerflow.trace_context import ensure_trace_context
|
||||||
from deerflow.uploads.manager import apply_upload_sandbox_permits
|
from deerflow.uploads.manager import apply_upload_sandbox_permits
|
||||||
from deerflow.utils.file_io import await_drained
|
from deerflow.utils.file_io import await_drained
|
||||||
from deerflow.utils.http_policy import control_plane_trust_env
|
|
||||||
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
|
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@ -142,7 +141,7 @@ _LIVE_RUN_STATUSES = frozenset({"pending", "running"})
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class _RunAdmission:
|
class _RunAdmission:
|
||||||
"""How far one inbound message's run request got, for dedupe and goal replies.
|
"""How far one inbound message's run request got, for dedupe release.
|
||||||
|
|
||||||
``requested`` flips right before the run request leaves the manager;
|
``requested`` flips right before the run request leaves the manager;
|
||||||
``confirmed`` once the Gateway proved it admitted the run (a run id, a
|
``confirmed`` once the Gateway proved it admitted the run (a run id, a
|
||||||
@ -151,7 +150,6 @@ class _RunAdmission:
|
|||||||
|
|
||||||
requested: bool = False
|
requested: bool = False
|
||||||
confirmed: bool = False
|
confirmed: bool = False
|
||||||
error: BaseException | None = None
|
|
||||||
|
|
||||||
|
|
||||||
_RUN_ADMISSION: ContextVar[_RunAdmission | None] = ContextVar("deerflow_channel_run_admission", default=None)
|
_RUN_ADMISSION: ContextVar[_RunAdmission | None] = ContextVar("deerflow_channel_run_admission", default=None)
|
||||||
@ -162,22 +160,11 @@ _RUN_ADMISSION: ContextVar[_RunAdmission | None] = ContextVar("deerflow_channel_
|
|||||||
# status errors mean the Gateway never admitted it.
|
# status errors mean the Gateway never admitted it.
|
||||||
_RUN_REQUEST_REACHED_GATEWAY_ERRORS = (httpx.ReadTimeout, httpx.ReadError, httpx.RemoteProtocolError)
|
_RUN_REQUEST_REACHED_GATEWAY_ERRORS = (httpx.ReadTimeout, httpx.ReadError, httpx.RemoteProtocolError)
|
||||||
|
|
||||||
# ``/goal from-chat`` makes the Gateway run a synthesis LLM call before it
|
|
||||||
# answers, which routinely exceeds the 10s budget the other ``/goal`` calls
|
|
||||||
# need for a checkpoint read/write — the live Feishu run timed out client-side
|
|
||||||
# while the Gateway went on to save the goal. Keep connect/write/pool short so
|
|
||||||
# a dead Gateway still fails fast, and extend only the read for the synthesis.
|
|
||||||
_GOAL_FROM_CHAT_TIMEOUT = httpx.Timeout(10.0, read=120.0)
|
|
||||||
|
|
||||||
|
|
||||||
class _RunFailedFrame(RuntimeError):
|
class _RunFailedFrame(RuntimeError):
|
||||||
"""The stream's ``error`` frame: the run ended, so a retry is not a duplicate."""
|
"""The stream's ``error`` frame: the run ended, so a retry is not a duplicate."""
|
||||||
|
|
||||||
|
|
||||||
class _ConversationSwitched(RuntimeError):
|
|
||||||
"""The chat no longer points at the thread whose goal was saved."""
|
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def _run_admission_scope(*, fresh: bool = False) -> Iterator[_RunAdmission]:
|
def _run_admission_scope(*, fresh: bool = False) -> Iterator[_RunAdmission]:
|
||||||
"""Install the admission record for one inbound message.
|
"""Install the admission record for one inbound message.
|
||||||
@ -208,11 +195,6 @@ def _note_run_confirmed() -> None:
|
|||||||
admission.confirmed = True
|
admission.confirmed = True
|
||||||
|
|
||||||
|
|
||||||
def _note_run_error(error: BaseException) -> None:
|
|
||||||
if (admission := _RUN_ADMISSION.get()) is not None:
|
|
||||||
admission.error = error
|
|
||||||
|
|
||||||
|
|
||||||
# Recovery replies must say the turn was dropped: the run was rejected before
|
# Recovery replies must say the turn was dropped: the run was rejected before
|
||||||
# starting, so the triggering message never reached the agent.
|
# starting, so the triggering message never reached the agent.
|
||||||
_STALE_PIN_RETRY_NOTE = "Your message was not processed — please send it again."
|
_STALE_PIN_RETRY_NOTE = "Your message was not processed — please send it again."
|
||||||
@ -319,59 +301,6 @@ _WECOM_MEDIA_COS_HOST_RE = re.compile(r"^ww-aibot-img-(?P<appid>\d+)\.cos\.[a-z0
|
|||||||
_METADATA_DROP_KEYS = frozenset({"raw_message", "ref_msg"})
|
_METADATA_DROP_KEYS = frozenset({"raw_message", "ref_msg"})
|
||||||
|
|
||||||
|
|
||||||
def _make_gateway_client(gateway_url: str) -> httpx.AsyncClient:
|
|
||||||
"""Build the short-lived client for Gateway control-plane calls.
|
|
||||||
|
|
||||||
A loopback Gateway must bypass ``HTTP_PROXY``, while an external one keeps
|
|
||||||
the environment proxy. The policy itself lives in
|
|
||||||
:mod:`deerflow.utils.http_policy` so this module never imports the sandbox
|
|
||||||
provider stack.
|
|
||||||
"""
|
|
||||||
return httpx.AsyncClient(trust_env=control_plane_trust_env(gateway_url))
|
|
||||||
|
|
||||||
|
|
||||||
# Mirrors the client ``langgraph_sdk.get_client()`` builds, so the loopback path
|
|
||||||
# below differs from it only in ``trust_env``.
|
|
||||||
_LANGGRAPH_SDK_RETRIES = 5
|
|
||||||
_LANGGRAPH_SDK_TIMEOUT = httpx.Timeout(connect=5, read=300, write=300, pool=5)
|
|
||||||
|
|
||||||
|
|
||||||
def _make_langgraph_client(base_url: str, headers: Mapping[str, str]) -> Any:
|
|
||||||
"""Build the ``langgraph_sdk`` async client used for channel run submission.
|
|
||||||
|
|
||||||
``get_client()`` always builds ``httpx.AsyncHTTPTransport(retries=5)`` on an
|
|
||||||
``httpx.AsyncClient`` that keeps ``trust_env`` at its default ``True`` and
|
|
||||||
exposes neither ``transport`` nor ``trust_env``, so a loopback Gateway would
|
|
||||||
still be sent through ``HTTP_PROXY``: control-plane calls are fixed but
|
|
||||||
``runs.wait`` keeps hitting the proxy-generated error of #7876. Loopback and
|
|
||||||
private Gateways therefore get an explicit ``trust_env=False`` client handed
|
|
||||||
to the SDK; external Gateways keep the stock ``get_client()`` path.
|
|
||||||
"""
|
|
||||||
from langgraph_sdk import __version__ as sdk_version
|
|
||||||
from langgraph_sdk import get_client
|
|
||||||
|
|
||||||
if control_plane_trust_env(base_url):
|
|
||||||
return get_client(url=base_url, headers=dict(headers))
|
|
||||||
|
|
||||||
try: # langgraph-sdk keeps the async client in a private package since 0.3
|
|
||||||
from langgraph_sdk._async import LangGraphClient
|
|
||||||
except ImportError: # older releases expose it directly on ``langgraph_sdk.client``
|
|
||||||
from langgraph_sdk.client import LangGraphClient
|
|
||||||
|
|
||||||
# ``get_client`` also folds an ``LANGGRAPH_API_KEY`` env var into the SDK
|
|
||||||
# headers; a control-plane Gateway authenticates with the internal auth
|
|
||||||
# headers above, so only those are forwarded here.
|
|
||||||
return LangGraphClient(
|
|
||||||
httpx.AsyncClient(
|
|
||||||
base_url=base_url,
|
|
||||||
transport=httpx.AsyncHTTPTransport(retries=_LANGGRAPH_SDK_RETRIES),
|
|
||||||
timeout=_LANGGRAPH_SDK_TIMEOUT,
|
|
||||||
headers={"User-Agent": f"langgraph-sdk-py/{sdk_version}", **headers},
|
|
||||||
trust_env=False,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _slim_metadata(meta: dict[str, Any]) -> dict[str, Any]:
|
def _slim_metadata(meta: dict[str, Any]) -> dict[str, Any]:
|
||||||
"""Return a shallow copy of *meta* with known-large keys removed."""
|
"""Return a shallow copy of *meta* with known-large keys removed."""
|
||||||
return {k: v for k, v in meta.items() if k not in _METADATA_DROP_KEYS}
|
return {k: v for k, v in meta.items() if k not in _METADATA_DROP_KEYS}
|
||||||
@ -670,7 +599,7 @@ def _is_run_status_result(result: Any) -> bool:
|
|||||||
return isinstance(result, dict) and "messages" not in result and isinstance(result.get("status"), str)
|
return isinstance(result, dict) and "messages" not in result and isinstance(result.get("status"), str)
|
||||||
|
|
||||||
|
|
||||||
def _turn_of_message(values: Any, message_id: str, *, thread_id: str | None = None) -> dict[str, Any] | None:
|
def _turn_of_message(values: Any, message_id: str) -> dict[str, Any] | None:
|
||||||
"""Thread *values* cut after the turn that answered human message *message_id*.
|
"""Thread *values* cut after the turn that answered human message *message_id*.
|
||||||
|
|
||||||
The cut ends before the next visible human message, so the response, artifact
|
The cut ends before the next visible human message, so the response, artifact
|
||||||
@ -682,7 +611,7 @@ def _turn_of_message(values: Any, message_id: str, *, thread_id: str | None = No
|
|||||||
for index, message in enumerate(messages):
|
for index, message in enumerate(messages):
|
||||||
if isinstance(message, dict) and message.get("type") == "human" and message.get("id") == message_id:
|
if isinstance(message, dict) and message.get("type") == "human" and message.get("id") == message_id:
|
||||||
end = next(
|
end = next(
|
||||||
(later for later in range(index + 1, len(messages)) if isinstance(messages[later], dict) and messages[later].get("type") == "human" and not _is_hidden_human_control_message(messages[later], thread_id=thread_id)),
|
(later for later in range(index + 1, len(messages)) if isinstance(messages[later], dict) and messages[later].get("type") == "human" and not _is_hidden_human_control_message(messages[later])),
|
||||||
len(messages),
|
len(messages),
|
||||||
)
|
)
|
||||||
return {**values, "messages": messages[:end]}
|
return {**values, "messages": messages[:end]}
|
||||||
@ -881,7 +810,7 @@ def _apply_explicit_model_choice(
|
|||||||
carrier["model_name"] = model_name
|
carrier["model_name"] = model_name
|
||||||
|
|
||||||
|
|
||||||
def _extract_response_text(result: dict | list, *, thread_id: str | None = None) -> str:
|
def _extract_response_text(result: dict | list) -> str:
|
||||||
"""Extract the last AI message text from a LangGraph runs.wait result.
|
"""Extract the last AI message text from a LangGraph runs.wait result.
|
||||||
|
|
||||||
``runs.wait`` returns the final state dict which contains a ``messages``
|
``runs.wait`` returns the final state dict which contains a ``messages``
|
||||||
@ -908,7 +837,7 @@ def _extract_response_text(result: dict | list, *, thread_id: str | None = None)
|
|||||||
|
|
||||||
# Stop at the last human message — anything before it is a previous turn
|
# Stop at the last human message — anything before it is a previous turn
|
||||||
if msg_type == "human":
|
if msg_type == "human":
|
||||||
if _is_hidden_human_control_message(msg, thread_id=thread_id):
|
if _is_hidden_human_control_message(msg):
|
||||||
continue
|
continue
|
||||||
break
|
break
|
||||||
|
|
||||||
@ -1217,7 +1146,7 @@ def _accumulate_stream_text(
|
|||||||
return buffers[message_id], message_id
|
return buffers[message_id], message_id
|
||||||
|
|
||||||
|
|
||||||
def _extract_artifacts(result: dict | list, *, thread_id: str | None = None) -> list[str]:
|
def _extract_artifacts(result: dict | list) -> list[str]:
|
||||||
"""Extract artifact paths from the last AI response cycle only.
|
"""Extract artifact paths from the last AI response cycle only.
|
||||||
|
|
||||||
Instead of reading the full accumulated ``artifacts`` state (which contains
|
Instead of reading the full accumulated ``artifacts`` state (which contains
|
||||||
@ -1238,39 +1167,27 @@ def _extract_artifacts(result: dict | list, *, thread_id: str | None = None) ->
|
|||||||
continue
|
continue
|
||||||
# Stop at the last human message — anything before it is a previous turn
|
# Stop at the last human message — anything before it is a previous turn
|
||||||
if msg.get("type") == "human":
|
if msg.get("type") == "human":
|
||||||
if _is_hidden_human_control_message(msg, thread_id=thread_id):
|
if _is_hidden_human_control_message(msg):
|
||||||
continue
|
continue
|
||||||
break
|
break
|
||||||
# Look for AI messages with present_files tool calls
|
# Look for AI messages with present_files tool calls
|
||||||
if msg.get("type") == "ai":
|
if msg.get("type") == "ai":
|
||||||
message_paths: list[str] = []
|
|
||||||
for tc in msg.get("tool_calls", []):
|
for tc in msg.get("tool_calls", []):
|
||||||
if isinstance(tc, dict) and tc.get("name") == "present_files":
|
if isinstance(tc, dict) and tc.get("name") == "present_files":
|
||||||
args = tc.get("args", {})
|
args = tc.get("args", {})
|
||||||
paths = args.get("filepaths", [])
|
paths = args.get("filepaths", [])
|
||||||
if isinstance(paths, list):
|
if isinstance(paths, list):
|
||||||
message_paths.extend(p for p in paths if isinstance(p, str))
|
artifacts.extend(p for p in paths if isinstance(p, str))
|
||||||
# The scan runs newest-first; prepend so earlier calls stay first.
|
return artifacts
|
||||||
artifacts[:0] = message_paths
|
|
||||||
# Match the ``merge_artifacts`` reducer: first-presented order, no repeats,
|
|
||||||
# so a file presented twice in one turn is uploaded once.
|
|
||||||
return list(dict.fromkeys(artifacts))
|
|
||||||
|
|
||||||
|
|
||||||
def _is_hidden_human_control_message(msg: Mapping[str, Any], *, thread_id: str | None = None) -> bool:
|
def _is_hidden_human_control_message(msg: Mapping[str, Any]) -> bool:
|
||||||
"""Distinguish internal control messages from real hidden user answers."""
|
"""Distinguish internal control messages from real hidden user answers."""
|
||||||
if msg.get("type") != "human":
|
if msg.get("type") != "human":
|
||||||
return False
|
return False
|
||||||
|
|
||||||
additional_kwargs = msg.get("additional_kwargs")
|
additional_kwargs = msg.get("additional_kwargs")
|
||||||
if not isinstance(additional_kwargs, Mapping):
|
if not isinstance(additional_kwargs, Mapping):
|
||||||
if additional_kwargs is not None:
|
|
||||||
logger.warning(
|
|
||||||
"Ignoring malformed message additional_kwargs for thread %s at row seq %s: expected a mapping, got %s",
|
|
||||||
thread_id or msg.get("thread_id", "<unknown>"),
|
|
||||||
msg.get("seq", "<unknown>"),
|
|
||||||
type(additional_kwargs).__name__,
|
|
||||||
)
|
|
||||||
return False
|
return False
|
||||||
|
|
||||||
return additional_kwargs.get("hide_from_ui") is True and read_human_input_response(additional_kwargs) is None
|
return additional_kwargs.get("hide_from_ui") is True and read_human_input_response(additional_kwargs) is None
|
||||||
@ -2178,21 +2095,14 @@ class ChannelManager:
|
|||||||
|
|
||||||
self._maybe_spawn_followup_watcher(thread_id, result, carrier_msg)
|
self._maybe_spawn_followup_watcher(thread_id, result, carrier_msg)
|
||||||
|
|
||||||
async def _publish_progress_update(self, msg: InboundMessage, thread_id: str, text: str, *, is_final: bool = False) -> None:
|
async def _publish_progress_update(self, msg: InboundMessage, thread_id: str, text: str) -> None:
|
||||||
"""Publish a turn update; progress by default, final for standalone replies.
|
|
||||||
|
|
||||||
A non-final outbound joins the run's stream: streaming channels update
|
|
||||||
the same message with later chunks (and the final answer), while QQ and
|
|
||||||
DingTalk card mode drop it before a stream exists. Callers that must
|
|
||||||
stay visible on every channel publish ``is_final=True``.
|
|
||||||
"""
|
|
||||||
await self.bus.publish_outbound(
|
await self.bus.publish_outbound(
|
||||||
OutboundMessage(
|
OutboundMessage(
|
||||||
channel_name=msg.channel_name,
|
channel_name=msg.channel_name,
|
||||||
chat_id=msg.chat_id,
|
chat_id=msg.chat_id,
|
||||||
thread_id=thread_id,
|
thread_id=thread_id,
|
||||||
text=text,
|
text=text,
|
||||||
is_final=is_final,
|
is_final=False,
|
||||||
thread_ts=msg.thread_ts,
|
thread_ts=msg.thread_ts,
|
||||||
connection_id=msg.connection_id,
|
connection_id=msg.connection_id,
|
||||||
owner_user_id=msg.owner_user_id,
|
owner_user_id=msg.owner_user_id,
|
||||||
@ -2401,9 +2311,11 @@ class ChannelManager:
|
|||||||
def _get_client(self):
|
def _get_client(self):
|
||||||
"""Return the ``langgraph_sdk`` async client, creating it on first use."""
|
"""Return the ``langgraph_sdk`` async client, creating it on first use."""
|
||||||
if self._client is None:
|
if self._client is None:
|
||||||
self._client = _make_langgraph_client(
|
from langgraph_sdk import get_client
|
||||||
self._langgraph_url,
|
|
||||||
{**create_internal_auth_headers(), **self._csrf_headers()},
|
self._client = get_client(
|
||||||
|
url=self._langgraph_url,
|
||||||
|
headers={**create_internal_auth_headers(), **self._csrf_headers()},
|
||||||
)
|
)
|
||||||
return self._client
|
return self._client
|
||||||
|
|
||||||
@ -3314,7 +3226,6 @@ class ChannelManager:
|
|||||||
extra_context: dict[str, Any] | None = None,
|
extra_context: dict[str, Any] | None = None,
|
||||||
*,
|
*,
|
||||||
bound_identity_checked: bool = False,
|
bound_identity_checked: bool = False,
|
||||||
expected_thread_id: str | None = None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
# Normal entry paths already run the bound-identity check in
|
# Normal entry paths already run the bound-identity check in
|
||||||
# _handle_message() or _handle_command(). Keep this default False so
|
# _handle_message() or _handle_command(). Keep this default False so
|
||||||
@ -3331,10 +3242,7 @@ class ChannelManager:
|
|||||||
# first message for the chat. topic_id may be None (e.g. Telegram
|
# first message for the chat. topic_id may be None (e.g. Telegram
|
||||||
# private chats) — the store handles this by using the "channel:chat_id"
|
# private chats) — the store handles this by using the "channel:chat_id"
|
||||||
# key without a topic suffix.
|
# key without a topic suffix.
|
||||||
if expected_thread_id is None:
|
|
||||||
thread_id, created = await self._get_or_create_thread(client, msg)
|
thread_id, created = await self._get_or_create_thread(client, msg)
|
||||||
else:
|
|
||||||
thread_id, created = expected_thread_id, False
|
|
||||||
if not created:
|
if not created:
|
||||||
logger.info("[Manager] reusing thread: thread_id=%s for topic_id=%s", thread_id, msg.topic_id)
|
logger.info("[Manager] reusing thread: thread_id=%s for topic_id=%s", thread_id, msg.topic_id)
|
||||||
await self._update_thread_channel_metadata(client, msg, thread_id)
|
await self._update_thread_channel_metadata(client, msg, thread_id)
|
||||||
@ -3374,7 +3282,6 @@ class ChannelManager:
|
|||||||
thread_id,
|
thread_id,
|
||||||
extra_context=extra_context,
|
extra_context=extra_context,
|
||||||
storage_user_id=storage_user_id,
|
storage_user_id=storage_user_id,
|
||||||
check_thread_binding=expected_thread_id is not None,
|
|
||||||
)
|
)
|
||||||
finally:
|
finally:
|
||||||
self._finish_serialized_thread_run(
|
self._finish_serialized_thread_run(
|
||||||
@ -3392,7 +3299,6 @@ class ChannelManager:
|
|||||||
*,
|
*,
|
||||||
extra_context: dict[str, Any] | None = None,
|
extra_context: dict[str, Any] | None = None,
|
||||||
storage_user_id: str | None = None,
|
storage_user_id: str | None = None,
|
||||||
check_thread_binding: bool = False,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
if storage_user_id is None:
|
if storage_user_id is None:
|
||||||
storage_user_id = _channel_storage_user_id(msg)
|
storage_user_id = _channel_storage_user_id(msg)
|
||||||
@ -3432,11 +3338,6 @@ class ChannelManager:
|
|||||||
uploaded = await _ingest_inbound_files(thread_id, msg, user_id=storage_user_id)
|
uploaded = await _ingest_inbound_files(thread_id, msg, user_id=storage_user_id)
|
||||||
human_message = _human_input_message_with_channel_response(msg.text, metadata=msg.metadata, original_content=original_text, files=uploaded or None, message_id=human_message_id)
|
human_message = _human_input_message_with_channel_response(msg.text, metadata=msg.metadata, original_content=original_text, files=uploaded or None, message_id=human_message_id)
|
||||||
|
|
||||||
# Metadata loading, queueing and file preparation may have awaited a
|
|
||||||
# /new. Keep the saved thread pinned and check its binding at handoff.
|
|
||||||
if check_thread_binding and await self._lookup_thread_id(msg) != thread_id:
|
|
||||||
raise _ConversationSwitched
|
|
||||||
|
|
||||||
if self._channel_supports_streaming(msg.channel_name):
|
if self._channel_supports_streaming(msg.channel_name):
|
||||||
await self._handle_streaming_chat(
|
await self._handle_streaming_chat(
|
||||||
client,
|
client,
|
||||||
@ -3487,7 +3388,6 @@ class ChannelManager:
|
|||||||
# behaviorally identical to the previous bare ``await``.
|
# behaviorally identical to the previous bare ``await``.
|
||||||
result = await client.runs.create(thread_id, assistant_id, **run_kwargs)
|
result = await client.runs.create(thread_id, assistant_id, **run_kwargs)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
_note_run_error(exc)
|
|
||||||
if _is_admitted_run_conflict(exc):
|
if _is_admitted_run_conflict(exc):
|
||||||
# An earlier delivery's run; it posts its own outcome.
|
# An earlier delivery's run; it posts its own outcome.
|
||||||
_note_run_confirmed()
|
_note_run_confirmed()
|
||||||
@ -3528,7 +3428,6 @@ class ChannelManager:
|
|||||||
**run_kwargs,
|
**run_kwargs,
|
||||||
)
|
)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
_note_run_error(exc)
|
|
||||||
if _is_admitted_run_conflict(exc):
|
if _is_admitted_run_conflict(exc):
|
||||||
_note_run_confirmed()
|
_note_run_confirmed()
|
||||||
logger.info("[Manager] run already admitted for this message: thread_id=%s (%s)", thread_id, exc)
|
logger.info("[Manager] run already admitted for this message: thread_id=%s (%s)", thread_id, exc)
|
||||||
@ -3542,7 +3441,6 @@ class ChannelManager:
|
|||||||
await self._send_error(msg, THREAD_BUSY_MESSAGE)
|
await self._send_error(msg, THREAD_BUSY_MESSAGE)
|
||||||
return
|
return
|
||||||
if step_limit_text := _step_limit_text(exc):
|
if step_limit_text := _step_limit_text(exc):
|
||||||
_note_run_confirmed()
|
|
||||||
logger.warning("[Manager] run stopped at its step limit: thread_id=%s", thread_id)
|
logger.warning("[Manager] run stopped at its step limit: thread_id=%s", thread_id)
|
||||||
await self._send_error(msg, step_limit_text)
|
await self._send_error(msg, step_limit_text)
|
||||||
# The run ended, as after a terminal error frame: release the key
|
# The run ended, as after a terminal error frame: release the key
|
||||||
@ -3564,9 +3462,9 @@ class ChannelManager:
|
|||||||
return
|
return
|
||||||
result = await self._admitted_turn(client, msg, thread_id) or result
|
result = await self._admitted_turn(client, msg, thread_id) or result
|
||||||
|
|
||||||
response_text = _extract_response_text(result, thread_id=thread_id)
|
response_text = _extract_response_text(result)
|
||||||
pending_clarification = _has_current_turn_clarification(result)
|
pending_clarification = _has_current_turn_clarification(result)
|
||||||
artifacts = _extract_artifacts(result, thread_id=thread_id)
|
artifacts = _extract_artifacts(result)
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"[Manager] agent response received: thread_id=%s, response_len=%d, artifacts=%d",
|
"[Manager] agent response received: thread_id=%s, response_len=%d, artifacts=%d",
|
||||||
@ -3619,7 +3517,7 @@ class ChannelManager:
|
|||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("[Manager] failed to read thread state for an admitted run: thread_id=%s", thread_id, exc_info=True)
|
logger.warning("[Manager] failed to read thread state for an admitted run: thread_id=%s", thread_id, exc_info=True)
|
||||||
return None
|
return None
|
||||||
return _turn_of_message(state.get("values") if isinstance(state, dict) else None, message_id, thread_id=thread_id)
|
return _turn_of_message(state.get("values") if isinstance(state, dict) else None, message_id)
|
||||||
|
|
||||||
async def _handle_streaming_chat(
|
async def _handle_streaming_chat(
|
||||||
self,
|
self,
|
||||||
@ -3696,7 +3594,7 @@ class ChannelManager:
|
|||||||
# Clarification text is only in the values snapshot;
|
# Clarification text is only in the values snapshot;
|
||||||
# publish it so the user sees the question mid-stream.
|
# publish it so the user sees the question mid-stream.
|
||||||
if _has_current_turn_clarification(data):
|
if _has_current_turn_clarification(data):
|
||||||
clarification_text = _extract_response_text(data, thread_id=thread_id)
|
clarification_text = _extract_response_text(data)
|
||||||
if clarification_text and clarification_text != latest_text:
|
if clarification_text and clarification_text != latest_text:
|
||||||
latest_text = clarification_text
|
latest_text = clarification_text
|
||||||
|
|
||||||
@ -3743,14 +3641,12 @@ class ChannelManager:
|
|||||||
pin_recovery = await self._stale_pin_recovery(msg, exc, expected_pin=expected_pin, thread_id=thread_id)
|
pin_recovery = await self._stale_pin_recovery(msg, exc, expected_pin=expected_pin, thread_id=thread_id)
|
||||||
pin_recovery_text = pin_recovery.text if pin_recovery else None
|
pin_recovery_text = pin_recovery.text if pin_recovery else None
|
||||||
finally:
|
finally:
|
||||||
if stream_error is not None:
|
|
||||||
_note_run_error(stream_error)
|
|
||||||
if replay_gap and last_values is None and not latest_text and stream_error is None:
|
if replay_gap and last_values is None and not latest_text and stream_error is None:
|
||||||
last_values = await self._admitted_turn(client, msg, thread_id)
|
last_values = await self._admitted_turn(client, msg, thread_id)
|
||||||
result = last_values if last_values is not None else {"messages": [{"type": "ai", "content": latest_text}]}
|
result = last_values if last_values is not None else {"messages": [{"type": "ai", "content": latest_text}]}
|
||||||
response_text = _extract_response_text(result, thread_id=thread_id)
|
response_text = _extract_response_text(result)
|
||||||
pending_clarification = _has_current_turn_clarification(result)
|
pending_clarification = _has_current_turn_clarification(result)
|
||||||
artifacts = _extract_artifacts(result, thread_id=thread_id)
|
artifacts = _extract_artifacts(result)
|
||||||
# Reuse the storage owner resolved by _handle_chat so artifact delivery
|
# Reuse the storage owner resolved by _handle_chat so artifact delivery
|
||||||
# matches the upload bucket and we avoid re-running _safe_user_id_for_run
|
# matches the upload bucket and we avoid re-running _safe_user_id_for_run
|
||||||
# (and its possible filesystem touch) on the streaming-error path.
|
# (and its possible filesystem touch) on the streaming-error path.
|
||||||
@ -3887,7 +3783,7 @@ class ChannelManager:
|
|||||||
reply = (
|
reply = (
|
||||||
"Available commands:\n"
|
"Available commands:\n"
|
||||||
"/bootstrap — Start a bootstrap session (enables agent setup)\n"
|
"/bootstrap — Start a bootstrap session (enables agent setup)\n"
|
||||||
"/goal [condition|clear|from-chat [hint]] — Set, show, clear, or compose the goal from the conversation\n"
|
"/goal [condition|clear] — Set, show, or clear an active goal\n"
|
||||||
"/new — Start a new conversation\n"
|
"/new — Start a new conversation\n"
|
||||||
"/status — Show current thread info\n"
|
"/status — Show current thread info\n"
|
||||||
"/models — List available models\n"
|
"/models — List available models\n"
|
||||||
@ -4128,27 +4024,6 @@ class ChannelManager:
|
|||||||
self._remember_thread_model(thread_id, name)
|
self._remember_thread_model(thread_id, name)
|
||||||
return f"Model '{name}' selected. It applies to this conversation from the next message."
|
return f"Model '{name}' selected. It applies to this conversation from the next message."
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _is_goal_in_flight_error(error: Exception) -> bool:
|
|
||||||
"""Recognize only the Gateway's known goal-write conflict, not arbitrary 409s."""
|
|
||||||
if not isinstance(error, httpx.HTTPStatusError) or error.response.status_code != 409:
|
|
||||||
return False
|
|
||||||
# POST (from-chat synthesis) saves the goal like PUT, so it carries the
|
|
||||||
# same "Set the goal ..." detail; recognizing it lets the IM reply reuse
|
|
||||||
# the busy wording for both the pre-check and the save-time race.
|
|
||||||
detail = {
|
|
||||||
"PUT": "Thread has a run in flight. Set the goal after the run finishes.",
|
|
||||||
"POST": "Thread has a run in flight. Set the goal after the run finishes.",
|
|
||||||
"DELETE": "Thread has a run in flight. Clear the goal after the run finishes.",
|
|
||||||
}.get(error.request.method)
|
|
||||||
if detail is None:
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
body = error.response.json()
|
|
||||||
except ValueError:
|
|
||||||
return False
|
|
||||||
return isinstance(body, Mapping) and body.get("detail") == detail
|
|
||||||
|
|
||||||
async def _goal_request(
|
async def _goal_request(
|
||||||
self,
|
self,
|
||||||
method: str,
|
method: str,
|
||||||
@ -4156,15 +4031,13 @@ class ChannelManager:
|
|||||||
*,
|
*,
|
||||||
headers: dict[str, str],
|
headers: dict[str, str],
|
||||||
json: dict[str, Any] | None = None,
|
json: dict[str, Any] | None = None,
|
||||||
suffix: str = "",
|
|
||||||
timeout: float | httpx.Timeout = 10,
|
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
async with _make_gateway_client(self._gateway_url) as http:
|
async with httpx.AsyncClient() as http:
|
||||||
request = getattr(http, method.lower())
|
request = getattr(http, method.lower())
|
||||||
kwargs: dict[str, Any] = {"timeout": timeout, "headers": {**headers, **self._csrf_headers()}}
|
kwargs: dict[str, Any] = {"timeout": 10, "headers": {**headers, **self._csrf_headers()}}
|
||||||
if json is not None:
|
if json is not None:
|
||||||
kwargs["json"] = json
|
kwargs["json"] = json
|
||||||
response = await request(f"{self._gateway_url}/api/threads/{quote(thread_id, safe='')}/goal{suffix}", **kwargs)
|
response = await request(f"{self._gateway_url}/api/threads/{quote(thread_id, safe='')}/goal", **kwargs)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
return response.json() or {}
|
return response.json() or {}
|
||||||
|
|
||||||
@ -4224,24 +4097,17 @@ class ChannelManager:
|
|||||||
return await self._goal_reply(msg, "cleared")
|
return await self._goal_reply(msg, "cleared")
|
||||||
try:
|
try:
|
||||||
await self._goal_request("delete", thread_id, headers=headers)
|
await self._goal_request("delete", thread_id, headers=headers)
|
||||||
except Exception as exc:
|
except Exception:
|
||||||
if self._is_goal_in_flight_error(exc):
|
|
||||||
return await self._goal_reply(msg, "busy")
|
|
||||||
logger.exception("Failed to clear goal through gateway")
|
logger.exception("Failed to clear goal through gateway")
|
||||||
return await self._goal_reply(msg, "clear_failed")
|
return await self._goal_reply(msg, "clear_failed")
|
||||||
return await self._goal_reply(msg, "cleared")
|
return await self._goal_reply(msg, "cleared")
|
||||||
|
|
||||||
if command.kind == "from_chat":
|
|
||||||
return await self._handle_goal_from_chat(msg, thread_id, guidance=command.objective, headers=headers)
|
|
||||||
|
|
||||||
if not thread_id:
|
if not thread_id:
|
||||||
thread_id, _ = await self._create_first_thread(self._get_client(), msg)
|
thread_id, _ = await self._create_first_thread(self._get_client(), msg)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
await self._goal_request("put", thread_id, headers=headers, json={"objective": command.objective})
|
await self._goal_request("put", thread_id, headers=headers, json={"objective": command.objective})
|
||||||
except Exception as exc:
|
except Exception:
|
||||||
if self._is_goal_in_flight_error(exc):
|
|
||||||
return await self._goal_reply(msg, "busy")
|
|
||||||
logger.exception("Failed to set goal through gateway")
|
logger.exception("Failed to set goal through gateway")
|
||||||
return await self._goal_reply(msg, "set_failed")
|
return await self._goal_reply(msg, "set_failed")
|
||||||
|
|
||||||
@ -4251,108 +4117,6 @@ class ChannelManager:
|
|||||||
await self._handle_chat(chat_msg, bound_identity_checked=True)
|
await self._handle_chat(chat_msg, bound_identity_checked=True)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
async def _handle_goal_from_chat(self, msg: InboundMessage, thread_id: str | None, *, guidance: str, headers: dict[str, str]) -> str | None:
|
|
||||||
"""``/goal from-chat``: synthesize an objective from the conversation, then start one run.
|
|
||||||
|
|
||||||
Unlike the literal ``set`` branch — which starts the run silently because
|
|
||||||
the user typed the objective — the objective here is model-synthesized,
|
|
||||||
so the user is shown what was saved before the run streams its output.
|
|
||||||
Saving and the busy/409 guard stay on the Gateway route; this only
|
|
||||||
reports the outcome and hands the saved objective to ``_handle_chat``.
|
|
||||||
Run handoff keeps this thread pinned and re-checks the binding after
|
|
||||||
confirmation delivery and preparation. Admission evidence distinguishes
|
|
||||||
rejected starts from requests whose response was lost.
|
|
||||||
"""
|
|
||||||
if not thread_id:
|
|
||||||
return await self._goal_reply(msg, "from_chat_no_thread")
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = await self._goal_request("post", thread_id, headers=headers, json={"guidance": guidance or None}, suffix="/from-chat", timeout=_GOAL_FROM_CHAT_TIMEOUT)
|
|
||||||
except Exception as exc:
|
|
||||||
return await self._goal_from_chat_error_reply(msg, exc)
|
|
||||||
|
|
||||||
goal = result.get("goal") if isinstance(result, Mapping) else None
|
|
||||||
objective = goal.get("objective") if isinstance(goal, Mapping) else None
|
|
||||||
if not isinstance(objective, str) or not objective.strip():
|
|
||||||
logger.warning("Goal from-chat returned no usable objective")
|
|
||||||
return await self._goal_reply(msg, "from_chat_failed")
|
|
||||||
|
|
||||||
from dataclasses import replace as _dc_replace
|
|
||||||
|
|
||||||
chat_msg = _dc_replace(msg, text=objective, msg_type=InboundMessageType.CHAT)
|
|
||||||
reply = None
|
|
||||||
may_be_running = False
|
|
||||||
with _run_admission_scope() as admission:
|
|
||||||
try:
|
|
||||||
if await self._lookup_thread_id(msg) != thread_id:
|
|
||||||
raise _ConversationSwitched
|
|
||||||
locale = await self._reply_locale.locale_for(_effective_owner_user_id(msg))
|
|
||||||
try:
|
|
||||||
# A separate final outbound keeps the saved objective visible
|
|
||||||
# alongside the run's stream on every IM provider.
|
|
||||||
await self._publish_progress_update(msg, thread_id, goal_reply_text("from_chat_saved", locale).format(objective=objective), is_final=True)
|
|
||||||
except Exception:
|
|
||||||
logger.warning("Goal from-chat saved but the confirmation card failed", exc_info=True)
|
|
||||||
await self._handle_chat(chat_msg, bound_identity_checked=True, expected_thread_id=thread_id)
|
|
||||||
except _ConversationSwitched:
|
|
||||||
reply = (await self._goal_reply(msg, "from_chat_conversation_switched")).format(objective=objective)
|
|
||||||
except Exception as exc:
|
|
||||||
_note_run_error(exc)
|
|
||||||
logger.warning("Goal from-chat run handoff failed", exc_info=True)
|
|
||||||
|
|
||||||
if reply is None and admission.error is not None and not admission.confirmed:
|
|
||||||
error = admission.error
|
|
||||||
rejected = isinstance(error, (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout))
|
|
||||||
if isinstance(error, (httpx.HTTPStatusError, APIStatusError)):
|
|
||||||
rejected = 400 <= error.response.status_code < 500
|
|
||||||
may_be_running = admission.requested and not rejected
|
|
||||||
reply = await self._goal_reply(msg, "from_chat_run_status_unknown" if may_be_running else "from_chat_saved_but_not_started")
|
|
||||||
if reply is not None:
|
|
||||||
# Use the saved thread even when its conversation binding cannot
|
|
||||||
# be read. Deliver the recovery advice before releasing dedupe.
|
|
||||||
await self._publish_progress_update(msg, thread_id, reply, is_final=True)
|
|
||||||
if admission.error is not None and not may_be_running:
|
|
||||||
await self._release_inbound_dedupe_key(msg)
|
|
||||||
return None
|
|
||||||
|
|
||||||
async def _goal_from_chat_error_reply(self, msg: InboundMessage, exc: Exception) -> str:
|
|
||||||
"""Map a from-chat Gateway error to the owner-language reply.
|
|
||||||
|
|
||||||
Only the known in-flight conflict, a client-side timeout, and the
|
|
||||||
structured synthesis outcomes (``unclear``, ``no_context``,
|
|
||||||
``pending_clarification``, ``goal_changed``) get their own wording;
|
|
||||||
anything else degrades to the generic retry reply. The model's ``reason``
|
|
||||||
is never read here, and the Gateway does not log it verbatim either.
|
|
||||||
"""
|
|
||||||
if self._is_goal_in_flight_error(exc):
|
|
||||||
return await self._goal_reply(msg, "busy")
|
|
||||||
if isinstance(exc, httpx.TimeoutException):
|
|
||||||
# The Gateway may finish synthesizing and save the goal after this
|
|
||||||
# client stops waiting, so the reply must not claim it failed.
|
|
||||||
logger.warning("Goal from-chat request timed out; the goal may still be saved", exc_info=True)
|
|
||||||
return await self._goal_reply(msg, "from_chat_timeout")
|
|
||||||
code = None
|
|
||||||
if isinstance(exc, httpx.HTTPStatusError):
|
|
||||||
try:
|
|
||||||
body = exc.response.json()
|
|
||||||
except ValueError:
|
|
||||||
body = None
|
|
||||||
detail = body.get("detail") if isinstance(body, Mapping) else None
|
|
||||||
if isinstance(detail, Mapping):
|
|
||||||
code = detail.get("code")
|
|
||||||
if code == "unclear":
|
|
||||||
return await self._goal_reply(msg, "from_chat_unclear")
|
|
||||||
if code == "no_context":
|
|
||||||
return await self._goal_reply(msg, "from_chat_no_context")
|
|
||||||
if code == "pending_clarification":
|
|
||||||
return await self._goal_reply(msg, "from_chat_pending_clarification")
|
|
||||||
if code == "goal_changed":
|
|
||||||
# The save's compare-and-swap found a newer checkpoint: another
|
|
||||||
# goal (or write) landed during synthesis, so nothing was saved.
|
|
||||||
return await self._goal_reply(msg, "from_chat_changed")
|
|
||||||
logger.warning("Goal from-chat failed through gateway", exc_info=True)
|
|
||||||
return await self._goal_reply(msg, "from_chat_failed")
|
|
||||||
|
|
||||||
async def _fetch_gateway_json(self, path: str, *, msg: InboundMessage | None = None) -> dict[str, Any]:
|
async def _fetch_gateway_json(self, path: str, *, msg: InboundMessage | None = None) -> dict[str, Any]:
|
||||||
"""GET a Gateway API path and return the parsed JSON body.
|
"""GET a Gateway API path and return the parsed JSON body.
|
||||||
|
|
||||||
@ -4360,7 +4124,7 @@ class ChannelManager:
|
|||||||
replies should catch and degrade (see ``_fetch_gateway``).
|
replies should catch and degrade (see ``_fetch_gateway``).
|
||||||
"""
|
"""
|
||||||
headers = _owner_headers(msg) if msg is not None else None
|
headers = _owner_headers(msg) if msg is not None else None
|
||||||
async with _make_gateway_client(self._gateway_url) as http:
|
async with httpx.AsyncClient() as http:
|
||||||
resp = await http.get(
|
resp = await http.get(
|
||||||
f"{self._gateway_url}{path}",
|
f"{self._gateway_url}{path}",
|
||||||
timeout=10,
|
timeout=10,
|
||||||
|
|||||||
@ -4,7 +4,6 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import math
|
|
||||||
import re
|
import re
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
@ -35,9 +34,6 @@ TELEGRAM_MAX_INBOUND_FILE_BYTES = 20_000_000
|
|||||||
# Keep Telegram cleanup inside the Gateway's five-second shutdown-hook budget.
|
# Keep Telegram cleanup inside the Gateway's five-second shutdown-hook budget.
|
||||||
TELEGRAM_SHUTDOWN_TIMEOUT_SECONDS = 4.0
|
TELEGRAM_SHUTDOWN_TIMEOUT_SECONDS = 4.0
|
||||||
TELEGRAM_BRIDGE_DRAIN_TIMEOUT_SECONDS = 1.0
|
TELEGRAM_BRIDGE_DRAIN_TIMEOUT_SECONDS = 1.0
|
||||||
# Outbound listeners await sends inline. Bound each flood wait while still
|
|
||||||
# allowing a full group rate-limit window.
|
|
||||||
TELEGRAM_MAX_OUTBOUND_RETRY_DELAY_SECONDS = 60.0
|
|
||||||
STREAM_EDIT_MIN_INTERVAL_SECONDS = 1.0
|
STREAM_EDIT_MIN_INTERVAL_SECONDS = 1.0
|
||||||
# Groups (negative chat_id) are capped at 20 messages/minute by Telegram,
|
# Groups (negative chat_id) are capped at 20 messages/minute by Telegram,
|
||||||
# so stream edits there must pace well below the private-chat 1 msg/s guideline.
|
# so stream edits there must pace well below the private-chat 1 msg/s guideline.
|
||||||
@ -327,20 +323,14 @@ class TelegramChannel(Channel):
|
|||||||
|
|
||||||
if not msg.is_final:
|
if not msg.is_final:
|
||||||
await self._send_stream_update(chat_id, key, msg.text, reply_to=self._parse_message_id(msg.thread_ts))
|
await self._send_stream_update(chat_id, key, msg.text, reply_to=self._parse_message_id(msg.thread_ts))
|
||||||
state = self._stream_messages.get(key)
|
|
||||||
if state is not None:
|
|
||||||
binding = (msg.thread_id, msg.connection_id, msg.owner_user_id, msg.thread_ts, state["message_id"])
|
|
||||||
if state.get("thread_binding") != binding and await self._remember_reply_thread(msg, state["message_id"], stream_state=state):
|
|
||||||
state["thread_binding"] = binding
|
|
||||||
return
|
return
|
||||||
|
|
||||||
state = self._stream_messages.pop(key, None)
|
state = self._stream_messages.pop(key, None)
|
||||||
if state is not None:
|
if state is not None:
|
||||||
await self._remember_reply_thread(msg, state["message_id"], stream_state=state)
|
|
||||||
if self._can_send_rich(msg.text) and await self._edit_rich_message(chat_id, state["message_id"], msg.text):
|
if self._can_send_rich(msg.text) and await self._edit_rich_message(chat_id, state["message_id"], msg.text):
|
||||||
self._last_bot_message[msg.chat_id] = state["message_id"]
|
self._last_bot_message[msg.chat_id] = state["message_id"]
|
||||||
return
|
return
|
||||||
await self._finalize_stream_message(chat_id, msg.chat_id, state, msg.text, msg)
|
await self._finalize_stream_message(chat_id, msg.chat_id, state, msg.text)
|
||||||
return
|
return
|
||||||
|
|
||||||
if self._can_send_rich(msg.text):
|
if self._can_send_rich(msg.text):
|
||||||
@ -350,51 +340,10 @@ class TelegramChannel(Channel):
|
|||||||
logger.warning("[Telegram] Rich Message send failed in chat=%s; falling back to plain text: %s", chat_id, exc)
|
logger.warning("[Telegram] Rich Message send failed in chat=%s; falling back to plain text: %s", chat_id, exc)
|
||||||
message_id = None
|
message_id = None
|
||||||
if message_id is not None:
|
if message_id is not None:
|
||||||
await self._remember_reply_thread(msg, message_id)
|
|
||||||
return
|
return
|
||||||
|
|
||||||
for chunk in self._split_message(msg.text):
|
for chunk in self._split_message(msg.text):
|
||||||
message_id = await self._send_new_message(chat_id, msg.chat_id, chunk, _max_retries=_max_retries)
|
await self._send_new_message(chat_id, msg.chat_id, chunk, _max_retries=_max_retries)
|
||||||
await self._remember_reply_thread(msg, message_id)
|
|
||||||
|
|
||||||
async def _remember_reply_thread(self, msg: OutboundMessage, message_id: int | None, *, stream_state: dict[str, Any] | None = None) -> bool:
|
|
||||||
"""Keep group replies on their source thread, without undoing a later /new."""
|
|
||||||
if not msg.thread_id or message_id is None or int(msg.chat_id) >= 0:
|
|
||||||
return True
|
|
||||||
store = self.config.get("channel_store")
|
|
||||||
if msg.connection_id:
|
|
||||||
# Bound conversations must never fall back to the shared legacy store.
|
|
||||||
if self._connection_repo is None or not msg.owner_user_id:
|
|
||||||
return False
|
|
||||||
elif store is None:
|
|
||||||
return False
|
|
||||||
topic_ids = {str(message_id)}
|
|
||||||
if msg.thread_ts:
|
|
||||||
topic_ids.add(msg.thread_ts)
|
|
||||||
try:
|
|
||||||
for topic_id in topic_ids:
|
|
||||||
if msg.connection_id:
|
|
||||||
await self._connection_repo.claim_thread_id(
|
|
||||||
connection_id=msg.connection_id,
|
|
||||||
owner_user_id=msg.owner_user_id,
|
|
||||||
provider=self.name,
|
|
||||||
external_conversation_id=msg.chat_id,
|
|
||||||
external_topic_id=topic_id,
|
|
||||||
thread_id=msg.thread_id,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
await store.claim_thread_id(self.name, msg.chat_id, msg.thread_id, topic_id=topic_id)
|
|
||||||
except Exception:
|
|
||||||
# Delivery has already succeeded. Retry the binding on the next update,
|
|
||||||
# not the Telegram send, which could duplicate a visible reply.
|
|
||||||
if stream_state is not None and stream_state.get("thread_binding_error_logged"):
|
|
||||||
logger.debug("[Telegram] reply thread mapping retry failed in chat=%s", msg.chat_id)
|
|
||||||
else:
|
|
||||||
logger.exception("[Telegram] failed to remember reply thread mapping in chat=%s", msg.chat_id)
|
|
||||||
if stream_state is not None:
|
|
||||||
stream_state["thread_binding_error_logged"] = True
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
async def _send_stream_update(self, chat_id: int, key: str, text: str, reply_to: int | None = None) -> None:
|
async def _send_stream_update(self, chat_id: int, key: str, text: str, reply_to: int | None = None) -> None:
|
||||||
"""Edit the in-flight streamed message with accumulated text.
|
"""Edit the in-flight streamed message with accumulated text.
|
||||||
@ -460,7 +409,7 @@ class TelegramChannel(Channel):
|
|||||||
state["last_edit_at"] = _monotonic()
|
state["last_edit_at"] = _monotonic()
|
||||||
state["last_text"] = display
|
state["last_text"] = display
|
||||||
|
|
||||||
async def _finalize_stream_message(self, chat_id: int, chat_key: str, state: dict[str, Any], text: str, msg: OutboundMessage) -> None:
|
async def _finalize_stream_message(self, chat_id: int, chat_key: str, state: dict[str, Any], text: str) -> None:
|
||||||
"""Apply the final text: edit the streamed message, splitting overflow into follow-ups."""
|
"""Apply the final text: edit the streamed message, splitting overflow into follow-ups."""
|
||||||
bot = self._application.bot
|
bot = self._application.bot
|
||||||
chunks = self._split_message(text or "")
|
chunks = self._split_message(text or "")
|
||||||
@ -474,12 +423,10 @@ class TelegramChannel(Channel):
|
|||||||
else:
|
else:
|
||||||
# Edit could not be applied (e.g. message deleted) — deliver the
|
# Edit could not be applied (e.g. message deleted) — deliver the
|
||||||
# first chunk as a fresh message with the standard retry policy.
|
# first chunk as a fresh message with the standard retry policy.
|
||||||
message_id = await self._send_new_message(chat_id, chat_key, chunks[0])
|
await self._send_new_message(chat_id, chat_key, chunks[0])
|
||||||
await self._remember_reply_thread(msg, message_id, stream_state=state)
|
|
||||||
|
|
||||||
for chunk in chunks[1:]:
|
for chunk in chunks[1:]:
|
||||||
message_id = await self._send_new_message(chat_id, chat_key, chunk)
|
await self._send_new_message(chat_id, chat_key, chunk)
|
||||||
await self._remember_reply_thread(msg, message_id, stream_state=state)
|
|
||||||
|
|
||||||
async def _edit_final_chunk(self, bot, chat_id: int, message_id: int, text: str) -> bool:
|
async def _edit_final_chunk(self, bot, chat_id: int, message_id: int, text: str) -> bool:
|
||||||
"""Edit with one rate-limit retry. Returns False if the edit could not be applied."""
|
"""Edit with one rate-limit retry. Returns False if the edit could not be applied."""
|
||||||
@ -598,24 +545,13 @@ class TelegramChannel(Channel):
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
input_file = await asyncio.to_thread(_load_telegram_input_file, attachment.actual_path, attachment.filename)
|
input_file = await asyncio.to_thread(_load_telegram_input_file, attachment.actual_path, attachment.filename)
|
||||||
from telegram.error import BadRequest
|
|
||||||
|
|
||||||
if attachment.is_image and attachment.size <= 10 * 1024 * 1024:
|
if attachment.is_image and attachment.size <= 10 * 1024 * 1024:
|
||||||
kwargs: dict[str, Any] = {"chat_id": chat_id, "photo": input_file}
|
kwargs: dict[str, Any] = {"chat_id": chat_id, "photo": input_file}
|
||||||
if reply_to:
|
if reply_to:
|
||||||
kwargs["reply_to_message_id"] = reply_to
|
kwargs["reply_to_message_id"] = reply_to
|
||||||
if chat_id > 0:
|
if chat_id > 0:
|
||||||
kwargs["allow_sending_without_reply"] = True
|
kwargs["allow_sending_without_reply"] = True
|
||||||
try:
|
|
||||||
sent = await bot.send_photo(**kwargs)
|
sent = await bot.send_photo(**kwargs)
|
||||||
except BadRequest as exc:
|
|
||||||
if str(exc).casefold() not in {"photo_invalid_dimensions", "image_process_failed"}:
|
|
||||||
raise
|
|
||||||
logger.info("[Telegram] photo rejected (%s); delivering as document: %s", exc, attachment.filename)
|
|
||||||
# A rejected photo can still be delivered unchanged as a file.
|
|
||||||
# Keep the reply target and reuse the already loaded bytes.
|
|
||||||
kwargs["document"] = kwargs.pop("photo")
|
|
||||||
sent = await bot.send_document(**kwargs)
|
|
||||||
else:
|
else:
|
||||||
kwargs = {"chat_id": chat_id, "document": input_file}
|
kwargs = {"chat_id": chat_id, "document": input_file}
|
||||||
if reply_to:
|
if reply_to:
|
||||||
@ -625,7 +561,6 @@ class TelegramChannel(Channel):
|
|||||||
sent = await bot.send_document(**kwargs)
|
sent = await bot.send_document(**kwargs)
|
||||||
|
|
||||||
self._last_bot_message[msg.chat_id] = sent.message_id
|
self._last_bot_message[msg.chat_id] = sent.message_id
|
||||||
await self._remember_reply_thread(msg, sent.message_id)
|
|
||||||
logger.info("[Telegram] file sent: %s to chat=%s", attachment.filename, msg.chat_id)
|
logger.info("[Telegram] file sent: %s to chat=%s", attachment.filename, msg.chat_id)
|
||||||
return True
|
return True
|
||||||
except Exception:
|
except Exception:
|
||||||
@ -982,24 +917,6 @@ class TelegramChannel(Channel):
|
|||||||
"last_text": last_text,
|
"last_text": last_text,
|
||||||
}
|
}
|
||||||
|
|
||||||
def _outbound_retry_delay(self, attempt: int, exc: Exception) -> float:
|
|
||||||
"""Extend the shared backoff for new plain or Rich final-reply sends.
|
|
||||||
|
|
||||||
Only sends through ``_send_with_retry`` use this policy. The running
|
|
||||||
acknowledgment and intermediate stream updates are best effort.
|
|
||||||
Final edits have their own retry path.
|
|
||||||
"""
|
|
||||||
delay = super()._outbound_retry_delay(attempt, exc)
|
|
||||||
if not self._is_retry_after(exc):
|
|
||||||
return delay
|
|
||||||
try:
|
|
||||||
hint = self._retry_after_seconds(exc)
|
|
||||||
except (TypeError, ValueError, OverflowError):
|
|
||||||
return delay
|
|
||||||
if not math.isfinite(hint):
|
|
||||||
return delay
|
|
||||||
return max(delay, min(hint, TELEGRAM_MAX_OUTBOUND_RETRY_DELAY_SECONDS))
|
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _is_retry_after(exc: Exception) -> bool:
|
def _is_retry_after(exc: Exception) -> bool:
|
||||||
return getattr(exc, "retry_after", None) is not None
|
return getattr(exc, "retry_after", None) is not None
|
||||||
@ -1041,7 +958,7 @@ class TelegramChannel(Channel):
|
|||||||
return chunks
|
return chunks
|
||||||
|
|
||||||
async def _send_running_reply(self, chat_id: str, reply_to_message_id: int) -> None:
|
async def _send_running_reply(self, chat_id: str, reply_to_message_id: int) -> None:
|
||||||
"""Send a best-effort acknowledgment before dispatching inbound work."""
|
"""Send a 'Working on it...' reply and register it as the stream target."""
|
||||||
if not self._application:
|
if not self._application:
|
||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
|
|||||||
@ -32,7 +32,6 @@ from app.channels.commands import is_known_channel_command
|
|||||||
from app.channels.connection_identity import attach_connection_identity
|
from app.channels.connection_identity import attach_connection_identity
|
||||||
from app.channels.credential_staging import cleanup_stale_credential_staging, credential_staging_prefix
|
from app.channels.credential_staging import cleanup_stale_credential_staging, credential_staging_prefix
|
||||||
from app.channels.message_bus import INBOUND_FILE_CLEANUP_KEY, InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment, cleanup_inbound_files
|
from app.channels.message_bus import INBOUND_FILE_CLEANUP_KEY, InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment, cleanup_inbound_files
|
||||||
from deerflow.utils.atomic_write import publish_bytes_atomically
|
|
||||||
from deerflow.utils.file_io import await_drained
|
from deerflow.utils.file_io import await_drained
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@ -1084,8 +1083,15 @@ class WechatChannel(Channel):
|
|||||||
def _update_longpoll_timeout(self, data: Mapping[str, Any]) -> None:
|
def _update_longpoll_timeout(self, data: Mapping[str, Any]) -> None:
|
||||||
if not self._respect_server_longpoll_timeout:
|
if not self._respect_server_longpoll_timeout:
|
||||||
return
|
return
|
||||||
timeout_ms = self._coerce_float(data.get("longpolling_timeout_ms"), 0.0)
|
raw_timeout = data.get("longpolling_timeout_ms")
|
||||||
if 1_000 <= timeout_ms <= 300_000:
|
if raw_timeout is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
timeout_ms = float(raw_timeout)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return
|
||||||
|
if timeout_ms <= 0:
|
||||||
|
return
|
||||||
self._server_longpoll_timeout_seconds = timeout_ms / 1000.0
|
self._server_longpoll_timeout_seconds = timeout_ms / 1000.0
|
||||||
|
|
||||||
def _base_info(self) -> dict[str, str]:
|
def _base_info(self) -> dict[str, str]:
|
||||||
@ -1701,10 +1707,7 @@ class WechatChannel(Channel):
|
|||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
self._cursor_path.parent.mkdir(parents=True, exist_ok=True)
|
self._cursor_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
# Publish through the shared helper: it fsyncs the staged bytes
|
self._cursor_path.write_text(json.dumps({"get_updates_buf": self._get_updates_buf}, ensure_ascii=False, indent=2), encoding="utf-8", newline="")
|
||||||
# before the rename (so a machine crash cannot leave a truncated
|
|
||||||
# cursor behind) and re-applies an existing file's mode.
|
|
||||||
publish_bytes_atomically(self._cursor_path, json.dumps({"get_updates_buf": self._get_updates_buf}, ensure_ascii=False, indent=2).encode("utf-8"))
|
|
||||||
except OSError:
|
except OSError:
|
||||||
logger.warning("[WeChat] failed to persist cursor state to %s", self._cursor_path)
|
logger.warning("[WeChat] failed to persist cursor state to %s", self._cursor_path)
|
||||||
|
|
||||||
|
|||||||
@ -9,7 +9,7 @@ from collections.abc import Awaitable, Callable, Mapping
|
|||||||
from typing import Any, cast
|
from typing import Any, cast
|
||||||
|
|
||||||
from app.channels.base import Channel, ChannelUnavailable, attachment_delivery_text
|
from app.channels.base import Channel, ChannelUnavailable, attachment_delivery_text
|
||||||
from app.channels.commands import is_known_channel_command, strip_leading_mentions
|
from app.channels.commands import is_known_channel_command
|
||||||
from app.channels.connection_identity import attach_connection_identity
|
from app.channels.connection_identity import attach_connection_identity
|
||||||
from app.channels.message_bus import (
|
from app.channels.message_bus import (
|
||||||
InboundMessage,
|
InboundMessage,
|
||||||
@ -40,10 +40,6 @@ else:
|
|||||||
_WECOM_ACK_REJECTION_PREFIX = "Reply ack error"
|
_WECOM_ACK_REJECTION_PREFIX = "Reply ack error"
|
||||||
|
|
||||||
|
|
||||||
def _group_workspace_id(aibotid: str | None, chatid: str) -> str:
|
|
||||||
return f"{aibotid}:group:{chatid}" if aibotid else f"group:{chatid}"
|
|
||||||
|
|
||||||
|
|
||||||
def _file_md5(path: str) -> str:
|
def _file_md5(path: str) -> str:
|
||||||
md5_hasher = hashlib.md5()
|
md5_hasher = hashlib.md5()
|
||||||
with open(path, "rb") as file_obj:
|
with open(path, "rb") as file_obj:
|
||||||
@ -438,6 +434,8 @@ class WeComChannel(Channel):
|
|||||||
logger.exception("[WeCom] failed to send attachment delivery feedback")
|
logger.exception("[WeCom] failed to send attachment delivery feedback")
|
||||||
|
|
||||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||||
|
if not msg.is_final:
|
||||||
|
return True
|
||||||
if not self._ws_client:
|
if not self._ws_client:
|
||||||
return False
|
return False
|
||||||
if not msg.thread_ts:
|
if not msg.thread_ts:
|
||||||
@ -447,11 +445,7 @@ class WeComChannel(Channel):
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
media_type = "image" if attachment.is_image else "file"
|
media_type = "image" if attachment.is_image else "file"
|
||||||
# aibot media upload limits: image 10 MB, voice 2 MB, video 10 MB,
|
size_limit = 2 * 1024 * 1024 if attachment.is_image else 20 * 1024 * 1024
|
||||||
# file 20 MB (https://developer.work.weixin.qq.com/document/path/101917).
|
|
||||||
# The image cap used to be 2 MB — the voice limit — which dropped every
|
|
||||||
# 2-10 MB image before it was uploaded.
|
|
||||||
size_limit = 10 * 1024 * 1024 if attachment.is_image else 20 * 1024 * 1024
|
|
||||||
if attachment.size > size_limit:
|
if attachment.size > size_limit:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"[WeCom] %s too large (%d bytes), skipping: %s",
|
"[WeCom] %s too large (%d bytes), skipping: %s",
|
||||||
@ -587,22 +581,7 @@ class WeComChannel(Channel):
|
|||||||
if handled:
|
if handled:
|
||||||
return
|
return
|
||||||
|
|
||||||
# WeCom group chats deliver "@bot /new" with the mention left in the text
|
inbound_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
|
||||||
# (Slack/Discord strip their own bot mention upstream). Skip a leading
|
|
||||||
# mention only for the command path so ordinary chat keeps any @mentions
|
|
||||||
# intact for the agent; the stripped form also flows into the inbound so
|
|
||||||
# ChannelManager._handle_command parses the bare command. Mirrors
|
|
||||||
# DingTalkChannel / FeishuChannel.
|
|
||||||
#
|
|
||||||
# This drops *any* leading mention, not only the bot's: WeCom text
|
|
||||||
# mentions carry display names, so the frame's `aibotid` cannot be
|
|
||||||
# matched against them and "@alice /status" is indistinguishable from
|
|
||||||
# "@bot /status". Feishu and DingTalk accept the same tradeoff;
|
|
||||||
# test_member_mention_before_a_command_also_classifies pins it.
|
|
||||||
command_text = strip_leading_mentions(text)
|
|
||||||
inbound_type = InboundMessageType.COMMAND if is_known_channel_command(command_text) else InboundMessageType.CHAT
|
|
||||||
if inbound_type is InboundMessageType.COMMAND:
|
|
||||||
text = command_text
|
|
||||||
inbound_metadata: dict[str, Any] = {
|
inbound_metadata: dict[str, Any] = {
|
||||||
"aibotid": body.get("aibotid"),
|
"aibotid": body.get("aibotid"),
|
||||||
"chattype": body.get("chattype"),
|
"chattype": body.get("chattype"),
|
||||||
@ -610,12 +589,8 @@ class WeComChannel(Channel):
|
|||||||
}
|
}
|
||||||
if body.get("chatid") is not None:
|
if body.get("chatid") is not None:
|
||||||
inbound_metadata["chatid"] = body.get("chatid")
|
inbound_metadata["chatid"] = body.get("chatid")
|
||||||
# A group's conversation is the group, a private chat's the sender.
|
|
||||||
# Keyed by sender alone, a member's private chat and every group they
|
|
||||||
# wrote in shared one thread wherever no group connection kept them apart.
|
|
||||||
group_chatid = str(body.get("chatid") or "").strip() if body.get("chattype") == "group" else ""
|
|
||||||
inbound = self._make_inbound(
|
inbound = self._make_inbound(
|
||||||
chat_id=group_chatid or user_id,
|
chat_id=user_id, # keep user's conversation in memory
|
||||||
user_id=user_id,
|
user_id=user_id,
|
||||||
text=text,
|
text=text,
|
||||||
msg_type=inbound_type,
|
msg_type=inbound_type,
|
||||||
@ -623,7 +598,7 @@ class WeComChannel(Channel):
|
|||||||
files=files or [],
|
files=files or [],
|
||||||
metadata=inbound_metadata,
|
metadata=inbound_metadata,
|
||||||
)
|
)
|
||||||
inbound.topic_id = user_id # one thread per member in each conversation
|
inbound.topic_id = user_id # keep the same thread
|
||||||
|
|
||||||
stream_id = generate_req_id("stream")
|
stream_id = generate_req_id("stream")
|
||||||
self._ws_frames[msg_id] = frame
|
self._ws_frames[msg_id] = frame
|
||||||
@ -641,8 +616,6 @@ class WeComChannel(Channel):
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
inbound = await self._attach_connection_identity(inbound)
|
inbound = await self._attach_connection_identity(inbound)
|
||||||
if group_chatid:
|
|
||||||
await self._carry_over_sender_keyed_group_thread(inbound, group_chatid)
|
|
||||||
self._commit_reserved_inbound(reservation, inbound)
|
self._commit_reserved_inbound(reservation, inbound)
|
||||||
finally:
|
finally:
|
||||||
reservation.release()
|
reservation.release()
|
||||||
@ -654,7 +627,7 @@ class WeComChannel(Channel):
|
|||||||
if chattype == "group":
|
if chattype == "group":
|
||||||
if not chatid or self._connection_repo is None:
|
if not chatid or self._connection_repo is None:
|
||||||
return inbound
|
return inbound
|
||||||
workspace_id = _group_workspace_id(aibotid, chatid)
|
workspace_id = f"{aibotid}:group:{chatid}" if aibotid else f"group:{chatid}"
|
||||||
inbound = await attach_connection_identity(inbound, repo=self._connection_repo, provider="wecom", workspace_id=workspace_id, fallback_without_workspace=False)
|
inbound = await attach_connection_identity(inbound, repo=self._connection_repo, provider="wecom", workspace_id=workspace_id, fallback_without_workspace=False)
|
||||||
if inbound.connection_id:
|
if inbound.connection_id:
|
||||||
return inbound
|
return inbound
|
||||||
@ -676,35 +649,6 @@ class WeComChannel(Channel):
|
|||||||
fallback_without_workspace=True,
|
fallback_without_workspace=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
async def _carry_over_sender_keyed_group_thread(self, inbound: InboundMessage, chatid: str) -> None:
|
|
||||||
"""Rebind a group connection's thread from the sender key it used to have.
|
|
||||||
|
|
||||||
Group messages were once keyed like private ones (``userid``, ``userid``).
|
|
||||||
A group connection belongs to one group, so its thread under that key is
|
|
||||||
this group's and moves to the group key once. A legacy group binding
|
|
||||||
spans every group of the bot; its old thread is left behind rather than
|
|
||||||
continued in whichever group writes first.
|
|
||||||
"""
|
|
||||||
repo = self._connection_repo
|
|
||||||
if repo is None or not inbound.connection_id or not inbound.owner_user_id:
|
|
||||||
return
|
|
||||||
aibotid = str(inbound.metadata.get("aibotid") or "").strip() or None
|
|
||||||
if inbound.workspace_id != _group_workspace_id(aibotid, chatid):
|
|
||||||
return
|
|
||||||
if await repo.get_thread_id(inbound.connection_id, inbound.chat_id, inbound.topic_id) is not None:
|
|
||||||
return
|
|
||||||
previous = await repo.get_thread_id(inbound.connection_id, inbound.user_id, inbound.user_id)
|
|
||||||
if previous is None:
|
|
||||||
return
|
|
||||||
await repo.claim_thread_id(
|
|
||||||
connection_id=inbound.connection_id,
|
|
||||||
owner_user_id=inbound.owner_user_id,
|
|
||||||
provider=self.name,
|
|
||||||
external_conversation_id=inbound.chat_id,
|
|
||||||
external_topic_id=inbound.topic_id,
|
|
||||||
thread_id=previous,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _bind_connection_from_connect_code(self, *, frame: dict[str, Any], user_id: str, code: str) -> bool:
|
async def _bind_connection_from_connect_code(self, *, frame: dict[str, Any], user_id: str, code: str) -> bool:
|
||||||
if self._connection_repo is None or not code:
|
if self._connection_repo is None or not code:
|
||||||
return False
|
return False
|
||||||
@ -727,7 +671,7 @@ class WeComChannel(Channel):
|
|||||||
if not chatid:
|
if not chatid:
|
||||||
await self._send_connection_reply(frame, await self._connect_reply_text("incomplete", owner_user_id=state["owner_user_id"]))
|
await self._send_connection_reply(frame, await self._connect_reply_text("incomplete", owner_user_id=state["owner_user_id"]))
|
||||||
return True
|
return True
|
||||||
workspace_id = _group_workspace_id(aibotid, chatid)
|
workspace_id = f"{aibotid}:group:{chatid}" if aibotid else f"group:{chatid}"
|
||||||
raw_group_name = str(body.get("chatname") or body.get("group_name") or "").strip()
|
raw_group_name = str(body.get("chatname") or body.get("group_name") or "").strip()
|
||||||
workspace_name = notification_group_display_name(state.get("metadata")) or raw_group_name[:256] or None
|
workspace_name = notification_group_display_name(state.get("metadata")) or raw_group_name[:256] or None
|
||||||
metadata: dict[str, Any] = {
|
metadata: dict[str, Any] = {
|
||||||
|
|||||||
@ -1,102 +1,264 @@
|
|||||||
# Gateway API (`app/gateway/`)
|
### Gateway API (`app/gateway/`)
|
||||||
|
|
||||||
The FastAPI application on port 8001. It serves the REST API, embeds the
|
Reject external `sandbox`, `thread_data`, `viewed_images`, `goal_outcome`. Non-null goals: valid text, int counters/limits >=0, no Overwrite; keep sparse data.
|
||||||
LangGraph-compatible agent runtime, and owns auth, CSRF, run admission and thread
|
|
||||||
lifecycle. Route-level rules are in [routers/AGENTS.md](routers/AGENTS.md). The
|
|
||||||
endpoint list is the OpenAPI schema at `/docs`; [API.md](../../docs/API.md)
|
|
||||||
describes it for humans.
|
|
||||||
|
|
||||||
## Verify
|
Studio retains sanitized creation metadata.
|
||||||
|
|
||||||
```bash
|
Capability Center's `business` adapter validates bundled credentials and creates
|
||||||
cd backend
|
an MCP connection. The MCP API permits only its isolated launcher and credential
|
||||||
python -m pytest tests -q -k "gateway or router or auth or run_manager"
|
environment keys; reject arbitrary commands and manifest-policy bypasses. Keep
|
||||||
make test-blocking-io # request handlers share one event loop
|
credentials out of tool schemas and preserve admin checks, masked atomic edits,
|
||||||
```
|
and MCP cache reloads.
|
||||||
|
|
||||||
Read these contracts before you change the matching area:
|
Memory shutdown resolves hot-reloaded config/backend, flushes and closes in one
|
||||||
[thread lifecycle](../../docs/THREAD_LIFECYCLE.md) (branch, regenerate, edit
|
`await_drained` operation. Resolve config off-loop inside the best-effort handler
|
||||||
replay, archive), [run event stream](../../docs/RUN_EVENT_STREAM.md),
|
so malformed edits cannot abort teardown. Cancellation waits for owned workers.
|
||||||
[run interaction policy](../../docs/RUN_INTERACTION_POLICY.md),
|
The host sets no close timeout: `close()` must be quick or internally bounded;
|
||||||
[auth design](../../docs/AUTH_DESIGN.md), [knowledge](../../docs/KNOWLEDGE.md).
|
budget it, flush timeout and other hooks within pod grace. Drain extension
|
||||||
|
`stop_services()` across cancellation before later resources unwind; each
|
||||||
|
service `stop()` has a 30-second `asyncio.timeout`.
|
||||||
|
|
||||||
## Trust boundary
|
Routers with persistent writes drain through
|
||||||
|
`app.gateway.persistent_writes.run_drained_write`.
|
||||||
|
|
||||||
- **Share tokens read only frozen snapshots.** See [sharing](../../../docs/guide/sharing.md).
|
`lifespan` first holds `hold_sandbox_shutdown_ownership()`. After
|
||||||
- **Server-owned run context.** A client can send run context in `body.context`
|
`langgraph_runtime` drains runs and extensions, close MCP (bounded, isolated),
|
||||||
and in `body.config`. Admit server-owned keys only from internally authenticated
|
then drain `shutdown_sandbox_provider()` off-loop in `finally`, even on
|
||||||
callers and strip them from both places for everyone else. When you add a
|
cancel. Never allocate for cleanup; log failures, stay retryable.
|
||||||
server-owned key, add it to that strip list in the same change.
|
|
||||||
- **Server-owned state.** Reject thread state fields that only the server may write
|
|
||||||
(sandbox, thread data, viewed images, goal outcome) and reserved metadata keys
|
|
||||||
when they come from a client. External run input cannot carry system or developer
|
|
||||||
messages.
|
|
||||||
- **Owner scoping.** Run-history helpers default to the request user and fail
|
|
||||||
closed when there is no user context. Pass `user_id=None` only in migration or
|
|
||||||
admin code that needs an unscoped read, and say why at the call site.
|
|
||||||
- **Thread IDs become paths.** Validate every caller-provided thread ID with
|
|
||||||
`deerflow.utils.thread_id` before it reaches persistence or the filesystem.
|
|
||||||
Legacy IDs stay readable and deletable, but they are never interpolated into a
|
|
||||||
host path.
|
|
||||||
- **Shared decisions.** Route permissions, model use and skill visibility are each
|
|
||||||
decided in one helper in `authz.py`. HTTP routes, decorator-only auth and the
|
|
||||||
Live Browser WebSocket all call it. A new entry point calls the same helper so
|
|
||||||
that it cannot drift from the others.
|
|
||||||
- **Tokens and cookies.** A request with an invalid Bearer token gets 401 and never
|
|
||||||
falls back to the session cookie; the CSRF middleware skips Bearer requests and
|
|
||||||
relies on this. Personal access tokens work only on allowlisted routes.
|
|
||||||
- **Proxy headers.** "Is this localhost" decisions use the direct `Host` header.
|
|
||||||
Forwarded headers are client-controlled unless the outer proxy replaces them.
|
|
||||||
- **Untrusted remote text.** MCP notifications and other remote events reach the
|
|
||||||
model framed as untrusted data, outside the trusted instruction.
|
|
||||||
|
|
||||||
## Runs and threads
|
Retention holds durable thread admission and the checkpoint lock before
|
||||||
|
checkpoint reads. Inspect root head/parent and leaf writes only; protect selectors,
|
||||||
|
require equal channel versions and invalidate caches before deletion. No scans,
|
||||||
|
stats or blob GC; failures stay non-fatal.
|
||||||
|
|
||||||
- **One writer per thread.** A run, a checkpoint write, an artifact write, a branch
|
`conversation_access.py` grants the opt-in read-only tool only for explicit
|
||||||
and a delete are all thread operations that share one durable uniqueness
|
`conversation_references` with effective `runs:read`; never infer grants from
|
||||||
constraint. Reserve through `RunManager.reserve_thread_operation()` or
|
messages or checkpoints. Pass the callback through `RunContext`, not serialized
|
||||||
`services.reserve_checkpoint_write()`. A separate lock would let two workers
|
config, and let the worker inject and clear it. `conversation_reader.py` shares
|
||||||
write the same thread. Outside run execution, checkpoint mutations — manual
|
HTTP transcript visibility/pagination without a Request; callers own authorization.
|
||||||
compaction, `POST /threads/{id}/state`, both goal routes (`PUT` / `DELETE
|
Exclude non-text/internal content. `RunCreateRequest` lifts SDK-compatible
|
||||||
/threads/{id}/goal`), and the `/goal from-chat` save (`POST
|
`context.conversation_references` into its top-level field before validation,
|
||||||
/threads/{id}/goal/from-chat`) — use `services.reserve_checkpoint_write()`.
|
then removes it from context; supplying both is 422. This shares bounds/error
|
||||||
`PUT /goal` creates a missing checkpoint; from-chat does not. From-chat
|
locations and keeps `merge_run_context_overrides` clear. Admission and
|
||||||
saves with `expected_checkpoint_id` pinned to the head its synthesis read:
|
`/api/features` use `conversation_references_enabled()` as the sole configured
|
||||||
a write that landed during the model call rejects the stale objective
|
predicate.
|
||||||
instead of clobbering the newer state, and the model call itself passes the
|
|
||||||
same `authorize_model_use` admission as the suggestions route.
|
|
||||||
- **Several workers.** In-memory run records exist only on the worker that owns the
|
|
||||||
run. Cancel, takeover and orphan recovery are compare-and-set operations on the
|
|
||||||
run store; use the store's claim and cancel primitives so that a lease renewal
|
|
||||||
between your read and your write is not overwritten. Multi-worker deployments
|
|
||||||
need `run_events.backend: db`.
|
|
||||||
- **Idempotency keys.** Scope an external `Idempotency-Key` to its owner and thread
|
|
||||||
before it reaches `RunManager`; the manager's index is process-wide.
|
|
||||||
- **`/wait` on a reused run** returns the durable run status and error. The latest
|
|
||||||
thread checkpoint can belong to a different run.
|
|
||||||
- **Stream gaps.** Stream bridges keep a bounded buffer. A client that asks for
|
|
||||||
events older than the buffer receives a `gap` event and the run continues.
|
|
||||||
- **`Content-Location` must stay CORS-exposed.** Run-creating routes return the run
|
|
||||||
ID in that header and the LangGraph SDK reads it from there only.
|
|
||||||
- **Run event contract.** Producer code, `deerflow/constants.py`, the event
|
|
||||||
catalog, `contracts/run_event_stream_contract.json` and
|
|
||||||
`docs/RUN_EVENT_STREAM.md` change together.
|
|
||||||
- **Thread membership in a project** is written by thread creation, branching and
|
|
||||||
the explicit move route. Run admission reads it and never changes it.
|
|
||||||
|
|
||||||
## Lifecycle
|
FastAPI listens on 8001. `GET /health` is liveness; `GET /health/ready` concurrently
|
||||||
|
probes startup snapshots of ORM, checkpointer/Store and stream bridge
|
||||||
|
(`StreamBridge.ping()`) under one deadline and per-kind gates. Unreachable or
|
||||||
|
unresolved gating backends return 503; `memory` reports `not_configured`;
|
||||||
|
provisioner `/health` is report-only. See `health.py`. `GATEWAY_ENABLE_DOCS=false`
|
||||||
|
disables `/docs`, `/redoc` and `/openapi.json`.
|
||||||
|
|
||||||
- **Probe file-backed managed subagent storage once during startup, off the
|
`build_run_config()` resolves the default LangGraph super-step budget from the
|
||||||
event loop.** Probe failures warn operators before their first create request;
|
hot-reloaded top-level `recursion_limit` setting. A valid request-level value
|
||||||
atomic create stays fail-closed rather than falling back to an overwrite.
|
takes precedence; invalid values fall back to that default, and
|
||||||
- **Shutdown order matters.** `lifespan` drains runs and extensions first, then
|
`max_recursion_limit` caps both sources.
|
||||||
closes MCP, then shuts the sandbox provider down off the event loop. Cleanup code
|
|
||||||
keeps running after cancellation, allocates nothing new, and logs failures
|
Durable MCP notifications use internal Agent runs. Keep their trusted delivery instruction outside the user-input boundary, and frame serialized remote events as untrusted before model invocation. Strict thread existence/ownership admission dead-letters events whose task outlives its deleted chat instead of recreating the thread. `start_run` marks run-manager admission conflicts with `BusyThreadConflict`; the notification launcher retries only that typed 409 and treats other 409 responses as permanent.
|
||||||
instead of raising.
|
|
||||||
- **Persistent writes in routers** go through
|
CORS defaults to same-origin behind nginx on 2026. Split-origin and port-forwarded
|
||||||
`app.gateway.persistent_writes.run_drained_write` so that a client disconnect
|
browser clients must set exact `GATEWAY_CORS_ORIGINS`, shared by `CORSMiddleware`
|
||||||
cannot leave a half-written file.
|
and `CSRFMiddleware`. Expose `Content-Location` through `CORS_EXPOSED_HEADERS`
|
||||||
- **LangGraph Studio.** `langgraph_studio.py` is loaded by LangGraph's file loader
|
(`csrf_middleware.py`): run-creating routes put the run ID there, and the LangGraph
|
||||||
before the runtime starts and without a `sys.modules` entry. Keep its
|
SDK uses only that header for run metadata. Without exposure, JS cannot read it,
|
||||||
annotations eager, and identify Studio callers by the upstream `StudioUser`
|
breaking `useStream.onCreated` and thread-gated actions.
|
||||||
type, because the identity string can be reused by any client.
|
|
||||||
|
Browser sessions belong to `app.gateway.auth.session_cookie`. Login's `remember_me`
|
||||||
|
flag never stores passwords. `SessionCookiePolicy` persists `HttpOnly access_token`
|
||||||
|
only for HTTPS/trusted-forwarded HTTPS, direct-host localhost HTTP, or explicit
|
||||||
|
operator opt-in; public HTTP otherwise uses session cookies. Session creators set
|
||||||
|
final `max_age` on `request.state`; CSRF mirrors it across login, password changes
|
||||||
|
and OIDC callbacks. A small `HttpOnly` preference cookie preserves the choice.
|
||||||
|
Logout clears auth cookies and suppresses CSRF re-issue.
|
||||||
|
|
||||||
|
OIDC redirects retain existing endpoint query bytes (RFC 6749 section 3.1).
|
||||||
|
Malformed IdP bodies raise `OIDCError` (502/`sso_failed`).
|
||||||
|
`tests/test_oidc_auth.py` pins query, state-cookie, nonce and PKCE agreement.
|
||||||
|
|
||||||
|
Personal Access Tokens (`app.gateway.auth.pat`, `Authorization: Bearer dfp_...`) run as their owning user: an invalid Bearer is a hard 401 with no cookie fallback, which keeps `CSRFMiddleware`'s Bearer skip safe (origin checks still run). Scopes narrow within the allowlisted threads/runs/projects routes (including `POST /api/threads/{id}/move`); every other authenticated route 403s PAT callers (admin included). PAT management and `/change-password` require session auth; only SHA-256 digests are stored (`0017`).
|
||||||
|
|
||||||
|
Thread→project membership is written by thread creation (`POST /api/threads` with
|
||||||
|
a validated `project_id`), branch creation (the new row inherits the source
|
||||||
|
thread's project; an archived/deleted project degrades the branch to unassigned
|
||||||
|
instead of failing), and explicit moves (`POST /api/threads/{id}/move`); run
|
||||||
|
admission never modifies membership; it only pins the resolved project context
|
||||||
|
read-only into the run context. The server-reserved `deerflow_project_id`
|
||||||
|
metadata key is a read-only exposure of the `threads_meta.project_id` column and
|
||||||
|
is stripped from client writes.
|
||||||
|
|
||||||
|
Localhost persistence deliberately reads the direct request `Host` and ignores `Forwarded` / `X-Forwarded-Host`. Scheme and auth-origin reconstruction still consume forwarding headers. The bundled nginx sets `X-Forwarded-Proto`, but preserves an upstream HTTPS value and does not overwrite every forwarded header, so the outer trusted proxy must replace or strip client-supplied forwarding headers before traffic reaches DeerFlow.
|
||||||
|
|
||||||
|
Standalone local LangGraph Studio is recognized only through the upstream
|
||||||
|
`Auth.types.StudioUser` principal type, never by its reusable identity string.
|
||||||
|
The type is resolved once at import; an older SDK without it degrades to normal
|
||||||
|
owner scoping.
|
||||||
|
For that principal's assistant reads/searches, `langgraph_auth.add_owner_filter`
|
||||||
|
selects genuine server-registered assistants plus assistants owned by Studio;
|
||||||
|
all other resources remain owner-scoped. Assistant create/update handlers make
|
||||||
|
both `user_id` and `created_by=user` server-owned, because LangGraph gives
|
||||||
|
`created_by=system` privileged ownership semantics during run creation. The
|
||||||
|
custom application module in `langgraph_studio.py` is imported before the
|
||||||
|
locked in-memory runtime lifespan. At that pre-runtime boundary it derives
|
||||||
|
genuine system assistant IDs from the CLI-provided graph registry, removes
|
||||||
|
their persisted active/version rows so graph registration recreates them, and
|
||||||
|
demotes every other legacy `created_by=system` marker in both active assistants
|
||||||
|
and version history. This must happen before runtime 0.30.0 loads and purges
|
||||||
|
system-marked rows; a user application lifespan is too late. LangGraph executes
|
||||||
|
this file-backed custom app without first registering its module in
|
||||||
|
`sys.modules`; keep its annotations eager so dataclass processing remains
|
||||||
|
compatible with that loader, and preserve the direct file-loader regression
|
||||||
|
test. An empty graph registry or absent persistence file is a no-op, while
|
||||||
|
persistence parse/write errors fail startup closed. The harness requires
|
||||||
|
in-memory runtime 0.30.0 or newer, and a persisted store containing no
|
||||||
|
expected registered assistant row emits a drift warning so changes to
|
||||||
|
LangGraph's internal persistence contract are observable. With current
|
||||||
|
create/update writes and all legacy versions sanitized, ordinary
|
||||||
|
owner-scoped assistant version selection remains enabled.
|
||||||
|
|
||||||
|
**Routers**:
|
||||||
|
|
||||||
|
| Router | Endpoints |
|
||||||
|
|--------|-----------|
|
||||||
|
| **Managed Models** (`/api/managed-models`) | Admin-only GET/PUT with revisions; streaming POST `/test` (DeepSeek thinking off). Clients/storage off-loop; YAML names reserved; keys private. |
|
||||||
|
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||||||
|
| **Features** (`/api/features`) | `GET /` - UI capabilities: hot-reloaded agents, guarded browser, startup MCP tasks, separate batch repository/worker states so history stays readable without a worker, `conversation_references` (`read_conversation` configured, per-run reference cap), knowledge scope selection, `scheduled_tasks`, and `thread_activity` |
|
||||||
|
| **Thread Activity** (`/api/thread-activity`) | `GET /?cursor=` - per-user feed over `runs.change_seq`: no cursor seeds; lists threads of the caller's server-originated runs, passes interactive rows; `read_version`. `POST /api/threads/{id}/read` and search `unread` use `thread_read_markers`. SQL only |
|
||||||
|
| **Knowledge** (`/api/knowledge`) | Authenticated, capability-driven read-only retrieval catalog and per-assistant `/capabilities`; provider adapters own resource authorization and mapping. Unsupported filters fail admission/execution without widening scope; management stays outside core |
|
||||||
|
| **Console** (`/api/console`) | Read-only cross-thread observability for the current user: `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly (no new `RunStore` methods); SQL only (503 on `database.backend: memory`). Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Mixed currencies null cost/currency fields. Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
|
||||||
|
| **MCP** (`/api/mcp`) | GET /config - raw/masked; PUT /config - bulk; PATCH /config - toggle; POST /config/servers - add; PUT /config/server - replace; DELETE /config/servers/{server_name:path} - bodyless. POST /cache/reset. Validate expanded, save raw; reload/reset; invalid -> 400. |
|
||||||
|
| **MCP Tasks** (`/api/threads/{id}/mcp-tasks`) | `GET /` - current user's durable tasks for one owned thread; `GET /{task_id}` - bounded result/input/status-error/cancellation-error detail, including cancellation attempt count, without remote task IDs or driver configuration |
|
||||||
|
| **Skills** (`/api/skills`) | `GET /` - list; `GET /{name}` - inspect; `PUT /{name}` - toggle; `POST /install` - install a thread-local .skill archive; `POST /install/upload` - admin-only multipart, authorized before parsing and capped at a 100 MiB file plus 1 MiB framing; `POST /reload` - refresh, then publish the shared skills reset marker |
|
||||||
|
| **Subagents** (`/api/subagents`) | Admin managed-worker CRUD and listing. |
|
||||||
|
| **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_probed` / `sandbox_runtime_ready` (whether `lark-cli` will be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/config/credentials` - atomically switch the caller's per-user Lark app after validating the new `app_id`/`app_secret` through the official CLI's live tenant-token probe, revoke/remove the previous OAuth tokens, and restore the prior credential tree if the switch fails; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants. Config and auth flows carry a server-issued, per-user generation persisted under the credential lock; a rejected direct switch leaves the current generation unchanged, stale completions return 409, and browser re-registration uses the same token-clearing/revocation transaction as direct credential switches. |
|
||||||
|
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||||
|
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload/convert PDF/PPT/Excel/Word; upload/shelf names share `normalize_filename` before staging; non-mounted sync lease ends when its last holder exits; `GET /list`; `DELETE /{filename}` - regular files, symlinks 404, keeps converted `.md` (#5672) |
|
||||||
|
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - branch a completed assistant turn with a replay checkpoint; inherited titles take next-free displayed sibling suffixes, including explicit/renamed ones, while explicit titles stay unchanged. Durable `branch` admission rejects races. Workspace files are not checkpointed: only a branch from the **latest** turn best-effort copies the workspace (`workspace_clone_mode="current_thread_best_effort"`); older turns skip it (`"skipped_historical_turn"`), so no later-timeline file leaks in. `sandbox`/`thread_data` channels are not copied (parent `sandbox_id` binds paths/release); the branch lazily acquires its own sandbox. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows get one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn at every persisted human message, including an allowlisted hidden `ask_clarification` reply): `run_id` is a turn identity, and regenerating an inherited answer supersedes its whole `run_id` in `GET /messages/page`, so one shared seed id deleted all inherited history (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - summarize older active context, deriving memory policy and bucket from the state-producing checkpoint rather than request `agent_name`, and block while a run is in flight; 409 unless idle; unexpected failures return a generic 500 detail |
|
||||||
|
| **Artifacts** (`/api/threads/{id}/artifacts`) | GET streams byte ranges; HTML/XML/XSL/`+xml` (also `.skill` members) and `?download=true` force attachments. PUT atomically replaces existing UTF-8 outputs on matching SHA-256; active runs conflict. Non-mounted sync uses a request lease and drains remote/local commit-or-rollback across cancellation before releasing reservations/leases. `resolve_outputs_confined_path` normalizes `..` and confines host paths against traversal/symlinks. Preserve POSIX ownership/mode where supported; Windows keeps temp-file permissions. Digest cache: path/dev/ino/ctime_ns/mtime_ns/size; identity invalidates same-size/mtime replacements. Large regular files use stat-prefixed ETags; If-Range requires matching ETags. Windows ctime is creation time. Keep cache reuse, size limits and worker hashing. |
|
||||||
|
| **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
|
||||||
|
| **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |
|
||||||
|
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block. Before the first journaled run, seed an empty feed from a checkpoint so legacy checkpoint-only history keeps its order and visibility; skip absent checkpoints or populated feeds. `POST /regenerate/prepare` - prepare clean input + checkpoint metadata for regenerating the latest completed or interrupted assistant answer, treating a protocol-valid hidden `human_input_response` as confirmed replay input while rejecting other hidden/control humans, preserving that response's hidden/correlation metadata, and carrying the latest non-empty thread title in graph input so resuming an older checkpoint cannot roll back a later manual rename (#4457); `POST /edit-regenerate/prepare` - prepare a checkpoint replay from the latest editable human turn with a replacement user message and edit replay metadata; it carries the current thread title the same way, but only when the replay base already has one — an untitled base belongs to a thread the title middleware has not named yet, so pinning the current title there would keep a name generated from the prompt the edit just replaced; `GET /` - newest 100 runs as an array; `GET /page` - keyset history page `{data, has_more, next_before_created_at, next_before_run_id}`; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/stream` hides action/wait; GET action 405 pre-owner; POST needs `runs:cancel`; `GET /{rid}/messages` - paginated per-run messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /{rid}/workspace-changes` - workspace/output file change summary and optional diffs; `GET/POST /{rid}/artifacts/archive` - receipt manifest / bounded ZIP; `GET /../messages` - legacy thread message array; `GET /../messages/page` - backward thread-global `seq` history page with middleware/subagent-AI/successful-regenerate/edit-replay filtering and page-run-scoped feedback enrichment; subagent AI callbacks remain available through run events while parent `task` ToolMessages stay visible for card restoration; `GET /../token-usage` - aggregate tokens plus an optional `context_usage` percentage. Context usage approximately counts messages from the latest materialized thread state through `build_thread_checkpoint_state_accessor`, so full and delta checkpoint modes expose the same input. The percentage uses the latest run's model and its `context_window`. |
|
||||||
|
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||||
|
| **Runs** (`/api/runs`) | `POST /stream`, `/wait` - stateless runs requiring `runs:create`; optional body `thread_id` is owner-checked. Scheduled-task create/update/resume/trigger also require `threads:write` plus `runs:create`. `GET /{rid}/messages`, `/feedback` - run messages/feedback |
|
||||||
|
| **GitHub Webhooks** (`/api/webhooks/github`) | `POST /` - receive GitHub App / repo webhook deliveries. Verifies `X-Hub-Signature-256` against `GITHUB_WEBHOOK_SECRET`; exempt from auth + CSRF because authenticity is enforced by HMAC. The route is fail-closed: mounted only when `GITHUB_WEBHOOK_SECRET` is set, or when explicit dev opt-in `DEER_FLOW_ALLOW_UNVERIFIED_GITHUB_WEBHOOKS=1` is set. Recognized events include `ping`, `issues`, `issue_comment`, `pull_request`, `pull_request_review`, and `pull_request_review_comment`; unknown events return 200 with `handled=false`. Fan-out runtime failures return 503, keeping the delivery recorded as failed for manual/API/scripted redelivery (GitHub does not automatically retry any failed delivery, 5xx included); permanent/non-retryable conditions such as `channels.github.enabled: false`, unknown events, malformed payloads, or unavailable channel service return 200 with a skipped/handled response. |
|
||||||
|
| **GitHub Event-Driven Agents** | Custom agents can declare a `github:` block in their `config.yaml` to bind to repos and event triggers. Webhook fan-out publishes one `InboundMessage` per matching binding to the channel bus; `GitHubChannel` routes those messages through `ChannelManager`. The response `dispatch` summarizes matched/fired/skipped agents. |
|
||||||
|
|
||||||
|
Edit replay and messages-input clarification may replace knowledge scope with a
|
||||||
|
validated client snapshot. Regenerate, null-input and `Command` resumes recover
|
||||||
|
scope from the genuine source user turn, skip hidden server context, and ignore
|
||||||
|
client scope. Treat `input.messages: null` as empty to allow promptless
|
||||||
|
conversation-reference runs.
|
||||||
|
|
||||||
|
Thread identifiers use the shared `deerflow.utils.thread_id` contract
|
||||||
|
`^[A-Za-z0-9_-]{1,64}$`. Caller-provided opaque IDs remain supported; UUIDs
|
||||||
|
are generated only for `None`, while explicit empty strings fail validation.
|
||||||
|
Gateway creation and state-producing request boundaries, embedded-client
|
||||||
|
entry points, filesystem/upload/event-store consumers, scheduled launches,
|
||||||
|
and the standalone Provisioner enforce the same contract before persistence
|
||||||
|
or workspace initialization. Route-addressable legacy IDs remain accepted by
|
||||||
|
pure reads and cleanup/control endpoints; deleting one best-effort removes
|
||||||
|
metadata and checkpoints but skips local filesystem cleanup, so the raw value
|
||||||
|
is never interpolated into a host path. New runs, workspace/sandbox
|
||||||
|
operations, and other state-producing mutations remain blocked.
|
||||||
|
|
||||||
|
**Message feed seq** (#4666): streaming `values` frames, `GET
|
||||||
|
/threads/{id}/state`, and `POST /threads/{id}/history` stamp serialized
|
||||||
|
messages with `additional_kwargs.deerflow_seq` so clients can place
|
||||||
|
checkpoint-kept messages against the paged feed; the REST reads resolve the
|
||||||
|
store via `threads.py::_optional_run_event_store` (a feed-less deployment
|
||||||
|
still reads threads), and `services.py::normalize_input` strips the
|
||||||
|
server-owned key from client input (#4380). Mechanism and identity rule:
|
||||||
|
`packages/harness/deerflow/runtime/AGENTS.md`.
|
||||||
|
|
||||||
|
**Workspace change review**: `packages/harness/deerflow/workspace_changes/`
|
||||||
|
captures a pre-run and post-run snapshot of the thread-owned `workspace` and
|
||||||
|
`outputs` directories. `runtime/runs/worker.py` performs the filesystem scan via
|
||||||
|
`asyncio.to_thread` and writes a `workspace_changes` event with category
|
||||||
|
`workspace` when changes exist. Uploads are intentionally excluded. Text diffs
|
||||||
|
are size-limited; binary, large, and sensitive-looking paths are persisted as
|
||||||
|
metadata only. Internal process-feedback directories never count as changes:
|
||||||
|
the scanner's `EXCLUDED_DIR_NAMES` drops `BROWSER_FRAMES_DIRNAME` (transient
|
||||||
|
browser screenshots) and `TOOL_RESULTS_DIRNAME` (the tool-output budget
|
||||||
|
middleware's default externalization subdir, `constants.py` is the shared
|
||||||
|
source of truth for both writers and the scanner), and the worker threads the
|
||||||
|
configured `tool_output.storage_subdir` through the snapshot capture as an
|
||||||
|
extra excluded dir name so custom storage locations stay excluded too.
|
||||||
|
|
||||||
|
**Run delivery receipts**: the worker derives delivery requirements from the
|
||||||
|
run's workspace snapshots rather than a client request option (files
|
||||||
|
created/modified under `/mnt/user-data/outputs`, minus internal
|
||||||
|
process-feedback exclusions) and idempotently persists a run-scoped
|
||||||
|
`run.delivery` receipt before the terminal run status; missing or
|
||||||
|
unverifiable `present_files` coverage downgrades the run to error, while runs
|
||||||
|
without changed outputs keep ordinary chat behavior. Journal mechanics
|
||||||
|
(callback attribution, receipt idempotency and retries, orphan recovery):
|
||||||
|
`packages/harness/deerflow/runtime/AGENTS.md`. Multi-worker deployments
|
||||||
|
require `run_events.backend: db` for shared, ordered delivery events; the
|
||||||
|
startup gate rejects memory/JSONL event stores when `GATEWAY_WORKERS > 1`
|
||||||
|
or `deployment.multi_instance: true`.
|
||||||
|
|
||||||
|
**RunManager / RunStore contract**:
|
||||||
|
- LangGraph-compatible run requests validate their supported subset before creating a run. `runtime/stream_modes.py` is the shared backend contract for public stream modes and the worker's `graph.astream` mapping; the public `messages-tuple` mode maps to LangGraph's internal `messages` mode, while public `messages`, `events`, and other unsupported modes are rejected instead of being dropped or replaced with `values`. `app/gateway/run_models.py::RunCreateRequest` is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (`if_not_exists="create"` plus `None` placeholders), returns 422 for unsupported values including `on_completion="complete"`, `on_completion="continue"`, and `multitask_strategy="enqueue"`, and forbids undeclared SDK options so fields such as `checkpoint_during` and `durability` cannot be silently discarded. A placeholder must still accept the stock SDK's own default: `langgraph_sdk` drops only `None` from its run payload, so `stream_resumable=False` reaches every request and means "non-resumable", which is what DeerFlow serves — rejecting it 422'd every IM channel run (#4466). `test_run_request_validation.py::test_gateway_accepts_langgraph_sdk_default_payload` pins the real SDK payload; channel tests mock the SDK and miss this drift.
|
||||||
|
- `RunManager.get()` is async; direct callers must `await` it.
|
||||||
|
- The history batch helpers `list_successful_regenerate_sources()`, `list_edit_regenerate_runs()`, and `get_many_by_thread()` default to `user_id=AUTO`: they resolve the request user and fail closed when no user context exists. Migration/admin callers that intentionally need an unscoped read must pass `user_id=None` explicitly.
|
||||||
|
- Edit-and-rerun visibility is derived from edit replay runs (`metadata.replay_kind="edit"` plus `regenerate_from_run_id`) by `RunManager.list_edit_replay_visibility()`: the newest attempt for each source run is authoritative. Pending/running/success attempts hide the original source run; failed, timed-out, or interrupted attempts hide only the failed attempt so the original conversation reappears.
|
||||||
|
- When a persistent `RunStore` is configured, `get()` and `list_by_thread()` hydrate historical runs from the store. In-memory records win for the same `run_id` so task, abort, and stream-control state stays attached to active local runs.
|
||||||
|
- Thread metadata status switches to `running` only after `RunManager.try_start()` succeeds. Pending-cancelled runs therefore skip the old `running` projection, while clients may observe the prior thread status during the short worker-startup window.
|
||||||
|
- `cancel()` returns a :class:`~deerflow.runtime.CancelOutcome` enum: `cancelled` (local cancel), `requested` (the non-owning worker durably recorded the first cancellation action for the live owner), `taken_over` (non-owning worker claimed the run because the owner's lease expired — marks it as `error`), `lease_valid_elsewhere` (legacy/custom store lacks the durable request primitive — caller retains the safe 409 + `Retry-After` fallback), `not_active_locally` (heartbeat disabled, preserving the old 409 path), `not_cancellable` (terminal state), or `unknown` (not found in memory or store). `create_or_reject(..., multitask_strategy="interrupt"|"rollback")` persists interrupted status through `RunStore.update_status()`, matching normal `set_status()` transitions.
|
||||||
|
- Interrupt/rollback admission registers the replacement before its best-effort persistence of locally interrupted predecessors. If the admitting caller is cancelled during that post-registration await, `RunManager` drains a shielded replacement cleanup before propagating `CancelledError`, including across repeated cancellation. The cleanup normally persists `interrupted`; if that best-effort transition fails, it retries the active-to-`interrupted` store transition strictly and verifies the result with the replacement's captured owner identity. A concurrent peer terminal transition wins and is synchronized back into the local record rather than being overwritten or deleted.
|
||||||
|
- Store-only hydrated runs are readable history. In multi-worker mode with heartbeat enabled, cancel on a store-only run records `runs.cancel_action` / `cancel_requested_at` while the owner's lease is live; the first action wins even if a retry later lands on the owner. `RunStore.request_cancel()` and owner completion through `finalize_if_not_cancelled()` are competing active-row CAS operations, so an accepted cancel cannot be overwritten by a later success. `RunStore.renew_lease()` renews and observes the request atomically in the SQL implementation. The owner then executes the normal process-local interrupt/rollback and terminal stream path without transferring the lease. An expired owner is taken over with the `orphan_recovered` stop reason so cross-worker consumers can synthesize END. `wait=true` and cancel-then-stream use the shared bridge to observe owner finalization; a non-standard process-local bridge returns accepted 202 instead of subscribing to an unreachable stream. In single-worker mode (heartbeat off), store-only runs still return 409.
|
||||||
|
- A local worker's `RunRecord.lease_expires_at` is the last durably confirmed ownership deadline. `_renew_leases()` bounds each renewal attempt by that deadline: transient store exceptions remain retryable while it is valid, but an exception or blocked call that reaches expiry sets the process-local `ownership_lost` fence, raises `abort_event`, and cancels the run task. Successful renewals collect durable cancellation actions; after all local renewals have been attempted, heartbeat only signals the corresponding process-local tasks, leaving status writes and rollback cleanup to the worker finalization path. Fenced workers do not perform subsequent journal/delivery-receipt, progress/completion/status, checkpoint/thread-metadata, or `on_run_completed` writes; the peer recovery path owns the terminal receipt. `RunStore.update_run_completion()` also refuses to replace a different terminal status, closing the peer-takeover/late-finalization race. `grace_seconds` delays peer reclamation for clock skew but is not extra execution time for an owner that can no longer confirm its lease. Already-committed remote tool side effects remain outside this local cancellation boundary.
|
||||||
|
- Startup/orphan reconciliation must claim stale active rows with `RunStore.claim_for_takeover()`, not a plain `update_status()`. The final claim re-checks `status` and lease expiry atomically, so a heartbeat renewal between the candidate scan and the recovery write keeps the run active.
|
||||||
|
- Run admission and independent writes are first-class thread operations. `runs.operation_kind` distinguishes `run` from `checkpoint_write`, `artifact_write`, `artifact_archive`, `branch`, and `delete`; every active kind shares the durable active-thread uniqueness constraint. New operation kinds must go through `RunStore.create_thread_operation_atomic()` and `RunManager.reserve_thread_operation()` rather than adding another lock or metadata marker. Live and lease-less reservations are non-interruptible; an expired leased reservation can be reclaimed immediately by interrupt/rollback admission without waiting for orphan reconciliation. Lease-less rows stay fail-closed because the store cannot distinguish a stale row from a live writer in another heartbeat-disabled worker; a rare failed delete therefore requires startup reconciliation, and heartbeat-disabled multi-worker deployment remains unsupported. Reservation bodies are attached to their caller task so loss detected by lease renewal cancels the writer before it can continue after takeover; the context manager translates that lease-loss cancellation to `ConflictError` after cleanup so Gateway mutation routes return a retryable 409 instead of dropping the HTTP request. The cleanup scope begins immediately after durable admission, including the await that attaches the caller task, so cancellation cannot strand a locally renewed pending reservation. A failed renewal is revalidated under the manager lock before cancellation; if the reservation completed and unregistered while the store update was in flight, its request task must not be cancelled after the write. Reservations are excluded from run history/reporting and from run-only helpers such as `list_by_thread()` and `has_inflight()`, release uses the captured owner rather than ambient user context, and local cleanup still runs when the best-effort store delete fails. `RunStore.create_run_atomic()` remains a deprecated compatibility shim for external stores that only admit normal runs; new stores must implement `create_thread_operation_atomic()` to support internal operation kinds.
|
||||||
|
- Gateway checkpoint mutations outside run execution must use `services.reserve_checkpoint_write()`, which composes the process-local thread lock with the durable `checkpoint_write` reservation. Manual compaction, `POST /threads/{id}/state`, and both goal mutation routes (`PUT` / `DELETE /threads/{id}/goal`, including creation of a missing goal checkpoint) use this boundary, so an existing run blocks the write and the reservation blocks new reject/interrupt/rollback runs across workers.
|
||||||
|
- Branch/state-update checkpoints copy only the source checkpoint's persisted `deerflow_agent_name`, never request metadata; missing or malformed bindings stay unbound so compaction fails closed.
|
||||||
|
- `DELETE /api/threads/{id}` holds a durable `delete` reservation for the whole cleanup and removes thread filesystem data (off-loop, drained on cancel), checkpoints, owner-scoped historical runs, run events, feedback and thread metadata best-effort. Historical-run cleanup removes only `operation_kind="run"` rows, so internal thread-operation rows — including the reservation protecting that request — stay durable until `RunManager.reserve_thread_operation()` exits. This cleans persisted rows; preventing an already-admitted write from re-creating state for a deleted thread is a separate thread-incarnation contract, not part of this cleanup.
|
||||||
|
- Creator SSE and `/wait` apply `on_disconnect` only to confirmed request disconnect or cancellation; bridge failures propagate without cancelling the run. `/wait` consumes bridge END; errors return SDK `__error__`, never stale state.
|
||||||
|
- Memory and Redis `StreamBridge` implementations retain only `stream_bridge.queue_maxsize` data events. A syntactically valid `Last-Event-ID` older than the retained watermark, or a live subscriber that falls behind it, yields `StreamGap` before any partial replay. `sse_consumer` maps that control item to an id-less SSE `gap` payload (`stream_replay_gap`) and intentionally leaves the run active; internal `/wait` consumers resume from its latest retained ID because they only need terminal completion. Redis checks bounds plus the non-blocking read in one transaction, using blocking `XREAD` only as a wake-up before repeating the atomic snapshot. For a no-cursor subscriber that established a wait on an empty stream, the first wake response remains provisional until that next snapshot verifies its tail is still retained; this closes the pre-first-delivery trimming window without changing malformed-cursor live tailing. The correctness tradeoff is one three-command snapshot pipeline per poll plus the blocking wake round trip while idle. Malformed cursor behavior remains backend-specific. Memory treats a syntactically numeric cursor below its watermark conservatively as a gap even when the evicted timestamp can no longer be verified; unknown ids at or above the watermark retain the legacy replay-from-earliest policy.
|
||||||
|
- Redis `StreamBridge` keys use a rolling retained-buffer TTL (`stream_bridge.stream_ttl_seconds`, refreshed on data and first-END writes) as a leak safety net, not as a run timeout. Startup and lease-driven periodic orphan recovery share one Gateway stream-terminalization path: after `RunManager` durably marks a run `error` with `stop_reason=orphan_recovered`, Gateway publishes `END_SENTINEL` and schedules stream cleanup. The periodic store scan, per-row status writes, and Gateway callback run as one supervised single-flight task, so a slow pass is skipped at the next interval instead of piling up or pausing the sole lease-renewal loop. Store retries have bounded attempts/backoff; an individual operation still relies on the database driver/pool timeout. `RunManager.shutdown()` gives active user runs priority within its shared deadline, then drains or cancels orphan recovery. Gateway tracks delayed recovered-stream cleanups and converts unfinished delays to immediate deletes before closing the bridge; the Redis TTL remains the outage safety net. Only startup recovery, before the runtime yields to requests, projects the latest affected thread to `error`; periodic recovery deliberately avoids that non-atomic projection because `ThreadMetaStore` has no `latest_run_id` conditional-update contract. Store-only SSE and `/wait` consumers wait for the bridge's real END marker after an ordinary durable terminal status, because status persistence can precede tail events. Terminal `orphan_recovered` and `scheduled_task_orphan_recovered` rows permit heartbeat fallback because their producer is gone; ordinary terminal rows still await END. Malformed `Last-Event-ID` reconnect values live-tail new Redis events rather than replaying the retained buffer. Keep cross-component recovery orchestration in Gateway through the generic `RunManager.on_orphans_recovered` callback; do not introduce a harness-to-app dependency. Callback failure warnings include every recovered `run_id` so operators can identify rows whose Gateway-side terminalization needs inspection.
|
||||||
|
- Thread-scoped create, stream and wait share an optional owner/thread-scoped `Idempotency-Key` (internal callers: owner-only, cross-thread reuse 409); never pass the raw external key to `RunManager`'s process-wide index. Stores resolve an existing key before `interrupt`/`rollback`, so a retry reuses rather than interrupts its run. Reuse requires matching input, assistant and conversation references; keyed resume stores a versioned strict typed-JSON SHA-256 (`true`, `1` and `1.0` differ) in the private `runs.idempotency_request_json` column, never public `kwargs`; ordinary input needs no extra identity. New readers accept the exact old-writer resume identity shape from `kwargs`, every response strips it; identity-less legacy resume fails closed. Validate strict resume JSON with or without a key; persist the identity only for keyed resume admission. Missing headers stay non-idempotent; stateless `/api/runs/*` is outside this contract (it creates a thread before admission).
|
||||||
|
- For a rolling upgrade, route every keyed resume submission/retry on create/stream/wait only to upgraded workers; otherwise pause that traffic or replace old workers first. Old workers ignore the private identity and can reuse `approve` for `deny` with null input. Old-reader schema compatibility does not protect admission. Upgraded workers reject identity-less legacy resumes with 409. Rollout details: `README.md`, `backend/docs/API.md`.
|
||||||
|
- Reused `/wait` returns durable status/error, never the latest thread checkpoint; `task is None` is not completion, and store-backed fields must refresh after waiting. Capture reuse before awaiting because the cached record's `idempotency_reused` flag is sticky. A creating `/runs/stream` retry whose terminal stream is gone opts into `gap`/`stream_replay_gap` via `emit_gap_on_missing_stream`, while observer joins and default/stateless consumers emit `end`; never infer this from disconnect policy or the sticky flag. A reused running `store_only` record on a process-local bridge returns 409 without `Retry-After`.
|
||||||
|
- Thread-scoped run creation accepts `checkpoint` / `checkpoint_id`; Gateway validates the checkpoint belongs to the request thread before writing `checkpoint_id` / `checkpoint_ns` into `config.configurable` for LangGraph branching. In `delta` checkpoint mode the worker rewrites that fork into a linear head write before the graph starts (see "A delta-mode run cannot fork" under Checkpoint Channel Modes), because delta state for a fork replays the abandoned sibling's writes.
|
||||||
|
- Thread-scoped Gateway runs evaluate an active `ThreadState.goal` after the visible turn completes. `runtime/goal.py` asks a non-thinking evaluator model to judge only visible conversation evidence and return a typed blocker; the evaluator model is created once per run and reused across hidden continuation checks. The evaluator runs after the graph root's tracing scope has already closed, so `create_goal_evaluator_model`/`evaluate_goal_completion` attach their own model-level tracing callbacks (`attach_tracing=True`) and inject Langfuse trace metadata (`thread_id`/`user_id`/`deerflow_trace_id`) directly onto the `ainvoke` call — the same standalone-caller pattern as `oneshot_llm.run_oneshot_llm` and `MemoryUpdater` (see Tracing System below). Satisfied goals are cleared; every non-satisfied evaluation — continuable or stand-down — is persisted with `last_evaluation` (the blocker, reason, and evidence summary; outcomes that stop the loop additionally record a `stand_down_reason` for observability), but only `goal_not_met_yet` evaluations are streamed as hidden `HumanMessage` continuations, and only when the turn did not end on an unanswered Human Input Card (`ask_clarification` or the sandbox network prompt; the question is still open, so the goal stands down with `blocker: "needs_user_input"` without calling the evaluator), a durable assistant end-of-turn checkpoint exists, the run has not been aborted, the run has not hit its token-budget hard stop (`stop_reason == "token_capped"`; the goal stands down with `stand_down_reason: "token_capped"`), the thread did not change during evaluation, and the no-progress breaker has not fired. The continuation cap is 8 — a hard maximum in the `0`–`8` range; callers requesting more are clamped (`set_goal`/TUI) or rejected with 422 (`PUT /goal`). The no-progress breaker keys on the latest visible assistant evidence (not the evaluator's free-text reason, which an LLM rewords every turn), so two consecutive continuations that add no new visible assistant output stop the loop after 2 attempts. Model-response cleanup helpers such as think-block stripping and code-fence stripping live in `deerflow.utils.llm_text` so `runtime/goal.py` and Gateway suggestion parsing share the same JSON-prep behavior.
|
||||||
|
- Run event stream changes must keep producer code, `deerflow/constants.py`, `runtime/events/catalog.py`, `contracts/run_event_stream_contract.json`, `backend/docs/RUN_EVENT_STREAM.md`, and `tests/test_run_event_stream_contract.py` in sync. The dependency-free constants module owns the persisted envelope limits (`event_type` 32 characters, `category` 16) and cross-layer workspace event identity; the catalog owns validated runtime definitions and categories. Dynamic middleware tags are limited to 21 characters after the `middleware:` prefix. The JSON contract owns payload schemas, backend-specific storage semantics, legacy aliases, and compatibility rules; conformance tests require both views and all producer groups to agree. `run.end.content` remains opaque and may retain nested Python values in memory while JSONL/database stores stringify non-JSON nested values, so consumers must not assume backend-identical nested output representations.
|
||||||
|
|
||||||
|
Scheduled-task PATCH: omitted `assistant_id` preserves the pin; explicit null resets to `lead_agent`.
|
||||||
|
|
||||||
|
Two origin keys; never read one for the other. `deerflow_scheduled_origin` marks the scheduled prompt message (`scheduled_goal_notes_contract.json`); `deerflow_origin` says who started a run/created a thread (`thread_origin_contract.json`), set only via `request.state.run_origin` or internal `body.metadata`.
|
||||||
|
|
||||||
|
**Thread lifecycle**: Before changing branching, regeneration, edit replay, or
|
||||||
|
archive/search behavior, read [Thread lifecycle invariants](../../docs/THREAD_LIFECYCLE.md).
|
||||||
|
It owns lineage and settled-checkpoint rules, legacy fallback boundaries, archive
|
||||||
|
filtering before pagination, owner isolation, and activity-time preservation.
|
||||||
|
|
||||||
|
MCP create/replace/state writes require unique installation IDs. DELETE validates
|
||||||
|
schema but permits residual legacy ID collisions. Explicit selections exclude
|
||||||
|
ambiguous IDs until repaired.
|
||||||
|
|
||||||
|
Capability skill discovery (`/api/capabilities/installations/skills`) reuses
|
||||||
|
`routers/skills.py::_filter_visible_skills` after its off-thread storage read.
|
||||||
|
It must preserve the ordinary skill listing's caller visibility and provider
|
||||||
|
failure policy rather than exposing the unfiltered user-scoped catalog.
|
||||||
|
|
||||||
|
Cap edit reads at `MAX_EDITABLE_ARTIFACT_BYTES + 1` despite stale `lstat`.
|
||||||
|
|
||||||
|
### Route and skill authorization
|
||||||
|
|
||||||
|
`authz.authorize_model_use` shares the `model:use` decision between model details
|
||||||
|
and follow-up suggestions, including the factory's first configured model when
|
||||||
|
the caller omits `model_name`. Suggestions check admission before their
|
||||||
|
best-effort LLM error handler: explicit denies return 403 without invoking the
|
||||||
|
model, while provider failures honor `authorization.fail_closed`.
|
||||||
|
|
||||||
|
`authz.py::resolve_route_permissions()` is shared by HTTP, decorator-only auth, and Live Browser WebSocket admission. It evaluates full `resource:action` targets; provider errors follow `authorization.fail_closed`, while disabled authorization preserves legacy permissions. Owner/admin checks stay separate. Live requires `threads:write` with `is_internal=False` after login/Origin checks; denial closes 4403 and setup errors close 4501. Admission is connection-scoped. On disconnect, Live drains owned `stop_screencast()` cleanup before releasing the session lease. Tests: `test_authorization_route_permissions.py`, `test_browser_readonly_security.py`, `test_auth.py`, `test_auth_middleware.py`.
|
||||||
|
|
||||||
|
Skill listing authorization mirrors the models pattern: `routers/skills.py` routes resolve `(provider, principal)` through `authz.py::resolve_skill_authorization()` — a thin sibling of `resolve_model_authorization`, both delegating to the shared `_resolve_route_scoped_authorization` core — and the shared `_filter_visible_skills` helper filters the user-scoped catalog (public + caller's custom skills) through `filter_resources(principal, "skill", ...)` by name. Three user-facing surfaces apply it: `GET /api/skills` (the frontend skill list / slash-command autocomplete), `GET /api/skills/custom`, and `GET /api/skills/{name}` — the detail endpoint returns the standard 404 for an invisible skill so it cannot become an existence oracle the filtered list closed (`get_model`'s 403 is an execution decision via `authorize("model", "use")`, which has no skills equivalent in this layer). Anonymous callers are unfiltered (mirroring `list_models`), and provider resolution/decision errors follow `authorization.fail_closed` (fail-closed → empty listing / 404, fail-open → full listing). Skill management endpoints (`install`/`upload`/`reload`/custom-skill CRUD) stay `require_admin_user`-gated, and runtime activation authorization is a separate layer. The built-in RBAC provider maps this to the per-role `skills` policy key. Tests: `tests/test_skills_listing_authorization.py`.
|
||||||
|
|
||||||
|
Batch workers pin `app.state.extensions`; never persist snapshots.
|
||||||
|
|
||||||
|
Personal MCP is owner-only; platform MCP shared.
|
||||||
|
|
||||||
|
### Web model provider presets
|
||||||
|
|
||||||
|
Managed model routes expose provider metadata; routing stays server-owned. Align validation, probing, storage, keys, and revisions.
|
||||||
|
|||||||
@ -49,8 +49,6 @@ from app.gateway.routers import (
|
|||||||
projects,
|
projects,
|
||||||
runs,
|
runs,
|
||||||
scheduled_tasks,
|
scheduled_tasks,
|
||||||
search,
|
|
||||||
shares,
|
|
||||||
skill_mutations,
|
skill_mutations,
|
||||||
skills,
|
skills,
|
||||||
subagent_batches,
|
subagent_batches,
|
||||||
@ -925,18 +923,11 @@ async def _gateway_lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
if subagent_batches_config.enabled and batch_repo is None:
|
if subagent_batches_config.enabled and batch_repo is None:
|
||||||
raise RuntimeError("subagent_batches.enabled requires database.backend sqlite or postgres")
|
raise RuntimeError("subagent_batches.enabled requires database.backend sqlite or postgres")
|
||||||
if batch_repo is not None:
|
if batch_repo is not None:
|
||||||
from app.subagent_batches.continuation import BatchContinuations
|
|
||||||
|
|
||||||
continuations = BatchContinuations(app)
|
|
||||||
app.state.batch_continuations = continuations
|
|
||||||
batch_service = SubagentBatchService(
|
batch_service = SubagentBatchService(
|
||||||
repository=batch_repo,
|
repository=batch_repo,
|
||||||
config=subagent_batches_config,
|
config=subagent_batches_config,
|
||||||
runtime_config=subagent_runtime_config,
|
runtime_config=subagent_runtime_config,
|
||||||
extensions=getattr(app.state, "extensions", None),
|
extensions=getattr(app.state, "extensions", None),
|
||||||
prepare_continuation=continuations.prepare,
|
|
||||||
dispatch_continuation=continuations.dispatch,
|
|
||||||
cancel_continuation=continuations.cancel,
|
|
||||||
)
|
)
|
||||||
app.state.subagent_batch_service = batch_service
|
app.state.subagent_batch_service = batch_service
|
||||||
if subagent_batches_config.enabled:
|
if subagent_batches_config.enabled:
|
||||||
@ -1381,9 +1372,7 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
|
|||||||
app.include_router(uploads.router)
|
app.include_router(uploads.router)
|
||||||
|
|
||||||
# Thread cleanup API is mounted at /api/threads/{thread_id}
|
# Thread cleanup API is mounted at /api/threads/{thread_id}
|
||||||
app.include_router(search.router)
|
|
||||||
app.include_router(threads.router)
|
app.include_router(threads.router)
|
||||||
app.include_router(shares.router)
|
|
||||||
|
|
||||||
# Per-user activity feed (own prefix, so /api/threads/{thread_id} never captures it)
|
# Per-user activity feed (own prefix, so /api/threads/{thread_id} never captures it)
|
||||||
app.include_router(thread_activity.router)
|
app.include_router(thread_activity.router)
|
||||||
|
|||||||
@ -243,15 +243,7 @@ def build_artifact_archive(
|
|||||||
reserved = frozenset(name.casefold() for name in {BROWSER_FRAMES_DIRNAME, TOOL_RESULTS_DIRNAME, *extra_reserved_dir_names})
|
reserved = frozenset(name.casefold() for name in {BROWSER_FRAMES_DIRNAME, TOOL_RESULTS_DIRNAME, *extra_reserved_dir_names})
|
||||||
members = [_member(root, path, reserved, deadline, root_components) for path in paths]
|
members = [_member(root, path, reserved, deadline, root_components) for path in paths]
|
||||||
collision_keys = [unicodedata.normalize("NFC", member.entry).casefold() for member in members]
|
collision_keys = [unicodedata.normalize("NFC", member.entry).casefold() for member in members]
|
||||||
collision_set = set(collision_keys)
|
if len(collision_keys) != len(set(collision_keys)):
|
||||||
if len(collision_keys) != len(collision_set):
|
|
||||||
raise _reject()
|
|
||||||
implicit_dirs = set()
|
|
||||||
for key in collision_keys:
|
|
||||||
parts = key.split("/")
|
|
||||||
for i in range(1, len(parts)):
|
|
||||||
implicit_dirs.add("/".join(parts[:i]))
|
|
||||||
if collision_set & implicit_dirs:
|
|
||||||
raise _reject()
|
raise _reject()
|
||||||
|
|
||||||
sizes = [member.initial.st_size for member in members]
|
sizes = [member.initial.st_size for member in members]
|
||||||
|
|||||||
@ -63,7 +63,7 @@ class OIDCValidationError(OIDCError):
|
|||||||
|
|
||||||
|
|
||||||
class OIDCUserInfoMismatch(OIDCError):
|
class OIDCUserInfoMismatch(OIDCError):
|
||||||
"""UserInfo sub is missing or does not match ID token sub."""
|
"""UserInfo sub does not match ID token sub."""
|
||||||
|
|
||||||
|
|
||||||
def _json_object(resp: httpx.Response, what: str) -> dict[str, Any]:
|
def _json_object(resp: httpx.Response, what: str) -> dict[str, Any]:
|
||||||
@ -391,10 +391,8 @@ class OIDCService:
|
|||||||
async def fetch_userinfo(self, metadata: OIDCMetadata, access_token: str, expected_sub: str) -> dict[str, Any]:
|
async def fetch_userinfo(self, metadata: OIDCMetadata, access_token: str, expected_sub: str) -> dict[str, Any]:
|
||||||
"""Fetch userinfo from the UserInfo endpoint.
|
"""Fetch userinfo from the UserInfo endpoint.
|
||||||
|
|
||||||
The response is returned only when its ``sub`` equals ``expected_sub``
|
Validates that the ``sub`` claim matches ``expected_sub``
|
||||||
(from the ID token). OIDC Core 5.3.2 requires that check, and a
|
(from the ID token) to prevent userinfo injection.
|
||||||
response without a ``sub`` cannot be tied to this login, so it raises
|
|
||||||
``OIDCUserInfoMismatch`` too instead of overriding the ID token claims.
|
|
||||||
"""
|
"""
|
||||||
if not metadata.userinfo_endpoint:
|
if not metadata.userinfo_endpoint:
|
||||||
return {}
|
return {}
|
||||||
@ -409,12 +407,7 @@ class OIDCService:
|
|||||||
raise OIDCError(f"UserInfo fetch failed: {exc}") from exc
|
raise OIDCError(f"UserInfo fetch failed: {exc}") from exc
|
||||||
userinfo = _json_object(resp, "UserInfo response")
|
userinfo = _json_object(resp, "UserInfo response")
|
||||||
|
|
||||||
sub = userinfo.get("sub")
|
if userinfo.get("sub") and userinfo["sub"] != expected_sub:
|
||||||
if sub is None or sub == "":
|
|
||||||
raise OIDCUserInfoMismatch("UserInfo response has no sub claim")
|
|
||||||
if not isinstance(sub, str):
|
|
||||||
raise OIDCUserInfoMismatch(f"UserInfo sub claim is not a string ({type(sub).__name__})")
|
|
||||||
if sub != expected_sub:
|
|
||||||
raise OIDCUserInfoMismatch("UserInfo sub does not match ID token sub")
|
raise OIDCUserInfoMismatch("UserInfo sub does not match ID token sub")
|
||||||
|
|
||||||
return userinfo
|
return userinfo
|
||||||
|
|||||||
@ -77,9 +77,6 @@ _PAT_ROUTE_RULES: tuple[tuple[frozenset[str], re.Pattern[str]], ...] = (
|
|||||||
(frozenset({"GET", "PATCH", "DELETE"}), re.compile(r"^/api/projects/[^/]+$")),
|
(frozenset({"GET", "PATCH", "DELETE"}), re.compile(r"^/api/projects/[^/]+$")),
|
||||||
(frozenset({"POST"}), re.compile(r"^/api/projects/[^/]+/(archive|restore)$")),
|
(frozenset({"POST"}), re.compile(r"^/api/projects/[^/]+/(archive|restore)$")),
|
||||||
(frozenset({"GET"}), re.compile(r"^/api/projects/[^/]+/threads$")),
|
(frozenset({"GET"}), re.compile(r"^/api/projects/[^/]+/threads$")),
|
||||||
(frozenset({"GET", "PATCH"}), re.compile(r"^/api/projects/[^/]+/memory$")),
|
|
||||||
(frozenset({"DELETE"}), re.compile(r"^/api/projects/[^/]+/memory/[^/]+$")),
|
|
||||||
(frozenset({"POST"}), re.compile(r"^/api/projects/[^/]+/memory/[^/]+/promote$")),
|
|
||||||
# Project document shelf (Phase 2 Slice B): enumerated per implemented
|
# Project document shelf (Phase 2 Slice B): enumerated per implemented
|
||||||
# route — list/upload on the collection, content and trash on the item.
|
# route — list/upload on the collection, content and trash on the item.
|
||||||
(frozenset({"GET", "POST"}), re.compile(r"^/api/projects/[^/]+/documents$")),
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/projects/[^/]+/documents$")),
|
||||||
|
|||||||
@ -6,7 +6,7 @@ from app.gateway.auth.models import User
|
|||||||
|
|
||||||
|
|
||||||
class LastAdminRemainsError(RuntimeError):
|
class LastAdminRemainsError(RuntimeError):
|
||||||
"""Raised when a role write would demote the last remaining active admin.
|
"""Raised when a role write would demote the last remaining admin.
|
||||||
|
|
||||||
Bootstrap lockout protection: raised inside the same serialized
|
Bootstrap lockout protection: raised inside the same serialized
|
||||||
transaction that would perform the write, so the check-then-act window
|
transaction that would perform the write, so the check-then-act window
|
||||||
@ -123,9 +123,8 @@ class UserRepository(ABC):
|
|||||||
Raises:
|
Raises:
|
||||||
UserNotFoundError: the target row does not exist.
|
UserNotFoundError: the target row does not exist.
|
||||||
LastAdminRemainsError: the write would demote the only remaining
|
LastAdminRemainsError: the write would demote the only remaining
|
||||||
active admin (the count and the write share one serialized
|
admin (the count and the write share one serialized
|
||||||
transaction — two concurrent demotions cannot both pass).
|
transaction — two concurrent demotions cannot both pass).
|
||||||
An already disabled admin may be assigned a different role.
|
|
||||||
"""
|
"""
|
||||||
raise NotImplementedError
|
raise NotImplementedError
|
||||||
|
|
||||||
|
|||||||
@ -316,12 +316,11 @@ class SQLiteUserRepository(UserRepository):
|
|||||||
row = await session.get(UserRow, user_id)
|
row = await session.get(UserRow, user_id)
|
||||||
if row is None:
|
if row is None:
|
||||||
raise UserNotFoundError(f"User {user_id} no longer exists")
|
raise UserNotFoundError(f"User {user_id} no longer exists")
|
||||||
if row.system_role == "admin" and not row.disabled and system_role != "admin":
|
if row.system_role == "admin" and system_role != "admin":
|
||||||
# Count ACTIVE admins only: a disabled admin cannot
|
# Count ACTIVE admins only: a disabled admin cannot
|
||||||
# authenticate, so counting them would let "disable B, then
|
# authenticate, so counting them would let "disable B, then
|
||||||
# demote self" (or a combined role+disable request) strand
|
# demote self" (or a combined role+disable request) strand
|
||||||
# the deployment with zero usable management credentials.
|
# the deployment with zero usable management credentials.
|
||||||
# Demoting an already disabled admin removes no active admin.
|
|
||||||
active_admins = await session.scalar(select(func.count()).select_from(UserRow).where(UserRow.system_role == "admin", UserRow.disabled.is_(False)))
|
active_admins = await session.scalar(select(func.count()).select_from(UserRow).where(UserRow.system_role == "admin", UserRow.disabled.is_(False)))
|
||||||
if active_admins is None or active_admins <= 1:
|
if active_admins is None or active_admins <= 1:
|
||||||
raise LastAdminRemainsError("cannot demote the last remaining admin")
|
raise LastAdminRemainsError("cannot demote the last remaining admin")
|
||||||
|
|||||||
@ -10,7 +10,6 @@ Fine-grained permission checks remain in authz.py decorators.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
import re
|
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
|
|
||||||
from fastapi import HTTPException, Request, Response
|
from fastapi import HTTPException, Request, Response
|
||||||
@ -30,7 +29,7 @@ from app.gateway.auth_disabled import (
|
|||||||
from app.gateway.authz import AuthContext, resolve_route_permissions
|
from app.gateway.authz import AuthContext, resolve_route_permissions
|
||||||
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
|
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
|
||||||
from app.gateway.request_path import get_request_route_path
|
from app.gateway.request_path import get_request_route_path
|
||||||
from deerflow.runtime.user_context import bind_authenticated_user, reset_current_user, set_current_user
|
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@ -61,11 +60,7 @@ _PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _is_public(path: str, method: str = "GET") -> bool:
|
def _is_public(path: str) -> bool:
|
||||||
# A share token grants only these reads. Forks and any future write routes
|
|
||||||
# under /shares still pass through session authentication and permissions.
|
|
||||||
if method in {"GET", "HEAD"} and re.fullmatch(r"/api/shares/[A-Za-z0-9_-]{43}(?:/artifacts/.+)?", path):
|
|
||||||
return True
|
|
||||||
stripped = path.rstrip("/")
|
stripped = path.rstrip("/")
|
||||||
if stripped in _PUBLIC_EXACT_PATHS:
|
if stripped in _PUBLIC_EXACT_PATHS:
|
||||||
return True
|
return True
|
||||||
@ -96,7 +91,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
|||||||
super().__init__(app)
|
super().__init__(app)
|
||||||
|
|
||||||
async def dispatch(self, request: Request, call_next: Callable) -> Response:
|
async def dispatch(self, request: Request, call_next: Callable) -> Response:
|
||||||
if _is_public(get_request_route_path(request), request.method):
|
if _is_public(get_request_route_path(request)):
|
||||||
return await call_next(request)
|
return await call_next(request)
|
||||||
|
|
||||||
internal_user = None
|
internal_user = None
|
||||||
@ -212,14 +207,6 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
# Browser extension pages are bound to the account that loaded their
|
|
||||||
# descriptor. Cookies can change in another tab before React observes it.
|
|
||||||
# This is a stale-view fence, not an authorization grant; CLI callers may
|
|
||||||
# omit it and all existing route/resource checks still apply.
|
|
||||||
expected_viewer = request.headers.get("x-deerflow-plugin-viewer")
|
|
||||||
if get_request_route_path(request).startswith("/api/extensions/") and expected_viewer is not None and expected_viewer != str(user.id):
|
|
||||||
return JSONResponse(status_code=409, content={"detail": "Account changed; reload this extension view."})
|
|
||||||
|
|
||||||
# Stamp both request.state.user (for the contextvar pattern)
|
# Stamp both request.state.user (for the contextvar pattern)
|
||||||
# and request.state.auth (so @require_permission's "auth is
|
# and request.state.auth (so @require_permission's "auth is
|
||||||
# None" branch short-circuits instead of running the entire
|
# None" branch short-circuits instead of running the entire
|
||||||
@ -239,8 +226,6 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
|||||||
request.state.auth = AuthContext(user=user, permissions=permissions)
|
request.state.auth = AuthContext(user=user, permissions=permissions)
|
||||||
token = set_current_user(user)
|
token = set_current_user(user)
|
||||||
try:
|
try:
|
||||||
verified_user_id = str(user.id) if auth_source in {AUTH_SOURCE_SESSION, AUTH_SOURCE_PAT} else None
|
|
||||||
with bind_authenticated_user(verified_user_id):
|
|
||||||
return await call_next(request)
|
return await call_next(request)
|
||||||
finally:
|
finally:
|
||||||
reset_current_user(token)
|
reset_current_user(token)
|
||||||
|
|||||||
@ -191,10 +191,7 @@ def _get_route_authorization_config() -> AuthorizationConfig:
|
|||||||
environments without a config.yaml), preserving legacy all-permissions behavior.
|
environments without a config.yaml), preserving legacy all-permissions behavior.
|
||||||
``get_app_config()`` stats and hashes ``config.yaml`` and
|
``get_app_config()`` stats and hashes ``config.yaml`` and
|
||||||
``extensions_config.json`` on every call, so async request paths use
|
``extensions_config.json`` on every call, so async request paths use
|
||||||
:func:`_route_authorization_config_async`, and call the synchronous
|
:func:`_route_authorization_config_async`.
|
||||||
helpers that read it (``resolve_model_authorization``,
|
|
||||||
``resolve_skill_authorization``, ``authorize_model_use``,
|
|
||||||
``authorize_sandbox_for_request``) through ``asyncio.to_thread``.
|
|
||||||
"""
|
"""
|
||||||
from deerflow.config.app_config import get_app_config
|
from deerflow.config.app_config import get_app_config
|
||||||
|
|
||||||
@ -823,7 +820,7 @@ async def try_acquire_sandbox_for_request(
|
|||||||
|
|
||||||
user = await get_optional_user_from_request(request) if request is not None else None
|
user = await get_optional_user_from_request(request) if request is not None else None
|
||||||
if user is not None:
|
if user is not None:
|
||||||
await asyncio.to_thread(authorize_sandbox_for_request, user, is_internal=_is_internal_caller(request, user), app_config=app_config)
|
authorize_sandbox_for_request(user, is_internal=_is_internal_caller(request, user), app_config=app_config)
|
||||||
except SandboxAuthorizationError:
|
except SandboxAuthorizationError:
|
||||||
logger.info("Sandbox sync skipped: sandbox execution not permitted for this caller (thread_id=%s)", thread_id)
|
logger.info("Sandbox sync skipped: sandbox execution not permitted for this caller (thread_id=%s)", thread_id)
|
||||||
return SandboxRequestLease(
|
return SandboxRequestLease(
|
||||||
|
|||||||
@ -194,11 +194,6 @@ class LarkAdapter:
|
|||||||
|
|
||||||
class SkillAdapter:
|
class SkillAdapter:
|
||||||
async def list_installations(self, context: AdapterContext) -> list[CapabilityInstallation]:
|
async def list_installations(self, context: AdapterContext) -> list[CapabilityInstallation]:
|
||||||
# The listing reports the enabled flag, which the storage layer fails
|
|
||||||
# closed on an unreadable extensions config; keep this read surface
|
|
||||||
# loud about the broken config instead of serving the degraded list.
|
|
||||||
# Blocking filesystem IO: always off the event loop, like the scan below.
|
|
||||||
await asyncio.to_thread(skills._assert_extensions_config_readable)
|
|
||||||
response = await asyncio.to_thread(lambda: skills._get_user_skill_storage(context.config).load_skills(enabled_only=False))
|
response = await asyncio.to_thread(lambda: skills._get_user_skill_storage(context.config).load_skills(enabled_only=False))
|
||||||
response = await skills._filter_visible_skills(context.request, context.config, response)
|
response = await skills._filter_visible_skills(context.request, context.config, response)
|
||||||
return [
|
return [
|
||||||
|
|||||||
@ -6,26 +6,6 @@ from collections.abc import Sequence
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
def task_interrupts_by_id(snapshot: Any) -> dict[str, list[Any]]:
|
|
||||||
"""Group non-empty task interrupts of one snapshot by task id."""
|
|
||||||
grouped: dict[str, list[Any]] = {}
|
|
||||||
for task in getattr(snapshot, "tasks", None) or ():
|
|
||||||
interrupts = list(getattr(task, "interrupts", ()) or ())
|
|
||||||
if interrupts:
|
|
||||||
grouped[task.id] = interrupts
|
|
||||||
return grouped
|
|
||||||
|
|
||||||
|
|
||||||
def is_hitl_approval_payload(value: Any) -> bool:
|
|
||||||
"""Whether an interrupt payload is a HITL tool-approval request."""
|
|
||||||
return isinstance(value, dict) and value.get("kind") == "hitl_approval"
|
|
||||||
|
|
||||||
|
|
||||||
def hitl_approval_payloads(grouped: dict[str, list[Any]]) -> list[dict[str, Any]]:
|
|
||||||
"""Flatten approval payloads from task_interrupts_by_id output."""
|
|
||||||
return [interrupt.value for interrupts in grouped.values() for interrupt in interrupts if is_hitl_approval_payload(interrupt.value)]
|
|
||||||
|
|
||||||
|
|
||||||
class CheckpointLineageError(RuntimeError):
|
class CheckpointLineageError(RuntimeError):
|
||||||
"""Raised when a requested checkpoint ancestor cannot be resolved safely."""
|
"""Raised when a requested checkpoint ancestor cannot be resolved safely."""
|
||||||
|
|
||||||
|
|||||||
@ -66,10 +66,7 @@ def build_context_usage_payload(*, token_count: int, max_context_tokens: int | N
|
|||||||
async def build_context_usage(request: Request, thread_id: str, run_store: Any, user_id: str | None = None) -> dict[str, Any] | None:
|
async def build_context_usage(request: Request, thread_id: str, run_store: Any, user_id: str | None = None) -> dict[str, Any] | None:
|
||||||
"""Return approximate usage for the latest materialized thread checkpoint."""
|
"""Return approximate usage for the latest materialized thread checkpoint."""
|
||||||
try:
|
try:
|
||||||
# ``get_config`` revalidates ``config.yaml`` and
|
app_config = get_config()
|
||||||
# ``extensions_config.json`` on every call (stat + full-file hash), so
|
|
||||||
# reading it here would put that IO on the loop every thread view pays.
|
|
||||||
app_config = await asyncio.to_thread(get_config)
|
|
||||||
except HTTPException:
|
except HTTPException:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|||||||
@ -5,14 +5,13 @@ from __future__ import annotations
|
|||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
from collections.abc import Awaitable, Callable, Mapping
|
from collections.abc import Awaitable, Callable
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
from urllib.parse import urlsplit
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
from fastapi import HTTPException, Request
|
from fastapi import HTTPException, Request
|
||||||
|
|
||||||
from app.gateway.conversation_reader import read_visible_message_page
|
from app.gateway.conversation_reader import read_visible_message_page
|
||||||
from app.gateway.csrf_middleware import _normalize_origin, _request_origin
|
|
||||||
from deerflow.constants import CONVERSATION_TOOL_NAME, CONVERSATION_TOOL_USE
|
from deerflow.constants import CONVERSATION_TOOL_NAME, CONVERSATION_TOOL_USE
|
||||||
from deerflow.utils.llm_text import strip_think_blocks
|
from deerflow.utils.llm_text import strip_think_blocks
|
||||||
from deerflow.utils.thread_id import validate_thread_id
|
from deerflow.utils.thread_id import validate_thread_id
|
||||||
@ -44,20 +43,13 @@ _BUDGET_TOO_SMALL = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _source_id(reference: str, request_origin: str | None) -> str:
|
def _source_id(reference: str, request_url: str) -> str:
|
||||||
"""URLs are local selectors on this DeerFlow's host, never network fetch targets.
|
"""URLs are same-origin local selectors, never network fetch targets."""
|
||||||
|
|
||||||
*request_origin* is the origin the client addressed, as the CSRF check
|
|
||||||
resolves it (forwarded host and port included). Only the host and port are
|
|
||||||
compared, not the scheme: behind a TLS proxy Gateway can see ``http`` for a
|
|
||||||
browser's ``https`` link, because uvicorn ignores ``X-Forwarded-Proto``
|
|
||||||
from an untrusted peer and a local nginx resends its own scheme.
|
|
||||||
"""
|
|
||||||
if "://" not in reference:
|
if "://" not in reference:
|
||||||
return validate_thread_id(reference)
|
return validate_thread_id(reference)
|
||||||
parsed = urlsplit(reference)
|
parsed = urlsplit(reference)
|
||||||
origin = _normalize_origin(f"{parsed.scheme}://{parsed.netloc}")
|
origin = urlsplit(request_url)
|
||||||
if origin is None or request_origin is None or urlsplit(origin).netloc != urlsplit(request_origin).netloc or parsed.query or parsed.fragment:
|
if parsed.scheme not in {"http", "https"} or (parsed.scheme, parsed.netloc) != (origin.scheme, origin.netloc) or parsed.query or parsed.fragment:
|
||||||
raise ValueError("Use a thread ID or a conversation URL from this DeerFlow origin")
|
raise ValueError("Use a thread ID or a conversation URL from this DeerFlow origin")
|
||||||
match = re.fullmatch(r"/workspace/(?:agents/[^/]+/)?chats/([^/]+)", parsed.path)
|
match = re.fullmatch(r"/workspace/(?:agents/[^/]+/)?chats/([^/]+)", parsed.path)
|
||||||
if match is None:
|
if match is None:
|
||||||
@ -65,39 +57,16 @@ def _source_id(reference: str, request_origin: str | None) -> str:
|
|||||||
return validate_thread_id(match.group(1))
|
return validate_thread_id(match.group(1))
|
||||||
|
|
||||||
|
|
||||||
def _visible_text(row: dict, *, thread_id: str | None = None) -> tuple[str, str] | None:
|
def _visible_text(row: dict) -> tuple[str, str] | None:
|
||||||
message = row.get("content")
|
message = row.get("content")
|
||||||
if not isinstance(message, Mapping):
|
if not isinstance(message, dict):
|
||||||
return None
|
return None
|
||||||
role = message.get("type") or message.get("role")
|
role = message.get("type") or message.get("role")
|
||||||
if not isinstance(role, str):
|
|
||||||
return None
|
|
||||||
role = {"human": "user", "ai": "assistant"}.get(role, role)
|
role = {"human": "user", "ai": "assistant"}.get(role, role)
|
||||||
extra = message.get("additional_kwargs")
|
extra = message.get("additional_kwargs") or {}
|
||||||
if extra is None:
|
|
||||||
extra = {}
|
|
||||||
elif not isinstance(extra, Mapping):
|
|
||||||
logger.warning(
|
|
||||||
"Ignoring malformed message additional_kwargs for thread %s at row seq %s: expected a mapping, got %s",
|
|
||||||
thread_id or "<unknown>",
|
|
||||||
row.get("seq", "<unknown>"),
|
|
||||||
type(extra).__name__,
|
|
||||||
)
|
|
||||||
extra = {}
|
|
||||||
if role not in {"user", "assistant"} or extra.get("hide_from_ui") or message.get("name") == "summary":
|
if role not in {"user", "assistant"} or extra.get("hide_from_ui") or message.get("name") == "summary":
|
||||||
return None
|
return None
|
||||||
metadata = row.get("metadata")
|
if str((row.get("metadata") or {}).get("caller", "")).startswith(("middleware:", "subagent:")):
|
||||||
if metadata is None:
|
|
||||||
metadata = {}
|
|
||||||
elif not isinstance(metadata, Mapping):
|
|
||||||
logger.warning(
|
|
||||||
"Ignoring malformed conversation metadata for thread %s at row seq %s: expected a mapping, got %s",
|
|
||||||
thread_id or "<unknown>",
|
|
||||||
row.get("seq", "<unknown>"),
|
|
||||||
type(metadata).__name__,
|
|
||||||
)
|
|
||||||
metadata = {}
|
|
||||||
if str(metadata.get("caller", "")).startswith(("middleware:", "subagent:")):
|
|
||||||
return None
|
return None
|
||||||
content = message.get("content")
|
content = message.get("content")
|
||||||
if role == "user" and isinstance(extra.get("original_user_content"), str):
|
if role == "user" and isinstance(extra.get("original_user_content"), str):
|
||||||
@ -189,8 +158,7 @@ def prepare_conversation_reader(
|
|||||||
if auth is None or not auth.is_authenticated or not auth.has_permission("runs", "read") or not user_id:
|
if auth is None or not auth.is_authenticated or not auth.has_permission("runs", "read") or not user_id:
|
||||||
raise HTTPException(status_code=403, detail="Permission denied: runs:read")
|
raise HTTPException(status_code=403, detail="Permission denied: runs:read")
|
||||||
try:
|
try:
|
||||||
request_origin = _request_origin(request)
|
ids = tuple(dict.fromkeys(_source_id(reference, str(request.url)) for reference in references))
|
||||||
ids = tuple(dict.fromkeys(_source_id(reference, request_origin) for reference in references))
|
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||||
return _prepare_owner_bound_conversation_reader(ids, user_id=user_id, run_context=run_context, run_manager=run_manager, app_config=app_config)
|
return _prepare_owner_bound_conversation_reader(ids, user_id=user_id, run_context=run_context, run_manager=run_manager, app_config=app_config)
|
||||||
@ -260,7 +228,7 @@ def _prepare_owner_bound_conversation_reader(
|
|||||||
found: dict[int, tuple[str, str, int]] = {}
|
found: dict[int, tuple[str, str, int]] = {}
|
||||||
|
|
||||||
def include_target(row: dict) -> bool:
|
def include_target(row: dict) -> bool:
|
||||||
parsed = _visible_text(row, thread_id=thread_id) if row["seq"] == message_seq else None
|
parsed = _visible_text(row) if row["seq"] == message_seq else None
|
||||||
if parsed is None:
|
if parsed is None:
|
||||||
return False
|
return False
|
||||||
role, text = parsed
|
role, text = parsed
|
||||||
@ -305,7 +273,7 @@ def _prepare_owner_bound_conversation_reader(
|
|||||||
parsed_text: dict[int, tuple[str, str]] = {}
|
parsed_text: dict[int, tuple[str, str]] = {}
|
||||||
|
|
||||||
def include_message(row: dict) -> bool:
|
def include_message(row: dict) -> bool:
|
||||||
parsed = _visible_text(row, thread_id=thread_id)
|
parsed = _visible_text(row)
|
||||||
if parsed is None:
|
if parsed is None:
|
||||||
return False
|
return False
|
||||||
role, text = parsed
|
role, text = parsed
|
||||||
|
|||||||
@ -8,7 +8,7 @@ access from a model argument or an ambient HTTP request.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
from collections.abc import Callable, Mapping
|
from collections.abc import Callable
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
@ -70,19 +70,8 @@ def _message_type(message: Any) -> str | None:
|
|||||||
return str(value) if value else None
|
return str(value) if value else None
|
||||||
|
|
||||||
|
|
||||||
def _is_thread_history_hidden_message_row(row: dict[str, Any], *, thread_id: str | None = None) -> bool:
|
def _is_thread_history_hidden_message_row(row: dict[str, Any]) -> bool:
|
||||||
metadata = row.get("metadata")
|
caller = str((row.get("metadata") or {}).get("caller", ""))
|
||||||
if metadata is None:
|
|
||||||
metadata = {}
|
|
||||||
elif not isinstance(metadata, Mapping):
|
|
||||||
logger.warning(
|
|
||||||
"Ignoring malformed conversation metadata for thread %s at row seq %s: expected a mapping, got %s",
|
|
||||||
thread_id or "<unknown>",
|
|
||||||
row.get("seq", "<unknown>"),
|
|
||||||
type(metadata).__name__,
|
|
||||||
)
|
|
||||||
metadata = {}
|
|
||||||
caller = str(metadata.get("caller", ""))
|
|
||||||
return caller.startswith("middleware:") or (caller.startswith("subagent:") and _message_type(row.get("content")) == "ai")
|
return caller.startswith("middleware:") or (caller.startswith("subagent:") and _message_type(row.get("content")) == "ai")
|
||||||
|
|
||||||
|
|
||||||
@ -121,7 +110,7 @@ async def scan_visible_thread_messages(
|
|||||||
if before_seq is not None and row["seq"] >= before_seq:
|
if before_seq is not None and row["seq"] >= before_seq:
|
||||||
reached_before_bound = True
|
reached_before_bound = True
|
||||||
break
|
break
|
||||||
if (not include_middleware and _is_thread_history_hidden_message_row(row, thread_id=thread_id)) or row.get("run_id") in hidden_run_ids or (message_filter is not None and not message_filter(row)):
|
if (not include_middleware and _is_thread_history_hidden_message_row(row)) or row.get("run_id") in hidden_run_ids or (message_filter is not None and not message_filter(row)):
|
||||||
continue
|
continue
|
||||||
visible.append(row)
|
visible.append(row)
|
||||||
if len(visible) == needed:
|
if len(visible) == needed:
|
||||||
@ -148,7 +137,7 @@ async def scan_visible_thread_messages(
|
|||||||
break
|
break
|
||||||
_validate_message_scan_rows(raw, thread_id=thread_id, scan_before=scan_before, scan_after=None)
|
_validate_message_scan_rows(raw, thread_id=thread_id, scan_before=scan_before, scan_after=None)
|
||||||
for row in reversed(raw):
|
for row in reversed(raw):
|
||||||
if (not include_middleware and _is_thread_history_hidden_message_row(row, thread_id=thread_id)) or row.get("run_id") in hidden_run_ids or (message_filter is not None and not message_filter(row)):
|
if (not include_middleware and _is_thread_history_hidden_message_row(row)) or row.get("run_id") in hidden_run_ids or (message_filter is not None and not message_filter(row)):
|
||||||
continue
|
continue
|
||||||
visible_desc.append(row)
|
visible_desc.append(row)
|
||||||
if len(visible_desc) == needed:
|
if len(visible_desc) == needed:
|
||||||
|
|||||||
@ -20,7 +20,6 @@ from __future__ import annotations
|
|||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import tempfile
|
|
||||||
from collections.abc import AsyncGenerator, Callable
|
from collections.abc import AsyncGenerator, Callable
|
||||||
from contextlib import AsyncExitStack, asynccontextmanager
|
from contextlib import AsyncExitStack, asynccontextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@ -385,27 +384,6 @@ def _validate_agent_storage(config: AppConfig) -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _probe_managed_subagent_file_storage() -> None:
|
|
||||||
"""Warn early if this volume cannot support create-only publication."""
|
|
||||||
from deerflow.config.paths import get_paths
|
|
||||||
|
|
||||||
root = get_paths().managed_subagents_dir
|
|
||||||
try:
|
|
||||||
root.mkdir(parents=True, exist_ok=True)
|
|
||||||
# A private directory keeps probe names out of the JSON catalog and
|
|
||||||
# lets independent Gateway workers probe without touching each other.
|
|
||||||
with tempfile.TemporaryDirectory(prefix=".hardlink-probe-", dir=root) as scratch:
|
|
||||||
source = Path(scratch) / "source"
|
|
||||||
source.touch()
|
|
||||||
os.link(source, Path(scratch) / "linked")
|
|
||||||
except OSError as exc:
|
|
||||||
logger.warning(
|
|
||||||
"Managed subagent hard-link probe failed for %s: %s. Managed subagent creation may fail on this volume. Use a volume with working hard links or set agent_storage.backend='db' with SQLite/Postgres.",
|
|
||||||
root,
|
|
||||||
exc,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_login_throttle_storage(config: AppConfig) -> None:
|
def _validate_login_throttle_storage(config: AppConfig) -> None:
|
||||||
"""Warn when a multi-process deployment counts login failures per process.
|
"""Warn when a multi-process deployment counts login failures per process.
|
||||||
|
|
||||||
@ -810,8 +788,6 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
|||||||
# Reject agent_storage.backend='db' on a non-durable database, and warn on
|
# Reject agent_storage.backend='db' on a non-durable database, and warn on
|
||||||
# node-divergent file storage under multi-worker Postgres.
|
# node-divergent file storage under multi-worker Postgres.
|
||||||
_validate_agent_storage(startup_config)
|
_validate_agent_storage(startup_config)
|
||||||
if getattr(getattr(startup_config, "agent_storage", None), "backend", None) == "file":
|
|
||||||
await asyncio.to_thread(_probe_managed_subagent_file_storage)
|
|
||||||
# Warn when login lockouts stay per-process under several Gateway processes.
|
# Warn when login lockouts stay per-process under several Gateway processes.
|
||||||
_validate_login_throttle_storage(startup_config)
|
_validate_login_throttle_storage(startup_config)
|
||||||
# IM chat-to-thread bindings need no companion warning: ``resolve_channel_store``
|
# IM chat-to-thread bindings need no companion warning: ``resolve_channel_store``
|
||||||
@ -889,9 +865,6 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
|||||||
|
|
||||||
# Initialize repositories — one get_session_factory() call for all.
|
# Initialize repositories — one get_session_factory() call for all.
|
||||||
sf = get_session_factory()
|
sf = get_session_factory()
|
||||||
from deerflow.approvals.store import ApprovalStore
|
|
||||||
|
|
||||||
app.state.approval_store = ApprovalStore(sf)
|
|
||||||
|
|
||||||
# The login throttle store is resolved once per process from the startup
|
# The login throttle store is resolved once per process from the startup
|
||||||
# snapshot and the engine above (auth.local.throttle_storage is
|
# snapshot and the engine above (auth.local.throttle_storage is
|
||||||
@ -925,15 +898,6 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
|||||||
run_events_config = getattr(config, "run_events", None)
|
run_events_config = getattr(config, "run_events", None)
|
||||||
app.state.run_events_config = run_events_config
|
app.state.run_events_config = run_events_config
|
||||||
app.state.run_event_store = make_run_event_store(run_events_config)
|
app.state.run_event_store = make_run_event_store(run_events_config)
|
||||||
if sf is not None:
|
|
||||||
from deerflow.persistence.run.producer import validate_producer_fencing
|
|
||||||
|
|
||||||
# An activated database also forbids a memory/JSONL downgrade.
|
|
||||||
await validate_producer_fencing(
|
|
||||||
sf,
|
|
||||||
mode=getattr(run_events_config, "producer_fencing", "off"),
|
|
||||||
cutover_id=getattr(run_events_config, "producer_cutover_id", None),
|
|
||||||
)
|
|
||||||
from deerflow.runtime.events.terminal import RecoveryTerminalPublisher
|
from deerflow.runtime.events.terminal import RecoveryTerminalPublisher
|
||||||
|
|
||||||
app.state.recovery_terminal_publisher = RecoveryTerminalPublisher(app.state.run_event_store, app.state.run_store)
|
app.state.recovery_terminal_publisher = RecoveryTerminalPublisher(app.state.run_event_store, app.state.run_store)
|
||||||
@ -1272,31 +1236,17 @@ def _selected_checkpoint_ids(config: dict[str, Any]) -> frozenset[str]:
|
|||||||
return frozenset(selected)
|
return frozenset(selected)
|
||||||
|
|
||||||
|
|
||||||
def _run_completion_hook(*, scheduled_task_service: Any | None, batch_continuations: Any | None = None):
|
def _run_completion_hook(*, scheduled_task_service: Any | None):
|
||||||
"""Notify the scheduler after ordinary run finalization."""
|
"""Notify the scheduler after ordinary run finalization."""
|
||||||
|
|
||||||
async def handle(record: Any) -> None:
|
async def handle(record: Any) -> None:
|
||||||
# Scheduler persistence is best-effort: a store outage must not skip the
|
|
||||||
# batch cancellation below, or claimed follow-ups of hard-stopped runs
|
|
||||||
# stay in `dispatching` forever (#7969).
|
|
||||||
try:
|
|
||||||
if scheduled_task_service is not None:
|
if scheduled_task_service is not None:
|
||||||
await scheduled_task_service.handle_run_completion(record)
|
await scheduled_task_service.handle_run_completion(record)
|
||||||
except Exception:
|
|
||||||
logger.warning("Scheduled task completion hook failed for run %s (non-fatal)", record.run_id, exc_info=True)
|
|
||||||
if batch_continuations is not None:
|
|
||||||
await batch_continuations.source_finished(record)
|
|
||||||
|
|
||||||
return handle
|
return handle
|
||||||
|
|
||||||
|
|
||||||
def get_agent_runs(request: Request):
|
|
||||||
"""Return the host-bound AgentRuns capability for *request*, or ``None``."""
|
|
||||||
host = getattr(request.app.state, "agent_runs_host", None)
|
|
||||||
# Internal/channel and PAT runs deliberately receive no retained delegation.
|
|
||||||
return host.bind(request) if host is not None else None
|
|
||||||
|
|
||||||
|
|
||||||
def get_run_context(request: Request) -> RunContext:
|
def get_run_context(request: Request) -> RunContext:
|
||||||
"""Build a :class:`RunContext` from ``app.state`` singletons.
|
"""Build a :class:`RunContext` from ``app.state`` singletons.
|
||||||
|
|
||||||
@ -1306,12 +1256,10 @@ def get_run_context(request: Request) -> RunContext:
|
|||||||
``event_store`` / ``run_events_config`` pair stays frozen to the snapshot
|
``event_store`` / ``run_events_config`` pair stays frozen to the snapshot
|
||||||
captured in :func:`langgraph_runtime` so callers never see a store bound to
|
captured in :func:`langgraph_runtime` so callers never see a store bound to
|
||||||
one backend paired with a config pointing at another.
|
one backend paired with a config pointing at another.
|
||||||
|
|
||||||
Loading ``app_config`` stats and hashes ``config.yaml`` (and reparses it
|
|
||||||
after an edit), so async callers build the context with
|
|
||||||
``asyncio.to_thread``.
|
|
||||||
"""
|
"""
|
||||||
agent_runs = get_agent_runs(request)
|
host = getattr(request.app.state, "agent_runs_host", None)
|
||||||
|
# Internal/channel and PAT runs deliberately receive no retained delegation.
|
||||||
|
agent_runs = host.bind(request) if host is not None else None
|
||||||
checkpointer = get_checkpointer(request)
|
checkpointer = get_checkpointer(request)
|
||||||
scheduled_task_service = getattr(request.app.state, "scheduled_task_service", None)
|
scheduled_task_service = getattr(request.app.state, "scheduled_task_service", None)
|
||||||
return RunContext(
|
return RunContext(
|
||||||
@ -1324,13 +1272,11 @@ def get_run_context(request: Request) -> RunContext:
|
|||||||
checkpoint_snapshot_frequency=getattr(request.app.state, "checkpoint_snapshot_frequency", None),
|
checkpoint_snapshot_frequency=getattr(request.app.state, "checkpoint_snapshot_frequency", None),
|
||||||
thread_store=get_thread_store(request),
|
thread_store=get_thread_store(request),
|
||||||
mcp_task_repo=getattr(request.app.state, "mcp_task_repo", None),
|
mcp_task_repo=getattr(request.app.state, "mcp_task_repo", None),
|
||||||
approval_store=getattr(request.app.state, "approval_store", None),
|
|
||||||
app_config=get_config(),
|
app_config=get_config(),
|
||||||
extensions=getattr(request.app.state, "extensions", None),
|
extensions=getattr(request.app.state, "extensions", None),
|
||||||
on_run_admitted=_run_admission_hook(checkpointer=checkpointer),
|
on_run_admitted=_run_admission_hook(checkpointer=checkpointer),
|
||||||
on_run_completed=_run_completion_hook(
|
on_run_completed=_run_completion_hook(
|
||||||
scheduled_task_service=scheduled_task_service,
|
scheduled_task_service=scheduled_task_service,
|
||||||
batch_continuations=getattr(request.app.state, "batch_continuations", None),
|
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@ -18,7 +18,7 @@ from pydantic import TypeAdapter, ValidationError
|
|||||||
|
|
||||||
from app.gateway.auth_disabled import AUTH_SOURCE_AUTH_DISABLED, AUTH_SOURCE_SESSION, get_auth_disabled_user, is_auth_disabled
|
from app.gateway.auth_disabled import AUTH_SOURCE_AUTH_DISABLED, AUTH_SOURCE_SESSION, get_auth_disabled_user, is_auth_disabled
|
||||||
from app.gateway.authz import AuthContext, resolve_route_permissions
|
from app.gateway.authz import AuthContext, resolve_route_permissions
|
||||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, make_origin
|
from deerflow.runtime.run_origin import make_origin
|
||||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||||
from deerflow.utils.thread_id import ThreadId
|
from deerflow.utils.thread_id import ThreadId
|
||||||
|
|
||||||
@ -73,32 +73,15 @@ class GatewayAgentRunsHost:
|
|||||||
return None
|
return None
|
||||||
return _BoundAgentRuns(self, str(user.id), user.token_version, source, frozenset(auth.permissions))
|
return _BoundAgentRuns(self, str(user.id), user.token_version, source, frozenset(auth.permissions))
|
||||||
|
|
||||||
def capture_batch_authority(self, runs, *, user_id: str) -> dict:
|
|
||||||
"""Capture only a native batch's existing host grant; never browser input."""
|
|
||||||
if not isinstance(runs, _BoundAgentRuns) or runs._host is not self or runs._user_id != user_id or not self.active:
|
|
||||||
raise AgentRunError(403, "Batch follow-up requires its originating Agent run authority")
|
|
||||||
return {"token_version": runs._token_version, "source": runs._source, "permissions": sorted(runs._permissions)}
|
|
||||||
|
|
||||||
async def restore_batch_authority(self, *, user_id: str, grant: dict):
|
|
||||||
"""Restore a persisted native batch grant with its original permission ceiling."""
|
|
||||||
source, version, permissions = grant.get("source"), grant.get("token_version"), grant.get("permissions")
|
|
||||||
if source not in (AUTH_SOURCE_SESSION, AUTH_SOURCE_AUTH_DISABLED) or type(version) is not int or not isinstance(permissions, list) or any(not isinstance(value, str) for value in permissions):
|
|
||||||
raise AgentRunError(403, "Invalid batch follow-up authority")
|
|
||||||
user = get_auth_disabled_user() if source == AUTH_SOURCE_AUTH_DISABLED and is_auth_disabled() else await self.load_user(user_id) if source == AUTH_SOURCE_SESSION and self.load_user is not None else None
|
|
||||||
if user is None or str(user.id) != user_id or user.token_version != version or user.needs_setup or getattr(user, "disabled", False):
|
|
||||||
raise AgentRunError(403, "Batch follow-up authorization revoked")
|
|
||||||
return _BoundAgentRuns(self, user_id, version, source, frozenset(permissions), origin=make_origin("subagent_batch"))
|
|
||||||
|
|
||||||
|
|
||||||
class _BoundAgentRuns:
|
class _BoundAgentRuns:
|
||||||
def __init__(self, host, user_id: str, token_version: int, source: str, permissions: frozenset[str], plugin_namespace: str | None = None, origin: dict | None = None):
|
def __init__(self, host, user_id: str, token_version: int, source: str, permissions: frozenset[str], plugin_namespace: str | None = None):
|
||||||
self._host = host
|
self._host = host
|
||||||
self._user_id = user_id
|
self._user_id = user_id
|
||||||
self._token_version = token_version
|
self._token_version = token_version
|
||||||
self._source = source
|
self._source = source
|
||||||
self._permissions = permissions
|
self._permissions = permissions
|
||||||
self._plugin_namespace = plugin_namespace
|
self._plugin_namespace = plugin_namespace
|
||||||
self._origin = origin
|
|
||||||
|
|
||||||
def for_plugin(self, namespace: str):
|
def for_plugin(self, namespace: str):
|
||||||
if not isinstance(namespace, str) or not namespace or len(namespace) > 96 or ":" in namespace:
|
if not isinstance(namespace, str) or not namespace or len(namespace) > 96 or ":" in namespace:
|
||||||
@ -130,7 +113,7 @@ class _BoundAgentRuns:
|
|||||||
request.state.auth = AuthContext(user, list(permissions))
|
request.state.auth = AuthContext(user, list(permissions))
|
||||||
# Host-set, never HTTP input: threads and runs created through this
|
# Host-set, never HTTP input: threads and runs created through this
|
||||||
# handle are marked as extension work (activity feed, unread state).
|
# handle are marked as extension work (activity feed, unread state).
|
||||||
request.state.run_origin = self._origin or make_origin("extension", namespace=self._plugin_namespace)
|
request.state.run_origin = make_origin("extension", namespace=self._plugin_namespace)
|
||||||
token = set_current_user(user)
|
token = set_current_user(user)
|
||||||
try:
|
try:
|
||||||
if thread_id is not None:
|
if thread_id is not None:
|
||||||
@ -163,56 +146,13 @@ class _BoundAgentRuns:
|
|||||||
raise AgentRunError(409, "Host assembly evidence unavailable")
|
raise AgentRunError(409, "Host assembly evidence unavailable")
|
||||||
return copy.deepcopy(evidence)
|
return copy.deepcopy(evidence)
|
||||||
|
|
||||||
async def validate_artifact_source(self, *, thread_id, run_id, idempotency_key):
|
async def read_artifact(self, *, thread_id, path):
|
||||||
from app.gateway.deps import get_run_manager
|
from app.gateway.extension_artifacts import read_output_snapshot
|
||||||
from app.gateway.extension_artifacts import validate_capture_support
|
|
||||||
from app.gateway.routers.thread_runs import _scope_http_run_idempotency_key
|
|
||||||
from deerflow.utils.file_io import await_drained
|
|
||||||
|
|
||||||
if self._plugin_namespace is None or not isinstance(idempotency_key, str) or not 1 <= len(idempotency_key) <= 200:
|
|
||||||
raise AgentRunError(422, "Artifact registration requires a plugin-scoped execution key")
|
|
||||||
async with self._request("runs:read", thread_id) as request:
|
|
||||||
run = await get_run_manager(request).get(run_id, user_id=self._user_id)
|
|
||||||
key = _scope_http_run_idempotency_key(request, thread_id, f"extension:{self._plugin_namespace}:{idempotency_key}")
|
|
||||||
origin = (run.metadata or {}).get(DEERFLOW_ORIGIN_KEY) if run is not None else None
|
|
||||||
if run is None or run.thread_id != thread_id or run.status != "running" or run.idempotency_key != key or origin != make_origin("extension", namespace=self._plugin_namespace):
|
|
||||||
raise AgentRunError(422, "Register files during the executing team run")
|
|
||||||
async with self._request("threads:read", thread_id) as request:
|
|
||||||
await await_drained(validate_capture_support(request, self._user_id, thread_id))
|
|
||||||
|
|
||||||
async def read_artifact(self, *, thread_id, path, expected_checkpoint_id=None):
|
|
||||||
from app.gateway.extension_artifacts import read_snapshot
|
|
||||||
from app.gateway.routers.artifacts import reserve_artifact_write
|
from app.gateway.routers.artifacts import reserve_artifact_write
|
||||||
from deerflow.runtime.runs.manager import ConflictError
|
|
||||||
from deerflow.utils.file_io import await_drained
|
|
||||||
|
|
||||||
async with self._request("threads:read", thread_id) as request:
|
async with self._request("threads:read", thread_id) as request:
|
||||||
try:
|
|
||||||
async with reserve_artifact_write(request, thread_id, user_id=self._user_id):
|
async with reserve_artifact_write(request, thread_id, user_id=self._user_id):
|
||||||
if expected_checkpoint_id is not None:
|
return await asyncio.to_thread(read_output_snapshot, self._user_id, thread_id, path)
|
||||||
if not isinstance(expected_checkpoint_id, str) or not expected_checkpoint_id:
|
|
||||||
raise AgentRunError(422, "Invalid producing checkpoint")
|
|
||||||
state = await self.get_state(thread_id=thread_id)
|
|
||||||
if state.get("checkpoint_id") != expected_checkpoint_id:
|
|
||||||
raise AgentRunError(412, "Producing conversation changed before artifact capture")
|
|
||||||
return await await_drained(read_snapshot(request, self._user_id, thread_id, path))
|
|
||||||
except ConflictError as exc:
|
|
||||||
raise AgentRunError(409, "Producing conversation is busy") from exc
|
|
||||||
|
|
||||||
async def import_artifact(self, *, thread_id, filename, content):
|
|
||||||
from app.gateway.extension_artifacts import import_snapshot
|
|
||||||
from app.gateway.routers.artifacts import reserve_artifact_write
|
|
||||||
from deerflow.runtime.runs.manager import ConflictError
|
|
||||||
from deerflow.utils.file_io import await_drained
|
|
||||||
|
|
||||||
if self._plugin_namespace is None:
|
|
||||||
raise AgentRunError(422, "Artifact import requires a plugin-scoped capability")
|
|
||||||
async with self._request("threads:write", thread_id) as request:
|
|
||||||
try:
|
|
||||||
async with reserve_artifact_write(request, thread_id, user_id=self._user_id):
|
|
||||||
return await await_drained(import_snapshot(request, self._user_id, thread_id, self._plugin_namespace, filename, content))
|
|
||||||
except ConflictError as exc:
|
|
||||||
raise AgentRunError(409, "Recipient conversation is busy") from exc
|
|
||||||
|
|
||||||
async def configuration_revision(self, *, assistant_ids: list[str], durable: bool = False) -> str:
|
async def configuration_revision(self, *, assistant_ids: list[str], durable: bool = False) -> str:
|
||||||
from deerflow.config.agents_config import load_agent_config, load_agent_soul
|
from deerflow.config.agents_config import load_agent_config, load_agent_soul
|
||||||
@ -258,8 +198,7 @@ class _BoundAgentRuns:
|
|||||||
response = await create_thread(ThreadCreateRequest(assistant_id=assistant_id, thread_id=thread_id, metadata=_json_object(metadata) or {}), request)
|
response = await create_thread(ThreadCreateRequest(assistant_id=assistant_id, thread_id=thread_id, metadata=_json_object(metadata) or {}), request)
|
||||||
return response.thread_id
|
return response.thread_id
|
||||||
|
|
||||||
async def _start(self, *, thread_id, input=None, resume=None, context=None, idempotency_key=None, execution_fence=None, artifact_imports=None):
|
async def _start(self, *, thread_id, input=None, resume=None, context=None, idempotency_key=None, execution_fence=None):
|
||||||
from app.gateway.extension_artifacts import validate_import_receipts
|
|
||||||
from app.gateway.routers.thread_runs import create_run
|
from app.gateway.routers.thread_runs import create_run
|
||||||
from app.gateway.run_models import RunCreateRequest
|
from app.gateway.run_models import RunCreateRequest
|
||||||
from app.gateway.services import resolve_thread_assistant_id
|
from app.gateway.services import resolve_thread_assistant_id
|
||||||
@ -270,16 +209,9 @@ class _BoundAgentRuns:
|
|||||||
run_input = _json_object(input)
|
run_input = _json_object(input)
|
||||||
run_context = _json_object(context)
|
run_context = _json_object(context)
|
||||||
command = _json_object({"resume": resume}) if resume is not None else None
|
command = _json_object({"resume": resume}) if resume is not None else None
|
||||||
imports = _json_object({"items": artifact_imports})["items"] if artifact_imports is not None else None
|
|
||||||
if imports is not None:
|
|
||||||
if self._plugin_namespace is None or resume is not None:
|
|
||||||
raise AgentRunError(422, "Artifact admission requires a plugin-scoped new run")
|
|
||||||
validate_import_receipts(imports)
|
|
||||||
if run_context is not None and any(key in run_context for key in ("agent_name", "is_bootstrap")):
|
if run_context is not None and any(key in run_context for key in ("agent_name", "is_bootstrap")):
|
||||||
raise AgentRunError(422, "Agent identity is bound to the thread")
|
raise AgentRunError(422, "Agent identity is bound to the thread")
|
||||||
async with self._request("runs:create", thread_id) as request:
|
async with self._request("runs:create", thread_id) as request:
|
||||||
if imports:
|
|
||||||
request.state.extension_artifact_imports = imports
|
|
||||||
if execution_fence is not None:
|
if execution_fence is not None:
|
||||||
if not isinstance(execution_fence, ExecutionFence) or self._plugin_namespace is None or execution_fence.namespace != self._plugin_namespace:
|
if not isinstance(execution_fence, ExecutionFence) or self._plugin_namespace is None or execution_fence.namespace != self._plugin_namespace:
|
||||||
raise AgentRunError(422, "Execution fence must belong to this plugin")
|
raise AgentRunError(422, "Execution fence must belong to this plugin")
|
||||||
@ -293,30 +225,8 @@ class _BoundAgentRuns:
|
|||||||
response = await create_run(thread_id, body, request, idempotency_key=f"extension:{self._plugin_namespace or ''}:{idempotency_key}" if idempotency_key is not None else None)
|
response = await create_run(thread_id, body, request, idempotency_key=f"extension:{self._plugin_namespace or ''}:{idempotency_key}" if idempotency_key is not None else None)
|
||||||
return _view(response)
|
return _view(response)
|
||||||
|
|
||||||
async def start(self, *, thread_id, input, context=None, idempotency_key=None, execution_fence=None, artifact_imports=None) -> AgentRun:
|
async def start(self, *, thread_id, input, context=None, idempotency_key=None, execution_fence=None) -> AgentRun:
|
||||||
return await self._start(thread_id=thread_id, input=input, context=context, idempotency_key=idempotency_key, execution_fence=execution_fence, artifact_imports=artifact_imports)
|
return await self._start(thread_id=thread_id, input=input, context=context, idempotency_key=idempotency_key, execution_fence=execution_fence)
|
||||||
|
|
||||||
async def start_batch_followup(self, *, thread_id, batch_id, input, context, config, execution_fence, goal_wakeup=None):
|
|
||||||
"""Native single follow-up; the existing SQL admission fences cancellation."""
|
|
||||||
from app.gateway.run_models import RunCreateRequest
|
|
||||||
from app.gateway.services import resolve_thread_assistant_id, start_run
|
|
||||||
from deerflow.subagents.batch_runtime import BATCH_CONTINUATION_NAMESPACE, batch_continuation_key
|
|
||||||
|
|
||||||
if self._origin != make_origin("subagent_batch") or execution_fence.namespace != BATCH_CONTINUATION_NAMESPACE or execution_fence.resource_id != batch_id:
|
|
||||||
raise AgentRunError(403, "Invalid native batch continuation")
|
|
||||||
async with self._request("runs:create", thread_id) as request:
|
|
||||||
request.state.extension_execution_fence = execution_fence
|
|
||||||
assistant_id = await resolve_thread_assistant_id(request, thread_id, fail_closed=True)
|
|
||||||
body = RunCreateRequest(
|
|
||||||
assistant_id=assistant_id,
|
|
||||||
input=_json_object(input),
|
|
||||||
context=_json_object(context),
|
|
||||||
config=_json_object(config),
|
|
||||||
metadata={"batch_goal_wakeup": _json_object(goal_wakeup)} if goal_wakeup is not None else None,
|
|
||||||
on_disconnect="continue",
|
|
||||||
)
|
|
||||||
record = await start_run(body, thread_id, request, idempotency_key=batch_continuation_key(batch_id), require_existing_thread=True)
|
|
||||||
return _view(record)
|
|
||||||
|
|
||||||
async def resume(self, *, thread_id, resume, idempotency_key=None) -> AgentRun:
|
async def resume(self, *, thread_id, resume, idempotency_key=None) -> AgentRun:
|
||||||
if resume is None:
|
if resume is None:
|
||||||
@ -335,40 +245,6 @@ class _BoundAgentRuns:
|
|||||||
async with self._request("threads:read", thread_id) as request:
|
async with self._request("threads:read", thread_id) as request:
|
||||||
return (await get_thread_state(thread_id, request)).model_dump(mode="json")
|
return (await get_thread_state(thread_id, request)).model_dump(mode="json")
|
||||||
|
|
||||||
async def send_input(self, *, thread_id, run_id, text, mode, request_id, context=None):
|
|
||||||
from app.gateway.routers.thread_runs import create_run_input
|
|
||||||
from app.gateway.run_inputs import RunInputRequest
|
|
||||||
|
|
||||||
async with self._request("runs:create", thread_id) as request:
|
|
||||||
body = RunInputRequest(request_id=request_id, text=text, mode=mode, context=_json_object(context) or {})
|
|
||||||
body = body.model_copy(update={"request_id": hashlib.sha256(f"{self._plugin_namespace or 'host'}:{body.request_id}".encode()).hexdigest()})
|
|
||||||
return await create_run_input(thread_id, run_id, body, request)
|
|
||||||
|
|
||||||
async def dispatch_input(self, *, thread_id, run_id, input_id):
|
|
||||||
from app.gateway.run_inputs import dispatch_one_input
|
|
||||||
|
|
||||||
async with self._request("runs:create", thread_id) as request:
|
|
||||||
return await dispatch_one_input(thread_id, run_id, input_id, request)
|
|
||||||
|
|
||||||
async def list_inputs(self, *, thread_id):
|
|
||||||
from app.gateway.routers.thread_runs import get_thread_inputs
|
|
||||||
|
|
||||||
async with self._request("threads:read", thread_id) as request:
|
|
||||||
return await get_thread_inputs(thread_id, request)
|
|
||||||
|
|
||||||
async def authorize_control(self, *, thread_id, run_id, clear_goal=False):
|
|
||||||
from app.gateway.deps import get_run_manager
|
|
||||||
|
|
||||||
async with self._request("runs:cancel", thread_id) as request:
|
|
||||||
run = await get_run_manager(request).get(run_id, user_id=self._user_id)
|
|
||||||
if run is None or run.thread_id != thread_id:
|
|
||||||
raise AgentRunError(404, "Run unavailable")
|
|
||||||
if run.status not in ("pending", "running"):
|
|
||||||
raise AgentRunError(409, "Run already stopped")
|
|
||||||
if clear_goal:
|
|
||||||
async with self._request("threads:write", thread_id):
|
|
||||||
pass
|
|
||||||
|
|
||||||
async def cancel(self, *, thread_id, run_id, execution_fence=None) -> None:
|
async def cancel(self, *, thread_id, run_id, execution_fence=None) -> None:
|
||||||
from app.gateway.routers.thread_runs import cancel_run
|
from app.gateway.routers.thread_runs import cancel_run
|
||||||
|
|
||||||
|
|||||||
@ -1,243 +1,24 @@
|
|||||||
"""Bounded output capture without symlink traversal or mutable-path rereads."""
|
"""Bounded output capture without symlink traversal or mutable-path rereads."""
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import logging
|
|
||||||
import os
|
import os
|
||||||
import re
|
|
||||||
import stat
|
import stat
|
||||||
import uuid
|
|
||||||
from collections.abc import AsyncIterator
|
|
||||||
from contextlib import AsyncExitStack, asynccontextmanager
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from deerflow_extension_api import AgentRunError
|
from deerflow_extension_api import AgentRunError
|
||||||
from fastapi import Request
|
|
||||||
|
|
||||||
from app.gateway.authz import try_acquire_sandbox_for_request
|
|
||||||
from app.gateway.path_utils import normalize_outputs_virtual_path
|
from app.gateway.path_utils import normalize_outputs_virtual_path
|
||||||
from deerflow.config.paths import get_paths
|
from deerflow.config.paths import get_paths
|
||||||
from deerflow.sandbox.sandbox import Sandbox
|
|
||||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
|
||||||
|
|
||||||
MAX_BYTES = 2 * 1024 * 1024
|
MAX_BYTES = 2 * 1024 * 1024
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def validate_import_receipts(values: object) -> None:
|
def read_output_snapshot(user_id, thread_id, path):
|
||||||
"""Reject malformed receipts or more than four bounded imports with 422."""
|
|
||||||
if not isinstance(values, list) or len(values) > 4:
|
|
||||||
raise AgentRunError(422, "Choose at most four imported artifacts")
|
|
||||||
for value in values:
|
|
||||||
if not isinstance(value, dict) or set(value) != {"filename", "path", "sha256", "size"}:
|
|
||||||
raise AgentRunError(422, "Invalid artifact import receipt")
|
|
||||||
filename = value["filename"]
|
|
||||||
if not isinstance(filename, str) or not filename or filename in (".", "..") or any(c in filename for c in "/\\") or any(ord(c) < 32 for c in filename):
|
|
||||||
raise AgentRunError(422, "Invalid artifact import filename")
|
|
||||||
try:
|
|
||||||
valid = len(filename.encode()) <= 255
|
|
||||||
except UnicodeError:
|
|
||||||
valid = False
|
|
||||||
if (
|
|
||||||
not valid
|
|
||||||
or value["path"] != "/mnt/user-data/uploads/" + filename
|
|
||||||
or not isinstance(value["sha256"], str)
|
|
||||||
or re.fullmatch(r"[0-9a-f]{64}", value["sha256"]) is None
|
|
||||||
or type(value["size"]) is not int
|
|
||||||
or not 0 <= value["size"] <= MAX_BYTES
|
|
||||||
):
|
|
||||||
raise AgentRunError(422, "Invalid artifact import receipt")
|
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
|
||||||
async def snapshot_sandbox(request: Request, user_id: str, thread_id: str) -> AsyncIterator[Sandbox | None]:
|
|
||||||
"""Yield a remote sandbox lease, or None for host-mounted thread data.
|
|
||||||
|
|
||||||
Permission denial raises 403 and an unavailable sandbox raises 503. Release
|
|
||||||
drains on cancellation; exact snapshots never fall back after a denied lease.
|
|
||||||
"""
|
|
||||||
from app.gateway.routers.uploads import _uses_thread_data_mounts
|
|
||||||
from deerflow.config.app_config import get_app_config
|
|
||||||
from deerflow.utils.file_io import await_drained, run_file_io
|
|
||||||
|
|
||||||
provider = get_sandbox_provider()
|
|
||||||
lease = None
|
|
||||||
try:
|
|
||||||
if not _uses_thread_data_mounts(provider):
|
|
||||||
lease = await try_acquire_sandbox_for_request(request, provider, thread_id, user_id=user_id, app_config=await run_file_io(get_app_config), owner_prefix="gateway:artifact-snapshot", release_on_last=True)
|
|
||||||
if lease.denied:
|
|
||||||
raise AgentRunError(403, "Sandbox permission denied for artifact snapshots")
|
|
||||||
if lease.sandbox is None:
|
|
||||||
raise AgentRunError(503, "Sandbox unavailable for artifact snapshots")
|
|
||||||
yield lease.sandbox if lease is not None else None
|
|
||||||
finally:
|
|
||||||
if lease is not None:
|
|
||||||
await await_drained(lease.release())
|
|
||||||
|
|
||||||
|
|
||||||
async def validate_capture_support(request: Request, user_id: str, thread_id: str) -> None:
|
|
||||||
"""Require POSIX capture and probe remote support before registration.
|
|
||||||
|
|
||||||
Raise 422 for unsupported capture, retaining lease authorization failures.
|
|
||||||
"""
|
|
||||||
from deerflow.sandbox.artifact_snapshot import validate_snapshot_support
|
|
||||||
from deerflow.utils.file_io import run_file_io
|
|
||||||
|
|
||||||
if os.name != "posix":
|
|
||||||
raise AgentRunError(422, "Artifact capture requires POSIX")
|
|
||||||
async with snapshot_sandbox(request, user_id, thread_id) as sandbox:
|
|
||||||
if sandbox is not None:
|
|
||||||
await run_file_io(validate_snapshot_support, sandbox)
|
|
||||||
|
|
||||||
|
|
||||||
def validate_import(filename: object, content: object) -> None:
|
|
||||||
"""Reject unsafe filenames, non-POSIX hosts or content over 2 MiB with 422."""
|
|
||||||
if os.name != "posix" or not isinstance(filename, str) or not filename or filename in (".", "..") or any(c in filename for c in "/\\") or any(ord(c) < 32 for c in filename):
|
|
||||||
raise AgentRunError(422, "Use a plain artifact filename on POSIX")
|
|
||||||
try:
|
|
||||||
valid = len(filename.encode()) <= 160
|
|
||||||
except UnicodeError:
|
|
||||||
valid = False
|
|
||||||
if not valid or not isinstance(content, bytes) or len(content) > MAX_BYTES:
|
|
||||||
raise AgentRunError(422, "Artifact filename is too long or content exceeds 2 MiB")
|
|
||||||
|
|
||||||
|
|
||||||
def write_import_snapshot(user_id: str, thread_id: str, namespace: str, filename: str, content: bytes) -> dict[str, Any]:
|
|
||||||
"""Publish a confined content-addressed upload and return its import receipt.
|
|
||||||
|
|
||||||
Reuse only identical regular files; retain existing edits and raise 422 for
|
|
||||||
unsafe or changed destinations. Run filesystem work off the event loop.
|
|
||||||
"""
|
|
||||||
validate_import(filename, content)
|
|
||||||
digest = hashlib.sha256(content).hexdigest()
|
|
||||||
identity = hashlib.sha256(namespace.encode() + b"\0" + filename.encode() + b"\0" + content).hexdigest()[:32]
|
|
||||||
destination = f"artifact-{identity}-{filename}"
|
|
||||||
paths = get_paths()
|
|
||||||
root = paths.base_dir.resolve()
|
|
||||||
relative = paths.sandbox_uploads_dir(thread_id, user_id=user_id).relative_to(paths.base_dir)
|
|
||||||
directory = os.open(root, os.O_RDONLY | os.O_DIRECTORY)
|
|
||||||
temporary = ".upload-" + uuid.uuid4().hex + ".part"
|
|
||||||
created = False
|
|
||||||
try:
|
|
||||||
for part in relative.parts:
|
|
||||||
try:
|
|
||||||
os.mkdir(part, mode=0o755, dir_fd=directory)
|
|
||||||
except FileExistsError:
|
|
||||||
pass
|
|
||||||
next_dir = os.open(part, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=directory)
|
|
||||||
os.close(directory)
|
|
||||||
directory = next_dir
|
|
||||||
fd = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=directory)
|
|
||||||
created = True
|
|
||||||
with os.fdopen(fd, "wb") as target:
|
|
||||||
target.write(content)
|
|
||||||
target.flush()
|
|
||||||
os.fsync(target.fileno())
|
|
||||||
os.fchmod(target.fileno(), 0o644)
|
|
||||||
try:
|
|
||||||
os.link(temporary, destination, src_dir_fd=directory, dst_dir_fd=directory, follow_symlinks=False)
|
|
||||||
except FileExistsError:
|
|
||||||
pass
|
|
||||||
os.unlink(temporary, dir_fd=directory)
|
|
||||||
created = False
|
|
||||||
fd = os.open(destination, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)
|
|
||||||
with os.fdopen(fd, "rb") as existing:
|
|
||||||
before = os.fstat(existing.fileno())
|
|
||||||
if not stat.S_ISREG(before.st_mode) or before.st_nlink != 1 or before.st_size != len(content):
|
|
||||||
raise AgentRunError(422, "Imported artifact was changed; existing file retained")
|
|
||||||
data = existing.read(MAX_BYTES + 1)
|
|
||||||
after = os.fstat(existing.fileno())
|
|
||||||
if data != content or (before.st_ino, before.st_size, before.st_mtime_ns, before.st_ctime_ns) != (after.st_ino, after.st_size, after.st_mtime_ns, after.st_ctime_ns):
|
|
||||||
raise AgentRunError(422, "Imported artifact was changed; existing file retained")
|
|
||||||
return {"filename": destination, "path": "/mnt/user-data/uploads/" + destination, "sha256": digest, "size": len(content)}
|
|
||||||
except OSError as exc:
|
|
||||||
raise AgentRunError(422, "Artifact import destination unavailable or unsafe") from exc
|
|
||||||
finally:
|
|
||||||
try:
|
|
||||||
if created:
|
|
||||||
os.unlink(temporary, dir_fd=directory)
|
|
||||||
except OSError:
|
|
||||||
logger.warning("Failed to clean up an abandoned artifact import staging file", exc_info=True)
|
|
||||||
finally:
|
|
||||||
os.close(directory)
|
|
||||||
|
|
||||||
|
|
||||||
async def import_snapshot(request: Request, user_id: str, thread_id: str, namespace: str, filename: str, content: bytes) -> dict[str, Any]:
|
|
||||||
"""Return a receipt after publishing identical host and remote uploads.
|
|
||||||
|
|
||||||
Retain existing edits; unsafe/changed snapshots raise 422 and lease failures
|
|
||||||
keep their authorization/availability status. Cancellation drains file I/O.
|
|
||||||
"""
|
|
||||||
from deerflow.sandbox.artifact_snapshot import publish_snapshot
|
|
||||||
from deerflow.utils.file_io import run_file_io
|
|
||||||
|
|
||||||
validate_import(filename, content)
|
|
||||||
async with snapshot_sandbox(request, user_id, thread_id) as sandbox:
|
|
||||||
result = await run_file_io(write_import_snapshot, user_id, thread_id, namespace, filename, content)
|
|
||||||
if sandbox is not None:
|
|
||||||
await run_file_io(publish_snapshot, sandbox, result["filename"], content)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
async def read_snapshot(request: Request, user_id: str, thread_id: str, path: str, *, max_bytes: int = MAX_BYTES) -> bytes:
|
|
||||||
"""Capture bounded output bytes through host storage or an authorized lease.
|
|
||||||
|
|
||||||
Non-output paths raise HTTP 400; unsafe files raise 422 and host mutation
|
|
||||||
during capture raises 409. Remote snapshot and lease errors retain their
|
|
||||||
status. Cancellation drains I/O.
|
|
||||||
"""
|
|
||||||
from deerflow.sandbox.artifact_snapshot import read_snapshot as read_remote
|
|
||||||
from deerflow.utils.file_io import run_file_io
|
|
||||||
|
|
||||||
path = normalize_outputs_virtual_path(path)
|
|
||||||
async with snapshot_sandbox(request, user_id, thread_id) as sandbox:
|
|
||||||
if sandbox is not None:
|
|
||||||
return await run_file_io(read_remote, sandbox, path.removeprefix("/mnt/user-data/outputs/"), max_bytes=max_bytes)
|
|
||||||
return await run_file_io(read_output_snapshot, user_id, thread_id, path, max_bytes=max_bytes)
|
|
||||||
|
|
||||||
|
|
||||||
async def verify_imports(request: Request, user_id: str, thread_id: str, receipts: list[dict[str, Any]], *, lifetime: AsyncExitStack | None = None) -> None:
|
|
||||||
"""Verify receipt bytes after new admission and before Agent construction.
|
|
||||||
|
|
||||||
A supplied lifetime retains the remote lease through execution; otherwise it
|
|
||||||
ends here. Missing, changed or inaccessible copies raise an artifact-specific
|
|
||||||
422. Cancellation drains file I/O before the enclosing lease is released.
|
|
||||||
"""
|
|
||||||
from deerflow.sandbox.artifact_snapshot import verify_snapshot
|
|
||||||
from deerflow.utils.file_io import run_file_io
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with AsyncExitStack() as temporary:
|
|
||||||
# A run retains its remote client through execution: releasing it
|
|
||||||
# after preflight could let the provider recycle the verified VM.
|
|
||||||
sandbox = await (lifetime or temporary).enter_async_context(snapshot_sandbox(request, user_id, thread_id))
|
|
||||||
for receipt in receipts:
|
|
||||||
data = await run_file_io(_read_host_snapshot, user_id, thread_id, receipt["filename"], "uploads")
|
|
||||||
if len(data) != receipt["size"] or hashlib.sha256(data).hexdigest() != receipt["sha256"]:
|
|
||||||
raise AgentRunError(422, "Imported copy changed")
|
|
||||||
if sandbox is not None:
|
|
||||||
await run_file_io(verify_snapshot, sandbox, receipt["filename"], receipt["sha256"], receipt["size"])
|
|
||||||
except (AgentRunError, OSError) as exc:
|
|
||||||
raise AgentRunError(422, "Attached artifact copy is unavailable or changed; inspect the team artifacts and send a new request") from exc
|
|
||||||
|
|
||||||
|
|
||||||
def read_output_snapshot(user_id: str, thread_id: str, path: str, *, max_bytes: int = MAX_BYTES) -> bytes:
|
|
||||||
"""Read bounded host output bytes without following links.
|
|
||||||
|
|
||||||
Non-output paths raise HTTP 400; unsafe files raise 422 and mutation during
|
|
||||||
capture raises 409. Run filesystem work off the event loop.
|
|
||||||
"""
|
|
||||||
if os.name != "posix" or not isinstance(path, str):
|
if os.name != "posix" or not isinstance(path, str):
|
||||||
raise AgentRunError(422, "Output capture requires POSIX and an output path")
|
raise AgentRunError(422, "Output capture requires POSIX and an output path")
|
||||||
path = normalize_outputs_virtual_path(path)
|
path = normalize_outputs_virtual_path(path)
|
||||||
return _read_host_snapshot(user_id, thread_id, path.removeprefix("/mnt/user-data/outputs/"), "outputs", max_bytes=max_bytes)
|
|
||||||
|
|
||||||
|
|
||||||
def _read_host_snapshot(user_id: str, thread_id: str, path: str, area: str, *, max_bytes: int = MAX_BYTES) -> bytes:
|
|
||||||
paths = get_paths()
|
paths = get_paths()
|
||||||
root = paths.base_dir.resolve()
|
root = paths.base_dir.resolve()
|
||||||
base = paths.sandbox_outputs_dir(thread_id, user_id=user_id) if area == "outputs" else paths.sandbox_uploads_dir(thread_id, user_id=user_id)
|
relative = paths.sandbox_outputs_dir(thread_id, user_id=user_id).relative_to(paths.base_dir)
|
||||||
relative = base.relative_to(paths.base_dir)
|
parts = (*relative.parts, *path.removeprefix("/mnt/user-data/outputs/").split("/"))
|
||||||
parts = (*relative.parts, *path.split("/"))
|
|
||||||
directory = os.open(root, os.O_RDONLY | os.O_DIRECTORY)
|
directory = os.open(root, os.O_RDONLY | os.O_DIRECTORY)
|
||||||
try:
|
try:
|
||||||
for part in parts[:-1]:
|
for part in parts[:-1]:
|
||||||
@ -247,11 +28,11 @@ def _read_host_snapshot(user_id: str, thread_id: str, path: str, area: str, *, m
|
|||||||
fd = os.open(parts[-1], os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)
|
fd = os.open(parts[-1], os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)
|
||||||
with os.fdopen(fd, "rb") as source:
|
with os.fdopen(fd, "rb") as source:
|
||||||
before = os.fstat(source.fileno())
|
before = os.fstat(source.fileno())
|
||||||
if not stat.S_ISREG(before.st_mode) or before.st_size > max_bytes or before.st_nlink != 1:
|
if not stat.S_ISREG(before.st_mode) or before.st_size > MAX_BYTES:
|
||||||
raise AgentRunError(422, "Artifact must be a regular, unlinked file within the capture limit")
|
raise AgentRunError(422, "Artifact must be a regular file of at most 2 MiB")
|
||||||
content = source.read(max_bytes + 1)
|
content = source.read(MAX_BYTES + 1)
|
||||||
after = os.fstat(source.fileno())
|
after = os.fstat(source.fileno())
|
||||||
if len(content) > max_bytes or (before.st_size, before.st_mtime_ns, before.st_ctime_ns) != (after.st_size, after.st_mtime_ns, after.st_ctime_ns):
|
if len(content) > MAX_BYTES or (before.st_size, before.st_mtime_ns, before.st_ctime_ns) != (after.st_size, after.st_mtime_ns, after.st_ctime_ns):
|
||||||
raise AgentRunError(409, "Artifact changed during capture")
|
raise AgentRunError(409, "Artifact changed during capture")
|
||||||
return content
|
return content
|
||||||
except OSError as exc:
|
except OSError as exc:
|
||||||
|
|||||||
@ -16,12 +16,6 @@ installation id, with a 55-minute TTL so we refresh well before the
|
|||||||
60-minute GitHub limit. If two webhook deliveries land within a few
|
60-minute GitHub limit. If two webhook deliveries land within a few
|
||||||
seconds we'll mint two tokens — that's fine and well under any
|
seconds we'll mint two tokens — that's fine and well under any
|
||||||
rate limit.
|
rate limit.
|
||||||
|
|
||||||
Building the App JWT reads the private key and signs with it, so the
|
|
||||||
async mint runs it in a worker thread. Parsing a PEM key is the costly
|
|
||||||
step (tens of ms for RSA-2048, holding the GIL, so a worker thread alone
|
|
||||||
would still stall the event loop), so the parsed key is reused for as
|
|
||||||
long as the configured PEM text stays the same.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@ -29,11 +23,9 @@ from __future__ import annotations
|
|||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import threading
|
|
||||||
import time
|
import time
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
import jwt
|
import jwt
|
||||||
@ -75,12 +67,6 @@ _install_locks: dict[int, asyncio.Lock] = {}
|
|||||||
# insert the per-installation lock, never while we hold one.
|
# insert the per-installation lock, never while we hold one.
|
||||||
_install_locks_lock = asyncio.Lock()
|
_install_locks_lock = asyncio.Lock()
|
||||||
|
|
||||||
# The most recently parsed App private key, as ``(pem_text, key)``. A
|
|
||||||
# changed PEM (a rotated env var or key file) is parsed again on the next
|
|
||||||
# mint. The lock keeps concurrent worker-thread mints from parsing twice.
|
|
||||||
_signing_key: tuple[str, Any] | None = None
|
|
||||||
_signing_key_lock = threading.Lock()
|
|
||||||
|
|
||||||
|
|
||||||
async def _lock_for(installation_id: int) -> asyncio.Lock:
|
async def _lock_for(installation_id: int) -> asyncio.Lock:
|
||||||
"""Return the lock dedicated to ``installation_id``, creating on demand."""
|
"""Return the lock dedicated to ``installation_id``, creating on demand."""
|
||||||
@ -128,24 +114,6 @@ def load_app_private_key() -> str:
|
|||||||
return p.read_text(encoding="utf-8")
|
return p.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
def _app_signing_key() -> Any:
|
|
||||||
"""Return the App private key ready for RS256, parsing it only when it changed.
|
|
||||||
|
|
||||||
The PEM is still read on every call, so a rotated key is used by the
|
|
||||||
next mint without a restart.
|
|
||||||
"""
|
|
||||||
global _signing_key
|
|
||||||
pem = load_app_private_key()
|
|
||||||
with _signing_key_lock:
|
|
||||||
cached = _signing_key
|
|
||||||
if cached is not None and cached[0] == pem:
|
|
||||||
return cached[1]
|
|
||||||
# Same parsing (and the same errors) as passing the PEM to jwt.encode.
|
|
||||||
key = jwt.get_algorithm_by_name("RS256").prepare_key(pem)
|
|
||||||
_signing_key = (pem, key)
|
|
||||||
return key
|
|
||||||
|
|
||||||
|
|
||||||
def mint_app_jwt(*, now: float | None = None) -> str:
|
def mint_app_jwt(*, now: float | None = None) -> str:
|
||||||
"""Sign a short-lived JWT identifying this App to GitHub.
|
"""Sign a short-lived JWT identifying this App to GitHub.
|
||||||
|
|
||||||
@ -164,7 +132,7 @@ def mint_app_jwt(*, now: float | None = None) -> str:
|
|||||||
# numeric App id rendered as a decimal string.
|
# numeric App id rendered as a decimal string.
|
||||||
"iss": str(app_id()),
|
"iss": str(app_id()),
|
||||||
}
|
}
|
||||||
return jwt.encode(payload, _app_signing_key(), algorithm="RS256")
|
return jwt.encode(payload, load_app_private_key(), algorithm="RS256")
|
||||||
|
|
||||||
|
|
||||||
async def _request_new_installation_token(
|
async def _request_new_installation_token(
|
||||||
@ -173,10 +141,8 @@ async def _request_new_installation_token(
|
|||||||
client: httpx.AsyncClient | None = None,
|
client: httpx.AsyncClient | None = None,
|
||||||
) -> _CachedToken:
|
) -> _CachedToken:
|
||||||
"""Hit ``POST /app/installations/{id}/access_tokens`` once."""
|
"""Hit ``POST /app/installations/{id}/access_tokens`` once."""
|
||||||
# Reading the key file and signing are blocking; keep them off the loop.
|
|
||||||
app_jwt = await asyncio.to_thread(mint_app_jwt)
|
|
||||||
headers = {
|
headers = {
|
||||||
"Authorization": f"Bearer {app_jwt}",
|
"Authorization": f"Bearer {mint_app_jwt()}",
|
||||||
"Accept": "application/vnd.github+json",
|
"Accept": "application/vnd.github+json",
|
||||||
"X-GitHub-Api-Version": "2022-11-28",
|
"X-GitHub-Api-Version": "2022-11-28",
|
||||||
}
|
}
|
||||||
@ -258,11 +224,6 @@ async def mint_installation_token(
|
|||||||
|
|
||||||
|
|
||||||
def _clear_token_cache_for_tests() -> None:
|
def _clear_token_cache_for_tests() -> None:
|
||||||
"""Drop every cached token and the parsed key. Tests reach for this between cases."""
|
"""Drop every cached token. Tests reach for this between cases."""
|
||||||
global _signing_key
|
|
||||||
_token_cache.clear()
|
_token_cache.clear()
|
||||||
_install_locks.clear()
|
_install_locks.clear()
|
||||||
# Under the lock, so a mint parsing in a worker thread finishes first
|
|
||||||
# instead of storing its key after the reset.
|
|
||||||
with _signing_key_lock:
|
|
||||||
_signing_key = None
|
|
||||||
|
|||||||
@ -95,7 +95,7 @@ def _build_index(agents: list[tuple[str, AgentConfig]]) -> _Registry:
|
|||||||
# it is here, so this branch is unreachable. Keep the
|
# it is here, so this branch is unreachable. Keep the
|
||||||
# guard for type-checker happiness.
|
# guard for type-checker happiness.
|
||||||
continue
|
continue
|
||||||
index.setdefault((binding.repo.lower(), event), []).append(GitHubAgentMatch(user_id=user_id, agent=cfg, trigger=resolved))
|
index.setdefault((binding.repo, event), []).append(GitHubAgentMatch(user_id=user_id, agent=cfg, trigger=resolved))
|
||||||
return index
|
return index
|
||||||
|
|
||||||
|
|
||||||
@ -145,4 +145,4 @@ def lookup_agents(
|
|||||||
and the pre-resolved trigger config for this specific event, so the
|
and the pre-resolved trigger config for this specific event, so the
|
||||||
caller does not need to walk the agent's ``bindings`` again.
|
caller does not need to walk the agent's ``bindings`` again.
|
||||||
"""
|
"""
|
||||||
return registry.get((repo.lower(), event), [])
|
return registry.get((repo, event), [])
|
||||||
|
|||||||
@ -81,11 +81,6 @@ def _comment_body(event: str, payload: dict[str, Any]) -> str:
|
|||||||
|
|
||||||
def _author_login(event: str, payload: dict[str, Any]) -> str | None:
|
def _author_login(event: str, payload: dict[str, Any]) -> str | None:
|
||||||
"""Login of the human who triggered the event, for ``allow_authors``."""
|
"""Login of the human who triggered the event, for ``allow_authors``."""
|
||||||
# Object authors need not be the actor on edits, labels, or dismissals.
|
|
||||||
# Keep the object fallback for payloads without a sender identity.
|
|
||||||
sender_login = (payload.get("sender") or {}).get("login")
|
|
||||||
if isinstance(sender_login, str):
|
|
||||||
return sender_login
|
|
||||||
if event in ("issue_comment", "pull_request_review_comment"):
|
if event in ("issue_comment", "pull_request_review_comment"):
|
||||||
login = (payload.get("comment") or {}).get("user", {}).get("login")
|
login = (payload.get("comment") or {}).get("user", {}).get("login")
|
||||||
elif event == "pull_request":
|
elif event == "pull_request":
|
||||||
|
|||||||
@ -51,7 +51,6 @@ from sqlalchemy import text
|
|||||||
|
|
||||||
from deerflow.persistence.engine import get_engine
|
from deerflow.persistence.engine import get_engine
|
||||||
from deerflow.utils.file_io import await_drained
|
from deerflow.utils.file_io import await_drained
|
||||||
from deerflow.utils.http_policy import control_plane_trust_env
|
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from deerflow.config.app_config import AppConfig
|
from deerflow.config.app_config import AppConfig
|
||||||
@ -314,7 +313,10 @@ def _make_provisioner_client(provisioner_url: str) -> httpx.AsyncClient:
|
|||||||
``NO_PROXY``, so without this a private provisioner behind a proxy-only
|
``NO_PROXY``, so without this a private provisioner behind a proxy-only
|
||||||
egress would be reported unreachable on every probe.
|
egress would be reported unreachable on every probe.
|
||||||
"""
|
"""
|
||||||
return httpx.AsyncClient(timeout=_PROBE_TIMEOUT_SECONDS, trust_env=control_plane_trust_env(provisioner_url))
|
# Lazy: importing the aio_sandbox package pulls in the whole provider stack.
|
||||||
|
from deerflow.community.aio_sandbox.backend import sandbox_http_trust_env
|
||||||
|
|
||||||
|
return httpx.AsyncClient(timeout=_PROBE_TIMEOUT_SECONDS, trust_env=sandbox_http_trust_env(provisioner_url))
|
||||||
|
|
||||||
|
|
||||||
async def _probe_provisioner(provisioner_url: str | None) -> str:
|
async def _probe_provisioner(provisioner_url: str | None) -> str:
|
||||||
|
|||||||
@ -1,74 +0,0 @@
|
|||||||
"""Host-authorized history copies under durable thread-operation admission."""
|
|
||||||
|
|
||||||
from collections.abc import AsyncIterator
|
|
||||||
from contextlib import asynccontextmanager
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
from fastapi import Request
|
|
||||||
|
|
||||||
from app.gateway.deps import get_run_manager, get_thread_store
|
|
||||||
from app.gateway.internal_auth import get_trusted_internal_owner_user_id
|
|
||||||
from deerflow.persistence.run.sql import RunRepository
|
|
||||||
from deerflow.persistence.thread_meta.sql import ThreadMetaRepository
|
|
||||||
from deerflow.runtime.events.producer import RunEventProducerRejected, history_seed_authority, producer_scope
|
|
||||||
from deerflow.runtime.events.store.db import DbRunEventStore
|
|
||||||
from deerflow.runtime.runs.manager import RunRecord
|
|
||||||
from deerflow.runtime.runs.schemas import ThreadOperationKind
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HistorySeedReservation:
|
|
||||||
record: RunRecord
|
|
||||||
thread: dict | None
|
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
|
||||||
async def reserve_history_seed(request: Request, thread_id: str) -> AsyncIterator[HistorySeedReservation | None]:
|
|
||||||
"""Keep snapshot reads and seed writes exclusive with delete and admission."""
|
|
||||||
events = request.app.state.run_event_store
|
|
||||||
manager = getattr(request.app.state, "run_manager", None)
|
|
||||||
runs = getattr(manager, "_store", None)
|
|
||||||
threads = getattr(request.app.state, "thread_store", None)
|
|
||||||
if not isinstance(events, DbRunEventStore):
|
|
||||||
yield None
|
|
||||||
return
|
|
||||||
if not isinstance(runs, RunRepository) or not isinstance(threads, ThreadMetaRepository):
|
|
||||||
if events._producer_fencing == "strict":
|
|
||||||
raise RunEventProducerRejected("History seeding requires shared SQL lifecycle stores")
|
|
||||||
yield None
|
|
||||||
return
|
|
||||||
if runs._sf.kw.get("bind") is not events._sf.kw.get("bind") or threads._sf.kw.get("bind") is not events._sf.kw.get("bind"):
|
|
||||||
raise RunEventProducerRejected("History seeding requires shared SQL lifecycle stores")
|
|
||||||
async with get_run_manager(request).reserve_thread_operation(
|
|
||||||
thread_id,
|
|
||||||
kind=ThreadOperationKind.checkpoint_write,
|
|
||||||
user_id=get_trusted_internal_owner_user_id(request),
|
|
||||||
) as reservation:
|
|
||||||
yield HistorySeedReservation(reservation, await threads.get(thread_id, user_id=None))
|
|
||||||
|
|
||||||
|
|
||||||
async def write_history_seed(
|
|
||||||
request: Request,
|
|
||||||
events: list[dict],
|
|
||||||
*,
|
|
||||||
reservation: HistorySeedReservation | None,
|
|
||||||
seed_kind: str,
|
|
||||||
thread: dict | None = None,
|
|
||||||
assistant_id: str | None = None,
|
|
||||||
) -> None:
|
|
||||||
store = request.app.state.run_event_store
|
|
||||||
if reservation is None:
|
|
||||||
await store.put_batch(events)
|
|
||||||
return
|
|
||||||
if thread is None:
|
|
||||||
# Only the checkpoint migration may materialize missing legacy metadata.
|
|
||||||
# The caller read that checkpoint while holding this same reservation.
|
|
||||||
if seed_kind != "checkpoint":
|
|
||||||
raise RunEventProducerRejected("Branch seeding requires its original thread identity")
|
|
||||||
threads = get_thread_store(request)
|
|
||||||
thread = reservation.thread
|
|
||||||
if thread is None:
|
|
||||||
thread = await threads.create(reservation.record.thread_id, assistant_id=assistant_id, user_id=reservation.record.user_id)
|
|
||||||
authority = history_seed_authority(reservation.record, thread, seed_kind=seed_kind)
|
|
||||||
with producer_scope(authority):
|
|
||||||
await store.put_batch(events)
|
|
||||||
@ -1,351 +0,0 @@
|
|||||||
"""Server-side validation for HITL tool-approval resume and admission.
|
|
||||||
|
|
||||||
All checks live here (called from ``services.start_run``) so every entry point
|
|
||||||
— HTTP routes, extension host, internal launchers — shares one choke point.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
from collections.abc import Mapping
|
|
||||||
from typing import TYPE_CHECKING, Any
|
|
||||||
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
from app.gateway.checkpoint_lineage import hitl_approval_payloads
|
|
||||||
from deerflow.approvals.protocol import combine_approval_payloads
|
|
||||||
from deerflow.runtime.checkpoint_history import recent_checkpoint_history
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class ApprovalRequestIds(dict):
|
|
||||||
"""Identity map plus the validated LangGraph interrupt-ID resume map."""
|
|
||||||
|
|
||||||
resume_map: dict | None = None
|
|
||||||
|
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
|
||||||
from deerflow.runtime.runs.manager import ApprovalAnswerHandle
|
|
||||||
|
|
||||||
|
|
||||||
def _error(status: int, code: str, message: str) -> HTTPException:
|
|
||||||
return HTTPException(status_code=status, detail={"code": code, "message": message})
|
|
||||||
|
|
||||||
|
|
||||||
def _responses_of(command: Any) -> list[Any] | None:
|
|
||||||
if not isinstance(command, Mapping):
|
|
||||||
return None
|
|
||||||
resume = command.get("resume")
|
|
||||||
if not isinstance(resume, Mapping):
|
|
||||||
return None
|
|
||||||
responses = resume.get("responses")
|
|
||||||
return responses if isinstance(responses, list) else None
|
|
||||||
|
|
||||||
|
|
||||||
def _all_reject(responses: list[Any] | None) -> bool:
|
|
||||||
return bool(responses) and all(isinstance(r, Mapping) and r.get("option_id") == "reject" for r in responses)
|
|
||||||
|
|
||||||
|
|
||||||
async def has_any_pending_interrupt(accessor: Any, read_config: dict[str, Any]) -> bool:
|
|
||||||
"""Whether a recent checkpoint has an unconsumed interrupt write.
|
|
||||||
Used for generic (non-hitl) resume admission where snapshot.next may be
|
|
||||||
empty after a head-moving write. The newest resume/interrupt marker wins,
|
|
||||||
so an answered pause cannot be resurrected by its older interrupt write.
|
|
||||||
"""
|
|
||||||
async for checkpoint in recent_checkpoint_history(accessor.checkpointer, read_config):
|
|
||||||
writes = list(getattr(checkpoint, "pending_writes", ()) or ())
|
|
||||||
if any(len(write) >= 2 and write[1] == "__resume__" for write in writes):
|
|
||||||
return False
|
|
||||||
if any(len(write) >= 2 and write[1] == "__interrupt__" for write in writes):
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
async def _pending_interrupts_from_checkpoint_history(checkpointer: Any, read_config: dict[str, Any]) -> dict[str, list[Any]]:
|
|
||||||
"""Read pending interrupts without materializing a graph state snapshot."""
|
|
||||||
async for checkpoint in recent_checkpoint_history(checkpointer, read_config):
|
|
||||||
writes = list(getattr(checkpoint, "pending_writes", ()) or ())
|
|
||||||
if any(len(write) >= 2 and write[1] == "__resume__" for write in writes):
|
|
||||||
return {}
|
|
||||||
interrupt_writes = [write for write in writes if len(write) >= 3 and write[1] == "__interrupt__"]
|
|
||||||
if not interrupt_writes:
|
|
||||||
continue
|
|
||||||
grouped: dict[str, list[Any]] = {}
|
|
||||||
for write in interrupt_writes:
|
|
||||||
interrupts = write[2] if isinstance(write[2], (list, tuple)) else (write[2],)
|
|
||||||
grouped.setdefault(str(write[0]), []).extend(interrupts)
|
|
||||||
return grouped
|
|
||||||
return {}
|
|
||||||
|
|
||||||
|
|
||||||
async def has_waiting_for_input_run(request: Any, thread_id: str) -> bool:
|
|
||||||
"""Whether any run in the thread is paused for input (any interrupt kind)."""
|
|
||||||
from app.gateway.deps import get_run_store
|
|
||||||
|
|
||||||
try:
|
|
||||||
run_store = get_run_store(request)
|
|
||||||
except (HTTPException, AttributeError) as exc:
|
|
||||||
if isinstance(exc, AttributeError) or exc.status_code == 503:
|
|
||||||
return False
|
|
||||||
raise
|
|
||||||
runs = await run_store.list_by_thread(thread_id, limit=5)
|
|
||||||
return any(run.get("status") == "waiting_for_input" for run in runs)
|
|
||||||
|
|
||||||
|
|
||||||
async def mark_approval_answered(request: Any, thread_id: str, *, expected_request_ids: dict[str, str] | None = None) -> ApprovalAnswerHandle | None:
|
|
||||||
"""Mark the validated paused run as answered and return its recovery handle."""
|
|
||||||
from app.gateway.deps import get_run_manager, get_run_store
|
|
||||||
|
|
||||||
try:
|
|
||||||
run_manager = get_run_manager(request)
|
|
||||||
except (HTTPException, AttributeError) as exc:
|
|
||||||
if isinstance(exc, HTTPException) and exc.status_code != 503:
|
|
||||||
raise
|
|
||||||
# Lightweight compositions may provide only a store. Keep their
|
|
||||||
# persistence on RunManager's retry and in-memory mutation path too.
|
|
||||||
from deerflow.runtime import RunManager
|
|
||||||
|
|
||||||
try:
|
|
||||||
run_manager = RunManager(store=get_run_store(request))
|
|
||||||
except (HTTPException, AttributeError) as store_exc:
|
|
||||||
if isinstance(store_exc, AttributeError) or store_exc.status_code == 503:
|
|
||||||
return None
|
|
||||||
raise
|
|
||||||
return await run_manager.mark_approval_answered(thread_id, expected_request_ids=expected_request_ids)
|
|
||||||
|
|
||||||
|
|
||||||
async def _interrupt_from_checkpoint_history(accessor: Any, read_config: dict[str, Any], payload: dict[str, Any]) -> tuple[str, Any] | None:
|
|
||||||
"""Recover the original LangGraph task identity after the head has moved."""
|
|
||||||
if accessor is None:
|
|
||||||
return None
|
|
||||||
if getattr(accessor, "checkpointer", None) is None:
|
|
||||||
return None
|
|
||||||
async for checkpoint in recent_checkpoint_history(accessor.checkpointer, read_config):
|
|
||||||
writes = list(getattr(checkpoint, "pending_writes", ()) or ())
|
|
||||||
if any(len(write) >= 2 and write[1] == "__resume__" for write in writes):
|
|
||||||
return None
|
|
||||||
for write in writes:
|
|
||||||
if len(write) < 3 or write[1] != "__interrupt__":
|
|
||||||
continue
|
|
||||||
interrupts = write[2] if isinstance(write[2], (list, tuple)) else (write[2],)
|
|
||||||
for pending_interrupt in interrupts:
|
|
||||||
if getattr(pending_interrupt, "value", None) == payload:
|
|
||||||
return str(write[0]), pending_interrupt
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
async def find_pending_interrupts(
|
|
||||||
request: Any,
|
|
||||||
thread_id: str,
|
|
||||||
*,
|
|
||||||
accessor: Any = None,
|
|
||||||
read_config: dict[str, Any] | None = None,
|
|
||||||
snapshot: Any = None,
|
|
||||||
) -> dict[str, list[Any]]:
|
|
||||||
"""Find pending interrupts from an existing snapshot or raw checkpoints.
|
|
||||||
|
|
||||||
The live snapshot catches a pause before the worker has finalized the run;
|
|
||||||
raw pending writes cover head-moving writes and restricted mutation graphs;
|
|
||||||
the durable run record remains the final fallback.
|
|
||||||
"""
|
|
||||||
from langgraph.types import Interrupt
|
|
||||||
|
|
||||||
from app.gateway.checkpoint_lineage import task_interrupts_by_id
|
|
||||||
from app.gateway.deps import get_checkpointer, get_run_store
|
|
||||||
|
|
||||||
if snapshot is None and accessor is not None and read_config is not None:
|
|
||||||
snapshot = await accessor.aget(read_config)
|
|
||||||
if snapshot is not None:
|
|
||||||
grouped = task_interrupts_by_id(snapshot)
|
|
||||||
if grouped:
|
|
||||||
return grouped
|
|
||||||
|
|
||||||
try:
|
|
||||||
run_store = get_run_store(request)
|
|
||||||
except (HTTPException, AttributeError) as exc:
|
|
||||||
if not isinstance(exc, AttributeError) and exc.status_code != 503:
|
|
||||||
raise
|
|
||||||
runs = []
|
|
||||||
else:
|
|
||||||
runs = await run_store.list_by_thread(thread_id, limit=10)
|
|
||||||
# A current loop snapshot is authoritative for ordinary polling. Update
|
|
||||||
# checkpoints can have moved the head past an unconsumed interrupt, and
|
|
||||||
# checkpoint-only compositions have no snapshot: retain recovery for both.
|
|
||||||
if (
|
|
||||||
snapshot is not None
|
|
||||||
and (getattr(snapshot, "metadata", None) or {}).get("source") == "loop"
|
|
||||||
and (getattr(snapshot, "config", None) or {}).get("configurable", {}).get("checkpoint_id")
|
|
||||||
and not (read_config or {}).get("configurable", {}).get("checkpoint_id")
|
|
||||||
and not any(run.get("status") == "waiting_for_input" for run in runs)
|
|
||||||
):
|
|
||||||
return {}
|
|
||||||
|
|
||||||
checkpointer = getattr(accessor, "checkpointer", None)
|
|
||||||
if checkpointer is None:
|
|
||||||
try:
|
|
||||||
checkpointer = get_checkpointer(request)
|
|
||||||
except (HTTPException, AttributeError) as exc:
|
|
||||||
if not isinstance(exc, AttributeError) and exc.status_code != 503:
|
|
||||||
raise
|
|
||||||
if checkpointer is not None:
|
|
||||||
grouped = await _pending_interrupts_from_checkpoint_history(
|
|
||||||
checkpointer,
|
|
||||||
read_config or {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}},
|
|
||||||
)
|
|
||||||
if grouped:
|
|
||||||
return grouped
|
|
||||||
|
|
||||||
for run in runs:
|
|
||||||
if run.get("status") != "waiting_for_input":
|
|
||||||
continue
|
|
||||||
payload = (run.get("metadata") or {}).get("hitl_approval")
|
|
||||||
if payload:
|
|
||||||
recovered = await _interrupt_from_checkpoint_history(accessor, read_config, payload) if read_config is not None else None
|
|
||||||
if recovered is not None:
|
|
||||||
task_id, pending_interrupt = recovered
|
|
||||||
return {task_id: [pending_interrupt]}
|
|
||||||
task_id = run.get("run_id") or "approval"
|
|
||||||
return {task_id: [Interrupt(value=payload, id=payload.get("request_id", task_id))]}
|
|
||||||
return {}
|
|
||||||
|
|
||||||
|
|
||||||
def _approval_request_ids(payload: dict[str, Any]) -> dict[str, str]:
|
|
||||||
"""Extract and validate the action identity binding for one HITL pause."""
|
|
||||||
expected_ids = payload.get("pending_tool_call_ids")
|
|
||||||
if not isinstance(expected_ids, list) or any(not isinstance(call_id, str) for call_id in expected_ids) or len(set(expected_ids)) != len(expected_ids):
|
|
||||||
raise _error(422, "invalid_approval_response", "pending approval has invalid tool_call_ids")
|
|
||||||
expected_set = set(expected_ids)
|
|
||||||
actions = payload.get("actions")
|
|
||||||
if not isinstance(actions, list):
|
|
||||||
raise _error(422, "invalid_approval_response", "pending approval has invalid actions")
|
|
||||||
expected_request_ids: dict[str, str] = {}
|
|
||||||
for action in actions:
|
|
||||||
if not isinstance(action, Mapping):
|
|
||||||
raise _error(422, "invalid_approval_response", "pending approval has invalid actions")
|
|
||||||
call_id = action.get("tool_call_id")
|
|
||||||
request_id = action.get("request_id")
|
|
||||||
if not isinstance(call_id, str) or call_id not in expected_set or call_id in expected_request_ids or not isinstance(request_id, str) or not request_id:
|
|
||||||
raise _error(422, "invalid_approval_response", "pending approval has invalid actions")
|
|
||||||
expected_request_ids[call_id] = request_id
|
|
||||||
if set(expected_request_ids) != expected_set:
|
|
||||||
raise _error(422, "invalid_approval_response", "pending approval has invalid actions")
|
|
||||||
return expected_request_ids
|
|
||||||
|
|
||||||
|
|
||||||
async def current_approval_request_ids(
|
|
||||||
request: Any,
|
|
||||||
thread_id: str,
|
|
||||||
*,
|
|
||||||
accessor: Any,
|
|
||||||
read_config: dict[str, Any],
|
|
||||||
) -> dict[str, str] | None:
|
|
||||||
"""Read the current live pause's identity for checkpoint-only recovery."""
|
|
||||||
pending = await find_pending_interrupts(request, thread_id, accessor=accessor, read_config=read_config)
|
|
||||||
payloads = hitl_approval_payloads(pending)
|
|
||||||
return _approval_request_ids(combine_approval_payloads(payloads)) if payloads else None
|
|
||||||
|
|
||||||
|
|
||||||
async def validate_resume_request(*, request: Any, thread_id: str, accessor: Any, read_config: dict[str, Any], enabled: bool, body: Any) -> dict[str, str] | None:
|
|
||||||
"""Validate a command.resume against the current pause, or raise 409/422.
|
|
||||||
|
|
||||||
Static breakpoints (next non-empty, no approval payload) pass through
|
|
||||||
unchanged. All-reject remains the escape hatch for the validated pause when
|
|
||||||
the feature is disabled or the payload was lost; attachment later verifies
|
|
||||||
that checkpoint-only recovery still targets that same pause.
|
|
||||||
"""
|
|
||||||
snapshot = await accessor.aget(read_config)
|
|
||||||
responses = _responses_of(getattr(body, "command", None))
|
|
||||||
all_reject = _all_reject(responses)
|
|
||||||
pending = await find_pending_interrupts(request, thread_id, accessor=accessor, read_config=read_config, snapshot=snapshot)
|
|
||||||
payloads = hitl_approval_payloads(pending)
|
|
||||||
has_waiting_run = await has_waiting_for_input_run(request, thread_id)
|
|
||||||
has_pending_interrupt = await has_any_pending_interrupt(accessor, read_config)
|
|
||||||
head_checkpoint_id = (getattr(snapshot, "config", None) or {}).get("configurable", {}).get("checkpoint_id")
|
|
||||||
if not pending and not has_waiting_run and not (getattr(snapshot, "next", None) or ()) and not has_pending_interrupt:
|
|
||||||
raise _error(409, "no_pending_approval", "No resumable pause exists for this thread")
|
|
||||||
|
|
||||||
if not payloads:
|
|
||||||
return None
|
|
||||||
|
|
||||||
if responses is None:
|
|
||||||
raise _error(422, "invalid_approval_response", "resume must carry a 'responses' list for hitl_approval")
|
|
||||||
expected_request_ids = ApprovalRequestIds(_approval_request_ids(combine_approval_payloads(payloads)))
|
|
||||||
action_by_request = {action["request_id"]: action for payload in payloads for action in payload["actions"]}
|
|
||||||
expected_ids = list(expected_request_ids)
|
|
||||||
expected_set = set(expected_ids)
|
|
||||||
if len(responses) != len(expected_ids):
|
|
||||||
raise _error(422, "invalid_approval_response", f"expected {len(expected_ids)} responses, got {len(responses)}")
|
|
||||||
seen_ids: set[str] = set()
|
|
||||||
for response in responses:
|
|
||||||
if not isinstance(response, Mapping):
|
|
||||||
raise _error(422, "invalid_approval_response", "each response must be an object")
|
|
||||||
response_id = response.get("tool_call_id")
|
|
||||||
if not isinstance(response_id, str) or response_id not in expected_set:
|
|
||||||
raise _error(422, "invalid_approval_response", "unknown tool_call_id")
|
|
||||||
if response.get("request_id") != expected_request_ids[response_id]:
|
|
||||||
raise _error(422, "invalid_approval_response", "stale or mismatched request_id")
|
|
||||||
if response_id in seen_ids:
|
|
||||||
raise _error(422, "invalid_approval_response", "duplicate tool_call_id")
|
|
||||||
seen_ids.add(response_id)
|
|
||||||
action = action_by_request[response["request_id"]]
|
|
||||||
supported = ("approve", "reject", "edit", "approve_thread") if all(payload.get("version") == 2 for payload in payloads) else ("approve", "reject")
|
|
||||||
if response.get("option_id") not in supported:
|
|
||||||
raise _error(422, "invalid_approval_response", f"unsupported option_id; expected one of {', '.join(repr(option) for option in supported)}")
|
|
||||||
if response.get("option_id") == "approve_thread" and not action.get("grant_allowed"):
|
|
||||||
raise _error(422, "invalid_approval_response", "This operation does not permit a thread grant")
|
|
||||||
if response.get("option_id") == "edit":
|
|
||||||
from jsonschema import Draft202012Validator
|
|
||||||
|
|
||||||
from deerflow.approvals.policy import canonical_json
|
|
||||||
|
|
||||||
args, schema = response.get("args"), action.get("args_schema")
|
|
||||||
try:
|
|
||||||
limit = min(payload.get("max_payload_bytes", 512 * 1024) for payload in payloads)
|
|
||||||
if not isinstance(args, dict) or not isinstance(schema, dict) or len(canonical_json(args).encode()) > limit:
|
|
||||||
raise ValueError("Invalid arguments")
|
|
||||||
Draft202012Validator(schema).validate(args)
|
|
||||||
if isinstance(schema.get("properties"), dict) and set(args) - set(schema["properties"]):
|
|
||||||
raise ValueError("Unknown or injected arguments")
|
|
||||||
except Exception as exc:
|
|
||||||
raise _error(422, "invalid_approval_arguments", "Edited arguments do not match the tool schema") from exc
|
|
||||||
|
|
||||||
if all(payload.get("version") == 2 for payload in payloads):
|
|
||||||
expected_request_ids.resume_map = {
|
|
||||||
item.id: {"responses": [response for response in responses if response["request_id"] in {action["request_id"] for action in item.value["actions"]}]}
|
|
||||||
for interrupts in pending.values()
|
|
||||||
for item in interrupts
|
|
||||||
if isinstance(getattr(item, "value", None), dict) and item.value.get("kind") == "hitl_approval"
|
|
||||||
}
|
|
||||||
|
|
||||||
if all_reject:
|
|
||||||
return expected_request_ids
|
|
||||||
|
|
||||||
if not enabled:
|
|
||||||
raise _error(409, "hitl_approval_disabled", "hitl_approval is currently disabled; resume with all-reject to clear the pause")
|
|
||||||
|
|
||||||
requested_checkpoint_id = getattr(body, "checkpoint_id", None)
|
|
||||||
body_checkpoint = getattr(body, "checkpoint", None)
|
|
||||||
if requested_checkpoint_id is None and isinstance(body_checkpoint, Mapping):
|
|
||||||
requested_checkpoint_id = body_checkpoint.get("checkpoint_id")
|
|
||||||
if requested_checkpoint_id and head_checkpoint_id and str(requested_checkpoint_id) != str(head_checkpoint_id):
|
|
||||||
raise _error(422, "checkpoint_not_head", "A resume must target the current head checkpoint, never an older pause checkpoint")
|
|
||||||
return expected_request_ids
|
|
||||||
|
|
||||||
|
|
||||||
async def reject_new_run_when_approval_pending(
|
|
||||||
request: Any,
|
|
||||||
thread_id: str,
|
|
||||||
*,
|
|
||||||
accessor: Any = None,
|
|
||||||
read_config: dict[str, Any] | None = None,
|
|
||||||
) -> None:
|
|
||||||
"""Reject a brand-new (non-resume) run while an approval is pending.
|
|
||||||
|
|
||||||
Without this guard a fresh input silently discards the pending approval
|
|
||||||
(measured behavior of LangGraph). Internal launchers catch this specific
|
|
||||||
error code to implement skip-and-retry semantics.
|
|
||||||
"""
|
|
||||||
pending = await find_pending_interrupts(request, thread_id, accessor=accessor, read_config=read_config)
|
|
||||||
if hitl_approval_payloads(pending):
|
|
||||||
raise _error(409, "approval_pending", "Thread has a pending tool approval; resume or reject it before starting a new run")
|
|
||||||
@ -1,78 +0,0 @@
|
|||||||
"""Bounded, read-only provider catalog retrieval. Never follow credentialed redirects."""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from deerflow.config.managed_model_providers import get_managed_model_provider
|
|
||||||
from deerflow.config.managed_models import ManagedModel
|
|
||||||
|
|
||||||
MAX_MODELS = 2000
|
|
||||||
MAX_BYTES = 2 * 1024 * 1024
|
|
||||||
|
|
||||||
|
|
||||||
class DiscoveryError(ValueError):
|
|
||||||
def __init__(self, code: str):
|
|
||||||
self.code = code
|
|
||||||
super().__init__(code)
|
|
||||||
|
|
||||||
|
|
||||||
async def fetch_model_catalog(profile: ManagedModel) -> tuple[list[str], bool]:
|
|
||||||
provider = get_managed_model_provider(profile.provider)
|
|
||||||
endpoint = provider.endpoint or profile.base_url
|
|
||||||
key = profile.api_key.get_secret_value() if profile.api_key else None
|
|
||||||
native = provider.protocol == "anthropic-messages"
|
|
||||||
url = endpoint.rstrip("/") + ("/v1/models" if native else "/models")
|
|
||||||
headers = {"Accept": "application/json", "Accept-Encoding": "identity"}
|
|
||||||
if native:
|
|
||||||
headers.update({"x-api-key": key or "", "anthropic-version": "2023-06-01"})
|
|
||||||
elif key:
|
|
||||||
headers["Authorization"] = f"Bearer {key}"
|
|
||||||
# HTTPX constructs TLS trust synchronously; keep that work off the event loop.
|
|
||||||
client = await asyncio.to_thread(httpx.AsyncClient, timeout=15, follow_redirects=False)
|
|
||||||
names: set[str] = set()
|
|
||||||
cursor = None
|
|
||||||
size = 0
|
|
||||||
try:
|
|
||||||
async with asyncio.timeout(20), client:
|
|
||||||
for _ in range(20):
|
|
||||||
params = {"limit": 100, **({"after_id": cursor} if cursor else {})} if native else None
|
|
||||||
async with client.stream("GET", url, headers=headers, params=params) as response:
|
|
||||||
if response.status_code in (401, 403):
|
|
||||||
raise DiscoveryError("authentication_failed")
|
|
||||||
if response.status_code in (404, 405):
|
|
||||||
raise DiscoveryError("discovery_unsupported")
|
|
||||||
if response.status_code != 200:
|
|
||||||
raise DiscoveryError("discovery_failed")
|
|
||||||
# Reject compression before HTTPX can allocate a decoded
|
|
||||||
# chunk larger than our response budget.
|
|
||||||
if response.headers.get("content-encoding", "identity").strip().lower() != "identity":
|
|
||||||
raise DiscoveryError("discovery_failed")
|
|
||||||
chunks = []
|
|
||||||
async for chunk in response.aiter_bytes():
|
|
||||||
size += len(chunk)
|
|
||||||
if size > MAX_BYTES:
|
|
||||||
raise DiscoveryError("response_limit")
|
|
||||||
chunks.append(chunk)
|
|
||||||
data = json.loads(b"".join(chunks))
|
|
||||||
if not isinstance(data, dict) or not isinstance(data.get("data"), list):
|
|
||||||
raise DiscoveryError("discovery_failed")
|
|
||||||
for item in data["data"]:
|
|
||||||
name = item.get("id") if isinstance(item, dict) else None
|
|
||||||
if not isinstance(name, str) or not name or len(name) > 200 or name != name.strip() or any(ord(c) < 32 or 127 <= ord(c) <= 159 for c in name):
|
|
||||||
continue
|
|
||||||
names.add(name)
|
|
||||||
if len(names) >= MAX_MODELS:
|
|
||||||
return sorted(names), True
|
|
||||||
if not native or not data.get("has_more"):
|
|
||||||
return sorted(names), bool(data.get("has_more"))
|
|
||||||
next_cursor = data.get("last_id")
|
|
||||||
if not isinstance(next_cursor, str) or not next_cursor or next_cursor == cursor or len(next_cursor) > 200:
|
|
||||||
raise DiscoveryError("discovery_failed")
|
|
||||||
cursor = next_cursor
|
|
||||||
return sorted(names), True
|
|
||||||
except DiscoveryError:
|
|
||||||
raise
|
|
||||||
except Exception:
|
|
||||||
raise DiscoveryError("discovery_failed") from None
|
|
||||||
@ -1,55 +1,100 @@
|
|||||||
# Gateway routers (`app/gateway/routers/`)
|
# Router contracts
|
||||||
|
|
||||||
One module per API area. The parent [Gateway guide](../AGENTS.md) covers trust
|
`GET /api/plugins/diagnostics` uses `require_admin_user` before reading the
|
||||||
boundaries and run admission; this file covers what every route handler has to do.
|
canonical runtime diagnostic snapshot. Return bounded entry-point identifiers,
|
||||||
|
known levels and fixed summaries; raw messages stay in logs. Never clear the
|
||||||
|
sink or expand PAT permissions.
|
||||||
|
|
||||||
## Verify
|
Branch metadata inherits valid `assistant_id` values verbatim. Validate with
|
||||||
|
`custom_agent_name` and clear invalid legacy values to null in the new row;
|
||||||
|
leave the source record unchanged.
|
||||||
|
|
||||||
```bash
|
Console usage attributes legacy rows without a per-model usage map to
|
||||||
cd backend
|
`model_name`, including their recorded `total_input_tokens`. Accumulate those
|
||||||
python -m pytest tests -q -k "<area>_router" # e.g. threads_router, console_router
|
inputs with newer usage buckets for the same model; do not infer cache hits.
|
||||||
make test-blocking-io # after adding any file, config or network read
|
Regression coverage lives in `tests/test_console_router.py`.
|
||||||
```
|
|
||||||
|
|
||||||
## Rules for a route handler
|
Shared scheduled-task update validation preserves an unchanged future `once`
|
||||||
|
instant on a live task inside `scheduler.min_once_delay_seconds`; new/changed
|
||||||
|
instants and terminal reactivation still enforce the minimum delay.
|
||||||
|
|
||||||
- **Authorize first.** Check the session, role or permission before the handler
|
Scheduled-task admission rejects nonexistent naive `run_at` and `end_at`
|
||||||
reads credentials, parses an upload or makes an outbound request. An
|
wall times through the shared Gateway datetime resolver. Keep this check out
|
||||||
unauthorized caller must not be able to trigger that work.
|
of runtime `next_run_at`: legacy tasks retain their accepted execution time,
|
||||||
- **Fixed text in 5xx responses.** Exceptions from storage, config loading and
|
including unchanged live once definitions. Explicit offsets retain their instant;
|
||||||
subprocesses contain host paths. Log the exception on the server and return a
|
ambiguous naive times keep the existing fold choice.
|
||||||
fixed detail string.
|
|
||||||
- **No blocking calls on the event loop.** `get_app_config()` stats and hashes the
|
The MCP task list accepts optional `TaskStatus` and `active_only` query filters.
|
||||||
config files on every call, and storage backends do file I/O. Call both through
|
Pass both to the repository before its limit; preserve `threads:read`, owner
|
||||||
`asyncio.to_thread`. The blocking-I/O suite fails the build otherwise.
|
checks, current-thread incarnation scoping, and the unfiltered array response.
|
||||||
- **Writes survive disconnects.** Wrap a persistent write in
|
|
||||||
`app.gateway.persistent_writes.run_drained_write`. A cancelled request then
|
Custom Agent packages are strict, versioned definition-only JSON documents.
|
||||||
still waits for the worker that owns the write, and no half-written state stays
|
Export explicitly projects the caller's portable `AgentConfig` fields and SOUL;
|
||||||
behind.
|
exclude `github` installation bindings, memory contents, conversations and
|
||||||
- **Catalog responses are projections.** A listing endpoint returns names and
|
credentials. Import may rename locally but must remain owner-scoped and
|
||||||
display fields. Provider paths, credentials and raw tool or server configuration
|
create-only: collisions return 409, never overwrite or upsert. Ordinary creation
|
||||||
stay on the server.
|
and import share `_persist_new_agent`, which uses `run_drained_write` so a
|
||||||
- **Provenance comes from the session.** Values that say who or what triggered an
|
cancelled request still drains the owned persistence worker.
|
||||||
action (for example the plugin `ActionContext.interaction`) are set from the
|
|
||||||
authenticated request only. Ignore the same field in a request body.
|
`GET /api/skills`, `/api/skills/custom` and `/api/skills/{name}` serve every
|
||||||
- **Unreadable configuration is not default configuration.** When a policy cannot
|
signed-in user, so their unexpected-failure 500s return a fixed detail. Load
|
||||||
be read, skip the optional work (cleanup, sweep) and keep serving. Applying a
|
errors (e.g. a malformed `extensions_config.json`) carry host paths and stay in
|
||||||
default policy can delete data the operator wanted to keep.
|
the server log.
|
||||||
- **Imports create, they do not overwrite.** Agent and skill imports are scoped to
|
|
||||||
the caller and return 409 on a name collision. Exports leave out credentials,
|
Thread upload, upload delete and shelf attach-to-thread 500s name the file but
|
||||||
memory, conversations and installation bindings.
|
not the cause: staging and unlink `OSError`s carry host paths and stay in the
|
||||||
- **Memory backup imports** replace the selected facts and require an explicit
|
server log.
|
||||||
`facts` array. Omitting it returns 422. An empty array clears that bucket.
|
|
||||||
- **Outbound requests from a route** (model discovery, connection tests) have a
|
Skill mutation operation lists return `next_cursor: null` when `has_more` is false, including empty pages after a supplied `after_id`. The next cursor is the final returned operation only when a further page exists.
|
||||||
time limit and a size limit, do not follow redirects, and return redacted
|
|
||||||
provider errors.
|
`GET /api/agent-tool-groups` projects only configured group names for the
|
||||||
- **Locks that several Gateway processes share** are file locks beside the data,
|
custom-Agent settings allowlist. Never expose tool provider paths, credentials,
|
||||||
taken off the event loop. A process mutex alone does not order writes between
|
or raw tool configuration through this catalog.
|
||||||
workers.
|
|
||||||
- **Path segments from clients.** A name that becomes a directory (agent name,
|
Custom Agent update guards apply legacy shared-file checks only when the
|
||||||
skill name, file name) is validated when it is created, including Windows
|
resolved store is `FileAgentStore`, including the `update_agent` tool. DB
|
||||||
reserved device names. Read and delete routes stay permissive so that data
|
imports retain source files for rollback; those files cannot veto SQL updates.
|
||||||
stored by an older release can still be exported and removed.
|
Keep store resolution and filesystem checks off-loop in the HTTP route.
|
||||||
- **Search results use the latest owner-visible message.** A historical jump
|
Agent PUT uses `preserve_non_managed_fields(..., new_tool_groups=...)` to clear
|
||||||
resolves only the requested thread. Use the shared history seed path for
|
an inherited knowledge default when changed groups remove knowledge access.
|
||||||
legacy checkpoints so it keeps its existing write reservation.
|
Explicit scope/null updates still win; unchanged groups are not a migration.
|
||||||
|
|
||||||
|
Agent/profile and deployment/personal MCP mutation 500s plus the
|
||||||
|
scheduled-task `invalid_assistant` 422 use fixed text. Log exceptions
|
||||||
|
server-side; they can carry host paths.
|
||||||
|
|
||||||
|
Committed MCP reconciliation failures are logged once by
|
||||||
|
`fail_mcp_reconciliation`, with only the underlying exception type.
|
||||||
|
`_run_drained_mcp_apply` passes `_McpCommittedReconciliationError` as an expected
|
||||||
|
error to `run_drained_write` to avoid a duplicate diagnostic. It still drains
|
||||||
|
retired-pool cleanup outside the config locks and reports the fixed-detail 500.
|
||||||
|
|
||||||
|
Custom-skill edit, delete and rollback hold the owning root's process mutex and
|
||||||
|
`.custom.mutation.lock` sidecar across predecessor read, storage mutation and
|
||||||
|
history append. Peer Gateway processes sharing that root must use this same
|
||||||
|
critical section to preserve history ordering. Keep the sidecar distinct from
|
||||||
|
the nested projection lock, retain it across requests, and acquire/release both
|
||||||
|
off-loop inside the drained worker. Unrelated users retain independent locks;
|
||||||
|
external filesystem writers do not participate in this history guarantee.
|
||||||
|
|
||||||
|
Plugin actions stamp `ActionContext.interaction=session` only from authenticated
|
||||||
|
session provenance. Never accept it from payloads or propagate it to model tools;
|
||||||
|
extensions use it to distinguish owner review from model/tool output.
|
||||||
|
|
||||||
|
Skill archive upload copies (`_copy_uploaded_skill_archive`) report the original
|
||||||
|
size-limit or read error when temporary cleanup or its warning log fails.
|
||||||
|
Successful copies still return the uploaded bytes unchanged. The upload route's
|
||||||
|
`finally` discards that archive through the same helper before closing the
|
||||||
|
multipart form. Form close is suppressed, so a close, cleanup, or warning
|
||||||
|
failure cannot turn a successful install into a 500, replace an install
|
||||||
|
error, or skip the discard. Coverage: `tests/test_skill_archive_upload_cleanup.py`.
|
||||||
|
|
||||||
|
State updates synchronize titles into thread metadata only for the root checkpoint
|
||||||
|
namespace. Nonroot writes preserve the conversation title, branch-title sequence
|
||||||
|
and activity timestamp. `tests/test_threads_router.py` checks metadata/search
|
||||||
|
isolation and root title synchronization in full and delta modes.
|
||||||
|
|
||||||
|
Unauthenticated auth routes (`/register`, `/setup-status`, `/providers`,
|
||||||
|
`/oauth/{provider}`, `/callback/{provider}`, the `/login/local` throttle) read
|
||||||
|
`get_app_config()` through `asyncio.to_thread`: each read stats and hashes the
|
||||||
|
config files. Coverage: `tests/blocking_io/test_auth_router_config_reads.py`.
|
||||||
|
|||||||
@ -96,7 +96,7 @@ class AgentUpdateRequest(BaseModel):
|
|||||||
"""Request body for updating a custom agent."""
|
"""Request body for updating a custom agent."""
|
||||||
|
|
||||||
display_name: AgentDisplayName | None = Field(default=None, description="Updated display name; null clears it")
|
display_name: AgentDisplayName | None = Field(default=None, description="Updated display name; null clears it")
|
||||||
description: str | None = Field(default=None, description="Updated description; null resets it to empty")
|
description: str | None = Field(default=None, description="Updated description")
|
||||||
model: str | None = Field(default=None, description="Updated model override")
|
model: str | None = Field(default=None, description="Updated model override")
|
||||||
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist")
|
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist")
|
||||||
mcp_plugins: list[str] | None = Field(default=None, description="MCP installation selection (None=all, []=none)")
|
mcp_plugins: list[str] | None = Field(default=None, description="MCP installation selection (None=all, []=none)")
|
||||||
@ -601,9 +601,7 @@ async def update_agent(name: str, body: AgentUpdateRequest, request: Request) ->
|
|||||||
if config_changed:
|
if config_changed:
|
||||||
updated = {
|
updated = {
|
||||||
"name": agent_cfg.name,
|
"name": agent_cfg.name,
|
||||||
# Explicit null resets to the AgentConfig default; storing None
|
"description": body.description if "description" in fields_set else agent_cfg.description,
|
||||||
# would make every later load of this agent fail validation.
|
|
||||||
"description": (body.description or "") if "description" in fields_set else agent_cfg.description,
|
|
||||||
}
|
}
|
||||||
if "display_name" in fields_set:
|
if "display_name" in fields_set:
|
||||||
updated["display_name"] = body.display_name or None
|
updated["display_name"] = body.display_name or None
|
||||||
|
|||||||
@ -4,21 +4,17 @@ import hashlib
|
|||||||
import logging
|
import logging
|
||||||
import mimetypes
|
import mimetypes
|
||||||
import os
|
import os
|
||||||
import secrets
|
|
||||||
import stat
|
import stat
|
||||||
import tempfile
|
import tempfile
|
||||||
import threading
|
|
||||||
import zipfile
|
import zipfile
|
||||||
from collections.abc import AsyncIterator
|
from collections.abc import AsyncIterator
|
||||||
from contextlib import asynccontextmanager
|
from contextlib import asynccontextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import BinaryIO
|
|
||||||
from urllib.parse import quote
|
from urllib.parse import quote
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Request
|
from fastapi import APIRouter, HTTPException, Request
|
||||||
from fastapi.responses import FileResponse, Response
|
from fastapi.responses import FileResponse, Response
|
||||||
from pydantic import BaseModel, Field
|
from pydantic import BaseModel, Field
|
||||||
from starlette.datastructures import MutableHeaders
|
|
||||||
|
|
||||||
from app.gateway.authz import SandboxRequestLease, require_permission, try_acquire_sandbox_for_request
|
from app.gateway.authz import SandboxRequestLease, require_permission, try_acquire_sandbox_for_request
|
||||||
from app.gateway.deps import get_run_manager
|
from app.gateway.deps import get_run_manager
|
||||||
@ -45,8 +41,6 @@ _SKILL_ARCHIVE_READ_CHUNK_SIZE = 64 * 1024
|
|||||||
MAX_EDITABLE_ARTIFACT_BYTES = 2 * 1024 * 1024
|
MAX_EDITABLE_ARTIFACT_BYTES = 2 * 1024 * 1024
|
||||||
MAX_VIEWED_IMAGE_BLOB_BYTES = 20 * 1024 * 1024
|
MAX_VIEWED_IMAGE_BLOB_BYTES = 20 * 1024 * 1024
|
||||||
_ARTIFACT_EDIT_TEMP_PREFIX = ".artifact-edit-"
|
_ARTIFACT_EDIT_TEMP_PREFIX = ".artifact-edit-"
|
||||||
_ARTIFACT_COPY_BEFORE_STREAM = os.name == "nt"
|
|
||||||
_artifact_digest_source = threading.local()
|
|
||||||
|
|
||||||
|
|
||||||
class ArtifactUpdateRequest(BaseModel):
|
class ArtifactUpdateRequest(BaseModel):
|
||||||
@ -340,7 +334,7 @@ def _read_artifact_payload(actual_path: Path, path: str, download: bool) -> tupl
|
|||||||
return ("inline_file", mime_type)
|
return ("inline_file", mime_type)
|
||||||
|
|
||||||
|
|
||||||
def _sha256_of_file(path: Path, *, pinned: tuple[BinaryIO, os.stat_result] | None = None) -> str:
|
def _sha256_of_file(path: Path) -> str:
|
||||||
"""Return the hex SHA-256 digest of *path* without loading it whole.
|
"""Return the hex SHA-256 digest of *path* without loading it whole.
|
||||||
|
|
||||||
Computing the digest on the Gateway lets the browser skip its own
|
Computing the digest on the Gateway lets the browser skip its own
|
||||||
@ -352,217 +346,45 @@ def _sha256_of_file(path: Path, *, pinned: tuple[BinaryIO, os.stat_result] | Non
|
|||||||
requests a browser issues while scrubbing/paginating a preview do not each
|
requests a browser issues while scrubbing/paginating a preview do not each
|
||||||
re-hash a potentially huge artifact from scratch (raised in PR review).
|
re-hash a potentially huge artifact from scratch (raised in PR review).
|
||||||
"""
|
"""
|
||||||
file_stat = path.stat() if pinned is None else pinned[1]
|
stat = path.stat()
|
||||||
# Sandbox syncs and other writers can atomically replace a same-size file
|
# Sandbox syncs and other writers can atomically replace a same-size file
|
||||||
# while preserving its mtime. Identity separates those generations; ctime
|
# while preserving its mtime. Identity separates those generations; ctime
|
||||||
# also invalidates in-place writes where the filesystem exposes change time.
|
# also invalidates in-place writes where the filesystem exposes change time.
|
||||||
key = (
|
return _sha256_of_file_cached(
|
||||||
str(path),
|
str(path),
|
||||||
file_stat.st_dev,
|
stat.st_dev,
|
||||||
file_stat.st_ino,
|
stat.st_ino,
|
||||||
file_stat.st_ctime_ns,
|
stat.st_ctime_ns,
|
||||||
file_stat.st_mtime_ns,
|
stat.st_mtime_ns,
|
||||||
file_stat.st_size,
|
stat.st_size,
|
||||||
)
|
)
|
||||||
if pinned is None:
|
|
||||||
return _sha256_of_file_cached(*key)
|
|
||||||
|
|
||||||
previous = getattr(_artifact_digest_source, "pinned", None)
|
|
||||||
_artifact_digest_source.pinned = (key, pinned[0])
|
|
||||||
try:
|
|
||||||
return _sha256_of_file_cached(*key)
|
|
||||||
finally:
|
|
||||||
_artifact_digest_source.pinned = previous
|
|
||||||
|
|
||||||
|
|
||||||
@functools.lru_cache(maxsize=256)
|
@functools.lru_cache(maxsize=256)
|
||||||
def _sha256_of_file_cached(path: str, device: int, inode: int, ctime_ns: int, mtime_ns: int, size: int) -> str:
|
def _sha256_of_file_cached(path: str, device: int, inode: int, ctime_ns: int, mtime_ns: int, size: int) -> str:
|
||||||
"""Cached SHA-256 of *path*; metadata arguments separate file generations."""
|
"""Cached SHA-256 of *path*; metadata arguments separate file generations."""
|
||||||
pinned = getattr(_artifact_digest_source, "pinned", None)
|
digest = hashlib.sha256()
|
||||||
# A nested lookup for a different file must hash its own bytes.
|
|
||||||
if pinned is None or pinned[0] != (path, device, inode, ctime_ns, mtime_ns, size):
|
|
||||||
with open(path, "rb") as handle:
|
with open(path, "rb") as handle:
|
||||||
return hashlib.file_digest(handle, "sha256").hexdigest()
|
for chunk in iter(lambda: handle.read(1 << 20), b""):
|
||||||
|
digest.update(chunk)
|
||||||
handle = pinned[1]
|
return digest.hexdigest()
|
||||||
previous_offset = handle.tell()
|
|
||||||
try:
|
|
||||||
handle.seek(0)
|
|
||||||
return hashlib.file_digest(handle, "sha256").hexdigest()
|
|
||||||
finally:
|
|
||||||
handle.seek(previous_offset)
|
|
||||||
|
|
||||||
|
|
||||||
def _artifact_etag(path: Path) -> str:
|
def _artifact_etag(path: Path) -> str:
|
||||||
"""Keep content revisions for editable files and metadata tags for large files."""
|
"""Keep content revisions for editable files and metadata-only tags for large files."""
|
||||||
file_stat = path.stat()
|
file_stat = path.stat()
|
||||||
if file_stat.st_size <= MAX_EDITABLE_ARTIFACT_BYTES:
|
if file_stat.st_size <= MAX_EDITABLE_ARTIFACT_BYTES:
|
||||||
# Populate the shared cache only with metadata from the hashed handle.
|
return f'"{_sha256_of_file(path)}"'
|
||||||
with path.open("rb") as handle:
|
|
||||||
return _artifact_etag_from_handle(path, handle, os.fstat(handle.fileno()))
|
|
||||||
return _artifact_stat_etag(file_stat)
|
|
||||||
|
|
||||||
|
|
||||||
def _artifact_stat_etag(file_stat: os.stat_result) -> str:
|
|
||||||
# Do not read oversized artifacts just to validate a small byte range.
|
# Do not read oversized artifacts just to validate a small byte range.
|
||||||
# The prefix keeps this opaque validator distinct from an editable SHA-256.
|
# The prefix keeps this opaque validator distinct from an editable SHA-256.
|
||||||
identity = f"{file_stat.st_dev}:{file_stat.st_ino}:{file_stat.st_ctime_ns}:{file_stat.st_mtime_ns}:{file_stat.st_size}"
|
identity = f"{file_stat.st_dev}:{file_stat.st_ino}:{file_stat.st_ctime_ns}:{file_stat.st_mtime_ns}:{file_stat.st_size}"
|
||||||
return f'"stat-{hashlib.sha256(identity.encode("ascii")).hexdigest()}"'
|
return f'"stat-{hashlib.sha256(identity.encode("ascii")).hexdigest()}"'
|
||||||
|
|
||||||
|
|
||||||
def _artifact_etag_from_handle(path: Path, handle: BinaryIO, file_stat: os.stat_result) -> str:
|
|
||||||
"""Build the validator from the same file generation the response will read."""
|
|
||||||
if file_stat.st_size <= MAX_EDITABLE_ARTIFACT_BYTES:
|
|
||||||
digest = _sha256_of_file(path, pinned=(handle, file_stat))
|
|
||||||
return f'"{digest}"'
|
|
||||||
return _artifact_stat_etag(file_stat)
|
|
||||||
|
|
||||||
|
|
||||||
def _prepare_artifact_stream(path: Path) -> tuple[BinaryIO, os.stat_result, str]:
|
|
||||||
"""Pin one file generation before response headers authorize a byte range."""
|
|
||||||
handle = path.open("rb")
|
|
||||||
try:
|
|
||||||
file_stat = os.fstat(handle.fileno())
|
|
||||||
if not stat.S_ISREG(file_stat.st_mode):
|
|
||||||
raise RuntimeError(f"File at path {path} is not a file.")
|
|
||||||
return handle, file_stat, _artifact_etag_from_handle(path, handle, file_stat)
|
|
||||||
except BaseException:
|
|
||||||
_close_artifact_handle(handle)
|
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
def _copy_artifact_ranges(handle: BinaryIO, ranges: list[tuple[int, int]]) -> BinaryIO:
|
|
||||||
"""Pack the selected ranges into a private temporary file in send order."""
|
|
||||||
snapshot = tempfile.TemporaryFile(mode="w+b")
|
|
||||||
try:
|
|
||||||
for start, end in ranges:
|
|
||||||
handle.seek(start)
|
|
||||||
remaining = end - start
|
|
||||||
while remaining:
|
|
||||||
chunk = handle.read(min(1 << 20, remaining))
|
|
||||||
if not chunk:
|
|
||||||
raise OSError("Artifact was truncated while preparing the response")
|
|
||||||
snapshot.write(chunk)
|
|
||||||
remaining -= len(chunk)
|
|
||||||
snapshot.seek(0)
|
|
||||||
return snapshot
|
|
||||||
except BaseException:
|
|
||||||
_close_artifact_handle(snapshot)
|
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
def _close_artifact_handle(handle: BinaryIO) -> None:
|
|
||||||
try:
|
|
||||||
handle.close()
|
|
||||||
except OSError:
|
|
||||||
logger.warning("Could not close artifact response handle", exc_info=True)
|
|
||||||
|
|
||||||
|
|
||||||
class _ArtifactFileResponse(FileResponse):
|
class _ArtifactFileResponse(FileResponse):
|
||||||
"""Require an ETag to authorize conditional ranges of replaceable artifacts."""
|
"""Require an ETag to authorize conditional ranges of replaceable artifacts."""
|
||||||
|
|
||||||
def __init__(self, *args, **kwargs) -> None:
|
|
||||||
super().__init__(*args, **kwargs)
|
|
||||||
self._artifact_handle: BinaryIO | None = None
|
|
||||||
|
|
||||||
async def __call__(self, scope, receive, send) -> None:
|
|
||||||
async def prepare() -> None:
|
|
||||||
self._artifact_handle, self.stat_result, etag = await asyncio.to_thread(_prepare_artifact_stream, Path(self.path))
|
|
||||||
self.headers["etag"] = etag
|
|
||||||
self.set_stat_headers(self.stat_result)
|
|
||||||
|
|
||||||
try:
|
|
||||||
await await_drained(prepare())
|
|
||||||
await super().__call__(scope, receive, send)
|
|
||||||
finally:
|
|
||||||
if self._artifact_handle is not None:
|
|
||||||
handle, self._artifact_handle = self._artifact_handle, None
|
|
||||||
await await_drained(asyncio.to_thread(_close_artifact_handle, handle))
|
|
||||||
|
|
||||||
async def _snapshot_windows_body(self, ranges: list[tuple[int, int]], send_header_only: bool) -> bool:
|
|
||||||
if not _ARTIFACT_COPY_BEFORE_STREAM:
|
|
||||||
return False
|
|
||||||
|
|
||||||
source = self._artifact_handle
|
|
||||||
assert source is not None
|
|
||||||
|
|
||||||
async def snapshot() -> None:
|
|
||||||
# Windows replacement needs the public handle closed. A private
|
|
||||||
# copy also leaves the artifact's link count unchanged for ZIP.
|
|
||||||
self._artifact_handle = None if send_header_only else await asyncio.to_thread(_copy_artifact_ranges, source, ranges)
|
|
||||||
await asyncio.to_thread(_close_artifact_handle, source)
|
|
||||||
|
|
||||||
await await_drained(snapshot())
|
|
||||||
return True
|
|
||||||
|
|
||||||
async def _handle_simple(self, send, send_header_only: bool, send_pathsend: bool) -> None:
|
|
||||||
assert self.stat_result is not None
|
|
||||||
await self._snapshot_windows_body([(0, self.stat_result.st_size)], send_header_only)
|
|
||||||
|
|
||||||
await send({"type": "http.response.start", "status": self.status_code, "headers": self.raw_headers})
|
|
||||||
if send_header_only:
|
|
||||||
await send({"type": "http.response.body", "body": b"", "more_body": False})
|
|
||||||
return
|
|
||||||
|
|
||||||
handle = self._artifact_handle
|
|
||||||
assert handle is not None
|
|
||||||
await await_drained(asyncio.to_thread(handle.seek, 0))
|
|
||||||
more_body = True
|
|
||||||
while more_body:
|
|
||||||
chunk = await await_drained(asyncio.to_thread(handle.read, self.chunk_size))
|
|
||||||
more_body = len(chunk) == self.chunk_size
|
|
||||||
await send({"type": "http.response.body", "body": chunk, "more_body": more_body})
|
|
||||||
|
|
||||||
async def _handle_single_range(self, send, start: int, end: int, file_size: int, send_header_only: bool) -> None:
|
|
||||||
snapshot = await self._snapshot_windows_body([(start, end)], send_header_only)
|
|
||||||
|
|
||||||
headers = MutableHeaders(raw=list(self.raw_headers))
|
|
||||||
headers["content-range"] = f"bytes {start}-{end - 1}/{file_size}"
|
|
||||||
headers["content-length"] = str(end - start)
|
|
||||||
await send({"type": "http.response.start", "status": 206, "headers": headers.raw})
|
|
||||||
if send_header_only:
|
|
||||||
await send({"type": "http.response.body", "body": b"", "more_body": False})
|
|
||||||
return
|
|
||||||
|
|
||||||
handle = self._artifact_handle
|
|
||||||
assert handle is not None
|
|
||||||
if not snapshot:
|
|
||||||
await await_drained(asyncio.to_thread(handle.seek, start))
|
|
||||||
more_body = True
|
|
||||||
while more_body:
|
|
||||||
chunk = await await_drained(asyncio.to_thread(handle.read, min(self.chunk_size, end - start)))
|
|
||||||
start += len(chunk)
|
|
||||||
more_body = len(chunk) == self.chunk_size and start < end
|
|
||||||
await send({"type": "http.response.body", "body": chunk, "more_body": more_body})
|
|
||||||
|
|
||||||
async def _handle_multiple_ranges(self, send, ranges: list[tuple[int, int]], file_size: int, send_header_only: bool) -> None:
|
|
||||||
snapshot = await self._snapshot_windows_body(ranges, send_header_only)
|
|
||||||
|
|
||||||
boundary = secrets.token_hex(13)
|
|
||||||
content_length, header_generator = self.generate_multipart(ranges, boundary, file_size, self.headers["content-type"])
|
|
||||||
headers = MutableHeaders(raw=list(self.raw_headers))
|
|
||||||
headers["content-type"] = f"multipart/byteranges; boundary={boundary}"
|
|
||||||
headers["content-length"] = str(content_length)
|
|
||||||
await send({"type": "http.response.start", "status": 206, "headers": headers.raw})
|
|
||||||
if send_header_only:
|
|
||||||
await send({"type": "http.response.body", "body": b"", "more_body": False})
|
|
||||||
return
|
|
||||||
|
|
||||||
handle = self._artifact_handle
|
|
||||||
assert handle is not None
|
|
||||||
for start, end in ranges:
|
|
||||||
await send({"type": "http.response.body", "body": header_generator(start, end), "more_body": True})
|
|
||||||
if not snapshot:
|
|
||||||
await await_drained(asyncio.to_thread(handle.seek, start))
|
|
||||||
while start < end:
|
|
||||||
chunk = await await_drained(asyncio.to_thread(handle.read, min(self.chunk_size, end - start)))
|
|
||||||
if not chunk:
|
|
||||||
raise OSError("Artifact was truncated while streaming a byte range")
|
|
||||||
start += len(chunk)
|
|
||||||
await send({"type": "http.response.body", "body": chunk, "more_body": True})
|
|
||||||
await send({"type": "http.response.body", "body": b"\r\n", "more_body": True})
|
|
||||||
await send({"type": "http.response.body", "body": f"--{boundary}--".encode("latin-1"), "more_body": False})
|
|
||||||
|
|
||||||
def _should_use_range(self, http_if_range: str) -> bool:
|
def _should_use_range(self, http_if_range: str) -> bool:
|
||||||
# Sandbox syncs can preserve mtimes across replacements, so Last-Modified
|
# Sandbox syncs can preserve mtimes across replacements, so Last-Modified
|
||||||
# is not a strong validator. Date-form If-Range must return the full file.
|
# is not a strong validator. Date-form If-Range must return the full file.
|
||||||
|
|||||||
@ -13,7 +13,6 @@ which persists no run history to report on.
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import math
|
|
||||||
from datetime import UTC, datetime, time, timedelta
|
from datetime import UTC, datetime, time, timedelta
|
||||||
from typing import NamedTuple
|
from typing import NamedTuple
|
||||||
|
|
||||||
@ -182,14 +181,6 @@ def _build_pricing_map() -> dict[str, _ModelPricing]:
|
|||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
logger.warning("console: ignoring malformed pricing on model %s", model_cfg.name)
|
logger.warning("console: ignoring malformed pricing on model %s", model_cfg.name)
|
||||||
continue
|
continue
|
||||||
# ``float("nan")`` / ``float(".inf")`` parse successfully, and NaN then
|
|
||||||
# survives every ``<=`` comparison below and every ``round()``, ending up
|
|
||||||
# in the JSON body where Starlette rejects it (allow_nan=False) -- a 500
|
|
||||||
# for the whole usage report. Treat non-finite prices as malformed, the
|
|
||||||
# same way the rest of the config layer does (allow_inf_nan=False).
|
|
||||||
if any(not math.isfinite(price) for price in (input_price, output_price, cache_hit_price) if price is not None):
|
|
||||||
logger.warning("console: ignoring non-finite pricing on model %s", model_cfg.name)
|
|
||||||
continue
|
|
||||||
if input_price <= 0 and output_price <= 0:
|
if input_price <= 0 and output_price <= 0:
|
||||||
continue
|
continue
|
||||||
model_currency = str(raw.get("currency") or "USD").strip().upper() or "USD"
|
model_currency = str(raw.get("currency") or "USD").strip().upper() or "USD"
|
||||||
@ -237,36 +228,8 @@ def _token_cost(input_tokens: int, output_tokens: int, price: _ModelPricing, cac
|
|||||||
return (uncached / 1_000_000) * price.input_per_million + (cache_read / 1_000_000) * hit_price + (output_tokens / 1_000_000) * price.output_per_million
|
return (uncached / 1_000_000) * price.input_per_million + (cache_read / 1_000_000) * hit_price + (output_tokens / 1_000_000) * price.output_per_million
|
||||||
|
|
||||||
|
|
||||||
def _token_count(value: object) -> int:
|
|
||||||
"""Coerce one stored token counter to a non-negative ``int``.
|
|
||||||
|
|
||||||
``token_usage_by_model`` is a JSON column written from provider-reported
|
|
||||||
usage, so a bucket value is not guaranteed to be a number: providers have
|
|
||||||
shipped counters as strings ("n/a", "1.0"), and rows written by older
|
|
||||||
build/import paths may hold anything JSON can express. The console already
|
|
||||||
skips buckets that are not dicts and pricing that is not numeric; coercing
|
|
||||||
counters the same way keeps one junk bucket from failing the whole report
|
|
||||||
with a 500. Booleans are rejected because ``int(True) == 1`` would invent a
|
|
||||||
token, and negatives cannot be spend.
|
|
||||||
|
|
||||||
A JSON integer is not bounded by the float range: ``10**400`` parses as an
|
|
||||||
exact ``int``, and ``_token_cost`` then divides it by a million, which
|
|
||||||
raises ``OverflowError`` on the way back to a float. Such a value is junk
|
|
||||||
in exactly the same way as ``"n/a"``, so it is rejected here too.
|
|
||||||
"""
|
|
||||||
if isinstance(value, bool):
|
|
||||||
return 0
|
|
||||||
try:
|
|
||||||
count = int(value) # type: ignore[arg-type]
|
|
||||||
if count < 0 or not math.isfinite(float(count)):
|
|
||||||
return 0
|
|
||||||
except (TypeError, ValueError, OverflowError):
|
|
||||||
return 0
|
|
||||||
return count
|
|
||||||
|
|
||||||
|
|
||||||
def _usage_is_missing(usage: dict) -> bool:
|
def _usage_is_missing(usage: dict) -> bool:
|
||||||
return _token_count(usage.get("missing_usage_calls")) > 0
|
return int(usage.get("missing_usage_calls") or 0) > 0
|
||||||
|
|
||||||
|
|
||||||
def _has_missing_usage(token_usage_by_model: object) -> bool:
|
def _has_missing_usage(token_usage_by_model: object) -> bool:
|
||||||
@ -300,11 +263,11 @@ def _run_cost(
|
|||||||
price = _lookup_pricing(pricing, model)
|
price = _lookup_pricing(pricing, model)
|
||||||
if price is None:
|
if price is None:
|
||||||
continue
|
continue
|
||||||
input_tokens = _token_count(usage.get("input_tokens"))
|
input_tokens = int(usage.get("input_tokens") or 0)
|
||||||
output_tokens = _token_count(usage.get("output_tokens"))
|
output_tokens = int(usage.get("output_tokens") or 0)
|
||||||
if input_tokens == 0 and output_tokens == 0:
|
if input_tokens == 0 and output_tokens == 0:
|
||||||
continue
|
continue
|
||||||
cost += _token_cost(input_tokens, output_tokens, price, _token_count(usage.get("cache_read_tokens")))
|
cost += _token_cost(input_tokens, output_tokens, price, int(usage.get("cache_read_tokens") or 0))
|
||||||
priced = True
|
priced = True
|
||||||
if priced:
|
if priced:
|
||||||
return cost
|
return cost
|
||||||
@ -548,9 +511,9 @@ async def console_usage(
|
|||||||
entry.runs += 1
|
entry.runs += 1
|
||||||
if not isinstance(usage, dict):
|
if not isinstance(usage, dict):
|
||||||
continue
|
continue
|
||||||
entry.tokens += _token_count(usage.get("total_tokens"))
|
entry.tokens += int(usage.get("total_tokens", 0) or 0)
|
||||||
entry.input_tokens += _token_count(usage.get("input_tokens"))
|
entry.input_tokens += int(usage.get("input_tokens") or 0)
|
||||||
entry.cache_read_tokens += _token_count(usage.get("cache_read_tokens"))
|
entry.cache_read_tokens += int(usage.get("cache_read_tokens") or 0)
|
||||||
price = _lookup_pricing(pricing, model)
|
price = _lookup_pricing(pricing, model)
|
||||||
if _usage_is_missing(usage):
|
if _usage_is_missing(usage):
|
||||||
missing_models.add(model.lower())
|
missing_models.add(model.lower())
|
||||||
@ -559,7 +522,7 @@ async def console_usage(
|
|||||||
# equal rates on an unrelated model are not aliases.
|
# equal rates on an unrelated model are not aliases.
|
||||||
missing_model_prices.add(id(price))
|
missing_model_prices.add(id(price))
|
||||||
if price is not None:
|
if price is not None:
|
||||||
model_cost = _token_cost(_token_count(usage.get("input_tokens")), _token_count(usage.get("output_tokens")), price, _token_count(usage.get("cache_read_tokens")))
|
model_cost = _token_cost(int(usage.get("input_tokens") or 0), int(usage.get("output_tokens") or 0), price, int(usage.get("cache_read_tokens") or 0))
|
||||||
entry.cost = round((entry.cost or 0.0) + model_cost, 6)
|
entry.cost = round((entry.cost or 0.0) + model_cost, 6)
|
||||||
elif row.model_name and run_tokens > 0:
|
elif row.model_name and run_tokens > 0:
|
||||||
# Legacy rows predating token_usage_by_model: fall back to the run's model.
|
# Legacy rows predating token_usage_by_model: fall back to the run's model.
|
||||||
|
|||||||
@ -7,8 +7,6 @@ request, while startup-scoped capabilities report the runtime that actually
|
|||||||
started.
|
started.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Literal
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, Request
|
from fastapi import APIRouter, Depends, Request
|
||||||
from pydantic import BaseModel, Field
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
@ -84,12 +82,6 @@ class ThreadActivityFeature(BaseModel):
|
|||||||
available: bool = Field(..., description="Whether GET /api/thread-activity, POST /api/threads/{thread_id}/read and the thread search `unread` field work (SQL persistence)")
|
available: bool = Field(..., description="Whether GET /api/thread-activity, POST /api/threads/{thread_id}/read and the thread search `unread` field work (SQL persistence)")
|
||||||
|
|
||||||
|
|
||||||
class ToolApprovalFeature(BaseModel):
|
|
||||||
enabled: bool = False
|
|
||||||
default_mode: Literal["ask", "auto", "full"] = "ask"
|
|
||||||
allowed_modes: list[Literal["ask", "auto", "full"]] = Field(default_factory=list)
|
|
||||||
|
|
||||||
|
|
||||||
class FeaturesResponse(BaseModel):
|
class FeaturesResponse(BaseModel):
|
||||||
"""Frontend-facing feature availability flags."""
|
"""Frontend-facing feature availability flags."""
|
||||||
|
|
||||||
@ -101,8 +93,6 @@ class FeaturesResponse(BaseModel):
|
|||||||
knowledge_base: KnowledgeBaseFeature
|
knowledge_base: KnowledgeBaseFeature
|
||||||
scheduled_tasks: ScheduledTasksFeature
|
scheduled_tasks: ScheduledTasksFeature
|
||||||
thread_activity: ThreadActivityFeature
|
thread_activity: ThreadActivityFeature
|
||||||
run_inputs: bool = False
|
|
||||||
tool_approval: ToolApprovalFeature = Field(default_factory=ToolApprovalFeature)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get(
|
@router.get(
|
||||||
@ -114,11 +104,8 @@ class FeaturesResponse(BaseModel):
|
|||||||
async def list_features(request: Request, config: AppConfig = Depends(get_config)) -> FeaturesResponse:
|
async def list_features(request: Request, config: AppConfig = Depends(get_config)) -> FeaturesResponse:
|
||||||
"""Return availability of optional frontend features."""
|
"""Return availability of optional frontend features."""
|
||||||
browser = browser_capability(config)
|
browser = browser_capability(config)
|
||||||
approval = getattr(config, "hitl_approval", None)
|
|
||||||
subagent_batch_worker_running = bool(getattr(request.app.state, "subagent_batches_available", False))
|
subagent_batch_worker_running = bool(getattr(request.app.state, "subagent_batches_available", False))
|
||||||
return FeaturesResponse(
|
return FeaturesResponse(
|
||||||
tool_approval=ToolApprovalFeature(enabled=approval.enabled, default_mode=approval.default_mode, allowed_modes=approval.allowed_modes) if approval is not None else ToolApprovalFeature(),
|
|
||||||
run_inputs=getattr(getattr(request.app.state, "run_manager", None), "input_store", None) is not None,
|
|
||||||
agents_api=AgentsApiFeature(enabled=config.agents_api.enabled),
|
agents_api=AgentsApiFeature(enabled=config.agents_api.enabled),
|
||||||
browser_control=BrowserControlFeature(enabled=browser.available),
|
browser_control=BrowserControlFeature(enabled=browser.available),
|
||||||
# MCP task bindings and the submitter are startup-scoped. Report the
|
# MCP task bindings and the submitter are startup-scoped. Report the
|
||||||
|
|||||||
@ -14,7 +14,6 @@ from pydantic import BaseModel, Field
|
|||||||
|
|
||||||
from app.gateway.authz import require_permission
|
from app.gateway.authz import require_permission
|
||||||
from app.gateway.deps import get_current_user, get_feedback_repo, get_run_store
|
from app.gateway.deps import get_current_user, get_feedback_repo, get_run_store
|
||||||
from deerflow.persistence.feedback import FeedbackAlreadyExistsError
|
|
||||||
from deerflow.utils.thread_id import ThreadId
|
from deerflow.utils.thread_id import ThreadId
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@ -134,7 +133,6 @@ async def create_feedback(
|
|||||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
|
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
|
||||||
|
|
||||||
feedback_repo = get_feedback_repo(request)
|
feedback_repo = get_feedback_repo(request)
|
||||||
try:
|
|
||||||
return await feedback_repo.create(
|
return await feedback_repo.create(
|
||||||
run_id=run_id,
|
run_id=run_id,
|
||||||
thread_id=thread_id,
|
thread_id=thread_id,
|
||||||
@ -143,11 +141,6 @@ async def create_feedback(
|
|||||||
message_id=body.message_id,
|
message_id=body.message_id,
|
||||||
comment=body.comment,
|
comment=body.comment,
|
||||||
)
|
)
|
||||||
except FeedbackAlreadyExistsError:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=409,
|
|
||||||
detail="Feedback already exists for this run; use PUT to update it",
|
|
||||||
) from None
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{thread_id}/runs/{run_id}/feedback", response_model=list[FeedbackResponse])
|
@router.get("/{thread_id}/runs/{run_id}/feedback", response_model=list[FeedbackResponse])
|
||||||
|
|||||||
@ -17,7 +17,7 @@ from deerflow.config.agents_config import load_agent_config
|
|||||||
from deerflow.config.app_config import AppConfig
|
from deerflow.config.app_config import AppConfig
|
||||||
from deerflow.knowledge import KnowledgeRequestContext, RetrievalCatalogPage
|
from deerflow.knowledge import KnowledgeRequestContext, RetrievalCatalogPage
|
||||||
from deerflow.knowledge.providers import get_knowledge_capabilities, resolve_knowledge_provider
|
from deerflow.knowledge.providers import get_knowledge_capabilities, resolve_knowledge_provider
|
||||||
from deerflow.runtime.user_context import get_authenticated_user_id, get_effective_user_id
|
from deerflow.runtime.user_context import get_effective_user_id
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
router = APIRouter(prefix="/api/knowledge", tags=["knowledge"])
|
router = APIRouter(prefix="/api/knowledge", tags=["knowledge"])
|
||||||
@ -94,8 +94,7 @@ async def list_retrieval_catalog_datasets(
|
|||||||
config: AppConfig = Depends(get_config),
|
config: AppConfig = Depends(get_config),
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
provider = await _scope_catalog(config, agent_name, request, "catalog")
|
provider = await _scope_catalog(config, agent_name, request, "catalog")
|
||||||
context = KnowledgeRequestContext(user_id=get_effective_user_id(), agent_name=agent_name, authenticated_user_id=get_authenticated_user_id())
|
return await _catalog_result(provider.catalog(context=KnowledgeRequestContext(user_id=get_effective_user_id(), agent_name=agent_name), page=page, page_size=page_size, search=search))
|
||||||
return await _catalog_result(provider.catalog(context=context, page=page, page_size=page_size, search=search))
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/retrieval-catalog/datasets/{dataset_id:path}/documents")
|
@router.get("/retrieval-catalog/datasets/{dataset_id:path}/documents")
|
||||||
@ -110,5 +109,4 @@ async def list_retrieval_catalog_documents(
|
|||||||
config: AppConfig = Depends(get_config),
|
config: AppConfig = Depends(get_config),
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
provider = await _scope_catalog(config, agent_name, request, "documents")
|
provider = await _scope_catalog(config, agent_name, request, "documents")
|
||||||
context = KnowledgeRequestContext(user_id=get_effective_user_id(), agent_name=agent_name, authenticated_user_id=get_authenticated_user_id())
|
return await _catalog_result(provider.documents(dataset_id, context=KnowledgeRequestContext(user_id=get_effective_user_id(), agent_name=agent_name), page=page, page_size=page_size, search=search))
|
||||||
return await _catalog_result(provider.documents(dataset_id, context=context, page=page, page_size=page_size, search=search))
|
|
||||||
|
|||||||
@ -8,7 +8,7 @@ from fastapi import APIRouter, HTTPException, Request
|
|||||||
from fastapi.exceptions import RequestValidationError
|
from fastapi.exceptions import RequestValidationError
|
||||||
from fastapi.routing import APIRoute
|
from fastapi.routing import APIRoute
|
||||||
from langchain_core.messages import HumanMessage
|
from langchain_core.messages import HumanMessage
|
||||||
from pydantic import BaseModel, ConfigDict, Field, SecretStr, field_validator, model_validator
|
from pydantic import BaseModel, ConfigDict, model_validator
|
||||||
|
|
||||||
from app.gateway.deps import require_admin_user
|
from app.gateway.deps import require_admin_user
|
||||||
from app.gateway.persistent_writes import run_drained_write
|
from app.gateway.persistent_writes import run_drained_write
|
||||||
@ -180,100 +180,3 @@ async def test_model(request: Request, body: SaveModelRequest):
|
|||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
# Provider exception strings may contain keys, URLs or response bodies.
|
# Provider exception strings may contain keys, URLs or response bodies.
|
||||||
return {"ok": False, "message": "authentication_failed" if getattr(exc, "status_code", None) in (401, 403) else "connection_failed"}
|
return {"ok": False, "message": "authentication_failed" if getattr(exc, "status_code", None) in (401, 403) else "connection_failed"}
|
||||||
|
|
||||||
|
|
||||||
class ModelConnection(BaseModel):
|
|
||||||
"""A transient connection draft; credentials never enter a browser query cache."""
|
|
||||||
|
|
||||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
|
||||||
provider: str = "openai"
|
|
||||||
base_url: str | None = None
|
|
||||||
api_key: SecretStr | None = None
|
|
||||||
saved_model_name: str | None = Field(default=None, max_length=100)
|
|
||||||
expected_revision: str | None = Field(default=None, max_length=100)
|
|
||||||
|
|
||||||
@model_validator(mode="after")
|
|
||||||
def valid_connection(self):
|
|
||||||
ManagedModel(provider=self.provider, base_url=self.base_url, api_key=self.api_key, model="catalog-check")
|
|
||||||
if bool(self.saved_model_name) != bool(self.expected_revision):
|
|
||||||
raise ValueError("Saved connection requires its revision")
|
|
||||||
return self
|
|
||||||
|
|
||||||
|
|
||||||
class ImportModelRequest(BaseModel):
|
|
||||||
model_config = ConfigDict(extra="forbid")
|
|
||||||
connection: ModelConnection
|
|
||||||
model: str = Field(min_length=1, max_length=200)
|
|
||||||
|
|
||||||
@field_validator("model")
|
|
||||||
@classmethod
|
|
||||||
def valid_id(cls, value):
|
|
||||||
if value != value.strip():
|
|
||||||
raise ValueError("Invalid model ID")
|
|
||||||
return ManagedModel.valid_model_id(value)
|
|
||||||
|
|
||||||
|
|
||||||
def _connection_profile(body: ModelConnection, model: str = "catalog-check") -> ManagedModel:
|
|
||||||
previous = None
|
|
||||||
if body.saved_model_name:
|
|
||||||
previous = next((item for item in ManagedModelStore().list() if item.name == body.saved_model_name), None)
|
|
||||||
if previous is None or previous.revision != body.expected_revision:
|
|
||||||
raise HTTPException(409, "Connection changed; reload before continuing")
|
|
||||||
profile = ManagedModel(provider=body.provider, base_url=body.base_url, api_key=body.api_key, model=model)
|
|
||||||
profile = profile.model_copy(update={"api_key": retained_api_key(profile, previous)})
|
|
||||||
if profile.provider != "openai-compatible" and not (profile.api_key and profile.api_key.get_secret_value()):
|
|
||||||
raise HTTPException(422, {"code": "missing_api_key", "message": "Enter the connection API key"})
|
|
||||||
return profile
|
|
||||||
|
|
||||||
|
|
||||||
def _same_connection(left: ManagedModel, right: ManagedModel) -> bool:
|
|
||||||
return (left.provider, left.base_url, left.api_key) == (right.provider, right.base_url, right.api_key)
|
|
||||||
|
|
||||||
|
|
||||||
def _existing_import(profile: ManagedModel) -> ManagedModel | None:
|
|
||||||
return next((item for item in ManagedModelStore().list() if item.model == profile.model and _same_connection(item, profile)), None)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/discover")
|
|
||||||
async def discover_models(request: Request, body: ModelConnection):
|
|
||||||
from app.gateway.model_discovery import DiscoveryError, fetch_model_catalog
|
|
||||||
|
|
||||||
await require_admin_user(request, detail=_ADMIN)
|
|
||||||
try:
|
|
||||||
profile = await asyncio.to_thread(_connection_profile, body)
|
|
||||||
ids, truncated = await fetch_model_catalog(profile)
|
|
||||||
records = await asyncio.to_thread(lambda: ManagedModelStore().list())
|
|
||||||
configured = {item.model for item in records if _same_connection(item, profile)}
|
|
||||||
return {"models": [{"id": mid, "configured": mid in configured} for mid in ids], "truncated": truncated}
|
|
||||||
except DiscoveryError as exc:
|
|
||||||
raise HTTPException(502, {"code": exc.code, "message": "Could not retrieve the model catalog"}) from None
|
|
||||||
except (OSError, ValueError):
|
|
||||||
raise HTTPException(503, "Managed model storage is unavailable") from None
|
|
||||||
|
|
||||||
|
|
||||||
def _save_import(profile: ManagedModel, connection: ModelConnection):
|
|
||||||
config = get_app_config()
|
|
||||||
reserved = {item.name for item in config.models if item.name not in config._managed_model_names}
|
|
||||||
source = (connection.saved_model_name, connection.expected_revision) if connection.saved_model_name and connection.expected_revision else None
|
|
||||||
try:
|
|
||||||
return ManagedModelStore().save(profile, expected_revision=None, reserved_names=reserved, deduplicate=True, expected_source=source).public()
|
|
||||||
except FileExistsError:
|
|
||||||
raise HTTPException(409, "Connection changed; reload before importing") from None
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/import")
|
|
||||||
async def import_model(request: Request, body: ImportModelRequest):
|
|
||||||
"""One bounded, retry-safe item. The UI sequences items and reports partial success."""
|
|
||||||
await require_admin_user(request, detail=_ADMIN)
|
|
||||||
try:
|
|
||||||
profile = await asyncio.to_thread(_connection_profile, body.connection, body.model)
|
|
||||||
existing = await asyncio.to_thread(_existing_import, profile)
|
|
||||||
if existing:
|
|
||||||
return {"status": "existing", "model": existing.public()}
|
|
||||||
probe = await test_model(request, SaveModelRequest(config=profile))
|
|
||||||
if not probe["ok"]:
|
|
||||||
return {"status": "failed", "message": probe["message"]}
|
|
||||||
saved = await run_drained_write("Import managed model", _save_import, (HTTPException,), profile, body.connection)
|
|
||||||
return {"status": "saved", "model": saved}
|
|
||||||
except (OSError, ValueError):
|
|
||||||
raise HTTPException(503, "Managed model storage is unavailable") from None
|
|
||||||
|
|||||||
@ -15,7 +15,6 @@ from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_valida
|
|||||||
from app.gateway.deps import require_admin_user
|
from app.gateway.deps import require_admin_user
|
||||||
from app.gateway.mcp_oauth_tokens import forget_unconfigured_mcp_oauth_tokens
|
from app.gateway.mcp_oauth_tokens import forget_unconfigured_mcp_oauth_tokens
|
||||||
from app.gateway.persistent_writes import run_drained_write
|
from app.gateway.persistent_writes import run_drained_write
|
||||||
from app.gateway.secret_masking import MASKED_VALUE, mask_secret_values, normalize_config_key
|
|
||||||
from deerflow.config.extensions_config import (
|
from deerflow.config.extensions_config import (
|
||||||
ExtensionsConfig,
|
ExtensionsConfig,
|
||||||
McpRoutingConfig,
|
McpRoutingConfig,
|
||||||
@ -458,7 +457,6 @@ class McpServerConfigResponse(BaseModel):
|
|||||||
url: str | None = Field(default=None, description="URL of the MCP server (for sse or http type)")
|
url: str | None = Field(default=None, description="URL of the MCP server (for sse or http type)")
|
||||||
headers: dict[str, str] = Field(default_factory=dict, description="HTTP headers to send (for sse or http type)")
|
headers: dict[str, str] = Field(default_factory=dict, description="HTTP headers to send (for sse or http type)")
|
||||||
oauth: McpOAuthConfigResponse | None = Field(default=None, description="OAuth configuration for MCP HTTP/SSE servers")
|
oauth: McpOAuthConfigResponse | None = Field(default=None, description="OAuth configuration for MCP HTTP/SSE servers")
|
||||||
agent_auth: bool = Field(default=False, description="Require an administrator-granted agent credential on this deployment HTTP server")
|
|
||||||
user_auth: McpUserScopedAuthConfigResponse | None = Field(default=None, description="Per-user credential injection for MCP HTTP/SSE servers")
|
user_auth: McpUserScopedAuthConfigResponse | None = Field(default=None, description="Per-user credential injection for MCP HTTP/SSE servers")
|
||||||
headers_from_context: McpContextHeadersConfigResponse | None = Field(default=None, description="Per-request credential injection for MCP HTTP/SSE servers: map header names to config.context.secrets keys")
|
headers_from_context: McpContextHeadersConfigResponse | None = Field(default=None, description="Per-request credential injection for MCP HTTP/SSE servers: map header names to config.context.secrets keys")
|
||||||
description: str = Field(default="", description="Human-readable description of what this MCP server provides")
|
description: str = Field(default="", description="Human-readable description of what this MCP server provides")
|
||||||
@ -554,19 +552,29 @@ class McpCacheResetResponse(BaseModel):
|
|||||||
message: str = Field(description="Human-readable reset status")
|
message: str = Field(description="Human-readable reset status")
|
||||||
|
|
||||||
|
|
||||||
_MASKED_VALUE = MASKED_VALUE
|
_MASKED_VALUE = "***"
|
||||||
_SENSITIVE_EXTRA_KEY_RE = re.compile(
|
_SENSITIVE_EXTRA_KEY_RE = re.compile(
|
||||||
r"(^|_)(api_key|apikey|access_key|private_key|client_secret|secret|token|password|passwd|credential|credentials|authorization|bearer)(_|$)",
|
r"(^|_)(api_key|apikey|access_key|private_key|client_secret|secret|token|password|passwd|credential|credentials|authorization|bearer)(_|$)",
|
||||||
re.IGNORECASE,
|
re.IGNORECASE,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_config_key(key: str) -> str:
|
||||||
|
with_boundaries = re.sub(r"(.)([A-Z][a-z]+)", r"\1_\2", key)
|
||||||
|
with_boundaries = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", with_boundaries)
|
||||||
|
return re.sub(r"[^a-z0-9]+", "_", with_boundaries.lower()).strip("_")
|
||||||
|
|
||||||
|
|
||||||
def _is_sensitive_extra_key(key: str) -> bool:
|
def _is_sensitive_extra_key(key: str) -> bool:
|
||||||
return bool(_SENSITIVE_EXTRA_KEY_RE.search(normalize_config_key(key)))
|
return bool(_SENSITIVE_EXTRA_KEY_RE.search(_normalize_config_key(key)))
|
||||||
|
|
||||||
|
|
||||||
def _mask_sensitive_extra_value(value: Any) -> Any:
|
def _mask_sensitive_extra_value(value: Any) -> Any:
|
||||||
return mask_secret_values(value, _is_sensitive_extra_key)
|
if isinstance(value, dict):
|
||||||
|
return {key: _MASKED_VALUE if _is_sensitive_extra_key(str(key)) else _mask_sensitive_extra_value(nested) for key, nested in value.items()}
|
||||||
|
if isinstance(value, list):
|
||||||
|
return [_mask_sensitive_extra_value(item) for item in value]
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
def _contains_masked_sensitive_extra_value(key: str, value: Any) -> bool:
|
def _contains_masked_sensitive_extra_value(key: str, value: Any) -> bool:
|
||||||
@ -1107,9 +1115,6 @@ def _merge_preserving_secrets(
|
|||||||
}
|
}
|
||||||
if preserve_omitted_fields and "user_auth" not in incoming.model_fields_set:
|
if preserve_omitted_fields and "user_auth" not in incoming.model_fields_set:
|
||||||
update["user_auth"] = existing.user_auth
|
update["user_auth"] = existing.user_auth
|
||||||
# Changing an ordinary field must not silently remove credential enforcement.
|
|
||||||
if "agent_auth" not in incoming.model_fields_set:
|
|
||||||
update["agent_auth"] = existing.agent_auth
|
|
||||||
if preserve_omitted_fields and "headers_from_context" not in incoming.model_fields_set:
|
if preserve_omitted_fields and "headers_from_context" not in incoming.model_fields_set:
|
||||||
update["headers_from_context"] = existing.headers_from_context
|
update["headers_from_context"] = existing.headers_from_context
|
||||||
if preserve_omitted_fields and "routing" not in incoming.model_fields_set:
|
if preserve_omitted_fields and "routing" not in incoming.model_fields_set:
|
||||||
|
|||||||
@ -1,7 +1,6 @@
|
|||||||
"""Memory API router for retrieving and managing global memory data."""
|
"""Memory API router for retrieving and managing global memory data."""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import re
|
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Request
|
from fastapi import APIRouter, HTTPException, Request
|
||||||
@ -10,13 +9,11 @@ from pydantic import BaseModel, Field, field_validator
|
|||||||
from app.gateway.authz import require_permission
|
from app.gateway.authz import require_permission
|
||||||
from app.gateway.internal_auth import get_trusted_internal_owner_user_id
|
from app.gateway.internal_auth import get_trusted_internal_owner_user_id
|
||||||
from app.gateway.persistent_writes import run_drained_write
|
from app.gateway.persistent_writes import run_drained_write
|
||||||
from app.gateway.secret_masking import mask_secret_values, normalize_config_key
|
from deerflow.agents.memory import MemoryConflictError, MemoryCorruptionError, MemoryManager, get_memory_manager
|
||||||
from deerflow.agents.memory import MemoryConflictError, MemoryCorruptionError, MemoryDuplicateFactIdError, MemoryManager, get_memory_manager
|
|
||||||
from deerflow.config.agents_config import AGENT_NAME_PATTERN
|
from deerflow.config.agents_config import AGENT_NAME_PATTERN
|
||||||
from deerflow.config.memory_config import get_memory_config
|
from deerflow.config.memory_config import get_memory_config
|
||||||
from deerflow.config.paths import make_safe_user_id
|
from deerflow.config.paths import make_safe_user_id
|
||||||
from deerflow.runtime.user_context import get_effective_user_id
|
from deerflow.runtime.user_context import get_effective_user_id
|
||||||
from deerflow.utils.host_paths import windows_incompatible_segment
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/api", tags=["memory"])
|
router = APIRouter(prefix="/api", tags=["memory"])
|
||||||
|
|
||||||
@ -120,12 +117,6 @@ class MemoryResponse(BaseModel):
|
|||||||
facts: list[Fact] = Field(default_factory=list)
|
facts: list[Fact] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
class MemoryImportRequest(MemoryResponse):
|
|
||||||
"""Replacement imports must explicitly supply the facts to restore."""
|
|
||||||
|
|
||||||
facts: list[Fact] = Field(description="Facts to restore. An empty list explicitly clears the selected fact bucket.")
|
|
||||||
|
|
||||||
|
|
||||||
def _map_memory_fact_value_error(exc: ValueError) -> HTTPException:
|
def _map_memory_fact_value_error(exc: ValueError) -> HTTPException:
|
||||||
"""Convert updater validation errors into stable API responses."""
|
"""Convert updater validation errors into stable API responses."""
|
||||||
if exc.args and exc.args[0] == "confidence":
|
if exc.args and exc.args[0] == "confidence":
|
||||||
@ -168,8 +159,6 @@ def _unsupported_501(manager: object, label: str) -> HTTPException:
|
|||||||
def _management_agent_name_or_501(
|
def _management_agent_name_or_501(
|
||||||
manager: MemoryManager,
|
manager: MemoryManager,
|
||||||
agent_name: str | None,
|
agent_name: str | None,
|
||||||
*,
|
|
||||||
for_creation: bool = False,
|
|
||||||
) -> str | None:
|
) -> str | None:
|
||||||
"""Validate an optional agent-scoped management target.
|
"""Validate an optional agent-scoped management target.
|
||||||
|
|
||||||
@ -179,16 +168,6 @@ def _management_agent_name_or_501(
|
|||||||
into a mutation of the wrong bucket. Preserve the caller's spelling here:
|
into a mutation of the wrong bucket. Preserve the caller's spelling here:
|
||||||
remote backends such as Mem0 use the agent name as a case-sensitive storage
|
remote backends such as Mem0 use the agent name as a case-sensitive storage
|
||||||
identity, while DeerMem performs its own lowercase canonicalization.
|
identity, while DeerMem performs its own lowercase canonicalization.
|
||||||
|
|
||||||
``for_creation`` also refuses names that cannot become a host path
|
|
||||||
component. DeerMem resolves an agent scope to
|
|
||||||
``{storage}/users/{uid}/agents/{name}/facts`` and creates it on the first
|
|
||||||
write; the character class above admits Windows reserved device names
|
|
||||||
(``NUL``, ``CON``, ``COM1``, ...), so that write raised
|
|
||||||
``FileNotFoundError`` and the endpoint's ``OSError`` branch answered
|
|
||||||
``500``. Only the two endpoints that create the scope set this: reads and
|
|
||||||
deletes stay permissive so a scope an older release stored can still be
|
|
||||||
exported and cleared.
|
|
||||||
"""
|
"""
|
||||||
if agent_name is None:
|
if agent_name is None:
|
||||||
return None
|
return None
|
||||||
@ -197,20 +176,6 @@ def _management_agent_name_or_501(
|
|||||||
status_code=422,
|
status_code=422,
|
||||||
detail=(f"Invalid agent name '{agent_name}'. Must match {AGENT_NAME_PATTERN.pattern} (letters, digits, and hyphens only)."),
|
detail=(f"Invalid agent name '{agent_name}'. Must match {AGENT_NAME_PATTERN.pattern} (letters, digits, and hyphens only)."),
|
||||||
)
|
)
|
||||||
if for_creation:
|
|
||||||
# Reuse the canonical Windows-name check that agent directories,
|
|
||||||
# thread IDs, uploads, sandbox writes and skill support files use.
|
|
||||||
# Sibling PR #7939 applies the same rule inside the shared
|
|
||||||
# ``validate_agent_name()`` (``deerflow.config.agents_config``); this
|
|
||||||
# router inlines ``AGENT_NAME_PATTERN`` and preserves caller spelling for
|
|
||||||
# case-sensitive backends rather than delegating to it, so keep the two
|
|
||||||
# copies in sync if the reserved set ever changes.
|
|
||||||
reason = windows_incompatible_segment(agent_name)
|
|
||||||
if reason is not None:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=422,
|
|
||||||
detail=f"Invalid agent name '{agent_name}'. Contains a {reason}.",
|
|
||||||
)
|
|
||||||
if getattr(manager, "supports_agent_scoped_management", False) is not True:
|
if getattr(manager, "supports_agent_scoped_management", False) is not True:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=501,
|
status_code=501,
|
||||||
@ -269,18 +234,6 @@ class FactPatchRequest(BaseModel):
|
|||||||
confidence: float | None = Field(default=None, ge=0.0, le=1.0, description="Confidence score (0-1)")
|
confidence: float | None = Field(default=None, ge=0.0, le=1.0, description="Confidence score (0-1)")
|
||||||
|
|
||||||
|
|
||||||
# Anchored at the end of the key: DeerMem knobs such as ``token_counting`` and
|
|
||||||
# ``guaranteed_token_budget`` are not secrets, and ``api_key_env`` names an
|
|
||||||
# environment variable rather than holding the key. This is a denylist: plural
|
|
||||||
# ``tokens`` stays out because ``max_injection_tokens`` is a knob, and bare
|
|
||||||
# ``key``, ``auth`` or ``session`` are too generic to mask.
|
|
||||||
_SECRET_CONFIG_KEY_RE = re.compile(r"(^|_)(api_?keys?|access_keys?|secret_keys?|private_keys?|auth_keys?|secrets?|token|passwords?|passwds?|credentials?|authorization|bearer|cookies?)$")
|
|
||||||
|
|
||||||
|
|
||||||
def _is_secret_config_key(key: str) -> bool:
|
|
||||||
return bool(_SECRET_CONFIG_KEY_RE.search(normalize_config_key(key)))
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryConfigResponse(BaseModel):
|
class MemoryConfigResponse(BaseModel):
|
||||||
"""Response model for memory configuration."""
|
"""Response model for memory configuration."""
|
||||||
|
|
||||||
@ -289,13 +242,7 @@ class MemoryConfigResponse(BaseModel):
|
|||||||
injection_enabled: bool = Field(..., description="Whether memory is injected into the system prompt (call-site gate).")
|
injection_enabled: bool = Field(..., description="Whether memory is injected into the system prompt (call-site gate).")
|
||||||
shutdown_flush_timeout_seconds: float = Field(..., description="Hard budget (s) to drain pending memory updates on Gateway graceful shutdown; must fit inside the pod's K8s terminationGracePeriodSeconds.")
|
shutdown_flush_timeout_seconds: float = Field(..., description="Hard budget (s) to drain pending memory updates on Gateway graceful shutdown; must fit inside the pod's K8s terminationGracePeriodSeconds.")
|
||||||
manager_class: str = Field(..., description="Active memory backend selector (backend name or dotted path).")
|
manager_class: str = Field(..., description="Active memory backend selector (backend name or dotted path).")
|
||||||
backend_config: dict = Field(..., description="Backend-private config (self-interpreted by the backend); credential values such as api_key are masked as ***.")
|
backend_config: dict = Field(..., description="Backend-private config (self-interpreted by the backend).")
|
||||||
|
|
||||||
@field_validator("backend_config")
|
|
||||||
@classmethod
|
|
||||||
def _mask_backend_secrets(cls, value: dict) -> dict:
|
|
||||||
"""Config loading already resolved ``$VAR`` references, so keys would be plaintext."""
|
|
||||||
return mask_secret_values(value, _is_secret_config_key, keep_unset=True)
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryStatusResponse(BaseModel):
|
class MemoryStatusResponse(BaseModel):
|
||||||
@ -445,7 +392,7 @@ async def clear_memory(request: Request, agent_name: str | None = None) -> Memor
|
|||||||
async def create_memory_fact_endpoint(body: FactCreateRequest, request: Request, agent_name: str | None = None) -> MemoryResponse:
|
async def create_memory_fact_endpoint(body: FactCreateRequest, request: Request, agent_name: str | None = None) -> MemoryResponse:
|
||||||
"""Create a single fact manually."""
|
"""Create a single fact manually."""
|
||||||
manager = await asyncio.to_thread(get_memory_manager)
|
manager = await asyncio.to_thread(get_memory_manager)
|
||||||
selected_agent = _management_agent_name_or_501(manager, agent_name, for_creation=True)
|
selected_agent = _management_agent_name_or_501(manager, agent_name)
|
||||||
scope_kwargs = _agent_scope_kwargs(selected_agent)
|
scope_kwargs = _agent_scope_kwargs(selected_agent)
|
||||||
try:
|
try:
|
||||||
memory_data, fact_id = await run_drained_write(
|
memory_data, fact_id = await run_drained_write(
|
||||||
@ -572,16 +519,13 @@ async def export_memory(request: Request, agent_name: str | None = None) -> Memo
|
|||||||
response_model=MemoryResponse,
|
response_model=MemoryResponse,
|
||||||
response_model_exclude_none=True,
|
response_model_exclude_none=True,
|
||||||
summary="Import Memory Data",
|
summary="Import Memory Data",
|
||||||
description=(
|
description="Import the default memory document, or replace only one selected agent's facts while preserving shared summaries.",
|
||||||
"Import the default memory document, or replace only one selected agent's facts while preserving shared summaries. "
|
|
||||||
"The request requires an explicit `facts` array. Omitting it returns HTTP 422, and an empty array clears the selected fact bucket."
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
@require_permission("memory", "write")
|
@require_permission("memory", "write")
|
||||||
async def import_memory(body: MemoryImportRequest, request: Request, agent_name: str | None = None) -> MemoryResponse:
|
async def import_memory(body: MemoryResponse, request: Request, agent_name: str | None = None) -> MemoryResponse:
|
||||||
"""Import memory data, preserving shared summaries for a selected agent."""
|
"""Import memory data, preserving shared summaries for a selected agent."""
|
||||||
manager = await asyncio.to_thread(get_memory_manager)
|
manager = await asyncio.to_thread(get_memory_manager)
|
||||||
selected_agent = _management_agent_name_or_501(manager, agent_name, for_creation=True)
|
selected_agent = _management_agent_name_or_501(manager, agent_name)
|
||||||
scope_kwargs = _agent_scope_kwargs(selected_agent)
|
scope_kwargs = _agent_scope_kwargs(selected_agent)
|
||||||
try:
|
try:
|
||||||
memory_data = await run_drained_write(
|
memory_data = await run_drained_write(
|
||||||
@ -596,8 +540,6 @@ async def import_memory(body: MemoryImportRequest, request: Request, agent_name:
|
|||||||
raise _unsupported_501(manager, "import memory") from None
|
raise _unsupported_501(manager, "import memory") from None
|
||||||
except (MemoryConflictError, MemoryCorruptionError) as exc:
|
except (MemoryConflictError, MemoryCorruptionError) as exc:
|
||||||
raise _map_memory_manager_error(exc) from exc
|
raise _map_memory_manager_error(exc) from exc
|
||||||
except MemoryDuplicateFactIdError as exc:
|
|
||||||
raise HTTPException(status_code=400, detail="Invalid memory import: fact IDs must be unique.") from exc
|
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
raise HTTPException(status_code=400, detail="Invalid memory import: facts must be a list of objects with non-empty content.") from exc
|
raise HTTPException(status_code=400, detail="Invalid memory import: facts must be a list of objects with non-empty content.") from exc
|
||||||
except OSError as exc:
|
except OSError as exc:
|
||||||
|
|||||||
@ -66,7 +66,6 @@ class ModelResponse(BaseModel):
|
|||||||
description: str | None = Field(None, description="Model description")
|
description: str | None = Field(None, description="Model description")
|
||||||
supports_thinking: bool = Field(default=False, description="Whether model supports thinking mode (deprecated: derived from `reasoning`)")
|
supports_thinking: bool = Field(default=False, description="Whether model supports thinking mode (deprecated: derived from `reasoning`)")
|
||||||
supports_reasoning_effort: bool = Field(default=False, description="Whether model supports reasoning effort (deprecated: derived from `reasoning`)")
|
supports_reasoning_effort: bool = Field(default=False, description="Whether model supports reasoning effort (deprecated: derived from `reasoning`)")
|
||||||
provider: str | None = Field(default=None, description="Managed provider display label; no endpoint or credentials")
|
|
||||||
reasoning: ReasoningCapabilitiesResponse = Field(..., description="Normalized reasoning capability contract")
|
reasoning: ReasoningCapabilitiesResponse = Field(..., description="Normalized reasoning capability contract")
|
||||||
|
|
||||||
|
|
||||||
@ -76,7 +75,6 @@ def _model_response(model: ModelConfig) -> ModelResponse:
|
|||||||
model=model.model,
|
model=model.model,
|
||||||
display_name=model.display_name,
|
display_name=model.display_name,
|
||||||
description=model.description,
|
description=model.description,
|
||||||
provider=model._managed_provider,
|
|
||||||
supports_thinking=model.supports_thinking,
|
supports_thinking=model.supports_thinking,
|
||||||
supports_reasoning_effort=model.supports_reasoning_effort,
|
supports_reasoning_effort=model.supports_reasoning_effort,
|
||||||
reasoning=ReasoningCapabilitiesResponse(**reasoning_capabilities_payload(resolve_reasoning_contract(model))),
|
reasoning=ReasoningCapabilitiesResponse(**reasoning_capabilities_payload(resolve_reasoning_contract(model))),
|
||||||
@ -197,7 +195,7 @@ async def list_models(
|
|||||||
owner = await _resolve_bound_owner_for_model_list(request)
|
owner = await _resolve_bound_owner_for_model_list(request)
|
||||||
principal_user = owner if owner is not None else user
|
principal_user = owner if owner is not None else user
|
||||||
try:
|
try:
|
||||||
provider, principal = await asyncio.to_thread(resolve_model_authorization, principal_user, is_internal=_is_internal_caller(request, user))
|
provider, principal = resolve_model_authorization(principal_user, is_internal=_is_internal_caller(request, user))
|
||||||
except _AuthorizationUnavailable as exc:
|
except _AuthorizationUnavailable as exc:
|
||||||
if exc.fail_closed:
|
if exc.fail_closed:
|
||||||
visible_models = []
|
visible_models = []
|
||||||
@ -263,6 +261,6 @@ async def get_model(
|
|||||||
# model exists but the role lacks permission to use it).
|
# model exists but the role lacks permission to use it).
|
||||||
user = await get_optional_user_from_request(request)
|
user = await get_optional_user_from_request(request)
|
||||||
if user is not None:
|
if user is not None:
|
||||||
await asyncio.to_thread(authorize_model_use, user, model_name, is_internal=_is_internal_caller(request, user), app_config=config)
|
authorize_model_use(user, model_name, is_internal=_is_internal_caller(request, user), app_config=config)
|
||||||
|
|
||||||
return _model_response(model)
|
return _model_response(model)
|
||||||
|
|||||||
@ -45,8 +45,6 @@ def _read_personal_config(user_id: str) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def _validate_personal_server(server: mcp.McpServerConfigResponse, *, admin: bool) -> None:
|
def _validate_personal_server(server: mcp.McpServerConfigResponse, *, admin: bool) -> None:
|
||||||
if server.agent_auth:
|
|
||||||
raise HTTPException(422, "agent_auth is available only for deployment MCP servers")
|
|
||||||
# A personal store must not grant ordinary users the deployment operator's
|
# A personal store must not grant ordinary users the deployment operator's
|
||||||
# ability to launch arbitrary host packages or query internal services.
|
# ability to launch arbitrary host packages or query internal services.
|
||||||
if not admin:
|
if not admin:
|
||||||
|
|||||||
@ -70,7 +70,6 @@ async def list_plugins(request: Request, response: Response):
|
|||||||
principal = _principal(request)
|
principal = _principal(request)
|
||||||
response.headers["Cache-Control"] = "private, no-store"
|
response.headers["Cache-Control"] = "private, no-store"
|
||||||
entries = []
|
entries = []
|
||||||
documentation = dict(request.app.state.extensions.documentation)
|
|
||||||
for source, plugin in request.app.state.extensions.plugins:
|
for source, plugin in request.app.state.extensions.plugins:
|
||||||
settings = plugin_settings(source, plugin)
|
settings = plugin_settings(source, plugin)
|
||||||
module = plugin.frontend
|
module = plugin.frontend
|
||||||
@ -94,36 +93,8 @@ async def list_plugins(request: Request, response: Response):
|
|||||||
"transport": transport,
|
"transport": transport,
|
||||||
"settings": {key: settings[key] for key in public},
|
"settings": {key: settings[key] for key in public},
|
||||||
"backend_actions": [action.name for action in plugin.backend],
|
"backend_actions": [action.name for action in plugin.backend],
|
||||||
"model_tools": [tool.name for tool in plugin.tools],
|
|
||||||
"documentation": documentation.get(source) if documentation.get(source, {}).get("id") == plugin.namespace else None,
|
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
registered_sources = {source for source, _ in request.app.state.extensions.plugins}
|
|
||||||
known_ids = {entry["namespace"] for entry in entries}
|
|
||||||
for source, page in request.app.state.extensions.documentation:
|
|
||||||
if source in registered_sources or page["id"] in known_ids:
|
|
||||||
continue
|
|
||||||
locales = page["locales"]
|
|
||||||
copy = next((value for tag, value in locales.items() if tag.lower() == "en-us"), None)
|
|
||||||
if copy is None:
|
|
||||||
copy = next((value for tag, value in locales.items() if tag.partition("-")[0] == "en"), next(iter(locales.values())))
|
|
||||||
entries.append(
|
|
||||||
{
|
|
||||||
"namespace": page["id"],
|
|
||||||
"title": copy["title"],
|
|
||||||
"description": copy["description"],
|
|
||||||
"viewer_id": principal.user_id,
|
|
||||||
"module": None,
|
|
||||||
"entry": None,
|
|
||||||
"transport": None,
|
|
||||||
"settings": {"enabled": True},
|
|
||||||
"backend_actions": [],
|
|
||||||
"model_tools": [],
|
|
||||||
"documentation": page,
|
|
||||||
"runtime_only": True,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
known_ids.add(page["id"])
|
|
||||||
return entries
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -1,6 +1,5 @@
|
|||||||
"""CRUD API for projects (Phase 1: organization only — no documents/trash)."""
|
"""CRUD API for projects (Phase 1: organization only — no documents/trash)."""
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import logging
|
import logging
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
|
|
||||||
@ -9,12 +8,9 @@ from pydantic import BaseModel, Field, field_validator
|
|||||||
|
|
||||||
from app.gateway.authz import require_permission
|
from app.gateway.authz import require_permission
|
||||||
from app.gateway.deps import get_config, get_project_repo, get_thread_store
|
from app.gateway.deps import get_config, get_project_repo, get_thread_store
|
||||||
from app.gateway.persistent_writes import run_drained_write
|
|
||||||
from deerflow.config.app_config import AppConfig, get_app_config
|
from deerflow.config.app_config import AppConfig, get_app_config
|
||||||
from deerflow.config.projects_config import ProjectsConfig
|
from deerflow.config.projects_config import ProjectsConfig
|
||||||
from deerflow.projects.memory import MAX_FACTS, get_project_memory_repository
|
|
||||||
from deerflow.runtime.secret_context import redact_metadata_secrets
|
from deerflow.runtime.secret_context import redact_metadata_secrets
|
||||||
from deerflow.runtime.user_context import get_effective_user_id
|
|
||||||
from deerflow.utils.time import coerce_iso
|
from deerflow.utils.time import coerce_iso
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@ -49,72 +45,6 @@ class ProjectListResponse(BaseModel):
|
|||||||
projects: list[ProjectResponse]
|
projects: list[ProjectResponse]
|
||||||
|
|
||||||
|
|
||||||
class ProjectMemoryFactResponse(BaseModel):
|
|
||||||
id: str
|
|
||||||
content: str
|
|
||||||
source_thread_id: str
|
|
||||||
created_at: str
|
|
||||||
|
|
||||||
|
|
||||||
class ProjectMemoryResponse(BaseModel):
|
|
||||||
enabled: bool
|
|
||||||
facts: list[ProjectMemoryFactResponse]
|
|
||||||
max_facts: int = MAX_FACTS
|
|
||||||
|
|
||||||
|
|
||||||
class ProjectMemoryPatch(BaseModel):
|
|
||||||
enabled: bool
|
|
||||||
|
|
||||||
|
|
||||||
class ProjectMemoryPromotion(BaseModel):
|
|
||||||
expected_instructions: str
|
|
||||||
|
|
||||||
|
|
||||||
class ProjectMemoryPromotionResponse(BaseModel):
|
|
||||||
instructions: str
|
|
||||||
memory: ProjectMemoryResponse
|
|
||||||
|
|
||||||
|
|
||||||
async def _memory_call(method: str, project_id: str, *args, **kwargs):
|
|
||||||
user_id = get_effective_user_id()
|
|
||||||
|
|
||||||
def call():
|
|
||||||
return getattr(get_project_memory_repository(), method)(project_id, user_id, *args, **kwargs)
|
|
||||||
|
|
||||||
result = await asyncio.to_thread(call) if method == "get" else await run_drained_write("Project memory update", call, (ValueError,))
|
|
||||||
if result is None:
|
|
||||||
raise _not_found()
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{project_id}/memory", response_model=ProjectMemoryResponse)
|
|
||||||
@require_permission("projects", "read")
|
|
||||||
async def get_project_memory(project_id: str, request: Request) -> ProjectMemoryResponse:
|
|
||||||
return ProjectMemoryResponse(**await _memory_call("get", project_id))
|
|
||||||
|
|
||||||
|
|
||||||
@router.patch("/{project_id}/memory", response_model=ProjectMemoryResponse)
|
|
||||||
@require_permission("projects", "write")
|
|
||||||
async def patch_project_memory(project_id: str, body: ProjectMemoryPatch, request: Request) -> ProjectMemoryResponse:
|
|
||||||
return ProjectMemoryResponse(**await _memory_call("set_enabled", project_id, body.enabled))
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/{project_id}/memory/{fact_id}", response_model=ProjectMemoryResponse)
|
|
||||||
@require_permission("projects", "write")
|
|
||||||
async def delete_project_memory(project_id: str, fact_id: str, request: Request) -> ProjectMemoryResponse:
|
|
||||||
return ProjectMemoryResponse(**await _memory_call("delete", project_id, fact_id))
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/{project_id}/memory/{fact_id}/promote", response_model=ProjectMemoryPromotionResponse)
|
|
||||||
@require_permission("projects", "write")
|
|
||||||
async def promote_project_memory(project_id: str, fact_id: str, body: ProjectMemoryPromotion, request: Request) -> ProjectMemoryPromotionResponse:
|
|
||||||
try:
|
|
||||||
result = await _memory_call("promote", project_id, fact_id, max_bytes=_projects_config().instructions_max_bytes, expected_instructions=body.expected_instructions)
|
|
||||||
except ValueError as exc:
|
|
||||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
|
||||||
return ProjectMemoryPromotionResponse(instructions=result["instructions"], memory=ProjectMemoryResponse(**result["memory"]))
|
|
||||||
|
|
||||||
|
|
||||||
class ProjectsConfigResponse(BaseModel):
|
class ProjectsConfigResponse(BaseModel):
|
||||||
"""The projects-block knobs the UI needs before it can validate client-side."""
|
"""The projects-block knobs the UI needs before it can validate client-side."""
|
||||||
|
|
||||||
|
|||||||
@ -1,160 +0,0 @@
|
|||||||
"""Workspace search and direct navigation to an owner-visible message."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import base64
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
from dataclasses import asdict
|
|
||||||
from typing import Literal
|
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Query, Request
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
from app.gateway.authz import require_permission
|
|
||||||
from app.gateway.deps import get_run_event_store, get_run_manager, get_thread_store
|
|
||||||
from app.gateway.search_corpus import GatewaySearchCorpus
|
|
||||||
from deerflow.utils.thread_id import ThreadId
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
router = APIRouter(prefix="/api", tags=["search"])
|
|
||||||
|
|
||||||
|
|
||||||
class WorkspaceSearchRequest(BaseModel):
|
|
||||||
query: str = Field(default="", max_length=256)
|
|
||||||
types: list[Literal["thread", "message"]] = Field(default_factory=lambda: ["thread", "message"], min_length=1, max_length=2)
|
|
||||||
archived: bool | None = Field(default=None, strict=True)
|
|
||||||
project_id: str | None = Field(default=None, max_length=64)
|
|
||||||
limit: int = Field(default=20, ge=1, le=100)
|
|
||||||
cursor: str | None = Field(default=None, max_length=4096)
|
|
||||||
|
|
||||||
|
|
||||||
def snippet(text: str, query: str) -> list[dict]:
|
|
||||||
"""Plain text only. Map case-fold expansions back to source characters."""
|
|
||||||
normalized = text.casefold()
|
|
||||||
position = normalized.find(query.casefold()) if query else -1
|
|
||||||
if position < 0:
|
|
||||||
return [{"text": text[:200] + ("…" if len(text) > 200 else "")}]
|
|
||||||
offsets = []
|
|
||||||
for index, character in enumerate(text):
|
|
||||||
offsets.extend([index] * len(character.casefold()))
|
|
||||||
if len(offsets) >= position + len(query.casefold()):
|
|
||||||
break
|
|
||||||
start, end = offsets[position], offsets[position + len(query.casefold()) - 1] + 1
|
|
||||||
left, right = max(0, start - 65), min(len(text), end + 110)
|
|
||||||
return [{"text": ("…" if left else "") + text[left:start]}, {"text": text[start:end], "matched": True}, {"text": text[end:right] + ("…" if right < len(text) else "")}]
|
|
||||||
|
|
||||||
|
|
||||||
def _corpus(request: Request) -> GatewaySearchCorpus:
|
|
||||||
return GatewaySearchCorpus(get_thread_store(request), get_run_event_store(request), get_run_manager(request), str(request.state.auth.require_user().id), request=request)
|
|
||||||
|
|
||||||
|
|
||||||
def _encode_cursor(scope: str, phase: str, position) -> str:
|
|
||||||
return base64.urlsafe_b64encode(json.dumps([scope, phase, position]).encode()).decode()
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/search")
|
|
||||||
@require_permission("threads", "read")
|
|
||||||
async def search_workspace(body: WorkspaceSearchRequest, request: Request):
|
|
||||||
corpus = _corpus(request)
|
|
||||||
query = body.query.strip()
|
|
||||||
scope = hashlib.sha256(json.dumps([corpus.user_id, query, sorted(set(body.types)), body.archived, body.project_id, "project_id" in body.model_fields_set]).encode()).hexdigest()
|
|
||||||
phase, position = "thread", None
|
|
||||||
if body.cursor:
|
|
||||||
try:
|
|
||||||
saved_scope, phase, position = json.loads(base64.urlsafe_b64decode(body.cursor))
|
|
||||||
if saved_scope != scope or phase not in {"thread", "message"}:
|
|
||||||
raise ValueError
|
|
||||||
if phase == "thread" and (not isinstance(position, list) or len(position) != 3 or not isinstance(position[0], int) or not all(isinstance(v, str) for v in position[1:])):
|
|
||||||
raise ValueError
|
|
||||||
if phase == "message" and position is not None and not isinstance(position, str):
|
|
||||||
raise ValueError
|
|
||||||
except (ValueError, TypeError) as exc:
|
|
||||||
raise HTTPException(400, "Invalid search cursor") from exc
|
|
||||||
all_threads = await corpus.threads()
|
|
||||||
threads = tuple(t for t in all_threads if (body.archived is None or t.archived == body.archived) and ("project_id" not in body.model_fields_set or t.project_id == body.project_id))
|
|
||||||
lookup = {t.thread_id: t for t in threads}
|
|
||||||
providers = getattr(getattr(request.app.state, "extensions", None), "workspace_search_providers", ())
|
|
||||||
messages_available = bool(providers) and request.state.auth.has_permission("runs", "read")
|
|
||||||
response = {"items": [], "cursor": None, "indexing": False, "messages_available": messages_available}
|
|
||||||
|
|
||||||
def thread_hit(thread):
|
|
||||||
return {
|
|
||||||
"type": "thread",
|
|
||||||
"thread_id": thread.thread_id,
|
|
||||||
"title": thread.title,
|
|
||||||
"agent_name": thread.agent_name,
|
|
||||||
"archived": thread.archived,
|
|
||||||
"project_id": thread.project_id,
|
|
||||||
"updated_at": thread.updated_at,
|
|
||||||
"snippet": snippet(thread.title, query),
|
|
||||||
}
|
|
||||||
|
|
||||||
if phase == "thread" and "thread" in body.types:
|
|
||||||
|
|
||||||
def key(thread):
|
|
||||||
title = thread.title.casefold()
|
|
||||||
return (2 if query and title == query.casefold() else 1, thread.updated_at, thread.thread_id)
|
|
||||||
|
|
||||||
matches = sorted((t for t in threads if not query or query.casefold() in t.title.casefold()), key=key, reverse=True)
|
|
||||||
if position is not None:
|
|
||||||
matches = [t for t in matches if key(t) < tuple(position)]
|
|
||||||
selected = matches[: body.limit]
|
|
||||||
response["items"].extend(thread_hit(t) for t in selected)
|
|
||||||
if len(matches) > body.limit:
|
|
||||||
response["cursor"] = _encode_cursor(scope, "thread", key(selected[-1]))
|
|
||||||
return response
|
|
||||||
if not query or "message" not in body.types or not messages_available:
|
|
||||||
return response
|
|
||||||
if len(response["items"]) == body.limit:
|
|
||||||
response["cursor"] = _encode_cursor(scope, "message", None)
|
|
||||||
return response
|
|
||||||
provider = providers[0][1]
|
|
||||||
provider_cursor = position if phase == "message" else None
|
|
||||||
# Invalidated candidates must not consume a visible result slot. Keep a
|
|
||||||
# bounded amount of work per HTTP page and expose continuation when needed.
|
|
||||||
for _ in range(5):
|
|
||||||
try:
|
|
||||||
page = await provider.search(corpus, query=query, threads=threads, limit=body.limit - len(response["items"]), cursor=provider_cursor)
|
|
||||||
except ValueError as exc:
|
|
||||||
raise HTTPException(400, "Invalid search cursor") from exc
|
|
||||||
except Exception:
|
|
||||||
logger.exception("Workspace search provider failed")
|
|
||||||
raise HTTPException(503, "Message search is temporarily unavailable") from None
|
|
||||||
response["indexing"] = page.indexing
|
|
||||||
for candidate in page.items:
|
|
||||||
thread = lookup.get(candidate.thread_id)
|
|
||||||
if thread is None or candidate.incarnation != thread.incarnation or not await corpus.owns(thread):
|
|
||||||
continue
|
|
||||||
# Re-read authoritative visible text before returning ANY snippet.
|
|
||||||
current = await corpus.latest_message(thread, candidate.message_id)
|
|
||||||
if current is None or current.seq != candidate.seq or query.casefold() not in current.text.casefold():
|
|
||||||
continue
|
|
||||||
hit = thread_hit(thread)
|
|
||||||
hit.update(type="message", message_id=current.message_id, seq=current.seq, role=current.role, updated_at=current.created_at, snippet=snippet(current.text, query))
|
|
||||||
response["items"].append(hit)
|
|
||||||
if page.indexing:
|
|
||||||
break
|
|
||||||
if page.cursor == provider_cursor and page.cursor is not None:
|
|
||||||
raise HTTPException(503, "Search pagination did not advance")
|
|
||||||
provider_cursor = page.cursor
|
|
||||||
if provider_cursor is None or len(response["items"]) >= body.limit:
|
|
||||||
break
|
|
||||||
if provider_cursor:
|
|
||||||
response["cursor"] = _encode_cursor(scope, "message", provider_cursor)
|
|
||||||
return response
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/threads/{thread_id}/messages/around")
|
|
||||||
@require_permission("runs", "read", owner_check=True, require_existing=True)
|
|
||||||
async def message_context(thread_id: ThreadId, request: Request, message_id: str = Query(min_length=1, max_length=512), seq: int | None = Query(default=None, ge=1), limit: int = Query(default=20, ge=1, le=50)):
|
|
||||||
corpus = _corpus(request)
|
|
||||||
window = await corpus.around(thread_id, message_id, seq=seq, limit=limit)
|
|
||||||
if window is None:
|
|
||||||
raise HTTPException(404, "Message is unavailable")
|
|
||||||
thread = await corpus.thread(thread_id)
|
|
||||||
if thread is None:
|
|
||||||
raise HTTPException(404, "Message is unavailable")
|
|
||||||
messages = [asdict(message) for row in window.pop("data") if (message := corpus.project(thread, row)) is not None]
|
|
||||||
return {**window, "messages": messages}
|
|
||||||
@ -1,130 +0,0 @@
|
|||||||
"""Owner-managed snapshots, public read capabilities, and authenticated imports."""
|
|
||||||
|
|
||||||
import logging
|
|
||||||
from datetime import UTC, datetime, timedelta
|
|
||||||
from typing import Annotated
|
|
||||||
from uuid import UUID
|
|
||||||
|
|
||||||
from deerflow_extension_api import AgentRunError
|
|
||||||
from fastapi import APIRouter, HTTPException, Path, Request
|
|
||||||
from fastapi.responses import JSONResponse, Response
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
from app.gateway.authz import require_permission
|
|
||||||
from app.gateway.routers.artifacts import _build_content_disposition, _slice_byte_range
|
|
||||||
from app.gateway.services import reserve_checkpoint_write
|
|
||||||
from app.gateway.share_fork import fork_share
|
|
||||||
from app.gateway.share_snapshot import capture_share
|
|
||||||
from deerflow.persistence.engine import get_session_factory
|
|
||||||
from deerflow.persistence.thread_shares.sql import ThreadShareRepository
|
|
||||||
from deerflow.runtime import ConflictError
|
|
||||||
from deerflow.runtime.user_context import get_effective_user_id
|
|
||||||
from deerflow.utils.file_io import await_drained
|
|
||||||
from deerflow.utils.thread_id import ThreadId
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
router = APIRouter(prefix="/api", tags=["shares"])
|
|
||||||
ShareToken = Annotated[str, Path(pattern=r"^[A-Za-z0-9_-]{43}$")]
|
|
||||||
PRIVATE_HEADERS = {"Cache-Control": "no-store", "Referrer-Policy": "no-referrer", "X-Robots-Tag": "noindex, nofollow", "X-Content-Type-Options": "nosniff"}
|
|
||||||
|
|
||||||
|
|
||||||
class ShareCreateRequest(BaseModel):
|
|
||||||
checkpoint_id: str | None = Field(default=None, max_length=128)
|
|
||||||
expires_in_seconds: int = Field(default=7 * 86400, ge=3600, le=30 * 86400)
|
|
||||||
|
|
||||||
|
|
||||||
class ShareForkRequest(BaseModel):
|
|
||||||
request_id: UUID
|
|
||||||
|
|
||||||
|
|
||||||
def share_repository() -> ThreadShareRepository:
|
|
||||||
factory = get_session_factory()
|
|
||||||
if factory is None:
|
|
||||||
raise HTTPException(503, "Sharing requires SQL persistence")
|
|
||||||
return ThreadShareRepository(factory)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/threads/{thread_id}/shares", status_code=201)
|
|
||||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
|
||||||
@require_permission("runs", "read")
|
|
||||||
async def create_share(thread_id: ThreadId, body: ShareCreateRequest, request: Request):
|
|
||||||
async def create():
|
|
||||||
repo = share_repository()
|
|
||||||
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
|
||||||
if await repo.count_active(thread_id, get_effective_user_id()) >= 10:
|
|
||||||
raise HTTPException(409, "Revoke an existing link before creating another")
|
|
||||||
captured = await capture_share(request, thread_id, body.checkpoint_id)
|
|
||||||
return await repo.create(**captured, expires_at=datetime.now(UTC) + timedelta(seconds=body.expires_in_seconds))
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = await await_drained(create())
|
|
||||||
return JSONResponse(result, status_code=201, headers=PRIVATE_HEADERS)
|
|
||||||
except ConflictError:
|
|
||||||
raise HTTPException(409, "Thread has work in flight; share it after the work finishes") from None
|
|
||||||
except AgentRunError as exc:
|
|
||||||
raise HTTPException(exc.status_code, str(exc)) from None
|
|
||||||
except HTTPException:
|
|
||||||
raise
|
|
||||||
except Exception:
|
|
||||||
logger.exception("Failed to create conversation share")
|
|
||||||
raise HTTPException(500, "Unable to create share") from None
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/threads/{thread_id}/shares")
|
|
||||||
@require_permission("threads", "read", owner_check=True, require_existing=True)
|
|
||||||
async def list_shares(thread_id: ThreadId, request: Request):
|
|
||||||
return JSONResponse(await share_repository().list_owned(thread_id, get_effective_user_id()), headers=PRIVATE_HEADERS)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/threads/{thread_id}/shares/{share_id}", status_code=204)
|
|
||||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
|
||||||
async def revoke_share(thread_id: ThreadId, share_id: str, request: Request):
|
|
||||||
if not await await_drained(share_repository().revoke(thread_id, share_id, get_effective_user_id())):
|
|
||||||
raise HTTPException(404, "Share not found")
|
|
||||||
return Response(status_code=204, headers=PRIVATE_HEADERS)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/shares/{share_token}")
|
|
||||||
async def read_share(share_token: ShareToken):
|
|
||||||
share = await share_repository().get_public(share_token)
|
|
||||||
if share is None:
|
|
||||||
raise HTTPException(404, "Share unavailable", headers=PRIVATE_HEADERS)
|
|
||||||
return JSONResponse({**share["snapshot"], "checkpoint_id": share["checkpoint_id"], "seq": share["seq"], "created_at": share["created_at"], "expires_at": share["expires_at"], "artifacts": share["artifacts"]}, headers=PRIVATE_HEADERS)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/shares/{share_token}/artifacts/{path:path}")
|
|
||||||
@router.head("/shares/{share_token}/artifacts/{path:path}")
|
|
||||||
async def read_share_artifact(share_token: ShareToken, path: str, request: Request, download: bool = False):
|
|
||||||
artifact = await share_repository().get_artifact(share_token, path)
|
|
||||||
if artifact is None:
|
|
||||||
raise HTTPException(404, "Shared artifact unavailable", headers=PRIVATE_HEADERS)
|
|
||||||
content, status, range_headers = _slice_byte_range(artifact["content"], request.headers.get("range"))
|
|
||||||
# Even a direct tab navigation cannot run active content with our origin.
|
|
||||||
# HTML previews render in an opaque sandbox and cannot contact the network.
|
|
||||||
headers = {
|
|
||||||
**PRIVATE_HEADERS,
|
|
||||||
**range_headers,
|
|
||||||
"Content-Disposition": _build_content_disposition("attachment" if download else "inline", path.rsplit("/", 1)[-1]),
|
|
||||||
"Content-Length": str(len(content)),
|
|
||||||
}
|
|
||||||
# Chromium's built-in PDF viewer cannot load in a CSP sandbox. The PDF
|
|
||||||
# media type and nosniff keep it outside the active HTML document surface.
|
|
||||||
if artifact["mime_type"] != "application/pdf":
|
|
||||||
headers["Content-Security-Policy"] = "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:; base-uri 'none'; form-action 'none'"
|
|
||||||
return Response(content if request.method != "HEAD" else b"", status_code=status, media_type=artifact["mime_type"], headers=headers)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/shares/{share_token}/fork")
|
|
||||||
@require_permission("threads", "write")
|
|
||||||
async def import_share(share_token: ShareToken, body: ShareForkRequest, request: Request):
|
|
||||||
try:
|
|
||||||
return JSONResponse(await await_drained(fork_share(request, share_repository(), share_token, str(body.request_id))), headers=PRIVATE_HEADERS)
|
|
||||||
except ConflictError:
|
|
||||||
raise HTTPException(409, "Import is already in progress; retry shortly") from None
|
|
||||||
except AgentRunError as exc:
|
|
||||||
raise HTTPException(exc.status_code, str(exc)) from None
|
|
||||||
except HTTPException:
|
|
||||||
raise
|
|
||||||
except Exception:
|
|
||||||
logger.exception("Failed to import conversation share")
|
|
||||||
raise HTTPException(500, "Unable to import share") from None
|
|
||||||
@ -338,20 +338,6 @@ async def _publish_skills_cache_reset(user_id: str | None) -> str | None:
|
|||||||
return await asyncio.to_thread(publish_skills_cache_reset, user_id=user_id)
|
return await asyncio.to_thread(publish_skills_cache_reset, user_id=user_id)
|
||||||
|
|
||||||
|
|
||||||
def _assert_extensions_config_readable() -> None:
|
|
||||||
"""Raise when the extensions config exists but cannot be read.
|
|
||||||
|
|
||||||
The storage layer fails closed on an unreadable config (every skill stays
|
|
||||||
listed, none is enabled), which keeps the runtime activation gate safe
|
|
||||||
without taking the listing down. The read surface must stay loud instead:
|
|
||||||
the ``enabled`` column it reports is derived from this config, so serving
|
|
||||||
a degraded list would present a broken deployment as a legitimately empty
|
|
||||||
one. The raised error reaches each handler's generic 500 branch, which
|
|
||||||
keeps the host path in the server log and out of the response body.
|
|
||||||
"""
|
|
||||||
ExtensionsConfig.from_file()
|
|
||||||
|
|
||||||
|
|
||||||
def _load_user_skills(config: AppConfig) -> list[Skill]:
|
def _load_user_skills(config: AppConfig) -> list[Skill]:
|
||||||
"""Load every skill visible to the caller's storage, enabled or not.
|
"""Load every skill visible to the caller's storage, enabled or not.
|
||||||
|
|
||||||
@ -361,7 +347,6 @@ def _load_user_skills(config: AppConfig) -> list[Skill]:
|
|||||||
each ``SKILL.md`` -- work that scales with the number of installed skills
|
each ``SKILL.md`` -- work that scales with the number of installed skills
|
||||||
(#5747 moved the same call off the loop for ``get_custom_skill``).
|
(#5747 moved the same call off the loop for ``get_custom_skill``).
|
||||||
"""
|
"""
|
||||||
_assert_extensions_config_readable()
|
|
||||||
return _get_user_skill_storage(config).load_skills(enabled_only=False)
|
return _get_user_skill_storage(config).load_skills(enabled_only=False)
|
||||||
|
|
||||||
|
|
||||||
@ -492,7 +477,7 @@ async def _filter_visible_skills(
|
|||||||
return skills
|
return skills
|
||||||
|
|
||||||
try:
|
try:
|
||||||
provider, principal = await asyncio.to_thread(resolve_skill_authorization, user, is_internal=_is_internal_caller(request, user))
|
provider, principal = resolve_skill_authorization(user, is_internal=_is_internal_caller(request, user))
|
||||||
except _AuthorizationUnavailable as exc:
|
except _AuthorizationUnavailable as exc:
|
||||||
return [] if exc.fail_closed else skills
|
return [] if exc.fail_closed else skills
|
||||||
|
|
||||||
|
|||||||
@ -8,7 +8,7 @@ from collections.abc import AsyncIterator
|
|||||||
from fastapi import APIRouter, HTTPException, Path, Query, Request
|
from fastapi import APIRouter, HTTPException, Path, Query, Request
|
||||||
from fastapi.responses import StreamingResponse
|
from fastapi.responses import StreamingResponse
|
||||||
|
|
||||||
from app.gateway.authz import require_cancel_permission_if, require_permission
|
from app.gateway.authz import require_permission
|
||||||
from app.gateway.deps import (
|
from app.gateway.deps import (
|
||||||
get_current_user,
|
get_current_user,
|
||||||
get_subagent_batch_repo,
|
get_subagent_batch_repo,
|
||||||
@ -110,7 +110,6 @@ async def cancel_batch(thread_id: ThreadId, batch_id: str, request: Request) ->
|
|||||||
if not getattr(request.app.state, "subagent_batches_available", False):
|
if not getattr(request.app.state, "subagent_batches_available", False):
|
||||||
raise HTTPException(status_code=503, detail="Subagent batch worker is not running")
|
raise HTTPException(status_code=503, detail="Subagent batch worker is not running")
|
||||||
_repo, user_id, _batch = await _owned_batch(request, thread_id, batch_id)
|
_repo, user_id, _batch = await _owned_batch(request, thread_id, batch_id)
|
||||||
require_cancel_permission_if(request, _batch.get("continuation_status") is not None)
|
|
||||||
result = await get_subagent_batch_service(request).cancel_batch(batch_id=batch_id, user_id=user_id)
|
result = await get_subagent_batch_service(request).cancel_batch(batch_id=batch_id, user_id=user_id)
|
||||||
if result is None:
|
if result is None:
|
||||||
raise HTTPException(status_code=404, detail="Subagent batch not found")
|
raise HTTPException(status_code=404, detail="Subagent batch not found")
|
||||||
@ -123,7 +122,7 @@ async def retry_batch_item(thread_id: ThreadId, batch_id: str, item_id: str, req
|
|||||||
repo, user_id, _batch = await _owned_batch(request, thread_id, batch_id)
|
repo, user_id, _batch = await _owned_batch(request, thread_id, batch_id)
|
||||||
item = await repo.retry_item(batch_id, item_id, user_id=user_id)
|
item = await repo.retry_item(batch_id, item_id, user_id=user_id)
|
||||||
if item is None:
|
if item is None:
|
||||||
raise HTTPException(status_code=409, detail="Only failed items of a batch that is not cancelled can be retried")
|
raise HTTPException(status_code=409, detail="Only failed items can be retried")
|
||||||
return item
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -35,8 +35,6 @@ class SubagentResponse(BaseModel):
|
|||||||
disallowed_tools: list[str] | None = None
|
disallowed_tools: list[str] | None = None
|
||||||
skills: list[str] | None = None
|
skills: list[str] | None = None
|
||||||
model: str = "inherit"
|
model: str = "inherit"
|
||||||
thinking_enabled: bool | None = None
|
|
||||||
reasoning_effort: str | None = None
|
|
||||||
max_turns: int = 50
|
max_turns: int = 50
|
||||||
timeout_seconds: int = 900
|
timeout_seconds: int = 900
|
||||||
enabled: bool = True
|
enabled: bool = True
|
||||||
@ -59,8 +57,6 @@ class ManagedSubagentCreateRequest(BaseModel):
|
|||||||
disallowed_tools: list[str] | None = None
|
disallowed_tools: list[str] | None = None
|
||||||
skills: list[str] | None = None
|
skills: list[str] | None = None
|
||||||
model: str = "inherit"
|
model: str = "inherit"
|
||||||
thinking_enabled: bool | None = None
|
|
||||||
reasoning_effort: str | None = None
|
|
||||||
max_turns: int = Field(default=50, ge=1)
|
max_turns: int = Field(default=50, ge=1)
|
||||||
timeout_seconds: int = Field(default=900, ge=1)
|
timeout_seconds: int = Field(default=900, ge=1)
|
||||||
enabled: bool = True
|
enabled: bool = True
|
||||||
@ -74,8 +70,6 @@ class ManagedSubagentUpdateRequest(BaseModel):
|
|||||||
disallowed_tools: list[str] | None = None
|
disallowed_tools: list[str] | None = None
|
||||||
skills: list[str] | None = None
|
skills: list[str] | None = None
|
||||||
model: str | None = None
|
model: str | None = None
|
||||||
thinking_enabled: bool | None = None
|
|
||||||
reasoning_effort: str | None = None
|
|
||||||
max_turns: int | None = Field(default=None, ge=1)
|
max_turns: int | None = Field(default=None, ge=1)
|
||||||
timeout_seconds: int | None = Field(default=None, ge=1)
|
timeout_seconds: int | None = Field(default=None, ge=1)
|
||||||
enabled: bool | None = None
|
enabled: bool | None = None
|
||||||
@ -88,9 +82,9 @@ def _validate_model(model: str, app_config) -> None:
|
|||||||
raise HTTPException(status_code=422, detail=f"Unknown model '{model}'. Use 'inherit' or a configured model name.")
|
raise HTTPException(status_code=422, detail=f"Unknown model '{model}'. Use 'inherit' or a configured model name.")
|
||||||
|
|
||||||
|
|
||||||
def _validate_path_name(name: str, *, for_creation: bool = False) -> str:
|
def _validate_path_name(name: str) -> str:
|
||||||
try:
|
try:
|
||||||
return normalize_managed_subagent_name(name, for_creation=for_creation)
|
return normalize_managed_subagent_name(name)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||||
|
|
||||||
@ -100,7 +94,7 @@ def _explicit_overrides(name: str, app_config) -> dict[str, Any]:
|
|||||||
if override is None:
|
if override is None:
|
||||||
return {}
|
return {}
|
||||||
result: dict[str, Any] = {}
|
result: dict[str, Any] = {}
|
||||||
for field in ("timeout_seconds", "max_turns", "model", "thinking_enabled", "reasoning_effort", "skills"):
|
for field in ("timeout_seconds", "max_turns", "model", "skills"):
|
||||||
if field not in override.model_fields_set:
|
if field not in override.model_fields_set:
|
||||||
continue
|
continue
|
||||||
value = getattr(override, field)
|
value = getattr(override, field)
|
||||||
@ -125,8 +119,6 @@ def _catalog(include_system_prompt: bool) -> SubagentsListResponse:
|
|||||||
disallowed_tools=definition.disallowed_tools,
|
disallowed_tools=definition.disallowed_tools,
|
||||||
skills=definition.skills,
|
skills=definition.skills,
|
||||||
model=definition.model,
|
model=definition.model,
|
||||||
thinking_enabled=definition.thinking_enabled,
|
|
||||||
reasoning_effort=definition.reasoning_effort,
|
|
||||||
max_turns=definition.max_turns,
|
max_turns=definition.max_turns,
|
||||||
timeout_seconds=definition.timeout_seconds,
|
timeout_seconds=definition.timeout_seconds,
|
||||||
source="builtin",
|
source="builtin",
|
||||||
@ -144,8 +136,6 @@ def _catalog(include_system_prompt: bool) -> SubagentsListResponse:
|
|||||||
disallowed_tools=definition.disallowed_tools,
|
disallowed_tools=definition.disallowed_tools,
|
||||||
skills=definition.skills,
|
skills=definition.skills,
|
||||||
model=definition.model,
|
model=definition.model,
|
||||||
thinking_enabled=definition.thinking_enabled,
|
|
||||||
reasoning_effort=definition.reasoning_effort,
|
|
||||||
max_turns=definition.max_turns,
|
max_turns=definition.max_turns,
|
||||||
timeout_seconds=definition.timeout_seconds,
|
timeout_seconds=definition.timeout_seconds,
|
||||||
source="config",
|
source="config",
|
||||||
@ -181,7 +171,6 @@ async def list_subagents(request: Request) -> SubagentsListResponse:
|
|||||||
@router.post("", response_model=SubagentResponse, status_code=201, summary="Create Managed Subagent")
|
@router.post("", response_model=SubagentResponse, status_code=201, summary="Create Managed Subagent")
|
||||||
async def create_managed_subagent(request: Request, body: ManagedSubagentCreateRequest) -> SubagentResponse:
|
async def create_managed_subagent(request: Request, body: ManagedSubagentCreateRequest) -> SubagentResponse:
|
||||||
await require_admin_user(request, detail=_ADMIN_REQUIRED_DETAIL)
|
await require_admin_user(request, detail=_ADMIN_REQUIRED_DETAIL)
|
||||||
_validate_path_name(body.name, for_creation=True)
|
|
||||||
try:
|
try:
|
||||||
definition = ManagedSubagentDefinition(**body.model_dump(exclude_none=True))
|
definition = ManagedSubagentDefinition(**body.model_dump(exclude_none=True))
|
||||||
except ValidationError as exc:
|
except ValidationError as exc:
|
||||||
|
|||||||
@ -1,4 +1,3 @@
|
|||||||
import asyncio
|
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
@ -39,11 +38,11 @@ class SuggestionsConfigResponse(BaseModel):
|
|||||||
|
|
||||||
|
|
||||||
_strip_markdown_code_fence = llm_text.strip_markdown_code_fence
|
_strip_markdown_code_fence = llm_text.strip_markdown_code_fence
|
||||||
_strip_think_blocks_outside_json_strings = llm_text.strip_think_blocks_outside_json_strings
|
_strip_think_blocks = llm_text.strip_think_blocks
|
||||||
|
|
||||||
|
|
||||||
def _parse_json_string_list(text: str) -> list[str] | None:
|
def _parse_json_string_list(text: str) -> list[str] | None:
|
||||||
candidate = _strip_think_blocks_outside_json_strings(text)
|
candidate = _strip_think_blocks(text)
|
||||||
candidate = _strip_markdown_code_fence(candidate)
|
candidate = _strip_markdown_code_fence(candidate)
|
||||||
start = candidate.find("[")
|
start = candidate.find("[")
|
||||||
end = candidate.rfind("]")
|
end = candidate.rfind("]")
|
||||||
@ -123,7 +122,7 @@ async def generate_suggestions(
|
|||||||
# default when the caller omits model_name. Keep permission failures out
|
# default when the caller omits model_name. Keep permission failures out
|
||||||
# of the best-effort LLM error handler below so a denial remains a 403.
|
# of the best-effort LLM error handler below so a denial remains a 403.
|
||||||
user = await get_current_user_from_request(request)
|
user = await get_current_user_from_request(request)
|
||||||
await asyncio.to_thread(authorize_model_use, user, body.model_name, is_internal=_is_internal_caller(request, user), app_config=config)
|
authorize_model_use(user, body.model_name, is_internal=_is_internal_caller(request, user), app_config=config)
|
||||||
|
|
||||||
system_instruction = (
|
system_instruction = (
|
||||||
"You are generating follow-up questions to help the user continue the conversation.\n"
|
"You are generating follow-up questions to help the user continue the conversation.\n"
|
||||||
|
|||||||
@ -16,7 +16,6 @@ import hashlib
|
|||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
import uuid
|
import uuid
|
||||||
from collections.abc import Mapping
|
|
||||||
from copy import deepcopy
|
from copy import deepcopy
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
from typing import Annotated, Any, Literal
|
from typing import Annotated, Any, Literal
|
||||||
@ -55,7 +54,6 @@ from app.gateway.conversation_reader import (
|
|||||||
from app.gateway.deps import get_current_user, get_feedback_repo, get_run_event_store, get_run_manager, get_run_store, get_stream_bridge
|
from app.gateway.deps import get_current_user, get_feedback_repo, get_run_event_store, get_run_manager, get_run_store, get_stream_bridge
|
||||||
from app.gateway.internal_auth import INTERNAL_SYSTEM_ROLE, get_trusted_internal_owner_user_id
|
from app.gateway.internal_auth import INTERNAL_SYSTEM_ROLE, get_trusted_internal_owner_user_id
|
||||||
from app.gateway.pagination import trim_run_message_page
|
from app.gateway.pagination import trim_run_message_page
|
||||||
from app.gateway.run_inputs import RunInputRequest, change_run_input, list_thread_inputs, submit_run_input
|
|
||||||
from app.gateway.run_models import RunCreateRequest
|
from app.gateway.run_models import RunCreateRequest
|
||||||
from app.gateway.services import (
|
from app.gateway.services import (
|
||||||
abuild_checkpoint_state_accessor,
|
abuild_checkpoint_state_accessor,
|
||||||
@ -66,19 +64,16 @@ from app.gateway.services import (
|
|||||||
wait_for_run_completion,
|
wait_for_run_completion,
|
||||||
)
|
)
|
||||||
from app.gateway.sse_headers import sse_response_headers
|
from app.gateway.sse_headers import sse_response_headers
|
||||||
from app.gateway.tool_approvals import ApprovalResponseBody, answer_approval, list_approvals, revoke_approval_grants
|
|
||||||
from app.gateway.utils import sanitize_log_param
|
from app.gateway.utils import sanitize_log_param
|
||||||
from deerflow.agents.human_input import read_human_input_response
|
from deerflow.agents.human_input import read_human_input_response
|
||||||
from deerflow.agents.middlewares.dynamic_context_middleware import strip_injected_user_message_id_suffix
|
from deerflow.agents.middlewares.dynamic_context_middleware import strip_injected_user_message_id_suffix
|
||||||
from deerflow.authz.sandbox_authz import safe_app_config_async
|
from deerflow.authz.sandbox_authz import safe_app_config_async
|
||||||
from deerflow.config.paths import get_paths, make_safe_user_id
|
from deerflow.config.paths import get_paths, make_safe_user_id
|
||||||
from deerflow.constants import DELIVERY_PRESENTATION_TOOL_NAMES
|
|
||||||
from deerflow.runtime import CancelOutcome, ConflictError, RunRecord, RunStatus, ThreadOperationKind, serialize_channel_values_for_api
|
from deerflow.runtime import CancelOutcome, ConflictError, RunRecord, RunStatus, ThreadOperationKind, serialize_channel_values_for_api
|
||||||
from deerflow.runtime.goal import is_active_goal
|
from deerflow.runtime.goal import is_active_goal
|
||||||
from deerflow.runtime.runs.store.base import format_run_cursor_created_at, normalize_run_created_at_iso
|
from deerflow.runtime.runs.store.base import format_run_cursor_created_at, normalize_run_created_at_iso
|
||||||
from deerflow.runtime.secret_context import redact_config_secrets, redact_metadata_secrets
|
from deerflow.runtime.secret_context import redact_config_secrets, redact_metadata_secrets
|
||||||
from deerflow.runtime.user_context import get_effective_user_id
|
from deerflow.runtime.user_context import get_effective_user_id
|
||||||
from deerflow.sandbox.exceptions import SandboxAuthorizationError
|
|
||||||
from deerflow.utils.llm_text import strip_leading_think_blocks
|
from deerflow.utils.llm_text import strip_leading_think_blocks
|
||||||
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY, get_original_user_content_text, message_to_text
|
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY, get_original_user_content_text, message_to_text
|
||||||
from deerflow.utils.thread_id import ThreadId
|
from deerflow.utils.thread_id import ThreadId
|
||||||
@ -86,50 +81,6 @@ from deerflow.workspace_changes import get_workspace_changes_response
|
|||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
router = APIRouter(prefix="/api/threads", tags=["runs"])
|
router = APIRouter(prefix="/api/threads", tags=["runs"])
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{thread_id}/tool-approvals")
|
|
||||||
@require_permission("threads", "read", owner_check=True, require_existing=True)
|
|
||||||
async def get_tool_approvals(thread_id: ThreadId, request: Request):
|
|
||||||
return await list_approvals(thread_id, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/{thread_id}/runs/{run_id}/tool-approvals/{request_id}")
|
|
||||||
@require_permission("runs", "create", owner_check=True, require_existing=True)
|
|
||||||
async def submit_tool_approval(thread_id: ThreadId, run_id: str, request_id: str, body: ApprovalResponseBody, request: Request):
|
|
||||||
return await answer_approval(thread_id, run_id, request_id, body, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/{thread_id}/tool-approvals/grants")
|
|
||||||
@require_permission("runs", "create", owner_check=True, require_existing=True)
|
|
||||||
async def revoke_tool_approval_grants(thread_id: ThreadId, request: Request):
|
|
||||||
return await revoke_approval_grants(thread_id, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/{thread_id}/runs/{run_id}/inputs")
|
|
||||||
@require_permission("runs", "create", owner_check=True, require_existing=True)
|
|
||||||
async def create_run_input(thread_id: ThreadId, run_id: str, body: RunInputRequest, request: Request):
|
|
||||||
return await submit_run_input(thread_id, run_id, body, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{thread_id}/inputs")
|
|
||||||
@require_permission("threads", "read", owner_check=True, require_existing=True)
|
|
||||||
async def get_thread_inputs(thread_id: ThreadId, request: Request):
|
|
||||||
return await list_thread_inputs(thread_id, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/{thread_id}/runs/{run_id}/inputs/{input_id}")
|
|
||||||
@require_permission("runs", "cancel", owner_check=True, require_existing=True)
|
|
||||||
async def remove_run_input(thread_id: ThreadId, run_id: str, input_id: str, request: Request):
|
|
||||||
return await change_run_input(thread_id, run_id, input_id, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/{thread_id}/runs/{run_id}/inputs/{input_id}/retry")
|
|
||||||
@require_permission("runs", "create", owner_check=True, require_existing=True)
|
|
||||||
async def retry_run_input(thread_id: ThreadId, run_id: str, input_id: str, request: Request):
|
|
||||||
return await change_run_input(thread_id, run_id, input_id, request, retry=True)
|
|
||||||
|
|
||||||
|
|
||||||
REGENERATE_HISTORY_SCAN_LIMIT = 200
|
REGENERATE_HISTORY_SCAN_LIMIT = 200
|
||||||
# Doubled to keep ~200 effective checkpoints when duration-only checkpoints
|
# Doubled to keep ~200 effective checkpoints when duration-only checkpoints
|
||||||
# (one per successful run in steady state) consume roughly half of history.
|
# (one per successful run in steady state) consume roughly half of history.
|
||||||
@ -230,7 +181,7 @@ def compute_run_durations(runs) -> dict[str, int]:
|
|||||||
return durations
|
return durations
|
||||||
|
|
||||||
|
|
||||||
def stamp_turn_duration_on_last_ai(messages, run_durations: dict[str, int], *, thread_id: str | None = None) -> None:
|
def stamp_turn_duration_on_last_ai(messages, run_durations: dict[str, int]) -> None:
|
||||||
"""Attach each run's elapsed seconds to that run's final visible AI message only.
|
"""Attach each run's elapsed seconds to that run's final visible AI message only.
|
||||||
|
|
||||||
``turn_duration`` is the run's wall-clock lifetime (``compute_run_durations``),
|
``turn_duration`` is the run's wall-clock lifetime (``compute_run_durations``),
|
||||||
@ -258,37 +209,10 @@ def stamp_turn_duration_on_last_ai(messages, run_durations: dict[str, int], *, t
|
|||||||
continue
|
continue
|
||||||
content = msg.get("content")
|
content = msg.get("content")
|
||||||
payload = content if isinstance(content, dict) else msg
|
payload = content if isinstance(content, dict) else msg
|
||||||
metadata = msg.get("metadata")
|
metadata = msg.get("metadata") or {}
|
||||||
if metadata is None:
|
|
||||||
metadata = {}
|
|
||||||
elif not isinstance(metadata, Mapping):
|
|
||||||
logger.warning(
|
|
||||||
"Ignoring malformed event metadata for thread %s at row seq %s: expected a mapping, got %s",
|
|
||||||
sanitize_log_param(thread_id or "<unknown>"),
|
|
||||||
msg.get("seq", "<unknown>"),
|
|
||||||
type(metadata).__name__,
|
|
||||||
)
|
|
||||||
metadata = {}
|
|
||||||
is_middleware = str(metadata.get("caller", "")).startswith("middleware:")
|
is_middleware = str(metadata.get("caller", "")).startswith("middleware:")
|
||||||
if payload.get("type") == "ai" and not is_middleware:
|
if payload.get("type") == "ai" and not is_middleware:
|
||||||
raw_additional_kwargs = payload.get("additional_kwargs")
|
payload.setdefault("additional_kwargs", {})["turn_duration"] = run_durations[rid]
|
||||||
if raw_additional_kwargs is None:
|
|
||||||
additional_kwargs: dict[str, Any] = {}
|
|
||||||
elif isinstance(raw_additional_kwargs, Mapping):
|
|
||||||
# A Mapping is not necessarily mutable (for example, a
|
|
||||||
# MappingProxyType from a serializer), so copy it before
|
|
||||||
# attaching the duration.
|
|
||||||
additional_kwargs = dict(raw_additional_kwargs)
|
|
||||||
else:
|
|
||||||
logger.warning(
|
|
||||||
"Ignoring malformed message additional_kwargs for thread %s at row seq %s: expected a mapping, got %s",
|
|
||||||
sanitize_log_param(thread_id or "<unknown>"),
|
|
||||||
msg.get("seq", "<unknown>"),
|
|
||||||
type(raw_additional_kwargs).__name__,
|
|
||||||
)
|
|
||||||
additional_kwargs = {}
|
|
||||||
payload["additional_kwargs"] = additional_kwargs
|
|
||||||
additional_kwargs["turn_duration"] = run_durations[rid]
|
|
||||||
stamped.add(rid)
|
stamped.add(rid)
|
||||||
|
|
||||||
|
|
||||||
@ -829,19 +753,9 @@ async def _require_successful_source_run(thread_id: str, run_id: str, request: R
|
|||||||
raise HTTPException(status_code=409, detail="Could not find source run for assistant message")
|
raise HTTPException(status_code=409, detail="Could not find source run for assistant message")
|
||||||
if _run_status_value(record) != RunStatus.success.value:
|
if _run_status_value(record) != RunStatus.success.value:
|
||||||
raise HTTPException(status_code=409, detail="Only successful assistant runs can be edited and rerun")
|
raise HTTPException(status_code=409, detail="Only successful assistant runs can be edited and rerun")
|
||||||
await _reject_multi_input_replay(run_id, request, user_id=user_id)
|
|
||||||
return record
|
return record
|
||||||
|
|
||||||
|
|
||||||
async def _reject_multi_input_replay(run_id: str, request: Request, *, user_id=None) -> None:
|
|
||||||
store = getattr(get_run_manager(request), "input_store", None)
|
|
||||||
if getattr(store, "supports_run_inputs", False) is not True:
|
|
||||||
return
|
|
||||||
row = await store.get(run_id, user_id=user_id)
|
|
||||||
if row is not None and any(entry.get("mode") == "steer" and entry.get("status") != "cancelled" for entry in (row.get("run_inputs") or {}).get("items", [])):
|
|
||||||
raise HTTPException(409, "This Run was adjusted while working; send a new message instead of replaying part of it")
|
|
||||||
|
|
||||||
|
|
||||||
async def _find_interrupted_target_run_id(
|
async def _find_interrupted_target_run_id(
|
||||||
thread_id: str,
|
thread_id: str,
|
||||||
source_human: Any,
|
source_human: Any,
|
||||||
@ -918,7 +832,6 @@ async def _prepare_regenerate_payload(thread_id: str, message_id: str, request:
|
|||||||
raise HTTPException(status_code=409, detail="Could not find the user message for this assistant response")
|
raise HTTPException(status_code=409, detail="Could not find the user message for this assistant response")
|
||||||
if target_run_id is None:
|
if target_run_id is None:
|
||||||
raise HTTPException(status_code=409, detail="Could not find source run for assistant message")
|
raise HTTPException(status_code=409, detail="Could not find source run for assistant message")
|
||||||
await _reject_multi_input_replay(target_run_id, request, user_id=await _run_scope_user_id(request, thread_id))
|
|
||||||
previous_human_id = _message_id(previous_human)
|
previous_human_id = _message_id(previous_human)
|
||||||
if not previous_human_id:
|
if not previous_human_id:
|
||||||
raise HTTPException(status_code=409, detail="The source user message is missing an id")
|
raise HTTPException(status_code=409, detail="The source user message is missing an id")
|
||||||
@ -1631,7 +1544,7 @@ async def list_thread_messages(
|
|||||||
run_durations = compute_run_durations(runs)
|
run_durations = compute_run_durations(runs)
|
||||||
|
|
||||||
if run_durations:
|
if run_durations:
|
||||||
stamp_turn_duration_on_last_ai(messages, run_durations, thread_id=thread_id)
|
stamp_turn_duration_on_last_ai(messages, run_durations)
|
||||||
|
|
||||||
return messages
|
return messages
|
||||||
|
|
||||||
@ -1700,7 +1613,7 @@ async def _enrich_thread_message_page(
|
|||||||
# endpoints already use ``stamp_turn_duration_on_last_ai``; the page
|
# endpoints already use ``stamp_turn_duration_on_last_ai``; the page
|
||||||
# endpoint was inlining the equivalent loop but stamping every AI row,
|
# endpoint was inlining the equivalent loop but stamping every AI row,
|
||||||
# which #4163 fixed for the other paths and missed here.
|
# which #4163 fixed for the other paths and missed here.
|
||||||
stamp_turn_duration_on_last_ai(data, run_durations, thread_id=thread_id)
|
stamp_turn_duration_on_last_ai(data, run_durations)
|
||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
@ -1765,7 +1678,7 @@ async def list_run_messages(
|
|||||||
if record:
|
if record:
|
||||||
durations = compute_run_durations([record])
|
durations = compute_run_durations([record])
|
||||||
if durations:
|
if durations:
|
||||||
stamp_turn_duration_on_last_ai(data, durations, thread_id=thread_id)
|
stamp_turn_duration_on_last_ai(data, durations)
|
||||||
|
|
||||||
return {"data": data, "has_more": has_more}
|
return {"data": data, "has_more": has_more}
|
||||||
|
|
||||||
@ -1822,15 +1735,8 @@ def _presented_files_from_delivery(events: list[dict]) -> list[str]:
|
|||||||
raise HTTPException(status_code=409, detail="This response has no verified artifact delivery")
|
raise HTTPException(status_code=409, detail="This response has no verified artifact delivery")
|
||||||
content = events[0].get("content")
|
content = events[0].get("content")
|
||||||
by_tool = content.get("by_tool") if isinstance(content, dict) else None
|
by_tool = content.get("by_tool") if isinstance(content, dict) else None
|
||||||
if not isinstance(by_tool, dict):
|
presented = by_tool.get("present_files") if isinstance(by_tool, dict) else None
|
||||||
raise HTTPException(status_code=409, detail="This response has no verified artifact delivery")
|
if not isinstance(presented, list) or not presented or any(not isinstance(path, str) for path in presented):
|
||||||
presented: list[str] = []
|
|
||||||
for tool_name in DELIVERY_PRESENTATION_TOOL_NAMES:
|
|
||||||
paths = by_tool.get(tool_name, [])
|
|
||||||
if not isinstance(paths, list):
|
|
||||||
raise HTTPException(status_code=409, detail="This response has no verified artifact delivery")
|
|
||||||
presented.extend(paths)
|
|
||||||
if not presented or any(not isinstance(path, str) for path in presented):
|
|
||||||
raise HTTPException(status_code=409, detail="This response has no verified artifact delivery")
|
raise HTTPException(status_code=409, detail="This response has no verified artifact delivery")
|
||||||
return presented
|
return presented
|
||||||
|
|
||||||
@ -1882,10 +1788,7 @@ async def create_run_artifact_archive(
|
|||||||
|
|
||||||
raw_owner_user_id = get_trusted_internal_owner_user_id(request)
|
raw_owner_user_id = get_trusted_internal_owner_user_id(request)
|
||||||
effective_user_id = make_safe_user_id(raw_owner_user_id) if raw_owner_user_id else get_effective_user_id()
|
effective_user_id = make_safe_user_id(raw_owner_user_id) if raw_owner_user_id else get_effective_user_id()
|
||||||
try:
|
|
||||||
app_config = await safe_app_config_async()
|
app_config = await safe_app_config_async()
|
||||||
except SandboxAuthorizationError as exc:
|
|
||||||
raise HTTPException(status_code=503, detail="Artifact archive is temporarily unavailable") from exc
|
|
||||||
custom_tool_output_dir = getattr(getattr(app_config, "tool_output", None), "storage_subdir", None)
|
custom_tool_output_dir = getattr(getattr(app_config, "tool_output", None), "storage_subdir", None)
|
||||||
extra_reserved_dir_names = {custom_tool_output_dir} if isinstance(custom_tool_output_dir, str) else set()
|
extra_reserved_dir_names = {custom_tool_output_dir} if isinstance(custom_tool_output_dir, str) else set()
|
||||||
paths = get_paths()
|
paths = get_paths()
|
||||||
@ -1951,18 +1854,12 @@ async def list_run_events(
|
|||||||
user_id = await _run_scope_user_id(request, thread_id)
|
user_id = await _run_scope_user_id(request, thread_id)
|
||||||
event_store = get_run_event_store(request)
|
event_store = get_run_event_store(request)
|
||||||
if event_types:
|
if event_types:
|
||||||
# Split on commas and normalise each entry: the value goes straight into
|
types = event_types.split(",")
|
||||||
# a SQL ``IN``, so a padded entry ("message, tool") would silently match
|
|
||||||
# nothing and return a 200 with a truncated audit trail. Blank entries
|
|
||||||
# carry no intent and are dropped; a filter that reduces to nothing is
|
|
||||||
# the same as no filter, matching the empty-string case above.
|
|
||||||
types = [name for entry in event_types.split(",") if (name := entry.strip())]
|
|
||||||
if len(types) > MAX_EVENT_TYPE_FILTERS:
|
if len(types) > MAX_EVENT_TYPE_FILTERS:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=422,
|
status_code=422,
|
||||||
detail=f"event_types accepts at most {MAX_EVENT_TYPE_FILTERS} values",
|
detail=f"event_types accepts at most {MAX_EVENT_TYPE_FILTERS} values",
|
||||||
)
|
)
|
||||||
types = types or None
|
|
||||||
else:
|
else:
|
||||||
types = None
|
types = None
|
||||||
events = await event_store.list_events(
|
events = await event_store.list_events(
|
||||||
|
|||||||
@ -12,7 +12,6 @@ matching the LangGraph Platform wire format expected by the
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import inspect
|
import inspect
|
||||||
import logging
|
import logging
|
||||||
import shutil
|
import shutil
|
||||||
@ -20,13 +19,13 @@ import uuid
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
|
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
|
||||||
from langgraph.checkpoint.base import empty_checkpoint
|
from langgraph.checkpoint.base import empty_checkpoint
|
||||||
from langgraph.types import Overwrite
|
from langgraph.types import Overwrite
|
||||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||||
from sqlalchemy.exc import IntegrityError
|
from sqlalchemy.exc import IntegrityError
|
||||||
|
|
||||||
from app.gateway.authz import _is_internal_caller, authorize_model_use, require_permission
|
from app.gateway.authz import require_permission
|
||||||
from app.gateway.checkpoint_lineage import (
|
from app.gateway.checkpoint_lineage import (
|
||||||
CheckpointLineageError,
|
CheckpointLineageError,
|
||||||
CheckpointParentMissingError,
|
CheckpointParentMissingError,
|
||||||
@ -38,9 +37,7 @@ from app.gateway.checkpoint_lineage import (
|
|||||||
resolve_history_versions,
|
resolve_history_versions,
|
||||||
resolve_stamp_candidate_versions,
|
resolve_stamp_candidate_versions,
|
||||||
)
|
)
|
||||||
from app.gateway.deps import get_checkpointer, get_config, get_current_user_from_request, get_run_event_store, get_run_manager, get_run_store
|
from app.gateway.deps import get_checkpointer, get_run_event_store, get_run_manager, get_run_store
|
||||||
from app.gateway.history_seed import reserve_history_seed, write_history_seed
|
|
||||||
from app.gateway.hitl_approval import find_pending_interrupts
|
|
||||||
from app.gateway.internal_auth import get_trusted_internal_owner_user_id
|
from app.gateway.internal_auth import get_trusted_internal_owner_user_id
|
||||||
from app.gateway.services import (
|
from app.gateway.services import (
|
||||||
abuild_checkpoint_state_accessor,
|
abuild_checkpoint_state_accessor,
|
||||||
@ -52,9 +49,7 @@ from app.gateway.services import (
|
|||||||
strip_server_owned_state_metadata,
|
strip_server_owned_state_metadata,
|
||||||
)
|
)
|
||||||
from app.gateway.utils import sanitize_log_param
|
from app.gateway.utils import sanitize_log_param
|
||||||
from deerflow.agents.human_input import extract_unanswered_human_input_request
|
|
||||||
from deerflow.agents.thread_state import THREAD_STATE_REDUCER_FIELDS
|
from deerflow.agents.thread_state import THREAD_STATE_REDUCER_FIELDS
|
||||||
from deerflow.config.app_config import AppConfig
|
|
||||||
from deerflow.config.paths import Paths, get_paths
|
from deerflow.config.paths import Paths, get_paths
|
||||||
from deerflow.config.summarization_config import ContextSize
|
from deerflow.config.summarization_config import ContextSize
|
||||||
from deerflow.persistence.thread_meta import PROJECT_FILTER_UNSET, THREAD_ARCHIVED_METADATA_KEY, THREAD_PINNED_METADATA_KEY, THREAD_PROJECT_METADATA_KEY, ThreadOwnershipConflictError
|
from deerflow.persistence.thread_meta import PROJECT_FILTER_UNSET, THREAD_ARCHIVED_METADATA_KEY, THREAD_PINNED_METADATA_KEY, THREAD_PROJECT_METADATA_KEY, ThreadOwnershipConflictError
|
||||||
@ -68,12 +63,11 @@ from deerflow.runtime.context_compaction import (
|
|||||||
ThreadCompactionResult,
|
ThreadCompactionResult,
|
||||||
compact_thread_context,
|
compact_thread_context,
|
||||||
)
|
)
|
||||||
from deerflow.runtime.context_keys import CHECKPOINT_AGENT_NAME_METADATA_KEY, checkpoint_agent_binding_metadata
|
from deerflow.runtime.context_keys import checkpoint_agent_binding_metadata
|
||||||
from deerflow.runtime.events.message_seq import stamp_messages_with_seq
|
from deerflow.runtime.events.message_seq import stamp_messages_with_seq
|
||||||
from deerflow.runtime.goal import (
|
from deerflow.runtime.goal import (
|
||||||
DEFAULT_MAX_GOAL_CONTINUATIONS,
|
DEFAULT_MAX_GOAL_CONTINUATIONS,
|
||||||
GOAL_OUTCOME_CHANNEL,
|
GOAL_OUTCOME_CHANNEL,
|
||||||
GoalWriteConflict,
|
|
||||||
build_goal_state,
|
build_goal_state,
|
||||||
ensure_thread_checkpoint,
|
ensure_thread_checkpoint,
|
||||||
goal_thread_lock,
|
goal_thread_lock,
|
||||||
@ -81,18 +75,11 @@ from deerflow.runtime.goal import (
|
|||||||
read_thread_goal,
|
read_thread_goal,
|
||||||
write_thread_goal,
|
write_thread_goal,
|
||||||
)
|
)
|
||||||
from deerflow.runtime.goal_synthesis import (
|
|
||||||
GoalSynthesisError,
|
|
||||||
GoalSynthesisUnclear,
|
|
||||||
build_synthesis_context,
|
|
||||||
synthesize_goal_objective,
|
|
||||||
)
|
|
||||||
from deerflow.runtime.journal import build_branch_history_seed_events
|
from deerflow.runtime.journal import build_branch_history_seed_events
|
||||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, admit_origin
|
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, admit_origin
|
||||||
from deerflow.runtime.runs.manager import ConflictError
|
from deerflow.runtime.runs.manager import ConflictError
|
||||||
from deerflow.runtime.runs.worker import RUN_MESSAGE_IDS_METADATA_KEY, valid_duration_entry, valid_run_message_id_entry
|
from deerflow.runtime.runs.worker import RUN_MESSAGE_IDS_METADATA_KEY, valid_duration_entry, valid_run_message_id_entry
|
||||||
from deerflow.runtime.secret_context import redact_metadata_secrets
|
from deerflow.runtime.secret_context import redact_metadata_secrets
|
||||||
from deerflow.runtime.serialization import serialize_lc_object
|
|
||||||
from deerflow.runtime.user_context import get_effective_user_id
|
from deerflow.runtime.user_context import get_effective_user_id
|
||||||
from deerflow.uploads.companions import register_companion, resolve_companion
|
from deerflow.uploads.companions import register_companion, resolve_companion
|
||||||
from deerflow.utils.file_io import await_drained, run_file_io
|
from deerflow.utils.file_io import await_drained, run_file_io
|
||||||
@ -139,13 +126,6 @@ def _checkpoint_mode_http_error(exc: Exception, thread_id: str) -> HTTPException
|
|||||||
# ``deerflow_origin`` marks a thread the server created for a schedule or an
|
# ``deerflow_origin`` marks a thread the server created for a schedule or an
|
||||||
# extension; only the server sets it (``create_thread`` below, run admission).
|
# extension; only the server sets it (``create_thread`` below, run admission).
|
||||||
_SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id", THREAD_PROJECT_METADATA_KEY, DEERFLOW_ORIGIN_KEY})
|
_SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id", THREAD_PROJECT_METADATA_KEY, DEERFLOW_ORIGIN_KEY})
|
||||||
# Checkpoint metadata the server writes and later reads back as trusted:
|
|
||||||
# LangGraph's lineage keys and delta counters (``step`` drives the next turn,
|
|
||||||
# ``parents`` the history walk, and ``counters_since_delta_snapshot`` is read
|
|
||||||
# by every run and state update) and the agent binding that gates memory
|
|
||||||
# writes. ``create_thread`` copies client metadata into the root checkpoint
|
|
||||||
# without them; the thread row keeps the client's metadata as sent.
|
|
||||||
_CHECKPOINT_OWNED_METADATA_KEYS: frozenset[str] = frozenset({"step", "source", "writes", "parents", "created_at", "counters_since_delta_snapshot", CHECKPOINT_AGENT_NAME_METADATA_KEY})
|
|
||||||
_SIDECAR_METADATA_KEY = "deerflow_sidecar"
|
_SIDECAR_METADATA_KEY = "deerflow_sidecar"
|
||||||
_BRANCH_METADATA_KEY = "deerflow_branch"
|
_BRANCH_METADATA_KEY = "deerflow_branch"
|
||||||
_BRANCH_TITLE_SEQUENCE_METADATA_KEY = "branch_title_sequence"
|
_BRANCH_TITLE_SEQUENCE_METADATA_KEY = "branch_title_sequence"
|
||||||
@ -530,8 +510,6 @@ class ThreadCreateRequest(BaseModel):
|
|||||||
class ThreadSearchRequest(BaseModel):
|
class ThreadSearchRequest(BaseModel):
|
||||||
"""Request body for searching threads."""
|
"""Request body for searching threads."""
|
||||||
|
|
||||||
query: str | None = Field(default=None, max_length=256, description="Literal case-insensitive title substring")
|
|
||||||
|
|
||||||
archived: bool | None = Field(default=None, strict=True, description="Archive filter; omitted includes all, false includes legacy unarchived threads")
|
archived: bool | None = Field(default=None, strict=True, description="Archive filter; omitted includes all, false includes legacy unarchived threads")
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Metadata filter (exact match)")
|
metadata: dict[str, Any] = Field(default_factory=dict, description="Metadata filter (exact match)")
|
||||||
project_id: str | None = Field(default=None, description="Filter by project; explicit null = unassigned threads; omit key for all")
|
project_id: str | None = Field(default=None, description="Filter by project; explicit null = unassigned threads; omit key for all")
|
||||||
@ -591,7 +569,7 @@ class ThreadPatchRequest(BaseModel):
|
|||||||
|
|
||||||
|
|
||||||
class ThreadStateUpdateRequest(BaseModel):
|
class ThreadStateUpdateRequest(BaseModel):
|
||||||
"""Request body for updating thread state (checkpoint mutation, not a resume API)."""
|
"""Request body for updating thread state (human-in-the-loop resume)."""
|
||||||
|
|
||||||
values: dict[str, Any] | None = Field(default=None, description="Channel values to merge")
|
values: dict[str, Any] | None = Field(default=None, description="Channel values to merge")
|
||||||
checkpoint_id: str | None = Field(default=None, description="Checkpoint to branch from")
|
checkpoint_id: str | None = Field(default=None, description="Checkpoint to branch from")
|
||||||
@ -617,12 +595,6 @@ class ThreadGoalResponse(BaseModel):
|
|||||||
goal: dict[str, Any] | None = Field(default=None, description="Current goal state, or null when no goal is active")
|
goal: dict[str, Any] | None = Field(default=None, description="Current goal state, or null when no goal is active")
|
||||||
|
|
||||||
|
|
||||||
class ThreadGoalFromChatRequest(BaseModel):
|
|
||||||
"""Request body for synthesizing a goal from the recent conversation."""
|
|
||||||
|
|
||||||
guidance: str | None = Field(default=None, max_length=500, description="Optional user hint steering the synthesized objective")
|
|
||||||
|
|
||||||
|
|
||||||
class ThreadCompactRequest(BaseModel):
|
class ThreadCompactRequest(BaseModel):
|
||||||
"""Request body for manually compacting a thread's active context."""
|
"""Request body for manually compacting a thread's active context."""
|
||||||
|
|
||||||
@ -650,13 +622,10 @@ class HistoryEntry(_MetadataRedactingResponse):
|
|||||||
|
|
||||||
checkpoint_id: str
|
checkpoint_id: str
|
||||||
parent_checkpoint_id: str | None = None
|
parent_checkpoint_id: str | None = None
|
||||||
created_at: str | None = Field(default=None, description="Checkpoint timestamp")
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
values: dict[str, Any] = Field(default_factory=dict)
|
values: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
created_at: str | None = None
|
||||||
next: list[str] = Field(default_factory=list)
|
next: list[str] = Field(default_factory=list)
|
||||||
# Only the head entry carries pending interrupts; older entries are history,
|
|
||||||
# never a pending-approval signal. Shape: {id, name, interrupts} only.
|
|
||||||
tasks: list[dict[str, Any]] = Field(default_factory=list)
|
|
||||||
|
|
||||||
|
|
||||||
class ThreadHistoryRequest(BaseModel):
|
class ThreadHistoryRequest(BaseModel):
|
||||||
@ -835,13 +804,6 @@ async def _delete_thread_data_with_reservation(thread_id: str, request: Request)
|
|||||||
# persisted runs/events/feedback and the thread_meta row all belong to the
|
# persisted runs/events/feedback and the thread_meta row all belong to the
|
||||||
# same owner, so they must not resolve their scope independently.
|
# same owner, so they must not resolve their scope independently.
|
||||||
user_id = get_effective_user_id()
|
user_id = get_effective_user_id()
|
||||||
run_manager = get_run_manager(request)
|
|
||||||
batch_repo = getattr(request.app.state, "subagent_batch_repo", None)
|
|
||||||
approval_store = getattr(request.app.state, "approval_store", None)
|
|
||||||
if approval_store is not None:
|
|
||||||
await await_drained(approval_store.delete_thread(thread_id, user_id=user_id))
|
|
||||||
if batch_repo is not None:
|
|
||||||
await batch_repo.suppress_thread_continuations(thread_id)
|
|
||||||
|
|
||||||
# Legacy IDs may predate the canonical filesystem-safe contract. They can
|
# Legacy IDs may predate the canonical filesystem-safe contract. They can
|
||||||
# still be removed from metadata/checkpoint stores, but must never be
|
# still be removed from metadata/checkpoint stores, but must never be
|
||||||
@ -860,15 +822,14 @@ async def _delete_thread_data_with_reservation(thread_id: str, request: Request)
|
|||||||
# being deleted under a run that has just been admitted.
|
# being deleted under a run that has just been admitted.
|
||||||
response = await await_drained(run_file_io(_delete_thread_data, thread_id, user_id=user_id))
|
response = await await_drained(run_file_io(_delete_thread_data, thread_id, user_id=user_id))
|
||||||
|
|
||||||
# Retain ownership if a configured checkpointer cannot delete the thread.
|
# Remove checkpoints (best-effort)
|
||||||
# Read routes treat rowless checkpoints as untracked legacy threads.
|
|
||||||
checkpointer = getattr(request.app.state, "checkpointer", None)
|
checkpointer = getattr(request.app.state, "checkpointer", None)
|
||||||
if checkpointer is not None:
|
if checkpointer is not None:
|
||||||
try:
|
try:
|
||||||
|
if hasattr(checkpointer, "adelete_thread"):
|
||||||
await checkpointer.adelete_thread(thread_id)
|
await checkpointer.adelete_thread(thread_id)
|
||||||
except Exception as exc:
|
except Exception:
|
||||||
logger.exception("Failed to delete checkpoints for thread %s", sanitize_log_param(thread_id))
|
logger.debug("Could not delete checkpoints for thread %s (not critical)", sanitize_log_param(thread_id))
|
||||||
raise HTTPException(status_code=500, detail="Failed to delete thread checkpoints.") from exc
|
|
||||||
|
|
||||||
# Remove historical runs (best-effort). Only ``operation_kind == "run"`` rows
|
# Remove historical runs (best-effort). Only ``operation_kind == "run"`` rows
|
||||||
# are deleted, so the durable thread-operation reservation protecting this
|
# are deleted, so the durable thread-operation reservation protecting this
|
||||||
@ -881,16 +842,6 @@ async def _delete_thread_data_with_reservation(thread_id: str, request: Request)
|
|||||||
except Exception:
|
except Exception:
|
||||||
logger.debug("Could not delete run records for thread %s (not critical)", sanitize_log_param(thread_id))
|
logger.debug("Could not delete run records for thread %s (not critical)", sanitize_log_param(thread_id))
|
||||||
|
|
||||||
# Drop this worker's in-memory copies too. ``RunManager.get()`` and
|
|
||||||
# ``list_by_thread()`` answer from the process-local registry before the
|
|
||||||
# store, so without this the deleted thread's runs — including ``kwargs``
|
|
||||||
# with the user's prompt — stay readable through ``GET /threads/{id}/runs``
|
|
||||||
# until the Gateway restarts, while the thread itself already 404s.
|
|
||||||
try:
|
|
||||||
await run_manager.discard_thread_runs(thread_id)
|
|
||||||
except Exception:
|
|
||||||
logger.debug("Could not evict in-memory runs for thread %s (not critical)", sanitize_log_param(thread_id))
|
|
||||||
|
|
||||||
# Remove persisted run events (best-effort). These are the user-visible
|
# Remove persisted run events (best-effort). These are the user-visible
|
||||||
# conversation history, not a cache: leaving them behind makes a deleted
|
# conversation history, not a cache: leaving them behind makes a deleted
|
||||||
# thread's feed readable again through GET /threads/{id}/messages. A legacy
|
# thread's feed readable again through GET /threads/{id}/messages. A legacy
|
||||||
@ -1014,17 +965,11 @@ async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadRe
|
|||||||
and an empty checkpoint (so state endpoints work immediately).
|
and an empty checkpoint (so state endpoints work immediately).
|
||||||
Idempotent: returns the existing record when ``thread_id`` already exists.
|
Idempotent: returns the existing record when ``thread_id`` already exists.
|
||||||
"""
|
"""
|
||||||
thread_id = resolve_thread_id(body.thread_id)
|
|
||||||
async with goal_thread_lock(thread_id):
|
|
||||||
return await _create_thread_locked(body, request, thread_id)
|
|
||||||
|
|
||||||
|
|
||||||
async def _create_thread_locked(body: ThreadCreateRequest, request: Request, thread_id: str) -> ThreadResponse:
|
|
||||||
"""Keep the idempotency read and initial writes in one lifecycle lock."""
|
|
||||||
from app.gateway.deps import get_thread_store
|
from app.gateway.deps import get_thread_store
|
||||||
|
|
||||||
checkpointer = get_checkpointer(request)
|
checkpointer = get_checkpointer(request)
|
||||||
thread_store = get_thread_store(request)
|
thread_store = get_thread_store(request)
|
||||||
|
thread_id = resolve_thread_id(body.thread_id)
|
||||||
now = now_iso()
|
now = now_iso()
|
||||||
thread_owner_user_id = get_trusted_internal_owner_user_id(request)
|
thread_owner_user_id = get_trusted_internal_owner_user_id(request)
|
||||||
thread_owner_kwargs = {"user_id": thread_owner_user_id} if thread_owner_user_id else {}
|
thread_owner_kwargs = {"user_id": thread_owner_user_id} if thread_owner_user_id else {}
|
||||||
@ -1061,25 +1006,17 @@ async def _create_thread_locked(body: ThreadCreateRequest, request: Request, thr
|
|||||||
raise HTTPException(status_code=404, detail="Thread not found") from None
|
raise HTTPException(status_code=404, detail="Thread not found") from None
|
||||||
except IntegrityError:
|
except IntegrityError:
|
||||||
# The idempotency read above and this insert are not atomic: a
|
# The idempotency read above and this insert are not atomic: a
|
||||||
# another worker's request for the same thread_id can commit in between, so
|
# concurrent request for the same thread_id can commit in between, so
|
||||||
# the SQL-backed store rejects ours on the duplicate primary key.
|
# the SQL-backed store rejects ours on the duplicate primary key.
|
||||||
# Honour the documented idempotency contract by resolving the
|
# Honour the documented idempotency contract by resolving the
|
||||||
# now-existing record — running the same owner reconciliation the fast
|
# now-existing record — running the same owner reconciliation the fast
|
||||||
# path does — instead of surfacing the conflict as a 500. Same-process
|
# path does — instead of surfacing the conflict as a 500. The memory
|
||||||
# HTTP creates share the lifecycle lock; the memory store retains its
|
# store serializes same-id creates under its per-thread lock and keeps
|
||||||
# historical overwrite behavior for direct callers.
|
# its historical overwrite behavior, so it does not reach this branch.
|
||||||
existing_record = await _resolve_existing_thread(thread_store, thread_id, thread_owner_user_id, thread_owner_kwargs)
|
existing_record = await _resolve_existing_thread(thread_store, thread_id, thread_owner_user_id, thread_owner_kwargs)
|
||||||
if existing_record is not None:
|
if existing_record is not None:
|
||||||
return _existing_thread_response(thread_id, existing_record)
|
return _existing_thread_response(thread_id, existing_record)
|
||||||
# The same-owner reconciliation above found nothing, yet the insert hit
|
# A duplicate-key error with no row we can read back is a real failure.
|
||||||
# a duplicate primary key — so the surviving row belongs to *another*
|
|
||||||
# user. That is the ownership-conflict case, which the route already
|
|
||||||
# maps ``ThreadOwnershipConflictError`` to: a non-disclosing 404. Read
|
|
||||||
# the row back bypassing the owner scope to confirm a foreign row
|
|
||||||
# exists before collapsing to a 500.
|
|
||||||
if await thread_store.get(thread_id, user_id=None) is not None:
|
|
||||||
raise HTTPException(status_code=404, detail="Thread not found") from None
|
|
||||||
# A duplicate-key error with no surviving row is a real failure.
|
|
||||||
logger.exception("Failed to write thread_meta for %s", sanitize_log_param(thread_id))
|
logger.exception("Failed to write thread_meta for %s", sanitize_log_param(thread_id))
|
||||||
raise HTTPException(status_code=500, detail="Failed to create thread")
|
raise HTTPException(status_code=500, detail="Failed to create thread")
|
||||||
except Exception:
|
except Exception:
|
||||||
@ -1090,14 +1027,12 @@ async def _create_thread_locked(body: ThreadCreateRequest, request: Request, thr
|
|||||||
# Write an empty checkpoint so state endpoints work immediately
|
# Write an empty checkpoint so state endpoints work immediately
|
||||||
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||||
try:
|
try:
|
||||||
# Client metadata cannot supply checkpoint-owned keys; see
|
|
||||||
# ``_CHECKPOINT_OWNED_METADATA_KEYS``.
|
|
||||||
ckpt_metadata = {
|
ckpt_metadata = {
|
||||||
**{key: value for key, value in metadata.items() if key not in _CHECKPOINT_OWNED_METADATA_KEYS},
|
|
||||||
"step": -1,
|
"step": -1,
|
||||||
"source": "input",
|
"source": "input",
|
||||||
"writes": None,
|
"writes": None,
|
||||||
"parents": {},
|
"parents": {},
|
||||||
|
**metadata,
|
||||||
"created_at": now,
|
"created_at": now,
|
||||||
}
|
}
|
||||||
await checkpointer.aput(config, empty_checkpoint(), ckpt_metadata, {})
|
await checkpointer.aput(config, empty_checkpoint(), ckpt_metadata, {})
|
||||||
@ -1286,9 +1221,9 @@ async def _branch_thread_with_reservation(
|
|||||||
# into a new thread. Preserve every valid spelling as create does.
|
# into a new thread. Preserve every valid spelling as create does.
|
||||||
branch_assistant_id = None
|
branch_assistant_id = None
|
||||||
|
|
||||||
async def _write_branch_row(project_id: str | None) -> dict:
|
async def _write_branch_row(project_id: str | None) -> None:
|
||||||
try:
|
try:
|
||||||
return await thread_store.create(
|
await thread_store.create(
|
||||||
new_thread_id,
|
new_thread_id,
|
||||||
assistant_id=branch_assistant_id,
|
assistant_id=branch_assistant_id,
|
||||||
display_name=display_name,
|
display_name=display_name,
|
||||||
@ -1303,13 +1238,13 @@ async def _branch_thread_with_reservation(
|
|||||||
raise HTTPException(status_code=500, detail="Failed to create branch") from None
|
raise HTTPException(status_code=500, detail="Failed to create branch") from None
|
||||||
|
|
||||||
try:
|
try:
|
||||||
branch_record = await _write_branch_row(source_project_id)
|
await _write_branch_row(source_project_id)
|
||||||
except ProjectNotAssignableError:
|
except ProjectNotAssignableError:
|
||||||
# The source project became unassignable (archived, or deleted in a
|
# The source project became unassignable (archived, or deleted in a
|
||||||
# race that cleared the source row's membership after our read): keep
|
# race that cleared the source row's membership after our read): keep
|
||||||
# the branch usable as an unassigned thread — the pre-inheritance
|
# the branch usable as an unassigned thread — the pre-inheritance
|
||||||
# behavior for sources without an active project.
|
# behavior for sources without an active project.
|
||||||
branch_record = await _write_branch_row(None)
|
await _write_branch_row(None)
|
||||||
|
|
||||||
# The thread feed (GET /messages, /messages/page) reads the run-event
|
# The thread feed (GET /messages, /messages/page) reads the run-event
|
||||||
# store, not checkpoints, and a fresh branch has no run_events — so the
|
# store, not checkpoints, and a fresh branch has no run_events — so the
|
||||||
@ -1319,7 +1254,6 @@ async def _branch_thread_with_reservation(
|
|||||||
# from. Best-effort: on failure the branch stays usable, with history
|
# from. Best-effort: on failure the branch stays usable, with history
|
||||||
# visible only through the checkpoint overlay until it is re-branched.
|
# visible only through the checkpoint overlay until it is re-branched.
|
||||||
try:
|
try:
|
||||||
async with reserve_history_seed(request, new_thread_id) as reservation:
|
|
||||||
seed_events = build_branch_history_seed_events(
|
seed_events = build_branch_history_seed_events(
|
||||||
_checkpoint_messages(snapshot),
|
_checkpoint_messages(snapshot),
|
||||||
thread_id=new_thread_id,
|
thread_id=new_thread_id,
|
||||||
@ -1327,9 +1261,7 @@ async def _branch_thread_with_reservation(
|
|||||||
parent_thread_id=thread_id,
|
parent_thread_id=thread_id,
|
||||||
)
|
)
|
||||||
if seed_events:
|
if seed_events:
|
||||||
event_store = get_run_event_store(request)
|
await get_run_event_store(request).put_batch(seed_events)
|
||||||
if reservation is None or not await event_store.list_messages(new_thread_id, limit=1, user_id=None):
|
|
||||||
await write_history_seed(request, seed_events, reservation=reservation, seed_kind="branch", thread=branch_record)
|
|
||||||
history_seed_mode = "seeded"
|
history_seed_mode = "seeded"
|
||||||
else:
|
else:
|
||||||
history_seed_mode = "skipped_empty"
|
history_seed_mode = "skipped_empty"
|
||||||
@ -1369,7 +1301,6 @@ async def search_threads(body: ThreadSearchRequest, request: Request) -> list[Th
|
|||||||
try:
|
try:
|
||||||
rows = await repo.search(
|
rows = await repo.search(
|
||||||
metadata=body.metadata or None,
|
metadata=body.metadata or None,
|
||||||
**({"query": body.query} if body.query is not None else {}),
|
|
||||||
status=body.status,
|
status=body.status,
|
||||||
**({"archived": body.archived} if body.archived is not None else {}),
|
**({"archived": body.archived} if body.archived is not None else {}),
|
||||||
project_id=project_filter,
|
project_id=project_filter,
|
||||||
@ -1541,7 +1472,6 @@ async def get_thread(thread_id: ThreadId, request: Request) -> ThreadResponse:
|
|||||||
updated_at=coerce_iso(record.get("updated_at", "")),
|
updated_at=coerce_iso(record.get("updated_at", "")),
|
||||||
metadata=record.get("metadata", {}),
|
metadata=record.get("metadata", {}),
|
||||||
values=serialize_channel_values_for_api(snapshot.values),
|
values=serialize_channel_values_for_api(snapshot.values),
|
||||||
interrupts={task_id: [serialize_lc_object(interrupt) for interrupt in interrupts] for task_id, interrupts in (await find_pending_interrupts(request, thread_id, accessor=accessor, read_config=config, snapshot=snapshot)).items()},
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@ -1567,21 +1497,11 @@ async def set_thread_goal(thread_id: ThreadId, body: ThreadGoalRequest, request:
|
|||||||
this endpoint creates the missing thread checkpoint on demand.
|
this endpoint creates the missing thread checkpoint on demand.
|
||||||
"""
|
"""
|
||||||
checkpointer = get_checkpointer(request)
|
checkpointer = get_checkpointer(request)
|
||||||
try:
|
|
||||||
accessor, read_config = await build_thread_checkpoint_state_accessor(request, thread_id=thread_id)
|
|
||||||
from app.gateway.hitl_approval import reject_new_run_when_approval_pending
|
|
||||||
|
|
||||||
await reject_new_run_when_approval_pending(request, thread_id, accessor=accessor, read_config=read_config)
|
|
||||||
except _CHECKPOINT_MODE_ERRORS as exc:
|
|
||||||
raise _checkpoint_mode_http_error(exc, thread_id) from exc
|
|
||||||
try:
|
try:
|
||||||
goal = build_goal_state(body.objective, max_continuations=body.max_continuations)
|
goal = build_goal_state(body.objective, max_continuations=body.max_continuations)
|
||||||
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
||||||
await _ensure_thread_for_goal(thread_id, request)
|
await _ensure_thread_for_goal(thread_id, request)
|
||||||
await write_thread_goal(checkpointer, thread_id, goal, as_node="goal", create_if_missing=True)
|
await write_thread_goal(checkpointer, thread_id, goal, as_node="goal", create_if_missing=True)
|
||||||
batch_repo = getattr(request.app.state, "subagent_batch_repo", None)
|
|
||||||
if batch_repo is not None:
|
|
||||||
await batch_repo.suppress_thread_continuations(thread_id)
|
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||||
except ConflictError:
|
except ConflictError:
|
||||||
@ -1594,154 +1514,14 @@ async def set_thread_goal(thread_id: ThreadId, body: ThreadGoalRequest, request:
|
|||||||
return ThreadGoalResponse(goal=goal)
|
return ThreadGoalResponse(goal=goal)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/{thread_id}/goal/from-chat", response_model=ThreadGoalResponse)
|
|
||||||
@require_permission("threads", "write", owner_check=True)
|
|
||||||
async def synthesize_thread_goal_from_chat(thread_id: ThreadId, body: ThreadGoalFromChatRequest, request: Request, app_config: AppConfig = Depends(get_config)) -> ThreadGoalResponse:
|
|
||||||
"""Synthesize a goal objective from the thread's recent conversation and save it.
|
|
||||||
|
|
||||||
The flow the #7050 first slice agreed on: a busy pre-check runs **before**
|
|
||||||
the synthesis LLM call (a busy thread must not spend a model call it could
|
|
||||||
not save under anyway), the LLM composes an objective preserving the agreed
|
|
||||||
constraints and stopping condition, and the save reuses ``PUT /goal``'s
|
|
||||||
reservation path — its ``ConflictError`` stays the authoritative busy
|
|
||||||
guard, because another run can start between the pre-check and the save.
|
|
||||||
|
|
||||||
Unlike ``PUT /goal`` (which mints a missing checkpoint for ``/chats/new``
|
|
||||||
pages), from-chat synthesizes *from* an existing conversation: no
|
|
||||||
checkpoint or no visible context is a 422 ``no_context``, never a new
|
|
||||||
thread. A conversation whose last turn is an unanswered human-input
|
|
||||||
request (a clarification, a sandbox-network approval, an extension prompt)
|
|
||||||
is a 422 ``pending_clarification``: the goal is not agreed yet, and
|
|
||||||
synthesizing anyway lets the model invent the missing decision. A pending
|
|
||||||
tool approval is a 409 like the other goal mutations, and the save
|
|
||||||
suppresses subagent-batch continuations queued under the previous goal.
|
|
||||||
The save is a compare-and-swap against the checkpoint the synthesis read:
|
|
||||||
a goal or checkpoint write that landed during the model call rejects the
|
|
||||||
stale objective with a 409 ``goal_changed`` instead of clobbering it.
|
|
||||||
The synthesis is read-only until the save; the caller starts the run through
|
|
||||||
its own existing run path.
|
|
||||||
"""
|
|
||||||
# Best-effort pre-check: has_inflight reads this worker's registry, so a
|
|
||||||
# multi-worker deployment can miss a peer's run. Missing it costs one
|
|
||||||
# wasted synthesis; the save-time reservation below still rejects. A
|
|
||||||
# failing pre-check degrades the same way rather than blocking the user.
|
|
||||||
try:
|
|
||||||
if await get_run_manager(request).has_inflight(thread_id):
|
|
||||||
raise HTTPException(status_code=409, detail="Thread has a run in flight. Set the goal after the run finishes.")
|
|
||||||
except HTTPException:
|
|
||||||
raise
|
|
||||||
except Exception:
|
|
||||||
logger.warning("Goal from-chat busy pre-check failed; falling through to synthesis", exc_info=True)
|
|
||||||
|
|
||||||
try:
|
|
||||||
accessor, config = await build_thread_checkpoint_state_accessor(request, thread_id=thread_id)
|
|
||||||
# Same gate as PUT/DELETE /goal: a pending tool approval must not be
|
|
||||||
# silently discarded by changing the goal underneath it.
|
|
||||||
from app.gateway.hitl_approval import reject_new_run_when_approval_pending
|
|
||||||
|
|
||||||
await reject_new_run_when_approval_pending(request, thread_id, accessor=accessor, read_config=config)
|
|
||||||
snapshot = await accessor.aget(config)
|
|
||||||
except _CHECKPOINT_MODE_ERRORS as exc:
|
|
||||||
raise _checkpoint_mode_http_error(exc, thread_id) from exc
|
|
||||||
except HTTPException:
|
|
||||||
raise
|
|
||||||
except Exception:
|
|
||||||
logger.exception("Failed to read thread %s state for goal synthesis", sanitize_log_param(thread_id))
|
|
||||||
raise HTTPException(status_code=500, detail="Failed to read thread state") from None
|
|
||||||
source_checkpoint_id = (snapshot.config or {}).get("configurable", {}).get("checkpoint_id")
|
|
||||||
if not source_checkpoint_id:
|
|
||||||
raise HTTPException(status_code=422, detail={"code": "no_context"})
|
|
||||||
|
|
||||||
values = snapshot.values or {}
|
|
||||||
messages = values.get("messages")
|
|
||||||
messages_list = messages if isinstance(messages, list) else []
|
|
||||||
if extract_unanswered_human_input_request(messages_list) is not None:
|
|
||||||
# The agreement is incomplete: the user still owes the assistant an
|
|
||||||
# answer. Running now would force the model to guess that decision.
|
|
||||||
raise HTTPException(status_code=422, detail={"code": "pending_clarification"})
|
|
||||||
summary_text = values.get("summary_text")
|
|
||||||
# Redaction is part of the builder so it runs before the budget can split
|
|
||||||
# an identifier; synthesize_goal_objective redacts the composed prompt once
|
|
||||||
# more as defense in depth.
|
|
||||||
context = build_synthesis_context(
|
|
||||||
messages_list,
|
|
||||||
summary_text if isinstance(summary_text, str) else None,
|
|
||||||
pii_redaction=getattr(app_config, "pii_redaction", None),
|
|
||||||
)
|
|
||||||
if not context.strip():
|
|
||||||
raise HTTPException(status_code=422, detail={"code": "no_context"})
|
|
||||||
|
|
||||||
# Model admission before the synthesis call: the route sends thread
|
|
||||||
# context to the deployment's default model (the same factory default
|
|
||||||
# run_oneshot_llm resolves when model_name is None), so a role denied
|
|
||||||
# that model must get the same 403 the suggestions route returns.
|
|
||||||
# Checked ahead of the model call, not at save time: a denied profile
|
|
||||||
# must never receive the conversation even transiently.
|
|
||||||
user = await get_current_user_from_request(request)
|
|
||||||
await asyncio.to_thread(authorize_model_use, user, None, is_internal=_is_internal_caller(request, user), app_config=app_config)
|
|
||||||
|
|
||||||
# The synthesis call runs outside every lock/reservation: a model call can
|
|
||||||
# take tens of seconds and must not hold the thread's write admission.
|
|
||||||
try:
|
|
||||||
objective = await synthesize_goal_objective(
|
|
||||||
context=context,
|
|
||||||
guidance=body.guidance or "",
|
|
||||||
app_config=app_config,
|
|
||||||
thread_id=thread_id,
|
|
||||||
)
|
|
||||||
except GoalSynthesisUnclear:
|
|
||||||
# The model's reason is derived from conversation content: it is never
|
|
||||||
# echoed to the user, and it is not logged verbatim either.
|
|
||||||
logger.info("Goal from-chat found no clear goal in thread %s", sanitize_log_param(thread_id))
|
|
||||||
raise HTTPException(status_code=422, detail={"code": "unclear"}) from None
|
|
||||||
except GoalSynthesisError:
|
|
||||||
logger.warning("Goal from-chat synthesis failed for thread %s", sanitize_log_param(thread_id), exc_info=True)
|
|
||||||
raise HTTPException(status_code=502, detail={"code": "synthesis_failed"}) from None
|
|
||||||
|
|
||||||
checkpointer = get_checkpointer(request)
|
|
||||||
goal = build_goal_state(objective)
|
|
||||||
try:
|
|
||||||
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
|
||||||
# Compare-and-swap against the head the synthesis read: a goal (or
|
|
||||||
# any checkpoint write) that landed during the model call means the
|
|
||||||
# objective was composed from a conversation that is no longer
|
|
||||||
# current, and saving it would silently clobber the newer state.
|
|
||||||
await write_thread_goal(checkpointer, thread_id, goal, as_node="goal", expected_checkpoint_id=source_checkpoint_id)
|
|
||||||
# Match PUT/DELETE /goal: a new goal supersedes queued subagent-batch
|
|
||||||
# continuations that were scheduled under the previous one.
|
|
||||||
batch_repo = getattr(request.app.state, "subagent_batch_repo", None)
|
|
||||||
if batch_repo is not None:
|
|
||||||
await batch_repo.suppress_thread_continuations(thread_id)
|
|
||||||
except GoalWriteConflict:
|
|
||||||
raise HTTPException(status_code=409, detail={"code": "goal_changed"}) from None
|
|
||||||
except ConflictError:
|
|
||||||
raise HTTPException(status_code=409, detail="Thread has a run in flight. Set the goal after the run finishes.") from None
|
|
||||||
except HTTPException:
|
|
||||||
raise
|
|
||||||
except Exception:
|
|
||||||
logger.exception("Failed to save from-chat goal for thread %s", sanitize_log_param(thread_id))
|
|
||||||
raise HTTPException(status_code=500, detail="Failed to set thread goal") from None
|
|
||||||
return ThreadGoalResponse(goal=goal)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/{thread_id}/goal", response_model=ThreadGoalResponse)
|
@router.delete("/{thread_id}/goal", response_model=ThreadGoalResponse)
|
||||||
@require_permission("threads", "write", owner_check=True)
|
@require_permission("threads", "write", owner_check=True)
|
||||||
async def clear_thread_goal(thread_id: ThreadId, request: Request) -> ThreadGoalResponse:
|
async def clear_thread_goal(thread_id: ThreadId, request: Request) -> ThreadGoalResponse:
|
||||||
"""Clear the active goal for a thread."""
|
"""Clear the active goal for a thread."""
|
||||||
checkpointer = get_checkpointer(request)
|
checkpointer = get_checkpointer(request)
|
||||||
try:
|
|
||||||
accessor, read_config = await build_thread_checkpoint_state_accessor(request, thread_id=thread_id)
|
|
||||||
from app.gateway.hitl_approval import reject_new_run_when_approval_pending
|
|
||||||
|
|
||||||
await reject_new_run_when_approval_pending(request, thread_id, accessor=accessor, read_config=read_config)
|
|
||||||
except _CHECKPOINT_MODE_ERRORS as exc:
|
|
||||||
raise _checkpoint_mode_http_error(exc, thread_id) from exc
|
|
||||||
try:
|
try:
|
||||||
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
||||||
await write_thread_goal(checkpointer, thread_id, None, as_node="goal")
|
await write_thread_goal(checkpointer, thread_id, None, as_node="goal")
|
||||||
batch_repo = getattr(request.app.state, "subagent_batch_repo", None)
|
|
||||||
if batch_repo is not None:
|
|
||||||
await batch_repo.suppress_thread_continuations(thread_id)
|
|
||||||
except ConflictError:
|
except ConflictError:
|
||||||
raise HTTPException(status_code=409, detail="Thread has a run in flight. Clear the goal after the run finishes.") from None
|
raise HTTPException(status_code=409, detail="Thread has a run in flight. Clear the goal after the run finishes.") from None
|
||||||
except LookupError:
|
except LookupError:
|
||||||
@ -1774,14 +1554,11 @@ async def compact_thread(thread_id: ThreadId, body: ThreadCompactRequest, reques
|
|||||||
# forward by checkpoint fork inheritance, so the base-schema mutation
|
# forward by checkpoint fork inheritance, so the base-schema mutation
|
||||||
# graph is sufficient (and avoids building the full lead graph per call).
|
# graph is sufficient (and avoids building the full lead graph per call).
|
||||||
try:
|
try:
|
||||||
accessor, read_config = build_checkpoint_state_mutation_accessor(
|
accessor, _ = build_checkpoint_state_mutation_accessor(
|
||||||
request,
|
request,
|
||||||
thread_id=thread_id,
|
thread_id=thread_id,
|
||||||
as_node="manual_compaction",
|
as_node="manual_compaction",
|
||||||
)
|
)
|
||||||
from app.gateway.hitl_approval import reject_new_run_when_approval_pending
|
|
||||||
|
|
||||||
await reject_new_run_when_approval_pending(request, thread_id)
|
|
||||||
except _CHECKPOINT_MODE_ERRORS as exc:
|
except _CHECKPOINT_MODE_ERRORS as exc:
|
||||||
raise _checkpoint_mode_http_error(exc, thread_id) from exc
|
raise _checkpoint_mode_http_error(exc, thread_id) from exc
|
||||||
keep = body.keep.to_tuple() if body.keep is not None else None
|
keep = body.keep.to_tuple() if body.keep is not None else None
|
||||||
@ -1816,32 +1593,6 @@ async def compact_thread(thread_id: ThreadId, body: ThreadCompactRequest, reques
|
|||||||
return _thread_compact_response(result)
|
return _thread_compact_response(result)
|
||||||
|
|
||||||
|
|
||||||
def _pending_interrupt_tasks(tasks_raw: Any, pending_interrupts: dict[str, list[Any]]) -> list[dict[str, Any]]:
|
|
||||||
"""Project pending interrupts without dropping run-store-only identities."""
|
|
||||||
tasks: list[dict[str, Any]] = []
|
|
||||||
matched_task_ids: set[str] = set()
|
|
||||||
for task in tasks_raw:
|
|
||||||
task_id = getattr(task, "id", "")
|
|
||||||
matched_task_ids.add(task_id)
|
|
||||||
tasks.append(
|
|
||||||
{
|
|
||||||
"id": task_id,
|
|
||||||
"name": getattr(task, "name", ""),
|
|
||||||
"interrupts": [serialize_lc_object(interrupt) for interrupt in pending_interrupts.get(task_id, [])],
|
|
||||||
}
|
|
||||||
)
|
|
||||||
for task_id, interrupts in pending_interrupts.items():
|
|
||||||
if task_id not in matched_task_ids:
|
|
||||||
tasks.append(
|
|
||||||
{
|
|
||||||
"id": task_id,
|
|
||||||
"name": "",
|
|
||||||
"interrupts": [serialize_lc_object(interrupt) for interrupt in interrupts],
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return tasks
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@router.get("/{thread_id}/state", response_model=ThreadStateResponse)
|
@router.get("/{thread_id}/state", response_model=ThreadStateResponse)
|
||||||
@require_permission("threads", "read", owner_check=True)
|
@require_permission("threads", "read", owner_check=True)
|
||||||
@ -1865,12 +1616,13 @@ async def get_thread_state(thread_id: ThreadId, request: Request) -> ThreadState
|
|||||||
checkpoint_id = snapshot_config.get("configurable", {}).get("checkpoint_id")
|
checkpoint_id = snapshot_config.get("configurable", {}).get("checkpoint_id")
|
||||||
if not checkpoint_id:
|
if not checkpoint_id:
|
||||||
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
|
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
|
||||||
|
|
||||||
parent_config = snapshot.parent_config or {}
|
parent_config = snapshot.parent_config or {}
|
||||||
pending_interrupts = await find_pending_interrupts(request, thread_id, accessor=accessor, read_config=config, snapshot=snapshot)
|
|
||||||
parent_checkpoint_id = parent_config.get("configurable", {}).get("checkpoint_id")
|
parent_checkpoint_id = parent_config.get("configurable", {}).get("checkpoint_id")
|
||||||
metadata = snapshot.metadata or {}
|
metadata = snapshot.metadata or {}
|
||||||
created_at = snapshot.created_at or metadata.get("created_at", "")
|
created_at = snapshot.created_at or metadata.get("created_at", "")
|
||||||
tasks = _pending_interrupt_tasks(snapshot.tasks or (), pending_interrupts)
|
tasks_raw = snapshot.tasks or ()
|
||||||
|
tasks = [{"id": getattr(task, "id", ""), "name": getattr(task, "name", "")} for task in tasks_raw]
|
||||||
|
|
||||||
values = serialize_channel_values_for_api(snapshot.values)
|
values = serialize_channel_values_for_api(snapshot.values)
|
||||||
messages = values.get("messages")
|
messages = values.get("messages")
|
||||||
@ -1896,9 +1648,7 @@ async def get_thread_state(thread_id: ThreadId, request: Request) -> ThreadState
|
|||||||
async def update_thread_state(thread_id: ThreadId, body: ThreadStateUpdateRequest, request: Request) -> ThreadStateResponse:
|
async def update_thread_state(thread_id: ThreadId, body: ThreadStateUpdateRequest, request: Request) -> ThreadStateResponse:
|
||||||
"""Replace selected thread-state fields through the materialized graph."""
|
"""Replace selected thread-state fields through the materialized graph."""
|
||||||
from app.gateway.deps import get_thread_store
|
from app.gateway.deps import get_thread_store
|
||||||
from app.gateway.hitl_approval import reject_new_run_when_approval_pending
|
|
||||||
|
|
||||||
await reject_new_run_when_approval_pending(request, thread_id)
|
|
||||||
thread_store = get_thread_store(request)
|
thread_store = get_thread_store(request)
|
||||||
# Validate external roles before materializing a graph or reserving a write.
|
# Validate external roles before materializing a graph or reserving a write.
|
||||||
values = strip_server_owned_state_metadata(dict(body.values or {}))
|
values = strip_server_owned_state_metadata(dict(body.values or {}))
|
||||||
@ -1962,7 +1712,6 @@ async def update_thread_state(thread_id: ThreadId, body: ThreadStateUpdateReques
|
|||||||
updates[GOAL_OUTCOME_CHANNEL] = None
|
updates[GOAL_OUTCOME_CHANNEL] = None
|
||||||
try:
|
try:
|
||||||
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
|
||||||
await reject_new_run_when_approval_pending(request, thread_id)
|
|
||||||
source_metadata = await accessor.aget_metadata(read_config)
|
source_metadata = await accessor.aget_metadata(read_config)
|
||||||
update_config = {
|
update_config = {
|
||||||
**read_config,
|
**read_config,
|
||||||
@ -1979,8 +1728,6 @@ async def update_thread_state(thread_id: ThreadId, body: ThreadStateUpdateReques
|
|||||||
raise HTTPException(status_code=409, detail="Thread has a run in flight. Update state after the run finishes.") from None
|
raise HTTPException(status_code=409, detail="Thread has a run in flight. Update state after the run finishes.") from None
|
||||||
except _CHECKPOINT_MODE_ERRORS as exc:
|
except _CHECKPOINT_MODE_ERRORS as exc:
|
||||||
raise _checkpoint_mode_http_error(exc, thread_id) from exc
|
raise _checkpoint_mode_http_error(exc, thread_id) from exc
|
||||||
except HTTPException:
|
|
||||||
raise
|
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("Failed to update state for thread %s", sanitize_log_param(thread_id))
|
logger.exception("Failed to update state for thread %s", sanitize_log_param(thread_id))
|
||||||
raise HTTPException(status_code=500, detail="Failed to update thread state")
|
raise HTTPException(status_code=500, detail="Failed to update thread state")
|
||||||
@ -2249,7 +1996,6 @@ async def get_thread_history(
|
|||||||
stamp_turn_duration_on_last_ai(
|
stamp_turn_duration_on_last_ai(
|
||||||
serialized_msgs,
|
serialized_msgs,
|
||||||
checkpoint_run_durations,
|
checkpoint_run_durations,
|
||||||
thread_id=thread_id,
|
|
||||||
)
|
)
|
||||||
raise
|
raise
|
||||||
|
|
||||||
@ -2300,7 +2046,6 @@ async def get_thread_history(
|
|||||||
stamp_turn_duration_on_last_ai(
|
stamp_turn_duration_on_last_ai(
|
||||||
serialized_msgs,
|
serialized_msgs,
|
||||||
resolved_run_durations,
|
resolved_run_durations,
|
||||||
thread_id=thread_id,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
except Exception:
|
except Exception:
|
||||||
@ -2316,14 +2061,10 @@ async def get_thread_history(
|
|||||||
serialized_msgs,
|
serialized_msgs,
|
||||||
)
|
)
|
||||||
|
|
||||||
next_tasks = list(snapshot.next or ())
|
|
||||||
if is_latest_checkpoint:
|
|
||||||
pending_interrupts = await find_pending_interrupts(request, thread_id, accessor=accessor, read_config=config, snapshot=snapshot)
|
|
||||||
head_tasks = _pending_interrupt_tasks(getattr(snapshot, "tasks", None) or (), pending_interrupts)
|
|
||||||
else:
|
|
||||||
head_tasks = [{"id": getattr(task, "id", ""), "name": getattr(task, "name", "")} for task in (getattr(snapshot, "tasks", None) or ())]
|
|
||||||
|
|
||||||
is_latest_checkpoint = False
|
is_latest_checkpoint = False
|
||||||
|
|
||||||
|
next_tasks = list(snapshot.next or ())
|
||||||
|
|
||||||
# Strip LangGraph internal keys from metadata
|
# Strip LangGraph internal keys from metadata
|
||||||
user_meta = {k: v for k, v in metadata.items() if k not in ("created_at", "updated_at", "step", "source", "writes", "parents", "run_durations", RUN_MESSAGE_IDS_METADATA_KEY)}
|
user_meta = {k: v for k, v in metadata.items() if k not in ("created_at", "updated_at", "step", "source", "writes", "parents", "run_durations", RUN_MESSAGE_IDS_METADATA_KEY)}
|
||||||
# Keep step for ordering context
|
# Keep step for ordering context
|
||||||
@ -2338,7 +2079,6 @@ async def get_thread_history(
|
|||||||
values=values,
|
values=values,
|
||||||
created_at=coerce_iso(snapshot.created_at or metadata.get("created_at", "")),
|
created_at=coerce_iso(snapshot.created_at or metadata.get("created_at", "")),
|
||||||
next=next_tasks,
|
next=next_tasks,
|
||||||
tasks=head_tasks,
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
except _CHECKPOINT_MODE_ERRORS as exc:
|
except _CHECKPOINT_MODE_ERRORS as exc:
|
||||||
|
|||||||
@ -14,7 +14,6 @@ not a server-enforced handshake (§11).
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import logging
|
import logging
|
||||||
import time
|
import time
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
@ -104,10 +103,13 @@ class PurgeResponse(BaseModel):
|
|||||||
|
|
||||||
|
|
||||||
def _projects_config() -> ProjectsConfig:
|
def _projects_config() -> ProjectsConfig:
|
||||||
"""Read the retention policy; an unavailable config must not shorten it."""
|
"""Projects config, falling back to defaults when the app config is unavailable."""
|
||||||
from deerflow.config.app_config import get_app_config
|
from deerflow.config.app_config import get_app_config
|
||||||
|
|
||||||
|
try:
|
||||||
return get_app_config().projects
|
return get_app_config().projects
|
||||||
|
except Exception:
|
||||||
|
return ProjectsConfig()
|
||||||
|
|
||||||
|
|
||||||
def _not_found() -> HTTPException:
|
def _not_found() -> HTTPException:
|
||||||
@ -147,11 +149,10 @@ async def list_trash_documents(
|
|||||||
repo = get_project_document_repo(request)
|
repo = get_project_document_repo(request)
|
||||||
user_id = get_effective_user_id()
|
user_id = get_effective_user_id()
|
||||||
try:
|
try:
|
||||||
config = await asyncio.to_thread(_projects_config)
|
|
||||||
await run_trash_retention_sweep(
|
await run_trash_retention_sweep(
|
||||||
repo,
|
repo,
|
||||||
get_paths(),
|
get_paths(),
|
||||||
retention_days=config.trash_retention_days,
|
retention_days=_projects_config().trash_retention_days,
|
||||||
user_id=user_id,
|
user_id=user_id,
|
||||||
include_reconciliation=_reconciliation_due(user_id),
|
include_reconciliation=_reconciliation_due(user_id),
|
||||||
)
|
)
|
||||||
|
|||||||
@ -1,193 +0,0 @@
|
|||||||
"""HTTP admission and ordinary AgentRuns dispatch for in-flight user inputs."""
|
|
||||||
|
|
||||||
import copy
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
from typing import Literal
|
|
||||||
from uuid import NAMESPACE_URL, uuid5
|
|
||||||
|
|
||||||
from fastapi import HTTPException, Request
|
|
||||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
|
||||||
|
|
||||||
from app.gateway.deps import get_agent_runs, get_current_user, get_run_manager
|
|
||||||
from deerflow.runtime.runs.inputs import InputCancelled, InputConflict, RunInputs
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class RunInputRequest(BaseModel):
|
|
||||||
model_config = ConfigDict(extra="forbid")
|
|
||||||
request_id: str = Field(min_length=1, max_length=96)
|
|
||||||
text: str = Field(min_length=1, max_length=16 * 1024)
|
|
||||||
mode: Literal["queue", "steer"]
|
|
||||||
context: dict = Field(default_factory=dict)
|
|
||||||
|
|
||||||
@model_validator(mode="after")
|
|
||||||
def bounded_context(self):
|
|
||||||
if len(json.dumps(self.context, ensure_ascii=True, allow_nan=False).encode()) > 32 * 1024:
|
|
||||||
raise ValueError("Input context exceeds 32 KiB")
|
|
||||||
return self
|
|
||||||
|
|
||||||
|
|
||||||
def _store(request):
|
|
||||||
store = get_run_manager(request).input_store
|
|
||||||
if store is None:
|
|
||||||
raise HTTPException(503, "Run inputs are unavailable")
|
|
||||||
return store
|
|
||||||
|
|
||||||
|
|
||||||
async def submit_run_input(thread_id: str, run_id: str, body: RunInputRequest, request: Request):
|
|
||||||
manager = get_run_manager(request)
|
|
||||||
user_id = await get_current_user(request)
|
|
||||||
if user_id is None:
|
|
||||||
raise HTTPException(401, "Authentication required")
|
|
||||||
run = await manager.get(run_id, user_id=user_id)
|
|
||||||
if run is None or run.thread_id != thread_id:
|
|
||||||
raise HTTPException(404, "Run not found")
|
|
||||||
if get_agent_runs(request) is None:
|
|
||||||
raise HTTPException(503, "Run inputs require an authenticated session")
|
|
||||||
row = await _store(request).get(run_id, user_id=user_id)
|
|
||||||
if row is None or row.get("run_inputs") is None:
|
|
||||||
raise HTTPException(409, "This Run has not enabled interactive inputs")
|
|
||||||
try:
|
|
||||||
from app.gateway.services import _CONTEXT_CONFIGURABLE_KEYS
|
|
||||||
from deerflow.runtime.run_origin import admit_origin
|
|
||||||
|
|
||||||
context = {key: value for key, value in body.context.items() if key in _CONTEXT_CONFIGURABLE_KEYS and key not in ("agent_name", "is_bootstrap")}
|
|
||||||
if isinstance(body.context.get("locale"), str):
|
|
||||||
context["locale"] = body.context["locale"]
|
|
||||||
source = "extension" if admit_origin(getattr(request.state, "run_origin", None)) is not None else "user"
|
|
||||||
item = await RunInputs(_store(request), run_id, user_id).submit(body.request_id, body.text, body.mode, context=context, source=source)
|
|
||||||
except InputConflict as exc:
|
|
||||||
raise HTTPException(409, str(exc)) from exc
|
|
||||||
return {**item, "target_run_id": run_id}
|
|
||||||
|
|
||||||
|
|
||||||
async def list_thread_inputs(thread_id: str, request: Request):
|
|
||||||
user_id = await get_current_user(request)
|
|
||||||
if user_id is None:
|
|
||||||
raise HTTPException(401, "Authentication required")
|
|
||||||
rows = await _store(request).list_input_runs(thread_id, user_id=user_id)
|
|
||||||
items = []
|
|
||||||
for row in reversed(rows):
|
|
||||||
for entry in (row.get("run_inputs") or {}).get("items", []):
|
|
||||||
status = entry["status"]
|
|
||||||
if status in ("pending", "applying") and (row["status"] not in ("pending", "running", "success") or row.get("stop_reason")):
|
|
||||||
status = "not_applied"
|
|
||||||
items.append({key: value for key, value in {**entry, "status": status, "target_run_id": row["run_id"]}.items() if key != "context"})
|
|
||||||
# Rank unresolved inputs (pending / not_applied / ...) ahead of terminal
|
|
||||||
# applied/cancelled receipts, prefer the newest entry within each group,
|
|
||||||
# cap at 32 records, then restore chronological order. A plain items[-32:]
|
|
||||||
# slice lets two full batches of receipts hide an older retryable input
|
|
||||||
# (issue #7972). If more than 32 inputs are unresolved, this deliberately
|
|
||||||
# returns 32 unresolved rows and no receipts for submission reconciliation;
|
|
||||||
# that trade-off follows the priority the issue asks for.
|
|
||||||
keep = sorted(range(len(items)), key=lambda i: (items[i]["status"] in ("applied", "cancelled"), -i))[:32]
|
|
||||||
return {"items": [items[i] for i in sorted(keep)], "active_run_id": await _store(request).active_input_run(thread_id, user_id=user_id)}
|
|
||||||
|
|
||||||
|
|
||||||
async def dispatch_queued_inputs(inputs: RunInputs, *, agent_runs, thread_id: str):
|
|
||||||
"""Use normal admission/authorization and its stable idempotency key.
|
|
||||||
|
|
||||||
This runs after the previous worker has fully finalized. No second main
|
|
||||||
Agent, task queue service or recovery scheduler is introduced.
|
|
||||||
"""
|
|
||||||
row = await inputs.store.get(inputs.run_id, user_id=inputs.user_id)
|
|
||||||
if row is None or row["status"] != "success" or row.get("stop_reason"):
|
|
||||||
return
|
|
||||||
state = await inputs.read()
|
|
||||||
for entry in state["items"]:
|
|
||||||
if entry["mode"] != "queue" or entry["status"] != "pending":
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
try:
|
|
||||||
run = await agent_runs.dispatch_input(thread_id=thread_id, run_id=inputs.run_id, input_id=entry["id"])
|
|
||||||
except InputCancelled:
|
|
||||||
# Cancellation can win after the snapshot, even during
|
|
||||||
# admission. The atomic claim proves no Run was started.
|
|
||||||
logger.info("Skipping cancelled queued input %s for Run %s", entry["id"], inputs.run_id)
|
|
||||||
continue
|
|
||||||
while run.status in ("pending", "running"):
|
|
||||||
# AgentRuns.wait retains its capped backoff and revocation
|
|
||||||
# checks; a long task exceeding a wait window is still live.
|
|
||||||
try:
|
|
||||||
run = await agent_runs.wait(thread_id=thread_id, run_id=run.run_id, timeout=60)
|
|
||||||
except TimeoutError:
|
|
||||||
continue
|
|
||||||
if run.status != "success" or run.stop_reason:
|
|
||||||
await inputs.pause()
|
|
||||||
return
|
|
||||||
except Exception:
|
|
||||||
# Unknown admission outcomes retain the original key and input;
|
|
||||||
# retrying can reconcile the same Run instead of running it twice.
|
|
||||||
logger.warning("Queued input remains undelivered for Run %s", inputs.run_id, exc_info=True)
|
|
||||||
return
|
|
||||||
|
|
||||||
|
|
||||||
async def dispatch_one_input(thread_id, run_id, input_id, request, *, retry=False):
|
|
||||||
from deerflow_extension_api.agent_runs import AgentRun
|
|
||||||
|
|
||||||
from app.gateway.routers.thread_runs import create_run, get_run
|
|
||||||
from app.gateway.run_models import RunCreateRequest
|
|
||||||
from app.gateway.services import resolve_thread_assistant_id
|
|
||||||
|
|
||||||
user_id = await get_current_user(request)
|
|
||||||
if user_id is None:
|
|
||||||
raise HTTPException(401, "Authentication required")
|
|
||||||
source = await _store(request).get(run_id, user_id=user_id)
|
|
||||||
if source is None or source["thread_id"] != thread_id:
|
|
||||||
raise HTTPException(404, "Run not found")
|
|
||||||
inputs = RunInputs(_store(request), run_id, user_id)
|
|
||||||
entry = await inputs.claim(input_id, retry=retry)
|
|
||||||
if entry.get("run_id"):
|
|
||||||
run = await get_run(thread_id, entry["run_id"], request)
|
|
||||||
return AgentRun(thread_id, run.run_id, run.status, run.assistant_id, run.stop_reason)
|
|
||||||
message_id = str(uuid5(NAMESPACE_URL, f"deerflow-dispatch:{inputs.run_id}:{input_id}"))
|
|
||||||
context = entry.get("context", {})
|
|
||||||
source_config = (source.get("kwargs") or {}).get("config") or {}
|
|
||||||
limit = source_config.get("recursion_limit")
|
|
||||||
config = {"recursion_limit": limit} if type(limit) is int else None
|
|
||||||
# Carry only the parent's admitted retrieval scope, never its files,
|
|
||||||
# credentials or framework markers. Normal admission rechecks access.
|
|
||||||
from deerflow.knowledge_scope import KNOWLEDGE_SCOPE_KEY
|
|
||||||
|
|
||||||
scope = None
|
|
||||||
for message in ((source.get("kwargs") or {}).get("input") or {}).get("messages", []):
|
|
||||||
if message.get("type", message.get("role")) in ("human", "user"):
|
|
||||||
scope = message.get("additional_kwargs", {}).get(KNOWLEDGE_SCOPE_KEY)
|
|
||||||
message = {"role": "user", "content": entry["text"], "id": message_id}
|
|
||||||
if scope is not None:
|
|
||||||
message["additional_kwargs"] = {KNOWLEDGE_SCOPE_KEY: copy.deepcopy(scope)}
|
|
||||||
if entry.get("source", "user") == "user":
|
|
||||||
# This fact comes from the stored, authenticated input, not from text
|
|
||||||
# or an arbitrary extension-supplied metadata flag.
|
|
||||||
request.state.run_origin = None
|
|
||||||
body = RunCreateRequest(assistant_id=await resolve_thread_assistant_id(request, thread_id, fail_closed=True), input={"messages": [message]}, context=context, config=config, on_disconnect="continue")
|
|
||||||
run = await create_run(thread_id, body, request, idempotency_key=f"run-input:{run_id}:{input_id}")
|
|
||||||
await inputs.dispatched(entry["id"], run.run_id)
|
|
||||||
return AgentRun(thread_id, run.run_id, run.status, run.assistant_id, run.stop_reason)
|
|
||||||
|
|
||||||
|
|
||||||
async def change_run_input(thread_id: str, run_id: str, input_id: str, request: Request, *, retry=False):
|
|
||||||
user_id = await get_current_user(request)
|
|
||||||
if user_id is None:
|
|
||||||
raise HTTPException(401, "Authentication required")
|
|
||||||
run = await get_run_manager(request).get(run_id, user_id=user_id)
|
|
||||||
if run is None or run.thread_id != thread_id:
|
|
||||||
raise HTTPException(404, "Run not found")
|
|
||||||
inputs = RunInputs(_store(request), run_id, user_id)
|
|
||||||
try:
|
|
||||||
if retry:
|
|
||||||
handle = get_agent_runs(request)
|
|
||||||
if handle is None:
|
|
||||||
raise HTTPException(503, "Run inputs require an authenticated session")
|
|
||||||
from deerflow.runtime.run_origin import admit_origin
|
|
||||||
|
|
||||||
if admit_origin(getattr(request.state, "run_origin", None)) is not None:
|
|
||||||
raise HTTPException(403, "An unapplied input requires an explicit user retry")
|
|
||||||
dispatched = await dispatch_one_input(thread_id, run_id, input_id, request, retry=True)
|
|
||||||
return {"run_id": dispatched.run_id}
|
|
||||||
await inputs.cancel(input_id)
|
|
||||||
return {"removed": True}
|
|
||||||
except InputConflict as exc:
|
|
||||||
raise HTTPException(409, str(exc)) from exc
|
|
||||||
@ -40,7 +40,7 @@ class RunCreateRequest(BaseModel):
|
|||||||
conversation_references: list[ConversationReference] = Field(
|
conversation_references: list[ConversationReference] = Field(
|
||||||
default_factory=list,
|
default_factory=list,
|
||||||
max_length=MAX_CONVERSATION_REFERENCES,
|
max_length=MAX_CONVERSATION_REFERENCES,
|
||||||
description="Explicit thread IDs or chat URLs on this DeerFlow host readable only during this run (opt-in read_conversation tool); SDK clients may send the same list as context.conversation_references",
|
description="Explicit thread IDs or same-origin chat URLs readable only during this run (opt-in read_conversation tool); SDK clients may send the same list as context.conversation_references",
|
||||||
)
|
)
|
||||||
webhook: None = Field(default=None, description="Compatibility placeholder; completion callbacks are not supported")
|
webhook: None = Field(default=None, description="Compatibility placeholder; completion callbacks are not supported")
|
||||||
checkpoint_id: str | None = Field(default=None, description="Resume from checkpoint")
|
checkpoint_id: str | None = Field(default=None, description="Resume from checkpoint")
|
||||||
|
|||||||
@ -26,6 +26,7 @@ from app.gateway.deps import get_config, get_local_provider, get_scheduled_task_
|
|||||||
from app.gateway.internal_auth import INTERNAL_SYSTEM_ROLE
|
from app.gateway.internal_auth import INTERNAL_SYSTEM_ROLE
|
||||||
from app.gateway.scheduled_task_errors import active_occurrence_error, repository_error, scheduler_error, tool_error_payload
|
from app.gateway.scheduled_task_errors import active_occurrence_error, repository_error, scheduler_error, tool_error_payload
|
||||||
from app.gateway.scheduled_task_validation import prepare_resume_updates, validate_scheduled_task_create, validate_scheduled_task_update
|
from app.gateway.scheduled_task_validation import prepare_resume_updates, validate_scheduled_task_create, validate_scheduled_task_update
|
||||||
|
from app.scheduler.service import TASK_CHANGED_ERROR
|
||||||
from deerflow.agents.interaction_policy import RunInteractionMode, RunInteractionPolicy
|
from deerflow.agents.interaction_policy import RunInteractionMode, RunInteractionPolicy
|
||||||
from deerflow.persistence.scheduled_tasks import ActiveScheduledTaskMutationConflict
|
from deerflow.persistence.scheduled_tasks import ActiveScheduledTaskMutationConflict
|
||||||
from deerflow.scheduler import host_notes
|
from deerflow.scheduler import host_notes
|
||||||
@ -61,6 +62,99 @@ _HOST_RUN_ERROR_KEYS = {
|
|||||||
_REASON_CODE = re.compile(r"[a-z_]+(:[a-z_]+)?")
|
_REASON_CODE = re.compile(r"[a-z_]+(:[a-z_]+)?")
|
||||||
_ZONE_ERRORS = (ValueError, KeyError, OSError, TypeError)
|
_ZONE_ERRORS = (ValueError, KeyError, OSError, TypeError)
|
||||||
|
|
||||||
|
# --- Trial consent -----------------------------------------------------------
|
||||||
|
|
||||||
|
_TRIAL_TRAILING_PUNCTUATION = ".!。！~～…"
|
||||||
|
_TRIAL_ACK_SEPARATORS = ",，、 "
|
||||||
|
_TRIAL_ACKS = tuple(sorted(("yes please", "yes", "sure", "ok", "okay", "please", "go ahead and", "alright", "好的", "好", "可以", "行", "嗯", "那就", "请", "麻烦"), key=len, reverse=True))
|
||||||
|
_TRIAL_PARTICLES = (" please", "吧", "呗", "啊", "呀", "哈", "看看")
|
||||||
|
_TRIAL_VERBS = ("run", "try", "test")
|
||||||
|
_TRIAL_TARGETS = ("it", "this task", "the task", "this")
|
||||||
|
_TRIAL_TIMINGS = ("now", "once", "once now", "right now", "now please")
|
||||||
|
_TRIAL_PHRASES = frozenset(
|
||||||
|
{f"{verb} {target} {timing}" for verb in _TRIAL_VERBS for target in _TRIAL_TARGETS for timing in _TRIAL_TIMINGS}
|
||||||
|
| {"run it", "try it", "run a trial", "run a trial now", "do a trial run", "do a test run", "do a trial run now", "do a test run now", "try it out", "test it now"}
|
||||||
|
| {
|
||||||
|
"先跑一次",
|
||||||
|
"现在跑一次",
|
||||||
|
"先试跑一次",
|
||||||
|
"现在试跑一次",
|
||||||
|
"先运行一次",
|
||||||
|
"现在运行一次",
|
||||||
|
"试跑一下",
|
||||||
|
"试跑一次",
|
||||||
|
"先试一下",
|
||||||
|
"先试试",
|
||||||
|
"试一下",
|
||||||
|
"试试",
|
||||||
|
"跑一次",
|
||||||
|
"跑一下",
|
||||||
|
"先跑一下",
|
||||||
|
"现在跑一下",
|
||||||
|
"马上跑一次",
|
||||||
|
"立即运行一次",
|
||||||
|
"运行一次",
|
||||||
|
"执行一次",
|
||||||
|
"先执行一次",
|
||||||
|
"现在执行一次",
|
||||||
|
"试运行一次",
|
||||||
|
"先试运行一次",
|
||||||
|
"现在试运行",
|
||||||
|
"立即试运行",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_trailing_punctuation(text: str) -> str:
|
||||||
|
return text.rstrip(_TRIAL_TRAILING_PUNCTUATION).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_acknowledgement(command: str) -> str | None:
|
||||||
|
"""Remove one leading "yes"/"好的"-style word; None when there is none."""
|
||||||
|
for ack in _TRIAL_ACKS:
|
||||||
|
if not command.startswith(ack):
|
||||||
|
continue
|
||||||
|
rest = command[len(ack) :]
|
||||||
|
# An English word must end at a separator ("okay" is not "ok" + "ay").
|
||||||
|
if ack.isascii() and rest and rest[0] not in _TRIAL_ACK_SEPARATORS:
|
||||||
|
continue
|
||||||
|
return rest.lstrip(_TRIAL_ACK_SEPARATORS)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _has_explicit_trial_request(user_text: str, task: Mapping[str, Any]) -> bool:
|
||||||
|
"""Accept a bounded direct-run request in the current turn, never a keyword in arbitrary text.
|
||||||
|
|
||||||
|
This is deliberately not a general intent parser: after removing up to
|
||||||
|
two leading acknowledgements and one trailing particle, the whole turn
|
||||||
|
must be one of the known direct requests. Bare confirmations, quoted,
|
||||||
|
conditional or compound turns, other tasks and task titles are rejected.
|
||||||
|
"""
|
||||||
|
if not isinstance(user_text, str) or not user_text.strip() or len(user_text) > 500:
|
||||||
|
return False
|
||||||
|
command = _strip_trailing_punctuation(" ".join(user_text.casefold().split()))
|
||||||
|
for _ in range(2):
|
||||||
|
rest = _strip_acknowledgement(command)
|
||||||
|
if rest is None:
|
||||||
|
break
|
||||||
|
command = rest
|
||||||
|
if not command:
|
||||||
|
return False
|
||||||
|
for particle in _TRIAL_PARTICLES:
|
||||||
|
if command.endswith(particle) and len(command) > len(particle):
|
||||||
|
command = _strip_trailing_punctuation(command[: -len(particle)])
|
||||||
|
break
|
||||||
|
phrases = set(_TRIAL_PHRASES)
|
||||||
|
# Titles are arbitrary prose and may contain quoted/conditional clauses.
|
||||||
|
# Only an opaque selector can safely be interpolated into a direct request.
|
||||||
|
task_id = task.get("id")
|
||||||
|
if isinstance(task_id, str) and 5 < len(task_id) <= 64 and task_id.startswith("task-") and task_id.isascii() and task_id.replace("-", "").replace("_", "").isalnum():
|
||||||
|
target = task_id.casefold()
|
||||||
|
phrases.update(f"{verb} {target} {timing}" for verb in _TRIAL_VERBS for timing in _TRIAL_TIMINGS)
|
||||||
|
phrases.update({f"先试跑{target}", f"现在试跑{target}", f"先运行{target}一次", f"现在运行{target}一次"})
|
||||||
|
return command in phrases
|
||||||
|
|
||||||
|
|
||||||
# --- Projections ---------------------------------------------------------------
|
# --- Projections ---------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
@ -451,15 +545,21 @@ class _SchedulerCapability:
|
|||||||
return self._result("note", task=await self._view(updated))
|
return self._result("note", task=await self._view(updated))
|
||||||
|
|
||||||
async def _trial(self, task: dict[str, Any]) -> dict[str, Any]:
|
async def _trial(self, task: dict[str, Any]) -> dict[str, Any]:
|
||||||
# Natural-language wording is not an execution authorization protocol.
|
if not _has_explicit_trial_request(self._original_user_text, task):
|
||||||
# The authenticated task-card/page button owns manual dispatch. This
|
raise scheduler_error(403, "trial_requires_direct_request", "A trial requires a direct request in the current user turn, such as 'Run this task now' or '先跑一次'; a task mention or bare confirmation is insufficient")
|
||||||
# capability has already checked owner, conversation and permissions.
|
result = await self._service.dispatch_task(task, now=datetime.now(UTC), trigger="manual")
|
||||||
return self._result(
|
outcome = result.get("outcome")
|
||||||
"trial",
|
error = result.get("error") or "Scheduled task trial could not be dispatched"
|
||||||
task=await self._view(task),
|
if outcome == "not_found":
|
||||||
requires_user_action="run_once_now",
|
raise scheduler_error(404, "task_not_found", error)
|
||||||
instruction="No trial has started. Use the task card's Run once now button, or open /workspace/scheduled-tasks and run this task there.",
|
if outcome == "conflict":
|
||||||
)
|
raise scheduler_error(409, "task_changed" if error == TASK_CHANGED_ERROR else "task_running", error)
|
||||||
|
if outcome == "failed":
|
||||||
|
raise scheduler_error(502, "trigger_failed", error)
|
||||||
|
current = await self._task_repo.get(task["id"], user_id=self._owner_id) or task
|
||||||
|
thread_id = result.get("thread_id")
|
||||||
|
trial = {"outcome": outcome, "existing": bool(result.get("existing")), "thread_id": thread_id if isinstance(thread_id, str) else None}
|
||||||
|
return self._result("trial", task=await self._view(current), trial=trial)
|
||||||
|
|
||||||
async def _pause_or_delete(self, action: str, task: dict[str, Any]) -> dict[str, Any]:
|
async def _pause_or_delete(self, action: str, task: dict[str, Any]) -> dict[str, Any]:
|
||||||
task_id = task["id"]
|
task_id = task["id"]
|
||||||
|
|||||||
@ -147,17 +147,10 @@ def validate_task_bounds(
|
|||||||
upcoming = _as_datetime(next_run_at)
|
upcoming = _as_datetime(next_run_at)
|
||||||
if end > now and upcoming is not None and end <= upcoming:
|
if end > now and upcoming is not None and end <= upcoming:
|
||||||
raise scheduler_error(422, "end_at_before_first_run", "end_at must allow at least one scheduled occurrence")
|
raise scheduler_error(422, "end_at_before_first_run", "end_at must allow at least one scheduled occurrence")
|
||||||
if require_limit_for_frequent and max_runs is None and end_at is None:
|
if require_limit_for_frequent and max_runs is None and end_at is None and is_frequent_schedule(schedule_type, schedule_spec):
|
||||||
if is_frequent_schedule(schedule_type, _require_schedule_object(schedule_spec)):
|
|
||||||
raise scheduler_error(422, "frequent_requires_limit", "Schedules more frequent than hourly require max_runs or end_at")
|
raise scheduler_error(422, "frequent_requires_limit", "Schedules more frequent than hourly require max_runs or end_at")
|
||||||
|
|
||||||
|
|
||||||
def _require_schedule_object(spec: Any) -> dict[str, Any]:
|
|
||||||
if not isinstance(spec, dict):
|
|
||||||
raise scheduler_error(422, "invalid_schedule", "schedule_spec must be an object")
|
|
||||||
return spec
|
|
||||||
|
|
||||||
|
|
||||||
def _normalized_stop_condition(value: Any) -> str | None:
|
def _normalized_stop_condition(value: Any) -> str | None:
|
||||||
if value is not None and not isinstance(value, str):
|
if value is not None and not isinstance(value, str):
|
||||||
raise scheduler_error(422, "invalid_stop_condition", "stop_condition must be a string", max_chars=MAX_STOP_CONDITION_CHARS)
|
raise scheduler_error(422, "invalid_stop_condition", "stop_condition must be a string", max_chars=MAX_STOP_CONDITION_CHARS)
|
||||||
@ -264,8 +257,7 @@ def _recompute_schedule(existing: dict[str, Any], updates: dict[str, Any], *, sc
|
|||||||
schedule_type = str(updates.get("schedule_type", existing["schedule_type"]))
|
schedule_type = str(updates.get("schedule_type", existing["schedule_type"]))
|
||||||
if schedule_type != existing["schedule_type"] and not isinstance(updates.get("schedule_spec"), dict):
|
if schedule_type != existing["schedule_type"] and not isinstance(updates.get("schedule_spec"), dict):
|
||||||
raise scheduler_error(422, "invalid_schedule", "changing schedule_type requires schedule_spec")
|
raise scheduler_error(422, "invalid_schedule", "changing schedule_type requires schedule_spec")
|
||||||
previous_spec = existing["schedule_spec"]
|
schedule_spec = dict(updates["schedule_spec"]) if isinstance(updates.get("schedule_spec"), dict) else dict(existing["schedule_spec"])
|
||||||
schedule_spec = dict(_require_schedule_object(updates.get("schedule_spec", previous_spec)))
|
|
||||||
timezone = str(updates.get("timezone", existing["timezone"]))
|
timezone = str(updates.get("timezone", existing["timezone"]))
|
||||||
try:
|
try:
|
||||||
if schedule_type == "once":
|
if schedule_type == "once":
|
||||||
@ -273,8 +265,7 @@ def _recompute_schedule(existing: dict[str, Any], updates: dict[str, Any], *, sc
|
|||||||
# definitions and terminal reactivation must pass today's validation.
|
# definitions and terminal reactivation must pass today's validation.
|
||||||
unchanged_definition = (
|
unchanged_definition = (
|
||||||
existing["schedule_type"] == "once"
|
existing["schedule_type"] == "once"
|
||||||
and isinstance(previous_spec, dict)
|
and schedule_spec.get("run_at") == existing["schedule_spec"].get("run_at")
|
||||||
and schedule_spec.get("run_at") == previous_spec.get("run_at")
|
|
||||||
and timezone == existing["timezone"]
|
and timezone == existing["timezone"]
|
||||||
and existing["status"] not in TERMINAL_TASK_STATUSES
|
and existing["status"] not in TERMINAL_TASK_STATUSES
|
||||||
and existing.get("next_run_at") is not None
|
and existing.get("next_run_at") is not None
|
||||||
@ -288,12 +279,10 @@ def _recompute_schedule(existing: dict[str, Any], updates: dict[str, Any], *, sc
|
|||||||
schedule_spec["cron"] = normalize_cron_expression(raw_cron)
|
schedule_spec["cron"] = normalize_cron_expression(raw_cron)
|
||||||
if schedule_type == "interval":
|
if schedule_type == "interval":
|
||||||
every_seconds = validate_interval_seconds(schedule_spec, scheduler_config.min_once_delay_seconds)
|
every_seconds = validate_interval_seconds(schedule_spec, scheduler_config.min_once_delay_seconds)
|
||||||
previous_seconds = None
|
|
||||||
if existing["schedule_type"] == "interval" and isinstance(previous_spec, dict):
|
|
||||||
try:
|
try:
|
||||||
previous_seconds = parse_interval_seconds(previous_spec)
|
previous_seconds = parse_interval_seconds(dict(existing["schedule_spec"])) if existing["schedule_type"] == "interval" else None
|
||||||
except ValueError:
|
except ValueError:
|
||||||
pass
|
previous_seconds = None
|
||||||
# Re-saving an unchanged cadence keeps the next occurrence.
|
# Re-saving an unchanged cadence keeps the next occurrence.
|
||||||
if previous_seconds == every_seconds and existing.get("next_run_at") is not None:
|
if previous_seconds == every_seconds and existing.get("next_run_at") is not None:
|
||||||
upcoming = existing["next_run_at"]
|
upcoming = existing["next_run_at"]
|
||||||
@ -390,11 +379,10 @@ def prepare_resume_updates(existing: dict[str, Any], renewal: dict[str, Any], *,
|
|||||||
if existing.get("status") == "enabled":
|
if existing.get("status") == "enabled":
|
||||||
return {}
|
return {}
|
||||||
renewal = {key: value for key, value in renewal.items() if key in {"max_runs", "end_at"}}
|
renewal = {key: value for key, value in renewal.items() if key in {"max_runs", "end_at"}}
|
||||||
schedule_spec = _require_schedule_object(existing["schedule_spec"])
|
|
||||||
upcoming = _as_datetime(existing.get("next_run_at"))
|
upcoming = _as_datetime(existing.get("next_run_at"))
|
||||||
if upcoming is None or upcoming <= now:
|
if upcoming is None or upcoming <= now:
|
||||||
try:
|
try:
|
||||||
upcoming = next_run_at(existing["schedule_type"], schedule_spec, existing["timezone"], now=now)
|
upcoming = next_run_at(existing["schedule_type"], existing["schedule_spec"], existing["timezone"], now=now)
|
||||||
except (ValueError, OverflowError) as exc:
|
except (ValueError, OverflowError) as exc:
|
||||||
raise scheduler_error(422, "invalid_schedule", str(exc)) from exc
|
raise scheduler_error(422, "invalid_schedule", str(exc)) from exc
|
||||||
if upcoming is None:
|
if upcoming is None:
|
||||||
|
|||||||
@ -1,172 +0,0 @@
|
|||||||
"""Permission-scoped transcript projection shared by search and message navigation."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
|
|
||||||
from deerflow_extension_api.search import SearchMessage, SearchMessagePage, SearchThread
|
|
||||||
|
|
||||||
from app.gateway.conversation_access import _visible_text
|
|
||||||
from app.gateway.conversation_reader import default_history_hidden_run_ids, read_visible_message_page
|
|
||||||
from deerflow.runtime.events.message_identity import message_identity
|
|
||||||
from deerflow.utils.time import coerce_iso
|
|
||||||
|
|
||||||
|
|
||||||
def visible_search_text(row: dict, *, thread_id: str):
|
|
||||||
"""Use the transcript reader and the chat UI's display-only exclusions."""
|
|
||||||
content = row.get("content")
|
|
||||||
if isinstance(content, dict) and content.get("name") in {"summary", "loop_warning", "todo_reminder", "todo_completion_reminder"}:
|
|
||||||
return None
|
|
||||||
visible = _visible_text(row, thread_id=thread_id)
|
|
||||||
if visible and visible[0] == "user":
|
|
||||||
text = re.sub(r"<(current_uploads|uploaded_files|slash_skill_activation)>[\s\S]*?</\1>", "", visible[1]).strip()
|
|
||||||
return ("user", text) if text else None
|
|
||||||
return visible
|
|
||||||
|
|
||||||
|
|
||||||
class GatewaySearchCorpus:
|
|
||||||
def __init__(self, thread_store, event_store, run_manager, user_id: str, *, request=None):
|
|
||||||
self.thread_store = thread_store
|
|
||||||
self.event_store = event_store
|
|
||||||
self.run_manager = run_manager
|
|
||||||
self.user_id = user_id
|
|
||||||
self.request = request
|
|
||||||
|
|
||||||
async def threads(self) -> tuple[SearchThread, ...]:
|
|
||||||
result = []
|
|
||||||
offset = 0
|
|
||||||
while True:
|
|
||||||
rows = await self.thread_store.search(user_id=self.user_id, limit=200, offset=offset)
|
|
||||||
for row in rows:
|
|
||||||
metadata = row.get("metadata") or {}
|
|
||||||
if metadata.get("deerflow_sidecar") is True:
|
|
||||||
continue
|
|
||||||
result.append(self.project_thread(row))
|
|
||||||
offset += len(rows)
|
|
||||||
if len(rows) < 200:
|
|
||||||
return tuple(result)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def project_thread(row) -> SearchThread:
|
|
||||||
from app.gateway.services import custom_agent_name
|
|
||||||
|
|
||||||
try:
|
|
||||||
agent_name = custom_agent_name(row.get("assistant_id"))
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
agent_name = None
|
|
||||||
metadata = row.get("metadata") or {}
|
|
||||||
return SearchThread(
|
|
||||||
thread_id=row["thread_id"],
|
|
||||||
incarnation=str(row.get("incarnation") or row.get("created_at") or ""),
|
|
||||||
title=row.get("display_name") or "",
|
|
||||||
updated_at=coerce_iso(row.get("updated_at", "")),
|
|
||||||
agent_name=agent_name,
|
|
||||||
project_id=metadata.get("deerflow_project_id"),
|
|
||||||
archived=metadata.get("deerflow_archived") is True,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def thread(self, thread_id: str) -> SearchThread | None:
|
|
||||||
row = await self.thread_store.get(thread_id, user_id=self.user_id)
|
|
||||||
if row is None or (row.get("metadata") or {}).get("deerflow_sidecar") is True:
|
|
||||||
return None
|
|
||||||
return self.project_thread(row)
|
|
||||||
|
|
||||||
async def seed_legacy_history(self, thread: SearchThread) -> None:
|
|
||||||
if self.request is None or await self.event_store.list_messages(thread.thread_id, limit=1, user_id=None):
|
|
||||||
return
|
|
||||||
row = await self.thread_store.get(thread.thread_id, user_id=self.user_id)
|
|
||||||
if row is None or self.project_thread(row).incarnation != thread.incarnation:
|
|
||||||
return
|
|
||||||
from app.gateway.services import ensure_checkpoint_history_seeded
|
|
||||||
|
|
||||||
await ensure_checkpoint_history_seeded(self.request, thread_id=thread.thread_id, assistant_id=row.get("assistant_id"))
|
|
||||||
|
|
||||||
async def owns(self, thread: SearchThread) -> bool:
|
|
||||||
row = await self.thread_store.get(thread.thread_id, user_id=self.user_id)
|
|
||||||
return bool(row and str(row.get("incarnation") or row.get("created_at") or "") == thread.incarnation and (row.get("metadata") or {}).get("deerflow_sidecar") is not True)
|
|
||||||
|
|
||||||
async def revision(self, thread: SearchThread) -> str | None:
|
|
||||||
if not await self.owns(thread):
|
|
||||||
return None
|
|
||||||
await self.seed_legacy_history(thread)
|
|
||||||
latest = await self.event_store.list_messages(thread.thread_id, limit=1, user_id=self.user_id)
|
|
||||||
hidden = await default_history_hidden_run_ids(self.run_manager, thread.thread_id, user_id=self.user_id)
|
|
||||||
revision = [thread.incarnation, thread.updated_at, latest[-1]["seq"] if latest else 0, sorted(hidden)]
|
|
||||||
return hashlib.sha256(json.dumps(revision).encode()).hexdigest()
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def project(thread: SearchThread, row: dict) -> SearchMessage | None:
|
|
||||||
visible = visible_search_text(row, thread_id=thread.thread_id)
|
|
||||||
content = row.get("content")
|
|
||||||
identity = message_identity(content) if isinstance(content, dict) else None
|
|
||||||
if not visible or not identity:
|
|
||||||
return None
|
|
||||||
return SearchMessage(thread.thread_id, thread.incarnation, identity.removeprefix("message:"), row["run_id"], row["seq"], visible[0], visible[1], coerce_iso(row.get("created_at", "")))
|
|
||||||
|
|
||||||
async def messages(self, thread: SearchThread, *, before_seq: int | None) -> SearchMessagePage:
|
|
||||||
if not await self.owns(thread):
|
|
||||||
return SearchMessagePage()
|
|
||||||
rows, more = await self._page(thread.thread_id, limit=200, before_seq=before_seq)
|
|
||||||
items = tuple(message for row in rows if (message := self.project(thread, row)) is not None)
|
|
||||||
return SearchMessagePage(items, rows[0]["seq"] if more else None)
|
|
||||||
|
|
||||||
async def _page(self, thread_id: str, *, limit: int, before_seq=None, after_seq=None):
|
|
||||||
return await read_visible_message_page(
|
|
||||||
event_store=self.event_store,
|
|
||||||
run_manager=self.run_manager,
|
|
||||||
thread_id=thread_id,
|
|
||||||
user_id=self.user_id,
|
|
||||||
limit=limit,
|
|
||||||
before_seq=before_seq,
|
|
||||||
after_seq=after_seq,
|
|
||||||
message_filter=lambda row: visible_search_text(row, thread_id=thread_id) is not None and message_identity(row.get("content", {})) is not None,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def around(self, thread_id: str, message_id: str, *, limit: int = 20, seq: int | None = None) -> dict | None:
|
|
||||||
row = await self.thread_store.get(thread_id, user_id=self.user_id)
|
|
||||||
if row is None or (row.get("metadata") or {}).get("deerflow_sidecar") is True:
|
|
||||||
return None
|
|
||||||
await self.seed_legacy_history(self.project_thread(row))
|
|
||||||
# Walk backward from the head so later versions win even when they
|
|
||||||
# live on another storage page. Sequence hints never select old text.
|
|
||||||
rows = await self.latest_rows(thread_id)
|
|
||||||
index = next((i for i, item in enumerate(rows) if message_identity(item["content"]) == f"message:{message_id}"), None)
|
|
||||||
if index is None or (seq is not None and rows[index]["seq"] != seq):
|
|
||||||
return None
|
|
||||||
target = rows[index]
|
|
||||||
seq = target["seq"]
|
|
||||||
left, right = max(0, index - limit), min(len(rows), index + limit + 1)
|
|
||||||
older, newer = rows[left:index], rows[index + 1 : right]
|
|
||||||
has_before, has_after = left > 0, right < len(rows)
|
|
||||||
# Recheck ownership/incarnation after the asynchronous history reads.
|
|
||||||
current = await self.thread_store.get(thread_id, user_id=self.user_id)
|
|
||||||
if current is None or current.get("incarnation") != row.get("incarnation"):
|
|
||||||
return None
|
|
||||||
return {"data": older + [target] + newer, "target_seq": seq, "has_before": has_before, "has_after": has_after}
|
|
||||||
|
|
||||||
async def latest_rows(self, thread_id: str) -> list[dict]:
|
|
||||||
latest = {}
|
|
||||||
before = None
|
|
||||||
while True:
|
|
||||||
rows, more = await self._page(thread_id, limit=200, before_seq=before)
|
|
||||||
for row in reversed(rows):
|
|
||||||
latest.setdefault(message_identity(row["content"]), row)
|
|
||||||
if not more:
|
|
||||||
break
|
|
||||||
before = rows[0]["seq"]
|
|
||||||
return sorted(latest.values(), key=lambda row: row["seq"])
|
|
||||||
|
|
||||||
async def latest_message(self, thread: SearchThread, message_id: str) -> SearchMessage | None:
|
|
||||||
if not await self.owns(thread):
|
|
||||||
return None
|
|
||||||
before = None
|
|
||||||
while True:
|
|
||||||
rows, more = await self._page(thread.thread_id, limit=200, before_seq=before)
|
|
||||||
for row in reversed(rows):
|
|
||||||
if message_identity(row["content"]) == f"message:{message_id}":
|
|
||||||
return self.project(thread, row)
|
|
||||||
if not more:
|
|
||||||
return None
|
|
||||||
before = rows[0]["seq"]
|
|
||||||
@ -1,30 +0,0 @@
|
|||||||
"""Shared masking for credential values that Gateway routes echo back from config."""
|
|
||||||
|
|
||||||
import re
|
|
||||||
from collections.abc import Callable
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
MASKED_VALUE = "***"
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_config_key(key: str) -> str:
|
|
||||||
"""Snake-case a config key so ``apiKey``, ``API-Key`` and ``api_key`` compare equal."""
|
|
||||||
with_boundaries = re.sub(r"(.)([A-Z][a-z]+)", r"\1_\2", key)
|
|
||||||
with_boundaries = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", with_boundaries)
|
|
||||||
return re.sub(r"[^a-z0-9]+", "_", with_boundaries.lower()).strip("_")
|
|
||||||
|
|
||||||
|
|
||||||
def mask_secret_values(value: Any, is_secret_key: Callable[[str], bool], *, keep_unset: bool = False) -> Any:
|
|
||||||
"""Return a copy of ``value`` with every value under a secret-named key masked, at any depth.
|
|
||||||
|
|
||||||
A secret-named key masks its whole value, containers included: a
|
|
||||||
``credentials`` mapping can hold keys under arbitrary names, so masking only
|
|
||||||
its secret-named leaves could pass one through. With ``keep_unset``, a
|
|
||||||
``None`` or ``""`` value stays as it is, so a response still shows that no
|
|
||||||
secret is configured.
|
|
||||||
"""
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return {key: MASKED_VALUE if is_secret_key(str(key)) and not (keep_unset and nested in (None, "")) else mask_secret_values(nested, is_secret_key, keep_unset=keep_unset) for key, nested in value.items()}
|
|
||||||
if isinstance(value, list):
|
|
||||||
return [mask_secret_values(item, is_secret_key, keep_unset=keep_unset) for item in value]
|
|
||||||
return value
|
|
||||||
@ -30,8 +30,7 @@ from langgraph.types import Command
|
|||||||
|
|
||||||
from app.gateway.auth_disabled import AUTH_DISABLED_USER_ID, AUTH_SOURCE_INTERNAL, get_auth_disabled_user, is_auth_disabled
|
from app.gateway.auth_disabled import AUTH_DISABLED_USER_ID, AUTH_SOURCE_INTERNAL, get_auth_disabled_user, is_auth_disabled
|
||||||
from app.gateway.authz import require_cancel_permission_if, resolve_route_permissions
|
from app.gateway.authz import require_cancel_permission_if, resolve_route_permissions
|
||||||
from app.gateway.deps import get_checkpointer, get_local_provider, get_run_context, get_run_manager, get_run_store, get_stream_bridge
|
from app.gateway.deps import get_checkpointer, get_local_provider, get_run_context, get_run_manager, get_stream_bridge
|
||||||
from app.gateway.history_seed import reserve_history_seed, write_history_seed
|
|
||||||
from app.gateway.internal_auth import (
|
from app.gateway.internal_auth import (
|
||||||
INTERNAL_OWNER_USER_ID_HEADER_NAME,
|
INTERNAL_OWNER_USER_ID_HEADER_NAME,
|
||||||
INTERNAL_SYSTEM_ROLE,
|
INTERNAL_SYSTEM_ROLE,
|
||||||
@ -55,7 +54,7 @@ from deerflow.agents.middlewares.tool_receipt import TOOL_RECEIPT_KEY, TOOL_RECE
|
|||||||
from deerflow.agents.middlewares.tool_transform_meta import TOOL_TRANSFORMS_KEY
|
from deerflow.agents.middlewares.tool_transform_meta import TOOL_TRANSFORMS_KEY
|
||||||
from deerflow.agents.middlewares.view_image_middleware import _IMAGE_CONTEXT_MESSAGE_MARKER_KEY
|
from deerflow.agents.middlewares.view_image_middleware import _IMAGE_CONTEXT_MESSAGE_MARKER_KEY
|
||||||
from deerflow.config.agents_config import load_agent_config
|
from deerflow.config.agents_config import load_agent_config
|
||||||
from deerflow.config.app_config import AppConfig, get_app_config
|
from deerflow.config.app_config import get_app_config
|
||||||
from deerflow.config.database_config import resolve_checkpoint_graph_cache_max
|
from deerflow.config.database_config import resolve_checkpoint_graph_cache_max
|
||||||
from deerflow.knowledge_scope import KNOWLEDGE_SCOPE_KEY, KNOWLEDGE_SCOPE_RUNTIME_KEY
|
from deerflow.knowledge_scope import KNOWLEDGE_SCOPE_KEY, KNOWLEDGE_SCOPE_RUNTIME_KEY
|
||||||
from deerflow.mcp_scope import (
|
from deerflow.mcp_scope import (
|
||||||
@ -92,7 +91,7 @@ from deerflow.runtime.checkpoint_mode import (
|
|||||||
)
|
)
|
||||||
from deerflow.runtime.checkpoint_state import graph_state_schema
|
from deerflow.runtime.checkpoint_state import graph_state_schema
|
||||||
from deerflow.runtime.client_context import sanitize_client_context
|
from deerflow.runtime.client_context import sanitize_client_context
|
||||||
from deerflow.runtime.context_keys import AGENT_AUTH_CONTEXT_KEY, PROJECT_CONTEXT_KEY
|
from deerflow.runtime.context_keys import PROJECT_CONTEXT_KEY
|
||||||
from deerflow.runtime.events.message_identity import MESSAGE_SEQ_KEY
|
from deerflow.runtime.events.message_identity import MESSAGE_SEQ_KEY
|
||||||
from deerflow.runtime.goal import goal_thread_lock
|
from deerflow.runtime.goal import goal_thread_lock
|
||||||
from deerflow.runtime.journal import build_checkpoint_history_seed_events
|
from deerflow.runtime.journal import build_checkpoint_history_seed_events
|
||||||
@ -106,7 +105,7 @@ from deerflow.runtime.secret_context import (
|
|||||||
validate_run_metadata_secrets,
|
validate_run_metadata_secrets,
|
||||||
)
|
)
|
||||||
from deerflow.runtime.stream_modes import normalize_stream_modes
|
from deerflow.runtime.stream_modes import normalize_stream_modes
|
||||||
from deerflow.runtime.user_context import get_current_user, reset_current_user, set_current_user
|
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||||
from deerflow.sandbox.lease import SANDBOX_SERVER_OWNED_CONTEXT_KEYS
|
from deerflow.sandbox.lease import SANDBOX_SERVER_OWNED_CONTEXT_KEYS
|
||||||
from deerflow.scheduler.runtime import scheduler_tools_enabled
|
from deerflow.scheduler.runtime import scheduler_tools_enabled
|
||||||
from deerflow.scheduler.schedules import validate_timezone
|
from deerflow.scheduler.schedules import validate_timezone
|
||||||
@ -158,7 +157,6 @@ async def reserve_checkpoint_write(
|
|||||||
|
|
||||||
_TERMINAL_RUN_STATUSES = {
|
_TERMINAL_RUN_STATUSES = {
|
||||||
RunStatus.success,
|
RunStatus.success,
|
||||||
RunStatus.waiting_for_input,
|
|
||||||
RunStatus.error,
|
RunStatus.error,
|
||||||
RunStatus.timeout,
|
RunStatus.timeout,
|
||||||
RunStatus.interrupted,
|
RunStatus.interrupted,
|
||||||
@ -166,8 +164,6 @@ _TERMINAL_RUN_STATUSES = {
|
|||||||
_TERMINAL_STREAM_RECOVERED = "recovered"
|
_TERMINAL_STREAM_RECOVERED = "recovered"
|
||||||
_TERMINAL_STREAM_PRODUCER_EXPIRED = "producer_expired"
|
_TERMINAL_STREAM_PRODUCER_EXPIRED = "producer_expired"
|
||||||
|
|
||||||
# Includes same-thread create admission and metadata I/O; timeout rejects strict
|
|
||||||
# starts and leaves non-strict starts without verified metadata (MCP fails closed).
|
|
||||||
_THREAD_METADATA_SETUP_TIMEOUT_SECONDS = 5.0
|
_THREAD_METADATA_SETUP_TIMEOUT_SECONDS = 5.0
|
||||||
|
|
||||||
# Message metadata a scheduled launch puts on its prompt: the task, run and the
|
# Message metadata a scheduled launch puts on its prompt: the task, run and the
|
||||||
@ -204,10 +200,6 @@ _SERVER_OWNED_MESSAGE_METADATA_KEYS = (
|
|||||||
SUBAGENT_RECEIPT_VERDICT_KEY,
|
SUBAGENT_RECEIPT_VERDICT_KEY,
|
||||||
SUBAGENT_ACCEPTANCE_VERDICT_KEY,
|
SUBAGENT_ACCEPTANCE_VERDICT_KEY,
|
||||||
UNTRUSTED_INPUT_KEY,
|
UNTRUSTED_INPUT_KEY,
|
||||||
"run_input_id",
|
|
||||||
"run_input_source",
|
|
||||||
"origin_task_id",
|
|
||||||
"subagent_pending",
|
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
| PROVENANCE_KEYS
|
| PROVENANCE_KEYS
|
||||||
@ -269,18 +261,6 @@ async def _ensure_thread_metadata(
|
|||||||
require_existing_thread: bool = False,
|
require_existing_thread: bool = False,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""Ensure an admitted run's thread exists without delaying task attachment."""
|
"""Ensure an admitted run's thread exists without delaying task attachment."""
|
||||||
async with goal_thread_lock(record.thread_id):
|
|
||||||
return await _ensure_thread_metadata_locked(run_ctx, record, owner_user_id=owner_user_id, require_existing_thread=require_existing_thread)
|
|
||||||
|
|
||||||
|
|
||||||
async def _ensure_thread_metadata_locked(
|
|
||||||
run_ctx: RunContext,
|
|
||||||
record: RunRecord,
|
|
||||||
*,
|
|
||||||
owner_user_id: str | None,
|
|
||||||
require_existing_thread: bool,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
"""Share create admission with HTTP while retaining detached task ownership."""
|
|
||||||
thread_store = run_ctx.thread_store
|
thread_store = run_ctx.thread_store
|
||||||
existing = await thread_store.get(record.thread_id)
|
existing = await thread_store.get(record.thread_id)
|
||||||
if existing is None and owner_user_id:
|
if existing is None and owner_user_id:
|
||||||
@ -570,17 +550,15 @@ def _strip_external_metadata_from_message_like(item: Any) -> Any:
|
|||||||
# unmarked ``name="summary"`` would reach the model on the guardrail's
|
# unmarked ``name="summary"`` would reach the model on the guardrail's
|
||||||
# genuine-user fallback. Treat it as empty for both steps rather than
|
# genuine-user fallback. Treat it as empty for both steps rather than
|
||||||
# returning early.
|
# returning early.
|
||||||
raw_kwargs = item.get("additional_kwargs")
|
source_kwargs = item.get("additional_kwargs")
|
||||||
source_kwargs = raw_kwargs if isinstance(raw_kwargs, dict) else {}
|
source_kwargs = source_kwargs if isinstance(source_kwargs, dict) else {}
|
||||||
additional_kwargs = {key: value for key, value in source_kwargs.items() if key not in _SERVER_OWNED_MESSAGE_METADATA_KEYS and key != ORIGINAL_USER_CONTENT_KEY}
|
additional_kwargs = {key: value for key, value in source_kwargs.items() if key not in _SERVER_OWNED_MESSAGE_METADATA_KEYS and key != ORIGINAL_USER_CONTENT_KEY}
|
||||||
if _is_human_message_like(item):
|
if _is_human_message_like(item):
|
||||||
additional_kwargs = _mark_untrusted_framework_markers(additional_kwargs, item.get("name"))
|
additional_kwargs = _mark_untrusted_framework_markers(additional_kwargs, item.get("name"))
|
||||||
if additional_kwargs == raw_kwargs:
|
if additional_kwargs == source_kwargs:
|
||||||
# Nothing to change — including the ordinary key-omitted message, which
|
# Nothing to change — including the ordinary key-omitted message, which
|
||||||
# must not gain an empty dict just by passing through here.
|
# must not gain an empty dict just by passing through here.
|
||||||
return item
|
return item
|
||||||
if "additional_kwargs" not in item and not additional_kwargs:
|
|
||||||
return item
|
|
||||||
return {**item, "additional_kwargs": additional_kwargs}
|
return {**item, "additional_kwargs": additional_kwargs}
|
||||||
|
|
||||||
|
|
||||||
@ -643,132 +621,7 @@ def _normalize_input_messages(
|
|||||||
|
|
||||||
#: State channels only the server writes. ``goal_outcome`` is the "goal met"
|
#: State channels only the server writes. ``goal_outcome`` is the "goal met"
|
||||||
#: record that clearing a satisfied goal writes; a caller copy would forge one.
|
#: record that clearing a satisfied goal writes; a caller copy would forge one.
|
||||||
SERVER_OWNED_STATE_CHANNELS = frozenset({"sandbox", "thread_data", "viewed_images", "goal_outcome", "applied_run_inputs"})
|
SERVER_OWNED_STATE_CHANNELS = frozenset({"sandbox", "thread_data", "viewed_images", "goal_outcome"})
|
||||||
|
|
||||||
#: The ``write_todos`` status contract the todo middleware renders.
|
|
||||||
_TODO_STATUSES = frozenset({"pending", "in_progress", "completed"})
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_external_todos(value: Any, *, location: str) -> list[dict[str, Any]] | None:
|
|
||||||
"""Reject todo lists the middleware cannot render.
|
|
||||||
|
|
||||||
``todos`` is a reducer channel a caller may Overwrite directly, and the todo
|
|
||||||
middleware reads it on every model call. A value that is not a list of
|
|
||||||
objects with string content would fail those calls on every turn instead of
|
|
||||||
being refused here, so validate the shape at the boundary the same way
|
|
||||||
``goal`` is.
|
|
||||||
"""
|
|
||||||
if value is None:
|
|
||||||
return None
|
|
||||||
if not isinstance(value, list):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location} must be a list of todo objects")
|
|
||||||
normalized: list[dict[str, Any]] = []
|
|
||||||
for index, entry in enumerate(value):
|
|
||||||
if not isinstance(entry, Mapping):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}[{index}] must be an object")
|
|
||||||
content = entry.get("content")
|
|
||||||
if not isinstance(content, str):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}[{index}].content must be a string")
|
|
||||||
status = entry.get("status", "pending")
|
|
||||||
if not isinstance(status, str) or status not in _TODO_STATUSES:
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}[{index}].status must be pending, in_progress, or completed")
|
|
||||||
normalized.append({**entry, "content": content, "status": status})
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_external_skill_context(value: Any, *, location: str) -> list[dict[str, Any]] | None:
|
|
||||||
"""Reject skill references the durable-context reducer cannot read.
|
|
||||||
|
|
||||||
``skill_context`` is a checkpointed reducer channel a caller may Overwrite.
|
|
||||||
``merge_skill_context`` normalizes each entry with ``entry.get(...)`` and
|
|
||||||
``str(entry["path"])``, so a non-object entry or a missing path raises while
|
|
||||||
the state update merges — failing the run with an unhandled
|
|
||||||
``AttributeError``/``KeyError`` instead of a 400 at this boundary. The other
|
|
||||||
fields are optional: the reducer coerces ``name``/``description`` and
|
|
||||||
``loaded_at`` itself, so only the path needs to be usable here.
|
|
||||||
"""
|
|
||||||
if value is None:
|
|
||||||
return None
|
|
||||||
if not isinstance(value, list):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location} must be a list of skill reference objects")
|
|
||||||
normalized: list[dict[str, Any]] = []
|
|
||||||
for index, entry in enumerate(value):
|
|
||||||
if not isinstance(entry, Mapping):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}[{index}] must be an object")
|
|
||||||
path = entry.get("path")
|
|
||||||
if not isinstance(path, str) or not path:
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}[{index}].path must be a non-empty string")
|
|
||||||
normalized.append({**entry, "name": str(entry.get("name") or "")})
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
|
|
||||||
def _reject_overwrite_marker(value: Mapping[str, Any], *, location: str) -> None:
|
|
||||||
"""Refuse a value LangGraph would apply with ``Overwrite`` semantics.
|
|
||||||
|
|
||||||
``Overwrite`` reaches the validator in two dict spellings: the sentinel form
|
|
||||||
``{"__overwrite__": value}`` and the dataclass-erased form an orjson round
|
|
||||||
trip produces, ``{"type": "__overwrite__", "value": value}``. Either one
|
|
||||||
bypasses the channel reducer and stores the caller's value directly on an
|
|
||||||
initialized channel, so a shape check alone cannot catch it.
|
|
||||||
"""
|
|
||||||
if "__overwrite__" in value or value.get("type") == "__overwrite__":
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location} cannot contain an Overwrite marker")
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_external_promoted(value: Any, *, location: str) -> dict[str, Any] | None:
|
|
||||||
"""Reject deferred-tool promotion records the reducer cannot merge.
|
|
||||||
|
|
||||||
``promoted`` is a checkpointed reducer channel a caller may Overwrite even
|
|
||||||
though only the server writes it. ``merge_promoted`` calls ``new.get(...)``
|
|
||||||
and indexes ``new["catalog_hash"]``/``new["names"]``, so a string, a missing
|
|
||||||
key or a non-list ``names`` raises while the state update merges — failing
|
|
||||||
the run with an unhandled ``AttributeError``/``KeyError``/``TypeError``
|
|
||||||
instead of a 400 at this boundary. On the thread-state route the bad shape
|
|
||||||
would also be persisted, so every later promotion write on that thread would
|
|
||||||
keep failing on it.
|
|
||||||
"""
|
|
||||||
if value is None:
|
|
||||||
return None
|
|
||||||
if not isinstance(value, Mapping):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location} must be an object with a catalog_hash and a names list")
|
|
||||||
_reject_overwrite_marker(value, location=location)
|
|
||||||
catalog_hash = value.get("catalog_hash")
|
|
||||||
if not isinstance(catalog_hash, str) or not catalog_hash:
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}.catalog_hash must be a non-empty string")
|
|
||||||
names = value.get("names")
|
|
||||||
if not isinstance(names, list) or not all(isinstance(name, str) for name in names):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}.names must be a list of strings")
|
|
||||||
turn_id = value.get("turn_id")
|
|
||||||
if turn_id is not None and not isinstance(turn_id, str):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}.turn_id must be a string")
|
|
||||||
return {**value, "catalog_hash": catalog_hash, "names": names}
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_external_tool_artifacts(value: Any, *, location: str) -> list[dict[str, Any]] | None:
|
|
||||||
"""Reject artifact-registry entries the reducer cannot key.
|
|
||||||
|
|
||||||
``tool_artifacts`` is a checkpointed reducer channel a caller may Overwrite.
|
|
||||||
``merge_tool_artifacts`` keys each entry by its ``handle``, so an entry whose
|
|
||||||
handle is missing or unhashable raises while the state update merges — a 500
|
|
||||||
instead of a 400 at this boundary. The thread-state route persists the value
|
|
||||||
with ``Overwrite`` before any reducer sees it, so the thread stays broken for
|
|
||||||
every later artifact capture. Require the registry's list-of-objects shape
|
|
||||||
with a non-empty string handle; the reducer also tolerates an already-poisoned
|
|
||||||
checkpoint.
|
|
||||||
"""
|
|
||||||
if value is None:
|
|
||||||
return None
|
|
||||||
if not isinstance(value, list):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location} must be a list of artifact objects")
|
|
||||||
normalized: list[dict[str, Any]] = []
|
|
||||||
for index, entry in enumerate(value):
|
|
||||||
if not isinstance(entry, Mapping):
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}[{index}] must be an object")
|
|
||||||
handle = entry.get("handle")
|
|
||||||
if not isinstance(handle, str) or not handle:
|
|
||||||
raise HTTPException(status_code=400, detail=f"{location}[{index}].handle must be a non-empty string")
|
|
||||||
normalized.append(dict(entry))
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_external_goal(value: Any, *, location: str) -> dict[str, Any] | None:
|
def _normalize_external_goal(value: Any, *, location: str) -> dict[str, Any] | None:
|
||||||
@ -777,7 +630,8 @@ def _normalize_external_goal(value: Any, *, location: str) -> dict[str, Any] | N
|
|||||||
return None
|
return None
|
||||||
if not isinstance(value, Mapping) or not isinstance(value.get("objective"), str):
|
if not isinstance(value, Mapping) or not isinstance(value.get("objective"), str):
|
||||||
raise HTTPException(status_code=400, detail=f"{location} must be an object with a string objective")
|
raise HTTPException(status_code=400, detail=f"{location} must be an object with a string objective")
|
||||||
_reject_overwrite_marker(value, location=location)
|
if "__overwrite__" in value or value.get("type") == "__overwrite__":
|
||||||
|
raise HTTPException(status_code=400, detail=f"{location} cannot contain an Overwrite marker")
|
||||||
try:
|
try:
|
||||||
objective = normalize_goal_objective(value["objective"])
|
objective = normalize_goal_objective(value["objective"])
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
@ -785,7 +639,7 @@ def _normalize_external_goal(value: Any, *, location: str) -> dict[str, Any] | N
|
|||||||
for field in ("continuation_count", "max_continuations", "no_progress_count", "max_no_progress_continuations"):
|
for field in ("continuation_count", "max_continuations", "no_progress_count", "max_no_progress_continuations"):
|
||||||
if field in value and (type(value[field]) is not int or value[field] < 0):
|
if field in value and (type(value[field]) is not int or value[field] < 0):
|
||||||
raise HTTPException(status_code=400, detail=f"{location}.{field} must be a non-negative integer")
|
raise HTTPException(status_code=400, detail=f"{location}.{field} must be a non-negative integer")
|
||||||
return {**{key: item for key, item in value.items() if key != "last_batch_wakeup_run_id"}, "objective": objective}
|
return {**value, "objective": objective}
|
||||||
|
|
||||||
|
|
||||||
def strip_server_owned_state_metadata(values: Mapping[str, Any]) -> dict[str, Any]:
|
def strip_server_owned_state_metadata(values: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
@ -814,29 +668,10 @@ def strip_server_owned_state_metadata(values: Mapping[str, Any]) -> dict[str, An
|
|||||||
for channel, value in values.items():
|
for channel, value in values.items():
|
||||||
if channel == "goal":
|
if channel == "goal":
|
||||||
stripped[channel] = _strip_external_metadata_from_message_like(_normalize_external_goal(value, location="values.goal"))
|
stripped[channel] = _strip_external_metadata_from_message_like(_normalize_external_goal(value, location="values.goal"))
|
||||||
elif channel == "todos":
|
|
||||||
normalized_todos = _normalize_external_todos(value, location="values.todos")
|
|
||||||
# Todos keep their extra fields, but they are still caller-supplied
|
|
||||||
# channel values: a dedicated validator must not bypass the
|
|
||||||
# per-entry metadata strip the generic list branch applies.
|
|
||||||
stripped[channel] = None if normalized_todos is None else [_strip_external_metadata_from_message_like(item) for item in normalized_todos]
|
|
||||||
elif channel == "messages" and value is not None:
|
elif channel == "messages" and value is not None:
|
||||||
stripped[channel] = _normalize_input_messages(value, location="values.messages")
|
stripped[channel] = _normalize_input_messages(value, location="values.messages")
|
||||||
elif channel == "delegations" and isinstance(value, list):
|
elif channel == "delegations" and isinstance(value, list):
|
||||||
stripped[channel] = [_strip_external_delegation_verdict(item) for item in value]
|
stripped[channel] = [_strip_external_delegation_verdict(item) for item in value]
|
||||||
elif channel == "skill_context":
|
|
||||||
normalized_skill_context = _normalize_external_skill_context(value, location="values.skill_context")
|
|
||||||
stripped[channel] = None if normalized_skill_context is None else [_strip_external_metadata_from_message_like(item) for item in normalized_skill_context]
|
|
||||||
elif channel == "promoted":
|
|
||||||
# A dedicated validator must not bypass the metadata strip the
|
|
||||||
# generic branches apply to caller-supplied channel values.
|
|
||||||
stripped[channel] = _strip_external_metadata_from_message_like(_normalize_external_promoted(value, location="values.promoted"))
|
|
||||||
elif channel == "tool_artifacts":
|
|
||||||
normalized_tool_artifacts = _normalize_external_tool_artifacts(value, location="values.tool_artifacts")
|
|
||||||
# Registry entries are rich dicts, so keep the per-entry metadata
|
|
||||||
# strip the generic list branch applied: a caller-crafted entry must
|
|
||||||
# not smuggle server-owned message metadata into the checkpoint.
|
|
||||||
stripped[channel] = None if normalized_tool_artifacts is None else [_strip_external_metadata_from_message_like(item) for item in normalized_tool_artifacts]
|
|
||||||
elif isinstance(value, list):
|
elif isinstance(value, list):
|
||||||
stripped[channel] = [_strip_external_metadata_from_message_like(item) for item in value]
|
stripped[channel] = [_strip_external_metadata_from_message_like(item) for item in value]
|
||||||
else:
|
else:
|
||||||
@ -905,26 +740,9 @@ def normalize_input(raw_input: dict[str, Any] | None, *, trusted_internal: bool
|
|||||||
cleaned = [_strip_external_delegation_verdict(entry) for entry in delegations]
|
cleaned = [_strip_external_delegation_verdict(entry) for entry in delegations]
|
||||||
if cleaned != delegations:
|
if cleaned != delegations:
|
||||||
result = {**result, "delegations": cleaned}
|
result = {**result, "delegations": cleaned}
|
||||||
if "skill_context" in result:
|
|
||||||
result = {**result, "skill_context": _normalize_external_skill_context(result["skill_context"], location="input.skill_context")}
|
|
||||||
if "promoted" in result:
|
|
||||||
result = {**result, "promoted": _normalize_external_promoted(result["promoted"], location="input.promoted")}
|
|
||||||
if "tool_artifacts" in result:
|
|
||||||
result = {**result, "tool_artifacts": _normalize_external_tool_artifacts(result["tool_artifacts"], location="input.tool_artifacts")}
|
|
||||||
# merge_goal ignores None, so only a goal value replaces the goal.
|
# merge_goal ignores None, so only a goal value replaces the goal.
|
||||||
if result.get("goal") is not None:
|
if result.get("goal") is not None:
|
||||||
normalized_goal = _normalize_external_goal(result["goal"], location="input.goal")
|
result = {**result, "goal": _normalize_external_goal(result["goal"], location="input.goal"), "goal_outcome": None}
|
||||||
result = {
|
|
||||||
**result,
|
|
||||||
"goal": _strip_external_metadata_from_message_like(normalized_goal),
|
|
||||||
"goal_outcome": None,
|
|
||||||
}
|
|
||||||
if "todos" in result:
|
|
||||||
normalized_todos = _normalize_external_todos(result["todos"], location="input.todos")
|
|
||||||
result = {
|
|
||||||
**result,
|
|
||||||
"todos": None if normalized_todos is None else [_strip_external_metadata_from_message_like(item) for item in normalized_todos],
|
|
||||||
}
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
@ -966,11 +784,11 @@ def _canonical_run_record_input(
|
|||||||
raw_input: dict[str, Any] | None,
|
raw_input: dict[str, Any] | None,
|
||||||
graph_input: object,
|
graph_input: object,
|
||||||
) -> dict[str, Any] | None:
|
) -> dict[str, Any] | None:
|
||||||
"""Persist the same normalized messages, goal, and todos that cross run admission.
|
"""Persist the same normalized messages that cross run admission.
|
||||||
|
|
||||||
The run record is a client-visible audit surface. Keeping the original raw
|
The run record is a client-visible audit surface. Keeping the original raw
|
||||||
message or state metadata there would preserve a non-canonical scope even
|
message there would preserve a non-canonical scope even though the graph
|
||||||
though the graph receives the validated form.
|
receives the validated form.
|
||||||
"""
|
"""
|
||||||
if not isinstance(graph_input, dict):
|
if not isinstance(graph_input, dict):
|
||||||
return raw_input
|
return raw_input
|
||||||
@ -978,9 +796,6 @@ def _canonical_run_record_input(
|
|||||||
messages = graph_input.get("messages")
|
messages = graph_input.get("messages")
|
||||||
if isinstance(messages, list):
|
if isinstance(messages, list):
|
||||||
canonical["messages"] = [message.model_dump(mode="json") if isinstance(message, BaseMessage) else message for message in messages]
|
canonical["messages"] = [message.model_dump(mode="json") if isinstance(message, BaseMessage) else message for message in messages]
|
||||||
for key in ("goal", "goal_outcome", "todos", "delegations"):
|
|
||||||
if key in graph_input:
|
|
||||||
canonical[key] = graph_input[key]
|
|
||||||
return canonical
|
return canonical
|
||||||
|
|
||||||
|
|
||||||
@ -1053,45 +868,6 @@ def _legacy_stored_idempotency_request(kwargs: dict[str, Any]) -> dict[str, Any]
|
|||||||
return identity
|
return identity
|
||||||
|
|
||||||
|
|
||||||
async def _find_admitted_run_for_idempotency_key(
|
|
||||||
run_mgr: RunManager,
|
|
||||||
thread_id: str,
|
|
||||||
idempotency_key: str,
|
|
||||||
*,
|
|
||||||
user_id: str | None,
|
|
||||||
) -> RunRecord | None:
|
|
||||||
"""Resolve an existing key on this thread and owner before pause checks."""
|
|
||||||
before_created_at: str | None = None
|
|
||||||
before_run_id: str | None = None
|
|
||||||
while True:
|
|
||||||
records = await run_mgr.list_by_thread(
|
|
||||||
thread_id,
|
|
||||||
user_id=user_id,
|
|
||||||
limit=100,
|
|
||||||
before_created_at=before_created_at,
|
|
||||||
before_run_id=before_run_id,
|
|
||||||
)
|
|
||||||
for record in records:
|
|
||||||
if record.idempotency_key == idempotency_key and record.user_id == user_id:
|
|
||||||
return record
|
|
||||||
if len(records) < 100:
|
|
||||||
return None
|
|
||||||
cursor = records[-1]
|
|
||||||
before_created_at = cursor.created_at
|
|
||||||
before_run_id = cursor.run_id
|
|
||||||
|
|
||||||
|
|
||||||
def _is_retryable_resume_start_failure(record: RunRecord | None) -> bool:
|
|
||||||
"""Whether admission failed before a resume worker could attach."""
|
|
||||||
return record is not None and record.status == RunStatus.error and isinstance(record.error, str) and record.error.startswith("Failed to attach run worker:")
|
|
||||||
|
|
||||||
|
|
||||||
async def _discard_failed_resume_admission(request: Request, run_mgr: RunManager, record: RunRecord) -> None:
|
|
||||||
"""Release a failed admission's key so an identical retry can start work."""
|
|
||||||
await get_run_store(request).delete(record.run_id, user_id=record.user_id)
|
|
||||||
await run_mgr.cleanup(record.run_id, delay=0)
|
|
||||||
|
|
||||||
|
|
||||||
_DEFAULT_ASSISTANT_ID = "lead_agent"
|
_DEFAULT_ASSISTANT_ID = "lead_agent"
|
||||||
|
|
||||||
|
|
||||||
@ -1104,7 +880,6 @@ _DEFAULT_ASSISTANT_ID = "lead_agent"
|
|||||||
_CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
|
_CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
|
||||||
{
|
{
|
||||||
"model_name",
|
"model_name",
|
||||||
"tool_approval_mode",
|
|
||||||
"mode",
|
"mode",
|
||||||
"thinking_enabled",
|
"thinking_enabled",
|
||||||
"reasoning_effort",
|
"reasoning_effort",
|
||||||
@ -1143,8 +918,6 @@ _SERVER_OWNED_RUNTIME_CONTEXT_KEYS: frozenset[str] = (
|
|||||||
"__run_loop_detection_recorder",
|
"__run_loop_detection_recorder",
|
||||||
"__run_tool_promotion_recorder",
|
"__run_tool_promotion_recorder",
|
||||||
"__run_tool_progress_recorder",
|
"__run_tool_progress_recorder",
|
||||||
"__run_input_control",
|
|
||||||
"__tool_approval_runtime",
|
|
||||||
"langgraph_auth_user",
|
"langgraph_auth_user",
|
||||||
"langgraph_auth_user_id",
|
"langgraph_auth_user_id",
|
||||||
THREAD_INCARNATION_METADATA_GUARD_KEY,
|
THREAD_INCARNATION_METADATA_GUARD_KEY,
|
||||||
@ -1152,7 +925,6 @@ _SERVER_OWNED_RUNTIME_CONTEXT_KEYS: frozenset[str] = (
|
|||||||
# at admission from threads_meta; a client-supplied value must
|
# at admission from threads_meta; a client-supplied value must
|
||||||
# never survive in either run-config section.
|
# never survive in either run-config section.
|
||||||
PROJECT_CONTEXT_KEY,
|
PROJECT_CONTEXT_KEY,
|
||||||
AGENT_AUTH_CONTEXT_KEY,
|
|
||||||
KNOWLEDGE_SCOPE_KEY,
|
KNOWLEDGE_SCOPE_KEY,
|
||||||
KNOWLEDGE_SCOPE_RUNTIME_KEY,
|
KNOWLEDGE_SCOPE_RUNTIME_KEY,
|
||||||
}
|
}
|
||||||
@ -1451,10 +1223,9 @@ _DEFAULT_RECURSION_LIMIT = 100
|
|||||||
_DEFAULT_MAX_RECURSION_LIMIT = 1000
|
_DEFAULT_MAX_RECURSION_LIMIT = 1000
|
||||||
|
|
||||||
|
|
||||||
def _resolve_gateway_recursion_limits(app_config: AppConfig | None = None) -> tuple[int, int]:
|
def _resolve_gateway_recursion_limits() -> tuple[int, int]:
|
||||||
"""Resolve the run default and ceiling from one hot-reloaded snapshot."""
|
"""Resolve the run default and ceiling from one hot-reloaded snapshot."""
|
||||||
try:
|
try:
|
||||||
if app_config is None:
|
|
||||||
app_config = get_app_config()
|
app_config = get_app_config()
|
||||||
raw = app_config.recursion_limit
|
raw = app_config.recursion_limit
|
||||||
max_limit = app_config.max_recursion_limit
|
max_limit = app_config.max_recursion_limit
|
||||||
@ -1537,17 +1308,11 @@ def _clamp_recursion_limit(value: Any, max_limit: int, default_limit: int) -> in
|
|||||||
return min(value, max_limit)
|
return min(value, max_limit)
|
||||||
|
|
||||||
|
|
||||||
# Error details echo caller-supplied agent names; bound them so an oversized
|
def custom_agent_name(assistant_id: str | None) -> str | None:
|
||||||
# value cannot bloat the response or any log line that records the detail.
|
|
||||||
_MAX_ECHOED_AGENT_NAME_CHARS = 200
|
|
||||||
|
|
||||||
|
|
||||||
def custom_agent_name(assistant_id: str | None, *, field: str = "assistant_id") -> str | None:
|
|
||||||
"""Return the custom agent *assistant_id* selects, or ``None`` for the lead agent.
|
"""Return the custom agent *assistant_id* selects, or ``None`` for the lead agent.
|
||||||
|
|
||||||
Custom names are normalized (trimmed, lowercased, ``_`` to ``-``) and must
|
Custom names are normalized (trimmed, lowercased, ``_`` to ``-``) and must
|
||||||
then contain only letters, digits and hyphens. *field* names the request
|
then contain only letters, digits and hyphens.
|
||||||
field in the error message.
|
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
ValueError: *assistant_id* cannot name an agent.
|
ValueError: *assistant_id* cannot name an agent.
|
||||||
@ -1556,7 +1321,7 @@ def custom_agent_name(assistant_id: str | None, *, field: str = "assistant_id")
|
|||||||
return None
|
return None
|
||||||
normalized = assistant_id.strip().lower().replace("_", "-")
|
normalized = assistant_id.strip().lower().replace("_", "-")
|
||||||
if not normalized or not re.fullmatch(r"[a-z0-9-]+", normalized):
|
if not normalized or not re.fullmatch(r"[a-z0-9-]+", normalized):
|
||||||
raise ValueError(f"Invalid {field} {assistant_id!r:.{_MAX_ECHOED_AGENT_NAME_CHARS}}: must contain only letters, digits, and hyphens after normalization.")
|
raise ValueError(f"Invalid assistant_id {assistant_id!r}: must contain only letters, digits, and hyphens after normalization.")
|
||||||
return normalized
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
@ -1566,7 +1331,6 @@ def build_run_config(
|
|||||||
metadata: dict[str, Any] | None,
|
metadata: dict[str, Any] | None,
|
||||||
*,
|
*,
|
||||||
assistant_id: str | None = None,
|
assistant_id: str | None = None,
|
||||||
app_config: AppConfig | None = None,
|
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""Build a RunnableConfig dict for the agent.
|
"""Build a RunnableConfig dict for the agent.
|
||||||
|
|
||||||
@ -1584,16 +1348,13 @@ def build_run_config(
|
|||||||
This mirrors the channel manager's ``_resolve_run_params`` logic. IM
|
This mirrors the channel manager's ``_resolve_run_params`` logic. IM
|
||||||
channel runs send no ``recursion_limit`` unless their session config or
|
channel runs send no ``recursion_limit`` unless their session config or
|
||||||
channel policy sets one, so they get the same configured default here.
|
channel policy sets one, so they get the same configured default here.
|
||||||
|
|
||||||
*app_config* is the snapshot the recursion limits come from; without it
|
|
||||||
they are read through ``get_app_config()``, which does file IO.
|
|
||||||
"""
|
"""
|
||||||
# Lead-agent recursion budget (LangGraph super-steps for the lead graph
|
# Lead-agent recursion budget (LangGraph super-steps for the lead graph
|
||||||
# only). Independent of subagent depth: a `task()` dispatch runs the whole
|
# only). Independent of subagent depth: a `task()` dispatch runs the whole
|
||||||
# subagent inside ONE lead tools-node step, and subagents enforce their own
|
# subagent inside ONE lead tools-node step, and subagents enforce their own
|
||||||
# limit via `subagents.max_turns`. Do not conflate this budget with the
|
# limit via `subagents.max_turns`. Do not conflate this budget with the
|
||||||
# general-purpose subagent's max_turns.
|
# general-purpose subagent's max_turns.
|
||||||
default_recursion_limit, max_recursion_limit = _resolve_gateway_recursion_limits(app_config)
|
default_recursion_limit, max_recursion_limit = _resolve_gateway_recursion_limits()
|
||||||
config: dict[str, Any] = {"recursion_limit": default_recursion_limit}
|
config: dict[str, Any] = {"recursion_limit": default_recursion_limit}
|
||||||
if request_config:
|
if request_config:
|
||||||
caller_configurable = request_config.get("configurable")
|
caller_configurable = request_config.get("configurable")
|
||||||
@ -2145,30 +1906,24 @@ async def ensure_checkpoint_history_seeded(
|
|||||||
if await event_store.list_messages(thread_id, limit=1, user_id=None):
|
if await event_store.list_messages(thread_id, limit=1, user_id=None):
|
||||||
return
|
return
|
||||||
|
|
||||||
checkpoint_config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
checkpoint_config = {
|
||||||
|
"configurable": {
|
||||||
|
"thread_id": thread_id,
|
||||||
|
"checkpoint_ns": "",
|
||||||
|
}
|
||||||
|
}
|
||||||
if await get_checkpointer(request).aget_tuple(checkpoint_config) is None:
|
if await get_checkpointer(request).aget_tuple(checkpoint_config) is None:
|
||||||
return
|
return
|
||||||
|
|
||||||
async def checkpoint_messages() -> list:
|
accessor, config = await abuild_checkpoint_state_accessor(
|
||||||
accessor, config = await abuild_checkpoint_state_accessor(request, thread_id=thread_id, assistant_id=assistant_id)
|
request,
|
||||||
|
thread_id=thread_id,
|
||||||
|
assistant_id=assistant_id,
|
||||||
|
)
|
||||||
snapshot = await accessor.aget(config)
|
snapshot = await accessor.aget(config)
|
||||||
values = getattr(snapshot, "values", None)
|
values = getattr(snapshot, "values", None)
|
||||||
messages = values.get("messages") if isinstance(values, dict) else None
|
messages = values.get("messages") if isinstance(values, dict) else None
|
||||||
return messages if isinstance(messages, list) else []
|
if not isinstance(messages, list) or not messages:
|
||||||
|
|
||||||
# Eligibility is advisory; an empty head needs no migration reservation.
|
|
||||||
# Discard this preview and reread under admission before any publication.
|
|
||||||
if not await checkpoint_messages():
|
|
||||||
return
|
|
||||||
|
|
||||||
async with reserve_history_seed(request, thread_id) as reservation:
|
|
||||||
if reservation is not None and await event_store.list_messages(thread_id, limit=1, user_id=None):
|
|
||||||
return
|
|
||||||
if await get_checkpointer(request).aget_tuple(checkpoint_config) is None:
|
|
||||||
return
|
|
||||||
|
|
||||||
messages = await checkpoint_messages()
|
|
||||||
if not messages:
|
|
||||||
return
|
return
|
||||||
|
|
||||||
events = build_checkpoint_history_seed_events(
|
events = build_checkpoint_history_seed_events(
|
||||||
@ -2178,7 +1933,7 @@ async def ensure_checkpoint_history_seeded(
|
|||||||
)
|
)
|
||||||
if not events:
|
if not events:
|
||||||
return
|
return
|
||||||
await write_history_seed(request, events, reservation=reservation, seed_kind="checkpoint", assistant_id=assistant_id)
|
await event_store.put_batch(events)
|
||||||
logger.info("Seeded %d checkpoint-history events for thread %s", len(events), thread_id)
|
logger.info("Seeded %d checkpoint-history events for thread %s", len(events), thread_id)
|
||||||
|
|
||||||
|
|
||||||
@ -2293,30 +2048,20 @@ async def _load_scope_agent_config(
|
|||||||
) -> Any | None:
|
) -> Any | None:
|
||||||
if not assistant_id or assistant_id == _DEFAULT_ASSISTANT_ID:
|
if not assistant_id or assistant_id == _DEFAULT_ASSISTANT_ID:
|
||||||
return None
|
return None
|
||||||
# The caller's config/context ``agent_name`` arrives here unvalidated, and
|
try:
|
||||||
# every run that names a custom agent passes through, with or without a
|
# The caller's config/context ``agent_name`` arrives here unvalidated.
|
||||||
# knowledge scope, so each rejection says what is wrong with the agent.
|
|
||||||
if not isinstance(assistant_id, str):
|
if not isinstance(assistant_id, str):
|
||||||
raise HTTPException(
|
raise ValueError("agent_name must be a string")
|
||||||
status_code=422,
|
normalized = assistant_id.strip().lower().replace("_", "-")
|
||||||
detail=f"Invalid agent_name: expected a string, got {type(assistant_id).__name__}.",
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
normalized = custom_agent_name(assistant_id, field="agent_name")
|
|
||||||
except ValueError as exc:
|
|
||||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
|
||||||
try:
|
|
||||||
return await asyncio.to_thread(
|
return await asyncio.to_thread(
|
||||||
load_agent_config,
|
load_agent_config,
|
||||||
normalized,
|
normalized,
|
||||||
user_id=user_id,
|
user_id=user_id,
|
||||||
)
|
)
|
||||||
except FileNotFoundError as exc:
|
except (FileNotFoundError, ValueError) as exc:
|
||||||
raise HTTPException(status_code=422, detail=f"Agent '{normalized:.{_MAX_ECHOED_AGENT_NAME_CHARS}}' not found") from exc
|
|
||||||
except ValueError as exc:
|
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=422,
|
status_code=422,
|
||||||
detail=f"Agent '{normalized:.{_MAX_ECHOED_AGENT_NAME_CHARS}}' configuration could not be loaded",
|
detail="knowledge_scope assistant configuration could not be resolved",
|
||||||
) from exc
|
) from exc
|
||||||
|
|
||||||
|
|
||||||
@ -2436,15 +2181,10 @@ async def start_run(
|
|||||||
stream_modes = normalize_stream_modes(body.stream_mode)
|
stream_modes = normalize_stream_modes(body.stream_mode)
|
||||||
bridge = get_stream_bridge(request)
|
bridge = get_stream_bridge(request)
|
||||||
run_mgr = get_run_manager(request)
|
run_mgr = get_run_manager(request)
|
||||||
# Building the context loads the live AppConfig, which stats and hashes
|
run_ctx = get_run_context(request)
|
||||||
# config.yaml and reparses it after an edit, so it runs in a worker. The
|
|
||||||
# rest of the run start reuses that snapshot.
|
|
||||||
run_ctx = await asyncio.to_thread(get_run_context, request)
|
|
||||||
app_config = run_ctx.app_config
|
|
||||||
admission_run_id: str | None = None
|
admission_run_id: str | None = None
|
||||||
producer_lease_started = False
|
producer_lease_started = False
|
||||||
producer_lease_handed_off = False
|
producer_lease_handed_off = False
|
||||||
validated_approval_request_ids: dict[str, str] | None = None
|
|
||||||
|
|
||||||
disconnect = DisconnectMode.cancel if body.on_disconnect == "cancel" else DisconnectMode.continue_
|
disconnect = DisconnectMode.cancel if body.on_disconnect == "cancel" else DisconnectMode.continue_
|
||||||
|
|
||||||
@ -2456,6 +2196,7 @@ async def start_run(
|
|||||||
|
|
||||||
# Validate model against the allowlist when a model_name is provided.
|
# Validate model against the allowlist when a model_name is provided.
|
||||||
if model_name:
|
if model_name:
|
||||||
|
app_config = get_app_config()
|
||||||
resolved = app_config.get_model_config(model_name)
|
resolved = app_config.get_model_config(model_name)
|
||||||
if resolved is None:
|
if resolved is None:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
@ -2516,43 +2257,10 @@ async def start_run(
|
|||||||
if admitted_origin not in EVIDENCE_ORIGINS:
|
if admitted_origin not in EVIDENCE_ORIGINS:
|
||||||
raise ValueError("Unsupported host evidence origin")
|
raise ValueError("Unsupported host evidence origin")
|
||||||
command = getattr(body, "command", None)
|
command = getattr(body, "command", None)
|
||||||
from app.gateway.hitl_approval import reject_new_run_when_approval_pending, validate_resume_request
|
|
||||||
|
|
||||||
current_user = get_current_user()
|
|
||||||
run_user_id = str(current_user.id) if current_user is not None else None
|
|
||||||
admitted_run = (
|
|
||||||
await _find_admitted_run_for_idempotency_key(
|
|
||||||
run_mgr,
|
|
||||||
thread_id,
|
|
||||||
idempotency_key,
|
|
||||||
user_id=run_user_id,
|
|
||||||
)
|
|
||||||
if idempotency_key is not None
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
if command and command.get("resume") is not None:
|
if command and command.get("resume") is not None:
|
||||||
graph_input = Command(resume=command["resume"])
|
graph_input = Command(resume=command["resume"])
|
||||||
stored_idempotency_request = (admitted_run.idempotency_request or _legacy_stored_idempotency_request(admitted_run.kwargs or {})) if admitted_run is not None else None
|
|
||||||
retry_failed_resume = _is_retryable_resume_start_failure(admitted_run) and stored_idempotency_request == idempotency_request
|
|
||||||
if admitted_run is None or retry_failed_resume:
|
|
||||||
accessor, read_config = await build_thread_checkpoint_state_accessor(
|
|
||||||
request,
|
|
||||||
thread_id=thread_id,
|
|
||||||
)
|
|
||||||
validated_approval_request_ids = await validate_resume_request(
|
|
||||||
request=request,
|
|
||||||
thread_id=thread_id,
|
|
||||||
accessor=accessor,
|
|
||||||
read_config=read_config,
|
|
||||||
enabled=app_config.hitl_approval.enabled,
|
|
||||||
body=body,
|
|
||||||
)
|
|
||||||
resume_map = getattr(validated_approval_request_ids, "resume_map", None)
|
|
||||||
if isinstance(resume_map, dict):
|
|
||||||
graph_input = Command(resume=resume_map)
|
|
||||||
else:
|
else:
|
||||||
if admitted_run is None:
|
# LangGraph continues pending work with None; {} starts new input.
|
||||||
await reject_new_run_when_approval_pending(request, thread_id)
|
|
||||||
graph_input = normalized_input if body.input is not None else None
|
graph_input = normalized_input if body.input is not None else None
|
||||||
# Keep the idempotency identity in the caller's pre-stamp shape. A
|
# Keep the idempotency identity in the caller's pre-stamp shape. A
|
||||||
# keyed retry can then reuse a run even though its stored run input has
|
# keyed retry can then reuse a run even though its stored run input has
|
||||||
@ -2582,7 +2290,7 @@ async def start_run(
|
|||||||
run_metadata[DEERFLOW_TRACE_METADATA_KEY] = ensure_trace_id()
|
run_metadata[DEERFLOW_TRACE_METADATA_KEY] = ensure_trace_id()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
config = build_run_config(thread_id, body.config, run_metadata, assistant_id=body.assistant_id, app_config=app_config)
|
config = build_run_config(thread_id, body.config, run_metadata, assistant_id=body.assistant_id)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
# Raised only for the caller's own config sections or assistant_id.
|
# Raised only for the caller's own config sections or assistant_id.
|
||||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||||
@ -2598,12 +2306,6 @@ async def start_run(
|
|||||||
# that carries agent configuration (model_name, thinking_enabled, etc.).
|
# that carries agent configuration (model_name, thinking_enabled, etc.).
|
||||||
# Only agent-relevant keys are forwarded; unknown keys (e.g. thread_id) are ignored.
|
# Only agent-relevant keys are forwarded; unknown keys (e.g. thread_id) are ignored.
|
||||||
merge_run_context_overrides(config, getattr(body, "context", None), internal=is_internal_caller)
|
merge_run_context_overrides(config, getattr(body, "context", None), internal=is_internal_caller)
|
||||||
from deerflow.approvals.policy import resolve_mode
|
|
||||||
|
|
||||||
try:
|
|
||||||
resolve_mode(app_config.hitl_approval, config.get("context") or {})
|
|
||||||
except (ValueError, TypeError) as exc:
|
|
||||||
raise HTTPException(422, detail={"code": "invalid_approval_mode", "message": "The selected tool approval mode is not allowed"}) from exc
|
|
||||||
if not is_internal_caller:
|
if not is_internal_caller:
|
||||||
# ``body.config`` is free-form and copied verbatim by
|
# ``body.config`` is free-form and copied verbatim by
|
||||||
# ``build_run_config``; scrub internal-only keys smuggled there.
|
# ``build_run_config``; scrub internal-only keys smuggled there.
|
||||||
@ -2697,20 +2399,6 @@ async def start_run(
|
|||||||
request_context=getattr(body, "context", None),
|
request_context=getattr(body, "context", None),
|
||||||
)
|
)
|
||||||
|
|
||||||
if agent_config is not None and getattr(run_ctx.app_config, "mcp_agent_auth", None):
|
|
||||||
from deerflow.mcp.agent_auth import McpAgentAuthorizationError, capture_agent_authority
|
|
||||||
|
|
||||||
try:
|
|
||||||
agent_authority = await run_assembly(
|
|
||||||
capture_agent_authority,
|
|
||||||
run_ctx.app_config,
|
|
||||||
user_id=owner_user_id or str(user.id),
|
|
||||||
agent_config=agent_config,
|
|
||||||
)
|
|
||||||
except McpAgentAuthorizationError as exc:
|
|
||||||
raise HTTPException(status_code=403, detail=str(exc)) from None
|
|
||||||
run_ctx = replace(run_ctx, agent_mcp_authority=agent_authority)
|
|
||||||
|
|
||||||
if scheduled_task_runtime is not None:
|
if scheduled_task_runtime is not None:
|
||||||
run_ctx = replace(run_ctx, scheduled_task_runtime=dict(scheduled_task_runtime))
|
run_ctx = replace(run_ctx, scheduled_task_runtime=dict(scheduled_task_runtime))
|
||||||
conversation_references = list(getattr(body, "conversation_references", None) or [])
|
conversation_references = list(getattr(body, "conversation_references", None) or [])
|
||||||
@ -2723,7 +2411,7 @@ async def start_run(
|
|||||||
user_id=owner_user_id or (str(user.id) if user is not None else None),
|
user_id=owner_user_id or (str(user.id) if user is not None else None),
|
||||||
run_context=run_ctx,
|
run_context=run_ctx,
|
||||||
run_manager=run_mgr,
|
run_manager=run_mgr,
|
||||||
app_config=app_config,
|
app_config=get_app_config(),
|
||||||
)
|
)
|
||||||
reader, source_ids = prepared
|
reader, source_ids = prepared
|
||||||
run_ctx = replace(run_ctx, conversation_reader=reader)
|
run_ctx = replace(run_ctx, conversation_reader=reader)
|
||||||
@ -2911,28 +2599,6 @@ async def start_run(
|
|||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("Scheduler tool capability is unavailable for this run", exc_info=True)
|
logger.warning("Scheduler tool capability is unavailable for this run", exc_info=True)
|
||||||
execution_context = replace(run_ctx, scheduler_capability=scheduler_capability) if scheduler_capability is not None else run_ctx
|
execution_context = replace(run_ctx, scheduler_capability=scheduler_capability) if scheduler_capability is not None else run_ctx
|
||||||
artifact_imports = getattr(request.state, "extension_artifact_imports", None)
|
|
||||||
artifact_lifetime = None
|
|
||||||
if artifact_imports:
|
|
||||||
from contextlib import AsyncExitStack
|
|
||||||
|
|
||||||
from deerflow_extension_api import AgentRunError
|
|
||||||
|
|
||||||
from app.gateway.extension_artifacts import verify_imports
|
|
||||||
|
|
||||||
artifact_lifetime = AsyncExitStack()
|
|
||||||
|
|
||||||
async def verify_artifacts():
|
|
||||||
# RunContext invokes this after durable admission and before
|
|
||||||
# Agent construction. A reused key never launches a worker.
|
|
||||||
try:
|
|
||||||
await await_drained(verify_imports(request, record.user_id, thread_id, artifact_imports, lifetime=artifact_lifetime))
|
|
||||||
except AgentRunError:
|
|
||||||
record.stop_reason = "artifact_conflict"
|
|
||||||
raise
|
|
||||||
|
|
||||||
execution_context = replace(execution_context, verify_artifacts=verify_artifacts)
|
|
||||||
try:
|
|
||||||
await run_agent(
|
await run_agent(
|
||||||
bridge,
|
bridge,
|
||||||
run_mgr,
|
run_mgr,
|
||||||
@ -2948,14 +2614,6 @@ async def start_run(
|
|||||||
knowledge_scope=admitted_knowledge_scope,
|
knowledge_scope=admitted_knowledge_scope,
|
||||||
**incarnation_kwargs,
|
**incarnation_kwargs,
|
||||||
)
|
)
|
||||||
finally:
|
|
||||||
if artifact_lifetime is not None:
|
|
||||||
await await_drained(artifact_lifetime.aclose())
|
|
||||||
if execution_context.agent_runs is not None and run_mgr.input_store is not None:
|
|
||||||
from app.gateway.run_inputs import dispatch_queued_inputs
|
|
||||||
from deerflow.runtime.runs.inputs import RunInputs
|
|
||||||
|
|
||||||
await dispatch_queued_inputs(RunInputs(run_mgr.input_store, record.run_id, record.user_id), agent_runs=execution_context.agent_runs, thread_id=thread_id)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
async with goal_thread_lock(thread_id):
|
async with goal_thread_lock(thread_id):
|
||||||
@ -2993,15 +2651,8 @@ async def start_run(
|
|||||||
"Stream producer tracking unavailable for run %s; admitting it with untracked (conservative) stream recovery",
|
"Stream producer tracking unavailable for run %s; admitting it with untracked (conservative) stream recovery",
|
||||||
sanitize_log_param(admission_run_id),
|
sanitize_log_param(admission_run_id),
|
||||||
)
|
)
|
||||||
|
|
||||||
async def admit_run() -> RunRecord:
|
|
||||||
try:
|
try:
|
||||||
if admitted_run is None and not (command and command.get("resume") is not None):
|
record = await run_mgr.create_or_reject(
|
||||||
# Validation above is intentionally early, but a pause can
|
|
||||||
# appear while this request prepares its admission. Recheck
|
|
||||||
# under the thread lock so a fresh input cannot replace it.
|
|
||||||
await reject_new_run_when_approval_pending(request, thread_id)
|
|
||||||
return await run_mgr.create_or_reject(
|
|
||||||
thread_id,
|
thread_id,
|
||||||
body.assistant_id,
|
body.assistant_id,
|
||||||
on_disconnect=disconnect,
|
on_disconnect=disconnect,
|
||||||
@ -3036,8 +2687,6 @@ async def start_run(
|
|||||||
# a different request for the run already admitted.
|
# a different request for the run already admitted.
|
||||||
raise HTTPException(status_code=409, detail="Idempotency-Key already used with a different request") from exc
|
raise HTTPException(status_code=409, detail="Idempotency-Key already used with a different request") from exc
|
||||||
|
|
||||||
record = await admit_run()
|
|
||||||
|
|
||||||
if record.idempotency_reused:
|
if record.idempotency_reused:
|
||||||
stored = record.kwargs or {}
|
stored = record.kwargs or {}
|
||||||
stored_input = stored.get("input")
|
stored_input = stored.get("input")
|
||||||
@ -3064,48 +2713,13 @@ async def start_run(
|
|||||||
status_code=409,
|
status_code=409,
|
||||||
detail="Idempotency-Key already used with a different request",
|
detail="Idempotency-Key already used with a different request",
|
||||||
)
|
)
|
||||||
if request_identity["kind"] == "resume" and _is_retryable_resume_start_failure(record):
|
|
||||||
# The original request never attached a worker. Preserve
|
|
||||||
# idempotency for real execution outcomes, but let this
|
|
||||||
# identical retry replace only that failed admission.
|
|
||||||
await _discard_failed_resume_admission(request, run_mgr, record)
|
|
||||||
record = await admit_run()
|
|
||||||
if record.idempotency_reused:
|
|
||||||
# A concurrent identical retry won the replacement.
|
|
||||||
return record
|
return record
|
||||||
else:
|
|
||||||
return record
|
|
||||||
worker = run_after_metadata(record)
|
|
||||||
answered_approval = None
|
|
||||||
try:
|
|
||||||
if command and command.get("resume") is not None:
|
|
||||||
from app.gateway.hitl_approval import _all_reject, _responses_of, current_approval_request_ids, mark_approval_answered
|
|
||||||
|
|
||||||
responses = _responses_of(command)
|
worker = run_after_metadata(record)
|
||||||
answered_approval = await mark_approval_answered(
|
try:
|
||||||
request,
|
|
||||||
thread_id,
|
|
||||||
expected_request_ids=(validated_approval_request_ids if responses is not None else None),
|
|
||||||
)
|
|
||||||
if answered_approval is None and responses is not None:
|
|
||||||
if not _all_reject(responses):
|
|
||||||
# Competing or stale approvals fail closed.
|
|
||||||
raise HTTPException(status_code=409, detail={"code": "no_pending_approval", "message": "No resumable pause exists for this thread"})
|
|
||||||
# A crashed attempt may have consumed the run row while
|
|
||||||
# leaving the same checkpoint pause live. Only that exact
|
|
||||||
# pause may use checkpoint-only recovery; a stale reject
|
|
||||||
# must never dismiss a newer approval.
|
|
||||||
current_request_ids = await current_approval_request_ids(
|
|
||||||
request,
|
|
||||||
thread_id,
|
|
||||||
accessor=accessor,
|
|
||||||
read_config=read_config,
|
|
||||||
)
|
|
||||||
if current_request_ids != validated_approval_request_ids:
|
|
||||||
raise HTTPException(status_code=409, detail={"code": "no_pending_approval", "message": "No resumable pause exists for this thread"})
|
|
||||||
# The stream lease was established before durable admission.
|
# The stream lease was established before durable admission.
|
||||||
# Once resume persistence succeeds, keep task attachment
|
# Keep the admission-to-task-attachment window await-free so
|
||||||
# await-free so cancellation cannot strand a pending row.
|
# cancellation cannot strand a pending row.
|
||||||
record.task = asyncio.create_task(worker)
|
record.task = asyncio.create_task(worker)
|
||||||
if producer_lease_started:
|
if producer_lease_started:
|
||||||
transfer_producer_lease = _declared_bridge_method(
|
transfer_producer_lease = _declared_bridge_method(
|
||||||
@ -3118,28 +2732,12 @@ async def start_run(
|
|||||||
record.task,
|
record.task,
|
||||||
)
|
)
|
||||||
producer_lease_handed_off = True
|
producer_lease_handed_off = True
|
||||||
except BaseException as exc:
|
except Exception as exc:
|
||||||
attachment_error = f"Failed to attach run worker: {exc}"
|
|
||||||
task = record.task
|
|
||||||
if task is None:
|
|
||||||
worker.close()
|
worker.close()
|
||||||
|
|
||||||
async def cleanup_failed_attachment() -> None:
|
|
||||||
if task is not None:
|
|
||||||
task.cancel()
|
|
||||||
await asyncio.gather(task, return_exceptions=True)
|
|
||||||
if record.task is task:
|
|
||||||
record.task = None
|
|
||||||
await run_mgr.fail_start_if_pending(
|
await run_mgr.fail_start_if_pending(
|
||||||
record.run_id,
|
record.run_id,
|
||||||
error=attachment_error,
|
error=f"Failed to attach run worker: {exc}",
|
||||||
)
|
)
|
||||||
if answered_approval is not None:
|
|
||||||
restored = await run_mgr.restore_approval_answer(answered_approval)
|
|
||||||
if not restored:
|
|
||||||
raise RuntimeError(f"Failed to restore approval run {answered_approval.run_id} after worker attachment failed")
|
|
||||||
|
|
||||||
await asyncio.shield(cleanup_failed_attachment())
|
|
||||||
raise
|
raise
|
||||||
except ConflictError as exc:
|
except ConflictError as exc:
|
||||||
raise BusyThreadConflict(str(exc)) from exc
|
raise BusyThreadConflict(str(exc)) from exc
|
||||||
@ -3216,7 +2814,7 @@ async def launch_scheduled_thread_run(
|
|||||||
input=graph_input,
|
input=graph_input,
|
||||||
command=None,
|
command=None,
|
||||||
metadata=metadata or {},
|
metadata=metadata or {},
|
||||||
config={"recursion_limit": await asyncio.to_thread(_resolve_scheduler_recursion_limit)},
|
config={"recursion_limit": _resolve_scheduler_recursion_limit()},
|
||||||
# ``user_id`` mirrors what IM channels put in ``body.context`` so
|
# ``user_id`` mirrors what IM channels put in ``body.context`` so
|
||||||
# runtime-context consumers without a ContextVar fallback (e.g.
|
# runtime-context consumers without a ContextVar fallback (e.g.
|
||||||
# user-scoped GuardrailMiddleware providers) see the owning user;
|
# user-scoped GuardrailMiddleware providers) see the owning user;
|
||||||
@ -3251,7 +2849,6 @@ async def launch_scheduled_thread_run(
|
|||||||
# scheduler service's own per-occurrence scope -- ensure_trace_context
|
# scheduler service's own per-occurrence scope -- ensure_trace_context
|
||||||
# keeps that trace instead of minting a competing one.
|
# keeps that trace instead of minting a competing one.
|
||||||
with ensure_trace_context():
|
with ensure_trace_context():
|
||||||
try:
|
|
||||||
record = await start_run(
|
record = await start_run(
|
||||||
body,
|
body,
|
||||||
thread_id,
|
thread_id,
|
||||||
@ -3260,14 +2857,6 @@ async def launch_scheduled_thread_run(
|
|||||||
evidence_origin="scheduled",
|
evidence_origin="scheduled",
|
||||||
scheduled_task_runtime=scheduled_task_runtime,
|
scheduled_task_runtime=scheduled_task_runtime,
|
||||||
)
|
)
|
||||||
except HTTPException as exc:
|
|
||||||
if isinstance(exc.detail, dict) and exc.detail.get("code") == "approval_pending":
|
|
||||||
# Skip-and-retry: an approval pause on the thread is temporary,
|
|
||||||
# not a permanent launch failure. The scheduler treats this as
|
|
||||||
# an overlap conflict and retries on its normal cadence.
|
|
||||||
logger.info("Scheduled launch skipped: thread %s has a pending approval", thread_id)
|
|
||||||
raise ConflictError(str(exc.detail)) from exc
|
|
||||||
raise
|
|
||||||
return {"run_id": record.run_id, "thread_id": record.thread_id}
|
return {"run_id": record.run_id, "thread_id": record.thread_id}
|
||||||
|
|
||||||
|
|
||||||
@ -3352,9 +2941,6 @@ async def launch_mcp_task_notification_run(
|
|||||||
require_existing_thread=True,
|
require_existing_thread=True,
|
||||||
)
|
)
|
||||||
except HTTPException as exc:
|
except HTTPException as exc:
|
||||||
if isinstance(exc.detail, dict) and exc.detail.get("code") == "approval_pending":
|
|
||||||
# Temporary condition, not a permanent notification failure.
|
|
||||||
raise ConflictError(str(exc.detail)) from exc
|
|
||||||
if isinstance(exc, BusyThreadConflict):
|
if isinstance(exc, BusyThreadConflict):
|
||||||
raise ConflictError(str(exc.detail)) from exc
|
raise ConflictError(str(exc.detail)) from exc
|
||||||
if exc.status_code in {400, 401, 403, 404, 409, 422, 501}:
|
if exc.status_code in {400, 401, 403, 404, 409, 422, 501}:
|
||||||
|
|||||||
@ -1,135 +0,0 @@
|
|||||||
"""Import a share into a fresh owner-scoped default-agent conversation."""
|
|
||||||
|
|
||||||
import os
|
|
||||||
import uuid
|
|
||||||
from contextlib import asynccontextmanager
|
|
||||||
|
|
||||||
from fastapi import HTTPException, Request
|
|
||||||
from langchain_core.messages import AIMessage, HumanMessage
|
|
||||||
from langgraph.types import Overwrite
|
|
||||||
|
|
||||||
from app.gateway.deps import get_checkpointer, get_run_event_store, get_run_manager, get_thread_store
|
|
||||||
from app.gateway.extension_artifacts import snapshot_sandbox
|
|
||||||
from app.gateway.history_seed import reserve_history_seed, write_history_seed
|
|
||||||
from app.gateway.persistent_writes import run_drained_write
|
|
||||||
from app.gateway.services import build_checkpoint_state_mutation_accessor
|
|
||||||
from app.gateway.share_snapshot import MAX_SHARE_FILE_BYTES, OUTPUT_PREFIX, artifact_path
|
|
||||||
from deerflow.config.paths import get_paths
|
|
||||||
from deerflow.persistence.thread_shares.sql import ThreadShareRepository
|
|
||||||
from deerflow.runtime import ThreadOperationKind
|
|
||||||
from deerflow.runtime.events.store.db import DbRunEventStore
|
|
||||||
from deerflow.runtime.goal import goal_thread_lock
|
|
||||||
from deerflow.runtime.journal import build_branch_history_seed_events
|
|
||||||
from deerflow.runtime.user_context import get_effective_user_id
|
|
||||||
from deerflow.sandbox.artifact_snapshot import publish_snapshot
|
|
||||||
from deerflow.utils.file_io import run_file_io
|
|
||||||
|
|
||||||
|
|
||||||
def _write_fork_files(owner: str, thread_id: str, artifacts: list[dict]) -> None:
|
|
||||||
"""A fresh UUID directory, descriptor-confined descendants and exclusive files."""
|
|
||||||
root = get_paths().sandbox_outputs_dir(thread_id, user_id=owner)
|
|
||||||
root.mkdir(mode=0o700, parents=True, exist_ok=False)
|
|
||||||
directory_flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
|
|
||||||
root_fd = os.open(root, directory_flags)
|
|
||||||
try:
|
|
||||||
for artifact in artifacts:
|
|
||||||
parts = artifact_path(artifact["path"]).split("/")
|
|
||||||
directory = os.dup(root_fd)
|
|
||||||
try:
|
|
||||||
for part in parts[:-1]:
|
|
||||||
try:
|
|
||||||
os.mkdir(part, 0o700, dir_fd=directory)
|
|
||||||
except FileExistsError:
|
|
||||||
pass
|
|
||||||
following = os.open(part, directory_flags, dir_fd=directory)
|
|
||||||
os.close(directory)
|
|
||||||
directory = following
|
|
||||||
os.fchmod(directory, 0o777)
|
|
||||||
fd = os.open(parts[-1], os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=directory)
|
|
||||||
with os.fdopen(fd, "wb") as target:
|
|
||||||
target.write(artifact["content"])
|
|
||||||
target.flush()
|
|
||||||
os.fchmod(target.fileno(), 0o666)
|
|
||||||
os.fsync(target.fileno())
|
|
||||||
finally:
|
|
||||||
os.close(directory)
|
|
||||||
# The bind mount is writable by the sandbox UID even when Gateway runs
|
|
||||||
# under a different UID or a restrictive umask.
|
|
||||||
os.fchmod(root_fd, 0o777)
|
|
||||||
finally:
|
|
||||||
os.close(root_fd)
|
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
|
||||||
async def _reserve_import(request: Request, thread_id: str, owner: str):
|
|
||||||
if isinstance(get_run_event_store(request), DbRunEventStore):
|
|
||||||
async with reserve_history_seed(request, thread_id) as reservation:
|
|
||||||
yield reservation
|
|
||||||
else:
|
|
||||||
async with get_run_manager(request).reserve_thread_operation(thread_id, kind=ThreadOperationKind.checkpoint_write, user_id=owner):
|
|
||||||
yield None
|
|
||||||
|
|
||||||
|
|
||||||
async def fork_share(request: Request, repo: ThreadShareRepository, token: str, request_id: str) -> dict:
|
|
||||||
share = await repo.get_public(token)
|
|
||||||
if share is None:
|
|
||||||
raise HTTPException(404, "Share unavailable")
|
|
||||||
owner = get_effective_user_id()
|
|
||||||
thread_id = str(uuid.uuid5(uuid.NAMESPACE_URL, f"deerflow-share:{owner}:{share['share_id']}:{request_id}"))
|
|
||||||
threads = get_thread_store(request)
|
|
||||||
async with goal_thread_lock(thread_id):
|
|
||||||
existing = await threads.get(thread_id, user_id=owner)
|
|
||||||
if existing is not None:
|
|
||||||
if existing.get("metadata", {}).get("share_import_complete") is True:
|
|
||||||
return {"thread_id": thread_id}
|
|
||||||
raise HTTPException(409, "The previous import did not finish; create a new copy")
|
|
||||||
artifacts = []
|
|
||||||
for descriptor in share["artifacts"]:
|
|
||||||
artifact = await repo.get_artifact(token, descriptor["path"])
|
|
||||||
if artifact is None:
|
|
||||||
raise HTTPException(404, "Share unavailable")
|
|
||||||
artifacts.append(artifact)
|
|
||||||
# Tool results become ordinary historical assistant text. The new thread
|
|
||||||
# inherits no tool execution, system prompt, Agent, project or auth grants.
|
|
||||||
messages = []
|
|
||||||
for item in share["snapshot"]["messages"]:
|
|
||||||
content = item["content"]
|
|
||||||
if item["type"] == "tool":
|
|
||||||
content = f"{item['name']}:\n{content}"
|
|
||||||
cls = HumanMessage if item["type"] == "human" else AIMessage
|
|
||||||
messages.append(cls(content=content, id=str(uuid.uuid4())))
|
|
||||||
title = str(share["snapshot"]["title"])[:256]
|
|
||||||
# Recheck after loading all bytes, before creating any new workspace state.
|
|
||||||
if await repo.get_public(token) is None:
|
|
||||||
raise HTTPException(404, "Share unavailable")
|
|
||||||
async with _reserve_import(request, thread_id, owner) as reservation:
|
|
||||||
existing = await threads.get(thread_id, user_id=owner)
|
|
||||||
if existing is not None:
|
|
||||||
if existing.get("metadata", {}).get("share_import_complete") is True:
|
|
||||||
return {"thread_id": thread_id}
|
|
||||||
raise HTTPException(409, "The previous import did not finish; create a new copy")
|
|
||||||
created = False
|
|
||||||
try:
|
|
||||||
record = await threads.create(thread_id, user_id=owner, display_name=title)
|
|
||||||
created = True
|
|
||||||
if artifacts:
|
|
||||||
await run_drained_write("Import shared artifacts", _write_fork_files, (), owner, thread_id, artifacts)
|
|
||||||
async with snapshot_sandbox(request, owner, thread_id) as sandbox:
|
|
||||||
if sandbox is not None:
|
|
||||||
for artifact in artifacts:
|
|
||||||
await run_file_io(publish_snapshot, sandbox, artifact["path"], artifact["content"], area="outputs", max_bytes=MAX_SHARE_FILE_BYTES)
|
|
||||||
accessor, config = build_checkpoint_state_mutation_accessor(request, thread_id=thread_id, as_node="branch")
|
|
||||||
await accessor.aupdate(config, {"messages": Overwrite(messages), "title": title, "artifacts": Overwrite([OUTPUT_PREFIX + a["path"] for a in artifacts])}, as_node="branch")
|
|
||||||
events = build_branch_history_seed_events(messages, thread_id=thread_id, run_id_prefix=f"branch-seed-{thread_id}", parent_thread_id=thread_id)
|
|
||||||
await write_history_seed(request, events, reservation=reservation, seed_kind="branch", thread=record)
|
|
||||||
await threads.update_metadata(thread_id, {"share_import_complete": True}, user_id=owner)
|
|
||||||
except Exception:
|
|
||||||
if created:
|
|
||||||
# Keep admission until rollback finishes, so a failed import
|
|
||||||
# cannot be continued while its files/checkpoints are removed.
|
|
||||||
await get_checkpointer(request).adelete_thread(thread_id)
|
|
||||||
await get_run_event_store(request).delete_by_thread(thread_id, user_id=owner)
|
|
||||||
await run_drained_write("Remove failed share import", get_paths().delete_thread_dir, (), thread_id, user_id=owner)
|
|
||||||
await threads.delete(thread_id, user_id=owner)
|
|
||||||
raise
|
|
||||||
return {"thread_id": thread_id}
|
|
||||||
@ -1,119 +0,0 @@
|
|||||||
"""Build a portable transcript under the caller's exclusive thread reservation."""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import mimetypes
|
|
||||||
from pathlib import PurePosixPath
|
|
||||||
|
|
||||||
from fastapi import HTTPException, Request
|
|
||||||
|
|
||||||
from app.gateway.conversation_access import _visible_text
|
|
||||||
from app.gateway.conversation_reader import read_visible_message_page
|
|
||||||
from app.gateway.deps import get_run_event_store, get_run_manager, get_thread_store
|
|
||||||
from app.gateway.extension_artifacts import read_snapshot
|
|
||||||
from app.gateway.services import build_thread_checkpoint_state_accessor
|
|
||||||
from deerflow.runtime import serialize_channel_values_for_api
|
|
||||||
from deerflow.runtime.user_context import get_effective_user_id
|
|
||||||
from deerflow.utils.file_io import run_file_io
|
|
||||||
|
|
||||||
MAX_SHARE_MESSAGES = 10_000
|
|
||||||
MAX_SHARE_JSON_BYTES = 10 * 1024 * 1024
|
|
||||||
MAX_SHARE_ARTIFACTS = 64
|
|
||||||
MAX_SHARE_FILE_BYTES = 20 * 1024 * 1024
|
|
||||||
MAX_SHARE_TOTAL_BYTES = 100 * 1024 * 1024
|
|
||||||
OUTPUT_PREFIX = "/mnt/user-data/outputs/"
|
|
||||||
|
|
||||||
|
|
||||||
def artifact_path(value: str) -> str:
|
|
||||||
"""Only canonical relative output paths are ever persisted or imported."""
|
|
||||||
path = value.removeprefix(OUTPUT_PREFIX)
|
|
||||||
if not path or len(path) > 1024 or path.startswith("/") or "\\" in path or any(ord(c) < 32 for c in path) or any(p in {"", ".", ".."} for p in path.split("/")) or str(PurePosixPath(path)) != path:
|
|
||||||
raise HTTPException(422, "Invalid shared artifact path")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def project_message(message: dict, index: int) -> dict | None:
|
|
||||||
"""Share visible text, not execution metadata, prompts or private attachments."""
|
|
||||||
extra = message.get("additional_kwargs") or {}
|
|
||||||
if not isinstance(extra, dict) or extra.get("hide_from_ui") or message.get("name") == "summary":
|
|
||||||
return None
|
|
||||||
visible = _visible_text({"content": message})
|
|
||||||
identity = str(message.get("id") or f"message-{index}")
|
|
||||||
if visible:
|
|
||||||
role, content = visible
|
|
||||||
return {"id": identity, "type": "human" if role == "user" else "ai", "content": content}
|
|
||||||
if message.get("type") == "tool":
|
|
||||||
content = message.get("content")
|
|
||||||
if not isinstance(content, str):
|
|
||||||
content = json.dumps(content, ensure_ascii=False)
|
|
||||||
return {"id": identity, "type": "tool", "name": str(message.get("name") or "tool"), "content": content}
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def project_transcript(messages: list[dict]) -> list[dict]:
|
|
||||||
projected: list[dict] = []
|
|
||||||
positions: dict[str, int] = {}
|
|
||||||
for index, message in enumerate(messages):
|
|
||||||
if not isinstance(message, dict):
|
|
||||||
continue
|
|
||||||
item = project_message(message, index)
|
|
||||||
if item is None:
|
|
||||||
continue
|
|
||||||
identity = item["id"].removesuffix("__user") if item["type"] == "human" else item["id"]
|
|
||||||
if identity in positions:
|
|
||||||
projected[positions[identity]] = item
|
|
||||||
else:
|
|
||||||
positions[identity] = len(projected)
|
|
||||||
projected.append(item)
|
|
||||||
return projected
|
|
||||||
|
|
||||||
|
|
||||||
async def capture_share(request: Request, thread_id: str, expected_checkpoint: str | None) -> dict:
|
|
||||||
owner = get_effective_user_id()
|
|
||||||
record = await get_thread_store(request).get(thread_id, user_id=owner)
|
|
||||||
if record is None:
|
|
||||||
raise HTTPException(404, "Thread not found")
|
|
||||||
accessor, config = await build_thread_checkpoint_state_accessor(request, thread_id=thread_id, fail_closed=True)
|
|
||||||
state = await accessor.aget(config)
|
|
||||||
checkpoint_id = (state.config or {}).get("configurable", {}).get("checkpoint_id")
|
|
||||||
if not checkpoint_id or state.next:
|
|
||||||
raise HTTPException(409, "Finish the current turn before sharing")
|
|
||||||
if expected_checkpoint is not None and checkpoint_id != expected_checkpoint:
|
|
||||||
raise HTTPException(409, "Thread changed; refresh before sharing")
|
|
||||||
rows: list[dict] = []
|
|
||||||
cursor = None
|
|
||||||
while True:
|
|
||||||
page, more = await read_visible_message_page(event_store=get_run_event_store(request), run_manager=get_run_manager(request), thread_id=thread_id, user_id=owner, limit=200, before_seq=cursor)
|
|
||||||
rows = page + rows
|
|
||||||
if len(rows) > MAX_SHARE_MESSAGES:
|
|
||||||
raise HTTPException(413, "Conversation exceeds the share limit")
|
|
||||||
if not more:
|
|
||||||
break
|
|
||||||
if not page or (cursor is not None and page[0]["seq"] >= cursor):
|
|
||||||
raise HTTPException(500, "Unable to capture complete conversation history")
|
|
||||||
cursor = page[0]["seq"]
|
|
||||||
values = serialize_channel_values_for_api(state.values)
|
|
||||||
source_messages = [row["content"] for row in rows] if rows else values.get("messages", [])
|
|
||||||
if len(source_messages) > MAX_SHARE_MESSAGES:
|
|
||||||
raise HTTPException(413, "Conversation exceeds the share limit")
|
|
||||||
messages = await run_file_io(project_transcript, source_messages)
|
|
||||||
if not messages:
|
|
||||||
raise HTTPException(409, "There are no visible messages to share")
|
|
||||||
snapshot = {"title": record.get("display_name") or values.get("title") or "Shared conversation", "messages": messages}
|
|
||||||
if len(json.dumps(snapshot, ensure_ascii=False).encode()) > MAX_SHARE_JSON_BYTES:
|
|
||||||
raise HTTPException(413, "Conversation exceeds the share limit")
|
|
||||||
paths = list(dict.fromkeys(values.get("artifacts") or []))
|
|
||||||
if len(paths) > MAX_SHARE_ARTIFACTS:
|
|
||||||
raise HTTPException(413, "Too many artifacts to share")
|
|
||||||
artifacts: list[dict] = []
|
|
||||||
total = 0
|
|
||||||
for path in paths:
|
|
||||||
if not isinstance(path, str) or not path.startswith(OUTPUT_PREFIX):
|
|
||||||
raise HTTPException(422, "Only output artifacts can be shared")
|
|
||||||
relative = artifact_path(path)
|
|
||||||
content = await read_snapshot(request, owner, thread_id, path, max_bytes=MAX_SHARE_FILE_BYTES)
|
|
||||||
total += len(content)
|
|
||||||
if total > MAX_SHARE_TOTAL_BYTES:
|
|
||||||
raise HTTPException(413, "Artifacts exceed the total share limit")
|
|
||||||
mime = await run_file_io(lambda: mimetypes.guess_type(relative)[0] or "application/octet-stream")
|
|
||||||
artifacts.append({"path": relative, "content": content, "mime_type": mime})
|
|
||||||
return {"thread_id": thread_id, "owner_id": owner, "incarnation": record.get("incarnation"), "checkpoint_id": checkpoint_id, "seq": max((row["seq"] for row in rows), default=0), "snapshot": snapshot, "artifacts": artifacts}
|
|
||||||
@ -1,105 +0,0 @@
|
|||||||
"""Owner-scoped HTTP decisions for live delegated calls and thread grants."""
|
|
||||||
|
|
||||||
import time
|
|
||||||
|
|
||||||
from fastapi import HTTPException, Request
|
|
||||||
from jsonschema import Draft202012Validator
|
|
||||||
from pydantic import BaseModel, ConfigDict, Field
|
|
||||||
|
|
||||||
from app.gateway.deps import get_current_user, get_run_manager, get_thread_store
|
|
||||||
from deerflow.approvals.policy import canonical_json
|
|
||||||
from deerflow.approvals.store import ApprovalConflict
|
|
||||||
from deerflow.utils.file_io import await_drained
|
|
||||||
|
|
||||||
|
|
||||||
class ApprovalResponseBody(BaseModel):
|
|
||||||
model_config = ConfigDict(extra="forbid")
|
|
||||||
responses: list[dict] = Field(min_length=1, max_length=32)
|
|
||||||
|
|
||||||
|
|
||||||
async def _scope(request, thread_id):
|
|
||||||
owner = await get_current_user(request)
|
|
||||||
if not owner:
|
|
||||||
raise HTTPException(401, "Authentication required")
|
|
||||||
thread = await get_thread_store(request).get(thread_id, user_id=owner)
|
|
||||||
if thread is None:
|
|
||||||
raise HTTPException(404, "Thread not found")
|
|
||||||
incarnation = thread.get("incarnation")
|
|
||||||
if not isinstance(incarnation, str):
|
|
||||||
raise HTTPException(409, "Thread approval identity is unavailable")
|
|
||||||
return (owner, thread_id, incarnation)
|
|
||||||
|
|
||||||
|
|
||||||
def _store(request):
|
|
||||||
store = getattr(request.app.state, "approval_store", None)
|
|
||||||
if store is None:
|
|
||||||
raise HTTPException(503, "Approval storage is unavailable")
|
|
||||||
return store
|
|
||||||
|
|
||||||
|
|
||||||
async def _active(request, row, owner):
|
|
||||||
if row["expires_at"] is not None and row["expires_at"] <= time.time():
|
|
||||||
return False
|
|
||||||
run = await get_run_manager(request).get(row["run_id"], user_id=owner)
|
|
||||||
return run is not None and run.thread_id == row["thread_id"] and run.status in ("running", "pending") and not run.ownership_lost and not run.abort_event.is_set()
|
|
||||||
|
|
||||||
|
|
||||||
async def list_approvals(thread_id, request):
|
|
||||||
scope = await _scope(request, thread_id)
|
|
||||||
rows = await _store(request).list(scope=scope, status="pending")
|
|
||||||
requests = [row["payload"] for row in rows if row["kind"] == "live" and await _active(request, row, scope[0])]
|
|
||||||
grants = await _store(request).list(scope=scope, grants=True)
|
|
||||||
return {"requests": requests, "grants": [{"request_id": row["request_id"], "operation": row["payload"]} for row in grants]}
|
|
||||||
|
|
||||||
|
|
||||||
async def answer_approval(thread_id, run_id, request_id, body, request: Request):
|
|
||||||
from app.gateway.auth_disabled import AUTH_SOURCE_INTERNAL, AUTH_SOURCE_PAT
|
|
||||||
|
|
||||||
if getattr(request.state, "auth_source", None) in (AUTH_SOURCE_INTERNAL, AUTH_SOURCE_PAT):
|
|
||||||
raise HTTPException(403, "Tool approval requires an interactive user session")
|
|
||||||
scope = await _scope(request, thread_id)
|
|
||||||
store = _store(request)
|
|
||||||
row = await store.get(request_id, scope=scope)
|
|
||||||
if row is None or row["run_id"] != run_id or row["kind"] != "live":
|
|
||||||
raise HTTPException(404, "Approval not found")
|
|
||||||
if not await _active(request, row, scope[0]):
|
|
||||||
raise HTTPException(409, "The originating run stopped or the approval expired")
|
|
||||||
actions = row["payload"]["actions"]
|
|
||||||
if len(body.responses) != len(actions):
|
|
||||||
raise HTTPException(422, "An answer is required for each operation")
|
|
||||||
by_id = {action["request_id"]: action for action in actions}
|
|
||||||
seen = set()
|
|
||||||
for response in body.responses:
|
|
||||||
response_id = response.get("request_id")
|
|
||||||
if not isinstance(response_id, str):
|
|
||||||
raise HTTPException(422, "Invalid approval request identity")
|
|
||||||
if not isinstance(response.get("value", ""), str) or len(response.get("value", "")) > 2000:
|
|
||||||
raise HTTPException(422, "Approval reason must be text of at most 2000 characters")
|
|
||||||
action = by_id.get(response_id)
|
|
||||||
if action is None or action["request_id"] in seen or response.get("tool_call_id") != action["tool_call_id"]:
|
|
||||||
raise HTTPException(422, "Stale or mismatched approval response")
|
|
||||||
seen.add(action["request_id"])
|
|
||||||
decision = response.get("option_id")
|
|
||||||
if decision not in row["payload"]["allowed_decisions"]:
|
|
||||||
raise HTTPException(422, "Unsupported approval decision")
|
|
||||||
if decision == "edit":
|
|
||||||
args, schema = response.get("args"), action.get("args_schema")
|
|
||||||
try:
|
|
||||||
if not isinstance(args, dict) or len(canonical_json(args).encode()) > row["payload"].get("max_payload_bytes", 512 * 1024):
|
|
||||||
raise ValueError("Invalid arguments")
|
|
||||||
Draft202012Validator(schema).validate(args)
|
|
||||||
if isinstance(schema.get("properties"), dict) and set(args) - set(schema["properties"]):
|
|
||||||
raise ValueError("Unknown or injected arguments")
|
|
||||||
except Exception as exc:
|
|
||||||
raise HTTPException(422, "Edited arguments do not match the tool schema") from exc
|
|
||||||
try:
|
|
||||||
await await_drained(store.decide(request_id, {"responses": body.responses}, scope=scope, fingerprint=row["fingerprint"]))
|
|
||||||
except ApprovalConflict as exc:
|
|
||||||
raise HTTPException(409, str(exc)) from exc
|
|
||||||
return {"accepted": True, "request_id": request_id}
|
|
||||||
|
|
||||||
|
|
||||||
async def revoke_approval_grants(thread_id, request):
|
|
||||||
scope = await _scope(request, thread_id)
|
|
||||||
await await_drained(_store(request).revoke(scope=scope))
|
|
||||||
return {"revoked": True}
|
|
||||||
@ -1689,60 +1689,11 @@ class McpTaskService:
|
|||||||
_CANCELLATION_DRAIN_TIMEOUT_SECONDS,
|
_CANCELLATION_DRAIN_TIMEOUT_SECONDS,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _log_pending_compensation_tasks(self, *, reason: str) -> None:
|
|
||||||
pending_count = sum(not task.done() for task in self._compensation_tasks)
|
|
||||||
if pending_count:
|
|
||||||
logger.warning(
|
|
||||||
"%s; %d MCP compensation task(s) remain in the background",
|
|
||||||
reason,
|
|
||||||
pending_count,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _drain_compensation_tasks(self, *, deadline: float) -> None:
|
|
||||||
"""Drain service-owned compensation work before Gateway dependencies close."""
|
|
||||||
loop = asyncio.get_running_loop()
|
|
||||||
while self._compensation_tasks:
|
|
||||||
tasks = tuple(self._compensation_tasks)
|
|
||||||
remaining = deadline - loop.time()
|
|
||||||
if remaining <= 0:
|
|
||||||
break
|
|
||||||
done, _ = await asyncio.wait(
|
|
||||||
tasks,
|
|
||||||
timeout=remaining,
|
|
||||||
return_when=asyncio.FIRST_COMPLETED,
|
|
||||||
)
|
|
||||||
if not done:
|
|
||||||
break
|
|
||||||
# Run completion callbacks before taking the next snapshot. A
|
|
||||||
# handoff may finish one task while registering another.
|
|
||||||
await asyncio.sleep(0)
|
|
||||||
|
|
||||||
self._log_pending_compensation_tasks(
|
|
||||||
reason="MCP compensation drain reached its deadline",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _drain_compensation_tasks_preserving_cancellation(self, *, deadline: float) -> None:
|
|
||||||
"""Finish the bounded drain before propagating caller cancellation."""
|
|
||||||
drain_task = asyncio.create_task(
|
|
||||||
self._drain_compensation_tasks(deadline=deadline),
|
|
||||||
name="deerflow-mcp-compensation-drain",
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
await asyncio.shield(drain_task)
|
|
||||||
except asyncio.CancelledError:
|
|
||||||
await wait_for_task_until(drain_task, deadline=deadline)
|
|
||||||
if drain_task.done():
|
|
||||||
drain_task.result()
|
|
||||||
raise
|
|
||||||
|
|
||||||
async def stop(self) -> None:
|
async def stop(self) -> None:
|
||||||
loop = asyncio.get_running_loop()
|
|
||||||
task = self._task
|
task = self._task
|
||||||
if task is None:
|
if task is None:
|
||||||
await self._drain_compensation_tasks_preserving_cancellation(
|
|
||||||
deadline=loop.time() + _CANCELLATION_DRAIN_TIMEOUT_SECONDS,
|
|
||||||
)
|
|
||||||
return
|
return
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
if self._stopping_task is not task:
|
if self._stopping_task is not task:
|
||||||
self._stopping_task = task
|
self._stopping_task = task
|
||||||
self._stop_deadline = loop.time() + _CANCELLATION_DRAIN_TIMEOUT_SECONDS
|
self._stop_deadline = loop.time() + _CANCELLATION_DRAIN_TIMEOUT_SECONDS
|
||||||
@ -1759,19 +1710,9 @@ class McpTaskService:
|
|||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
if not await wait_for_task_until(task, deadline=deadline):
|
if not await wait_for_task_until(task, deadline=deadline):
|
||||||
self._log_stop_timeout(task)
|
self._log_stop_timeout(task)
|
||||||
self._log_pending_compensation_tasks(
|
|
||||||
reason="MCP task poller cleanup reached its deadline before compensation drain could start",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
await self._drain_compensation_tasks_preserving_cancellation(deadline=deadline)
|
|
||||||
raise
|
raise
|
||||||
if task not in done:
|
if task not in done:
|
||||||
self._log_stop_timeout(task)
|
self._log_stop_timeout(task)
|
||||||
self._log_pending_compensation_tasks(
|
|
||||||
reason="MCP task poller cleanup reached its deadline before compensation drain could start",
|
|
||||||
)
|
|
||||||
return
|
|
||||||
await self._drain_compensation_tasks_preserving_cancellation(deadline=deadline)
|
|
||||||
|
|
||||||
async def _run_loop(self) -> None:
|
async def _run_loop(self) -> None:
|
||||||
while not self._stop.is_set():
|
while not self._stop.is_set():
|
||||||
|
|||||||
@ -1,26 +0,0 @@
|
|||||||
# Scheduler failure isolation
|
|
||||||
|
|
||||||
Scheduled work shares one polling cycle. Treat an individual occurrence as the
|
|
||||||
failure boundary so one damaged record cannot repeatedly prevent unrelated
|
|
||||||
owners' work from progressing. Apply this convention to maintenance passes as
|
|
||||||
well as launches: failing to clean up an older occurrence is not a reason to
|
|
||||||
withhold admission from a healthy task.
|
|
||||||
|
|
||||||
An exception during dispatch does not establish whether Gateway accepted the
|
|
||||||
run. Leave that uncertainty to durable queue and lease recovery. Eagerly
|
|
||||||
clearing a lease or starting a replacement from the exception handler can
|
|
||||||
duplicate work that already started. Cleanup is appropriate only when the
|
|
||||||
operation's outcome and ownership are known.
|
|
||||||
|
|
||||||
Containment preserves progress, not success. Keep failures observable and let
|
|
||||||
normal recovery retry the affected occurrence. Apply queue age before building
|
|
||||||
the fair drain batch: expired rows must not consume owner ranks or batch slots,
|
|
||||||
or a failed timeout sweep can starve healthy work indefinitely. Queue timeout is
|
|
||||||
enforced again when a waiting row is claimed for launch. If timeout cleanup
|
|
||||||
fails, the row remains queued for the next recovery pass but is no longer
|
|
||||||
eligible to start; cleanup failure must never turn expired work into a live run.
|
|
||||||
Failure to read the queue or claim due work still prevents that stage from
|
|
||||||
progressing; the next poll retries it.
|
|
||||||
|
|
||||||
For user-visible behavior and deployment requirements, see the
|
|
||||||
[scheduled tasks guide](../../../../docs/guide/scheduled-tasks.md).
|
|
||||||
@ -29,7 +29,7 @@ from deerflow.persistence.user.model import UserPreferenceRow
|
|||||||
from deerflow.runtime import ConflictError, RunRecord
|
from deerflow.runtime import ConflictError, RunRecord
|
||||||
from deerflow.runtime.run_origin import origin_kind_of
|
from deerflow.runtime.run_origin import origin_kind_of
|
||||||
from deerflow.scheduler.host_notes import RUN_ERROR_DELETED_WHILE_QUEUED, RUN_ERROR_INTERRUPTED, RUN_ERROR_LEASE_LOST, RUN_ERROR_PAUSED_WHILE_QUEUED, RUN_ERROR_QUEUE_TIMEOUT, RUN_ERROR_RESTARTED
|
from deerflow.scheduler.host_notes import RUN_ERROR_DELETED_WHILE_QUEUED, RUN_ERROR_INTERRUPTED, RUN_ERROR_LEASE_LOST, RUN_ERROR_PAUSED_WHILE_QUEUED, RUN_ERROR_QUEUE_TIMEOUT, RUN_ERROR_RESTARTED
|
||||||
from deerflow.scheduler.schedules import pauses_unschedulable, should_advance_schedule_cursor, stored_next_run_at
|
from deerflow.scheduler.schedules import next_run_at, should_advance_schedule_cursor
|
||||||
from deerflow.scheduler.stop_rule import launch_prompt
|
from deerflow.scheduler.stop_rule import launch_prompt
|
||||||
from deerflow.trace_context import ensure_trace_context
|
from deerflow.trace_context import ensure_trace_context
|
||||||
from deerflow.utils.thread_id import validate_thread_id
|
from deerflow.utils.thread_id import validate_thread_id
|
||||||
@ -450,15 +450,10 @@ class ScheduledTaskService:
|
|||||||
else:
|
else:
|
||||||
await self._reconcile_active_state(now=now)
|
await self._reconcile_active_state(now=now)
|
||||||
else:
|
else:
|
||||||
try:
|
|
||||||
await self._task_run_repo.recover_expired_launch_claims(
|
await self._task_run_repo.recover_expired_launch_claims(
|
||||||
error=_LEASE_RECOVERY_ERROR,
|
error=_LEASE_RECOVERY_ERROR,
|
||||||
now=now,
|
now=now,
|
||||||
)
|
)
|
||||||
except Exception:
|
|
||||||
# Like the reconciliation below: a failed recovery pass must
|
|
||||||
# not keep every other due task from being admitted.
|
|
||||||
logger.exception("Failed to recover expired scheduled launch claims")
|
|
||||||
try:
|
try:
|
||||||
finalized = await self._task_repo.reconcile_terminal_runs(now=now)
|
finalized = await self._task_repo.reconcile_terminal_runs(now=now)
|
||||||
if finalized:
|
if finalized:
|
||||||
@ -468,13 +463,7 @@ class ScheduledTaskService:
|
|||||||
# failure (e.g. the candidate query) must not skip admission or
|
# failure (e.g. the candidate query) must not skip admission or
|
||||||
# queue draining for this poll.
|
# queue draining for this poll.
|
||||||
logger.exception("Failed to reconcile terminal scheduled runs")
|
logger.exception("Failed to reconcile terminal scheduled runs")
|
||||||
try:
|
|
||||||
await self._expire_waiting_runs(now=now)
|
await self._expire_waiting_runs(now=now)
|
||||||
except Exception:
|
|
||||||
# A failed timeout sweep must not skip queue draining or this
|
|
||||||
# poll's due-task admission. Individual rows are also isolated by
|
|
||||||
# expire_queued_runs; this guard covers the candidate query/session.
|
|
||||||
logger.exception("Failed to expire queued scheduled runs")
|
|
||||||
await self._drain_queue(now=now)
|
await self._drain_queue(now=now)
|
||||||
# Admission and execution capacity are separate. Due occurrences are
|
# Admission and execution capacity are separate. Due occurrences are
|
||||||
# persisted even when all execution slots are busy; claim_queued_run()
|
# persisted even when all execution slots are busy; claim_queued_run()
|
||||||
@ -486,17 +475,7 @@ class ScheduledTaskService:
|
|||||||
limit=self._max_concurrent_runs,
|
limit=self._max_concurrent_runs,
|
||||||
)
|
)
|
||||||
for task in claimed:
|
for task in claimed:
|
||||||
try:
|
|
||||||
await self.dispatch_task(task, now=now, trigger="scheduled")
|
await self.dispatch_task(task, now=now, trigger="scheduled")
|
||||||
except Exception:
|
|
||||||
# A per-record failure inside one task's dispatch must not abort
|
|
||||||
# the rest of this poll's claimed batch: every claimed task
|
|
||||||
# already holds a lease, and the next tick would skip them the
|
|
||||||
# same way. Claim-level store failures stay loud above the loop.
|
|
||||||
logger.exception(
|
|
||||||
"Scheduled task %s: dispatch failed; continuing with the claimed batch",
|
|
||||||
task.get("id"),
|
|
||||||
)
|
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _is_overlap_conflict(exc: Exception) -> bool:
|
def _is_overlap_conflict(exc: Exception) -> bool:
|
||||||
@ -551,39 +530,22 @@ class ScheduledTaskService:
|
|||||||
*,
|
*,
|
||||||
trigger: str,
|
trigger: str,
|
||||||
now: datetime,
|
now: datetime,
|
||||||
) -> tuple[datetime | str | None, str | None]:
|
) -> datetime | str | None:
|
||||||
"""Keep a recurring task's cadence when a manual trial launches.
|
"""Keep a recurring task's cadence when a manual trial launches.
|
||||||
|
|
||||||
A manual run is an extra trial and must not consume the task's next
|
A manual run is an extra trial and must not consume the task's next
|
||||||
scheduled occurrence. One-time tasks retain their existing boundary
|
scheduled occurrence. One-time tasks retain their existing boundary
|
||||||
semantics, so only recurring manual launches keep the stored value.
|
semantics, so only recurring manual launches keep the stored value.
|
||||||
The second value is why a stored schedule no longer computes; the
|
|
||||||
launch bookkeeping then pauses the task instead of raising out of the
|
|
||||||
poll (see ``_unschedulable_outcome``).
|
|
||||||
"""
|
"""
|
||||||
if not should_advance_schedule_cursor(task["schedule_type"], trigger=trigger):
|
if not should_advance_schedule_cursor(task["schedule_type"], trigger=trigger):
|
||||||
return task.get("next_run_at"), None
|
return task.get("next_run_at")
|
||||||
return stored_next_run_at(
|
return next_run_at(
|
||||||
task["schedule_type"],
|
task["schedule_type"],
|
||||||
task["schedule_spec"],
|
task["schedule_spec"],
|
||||||
task["timezone"],
|
task["timezone"],
|
||||||
now=now,
|
now=now,
|
||||||
)
|
)
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _unschedulable_outcome(
|
|
||||||
task: dict[str, Any],
|
|
||||||
*,
|
|
||||||
status: str,
|
|
||||||
last_error: str | None,
|
|
||||||
schedule_error: str | None,
|
|
||||||
) -> tuple[str, str | None]:
|
|
||||||
"""The status and ``last_error`` to write, pausing an unschedulable task."""
|
|
||||||
if not pauses_unschedulable(task["schedule_type"], status, schedule_error):
|
|
||||||
return status, last_error
|
|
||||||
logger.warning("Paused scheduled task %s: %s", task["id"], schedule_error)
|
|
||||||
return "paused", schedule_error
|
|
||||||
|
|
||||||
async def dispatch_task(
|
async def dispatch_task(
|
||||||
self,
|
self,
|
||||||
task: dict[str, Any],
|
task: dict[str, Any],
|
||||||
@ -608,21 +570,15 @@ class ScheduledTaskService:
|
|||||||
# surface as HTTP 500 on manual trigger and, in the poller, abort
|
# surface as HTTP 500 on manual trigger and, in the poller, abort
|
||||||
# the rest of the claimed batch every cycle while the task itself
|
# the rest of the claimed batch every cycle while the task itself
|
||||||
# is never marked with last_error.
|
# is never marked with last_error.
|
||||||
next_at, schedule_error = self._next_run_at_after_launch(task, trigger=trigger, now=now)
|
task_status = self._task_status_for_failure(task, trigger=trigger)
|
||||||
task_status, last_error = self._unschedulable_outcome(
|
|
||||||
task,
|
|
||||||
status=self._task_status_for_failure(task, trigger=trigger),
|
|
||||||
last_error=str(exc),
|
|
||||||
schedule_error=schedule_error,
|
|
||||||
)
|
|
||||||
await self._task_repo.update_after_launch(
|
await self._task_repo.update_after_launch(
|
||||||
task["id"],
|
task["id"],
|
||||||
status=task_status,
|
status=task_status,
|
||||||
next_run_at=next_at,
|
next_run_at=self._next_run_at_after_launch(task, trigger=trigger, now=now),
|
||||||
last_run_at=now,
|
last_run_at=now,
|
||||||
last_run_id=None,
|
last_run_id=None,
|
||||||
last_thread_id=execution_thread_id,
|
last_thread_id=execution_thread_id,
|
||||||
last_error=last_error,
|
last_error=str(exc),
|
||||||
increment_run_count=False,
|
increment_run_count=False,
|
||||||
expected_lease_owner=expected_lease_owner,
|
expected_lease_owner=expected_lease_owner,
|
||||||
)
|
)
|
||||||
@ -710,7 +666,6 @@ class ScheduledTaskService:
|
|||||||
queued: dict[str, Any],
|
queued: dict[str, Any],
|
||||||
*,
|
*,
|
||||||
now: datetime,
|
now: datetime,
|
||||||
created_after: datetime | None = None,
|
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""Turn one queued occurrence into a live run under its own trace scope.
|
"""Turn one queued occurrence into a live run under its own trace scope.
|
||||||
|
|
||||||
@ -719,17 +674,9 @@ class ScheduledTaskService:
|
|||||||
sharing one id across a whole poll cycle. A manual trigger arrives
|
sharing one id across a whole poll cycle. A manual trigger arrives
|
||||||
inside a Gateway request and keeps that request's trace instead, so
|
inside a Gateway request and keeps that request's trace instead, so
|
||||||
the launched run stays correlated with the call that asked for it.
|
the launched run stays correlated with the call that asked for it.
|
||||||
|
|
||||||
``created_after`` applies the queue-age cutoff when retrying a row
|
|
||||||
from the waiting queue. A freshly admitted row has not waited.
|
|
||||||
"""
|
"""
|
||||||
with ensure_trace_context():
|
with ensure_trace_context():
|
||||||
return await self._launch_queued_occurrence(
|
return await self._launch_queued_occurrence(task, queued, now=now)
|
||||||
task,
|
|
||||||
queued,
|
|
||||||
now=now,
|
|
||||||
created_after=created_after,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _launch_queued_occurrence(
|
async def _launch_queued_occurrence(
|
||||||
self,
|
self,
|
||||||
@ -737,7 +684,6 @@ class ScheduledTaskService:
|
|||||||
queued: dict[str, Any],
|
queued: dict[str, Any],
|
||||||
*,
|
*,
|
||||||
now: datetime,
|
now: datetime,
|
||||||
created_after: datetime | None = None,
|
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
task_run_id = queued["id"]
|
task_run_id = queued["id"]
|
||||||
execution_thread_id = queued["thread_id"]
|
execution_thread_id = queued["thread_id"]
|
||||||
@ -749,7 +695,6 @@ class ScheduledTaskService:
|
|||||||
lease_seconds=self._lease_seconds,
|
lease_seconds=self._lease_seconds,
|
||||||
global_max_concurrent_runs=self._max_concurrent_runs,
|
global_max_concurrent_runs=self._max_concurrent_runs,
|
||||||
per_user_max_concurrent_runs=self._max_concurrent_runs_per_user,
|
per_user_max_concurrent_runs=self._max_concurrent_runs_per_user,
|
||||||
created_after=created_after,
|
|
||||||
)
|
)
|
||||||
if claimed is None:
|
if claimed is None:
|
||||||
return self._queued_result(task_run_id, execution_thread_id)
|
return self._queued_result(task_run_id, execution_thread_id)
|
||||||
@ -791,13 +736,8 @@ class ScheduledTaskService:
|
|||||||
launch_succeeded = True
|
launch_succeeded = True
|
||||||
launched_run_id = result["run_id"]
|
launched_run_id = result["run_id"]
|
||||||
launched_thread_id = result["thread_id"]
|
launched_thread_id = result["thread_id"]
|
||||||
next_at, schedule_error = self._next_run_at_after_launch(task, trigger=trigger, now=now)
|
next_at = self._next_run_at_after_launch(task, trigger=trigger, now=now)
|
||||||
task_status, last_error = self._unschedulable_outcome(
|
task_status = self._task_status_for_launch(task, trigger=trigger, next_at=next_at)
|
||||||
task,
|
|
||||||
status=self._task_status_for_launch(task, trigger=trigger, next_at=next_at),
|
|
||||||
last_error=None,
|
|
||||||
schedule_error=schedule_error,
|
|
||||||
)
|
|
||||||
await self._record_launched_run(
|
await self._record_launched_run(
|
||||||
task_run_id=task_run_id,
|
task_run_id=task_run_id,
|
||||||
task_id=task["id"],
|
task_id=task["id"],
|
||||||
@ -811,7 +751,7 @@ class ScheduledTaskService:
|
|||||||
last_run_at=now,
|
last_run_at=now,
|
||||||
last_run_id=launched_run_id,
|
last_run_id=launched_run_id,
|
||||||
last_thread_id=launched_thread_id,
|
last_thread_id=launched_thread_id,
|
||||||
last_error=last_error,
|
last_error=None,
|
||||||
increment_run_count=True,
|
increment_run_count=True,
|
||||||
task_run_id=task_run_id,
|
task_run_id=task_run_id,
|
||||||
# Same race as the run-row write above: a fast-failing run's
|
# Same race as the run-row write above: a fast-failing run's
|
||||||
@ -834,6 +774,8 @@ class ScheduledTaskService:
|
|||||||
)
|
)
|
||||||
return self._queued_result(task_run_id, execution_thread_id, error=str(exc))
|
return self._queued_result(task_run_id, execution_thread_id, error=str(exc))
|
||||||
|
|
||||||
|
next_at = self._next_run_at_after_launch(task, trigger=trigger, now=now)
|
||||||
|
|
||||||
if launch_succeeded:
|
if launch_succeeded:
|
||||||
# _launch_run succeeded, so a run is live even though
|
# _launch_run succeeded, so a run is live even though
|
||||||
# post-launch bookkeeping raised. Keep the task-run row
|
# post-launch bookkeeping raised. Keep the task-run row
|
||||||
@ -844,13 +786,7 @@ class ScheduledTaskService:
|
|||||||
# if the DB is still down the row stays "queued" -- still
|
# if the DB is still down the row stays "queued" -- still
|
||||||
# active, still holding the slot -- so we log and still report
|
# active, still holding the slot -- so we log and still report
|
||||||
# the run as launched so callers know a run is in flight.
|
# the run as launched so callers know a run is in flight.
|
||||||
next_at, schedule_error = self._next_run_at_after_launch(task, trigger=trigger, now=now)
|
task_status = self._task_status_for_launch(task, trigger=trigger, next_at=next_at)
|
||||||
task_status, last_error = self._unschedulable_outcome(
|
|
||||||
task,
|
|
||||||
status=self._task_status_for_launch(task, trigger=trigger, next_at=next_at),
|
|
||||||
last_error=None,
|
|
||||||
schedule_error=schedule_error,
|
|
||||||
)
|
|
||||||
try:
|
try:
|
||||||
await self._record_launched_run(
|
await self._record_launched_run(
|
||||||
task_run_id=task_run_id,
|
task_run_id=task_run_id,
|
||||||
@ -880,7 +816,7 @@ class ScheduledTaskService:
|
|||||||
# on a task whose run is actively running; the real
|
# on a task whose run is actively running; the real
|
||||||
# terminal outcome is written by handle_run_completion.
|
# terminal outcome is written by handle_run_completion.
|
||||||
# The transient itself is logged above.
|
# The transient itself is logged above.
|
||||||
last_error=last_error,
|
last_error=None,
|
||||||
increment_run_count=True,
|
increment_run_count=True,
|
||||||
task_run_id=task_run_id,
|
task_run_id=task_run_id,
|
||||||
protect_terminal=True,
|
protect_terminal=True,
|
||||||
@ -1043,22 +979,8 @@ class ScheduledTaskService:
|
|||||||
queued_rows = await self._task_run_repo.list_queued_runs(
|
queued_rows = await self._task_run_repo.list_queued_runs(
|
||||||
limit=max(16, self._max_concurrent_runs * 4),
|
limit=max(16, self._max_concurrent_runs * 4),
|
||||||
per_user_max_concurrent_runs=self._max_concurrent_runs_per_user,
|
per_user_max_concurrent_runs=self._max_concurrent_runs_per_user,
|
||||||
created_after=now - timedelta(seconds=self._queue_timeout_seconds),
|
|
||||||
)
|
)
|
||||||
for queued in queued_rows:
|
for queued in queued_rows:
|
||||||
try:
|
|
||||||
await self._drain_queued_run(queued, now=now)
|
|
||||||
except Exception:
|
|
||||||
# A storage failure on one occurrence must not leave every
|
|
||||||
# later queued row and this poll's due tasks undispatched. The
|
|
||||||
# failed row keeps its queue/lease state for the next recovery
|
|
||||||
# pass, and queue_timeout still bounds how long it can wait.
|
|
||||||
logger.exception(
|
|
||||||
"Failed to drain scheduled occurrence %s; continuing with the remaining queue",
|
|
||||||
queued.get("id"),
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _drain_queued_run(self, queued: dict[str, Any], *, now: datetime) -> None:
|
|
||||||
await self._task_repo.release_queued_admission_lease(queued["task_id"])
|
await self._task_repo.release_queued_admission_lease(queued["task_id"])
|
||||||
task = await self._task_repo.get_internal(queued["task_id"])
|
task = await self._task_repo.get_internal(queued["task_id"])
|
||||||
if task is None:
|
if task is None:
|
||||||
@ -1068,7 +990,7 @@ class ScheduledTaskService:
|
|||||||
error=RUN_ERROR_DELETED_WHILE_QUEUED,
|
error=RUN_ERROR_DELETED_WHILE_QUEUED,
|
||||||
finished_at=now,
|
finished_at=now,
|
||||||
)
|
)
|
||||||
return
|
continue
|
||||||
# Pausing suppresses automatic occurrences, but a manual trigger is
|
# Pausing suppresses automatic occurrences, but a manual trigger is
|
||||||
# an explicit request and has always been allowed to run without
|
# an explicit request and has always been allowed to run without
|
||||||
# resuming the schedule. A later pause still cancels an already
|
# resuming the schedule. A later pause still cancels an already
|
||||||
@ -1080,15 +1002,8 @@ class ScheduledTaskService:
|
|||||||
error=RUN_ERROR_PAUSED_WHILE_QUEUED,
|
error=RUN_ERROR_PAUSED_WHILE_QUEUED,
|
||||||
finished_at=now,
|
finished_at=now,
|
||||||
)
|
)
|
||||||
return
|
continue
|
||||||
# The timeout sweep may have failed for this row. Enforce the same
|
await self._attempt_queued_run(task, queued, now=now)
|
||||||
# cutoff in the atomic claim so cleanup retries cannot start it.
|
|
||||||
await self._attempt_queued_run(
|
|
||||||
task,
|
|
||||||
queued,
|
|
||||||
now=now,
|
|
||||||
created_after=now - timedelta(seconds=self._queue_timeout_seconds),
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _expire_waiting_runs(self, *, now: datetime) -> None:
|
async def _expire_waiting_runs(self, *, now: datetime) -> None:
|
||||||
await self._task_run_repo.expire_queued_runs(
|
await self._task_run_repo.expire_queued_runs(
|
||||||
|
|||||||
@ -1,144 +0,0 @@
|
|||||||
"""Native batch follow-ups through the existing revocable AgentRuns host."""
|
|
||||||
|
|
||||||
import copy
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
|
|
||||||
from deerflow_extension_api.agent_runs import AgentRunError, ExecutionFence
|
|
||||||
|
|
||||||
from deerflow.agents.human_input import extract_unanswered_human_input_request
|
|
||||||
from deerflow.mcp_scope import THREAD_INCARNATION_CONTEXT_KEY
|
|
||||||
from deerflow.runtime.goal import goal_waits_for_external_result, read_thread_goal
|
|
||||||
from deerflow.runtime.runs.schemas import GOAL_RUN_HARD_STOP_REASONS
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
_TRUSTED = object()
|
|
||||||
|
|
||||||
|
|
||||||
class BatchContinuations:
|
|
||||||
def __init__(self, app):
|
|
||||||
self.app = app
|
|
||||||
|
|
||||||
async def prepare(self, request):
|
|
||||||
from app.gateway.services import _CONTEXT_CONFIGURABLE_KEYS, _MODEL_SELECTION_CONTEXT_KEYS
|
|
||||||
|
|
||||||
host = self.app.state.agent_runs_host
|
|
||||||
grant = host.capture_batch_authority(request.continuation_authority, user_id=request.user_id)
|
|
||||||
source = await request.continuation_authority.get(thread_id=request.thread_id, run_id=request.run_id)
|
|
||||||
if source.status not in ("pending", "running"):
|
|
||||||
raise AgentRunError(409, "The originating Run has already stopped")
|
|
||||||
record = await self.app.state.run_manager.get(request.run_id, user_id=request.user_id)
|
|
||||||
if record is None or record.user_id != request.user_id or record.thread_id != request.thread_id:
|
|
||||||
raise AgentRunError(403, "Batch follow-up must belong to its originating Run")
|
|
||||||
captured = request.continuation_context or {}
|
|
||||||
if (record.metadata or {}).get("deerflow_origin", {}).get("kind") == "subagent_batch" and (captured.get("goal") or {}).get("status") != "active":
|
|
||||||
raise AgentRunError(409, "A batch follow-up cannot register another automatic follow-up")
|
|
||||||
context = {key: value for key, value in captured.get("context", {}).items() if key in _CONTEXT_CONFIGURABLE_KEYS | _MODEL_SELECTION_CONTEXT_KEYS and key not in ("agent_name", "is_bootstrap")}
|
|
||||||
if isinstance(captured.get("context", {}).get("locale"), str):
|
|
||||||
context["locale"] = captured["context"]["locale"]
|
|
||||||
scope = request.execution_spec.get("knowledge_scope")
|
|
||||||
return {
|
|
||||||
"prompt": request.continuation,
|
|
||||||
"grant": grant,
|
|
||||||
"thread_incarnation": request.execution_spec.get(THREAD_INCARNATION_CONTEXT_KEY),
|
|
||||||
"goal": copy.deepcopy(captured.get("goal")),
|
|
||||||
"context": copy.deepcopy(context),
|
|
||||||
"config": copy.deepcopy(captured.get("config", {})),
|
|
||||||
"knowledge_scope": copy.deepcopy(scope),
|
|
||||||
}
|
|
||||||
|
|
||||||
async def dispatch(self, batch):
|
|
||||||
continuation = batch["continuation"]
|
|
||||||
try:
|
|
||||||
runs = await self.app.state.agent_runs_host.restore_batch_authority(user_id=batch["user_id"], grant=continuation["grant"])
|
|
||||||
existing_id = await self.app.state.subagent_batch_repo.continuation_run_id(batch["id"], user_id=batch["user_id"])
|
|
||||||
if existing_id is not None:
|
|
||||||
existing = await runs.get(thread_id=batch["thread_id"], run_id=existing_id)
|
|
||||||
return {"status": "dispatched", "run_id": existing.run_id}
|
|
||||||
source = await runs.get(thread_id=batch["thread_id"], run_id=batch["run_id"])
|
|
||||||
if source.status in ("pending", "running"):
|
|
||||||
return None
|
|
||||||
if source.status != "success" or source.stop_reason in GOAL_RUN_HARD_STOP_REASONS:
|
|
||||||
return {"status": "suppressed"}
|
|
||||||
thread = await self.app.state.thread_store.get(batch["thread_id"], user_id=batch["user_id"])
|
|
||||||
if thread is None or thread.get("incarnation") != continuation.get("thread_incarnation"):
|
|
||||||
return {"status": "suppressed"}
|
|
||||||
state = await runs.get_state(thread_id=batch["thread_id"])
|
|
||||||
values = state.get("values", {})
|
|
||||||
if extract_unanswered_human_input_request(values.get("messages")) is not None:
|
|
||||||
return {"status": "waiting"}
|
|
||||||
goal = values.get("goal")
|
|
||||||
goal_identity = {key: goal.get(key) for key in ("created_at", "status", "objective")} if isinstance(goal, dict) else None
|
|
||||||
expected_goal = continuation.get("goal")
|
|
||||||
if goal_identity != expected_goal:
|
|
||||||
return {"status": "suppressed"}
|
|
||||||
goal_wakeup = None
|
|
||||||
if expected_goal is not None:
|
|
||||||
latest = await self.app.state.subagent_batch_repo.latest_run_state(batch["thread_id"], user_id=batch["user_id"])
|
|
||||||
if latest and latest["status"] in ("pending", "running"):
|
|
||||||
return None
|
|
||||||
if latest and (latest["status"] != "success" or latest.get("stop_reason") in GOAL_RUN_HARD_STOP_REASONS):
|
|
||||||
return {"status": "suppressed"}
|
|
||||||
if not goal_waits_for_external_result(goal, expected_goal):
|
|
||||||
return None if not (goal.get("last_evaluation") or {}).get("stand_down_reason") else {"status": "suppressed"}
|
|
||||||
goal_wakeup = expected_goal
|
|
||||||
public = await self.app.state.subagent_batch_repo.get_batch(batch["id"], user_id=batch["user_id"])
|
|
||||||
if public is None or public.get("continuation_status") == "cancelled" or public["status"] == "cancelled":
|
|
||||||
return {"status": "suppressed"}
|
|
||||||
if public["status"] not in ("completed", "failed"):
|
|
||||||
return None
|
|
||||||
# Stable input is essential after an unknown admission outcome.
|
|
||||||
# Counts/results stay in the existing owner-scoped result reader.
|
|
||||||
content = "The requested background batch has finished. Continue the agreed follow-up using its stored results; preserve any failures or missing information.\n" + json.dumps(
|
|
||||||
{"batch_id": batch["id"], "follow_up": continuation["prompt"]}, ensure_ascii=False
|
|
||||||
)
|
|
||||||
from deerflow.knowledge_scope import KNOWLEDGE_SCOPE_KEY
|
|
||||||
|
|
||||||
metadata = {"hide_from_ui": True, "untrusted_input": True}
|
|
||||||
if continuation.get("knowledge_scope") is not None:
|
|
||||||
metadata[KNOWLEDGE_SCOPE_KEY] = continuation["knowledge_scope"]
|
|
||||||
result = await runs.start_batch_followup(
|
|
||||||
thread_id=batch["thread_id"],
|
|
||||||
batch_id=batch["id"],
|
|
||||||
input={"messages": [{"role": "user", "content": content, "additional_kwargs": metadata}]},
|
|
||||||
context=continuation.get("context"),
|
|
||||||
config=continuation.get("config"),
|
|
||||||
execution_fence=ExecutionFence(**batch["execution_fence"]),
|
|
||||||
goal_wakeup=goal_wakeup,
|
|
||||||
)
|
|
||||||
return {"status": "dispatched", "run_id": result.run_id}
|
|
||||||
except AgentRunError as exc:
|
|
||||||
if exc.status_code == 409:
|
|
||||||
return None
|
|
||||||
if exc.status_code in (400, 401, 403, 404, 422):
|
|
||||||
return {"status": "suppressed"}
|
|
||||||
raise
|
|
||||||
|
|
||||||
async def cancel(self, batch_id, user_id, *, thread_id=None, authority=_TRUSTED, authorize_only=False):
|
|
||||||
if authority is not _TRUSTED:
|
|
||||||
self.app.state.agent_runs_host.capture_batch_authority(authority, user_id=user_id)
|
|
||||||
async with authority._request("runs:cancel", thread_id):
|
|
||||||
if authorize_only:
|
|
||||||
return
|
|
||||||
elif authorize_only:
|
|
||||||
return
|
|
||||||
run_id = await self.app.state.subagent_batch_repo.continuation_run_id(batch_id, user_id=user_id)
|
|
||||||
if run_id is None:
|
|
||||||
return
|
|
||||||
record = await self.app.state.run_manager.get(run_id, user_id=user_id)
|
|
||||||
if record is not None:
|
|
||||||
await self.app.state.run_manager.cancel(run_id)
|
|
||||||
|
|
||||||
async def source_finished(self, record):
|
|
||||||
if record.stop_reason == "goal_wakeup_superseded":
|
|
||||||
return
|
|
||||||
if record.status == "success" and record.stop_reason not in GOAL_RUN_HARD_STOP_REASONS:
|
|
||||||
return
|
|
||||||
for batch in await self.app.state.subagent_batch_repo.list_continuation_batches_for_run(record.run_id):
|
|
||||||
try:
|
|
||||||
await self.app.state.subagent_batch_service.cancel_batch(batch_id=batch["id"], user_id=batch["user_id"])
|
|
||||||
except Exception:
|
|
||||||
logger.warning("Batch %s cancellation failed for run %s (non-fatal)", batch["id"], record.run_id, exc_info=True)
|
|
||||||
goal = await read_thread_goal(self.app.state.checkpointer, record.thread_id)
|
|
||||||
if goal and goal.get("status") == "active" and goal.get("created_at"):
|
|
||||||
await self.app.state.subagent_batch_repo.suppress_thread_continuations(record.thread_id, goal_created_at=goal["created_at"])
|
|
||||||
@ -188,11 +188,6 @@ only letters, digits and hyphens; any other value is rejected with `422`.
|
|||||||
Branches preserve valid stored IDs, including their original spelling. If a
|
Branches preserve valid stored IDs, including their original spelling. If a
|
||||||
legacy source has an invalid ID, its new branch stores `null` and uses the lead agent.
|
legacy source has an invalid ID, its new branch stores `null` and uses the lead agent.
|
||||||
|
|
||||||
`metadata` is stored on the thread as sent and copied into its first
|
|
||||||
checkpoint, except for the keys the server owns there: `step`, `source`,
|
|
||||||
`writes`, `parents`, `created_at`, `counters_since_delta_snapshot` and
|
|
||||||
`deerflow_agent_name`.
|
|
||||||
|
|
||||||
**Response:**
|
**Response:**
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@ -356,7 +351,7 @@ does not accept this header and keeps the existing missing-stream close of
|
|||||||
**Run Option Compatibility:**
|
**Run Option Compatibility:**
|
||||||
- Supported concurrency strategies: `reject`, `rollback`, and `interrupt`
|
- Supported concurrency strategies: `reject`, `rollback`, and `interrupt`
|
||||||
- Compatibility default: `if_not_exists="create"`; this matches DeerFlow's current behavior
|
- Compatibility default: `if_not_exists="create"`; this matches DeerFlow's current behavior
|
||||||
- Artifact delivery is enforced automatically when a run creates or modifies regular files under `/mnt/user-data/outputs`. `present_files` or `present_shared_artifact` must present at least one path produced by the current run (or a directory containing it), and the terminal receipt must be persisted; presenting only an unrelated file does not satisfy delivery. Runs without changed outputs retain ordinary conversational behavior. `artifact_delivery` is not a client-settable run option.
|
- Artifact delivery is enforced automatically when a run creates or modifies regular files under `/mnt/user-data/outputs`. `present_files` must present at least one path produced by the current run (or a directory containing it), and the terminal receipt must be persisted; presenting only an unrelated file does not satisfy delivery. Runs without changed outputs retain ordinary conversational behavior. `artifact_delivery` is not a client-settable run option.
|
||||||
- Unsupported options return `422`: `webhook`, `stream_resumable=true`, `after_seconds`, `feedback_keys`, any non-null `on_completion` value (including the SDK values `"complete"` and `"continue"`), `if_not_exists="reject"`, and `multitask_strategy="enqueue"`
|
- Unsupported options return `422`: `webhook`, `stream_resumable=true`, `after_seconds`, `feedback_keys`, any non-null `on_completion` value (including the SDK values `"complete"` and `"continue"`), `if_not_exists="reject"`, and `multitask_strategy="enqueue"`
|
||||||
- `stream_resumable=false` is accepted: it is the LangGraph SDK's default and requests the non-resumable stream DeerFlow already serves
|
- `stream_resumable=false` is accepted: it is the LangGraph SDK's default and requests the non-resumable stream DeerFlow already serves
|
||||||
- Undeclared SDK options, including `checkpoint_during` and `durability`, also return `422` instead of being silently discarded
|
- Undeclared SDK options, including `checkpoint_during` and `durability`, also return `422` instead of being silently discarded
|
||||||
@ -510,15 +505,10 @@ Gateway API callers can attach up to three explicit references to create/stream/
|
|||||||
|
|
||||||
A reference is a valid thread ID or an absolute `/workspace/chats/{thread_id}` URL
|
A reference is a valid thread ID or an absolute `/workspace/chats/{thread_id}` URL
|
||||||
(also `/workspace/agents/{agent_name}/chats/{thread_id}` for custom agents)
|
(also `/workspace/agents/{agent_name}/chats/{thread_id}` for custom agents)
|
||||||
on the host and port the client sent the run request to, without query or fragment.
|
with the same scheme and authority as the run request, without query or fragment.
|
||||||
Gateway takes that host from `Forwarded` / `X-Forwarded-Host` / `Host`, as the CSRF
|
URLs are parsed as local selectors and are never fetched. For split-origin clients
|
||||||
check does, and appends `X-Forwarded-Port` when that host carries no port of its own.
|
or internal proxies, pass the thread ID. The field is separate from message text:
|
||||||
A stray `X-Forwarded-Port: 8443` therefore turns `deerflow.example` into
|
links in pasted documents, tool results, or previous messages grant no access.
|
||||||
`deerflow.example:8443`, and a link without that port gets a 422. The scheme is
|
|
||||||
not compared, since behind a TLS proxy Gateway can see `http` for an `https` link. URLs are parsed as local selectors and are never fetched.
|
|
||||||
For split-origin clients or internal proxies, pass the thread ID. The field is
|
|
||||||
separate from message text: links in pasted documents, tool results, or previous
|
|
||||||
messages grant no access.
|
|
||||||
The server supplies source IDs to the model as background user-role data and
|
The server supplies source IDs to the model as background user-role data and
|
||||||
binds the reader to this run's references and authenticated identity.
|
binds the reader to this run's references and authenticated identity.
|
||||||
|
|
||||||
@ -1468,7 +1458,7 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
|
|||||||
|
|
||||||
### Thread Cleanup
|
### Thread Cleanup
|
||||||
|
|
||||||
Delete a thread's checkpoint data and DeerFlow-managed files and metadata.
|
Remove DeerFlow-managed local thread files under `.deer-flow/threads/{thread_id}` after the LangGraph thread itself has been deleted.
|
||||||
|
|
||||||
```http
|
```http
|
||||||
DELETE /api/threads/{thread_id}
|
DELETE /api/threads/{thread_id}
|
||||||
@ -1485,14 +1475,6 @@ DELETE /api/threads/{thread_id}
|
|||||||
**Error behavior:**
|
**Error behavior:**
|
||||||
- `422` for invalid thread IDs
|
- `422` for invalid thread IDs
|
||||||
- `500` returns a generic `{"detail": "Failed to delete local thread data."}` response while full exception details stay in server logs
|
- `500` returns a generic `{"detail": "Failed to delete local thread data."}` response while full exception details stay in server logs
|
||||||
- `500` with `{"detail": "Failed to delete thread checkpoints."}` when a configured
|
|
||||||
checkpointer cannot delete the thread's checkpoints. The owner row is retained
|
|
||||||
for a retry.
|
|
||||||
- `409` with `Thread has work in flight. Delete it after the work finishes.` when
|
|
||||||
thread work blocks deletion. After a checkpoint error, a retry can also return
|
|
||||||
this response until the durable delete reservation is released or reconciled.
|
|
||||||
In a single-worker deployment with ownership heartbeats disabled, a stranded
|
|
||||||
reservation is reconciled at Gateway startup.
|
|
||||||
|
|
||||||
Deleting a thread also removes every user's read marker for it.
|
Deleting a thread also removes every user's read marker for it.
|
||||||
|
|
||||||
@ -2288,38 +2270,6 @@ curl -X POST http://localhost:2026/api/langgraph/threads/abc123/runs/stream \
|
|||||||
> section for details. Scheduled-task launches use
|
> section for details. Scheduled-task launches use
|
||||||
> `scheduler.recursion_limit` from `config.yaml` instead of a client body.
|
> `scheduler.recursion_limit` from `config.yaml` instead of a client body.
|
||||||
|
|
||||||
## Workspace search
|
|
||||||
|
|
||||||
`POST /api/threads/search` additionally accepts an optional `query` (up to 256
|
|
||||||
characters). It matches a trimmed, case-insensitive literal substring of the
|
|
||||||
conversation title before pagination, preserving the existing array response.
|
|
||||||
|
|
||||||
`POST /api/search` requires `threads:read` and returns grouped title/message
|
|
||||||
matches for the current user. Message results additionally require `runs:read`
|
|
||||||
and an installed `workspace_search` extension provider.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"query":"报告","types":["thread","message"],"archived":null,"limit":20,"cursor":null}
|
|
||||||
```
|
|
||||||
|
|
||||||
`types` defaults to both kinds; `limit` is 1–100. `archived` is a nullable boolean;
|
|
||||||
`project_id` omitted includes every project, explicit null selects unassigned
|
|
||||||
threads. Blank queries return recent titles. The response is
|
|
||||||
`{items, cursor, indexing, messages_available}`. Each item carries `type`,
|
|
||||||
`thread_id`, `title`, `agent_name`, `archived`, `project_id`, `updated_at` and
|
|
||||||
`snippet` (plain-text `{text, matched?}` segments). Message hits also carry the
|
|
||||||
canonical `message_id`, thread-global `seq` and `role`. Preserve the query and
|
|
||||||
filters with each opaque cursor; invalid cursors return 400. When `indexing` is
|
|
||||||
true, retry the current page until historical backfill completes. Provider errors
|
|
||||||
return 503; without a provider, title search stays available.
|
|
||||||
|
|
||||||
`GET /api/threads/{thread_id}/messages/around?message_id=...&seq=...&limit=20`
|
|
||||||
requires `runs:read` and thread ownership. `seq` is an optional exact version hint;
|
|
||||||
`limit` (1–50) bounds each side of the target. The response contains `messages`,
|
|
||||||
`target_seq`, `has_before` and `has_after`. Hidden/superseded targets and stale
|
|
||||||
version hints return 404. Earlier/later navigation uses the boundary message as
|
|
||||||
the next anchor. The UI keeps these windows separate from live history.
|
|
||||||
|
|
||||||
## Chat archive and restore
|
## Chat archive and restore
|
||||||
|
|
||||||
`POST /api/threads/search` accepts `archived: true` for archived chats or
|
`POST /api/threads/search` accepts `archived: true` for archived chats or
|
||||||
|
|||||||
@ -109,7 +109,7 @@ The web conversation delete flow first deletes Gateway-managed thread state thro
|
|||||||
|
|
||||||
**Trash tier (spec §8).** Trashed rows are invisible to the shelf index, tools, and listing APIs; they surface only through the `/api/trash` routes with their `trash_origin` display snapshot. Restore is a database re-point — `stored_relpath` is projects-root-relative, so no file ever moves (spec §10.6) — inside one transaction that locks the target active project row, then the source and any merge-candidate rows in ID order; identical active bytes in the target merge (trash row deleted, discarded namespace unlinked best-effort after commit), and a read-only existence/size check under the document lock rejects restores of missing or size-mismatched content with `409 content_missing`, leaving the row trashed. Purge holds the document-row lock continuously across trashed-state revalidation, unlink of the original and `derived/converted.md`, row deletion, and commit: already-absent content counts as removed, any other unlink error rolls back and keeps the row retryable, and retention callers revalidate the selected trash timestamp and cutoff under the lock so a restored-and-re-trashed row is never purged on its former expiry. Empty trash (`POST /api/trash/purge`) runs that same guarded purge once per trashed row of the caller with no age filter — the confirmation covers the whole listing — so the retention cutoff gates only the sweep's candidate selection. The retention sweep runs lazily on `GET /api/trash/documents` and once at gateway startup (no daemon): it purges rows at or past `projects.trash_retention_days` through the same guarded path, then reconciles storage — `.staging/*` entries and unreferenced files older than 24 hours are removed (nothing younger is ever collected; any active or trashed row protects its whole namespace, even after restore to another project) — and detects rows with missing or size-mismatched content, logging them at warning and surfacing them as `content_missing` without ever deleting the row (the row is the user's only record; disposal always starts with the user trashing it).
|
**Trash tier (spec §8).** Trashed rows are invisible to the shelf index, tools, and listing APIs; they surface only through the `/api/trash` routes with their `trash_origin` display snapshot. Restore is a database re-point — `stored_relpath` is projects-root-relative, so no file ever moves (spec §10.6) — inside one transaction that locks the target active project row, then the source and any merge-candidate rows in ID order; identical active bytes in the target merge (trash row deleted, discarded namespace unlinked best-effort after commit), and a read-only existence/size check under the document lock rejects restores of missing or size-mismatched content with `409 content_missing`, leaving the row trashed. Purge holds the document-row lock continuously across trashed-state revalidation, unlink of the original and `derived/converted.md`, row deletion, and commit: already-absent content counts as removed, any other unlink error rolls back and keeps the row retryable, and retention callers revalidate the selected trash timestamp and cutoff under the lock so a restored-and-re-trashed row is never purged on its former expiry. Empty trash (`POST /api/trash/purge`) runs that same guarded purge once per trashed row of the caller with no age filter — the confirmation covers the whole listing — so the retention cutoff gates only the sweep's candidate selection. The retention sweep runs lazily on `GET /api/trash/documents` and once at gateway startup (no daemon): it purges rows at or past `projects.trash_retention_days` through the same guarded path, then reconciles storage — `.staging/*` entries and unreferenced files older than 24 hours are removed (nothing younger is ever collected; any active or trashed row protects its whole namespace, even after restore to another project) — and detects rows with missing or size-mismatched content, logging them at warning and surfacing them as `content_missing` without ever deleting the row (the row is the user's only record; disposal always starts with the user trashing it).
|
||||||
|
|
||||||
**Document summaries (default on).** `project_documents.summary` is a nullable, best-effort LLM one-liner. The two shelf write routes enqueue a job after row creation (`projects/summaries.py`, enabled by default; disabled with `projects.summaries_enabled: false`); a bounded worker pool (`summary_concurrency` + bounded `summary_queue_size`, full queue drops) resolves text through `read_text_serving_path` (never re-deriving the convertible-first rule), calls the model via `run_oneshot_llm` with a caller-side wait limit (the worker slot is held until the provider call actually finishes; late results are discarded), rejects length-capped output through the shared provider-normalized detectors, and applies a write-time contract (unwrap → single line → empty is NULL → byte cap on a character boundary) before persisting with `updated_at` pinned so a landing summary never reorders the shelf. The summary rides the row's lifecycle (trash/restore/purge need no hooks), renders into the `<documents>` index and `list_project_documents` tag-neutralized at both exits, and is counted through `eligible/success/failed/skipped/queue_dropped/uncapped_accepted` metrics. All failure modes leave `summary = NULL`, which renders exactly like a shelf without the feature.
|
**Document summaries (opt-in).** `project_documents.summary` is a nullable, best-effort LLM one-liner. The two shelf write routes enqueue a job after row creation (`projects/summaries.py`, off unless `projects.summaries_enabled`); a bounded worker pool (`summary_concurrency` + bounded `summary_queue_size`, full queue drops) resolves text through `read_text_serving_path` (never re-deriving the convertible-first rule), calls the model via `run_oneshot_llm` with a caller-side wait limit (the worker slot is held until the provider call actually finishes; late results are discarded), rejects length-capped output through the shared provider-normalized detectors, and applies a write-time contract (unwrap → single line → empty is NULL → byte cap on a character boundary) before persisting with `updated_at` pinned so a landing summary never reorders the shelf. The summary rides the row's lifecycle (trash/restore/purge need no hooks), renders into the `<documents>` index and `list_project_documents` tag-neutralized at both exits, and is counted through `eligible/success/failed/skipped/queue_dropped/uncapped_accepted` metrics. All failure modes leave `summary = NULL`, which renders exactly like a shelf without the feature.
|
||||||
|
|
||||||
### Agent Architecture
|
### Agent Architecture
|
||||||
|
|
||||||
|
|||||||
@ -1494,7 +1494,7 @@ done
|
|||||||
docker logs deer-flow-gateway 2>&1 | grep -E "ChannelManager|channel" | head -10
|
docker logs deer-flow-gateway 2>&1 | grep -E "ChannelManager|channel" | head -10
|
||||||
```
|
```
|
||||||
|
|
||||||
**预期：** 无 auth 相关错误。渠道不依赖浏览器 cookie；服务端通过内部认证头接受请求，run 归入绑定 owner 的用户桶（未绑定时为规范化后的平台用户 ID；仅 `DEER_FLOW_AUTH_DISABLED=1` 时为 `default`）。
|
**预期：** 无 auth 相关错误。渠道不依赖浏览器 cookie；服务端通过内部认证头把请求归入 `default` 用户桶。
|
||||||
|
|
||||||
#### TC-DOCKER-05: reset_admin 密码写入 0600 凭证文件（不再走日志）
|
#### TC-DOCKER-05: reset_admin 密码写入 0600 凭证文件（不再走日志）
|
||||||
|
|
||||||
|
|||||||
@ -127,7 +127,7 @@ python -c "import secrets; print(secrets.token_urlsafe(32))"
|
|||||||
- **本地开发**（`make dev`）：Gateway embedded runtime 完全兼容；无 admin 时访问 `/setup` 初始化
|
- **本地开发**（`make dev`）：Gateway embedded runtime 完全兼容；无 admin 时访问 `/setup` 初始化
|
||||||
- **Gateway embedded runtime**：标准脚本、Docker dev 和生产部署均通过 Gateway 提供认证与 LangGraph-compatible API
|
- **Gateway embedded runtime**：标准脚本、Docker dev 和生产部署均通过 Gateway 提供认证与 LangGraph-compatible API
|
||||||
- **Docker 部署**：完全兼容，`.deer-flow/data/deerflow.db` 需持久化卷挂载
|
- **Docker 部署**：完全兼容，`.deer-flow/data/deerflow.db` 需持久化卷挂载
|
||||||
- **IM 渠道**（Feishu/Slack/Telegram）：通过 Gateway 内部认证通信。已通过 `/connect` 绑定的发送者使用绑定 owner 的用户桶，未绑定的发送者使用规范化后的平台用户 ID；只有设置 `DEER_FLOW_AUTH_DISABLED=1` 的单用户模式才统一使用 `default` 用户桶。详见 [IM_CHANNEL_CONNECTIONS.md](IM_CHANNEL_CONNECTIONS.md)
|
- **IM 渠道**（Feishu/Slack/Telegram）：通过 Gateway 内部认证通信，使用 `default` 用户桶
|
||||||
- **DeerFlowClient**（嵌入式）：不经过 HTTP，不受认证影响
|
- **DeerFlowClient**（嵌入式）：不经过 HTTP，不受认证影响
|
||||||
|
|
||||||
## 故障排查
|
## 故障排查
|
||||||
|
|||||||
@ -1,64 +0,0 @@
|
|||||||
# Native batch dependencies and follow-up
|
|
||||||
|
|
||||||
Optional fields extend the existing SQL-backed batch service. Omitting them keeps
|
|
||||||
independent batch execution. Items remain read-only or idempotent under the existing
|
|
||||||
retry/recovery policy.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
|
|
||||||
Use `depends_on` to name prerequisite item keys in the same batch. Unknown keys,
|
|
||||||
self-references and cycles are rejected before saving. Dependent items consume no
|
|
||||||
worker slot until every prerequisite succeeds without a guardrail stop reason.
|
|
||||||
Retryable attempts keep dependents waiting; exhausted failures, cancellation or
|
|
||||||
terminal capped results fail unstarted descendants. Independent items continue.
|
|
||||||
Retrying a failed prerequisite by hand does not requeue the descendants it already
|
|
||||||
failed; retry each blocked item as well.
|
|
||||||
|
|
||||||
Item prompts must name their input/output paths. Ordering does not certify factual
|
|
||||||
accuracy or make advisory acceptance mandatory. Use existing result readers; full
|
|
||||||
reports are not copied into every dependent's context.
|
|
||||||
|
|
||||||
## Requested follow-up
|
|
||||||
|
|
||||||
`continuation` describes remaining work the user already requested. After terminal
|
|
||||||
batch results are saved, the lead performs that work in the original conversation
|
|
||||||
through ordinary Run admission. Preserve the originating model selection, reasoning,
|
|
||||||
limits and admitted retrieval scope. Busy conversations wait. While a question is
|
|
||||||
unanswered the claim is kept until its lease expires, so the conversation is not
|
|
||||||
reread on every poll.
|
|
||||||
|
|
||||||
A stable idempotency key reconciles lost admission replies without another Run.
|
|
||||||
Pending claims rotate so busy conversations do not starve others. Manual item retries
|
|
||||||
hold an unadmitted follow-up but do not re-arm one already dispatched. Without a Goal,
|
|
||||||
a follow-up cannot request another automatic follow-up.
|
|
||||||
|
|
||||||
An active Goal wakes only from its own `external_wait`, consuming one existing
|
|
||||||
continuation allowance, once even after recovery. Other blockers, no-progress limits
|
|
||||||
and hard stops remain effective. Goal replacement, clearing and thread deletion
|
|
||||||
suppress pending work; a failed, cancelled or hard-stopped source Run cannot restart.
|
|
||||||
|
|
||||||
## Authority, cancellation and recovery
|
|
||||||
|
|
||||||
Follow-up needs a session-authorized Gateway AgentRuns capability or local auth-disabled
|
|
||||||
mode. Internal/PAT callers and unsupported embedded runtimes retain ordinary batches.
|
|
||||||
Persist only the original permission ceiling, account token version and auth source;
|
|
||||||
no cookie, API key or Python capability. Restore checks current account/permissions,
|
|
||||||
conversation ownership and incarnation. Private continuation data is not returned.
|
|
||||||
|
|
||||||
Batch cancellation revokes the existing SQL execution fence and stops an admitted
|
|
||||||
follow-up, including after a lost launch reply. HTTP and tool cancellation of a batch
|
|
||||||
with a follow-up also require `runs:cancel`. Trusted internal cleanup retains its existing
|
|
||||||
path. Claim ownership/expiry lives only in the existing execution-lease table.
|
|
||||||
Completed item results remain available. The Web panel discovers follow-up Runs
|
|
||||||
through existing observers; refresh never launches work. Chat Stop remains available.
|
|
||||||
|
|
||||||
## Upgrade and verification
|
|
||||||
|
|
||||||
Migration `0052_batch_continuation` adds dependency edges and nullable follow-up data,
|
|
||||||
status and Run ID. Existing rows need no backfill. Upgrade all batch workers before
|
|
||||||
using dependencies. Do not downgrade with opted-in work outstanding; downgrade drops
|
|
||||||
new metadata while retaining items and reports.
|
|
||||||
|
|
||||||
Focused regressions cover dependency/retry ordering, SQL claims and recovery,
|
|
||||||
permission revocation, admission races, Goal limits and page reconnection. PostgreSQL
|
|
||||||
coverage uses `TEST_POSTGRES_URI`; browser backend timing is simulated.
|
|
||||||
@ -323,7 +323,7 @@ are:
|
|||||||
|
|
||||||
| Provider ID | Service / credential scope | Endpoint | Protocol / authentication |
|
| Provider ID | Service / credential scope | Endpoint | Protocol / authentication |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `openai` | OpenAI API | `https://api.openai.com/v1` | Responses for the GPT-6 IDs below; otherwise Chat Completions / Bearer |
|
| `openai` | OpenAI API | `https://api.openai.com/v1` | Chat Completions / Bearer |
|
||||||
| `anthropic` | Anthropic API | `https://api.anthropic.com` | Native Messages `/v1/messages` / `x-api-key` |
|
| `anthropic` | Anthropic API | `https://api.anthropic.com` | Native Messages `/v1/messages` / `x-api-key` |
|
||||||
| `deepseek` | DeepSeek API | `https://api.deepseek.com` | Chat Completions / Bearer |
|
| `deepseek` | DeepSeek API | `https://api.deepseek.com` | Chat Completions / Bearer |
|
||||||
| `qwen-cn` | Alibaba Cloud Bailian / Qwen, China | `https://dashscope.aliyuncs.com/compatible-mode/v1` | Chat Completions / Bearer |
|
| `qwen-cn` | Alibaba Cloud Bailian / Qwen, China | `https://dashscope.aliyuncs.com/compatible-mode/v1` | Chat Completions / Bearer |
|
||||||
@ -334,7 +334,6 @@ are:
|
|||||||
| `zai` | Z.AI, international ordinary API | `https://api.z.ai/api/paas/v4` | Chat Completions / Bearer |
|
| `zai` | Z.AI, international ordinary API | `https://api.z.ai/api/paas/v4` | Chat Completions / Bearer |
|
||||||
| `minimax` | MiniMax, international API | `https://api.minimax.io/v1` | Chat Completions / Bearer |
|
| `minimax` | MiniMax, international API | `https://api.minimax.io/v1` | Chat Completions / Bearer |
|
||||||
| `minimax-cn` | MiniMax, China API | `https://api.minimaxi.com/v1` | Chat Completions / Bearer |
|
| `minimax-cn` | MiniMax, China API | `https://api.minimaxi.com/v1` | Chat Completions / Bearer |
|
||||||
| `stepfun` | StepFun, ordinary API | `https://api.stepfun.com/v1` | Chat Completions / Bearer |
|
|
||||||
| `openai-compatible` | Custom OpenAI-compatible, including unauthenticated local endpoints | Administrator-supplied HTTP(S) base URL | Chat Completions / optional key |
|
| `openai-compatible` | Custom OpenAI-compatible, including unauthenticated local endpoints | Administrator-supplied HTTP(S) base URL | Chat Completions / optional key |
|
||||||
|
|
||||||
Keys for different regions and Coding Plans are not interchangeable. These
|
Keys for different regions and Coding Plans are not interchangeable. These
|
||||||
@ -343,47 +342,6 @@ Plan from the model name. Plans/regions without a preset can use the custom
|
|||||||
OpenAI-compatible option if their documented protocol is compatible; native
|
OpenAI-compatible option if their documented protocol is compatible; native
|
||||||
protocols need a verified backend preset or operator YAML configuration.
|
protocols need a verified backend preset or operator YAML configuration.
|
||||||
|
|
||||||
The StepFun preset uses the existing `PatchedChatStepFun` adapter to capture
|
|
||||||
`reasoning` / `reasoning_content` in responses and replay reasoning history.
|
|
||||||
Both connection probes and chat send the documented `max_tokens` output bound.
|
|
||||||
Model IDs remain free-form; this provider registration does not infer thinking,
|
|
||||||
vision or context-window capabilities from their names. The preset is for
|
|
||||||
ordinary API credentials. Step Plan has a separate `/step_plan/v1` endpoint;
|
|
||||||
use the custom OpenAI-compatible option for that channel. See the official
|
|
||||||
[OpenAI migration guide](https://platform.stepfun.com/docs/zh/guides/developer/openai)
|
|
||||||
and [Chat Completions reference](https://platform.stepfun.com/docs/zh/api-reference/chat/chat-completion-create).
|
|
||||||
|
|
||||||
Settings → Models → **Fetch and add models** reads the connection's model
|
|
||||||
catalog without changing configuration. Administrators can reuse an existing
|
|
||||||
saved connection without exposing its key, search IDs, select visible results,
|
|
||||||
and test/add the selection. Each selected new model must pass the existing
|
|
||||||
streaming tool-call probe before it is enabled. Tests can incur provider charges.
|
|
||||||
Failures are displayed individually; stop/retry preserves successful imports and
|
|
||||||
deduplicates by provider, endpoint, credential and model ID. Existing disabled
|
|
||||||
profiles remain disabled. Renaming and enabling/disabling stay in the model editor.
|
|
||||||
|
|
||||||
Discovery uses the provider's model-list endpoint (including Anthropic pagination)
|
|
||||||
and reports unsupported endpoints for manual entry. It follows no redirects and
|
|
||||||
caps reads at 20 seconds, 2 MiB and 2,000 model IDs/20 pages; truncated catalogs are
|
|
||||||
marked. IDs do not establish chat/tool capabilities. No background synchronization
|
|
||||||
or automatic enabling occurs just because a model appears in the catalog.
|
|
||||||
|
|
||||||
Chat model pickers search display names, configured names, IDs and provider labels,
|
|
||||||
with provider filtering and favorites/recent groups. The eight recent selections
|
|
||||||
are browser-local and user-scoped, and only currently available models appear.
|
|
||||||
Selecting a model continues to use the existing conversation/account preferences.
|
|
||||||
|
|
||||||
The `openai` preset recognizes the exact IDs `gpt-6-astra`, `gpt-6.1-sol`,
|
|
||||||
`gpt-6.1-terra`, `gpt-6-sol` and `gpt-6-luna`. They use the Responses API
|
|
||||||
for both the connection probe and chat, including tool calls and tool-result
|
|
||||||
follow-ups. Astra and GPT-6.1 Sol require thinking, default to `medium`, and
|
|
||||||
accept `low`, `medium`, `high`, `xhigh` and `max`. The other three also accept
|
|
||||||
`none`: disabling thinking uses `none`, while enabling it without an explicit
|
|
||||||
effort uses `medium`. Generic `minimal` maps to `low`. Image input is enabled.
|
|
||||||
Explicit capability overrides remain supported; routing stays provider-owned.
|
|
||||||
Unknown IDs, older OpenAI presets and custom endpoints keep their existing
|
|
||||||
behavior. See OpenAI's [GPT-6 protocol and parameter guide](https://developers.openai.com/api/docs/guides/latest-model).
|
|
||||||
|
|
||||||
The `moonshot-cn` preset recognizes the exact IDs `kimi-k3.1-pro` and
|
The `moonshot-cn` preset recognizes the exact IDs `kimi-k3.1-pro` and
|
||||||
`kimi-k3.1-pro-highspeed`. Both require thinking, preserve reasoning history,
|
`kimi-k3.1-pro-highspeed`. Both require thinking, preserve reasoning history,
|
||||||
and accept top-level `reasoning_effort` values `low`, `high` and `max`, with
|
and accept top-level `reasoning_effort` values `low`, `high` and `max`, with
|
||||||
@ -855,7 +813,7 @@ Notes:
|
|||||||
scripts/dev_multi_instance.sh up # containers, config, Gateway A, then B, then nginx
|
scripts/dev_multi_instance.sh up # containers, config, Gateway A, then B, then nginx
|
||||||
scripts/dev_multi_instance.sh check # automated cross-instance checks (below)
|
scripts/dev_multi_instance.sh check # automated cross-instance checks (below)
|
||||||
scripts/dev_multi_instance.sh stop b --kill # SIGKILL B to simulate a crash; `start b` brings it back
|
scripts/dev_multi_instance.sh stop b --kill # SIGKILL B to simulate a crash; `start b` brings it back
|
||||||
scripts/dev_multi_instance.sh logs a -f # follow Gateway A; also b, nginx, or all (both Gateways and nginx)
|
scripts/dev_multi_instance.sh logs a -f
|
||||||
scripts/dev_multi_instance.sh down
|
scripts/dev_multi_instance.sh down
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -933,16 +891,6 @@ agent_storage:
|
|||||||
- `agent_storage` is restart-required (the backend is captured at Gateway lifespan startup).
|
- `agent_storage` is restart-required (the backend is captured at Gateway lifespan startup).
|
||||||
- In a multi-worker Postgres deployment (`GATEWAY_WORKERS > 1`), leaving `agent_storage.backend: file` logs a startup warning — agents written to one node's local disk are invisible to the others, which is exactly the divergence the `db` backend fixes.
|
- In a multi-worker Postgres deployment (`GATEWAY_WORKERS > 1`), leaving `agent_storage.backend: file` logs a startup warning — agents written to one node's local disk are invisible to the others, which is exactly the divergence the `db` backend fixes.
|
||||||
|
|
||||||
Managed subagents also follow this backend selection. With `file`, their directory
|
|
||||||
must support hard links: creation atomically publishes a fully written temporary
|
|
||||||
file without replacing an existing name. Workers sharing that directory therefore
|
|
||||||
return one HTTP 201 and one HTTP 409 for concurrent same-name creates. Other
|
|
||||||
filesystem errors propagate rather than falling back to an overwrite.
|
|
||||||
Gateway startup probes hard-link creation once per worker in that directory,
|
|
||||||
off the event loop, and logs a warning if the probe fails. Startup continues;
|
|
||||||
use a volume with working hard links or migrate to `backend: db` before creating
|
|
||||||
managed subagents on an incompatible volume. Probe files are removed afterward.
|
|
||||||
|
|
||||||
Migrating an existing install from `file` to `db`:
|
Migrating an existing install from `file` to `db`:
|
||||||
|
|
||||||
Run from the repository root using the backend's `uv` environment. The importer
|
Run from the repository root using the backend's `uv` environment. The importer
|
||||||
@ -1314,8 +1262,8 @@ offline mocked tests verify request composition and local filtering, not provide
|
|||||||
retrieval behavior. No paid API calls are needed for the regression suite.
|
retrieval behavior. No paid API calls are needed for the regression suite.
|
||||||
|
|
||||||
**Built-in Tools**:
|
**Built-in Tools**:
|
||||||
- `web_search` - Search the web (DuckDuckGo, Tavily, Brave, Serper, Serply, Exa, InfoQuest, Tencent Cloud WSA, Firecrawl, fastCRW, GroundRoute, Sofya, SpecStream, You.com)
|
- `web_search` - Search the web (DuckDuckGo, Tavily, Brave, Serper, Serply, Exa, InfoQuest, Tencent Cloud WSA, Firecrawl, fastCRW, GroundRoute, Sofya, You.com)
|
||||||
- `web_fetch` - Fetch web pages (Jina AI, Crawl4AI, Exa, InfoQuest, Firecrawl, fastCRW, GroundRoute, Browserless, Sofya, SpecStream, Unbrowse)
|
- `web_fetch` - Fetch web pages (Jina AI, Crawl4AI, Exa, InfoQuest, Firecrawl, fastCRW, GroundRoute, Browserless, Sofya, Unbrowse)
|
||||||
- `web_capture` - Capture rendered webpage screenshots as artifacts (Browserless)
|
- `web_capture` - Capture rendered webpage screenshots as artifacts (Browserless)
|
||||||
- `image_search` - Search for reference images (DuckDuckGo, InfoQuest, Serper, Brave)
|
- `image_search` - Search for reference images (DuckDuckGo, InfoQuest, Serper, Brave)
|
||||||
- `ls` - List directory contents
|
- `ls` - List directory contents
|
||||||
@ -1382,14 +1330,6 @@ shared (`100.64.0.0/10`, used by CGNAT and Tailscale), other non-global, or
|
|||||||
cloud-metadata addresses; set `allow_private_addresses: true` only when you
|
cloud-metadata addresses; set `allow_private_addresses: true` only when you
|
||||||
intentionally point the tool at an internal target.
|
intentionally point the tool at an internal target.
|
||||||
|
|
||||||
Each Browserless tool entry reads its request timeout from `timeout_s`, with
|
|
||||||
`timeout` as an alias when `timeout_s` is absent. The default is 30 seconds.
|
|
||||||
Zero, negative, and non-finite values fall back to the default with a warning.
|
|
||||||
Strings that cannot be parsed as numbers also trigger this fallback. HTTPX can
|
|
||||||
time out immediately for non-positive values. It raises an error for NaN and
|
|
||||||
applies no deadline for positive infinity. Browserless preserves positive finite
|
|
||||||
values in seconds, including very large values.
|
|
||||||
|
|
||||||
Both `web_fetch` (Browserless provider) and `web_capture` need a running
|
Both `web_fetch` (Browserless provider) and `web_capture` need a running
|
||||||
Browserless instance. You can point `base_url` at [Browserless Cloud](https://www.browserless.io/)
|
Browserless instance. You can point `base_url` at [Browserless Cloud](https://www.browserless.io/)
|
||||||
(set `BROWSERLESS_TOKEN`) or run one locally with Docker:
|
(set `BROWSERLESS_TOKEN`) or run one locally with Docker:
|
||||||
@ -1656,7 +1596,6 @@ sandbox:
|
|||||||
request_timeout: 30 # management request timeout seconds
|
request_timeout: 30 # management request timeout seconds
|
||||||
ready_timeout: 30 # create/readiness timeout seconds
|
ready_timeout: 30 # create/readiness timeout seconds
|
||||||
use_server_proxy: false # proxy execd/file traffic through server
|
use_server_proxy: false # proxy execd/file traffic through server
|
||||||
secure_access: false # per-sandbox endpoint token; Kubernetes + gateway mode only
|
|
||||||
sandbox_timeout: 14400 # remote lifetime; 0 = explicit cleanup
|
sandbox_timeout: 14400 # remote lifetime; 0 = explicit cleanup
|
||||||
bash_command_timeout: 600 # default remote command timeout seconds
|
bash_command_timeout: 600 # default remote command timeout seconds
|
||||||
replicas: 3 # active + warm cap per gateway process
|
replicas: 3 # active + warm cap per gateway process
|
||||||
@ -1665,10 +1604,6 @@ sandbox:
|
|||||||
PYTHONUNBUFFERED: "1"
|
PYTHONUNBUFFERED: "1"
|
||||||
```
|
```
|
||||||
|
|
||||||
`use_server_proxy` and `secure_access` accept YAML booleans. Omitted or `null`
|
|
||||||
values default to `false`. Quoted values such as `"false"` cause a provider
|
|
||||||
startup error.
|
|
||||||
|
|
||||||
Install the optional SDK before selecting this provider:
|
Install the optional SDK before selecting this provider:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@ -1996,7 +1931,7 @@ while WSL resolves it to `/mnt/d/data/report.csv`. WSL-native files can instead
|
|||||||
use `/home/alice/...`. Native Windows paths can use forward slashes as above,
|
use `/home/alice/...`. Native Windows paths can use forward slashes as above,
|
||||||
or single-quoted backslashes (`'D:\data'`). Plain, unquoted YAML scalars also keep
|
or single-quoted backslashes (`'D:\data'`). Plain, unquoted YAML scalars also keep
|
||||||
backslashes literal; only double-quoted scalars interpret backslash escapes.
|
backslashes literal; only double-quoted scalars interpret backslash escapes.
|
||||||
These are path examples; the [local development prerequisites](../../docs/guide/deployment.md#option-2-local-development)
|
These are path examples; the [local development prerequisites](../../README.md#option-2-local-development)
|
||||||
still apply, including the documented WSL limitations of the startup scripts.
|
still apply, including the documented WSL limitations of the startup scripts.
|
||||||
|
|
||||||
**Gateway in Docker:** both bundled Compose files mount the repository's
|
**Gateway in Docker:** both bundled Compose files mount the repository's
|
||||||
@ -2100,7 +2035,6 @@ models:
|
|||||||
- `SERPLY_API_KEY` - [Serply](https://serply.io) key for `web_search` (Google Search, plus Google News and Google Scholar via `vertical`)
|
- `SERPLY_API_KEY` - [Serply](https://serply.io) key for `web_search` (Google Search, plus Google News and Google Scholar via `vertical`)
|
||||||
- `GROUNDROUTE_API_KEY` - GroundRoute meta-search API key for `web_search` and `web_fetch` (routes across Serper, Brave, Exa, Tavily, Firecrawl, Perplexity with gain-share pricing)
|
- `GROUNDROUTE_API_KEY` - GroundRoute meta-search API key for `web_search` and `web_fetch` (routes across Serper, Brave, Exa, Tavily, Firecrawl, Perplexity with gain-share pricing)
|
||||||
- `SOFYA_API_KEY` - [Sofya](https://sofya.co) key for `web_search` and `web_fetch`
|
- `SOFYA_API_KEY` - [Sofya](https://sofya.co) key for `web_search` and `web_fetch`
|
||||||
- `SPECSTREAM_API_KEY` - [SpecStream](https://specstream.ai) key for `web_search` and `web_fetch`
|
|
||||||
- `UNBROWSE_API_KEY` - [Unbrowse](https://unbrowse.ai) key for `web_fetch`
|
- `UNBROWSE_API_KEY` - [Unbrowse](https://unbrowse.ai) key for `web_fetch`
|
||||||
- `YDC_API_KEY` - You.com Search API key for `web_search` (optional: without it the provider uses You.com's keyless free-tier endpoint)
|
- `YDC_API_KEY` - You.com Search API key for `web_search` (optional: without it the provider uses You.com's keyless free-tier endpoint)
|
||||||
- `BROWSERLESS_TOKEN` - Browserless token for `web_fetch` (Browserless provider) and `web_capture`, sent as the `token` query parameter (required by Browserless Cloud and by a self-hosted instance started with `TOKEN`)
|
- `BROWSERLESS_TOKEN` - Browserless token for `web_fetch` (Browserless provider) and `web_capture`, sent as the `token` query parameter (required by Browserless Cloud and by a self-hosted instance started with `TOKEN`)
|
||||||
@ -2112,8 +2046,6 @@ models:
|
|||||||
- `DEER_FLOW_SKILLS_PATH` - Skills directory when `skills.path` is omitted
|
- `DEER_FLOW_SKILLS_PATH` - Skills directory when `skills.path` is omitted
|
||||||
- `GATEWAY_ENABLE_DOCS` - Set to `false` to disable Swagger UI (`/docs`), ReDoc (`/redoc`), and OpenAPI schema (`/openapi.json`) endpoints (default: `true`)
|
- `GATEWAY_ENABLE_DOCS` - Set to `false` to disable Swagger UI (`/docs`), ReDoc (`/redoc`), and OpenAPI schema (`/openapi.json`) endpoints (default: `true`)
|
||||||
|
|
||||||
Search, fetch and capture providers treat an example value that `.env.example` ships (such as `your-jina-api-key`, matched exactly) as unset, in `config.yaml` and in the environment alike: a keyless provider such as the Jina Reader uses its free tier, and a provider that needs a key reports it missing. Any other value is sent as written, so a self-hosted token you chose keeps working. Browserless reads `BROWSERLESS_TOKEN` only when its `token` field is omitted, so a blank or placeholder `token` is not replaced by the variable. `make doctor` applies the same rules and names the variable or field that still holds a placeholder, including `JINA_API_KEY` and `YDC_API_KEY` for the keyless providers.
|
|
||||||
|
|
||||||
## Backend dotenv selection
|
## Backend dotenv selection
|
||||||
|
|
||||||
Set `DEER_FLOW_ENV_FILE` in the backend process environment **before startup**
|
Set `DEER_FLOW_ENV_FILE` in the backend process environment **before startup**
|
||||||
@ -2280,92 +2212,6 @@ only when the agent is expected to edit files or run commands for a trusted
|
|||||||
task.
|
task.
|
||||||
|
|
||||||
|
|
||||||
## HITL Tool Approval
|
|
||||||
|
|
||||||
Set `hitl_approval.enabled: true` to gate bound tools immediately before execution.
|
|
||||||
The default is disabled for existing installations. When enabled, unknown tools,
|
|
||||||
file writes, shell commands, and both personal and deployment MCP tools are
|
|
||||||
reviewed. Concrete host read/navigation tools can run directly; an MCP server's
|
|
||||||
read-only annotation never grants permission.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
hitl_approval:
|
|
||||||
enabled: true
|
|
||||||
default_mode: ask
|
|
||||||
allowed_modes: [ask, auto]
|
|
||||||
review_unknown_tools: true
|
|
||||||
tools: []
|
|
||||||
allow_tools: []
|
|
||||||
deny_tools: []
|
|
||||||
always_ask_tools: []
|
|
||||||
reviewer_model: null
|
|
||||||
reviewer_timeout_seconds: 30
|
|
||||||
reviewer_max_input_bytes: 65536
|
|
||||||
max_payload_bytes: 524288
|
|
||||||
child_approval_timeout_seconds: 300
|
|
||||||
non_interactive: deny
|
|
||||||
max_actions: 32
|
|
||||||
```
|
|
||||||
|
|
||||||
| Mode | Behavior |
|
|
||||||
| --- | --- |
|
|
||||||
| `ask` | Present an approval card for operations requiring review. |
|
|
||||||
| `auto` | Independently assess the concrete operation against user intent. Allow low/medium-risk authorized operations; ask for uncertainty or high risk; deny critical violations. |
|
|
||||||
| `full` | Skip individual approvals within mandatory deployment and account restrictions. |
|
|
||||||
|
|
||||||
Clients select `context.tool_approval_mode` per run from the operator's `allowed_modes`.
|
|
||||||
The default ceiling permits Ask and Auto. Full requires explicitly adding `full`
|
|
||||||
to that operator-owned list; the client selection is not persisted server-side
|
|
||||||
and cannot extend the ceiling.
|
|
||||||
This setting is separate from Flash/Thinking/Pro/Ultra. `deny_tools` blocks in
|
|
||||||
every mode. `always_ask_tools` requires a human in every mode and cannot receive
|
|
||||||
a chat grant. Uninspectable tool implementations cannot receive chat grants.
|
|
||||||
Implementation fingerprints include both execution callables and their callable
|
|
||||||
adapters, so changing an asynchronous implementation requires fresh approval. `allow_tools` is an explicit operator exception. `tools` adds
|
|
||||||
otherwise routine tools to review; disabling `review_unknown_tools` restores a
|
|
||||||
selected-tool policy. These are trusted deployment settings, never model output.
|
|
||||||
|
|
||||||
Auto uses a separate, tool-free model invocation. `reviewer_model` names an
|
|
||||||
existing configured profile; null reuses the effective execution profile. No new
|
|
||||||
provider or credentials are required. Review input contains original user intent,
|
|
||||||
tool identity/schema, and complete arguments, treating tool material as untrusted
|
|
||||||
data. Reviewer callbacks cannot enter the Agent message stream; the host records
|
|
||||||
its token usage. Each review adds model latency and usage. Invalid output,
|
|
||||||
timeout, provider failure, or oversized input requests human approval. In an
|
|
||||||
unattended run these outcomes deny execution. Model review supplements the host's
|
|
||||||
rules and sandbox; it is not a security isolation boundary.
|
|
||||||
|
|
||||||
Cards show complete arguments and support approve, reject with a reason, edit
|
|
||||||
and approve, or allow the exact operation for this chat. Edits are validated
|
|
||||||
against the public tool schema and rerun through mandatory authorization. Injected
|
|
||||||
runtime fields cannot be edited. Grants bind owner, thread incarnation, tool
|
|
||||||
implementation/schema/connection, arguments, and policy. A changed MCP connection
|
|
||||||
requires fresh approval. Revoke grants from the Tool permissions menu. SQL-backed
|
|
||||||
deployments persist grants; memory-only deployments keep them for the process.
|
|
||||||
|
|
||||||
Root pauses use native checkpoints and block fresh runs until resolved. Native
|
|
||||||
parallel interrupts resume by their individual IDs; decisions cannot approve a
|
|
||||||
stale proposal. Root pauses have no time expiry; policy/tool changes fail closed,
|
|
||||||
and disabling the feature still permits the existing all-reject escape hatch.
|
|
||||||
The full card must fit `max_payload_bytes`; oversized cards deny execution rather
|
|
||||||
than hiding part of the operation. `max_actions` bounds calls in a model turn;
|
|
||||||
zero removes that count limit.
|
|
||||||
|
|
||||||
Delegated calls use an owner-scoped live approval endpoint and suspend the original
|
|
||||||
child invocation. The wait expires after `child_approval_timeout_seconds`, or the
|
|
||||||
child's execution deadline if earlier. Cancellation, lost run ownership and worker
|
|
||||||
restart invalidate pending child approvals; they do not replay the delegated task.
|
|
||||||
The UI polls pending child cards while the parent is active. Authenticated session
|
|
||||||
ownership is required to answer these cards. Requests and audit records are deleted
|
|
||||||
with their thread. Scheduled, webhook and other unattended origins cannot open a
|
|
||||||
human wait. The legacy `non_interactive: allow` explicitly bypasses Ask for those
|
|
||||||
root runs; it never overrides mandatory deny/ask rules, Auto uncertainty, or an
|
|
||||||
unresumable child.
|
|
||||||
|
|
||||||
Approval permits an execution attempt. Guardrails, read-before-write rules,
|
|
||||||
sandbox restrictions and tool-account authorization still run afterward.
|
|
||||||
|
|
||||||
|
|
||||||
## Best Practices
|
## Best Practices
|
||||||
|
|
||||||
1. **Place `config.yaml` in project root** - Set `DEER_FLOW_PROJECT_ROOT` if the runtime starts elsewhere
|
1. **Place `config.yaml` in project root** - Set `DEER_FLOW_PROJECT_ROOT` if the runtime starts elsewhere
|
||||||
|
|||||||
@ -180,8 +180,6 @@ sequenceDiagram
|
|||||||
|
|
||||||
Why a string and not a closure: `run_context` is JSON-encoded by the `langgraph_sdk` HTTP client before reaching Gateway. A Python callable does not survive that serialization. The harness side (`_github_env_from_runtime`) already accepts either shape, but only `str` round-trips through the SDK transport.
|
Why a string and not a closure: `run_context` is JSON-encoded by the `langgraph_sdk` HTTP client before reaching Gateway. A Python callable does not survive that serialization. The harness side (`_github_env_from_runtime`) already accepts either shape, but only `str` round-trips through the SDK transport.
|
||||||
|
|
||||||
**App JWT signing**: the mint builds the App JWT in a worker thread, because it reads `GITHUB_APP_PRIVATE_KEY_PATH` and signs with the key. The key is still read on every mint, but it is parsed again only when its PEM text changes (parsing an RSA-2048 key takes tens of milliseconds and holds the GIL). A rotated key, inline or on disk, is used by the next mint without a restart.
|
|
||||||
|
|
||||||
**Token TTL caveat**: GitHub installation tokens are valid for ~1 hour. Most agent runs finish well inside that window. Truly long coder runs (multi-hour refactors at the higher `recursion_limit=250` ceiling) may see a 401 on a late `git push` / `gh pr create`. Auto-refresh past the 1h TTL is intentionally deferred — it requires registering a token-provider lookup on the harness side, which crosses the harness/app boundary (`tests/test_harness_boundary.py`). Until refresh ships, long runs should finish GitHub writes before expiry or accept the loss. If minting fails (bad App id, wrong installation_id, missing private key), the agent still runs without push/write credentials — read-only is better than no response.
|
**Token TTL caveat**: GitHub installation tokens are valid for ~1 hour. Most agent runs finish well inside that window. Truly long coder runs (multi-hour refactors at the higher `recursion_limit=250` ceiling) may see a 401 on a late `git push` / `gh pr create`. Auto-refresh past the 1h TTL is intentionally deferred — it requires registering a token-provider lookup on the harness side, which crosses the harness/app boundary (`tests/test_harness_boundary.py`). Until refresh ships, long runs should finish GitHub writes before expiry or accept the loss. If minting fails (bad App id, wrong installation_id, missing private key), the agent still runs without push/write credentials — read-only is better than no response.
|
||||||
|
|
||||||
## Thread-create Race Recovery
|
## Thread-create Race Recovery
|
||||||
|
|||||||
@ -27,12 +27,6 @@ Because selecting an agent always creates a new thread instead of mutating the c
|
|||||||
|
|
||||||
`/goal <condition>` sets a goal on the chat's conversation and starts a run with the condition as the task; `/goal clear` clears it. `/goal` alone reports what the web goal bar shows: the goal, `In progress.` while a run is pending or running, and once auto-continue has stood down, why (for example `Stopped: the continuation limit was reached.` or `Waiting for you: it needs your input.`). A met goal is cleared and recorded in the thread state, so `/goal` answers `Goal met: <goal>` until the next goal change. Reasons come only from host-written codes; the goal checker's notes and provider errors are never sent.
|
`/goal <condition>` sets a goal on the chat's conversation and starts a run with the condition as the task; `/goal clear` clears it. `/goal` alone reports what the web goal bar shows: the goal, `In progress.` while a run is pending or running, and once auto-continue has stood down, why (for example `Stopped: the continuation limit was reached.` or `Waiting for you: it needs your input.`). A met goal is cleared and recorded in the thread state, so `/goal` answers `Goal met: <goal>` until the next goal change. Reasons come only from host-written codes; the goal checker's notes and provider errors are never sent.
|
||||||
|
|
||||||
`/goal from-chat [hint]` composes the goal from the conversation instead of taking the text literally. The Gateway reads the thread summary plus recent agreement turns — visible user/assistant text, the assistant's clarification questions, and the user's answers to clarification cards — asks a non-thinking model (the deployment default; the goal evaluator instead uses the run's recorded model) to preserve the constraints and stopping condition agreed in the chat, and saves the objective through the existing goal route; the manager then sends the saved objective as its own final message and starts exactly one run through the ordinary chat path. An optional hint after the command adds a constraint without restating the whole goal. Evidence tool output, subagent chatter and attachments are not part of the synthesis context: they inform execution, not the objective. If the conversation does not yield a clear goal, nothing is saved or started and the reply explains why; a conversation whose last turn is an unanswered human-input request (a clarification, a sandbox-network approval, another `human_input_request`) is reported as pending instead of being synthesized, because the agreement is incomplete; a busy thread is rejected before the model call when this Gateway worker owns the run, while a run admitted by another worker is still rejected by the save-time reservation (so the model call may already have happened). Because the saved objective is shown first, a run that cannot start is followed by the channel's busy or error reply. Like `PUT`/`DELETE /goal`, the save is rejected while a tool approval is pending, so a new goal cannot silently discard it. Model-generated failure reasons are never sent back to the chat.
|
|
||||||
|
|
||||||
The synthesis model call is subject to the same model-use authorization as other model-backed routes: a role denied the deployment default model gets the route's 403 and the conversation is never sent to that model. The save is a compare-and-swap against the checkpoint the synthesis read — a goal or checkpoint write that lands during the model call rejects the stale objective with a 409, and the channel replies that the conversation changed and nothing was saved. If the chat switches conversations (`/new`) while synthesis is in flight, the goal stays saved in the original thread but the manager does not start a run in the new conversation; the reply says so and carries the saved objective.
|
|
||||||
|
|
||||||
Run handoff stays pinned to the thread where the goal was saved and checks the conversation binding again after confirmation delivery and run preparation. A rejected start gets an explicit goal-saved, task-not-started reply. If the response is lost after submission, the reply says the task may still be running and asks the user to check the conversation in DeerFlow before retrying.
|
|
||||||
|
|
||||||
DeerFlow's own replies here and to `/connect` use the binding owner's web UI language (the `locale` user preference), else `channel_connections.notification_locale` (`en-US` by default). A reply with no known owner, such as one to an invalid or expired code, uses the configured default. Agent replies are not translated.
|
DeerFlow's own replies here and to `/connect` use the binding owner's web UI language (the `locale` user preference), else `channel_connections.notification_locale` (`en-US` by default). A reply with no known owner, such as one to an invalid or expired code, uses the configured default. Agent replies are not translated.
|
||||||
|
|
||||||
Connect codes are deliberately **bind-time defenses**, not chat-time defenses. After binding, ordinary `allowed_users` continue to gate regular messages exactly as before.
|
Connect codes are deliberately **bind-time defenses**, not chat-time defenses. After binding, ordinary `allowed_users` continue to gate regular messages exactly as before.
|
||||||
@ -193,16 +187,6 @@ graph TB
|
|||||||
Msg --> Stream3 --> Stream --> Mid1 --> Mid2 --> Final
|
Msg --> Stream3 --> Stream --> Mid1 --> Mid2 --> Final
|
||||||
```
|
```
|
||||||
|
|
||||||
Feishu rejects a card whose request body exceeds 30 KB (`230025`). The channel
|
|
||||||
budgets card text at 24 KB of that body, measured as the bytes the text adds
|
|
||||||
after both JSON escapes, and sends CJK unescaped (3 bytes per character, about
|
|
||||||
8,000 per card). A streamed update that no longer fits shows the part that does,
|
|
||||||
ending in `…`. A longer final answer fills the running card and continues in up
|
|
||||||
to 10 cards replied to the same source message, split at line breaks; past that
|
|
||||||
the last card is clipped and marked `... (truncated)`. A retry of a partially
|
|
||||||
delivered answer resumes after the cards already sent, and a reply to any of the
|
|
||||||
cards continues the conversation.
|
|
||||||
|
|
||||||
For the special GitHub case (`fire_and_forget=True` channel policy), the manager calls `runs.create()` and returns once the run is `pending` — no outbound reply, because GitHub agents post via the `gh` CLI from inside their sandbox. See [GITHUB_AGENTS.md](GITHUB_AGENTS.md) for the full GitHub flow.
|
For the special GitHub case (`fire_and_forget=True` channel policy), the manager calls `runs.create()` and returns once the run is `pending` — no outbound reply, because GitHub agents post via the `gh` CLI from inside their sandbox. See [GITHUB_AGENTS.md](GITHUB_AGENTS.md) for the full GitHub flow.
|
||||||
|
|
||||||
## Attachment delivery feedback
|
## Attachment delivery feedback
|
||||||
@ -844,12 +828,6 @@ chat identity. That compatibility applies only to the old group-binding shape;
|
|||||||
a new private binding never authorizes messages from an unbound group. Bind a
|
a new private binding never authorizes messages from an unbound group. Bind a
|
||||||
specific group again with a name to make it available as a notification target.
|
specific group again with a name to make it available as a notification target.
|
||||||
|
|
||||||
Each WeCom group is its own conversation, with one DeerFlow thread per member;
|
|
||||||
a private chat is keyed by the sender alone. A group bound with its own
|
|
||||||
`/connect` keeps the thread it had before groups got their own conversation. A
|
|
||||||
legacy group binding covers every group of the bot, so its earlier thread is
|
|
||||||
not continued in any of them; each group starts a new one.
|
|
||||||
|
|
||||||
### WeChat private notifications
|
### WeChat private notifications
|
||||||
|
|
||||||
Use the existing iLink QR/bot setup and bind in a private chat. Send one private
|
Use the existing iLink QR/bot setup and bind in a private chat. Send one private
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user