deer-flow/examples/deerflow-extension-lightrag/VALIDATION.md

Local Phase 1 integration validation — 2026-10-20

The new Phase 1 retrieval chain was exercised against real local services on Apple Silicon macOS with OrbStack. The corpus contains explicitly synthetic Chinese people, organizations, project relations and policy text. GLM-5.3 received only these synthetic fixtures and integration prompts.

Deployment

• LightRAG core 1.5.7, API 0344; official ghcr.io/hkuds/lightrag:v1.5.7 arm64 image pinned to sha256:04716d2f10557342e033395c6490903ed0849a41ac5f89008736a7383e8bad7a. API published at 127.0.0.1:19621, with an explicit API key.
• WeKnora v0.8.3, reusing the document-only local validation stack described in its validation record. API: 127.0.0.1:18082; upstream UI: 127.0.0.1:18081.
• Native Ollama 0.40.4, local bge-m3, 1024 embedding dimensions. Both container connectivity and a two-input embedding request were verified.
• GLM glm-5.3, OpenAI-compatible endpoint configured in local .env.local. Provider credentials and resource IDs are outside the repository. LightRAG connects to native embeddings through host.docker.internal:11434.
• LightRAG uses its file-backed stores: NetworkX graph, vector and JSON stores persisted in dedicated named volumes. No prebuilt mock graph was loaded.

Synthetic data and processing

A new published WeKnora Wiki entry states that Lin Lan (林岚) leads the Cedar project (青杉计划), funded by Star River Laboratory (星河实验室). A dedicated retrieve-only key is restricted to this Wiki base and the existing synthetic document base. The original document fixtures were retained.

LightRAG ingested synthetic-cedar-relations.md through POST /documents/text. GLM extracted 5 entities and 5 relations; bge-m3 embedded them and the one original text chunk. Upstream status reached processed with chunks_count=1. Actual relations include the project lead, funding and employment connections.

The initial extraction with default GLM reasoning remained in processing for several minutes. The test pipeline was cancelled and the container recreated with OPENAI_LLM_EXTRA_BODY='{"thinking":{"type":"disabled"}}', OPENAI_LLM_MAX_TOKENS=6000 and ENTITY_EXTRACTION_USE_JSON=true. Recovery of the same document completed successfully. No provider adapter code or source mapping was changed to make the live responses pass. These settings are local validation choices, not new DeerFlow defaults.

Observed results

Check	Result
Existing WeKnora document retrieval live regression	1 passed
Published Wiki retrieval with restricted key and selected base	Passed
Wiki unbound user and unsupported document filtering	Rejected
Real LightRAG relation retrieval from the generated graph	Passed
Relation source_id joins to actual returned original chunks	Attributable document evidence returned
LightRAG unbound user, selected scope and disabled scope	Rejected before retrieval
Real GLM document + Wiki tool loop	Both tools invoked; 3 validated cited sources
Real GLM document + Wiki + relation tool loop	All three tools invoked and each supplied attributable evidence
Final answer source handles	Every cited handle belongs to actual tool evidence
JSON history serialization and replay	Source records preserved
Shared source forwarding and pruning	Cited evidence retained; removed evidence drops sources
New opt-in live test module including GLM	3 passed in 23.67 seconds
Focused offline provider/tool tests with live opt-ins absent	81 passed, 3 skipped

The generative test executes a real LangChain agent loop using DeerFlow's StructuredTools, extension snapshot, trusted runtime context, shared source formatter and KnowledgeScopeMiddleware. Tool calls are generated by GLM, not inserted as predetermined AIMessage calls.

Reproduction

Set DEER_FLOW_RUN_LIVE_TESTS=1 and supply:

• Wiki: WEKNORA_BASE_URL, WEKNORA_TENANT_ID, WEKNORA_WIKI_BASE_ID, WEKNORA_API_KEY, WEKNORA_WIKI_QUERY.
• Graph: LIGHTRAG_BASE_URL, LIGHTRAG_API_KEY, LIGHTRAG_RELATION_QUERY.
• Optional generative check: KNOWLEDGE_RUN_MODEL_TEST=1, E2E_LLM_MODEL, E2E_LLM_BASE_URL, E2E_LLM_API_KEY, WEKNORA_KNOWLEDGE_BASE_ID and KNOWLEDGE_MODEL_QUESTION. The question should cover attributable facts in all three existing fixtures. The WeKnora key must read both configured bases.

From the repository root:

uv run --no-sync --project backend pytest \
  backend/tests/test_knowledge_phase1_live.py -m live -q --override-ini=addopts=

Tests are read-only and require already indexed fixtures; they do not upload, create pages, or change upstream credentials. The generative check has a separate opt-in and incurs model API usage. Without it, only the provider checks run. No credentials or raw API responses should be added to this record.

Real DeerFlow browser acceptance

On 2026-10-20 the current checkout was started with the actual Gateway, Next.js frontend and an OrbStack nginx ingress at loopback port 2026. An isolated local runtime home, SQLite persistence and the local auth-disabled synthetic default user were used. Both providers bind that exact user. The browser submitted real chat messages through the normal streaming runtime; no browser API mocks were used.

Session: a598bdcd-1acc-49e8-adcd-70f95b704c00.

Browser check	Observed result
First real chat	All three requested tools called; completed in 26 seconds
Document facts	680 RMB/night and 15 calendar days, correct clickable document citation
Relation facts	林岚 leads 青杉计划; 星河实验室 funds it; citation opens the actual original chunk
Initial Wiki query	Model expanded query to 青杉 计划 负责人 资助机构 林岚 星河实验室; zero hits, honestly reported
Follow-up Wiki query	Exact 青杉 returned the published entry; completed in 9 seconds
Wiki citation	Opens provider/base and summary containing CEDAR-WIKI-2026
Browser refresh	Document, Wiki and relation citation buttons all reopen their original evidence
Persisted Gateway state	Actual tool calls and source artifacts retained (document 2, relation 5, Wiki follow-up 1)

Core retrieval, citation clicks and browser history replay passed. This is not an unqualified chat-UX pass: this GLM endpoint emitted English reasoning text and a closing </think> in ordinary message content, visible in the browser, even with the local thinking.type=disabled setting. Wiki keyword expansion also needed a user follow-up to get a hit. Neither behavior was hidden or mocked away. The mixed-provider configuration does not advertise catalog/dataset selection, so the UI hides its knowledge scope selector; disabled scope was not exercised through this browser session (the earlier tool-level checks still apply).

Local reproduction and evidence (outside version control): start-browser.py, browser-state.json, and browser-{document,wiki,graph}-replay.jpg. The launcher uses private fixture credentials and .env.local; do not commit them. To reproduce the interaction, ask all three tools for the Cedar reimbursement, lead and funding facts, follow up with an exact Wiki query 青杉, open each source, refresh the chat URL and reopen each source.

Browser fixes and re-acceptance

The two chat issues above were reproduced and fixed on 2026-10-20:

• WeKnora's lexical Wiki endpoint returned no pages for the complete query 青杉 计划 负责人 资助机构 林岚 星河实验室. The extension now retries an empty successful result with up to four distinct whitespace-separated keywords of at least two characters, keeping the same authorized KB and credentials. Duplicate title/excerpt pairs are merged. Schema, authorization and HTTP errors still fail immediately; nonempty original results do not trigger fallback.
• The local GLM-compatible proxy ignores the native thinking.type=disabled switch. Direct synthetic probes reproduced reasoning embedded in content with that payload. The same endpoint returned clean content with the chat-template switch below. The local browser launcher and model live-test runner now use it; no frontend text stripping or global change to official GLM profiles was made.

# For the tested proxy's glm-5.3 deployment, not a universal GLM default.
extra_body:
  chat_template_kwargs:
    enable_thinking: false

New browser session: 81bf7e33-d0a2-4cd4-9dfe-f0d8b2cd1ee5. The first turn explicitly used the complete previously failing Wiki query, called all three tools, and returned all four correct facts with three actual source citations, in 26 seconds. The Wiki source opened its published summary. After refresh the citation reopened with the same evidence. A second turn used that same long query and returned a clean one-sentence answer with a real Wiki citation in 5 seconds. Neither turn displayed English reasoning or think tags. Persisted Gateway messages independently confirm the long query, Wiki evidence in both turns, clean assistant content, and all three cited source handles.

• Wiki adapter regressions: 65 passed, including bounded retries, deduplication, nonempty-result preservation, single/short terms, and fail-closed errors.
• Provider/model factory focused run: 243 passed, 3 skipped (before the last three additional keyword edge cases).
• Strict blocking-I/O suite after the fix: 264 passed, 1 skipped.
• Full backend regression: 34,543 passed, 669 skipped, 1 failed across four shards. The failure is the previously reproduced Node reporter acceptance case failing-todo-failure-name-color-spec; it also fails in isolation. A TUI timing failure observed during the baseline passed both isolation and rerun.
• Real provider + model live checks: 3 passed in 26.79 seconds, using the long Wiki query and clean-answer assertion.
• Live model test accepts optional JSON E2E_LLM_EXTRA_BODY; set KNOWLEDGE_EXPECT_CLEAN_ANSWER=1 to reject inline think tags in assistant content.

Local evidence: browser-fixed-state.json, browser-fixed-final.jpg, browser-fixed-replay.jpg, probe-thinking.py, and the corrected start-browser.py/browser-config.yaml under the local validation directory. The two previously reported issues pass this re-acceptance. Historical messages from the earlier session are retained as originally generated.

Limits

This validates provider retrieval, real model/tool/evidence execution and local browser chat replay. It is not a concurrency/load test or production authorization audit. Source forwarding is tested through the shared helper; no second generative subagent was launched in this live run.

Relation evidence consists of upstream one-edge paths. Multi-hop discovery, graph completeness, WeKnora entity-graph HTTP access, graph visualization and management pipelines remain outside this delivery. Upstream extraction during fixture preparation does not implement DeerFlow's planned Phase 2 pipeline.

Shared-evidence budget review follow-up

The adapter now emits each referenced chunk ID once, before its first edge, and labels every edge with the corresponding result positions. A regression with 20 edges sharing a long chunk preserves all 20 edges under the shared 8,000-character budget with one source record. A second regression covers partially shared chunks, missing IDs and prefix truncation without dangling references. Graph descriptions remain unattributed; only original chunk text receives a source citation.

Validation after this change: relation adapter tests 10 passed; real WeKnora/LightRAG/GLM checks 3 passed in 22.80 seconds after restarting the local provider services; blocking-I/O 266 passed, 1 skipped. Extension catalog E2E 10 passed, including English/Chinese LightRAG descriptions and catalog discovery with the runtime API unavailable. The old E2E expectation of 10 bundled extensions was updated to 11.

Hit-budget boundary follow-up — 2026-10-21

The last retained edge now says Evidence omitted for this edge: bounded view. when the 98-hit threshold prevented its first valid chunk from being emitted. Missing/unmatched chunk IDs retain the distinct no-retrieved-evidence wording. A 50-edge/50-chunk regression fails before the change and passes afterwards. The repeated test-loop re import was moved to module scope.

The preceding remote head passed all four backend shards, frontend unit tests, blocking-I/O, lint, package checks and guidance validation. Both browser CI jobs failed before tests because ubuntu-latest rolled to Ubuntu 26.04, which the locked Playwright 1.59.1 cannot install Chromium dependencies on. Those two jobs now explicitly use Ubuntu 24.04, with a workflow regression checking both runner selection and preservation of the Chromium dependency-install step.

Runner regression correction — 2026-10-22

Maintainers restored both browser jobs to ubuntu-latest; all four E2E shards and full-stack replay passed remotely at 7e7710d03. The earlier backend regression still pinned Ubuntu 24.04 and was the sole failure in backend shards 1 and 3. It now checks Chromium dependency installation precedes browser test execution, without imposing a runner version. The maintainer-selected runner and E2E sharding remain unchanged.