deer-flow/examples/deerflow-extension-lightrag/VALIDATION.md

# Local Phase 1 integration validation — 2026-10-20

The new Phase 1 retrieval chain was exercised against real local services on
Apple Silicon macOS with OrbStack. The corpus contains explicitly synthetic
Chinese people, organizations, project relations and policy text. GLM-5.3
received only these synthetic fixtures and integration prompts.

## Deployment

- LightRAG core **1.5.7**, API **0344**; official `ghcr.io/hkuds/lightrag:v1.5.7`
  arm64 image pinned to
  `sha256:04716d2f10557342e033395c6490903ed0849a41ac5f89008736a7383e8bad7a`.
  API published at `127.0.0.1:19621`, with an explicit API key.
- WeKnora **v0.8.3**, reusing the document-only local validation stack described
  in [its validation record](../deerflow-extension-weknora/VALIDATION.md).
  API: `127.0.0.1:18082`; upstream UI: `127.0.0.1:18081`.
- Native Ollama **0.40.4**, local **bge-m3**, 1024 embedding dimensions.
  Both container connectivity and a two-input embedding request were verified.
- GLM **glm-5.3**, OpenAI-compatible endpoint configured in local `.env.local`.
  Provider credentials and resource IDs are outside the repository. LightRAG
  connects to native embeddings through `host.docker.internal:11434`.
- LightRAG uses its file-backed stores: NetworkX graph, vector and JSON stores
  persisted in dedicated named volumes. No prebuilt mock graph was loaded.

## Synthetic data and processing

A new published WeKnora Wiki entry states that Lin Lan (林岚) leads the Cedar
project (青杉计划), funded by Star River Laboratory (星河实验室). A dedicated
`retrieve`-only key is restricted to this Wiki base and the existing synthetic
document base. The original document fixtures were retained.

LightRAG ingested `synthetic-cedar-relations.md` through `POST /documents/text`.
GLM extracted **5 entities and 5 relations**; bge-m3 embedded them and the one
original text chunk. Upstream status reached `processed` with `chunks_count=1`.
Actual relations include the project lead, funding and employment connections.

The initial extraction with default GLM reasoning remained in processing for
several minutes. The test pipeline was cancelled and the container recreated
with `OPENAI_LLM_EXTRA_BODY='{"thinking":{"type":"disabled"}}'`,
`OPENAI_LLM_MAX_TOKENS=6000` and `ENTITY_EXTRACTION_USE_JSON=true`. Recovery of
the same document completed successfully. No provider adapter code or source
mapping was changed to make the live responses pass. These settings are local
validation choices, not new DeerFlow defaults.

## Observed results

| Check | Result |
| --- | --- |
| Existing WeKnora document retrieval live regression | 1 passed |
| Published Wiki retrieval with restricted key and selected base | Passed |
| Wiki unbound user and unsupported document filtering | Rejected |
| Real LightRAG relation retrieval from the generated graph | Passed |
| Relation `source_id` joins to actual returned original chunks | Attributable document evidence returned |
| LightRAG unbound user, selected scope and disabled scope | Rejected before retrieval |
| Real GLM document + Wiki tool loop | Both tools invoked; 3 validated cited sources |
| Real GLM document + Wiki + relation tool loop | All three tools invoked and each supplied attributable evidence |
| Final answer source handles | Every cited handle belongs to actual tool evidence |
| JSON history serialization and replay | Source records preserved |
| Shared source forwarding and pruning | Cited evidence retained; removed evidence drops sources |
| New opt-in live test module including GLM | **3 passed in 23.67 seconds** |
| Focused offline provider/tool tests with live opt-ins absent | **81 passed, 3 skipped** |

The generative test executes a real LangChain agent loop using DeerFlow's
StructuredTools, extension snapshot, trusted runtime context, shared source
formatter and `KnowledgeScopeMiddleware`. Tool calls are generated by GLM,
not inserted as predetermined AIMessage calls.

## Reproduction

Set `DEER_FLOW_RUN_LIVE_TESTS=1` and supply:

- Wiki: `WEKNORA_BASE_URL`, `WEKNORA_TENANT_ID`, `WEKNORA_WIKI_BASE_ID`,
  `WEKNORA_API_KEY`, `WEKNORA_WIKI_QUERY`.
- Graph: `LIGHTRAG_BASE_URL`, `LIGHTRAG_API_KEY`, `LIGHTRAG_RELATION_QUERY`.
- Optional generative check: `KNOWLEDGE_RUN_MODEL_TEST=1`, `E2E_LLM_MODEL`,
  `E2E_LLM_BASE_URL`, `E2E_LLM_API_KEY`, `WEKNORA_KNOWLEDGE_BASE_ID` and
  `KNOWLEDGE_MODEL_QUESTION`. The question should cover attributable facts in
  all three existing fixtures. The WeKnora key must read both configured bases.

From the repository root:

```sh
uv run --no-sync --project backend pytest \
  backend/tests/test_knowledge_phase1_live.py -m live -q --override-ini=addopts=
```

Tests are read-only and require already indexed fixtures; they do not upload,
create pages, or change upstream credentials. The generative check has a
separate opt-in and incurs model API usage. Without it, only the provider checks
run. No credentials or raw API responses should be added to this record.

## Real DeerFlow browser acceptance

On 2026-10-20 the current checkout was started with the actual Gateway, Next.js
frontend and an OrbStack nginx ingress at loopback port 2026. An isolated local
runtime home, SQLite persistence and the local auth-disabled synthetic `default`
user were used. Both providers bind that exact user. The browser submitted real
chat messages through the normal streaming runtime; no browser API mocks were used.

Session: `a598bdcd-1acc-49e8-adcd-70f95b704c00`.

| Browser check | Observed result |
| --- | --- |
| First real chat | All three requested tools called; completed in 26 seconds |
| Document facts | 680 RMB/night and 15 calendar days, correct clickable document citation |
| Relation facts | 林岚 leads 青杉计划; 星河实验室 funds it; citation opens the actual original chunk |
| Initial Wiki query | Model expanded query to `青杉 计划 负责人 资助机构 林岚 星河实验室`; zero hits, honestly reported |
| Follow-up Wiki query | Exact `青杉` returned the published entry; completed in 9 seconds |
| Wiki citation | Opens provider/base and summary containing `CEDAR-WIKI-2026` |
| Browser refresh | Document, Wiki and relation citation buttons all reopen their original evidence |
| Persisted Gateway state | Actual tool calls and source artifacts retained (document 2, relation 5, Wiki follow-up 1) |

Core retrieval, citation clicks and browser history replay passed. This is **not
an unqualified chat-UX pass**: this GLM endpoint emitted English reasoning text
and a closing `</think>` in ordinary message content, visible in the browser,
even with the local `thinking.type=disabled` setting. Wiki keyword expansion also
needed a user follow-up to get a hit. Neither behavior was hidden or mocked away.
The mixed-provider configuration does not advertise catalog/dataset selection,
so the UI hides its knowledge scope selector; disabled scope was not exercised
through this browser session (the earlier tool-level checks still apply).

Local reproduction and evidence (outside version control):
`start-browser.py`,
`browser-state.json`, and `browser-{document,wiki,graph}-replay.jpg`.
The launcher uses private fixture credentials and `.env.local`; do not commit them.
To reproduce the interaction, ask all three tools for the Cedar reimbursement,
lead and funding facts, follow up with an exact Wiki query `青杉`, open each source,
refresh the chat URL and reopen each source.

## Browser fixes and re-acceptance

The two chat issues above were reproduced and fixed on 2026-10-20:

- WeKnora's lexical Wiki endpoint returned no pages for the complete query
  `青杉 计划 负责人 资助机构 林岚 星河实验室`. The extension now retries an
  empty successful result with up to four distinct whitespace-separated keywords
  of at least two characters, keeping the same authorized KB and credentials.
  Duplicate title/excerpt pairs are merged. Schema, authorization and HTTP errors
  still fail immediately; nonempty original results do not trigger fallback.
- The local GLM-compatible proxy ignores the native `thinking.type=disabled`
  switch. Direct synthetic probes reproduced reasoning embedded in content with
  that payload. The same endpoint returned clean content with the chat-template
  switch below. The local browser launcher and model live-test runner now use it;
  no frontend text stripping or global change to official GLM profiles was made.

```yaml
# For the tested proxy's glm-5.3 deployment, not a universal GLM default.
extra_body:
  chat_template_kwargs:
    enable_thinking: false
```

New browser session: `81bf7e33-d0a2-4cd4-9dfe-f0d8b2cd1ee5`.
The first turn explicitly used the complete previously failing Wiki query,
called all three tools, and returned all four correct facts with **three actual
source citations**, in 26 seconds. The Wiki source opened its published summary.
After refresh the citation reopened with the same evidence. A second turn used
that same long query and returned a clean one-sentence answer with a real Wiki
citation in 5 seconds. Neither turn displayed English reasoning or think tags.
Persisted Gateway messages independently confirm the long query, Wiki evidence
in both turns, clean assistant content, and all three cited source handles.

- Wiki adapter regressions: **65 passed**, including bounded retries, deduplication,
  nonempty-result preservation, single/short terms, and fail-closed errors.
- Provider/model factory focused run: **243 passed, 3 skipped** (before the last
  three additional keyword edge cases).
- Strict blocking-I/O suite after the fix: **264 passed, 1 skipped**.
- Full backend regression: **34,543 passed, 669 skipped, 1 failed** across four
  shards. The failure is the previously reproduced Node reporter acceptance
  case `failing-todo-failure-name-color-spec`; it also fails in isolation. A TUI
  timing failure observed during the baseline passed both isolation and rerun.
- Real provider + model live checks: **3 passed in 26.79 seconds**, using the long
  Wiki query and clean-answer assertion.
- Live model test accepts optional JSON `E2E_LLM_EXTRA_BODY`; set
  `KNOWLEDGE_EXPECT_CLEAN_ANSWER=1` to reject inline think tags in assistant content.

Local evidence: `browser-fixed-state.json`, `browser-fixed-final.jpg`,
`browser-fixed-replay.jpg`, `probe-thinking.py`, and the corrected
`start-browser.py`/`browser-config.yaml` under the local validation directory.
The two previously reported issues pass this re-acceptance. Historical messages
from the earlier session are retained as originally generated.

## Limits

This validates provider retrieval, real model/tool/evidence execution and local
browser chat replay. It is not a concurrency/load test or production
authorization audit. Source forwarding is tested through the shared helper;
no second generative subagent was launched in this live run.

Relation evidence consists of upstream one-edge paths. Multi-hop discovery,
graph completeness, WeKnora entity-graph HTTP access, graph visualization and
management pipelines remain outside this delivery. Upstream extraction during
fixture preparation does not implement DeerFlow's planned Phase 2 pipeline.

## Shared-evidence budget review follow-up

The adapter now emits each referenced chunk ID once, before its first edge,
and labels every edge with the corresponding result positions. A regression
with 20 edges sharing a long chunk preserves all 20 edges under the shared
8,000-character budget with one source record. A second regression covers
partially shared chunks, missing IDs and prefix truncation without dangling
references. Graph descriptions remain unattributed; only original chunk text
receives a source citation.

Validation after this change: relation adapter tests **10 passed**; real
WeKnora/LightRAG/GLM checks **3 passed in 22.80 seconds** after restarting the
local provider services; blocking-I/O **266 passed, 1 skipped**. Extension
catalog E2E **10 passed**, including English/Chinese LightRAG descriptions and
catalog discovery with the runtime API unavailable. The old E2E expectation
of 10 bundled extensions was updated to 11.

## Hit-budget boundary follow-up — 2026-10-21

The last retained edge now says `Evidence omitted for this edge: bounded view.`
when the 98-hit threshold prevented its first valid chunk from being emitted.
Missing/unmatched chunk IDs retain the distinct no-retrieved-evidence wording.
A 50-edge/50-chunk regression fails before the change and passes afterwards.
The repeated test-loop `re` import was moved to module scope.

The preceding remote head passed all four backend shards, frontend unit tests,
blocking-I/O, lint, package checks and guidance validation. Both browser CI jobs
failed before tests because ubuntu-latest rolled to Ubuntu 26.04, which the
locked Playwright 1.59.1 cannot install Chromium dependencies on. Those two
jobs now explicitly use Ubuntu 24.04, with a workflow regression checking both
runner selection and preservation of the Chromium dependency-install step.

## Runner regression correction — 2026-10-22

Maintainers restored both browser jobs to `ubuntu-latest`; all four E2E shards
and full-stack replay passed remotely at `7e7710d03`. The earlier backend
regression still pinned Ubuntu 24.04 and was the sole failure in backend shards
1 and 3. It now checks Chromium dependency installation precedes browser test
execution, without imposing a runner version. The maintainer-selected runner
and E2E sharding remain unchanged.