mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
feat(knowledge): add WeKnora read-only extension (phase 3) (#6950)
* feat(knowledge): add WeKnora read-only extension (phase 3) * fix(weknora): complete extension catalog and harden response validation * test(extensions): cover WeKnora discovery in browser catalog * docs(weknora): explain extension and MCP use cases * docs(weknora): correct guide language and sync Chinese overview --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Xuehao Xu
GitHub
parent
158c54508f
commit
c056ee556a
@ -1440,8 +1440,12 @@ are rejected without truncation.
|
||||
RAGFlow keeps its existing configuration and citations, and search-only
|
||||
Providers remain usable without a catalog or management page. See the
|
||||
[offline Provider example](examples/deerflow-extension-knowledge/README.md)
|
||||
for extension registration and configuration. WeKnora's adapter is planned for
|
||||
the next phase.
|
||||
for extension registration and configuration. The installable
|
||||
[WeKnora read-only extension](examples/deerflow-extension-weknora/README.md)
|
||||
targets WeKnora v0.8.3 and adds knowledge-base/document selection and shared
|
||||
retrieval citations. Configure explicit user-to-workspace credential bindings;
|
||||
resource authorization remains with WeKnora. The extension directory includes
|
||||
its installation guide.
|
||||
|
||||
Skills are what make DeerFlow do *almost anything*.
|
||||
|
||||
|
||||
@ -727,6 +727,11 @@ DeerFlow 可连接租户级 RAGFlow，并通过 `knowledge_search` 按 embedding
|
||||
|
||||
本版不在工作区侧边栏增加独立的“知识库”入口，也不提供 DeerFlow 知识库管理页面；知识库和文件的创建、上传、解析与删除仍直接在 RAGFlow 中完成。
|
||||
|
||||
运维还可独立安装 [WeKnora 只读扩展](examples/deerflow-extension-weknora/README.md)，
|
||||
对接 WeKnora v0.8.3，使用知识库/文档范围选择器和共享检索引用。
|
||||
需显式配置 DeerFlow 用户到 WeKnora 工作空间的凭据绑定；资源授权仍由 WeKnora 执行。
|
||||
安装与配置步骤见扩展目录中的英文指南。
|
||||
|
||||
Gateway 生成后续建议时，现在会先把普通字符串输出和 block/list 风格的富文本内容统一归一化，再去解析 JSON 数组响应，因此不同 provider 的内容包装方式不会再悄悄把建议吞掉。
|
||||
|
||||
Web UI 支持从已完成的 assistant 回复分叉出一个新的主对话。自动继承的分叉标题会使用下一个空闲的数字后缀（`标题 (2)`、`标题 (3)`……）；显式指定或手动重命名得到的同名后缀也会占号，即使它没有生成序号 metadata，后续自动分叉也不会与它重名。API 调用方显式提供的标题保持不变；重命名会清除旧的生成序号，因此从新标题继续自动分叉时会重新从 `(2)` 开始。最近对话列表还会把已加载的分叉直接排列在已加载的父对话下方，并显示低干扰的树形连接线。父对话尚未加载、谱系数据错误或成环、父子置顶状态不一致时，分叉会安全地保留在顶层，不会被隐藏或跨越置顶边界移动。新 thread 会保留该轮回复的 checkpoint 以及用户消息之前的重放 checkpoint，因此分叉后可以立即重新生成该回复。对于缺少 checkpoint 父链接的旧历史或导入历史，Gateway 会进行有界的时间顺序查找；如果不存在更早的重放 checkpoint，分叉仍会按旧版单-checkpoint 形态成功创建，但无法重新生成继承的回复。已有的单-checkpoint 分叉会保持不变，不会通过不安全的 checkpoint 复制尝试修复。只有从最新回合分叉时才会尽力复制当前 thread 的工作区文件；从历史回合分叉不会带入后续时间线创建的文件。
|
||||
|
||||
@ -1,11 +1,13 @@
|
||||
# Read-only knowledge contract (#6269, phases 1–2)
|
||||
# Read-only knowledge contract (#6269, phases 1–3)
|
||||
|
||||
`deerflow.knowledge` owns `RetrievalResult`, `RetrievalHit`, `KnowledgeSource`,
|
||||
`KnowledgeCapabilities`, and the search-only `RetrievalProvider` protocol.
|
||||
Search-only providers need no knowledge management service, resource store,
|
||||
catalog, or scope-filter implementation. Scope admission, the read-only catalog
|
||||
and selection UI consume the same operation-level capabilities. LightRAG still
|
||||
does not advertise dataset/document filters; WeKnora remains phase 3.
|
||||
does not advertise dataset/document filters. The phase 3
|
||||
[WeKnora extension](../../examples/deerflow-extension-weknora/README.md) implements
|
||||
all six read-only operations against the v0.8.3 API.
|
||||
|
||||
## Provider registration and optional operations
|
||||
|
||||
@ -35,10 +37,10 @@ provider identifiers, including non-UUID identifiers. Gateway validates this
|
||||
bounded public projection and never returns raw provider payloads.
|
||||
Dataset and document IDs are limited to **256 Unicode code points** throughout
|
||||
catalog output, URL parameters and scope snapshots. Overlong IDs are rejected,
|
||||
never truncated. A phase 3 WeKnora adapter must map space-qualified resources
|
||||
to stable, unambiguous IDs within this bound and resolve them inside the
|
||||
provider; the earlier `ks1_` encoding allowing 512 characters is not the public
|
||||
contract. Reuse provider API/mapping logic without adding a second discovery
|
||||
never truncated. The WeKnora extension maps endpoint/tenant/base tuples to stable
|
||||
`wk1_` SHA-256 IDs (68 ASCII characters), resolving them from the authorized
|
||||
catalog on each request without a mapping store. The earlier `ks1_` encoding
|
||||
allowing 512 characters is not the public contract. Reuse provider API/mapping logic without adding a second discovery
|
||||
protocol or a DeerFlow ACL store.
|
||||
|
||||
`KnowledgeRequestContext` supplies the host user ID, Agent name and optional
|
||||
@ -251,3 +253,29 @@ uv run pytest tests/test_knowledge_contract_live.py -m live -q
|
||||
Set `LIGHTRAG_API_KEY` in the environment if required. This test performs no
|
||||
upload or resource changes. It requires attributable hits and checks each
|
||||
citation against the real returned chunk evidence.
|
||||
|
||||
## WeKnora read-only extension
|
||||
|
||||
The independently packaged `examples/deerflow-extension-weknora` registers
|
||||
`weknora` through the existing registry. Its operator configuration binds exact
|
||||
host users (and optionally current executor names) to credential environment
|
||||
variables, endpoint, tenant and optional base allowlist. No automatic tenant
|
||||
inference, default user binding, resource ACL storage or new core API is added.
|
||||
All bound users share the configured credential's upstream visibility; the
|
||||
plugin does not impersonate a WeKnora human user.
|
||||
|
||||
The adapter uses v0.8.3's KB-bound `hybrid-search` with `query_text` and optional
|
||||
`knowledge_ids`, disabling context enrichment. This supports KB-restricted
|
||||
read-only API keys; the global search endpoint rejects their document targets
|
||||
and unions base/document target lists. Raw recall needs no chat/rerank model.
|
||||
The adapter validates all selected resources, checks returned document
|
||||
membership/status, and fails the request on access errors without retrying an
|
||||
unrestricted search. A missing title remains unattributed; pages and URLs are
|
||||
omitted because the mapped response does not establish those locators.
|
||||
|
||||
See the extension README for immutable API references, installation, limits and
|
||||
opt-in real-provider verification. Offline synthetic responses exercise the
|
||||
actual registry, Gateway catalog, ToolNode, shared evidence and replay path.
|
||||
Live validation needs an indexed instance and is not implied by offline passes.
|
||||
The extension's [validation record](../../examples/deerflow-extension-weknora/VALIDATION.md)
|
||||
documents the local v0.8.3 / Ollama bge-m3 run.
|
||||
|
||||
40
backend/tests/blocking_io/test_weknora_extension.py
Normal file
40
backend/tests/blocking_io/test_weknora_extension.py
Normal file
@ -0,0 +1,40 @@
|
||||
"""Exercise real HTTPX/SSL construction with only the network response replaced."""
|
||||
|
||||
import asyncio
|
||||
import ssl
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def weknora_provider(monkeypatch):
|
||||
monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[3] / "examples/deerflow-extension-weknora"))
|
||||
from deerflow_extension_weknora import WeKnoraProvider
|
||||
|
||||
monkeypatch.setenv("WEKNORA_BLOCKING_TEST_KEY", "synthetic")
|
||||
return WeKnoraProvider({"spaces": [{"base_url": "https://weknora.invalid", "tenant_id": "1", "api_key_env": "WEKNORA_BLOCKING_TEST_KEY", "user_ids": ["alice"]}]})
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_ssl_setup_is_off_loop(weknora_provider, monkeypatch):
|
||||
from deerflow.knowledge import KnowledgeRequestContext
|
||||
|
||||
async def send(self, request, **kwargs):
|
||||
assert request.headers["X-Tenant-ID"] == "1"
|
||||
return httpx.Response(200, request=request, json={"success": True, "data": [{"id": "kb", "name": "Synthetic"}]})
|
||||
|
||||
original = ssl.create_default_context
|
||||
|
||||
def ssl_context(*args, **kwargs):
|
||||
try:
|
||||
asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
return original(*args, **kwargs)
|
||||
raise AssertionError("SSL trust loading must run off the event loop")
|
||||
|
||||
monkeypatch.setattr(ssl, "create_default_context", ssl_context)
|
||||
monkeypatch.setattr(httpx.AsyncClient, "send", send)
|
||||
catalog = await weknora_provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="")
|
||||
assert catalog.items[0].name == "Synthetic"
|
||||
@ -55,6 +55,7 @@ EXPECTED_GUIDANCE_PATHS = {
|
||||
"scripts/AGENTS.md",
|
||||
"examples/deerflow-extension-agent-teams/AGENTS.md",
|
||||
"examples/deerflow-extension-workflows/AGENTS.md",
|
||||
"examples/deerflow-extension-weknora/AGENTS.md",
|
||||
}
|
||||
|
||||
|
||||
|
||||
400
backend/tests/test_weknora_extension.py
Normal file
400
backend/tests/test_weknora_extension.py
Normal file
@ -0,0 +1,400 @@
|
||||
"""Synthetic v0.8.3 responses; never contacts an external provider."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from deerflow.knowledge import KnowledgeRequestContext, format_result, sources_from_messages
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def adapter(monkeypatch):
|
||||
monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[2] / "examples/deerflow-extension-weknora"))
|
||||
from deerflow_extension_weknora import WeKnoraProvider
|
||||
|
||||
monkeypatch.setenv("WEKNORA_TEST_KEY", "synthetic-secret")
|
||||
calls = []
|
||||
document = {"id": "doc-1", "knowledge_base_id": "kb-1", "title": "Synthetic guide", "parse_status": "completed", "enable_status": "enabled"}
|
||||
state = {"document": document, "hits": [{"knowledge_id": "doc-1", "content": "Cedar is synthetic evidence.", "knowledge_title": "Synthetic guide", "score": 0.8}]}
|
||||
|
||||
def handler(request):
|
||||
calls.append(request)
|
||||
assert request.headers["X-API-Key"] == "synthetic-secret"
|
||||
assert request.headers["X-Tenant-ID"] in {"1", "2"}
|
||||
if state.get("status"):
|
||||
return httpx.Response(state["status"], text="synthetic-secret upstream body")
|
||||
if request.url.path == "/api/v1/knowledge-bases":
|
||||
data = state.get("bases", [{"id": "kb-1", "name": "Synthetic base"}])
|
||||
elif request.url.path == "/api/v1/knowledge-bases/kb-1/knowledge":
|
||||
return httpx.Response(200, json={"success": True, "data": [state["document"]], **state.get("pagination", {"total": 1})})
|
||||
elif request.url.path == "/api/v1/knowledge/doc-1":
|
||||
data = state["document"]
|
||||
elif request.url.path == "/api/v1/knowledge-search":
|
||||
return httpx.Response(403, json={"success": False, "error": "KB-restricted keys reject global knowledge_ids"})
|
||||
elif request.url.path == "/api/v1/knowledge-bases/kb-1/hybrid-search":
|
||||
data = state["hits"]
|
||||
else:
|
||||
return httpx.Response(404, json={"success": False})
|
||||
return httpx.Response(200, json={"success": True, "data": data})
|
||||
|
||||
config = {"spaces": [{"base_url": "https://weknora.example", "tenant_id": "1", "api_key_env": "WEKNORA_TEST_KEY", "user_ids": ["alice"], "agent_names": ["lead_agent", "researcher"]}]}
|
||||
transport = httpx.MockTransport(handler)
|
||||
return WeKnoraProvider(config, transport=transport), calls, state, config, transport
|
||||
|
||||
|
||||
async def selection(provider, *, docs=True):
|
||||
context = KnowledgeRequestContext("alice", "lead_agent")
|
||||
page = await provider.catalog(context=context, page=1, page_size=20, search="")
|
||||
dataset_id = page.items[0].id
|
||||
scope = {"version": 1, "mode": "selected", "dataset_ids": [dataset_id]}
|
||||
if docs:
|
||||
scope["document_filters"] = [{"dataset_id": dataset_id, "document_ids": ["doc-1"]}]
|
||||
return KnowledgeRequestContext("alice", "lead_agent", scope)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_catalog_documents_search_shared_evidence(adapter):
|
||||
provider, calls, _, _, _ = adapter
|
||||
ctx = await selection(provider)
|
||||
page = await provider.documents(ctx.scope["dataset_ids"][0], context=ctx, page=1, page_size=20, search="guide")
|
||||
assert page.items[0].selectable and page.items[0].id == "doc-1"
|
||||
assert calls[-1].url.params["keyword"] == "guide"
|
||||
result = await provider.search_scoped("Cedar", context=ctx)
|
||||
body = json.loads(next(r.content for r in calls if r.method == "POST"))
|
||||
assert body["knowledge_ids"] == ["doc-1"]
|
||||
assert "knowledge_base_ids" not in body
|
||||
assert body["query_text"] == "Cedar"
|
||||
assert body["skip_context_enrichment"] is True
|
||||
assert next(r for r in calls if r.method == "POST").url.path == "/api/v1/knowledge-bases/kb-1/hybrid-search"
|
||||
content, artifact = format_result(result)
|
||||
sources = sources_from_messages([{"type": "tool", "content": content, "artifact": artifact}])
|
||||
assert len(sources) == 1 and sources[0].provider == "weknora"
|
||||
assert sources[0].excerpt == "Cedar is synthetic evidence."
|
||||
assert sources[0].page is None and sources[0].url is None
|
||||
assert sources_from_messages([{"type": "tool", "content": "removed", "artifact": artifact}]) == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("user,agent", [("mallory", "lead_agent"), ("alice", "unbound"), ("alice", None)])
|
||||
async def test_unbound_identity_cannot_make_requests(adapter, user, agent):
|
||||
provider, calls, *_ = adapter
|
||||
ctx = KnowledgeRequestContext(user, agent)
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.catalog(context=ctx, page=1, page_size=20, search="")
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
assert not calls
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_context_search_is_not_an_authorization_bypass(adapter):
|
||||
provider, calls, *_ = adapter
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.search("Cedar")
|
||||
assert not calls
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("scope", [{"version": 1, "mode": "disabled"}, {"mode": "selected", "dataset_ids": []}])
|
||||
async def test_disabled_or_invalid_scope_never_calls_upstream(adapter, scope):
|
||||
provider, calls, *_ = adapter
|
||||
with pytest.raises(ValueError):
|
||||
await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent", scope))
|
||||
assert not calls
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("change", [{"knowledge_base_id": "other"}, {"id": "other"}, {"enable_status": "disabled"}, {"parse_status": "pending"}])
|
||||
async def test_document_validation_precedes_search(adapter, change):
|
||||
provider, calls, state, *_ = adapter
|
||||
ctx = await selection(provider)
|
||||
state["document"].update(change)
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
assert not any(r.method == "POST" for r in calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unselected_result_fails_entire_request(adapter):
|
||||
provider, calls, state, *_ = adapter
|
||||
ctx = await selection(provider)
|
||||
state["hits"].append({"knowledge_id": "other", "content": "must never escape"})
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
assert len([r for r in calls if r.method == "POST"]) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dataset_search_verifies_returned_document_membership(adapter):
|
||||
provider, _, state, *_ = adapter
|
||||
ctx = await selection(provider, docs=False)
|
||||
state["document"]["knowledge_base_id"] = "other"
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("status,error", [(401, PermissionError), (403, PermissionError), (404, FileNotFoundError), (500, RuntimeError), (302, RuntimeError)])
|
||||
async def test_http_errors_are_redacted_and_never_retried(adapter, status, error):
|
||||
provider, calls, state, *_ = adapter
|
||||
state["status"] = status
|
||||
with pytest.raises(error) as exc:
|
||||
await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="")
|
||||
assert "synthetic-secret" not in str(exc.value)
|
||||
assert len(calls) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multi_space_ids_are_stable_bounded_and_isolated(adapter):
|
||||
provider, _, _, config, transport = adapter
|
||||
from deerflow_extension_weknora import WeKnoraProvider
|
||||
|
||||
config["spaces"].append({**config["spaces"][0], "tenant_id": "2", "user_ids": ["bob"]})
|
||||
multi = WeKnoraProvider(config, transport=transport)
|
||||
alice = await multi.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="")
|
||||
bob = await multi.catalog(context=KnowledgeRequestContext("bob", "lead_agent"), page=1, page_size=20, search="")
|
||||
old = await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="")
|
||||
assert alice.items[0].id == old.items[0].id != bob.items[0].id
|
||||
assert len(alice.items[0].id) <= 256
|
||||
with pytest.raises(PermissionError):
|
||||
await multi.search_scoped("Cedar", context=KnowledgeRequestContext("bob", "lead_agent", {"version": 1, "mode": "selected", "dataset_ids": [alice.items[0].id]}))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_scope_still_sends_explicit_targets_and_subagent_works(adapter):
|
||||
provider, calls, *_ = adapter
|
||||
await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "researcher"))
|
||||
body = json.loads(next(r.content for r in calls if r.method == "POST"))
|
||||
assert "knowledge_base_ids" not in body and "knowledge_ids" not in body
|
||||
assert next(r for r in calls if r.method == "POST").url.path == "/api/v1/knowledge-bases/kb-1/hybrid-search"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_secrets_redacted_and_missing_attribution_not_invented(adapter):
|
||||
provider, _, state, *_ = adapter
|
||||
state["hits"][0].update(content="synthetic-secret evidence", knowledge_title="")
|
||||
state["document"]["title"] = ""
|
||||
result = await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent"))
|
||||
assert "synthetic-secret" not in result.model_dump_json()
|
||||
assert result.hits[0].source is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("bad_id", ["..", "../secret", "%2e%2e", "a/b", "x" * 257])
|
||||
async def test_unsafe_native_ids_fail_before_path_request(adapter, bad_id):
|
||||
provider, calls, *_ = adapter
|
||||
ctx = await selection(provider)
|
||||
ctx.scope["document_filters"][0]["document_ids"] = [bad_id]
|
||||
with pytest.raises(ValueError):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
assert not any("/knowledge/" in r.url.path or r.method == "POST" for r in calls)
|
||||
|
||||
|
||||
def test_configuration_rejects_implicit_authorization_and_credentials(adapter):
|
||||
from deerflow_extension_weknora import WeKnoraProvider
|
||||
|
||||
_, _, _, config, _ = adapter
|
||||
for change in ({"tenant_id": None}, {"user_ids": []}, {"base_url": "https://user:secret@example.com"}, {"api_key_env": ""}, {"knowledge_base_ids": []}):
|
||||
with pytest.raises(ValueError):
|
||||
WeKnoraProvider({"spaces": [{**config["spaces"][0], **change}]})
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_packaged_registration_real_tool_replay_budget_and_delegation(adapter, monkeypatch):
|
||||
from types import SimpleNamespace
|
||||
|
||||
from langchain_core.messages import AIMessage
|
||||
from langgraph.graph import END, START, MessagesState, StateGraph
|
||||
from langgraph.prebuilt import ToolNode
|
||||
|
||||
from deerflow.extensions import EXTENSION_SNAPSHOT_CONTEXT_KEY, load_extensions
|
||||
from deerflow.extensions.loader import ExtensionSpec
|
||||
from deerflow.knowledge import tools
|
||||
from deerflow.knowledge.providers import GENERIC_KNOWLEDGE_TOOL, resolve_knowledge_provider
|
||||
from deerflow.knowledge.sources import prune_sources
|
||||
from deerflow.runtime.serialization import serialize_channel_values_for_api
|
||||
from deerflow.tools.builtins.task_tool import _task_result_command
|
||||
|
||||
_, _, _, config, transport = adapter
|
||||
loaded, diagnostics = load_extensions([ExtensionSpec(use="deerflow_extension_weknora:install", config=config)])
|
||||
assert diagnostics == []
|
||||
app_config = SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(use=GENERIC_KNOWLEDGE_TOOL, model_extra={"provider": "weknora"}))
|
||||
provider = resolve_knowledge_provider(app_config, extensions=loaded)
|
||||
provider._transport = transport
|
||||
ctx = await selection(provider)
|
||||
monkeypatch.setattr(tools, "get_app_config", lambda: app_config)
|
||||
builder = StateGraph(MessagesState, context_schema=dict)
|
||||
builder.add_node("tools", ToolNode([tools.knowledge_search_tool]))
|
||||
builder.add_edge(START, "tools")
|
||||
builder.add_edge("tools", END)
|
||||
result = await builder.compile().ainvoke(
|
||||
{"messages": [AIMessage(content="", tool_calls=[{"id": "call", "name": "knowledge_search", "args": {"query": "Cedar"}}])]},
|
||||
context={"user_id": "alice", "is_subagent": True, "agent_id": "researcher", "__knowledge_scope_execution": ctx.scope, EXTENSION_SNAPSHOT_CONTEXT_KEY: loaded},
|
||||
)
|
||||
message = result["messages"][-1]
|
||||
sources = sources_from_messages([message])
|
||||
assert len(sources) == 1 and sources[0].provider == "weknora"
|
||||
persisted = json.loads(json.dumps(serialize_channel_values_for_api({"messages": [message]})))
|
||||
assert sources_from_messages(persisted["messages"]) == sources
|
||||
command = _task_result_command(tool_call_id="task", status="completed", result=message.content, knowledge_sources=[s.model_dump(mode="json", exclude_none=True) for s in sources])
|
||||
assert sources_from_messages(command.update["messages"]) == sources
|
||||
_, pruned = prune_sources("truncated", message.artifact)
|
||||
assert pruned["knowledge_sources"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_key_fails_closed(adapter, monkeypatch):
|
||||
provider, calls, *_ = adapter
|
||||
monkeypatch.delenv("WEKNORA_TEST_KEY")
|
||||
with pytest.raises(RuntimeError):
|
||||
await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent"))
|
||||
assert not calls
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_allowlist_and_missing_configured_base(adapter):
|
||||
from deerflow_extension_weknora import WeKnoraProvider
|
||||
|
||||
_, calls, state, config, transport = adapter
|
||||
config["spaces"][0]["knowledge_base_ids"] = ["kb-1"]
|
||||
provider = WeKnoraProvider(config, transport=transport)
|
||||
state["bases"] = [{"id": "kb-1", "name": "Allowed"}, {"id": "kb-2", "name": "Excluded"}]
|
||||
page = await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="")
|
||||
assert [item.name for item in page.items] == ["Allowed"]
|
||||
state["bases"] = [{"id": "kb-2", "name": "Excluded"}]
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent"))
|
||||
assert not any(r.method == "POST" for r in calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_catalog_never_sends_unconstrained_search(adapter):
|
||||
provider, calls, state, *_ = adapter
|
||||
state["bases"] = None
|
||||
result = await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent"))
|
||||
assert result.hits == [] and not any(r.method == "POST" for r in calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [[], {"success": False, "data": []}, {"success": True}, {"success": True, "data": "bad"}])
|
||||
async def test_malformed_upstream_response_fails_closed(adapter, payload):
|
||||
provider, *_ = adapter
|
||||
provider._transport = httpx.MockTransport(lambda _: httpx.Response(200, json=payload))
|
||||
with pytest.raises(RuntimeError):
|
||||
await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_response_size_limit(adapter):
|
||||
provider, *_ = adapter
|
||||
provider._transport = httpx.MockTransport(lambda _: httpx.Response(200, content=b"x" * (4 * 1024 * 1024 + 1)))
|
||||
with pytest.raises(RuntimeError, match="size"):
|
||||
await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_document_selections_checked_before_any_search(adapter):
|
||||
provider, calls, *_ = adapter
|
||||
ctx = await selection(provider)
|
||||
ctx.scope["document_filters"][0]["document_ids"].append("missing")
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
assert not any(r.method == "POST" for r in calls)
|
||||
|
||||
|
||||
def test_gateway_catalog_uses_authenticated_user_with_real_adapter(adapter):
|
||||
from _router_auth_helpers import make_authed_test_app
|
||||
from deerflow_extension_weknora import WeKnoraProvider
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.deps import get_config
|
||||
from app.gateway.routers import knowledge
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.extensions import ExtensionRegistry
|
||||
|
||||
_, calls, _, config, transport = adapter
|
||||
user = User(email="weknora@example.com", password_hash="unused", system_role="user")
|
||||
config["spaces"][0]["user_ids"] = [str(user.id)]
|
||||
provider = WeKnoraProvider(config, transport=transport)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("deerflow_extension_weknora:install"):
|
||||
registry.knowledge_provider("weknora", provider)
|
||||
app = make_authed_test_app(user_factory=lambda: user, bind_current_user=True)
|
||||
app.state.extensions = registry.build()
|
||||
app.include_router(knowledge.router)
|
||||
app_config = AppConfig.model_validate(
|
||||
{
|
||||
"models": [],
|
||||
"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"},
|
||||
"knowledge_base": {"enabled": True, "scope_selection_enabled": True},
|
||||
"tools": [{"name": "knowledge_search", "group": "knowledge", "use": "deerflow.knowledge.tools:knowledge_search_tool", "provider": "weknora"}],
|
||||
}
|
||||
)
|
||||
app.dependency_overrides[get_config] = lambda: app_config
|
||||
with TestClient(app) as client:
|
||||
cap = client.get("/api/knowledge/capabilities?agent_name=lead_agent")
|
||||
assert cap.status_code == 200 and cap.json()["scope_selection_enabled"] is True
|
||||
catalog = client.get("/api/knowledge/retrieval-catalog/datasets?agent_name=lead_agent&user_id=mallory")
|
||||
assert catalog.status_code == 200
|
||||
dataset_id = catalog.json()["items"][0]["id"]
|
||||
documents = client.get(f"/api/knowledge/retrieval-catalog/datasets/{dataset_id}/documents?agent_name=lead_agent")
|
||||
assert documents.status_code == 200 and documents.json()["items"][0]["selectable"] is True
|
||||
assert client.get("/api/knowledge/retrieval-catalog/datasets/unknown/documents?agent_name=lead_agent").status_code == 403
|
||||
assert all(r.method == "GET" for r in calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_oversized_hit_list_does_not_cause_unbounded_detail_requests(adapter):
|
||||
provider, calls, state, *_ = adapter
|
||||
state["hits"] = [{"knowledge_id": f"doc-{i}", "content": "synthetic"} for i in range(101)]
|
||||
with pytest.raises(RuntimeError, match="result limit"):
|
||||
await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent"))
|
||||
assert not any("/knowledge/" in r.url.path for r in calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("pagination", [{}, {"total": None}, {"total": "1"}, {"total": True}, {"total": -1}, {"total": 1.5}])
|
||||
async def test_invalid_document_total_is_explicit_provider_error(adapter, pagination):
|
||||
provider, calls, state, *_ = adapter
|
||||
ctx = await selection(provider)
|
||||
state["pagination"] = pagination
|
||||
with pytest.raises(RuntimeError, match="Invalid WeKnora document total"):
|
||||
await provider.documents(ctx.scope["dataset_ids"][0], context=ctx, page=1, page_size=20, search="")
|
||||
assert not any(r.method == "POST" for r in calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("target", ["hits", "document"])
|
||||
@pytest.mark.parametrize("data", ["bad", [], ["bad"], 42])
|
||||
async def test_malformed_retrieval_payloads_fail_without_retry(adapter, target, data):
|
||||
provider, calls, state, *_ = adapter
|
||||
ctx = await selection(provider)
|
||||
state[target] = data
|
||||
if target == "hits" and data == []:
|
||||
assert (await provider.search_scoped("Cedar", context=ctx)).hits == []
|
||||
else:
|
||||
with pytest.raises(RuntimeError, match="Invalid WeKnora"):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
assert len([r for r in calls if r.method == "POST"]) == (1 if target == "hits" else 0)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retrieval_response_size_limit_without_retry(adapter):
|
||||
provider, calls, _, _, transport = adapter
|
||||
ctx = await selection(provider, docs=False)
|
||||
posts = []
|
||||
|
||||
async def handler(request):
|
||||
if request.method == "POST":
|
||||
posts.append(request)
|
||||
return httpx.Response(200, content=b"x" * (4 * 1024 * 1024 + 1))
|
||||
return await transport.handle_async_request(request)
|
||||
|
||||
provider._transport = httpx.MockTransport(handler)
|
||||
with pytest.raises(RuntimeError, match="size"):
|
||||
await provider.search_scoped("Cedar", context=ctx)
|
||||
assert len(posts) == 1
|
||||
assert not any("/knowledge/" in r.url.path for r in calls)
|
||||
88
backend/tests/test_weknora_extension_live.py
Normal file
88
backend/tests/test_weknora_extension_live.py
Normal file
@ -0,0 +1,88 @@
|
||||
"""Opt-in read-only WeKnora v0.8.3 integration; needs existing indexed evidence."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.live
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_live_selected_document_evidence(monkeypatch):
|
||||
if os.getenv("DEER_FLOW_RUN_LIVE_TESTS") != "1":
|
||||
pytest.skip("live knowledge checks are opt-in")
|
||||
names = ["WEKNORA_BASE_URL", "WEKNORA_TENANT_ID", "WEKNORA_KNOWLEDGE_BASE_ID", "WEKNORA_DOCUMENT_ID", "WEKNORA_EVIDENCE_QUERY", "WEKNORA_API_KEY"]
|
||||
if any(not os.getenv(name) for name in names):
|
||||
pytest.fail("Set all documented WEKNORA_* variables for live verification")
|
||||
monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[2] / "examples/deerflow-extension-weknora"))
|
||||
from deerflow_extension_weknora import WeKnoraProvider
|
||||
|
||||
from deerflow.knowledge import KnowledgeRequestContext, format_result, sources_from_messages
|
||||
|
||||
provider = WeKnoraProvider(
|
||||
{"spaces": [{"base_url": os.environ["WEKNORA_BASE_URL"], "tenant_id": os.environ["WEKNORA_TENANT_ID"], "api_key_env": "WEKNORA_API_KEY", "user_ids": ["live-probe"], "knowledge_base_ids": [os.environ["WEKNORA_KNOWLEDGE_BASE_ID"]]}]}
|
||||
)
|
||||
context = KnowledgeRequestContext("live-probe", "lead_agent")
|
||||
catalog = await provider.catalog(context=context, page=1, page_size=100, search="")
|
||||
assert len(catalog.items) == 1, "Configured base must be visible"
|
||||
dataset_id = catalog.items[0].id
|
||||
document_id = os.environ["WEKNORA_DOCUMENT_ID"]
|
||||
# Traverse actual catalog pagination, not a guessed first-page selection.
|
||||
page = 1
|
||||
while True:
|
||||
documents = await provider.documents(dataset_id, context=context, page=page, page_size=100, search="")
|
||||
match = next((item for item in documents.items if item.id == document_id), None)
|
||||
if match:
|
||||
assert match.selectable, "Configured document must be searchable"
|
||||
break
|
||||
assert documents.items and page * 100 < documents.total and page < 100, "Configured document not found within the bounded live scan"
|
||||
page += 1
|
||||
scope = {"version": 1, "mode": "selected", "dataset_ids": [dataset_id], "document_filters": [{"dataset_id": dataset_id, "document_ids": [document_id]}]}
|
||||
result = await provider.search_scoped(os.environ["WEKNORA_EVIDENCE_QUERY"], context=KnowledgeRequestContext("live-probe", "lead_agent", scope))
|
||||
content, artifact = format_result(result)
|
||||
persisted = json.loads(json.dumps({"type": "tool", "content": content, "artifact": artifact}))
|
||||
sources = sources_from_messages([persisted])
|
||||
assert sources, "Live query must retrieve attributable document evidence"
|
||||
for source in sources:
|
||||
assert source.provider == "weknora" and source.page is None and source.url is None
|
||||
assert any(hit.source and hit.source.title == source.title and source.excerpt.rstrip("…") in hit.content for hit in result.hits)
|
||||
|
||||
# Exercise the host's actual tool/artifact path with the live provider.
|
||||
# The AIMessage is deterministic: this checks tool execution, not LLM quality.
|
||||
from types import SimpleNamespace
|
||||
|
||||
from langchain_core.messages import AIMessage
|
||||
from langgraph.graph import END, START, MessagesState, StateGraph
|
||||
from langgraph.prebuilt import ToolNode
|
||||
|
||||
from deerflow.extensions import EXTENSION_SNAPSHOT_CONTEXT_KEY, ExtensionRegistry
|
||||
from deerflow.knowledge import tools
|
||||
from deerflow.knowledge.sources import prune_sources
|
||||
from deerflow.runtime.serialization import serialize_channel_values_for_api
|
||||
from deerflow.tools.builtins.task_tool import _task_result_command
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("deerflow_extension_weknora:install"):
|
||||
registry.knowledge_provider("weknora", provider)
|
||||
config = SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(use="deerflow.knowledge.tools:knowledge_search_tool", model_extra={"provider": "weknora"}))
|
||||
monkeypatch.setattr(tools, "get_app_config", lambda: config)
|
||||
builder = StateGraph(MessagesState, context_schema=dict)
|
||||
builder.add_node("tools", ToolNode([tools.knowledge_search_tool]))
|
||||
builder.add_edge(START, "tools")
|
||||
builder.add_edge("tools", END)
|
||||
state = await builder.compile().ainvoke(
|
||||
{"messages": [AIMessage(content="", tool_calls=[{"id": "live-search", "name": "knowledge_search", "args": {"query": os.environ["WEKNORA_EVIDENCE_QUERY"]}}])]},
|
||||
context={"user_id": "live-probe", "is_subagent": True, "agent_id": "researcher", "__knowledge_scope_execution": scope, EXTENSION_SNAPSHOT_CONTEXT_KEY: registry.build()},
|
||||
)
|
||||
message = state["messages"][-1]
|
||||
tool_sources = sources_from_messages([message])
|
||||
assert tool_sources and all(source.provider == "weknora" for source in tool_sources)
|
||||
history = json.loads(json.dumps(serialize_channel_values_for_api({"messages": [message]})))
|
||||
assert sources_from_messages(history["messages"]) == tool_sources
|
||||
forwarded = _task_result_command(tool_call_id="live-task", status="completed", result=message.content, knowledge_sources=[source.model_dump(mode="json", exclude_none=True) for source in tool_sources])
|
||||
assert sources_from_messages(forwarded.update["messages"]) == tool_sources
|
||||
_, pruned = prune_sources("removed by budget", message.artifact)
|
||||
assert pruned["knowledge_sources"] == []
|
||||
with pytest.raises(PermissionError):
|
||||
await provider.search_scoped("test", context=KnowledgeRequestContext("unbound-live-user", "lead_agent", scope))
|
||||
20
examples/deerflow-extension-weknora/AGENTS.md
Normal file
20
examples/deerflow-extension-weknora/AGENTS.md
Normal file
@ -0,0 +1,20 @@
|
||||
# WeKnora read-only extension
|
||||
|
||||
Keep provider-specific code in this independently packaged extension. Use
|
||||
`registry.knowledge_provider` and the public `deerflow.knowledge` contracts;
|
||||
core owns scope admission, formatting, budgets and citation evidence.
|
||||
|
||||
API reference: WeKnora v0.8.3 at `3e22822c1d2397be730c458c7539d0eb8c113a24`.
|
||||
Preserve per-request operator bindings and upstream authorization. Never turn
|
||||
missing bindings, invalid selections, empty allowlists, or upstream failures
|
||||
into an unrestricted search. Use KB-bound `hybrid-search` with optional
|
||||
`knowledge_ids` and disabled context enrichment. The global search endpoint
|
||||
rejects document targets for KB-restricted keys and unions base/document target
|
||||
lists. Never broaden credentials to work around that. Verify result membership.
|
||||
Reject malformed document totals and detail shapes as provider errors; reserve
|
||||
permission errors for resource identity, membership and searchability failures.
|
||||
Dataset IDs derive from endpoint/tenant/native base, not credentials or labels.
|
||||
|
||||
Tests live in `backend/tests/test_weknora_extension*.py`; live tests require
|
||||
explicit opt-in and an already indexed instance. Do not fabricate a live result.
|
||||
See README for version references, configuration and the authorization model.
|
||||
185
examples/deerflow-extension-weknora/README.md
Normal file
185
examples/deerflow-extension-weknora/README.md
Normal file
@ -0,0 +1,185 @@
|
||||
# WeKnora read-only knowledge provider
|
||||
|
||||
An independently installable extension for the phase 3 read-only delivery of
|
||||
[#6269](https://github.com/bytedance/deer-flow/issues/6269). It uses the existing
|
||||
DeerFlow scope selector, generic `knowledge_search`, shared citation validation,
|
||||
source budgets, delegation and history rendering. No extra frontend is needed.
|
||||
|
||||
The API mapping is checked against **WeKnora v0.8.3**, commit
|
||||
[`3e22822c1d2397be730c458c7539d0eb8c113a24`](https://github.com/Tencent/WeKnora/tree/3e22822c1d2397be730c458c7539d0eb8c113a24).
|
||||
Other releases require verification. Offline tests use explicitly synthetic responses. A real local deployment
|
||||
with Ollama `bge-m3` has also passed; see [the validation record](VALIDATION.md)
|
||||
for the exact environment, checks and limits.
|
||||
|
||||
## Choosing this extension or MCP
|
||||
|
||||
Use [WeKnora's built-in MCP server](https://github.com/Tencent/WeKnora/blob/3e22822c1d2397be730c458c7539d0eb8c113a24/website-docs/03-features/08-mcp.md)
|
||||
when you want to expose WeKnora tools to DeerFlow or another MCP client.
|
||||
Choose this read-only extension when you want DeerFlow's native knowledge-base
|
||||
and document scope selector, shared citations, and retrieval evidence preserved
|
||||
across delegated tasks and conversation replay.
|
||||
|
||||
## Install and configure
|
||||
|
||||
Requires a host containing phase 2 (#6904), extension-api >=0.2.7, and an indexed
|
||||
WeKnora v0.8.3 workspace. From the DeerFlow repository root:
|
||||
|
||||
```bash
|
||||
make extension-install SOURCE=/absolute/path/to/deer-flow/examples/deerflow-extension-weknora
|
||||
```
|
||||
|
||||
Set `WEKNORA_API_KEY` in the Gateway process environment (or its Docker
|
||||
container environment). Do not put a key into the URL or commit credentials.
|
||||
Edit the installed plugin's existing record in root `config.yaml`, and merge
|
||||
the knowledge/tool settings with your existing entries:
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
- use: deerflow_extension_weknora:install
|
||||
enabled: true
|
||||
required: true
|
||||
config:
|
||||
timeout: 30
|
||||
match_count: 8
|
||||
spaces:
|
||||
- base_url: https://weknora.example.com
|
||||
tenant_id: "42"
|
||||
api_key_env: WEKNORA_API_KEY
|
||||
user_ids: ["YOUR_DEERFLOW_USER_ID"]
|
||||
# Optional restriction to exact current executor names, including
|
||||
# each subagent that should search. Omit to allow all agents for
|
||||
# the explicitly bound users.
|
||||
agent_names: ["lead_agent", "researcher"]
|
||||
# Optional operator restriction; omission permits bases visible to
|
||||
# this connection. An empty list is rejected, never treated as all.
|
||||
knowledge_base_ids: ["YOUR_WEKNORA_KB_ID"]
|
||||
|
||||
knowledge_base:
|
||||
enabled: true
|
||||
scope_selection_enabled: true
|
||||
|
||||
tool_groups:
|
||||
- name: knowledge
|
||||
|
||||
tools:
|
||||
- name: knowledge_search
|
||||
group: knowledge
|
||||
use: deerflow.knowledge.tools:knowledge_search_tool
|
||||
provider: weknora
|
||||
```
|
||||
|
||||
Preserve extension-manager fields such as `name` and `package` if present;
|
||||
replace the existing `knowledge_search` entry instead of adding a second one.
|
||||
Restart Gateway after installation or configuration changes. Select a knowledge
|
||||
base (optionally individual documents) using the chat's existing scope selector,
|
||||
then ask a question matching indexed content. Sources show the retrieved title,
|
||||
excerpt and base name. This adapter does not invent page numbers or source URLs
|
||||
from chunk offsets, storage paths or response indexes.
|
||||
|
||||
`user_ids` are exact server-authenticated DeerFlow IDs, not display names or
|
||||
model-supplied values. Resolve them from your deployment's user administration;
|
||||
for an unauthenticated/single-user installation use its actual runtime user ID.
|
||||
There is no default or wildcard binding. To bind multiple workspaces, add a
|
||||
`spaces` entry per endpoint/tenant pair, each with its own explicit users and
|
||||
credential environment variable. Configure each endpoint/tenant only once.
|
||||
|
||||
## Authorization and scope
|
||||
|
||||
Each operation resolves the trusted current user and executor against the
|
||||
operator-owned connection bindings. It sends `X-API-Key` and `X-Tenant-ID` to
|
||||
WeKnora, whose API-key permissions and workspace/resource authorization remain
|
||||
final. The binding is a credential-routing decision: every listed user receives
|
||||
the resource visibility of that credential, further restricted by the optional
|
||||
base allowlist. It does **not** impersonate a WeKnora human user. Use dedicated
|
||||
least-privilege keys and only bind users intended to share that visibility.
|
||||
|
||||
Tenant-scoped keys must match the configured tenant. Platform keys always need
|
||||
an explicit tenant and user binding here; unbound platform discovery and
|
||||
fallback-to-all behavior are not supported. Missing bindings or keys fail
|
||||
closed. API-key roles/capabilities must allow base/document reads and knowledge
|
||||
search; restricted keys that cannot enumerate the configured bases are rejected.
|
||||
|
||||
- `all` searches explicit accessible bases in the user's bindings, never an
|
||||
omitted target list. At most 100 bases across at most 20 bound spaces are
|
||||
supported; narrow larger connections with `knowledge_base_ids`.
|
||||
- `selected` validates every base and document before searching. Unknown,
|
||||
inaccessible, disabled, or not-yet-searchable documents fail the complete
|
||||
request. WeKnora `completed` and `finalizing` documents with `enabled` status
|
||||
are selectable.
|
||||
- Retrieval uses `POST /api/v1/knowledge-bases/{id}/hybrid-search` with
|
||||
`query_text` and optional `knowledge_ids`. The path fixes the authorized base;
|
||||
document IDs narrow it. Context enrichment is disabled to keep matched
|
||||
evidence within that selection. Mixed selections use a request per base.
|
||||
This raw-recall API needs only an embedding model, with no reranker or chat
|
||||
model. The global `/knowledge-search` API rejects document IDs with
|
||||
KB-restricted API keys, and combining base/document targets there uses union
|
||||
semantics. The adapter never broadens key permissions to work around that.
|
||||
- Retrieved documents are checked against the authoritative detail endpoint for
|
||||
base membership and searchable state; unexpected evidence fails the request.
|
||||
No unrestricted retry, partial result on an error, or redirect is allowed.
|
||||
- Dataset IDs are `wk1_` plus SHA-256 of a JSON tuple of normalized endpoint,
|
||||
tenant and native base ID (68 ASCII characters). Resolution is recomputed
|
||||
from the currently authorized catalog without an ID/ACL database. Credential
|
||||
rotation preserves IDs; endpoint or tenant changes invalidate old selections.
|
||||
Document IDs remain native IDs inside the dataset filter. Native WeKnora IDs
|
||||
accept only ASCII letters, digits, `_` and `-`, up to 256 characters.
|
||||
- The same bindings apply to catalog, primary execution and subagents. Agent
|
||||
restrictions match the **current executor**, not the parent assistant.
|
||||
Restored scopes are re-authorized against current upstream permissions.
|
||||
|
||||
API calls are asynchronous, use a bounded timeout and 4 MiB response limit,
|
||||
and suppress upstream bodies in errors. A response exceeding 100 hits fails;
|
||||
at most 100 hits are admitted per tool call before core applies its text budget. Current connection secrets are redacted
|
||||
from text evidence. Shared source formatting and output budgets remain in core.
|
||||
|
||||
## Verification
|
||||
|
||||
Offline regression tests from the repository root:
|
||||
|
||||
```bash
|
||||
uv run --project backend --no-sync python -m pytest \
|
||||
backend/tests/test_weknora_extension.py -q
|
||||
```
|
||||
|
||||
The tests cover connection isolation, filters without widening, missing/denied
|
||||
resources, malformed responses, package registration, real ToolNode execution,
|
||||
subagent identity, shared evidence, serialization/replay and budget pruning.
|
||||
Run existing generic scope-admission and frontend citation suites as well.
|
||||
|
||||
An opt-in, read-only live check uses an **already indexed document**. Set the key
|
||||
in the environment before invoking it; do not paste keys into shell history.
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
DEER_FLOW_RUN_LIVE_TESTS=1 \
|
||||
WEKNORA_BASE_URL=https://weknora.example.com \
|
||||
WEKNORA_TENANT_ID=42 \
|
||||
WEKNORA_KNOWLEDGE_BASE_ID=your-base-id \
|
||||
WEKNORA_DOCUMENT_ID=your-indexed-document-id \
|
||||
WEKNORA_EVIDENCE_QUERY='a question matching that document' \
|
||||
uv run --project backend --no-sync python -m pytest \
|
||||
backend/tests/test_weknora_extension_live.py -m live -q
|
||||
```
|
||||
|
||||
The test expects `WEKNORA_API_KEY`, performs no upload/write operation, verifies
|
||||
catalog → selected document → retrieval → actual ToolNode → shared source/replay,
|
||||
subagent forwarding and evidence pruning, and requires attributable results.
|
||||
It supplies a deterministic tool call rather than invoking a generative model. Record the deployed WeKnora version, test result and UI
|
||||
smoke-test result before claiming live delivery. Do not include keys or private
|
||||
retrieval text in committed validation records.
|
||||
|
||||
## Scope and attribution
|
||||
|
||||
API mapping and filter-validation design were informed by
|
||||
[superbuzzy's integration branch](https://github.com/superbuzzy/deer-flow/tree/f67609f3ab54c6d243d6f0ee1f82f2253b018b33),
|
||||
especially `community/weknora/{provider,documents,retrieval}.py`, and checked
|
||||
against v0.8.3's `internal/handler/knowledgebase.go`, `internal/handler/session/{types,qa}.go`,
|
||||
`internal/handler/knowledge.go`, `internal/types/search.go` and
|
||||
`internal/middleware/auth.go`.
|
||||
|
||||
This package registers the current `RetrievalProvider` contract rather than
|
||||
importing the older branch's provider discovery or native citation format.
|
||||
Uploads, deletion, processing management, Wiki/Graph, a knowledge management
|
||||
page and new DeerFlow ACL persistence remain outside the read-only delivery.
|
||||
Shared-contract durable batch source forwarding retains the existing core
|
||||
limitation; ordinary subagent forwarding uses the shared evidence path.
|
||||
96
examples/deerflow-extension-weknora/VALIDATION.md
Normal file
96
examples/deerflow-extension-weknora/VALIDATION.md
Normal file
@ -0,0 +1,96 @@
|
||||
# Local integration validation — 2026-10-15
|
||||
|
||||
A real local WeKnora deployment and local Ollama embeddings were exercised on
|
||||
Apple Silicon macOS. The corpus is explicitly synthetic Chinese text; no
|
||||
external model API, real customer corpus, or production credentials were used.
|
||||
|
||||
## Environment
|
||||
|
||||
- WeKnora API/UI: v0.8.3. API runtime reports version `0.8.3`, standard edition,
|
||||
Go 1.26.0, linux/arm64, build time 2026-10-14 07:53:47 UTC.
|
||||
- API image: `wechatopenai/weknora-app:v0.8.3`, multi-platform manifest digest
|
||||
`sha256:b5f96b13c2f69779af8c7291264cafbda64c208398b019e3a6223c6f3ea82031`;
|
||||
selected arm64 manifest
|
||||
`sha256:e9b2f77cbd0edad22a693844d27029a1881718687e77804028f6d3306f7340b1`.
|
||||
- PostgreSQL/vector retrieval: `paradedb/paradedb:v0.25.11-pg17`;
|
||||
task queue: `redis:7.0-alpine`; container runtime: OrbStack.
|
||||
- Ollama 0.40.4, native macOS; model `bge-m3`, 1024 embedding dimensions,
|
||||
model context length 8192, embedding-only capability.
|
||||
- Model weights: official Ollama `library/bge-m3` model blob, 1,157,671,200
|
||||
bytes, verified SHA-256
|
||||
`daec91ffb5dd0c27411bd71f29932917c49cf529a641d0168496c3a501e3062c`.
|
||||
The machine's Fake-IP DNS made Ollama reject the registry redirect. The
|
||||
official blob was downloaded over HTTPS, digest-verified, and imported with
|
||||
`ollama create bge-m3`; no redirect guard was disabled.
|
||||
- All host ports are loopback-bound. Container API calls reach the native
|
||||
Ollama service through `host.docker.internal`.
|
||||
|
||||
## Corpus and credential setup
|
||||
|
||||
Created a dedicated local workspace and document knowledge base through the
|
||||
WeKnora API. Its embedding model is local `bge-m3`; vector indexing is enabled.
|
||||
Summary generation, generated questions, graph and Wiki are disabled, so this
|
||||
fixture does not require a chat or rerank model. Manual Markdown ingestion
|
||||
uses WeKnora's own processing and embedding pipeline.
|
||||
|
||||
Two documents reached `parse_status=completed`, `enable_status=enabled`:
|
||||
|
||||
1. Synthetic Cedar travel policy: nightly accommodation cap 680 CNY, submission
|
||||
within 15 days, unique marker `CEDAR-680-15`.
|
||||
2. Synthetic Blue Whale training policy: budget 2400 CNY, report within 7 days,
|
||||
unique marker `BLUEWHALE-2400-7`.
|
||||
|
||||
The integration key has **only `retrieve` capability**, `full_access=false`,
|
||||
and an upstream allowlist containing the test base. A second empty base exists
|
||||
in the same workspace solely to test rejection by that API-key allowlist.
|
||||
Secrets and local resource IDs are kept outside the repository.
|
||||
|
||||
## Observed checks
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Native Ollama embeds two Chinese strings | Two 1024-dimensional vectors |
|
||||
| Container reaches native Ollama | Version endpoint succeeds |
|
||||
| API health / frontend page / frontend-proxied login | HTTP 200 |
|
||||
| Catalog → selected document → live retrieval → shared evidence | Passed opt-in pytest |
|
||||
| Actual DeerFlow ToolNode with executing subagent identity | Live evidence returned |
|
||||
| ToolMessage serialization/replay and parent source forwarding | Valid sources preserved |
|
||||
| Source pruning after evidence removal | Empty source list |
|
||||
| Select Cedar document | Cedar marker present; Blue Whale marker absent |
|
||||
| Select Blue Whale document | Blue Whale marker present; Cedar marker absent |
|
||||
| All-accessible scope within configured base | Cedar evidence retrieved |
|
||||
| Unknown document selection | Rejected without unrestricted retry |
|
||||
| Unbound DeerFlow user | Rejected |
|
||||
| Restricted API key reads the second base | HTTP 403 |
|
||||
| Read-only API key attempts a synthetic draft write | HTTP 403; no document created |
|
||||
|
||||
The live pytest is `backend/tests/test_weknora_extension_live.py`; it supplies a
|
||||
deterministic tool call and requires an already indexed document. This tests
|
||||
real retrieval and tool/artifact integration, **not generative answer quality**.
|
||||
The separate boundary probes used the two synthetic documents described above.
|
||||
No interactive DeerFlow browser chat or external provider deployment is claimed.
|
||||
|
||||
## Compatibility finding fixed during this run
|
||||
|
||||
The initial adapter used `/api/v1/knowledge-search`. With the correctly scoped
|
||||
read-only key, WeKnora v0.8.3 returned HTTP 403:
|
||||
`API key scope does not allow knowledge_ids without a verified knowledge base`.
|
||||
Passing base IDs alongside document IDs is unsafe because that endpoint unions
|
||||
those targets; granting broader key permissions would also weaken isolation.
|
||||
|
||||
The adapter now uses
|
||||
`POST /api/v1/knowledge-bases/{id}/hybrid-search` with `query_text`, optional
|
||||
`knowledge_ids`, and `skip_context_enrichment=true`. The path enforces the base
|
||||
binding and document IDs narrow it. This endpoint supplies raw recall without
|
||||
requiring a reranker. A failing offline regression was added before the change;
|
||||
the restricted-key live test and the two-document isolation probes pass after it.
|
||||
|
||||
## Regression checks after the compatibility fix
|
||||
|
||||
- Full backend offline suite: **31,897 passed** (four duration-balanced shards;
|
||||
optional/environment-dependent tests retain their reported skips).
|
||||
- Strict blocking-I/O suite: **257 passed, 1 skipped**.
|
||||
- Focused adapter plus its SSL/blocking regression: **40 passed**.
|
||||
- Opt-in live test against this deployment: **1 passed**, including the real
|
||||
ToolNode, replay, forwarding and pruning checks above.
|
||||
- Ruff lint/format, extension wheel packaging and agent-guidance checks passed.
|
||||
@ -0,0 +1,13 @@
|
||||
"""Installable WeKnora read-only provider; all evidence goes through core."""
|
||||
|
||||
from deerflow_extension_api import extension
|
||||
|
||||
from .provider import WeKnoraProvider
|
||||
|
||||
__all__ = ["WeKnoraProvider", "install"]
|
||||
|
||||
|
||||
@extension(api="0.2.7", name="weknora")
|
||||
def install(registry, config):
|
||||
if not registry.knowledge_provider("weknora", WeKnoraProvider(dict(config))):
|
||||
raise RuntimeError("This host does not support knowledge provider registration")
|
||||
@ -0,0 +1,62 @@
|
||||
"""Operator-owned connection routing; no resource ACL database or client metadata."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from pydantic import AnyHttpUrl, BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
|
||||
def native_id(value: object) -> str:
|
||||
# WeKnora uses UUID-like IDs. Reject path syntax before constructing URLs.
|
||||
if not isinstance(value, str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,256}", value):
|
||||
raise ValueError("Invalid WeKnora resource ID")
|
||||
return value
|
||||
|
||||
|
||||
class Space(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", frozen=True)
|
||||
|
||||
base_url: AnyHttpUrl
|
||||
tenant_id: str = Field(pattern=r"^[1-9][0-9]{0,19}$")
|
||||
api_key_env: str = Field(pattern=r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
user_ids: tuple[str, ...] = Field(min_length=1)
|
||||
agent_names: tuple[str, ...] | None = Field(default=None, min_length=1)
|
||||
knowledge_base_ids: tuple[str, ...] | None = Field(default=None, min_length=1, max_length=100)
|
||||
|
||||
@field_validator("base_url")
|
||||
@classmethod
|
||||
def safe_url(cls, value: AnyHttpUrl) -> AnyHttpUrl:
|
||||
if value.username is not None or value.password is not None or value.query is not None or value.fragment is not None:
|
||||
raise ValueError("base_url must not contain credentials, query or fragment")
|
||||
return value
|
||||
|
||||
@field_validator("user_ids", "agent_names")
|
||||
@classmethod
|
||||
def identities(cls, value: tuple[str, ...] | None) -> tuple[str, ...] | None:
|
||||
if value is not None and any(not item.strip() or item != item.strip() for item in value):
|
||||
raise ValueError("Identity bindings must be nonblank exact host identities")
|
||||
return value
|
||||
|
||||
@field_validator("knowledge_base_ids")
|
||||
@classmethod
|
||||
def resource_ids(cls, value: tuple[str, ...] | None) -> tuple[str, ...] | None:
|
||||
if value is not None:
|
||||
for item in value:
|
||||
native_id(item)
|
||||
return value
|
||||
|
||||
|
||||
class Settings(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", frozen=True)
|
||||
|
||||
spaces: tuple[Space, ...] = Field(min_length=1, max_length=20)
|
||||
timeout: float = Field(default=30, gt=0, le=120, allow_inf_nan=False)
|
||||
match_count: int = Field(default=8, ge=1, le=100, strict=True)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def unique_connections(self) -> Settings:
|
||||
keys = [(str(space.base_url).rstrip("/"), space.tenant_id) for space in self.spaces]
|
||||
if len(set(keys)) != len(keys):
|
||||
raise ValueError("Configure each endpoint/tenant once, with its explicit user bindings")
|
||||
return self
|
||||
@ -0,0 +1,234 @@
|
||||
"""WeKnora v0.8.3 read-only adapter using DeerFlow's phase 2 contracts.
|
||||
|
||||
API/validation approach informed by superbuzzy's feat/weknora-integration;
|
||||
this package uses current contracts, request-local bindings and shared sources.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from deerflow.knowledge import KnowledgeCapabilities, KnowledgeRequestContext, KnowledgeSource, RetrievalCatalogPage, RetrievalHit, RetrievalResult
|
||||
from deerflow.knowledge_scope import execution_scope
|
||||
|
||||
from .config import Settings, Space, native_id
|
||||
|
||||
_MAX_RESPONSE_BYTES = 4 * 1024 * 1024
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Binding:
|
||||
space: Space
|
||||
key: str = field(repr=False)
|
||||
|
||||
def text(self, value: object, limit: int) -> str:
|
||||
return value.replace(self.key, "[REDACTED]").strip()[:limit] if isinstance(value, str) else ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Dataset:
|
||||
binding: _Binding
|
||||
native_id: str
|
||||
name: str
|
||||
|
||||
@property
|
||||
def id(self) -> str:
|
||||
# Recomputed from authorized catalogs; no reverse-map database, truncation,
|
||||
# key material or ambiguous concatenation. Key rotation preserves IDs.
|
||||
identity = [str(self.binding.space.base_url).rstrip("/"), self.binding.space.tenant_id, self.native_id]
|
||||
return "wk1_" + hashlib.sha256(json.dumps(identity, ensure_ascii=False).encode()).hexdigest()
|
||||
|
||||
|
||||
def _items(data: object) -> list[dict[str, Any]]:
|
||||
# Go nil slices are encoded as null, including an empty search result.
|
||||
if data is None:
|
||||
return []
|
||||
if not isinstance(data, list) or any(not isinstance(item, dict) for item in data):
|
||||
raise RuntimeError("Invalid WeKnora list response")
|
||||
return data
|
||||
|
||||
|
||||
def _searchable(document: dict[str, Any]) -> bool:
|
||||
return document.get("parse_status") in {"completed", "finalizing"} and document.get("enable_status") == "enabled"
|
||||
|
||||
|
||||
def _pagination(page: int, page_size: int) -> None:
|
||||
if type(page) is not int or page < 1 or type(page_size) is not int or not 1 <= page_size <= 100:
|
||||
raise ValueError("Invalid catalog pagination")
|
||||
|
||||
|
||||
class WeKnoraProvider:
|
||||
capabilities = KnowledgeCapabilities(operations={"search", "sources", "catalog", "documents", "dataset_filter", "document_filter"})
|
||||
|
||||
def __init__(self, config: dict[str, Any], *, transport: httpx.AsyncBaseTransport | None = None):
|
||||
self.settings = Settings.model_validate(config)
|
||||
self._transport = transport
|
||||
|
||||
def _bindings(self, context: KnowledgeRequestContext) -> list[_Binding]:
|
||||
bindings = []
|
||||
for space in self.settings.spaces:
|
||||
if context.user_id not in space.user_ids or (space.agent_names is not None and context.agent_name not in space.agent_names):
|
||||
continue
|
||||
key = os.getenv(space.api_key_env, "").strip()
|
||||
if not key or not key.isascii() or any(ord(char) < 33 or ord(char) == 127 for char in key):
|
||||
raise RuntimeError("WeKnora API key environment variable is missing or invalid")
|
||||
bindings.append(_Binding(space, key))
|
||||
if not bindings:
|
||||
raise PermissionError("No WeKnora space is bound to this user and executing agent")
|
||||
return bindings
|
||||
|
||||
async def _request(self, binding: _Binding, method: str, path: str, **kwargs: Any) -> dict[str, Any]:
|
||||
# Client per request prevents tenant/credential mutation across users and
|
||||
# event loops. SSL trust loading runs off-loop. Never follow redirects
|
||||
# carrying privileged credentials.
|
||||
try:
|
||||
async with await asyncio.to_thread(
|
||||
httpx.AsyncClient,
|
||||
base_url=str(binding.space.base_url).rstrip("/"),
|
||||
headers={"X-API-Key": binding.key, "X-Tenant-ID": binding.space.tenant_id},
|
||||
timeout=self.settings.timeout,
|
||||
follow_redirects=False,
|
||||
transport=self._transport,
|
||||
) as client:
|
||||
async with client.stream(method, path, **kwargs) as response:
|
||||
if response.status_code in {401, 403}:
|
||||
raise PermissionError("WeKnora denied resource access")
|
||||
if response.status_code == 404:
|
||||
raise FileNotFoundError("WeKnora resource is unavailable")
|
||||
if response.status_code != 200:
|
||||
raise RuntimeError("WeKnora request failed")
|
||||
raw = bytearray()
|
||||
async for part in response.aiter_bytes():
|
||||
raw.extend(part)
|
||||
if len(raw) > _MAX_RESPONSE_BYTES:
|
||||
raise RuntimeError("WeKnora response exceeds the supported size")
|
||||
payload = json.loads(raw)
|
||||
except (httpx.HTTPError, ValueError):
|
||||
# No raw body, URL, request headers or credential-bearing cause.
|
||||
raise RuntimeError("WeKnora request failed or returned invalid JSON") from None
|
||||
if not isinstance(payload, dict) or payload.get("success") is not True or "data" not in payload:
|
||||
raise RuntimeError("WeKnora returned an unsuccessful response")
|
||||
return payload
|
||||
|
||||
async def _datasets(self, context: KnowledgeRequestContext) -> dict[str, _Dataset]:
|
||||
result: dict[str, _Dataset] = {}
|
||||
for binding in self._bindings(context):
|
||||
payload = await self._request(binding, "GET", "/api/v1/knowledge-bases")
|
||||
bases = _items(payload["data"])
|
||||
allowed = binding.space.knowledge_base_ids
|
||||
found = {native_id(base.get("id")) for base in bases}
|
||||
if allowed is not None and not set(allowed) <= found:
|
||||
raise PermissionError("A configured WeKnora knowledge base is unavailable")
|
||||
for base in bases:
|
||||
base_id = native_id(base.get("id"))
|
||||
if allowed is not None and base_id not in allowed:
|
||||
continue
|
||||
dataset = _Dataset(binding, base_id, binding.text(base.get("name"), 500) or "Unnamed knowledge base")
|
||||
if dataset.id in result:
|
||||
raise RuntimeError("Duplicate WeKnora knowledge base")
|
||||
result[dataset.id] = dataset
|
||||
if len(result) > 100:
|
||||
raise RuntimeError("Bind at most 100 knowledge bases using knowledge_base_ids")
|
||||
return result
|
||||
|
||||
async def catalog(self, *, context: KnowledgeRequestContext, page: int, page_size: int, search: str) -> RetrievalCatalogPage:
|
||||
_pagination(page, page_size)
|
||||
datasets = await self._datasets(context)
|
||||
items = [{"id": ds.id, "name": ds.name, "selectable": True} for ds in datasets.values() if search.strip().casefold() in ds.name.casefold()]
|
||||
start = (page - 1) * page_size
|
||||
return RetrievalCatalogPage(items=items[start : start + page_size], total=len(items), page=page, page_size=page_size)
|
||||
|
||||
async def documents(self, dataset_id: str, *, context: KnowledgeRequestContext, page: int, page_size: int, search: str) -> RetrievalCatalogPage:
|
||||
_pagination(page, page_size)
|
||||
datasets = await self._datasets(context)
|
||||
if dataset_id not in datasets:
|
||||
raise PermissionError("Selected WeKnora knowledge base is unavailable")
|
||||
ds = datasets[dataset_id]
|
||||
payload = await self._request(ds.binding, "GET", f"/api/v1/knowledge-bases/{ds.native_id}/knowledge", params={"page": page, "page_size": page_size, "keyword": search})
|
||||
items = []
|
||||
for doc in _items(payload["data"]):
|
||||
if doc.get("knowledge_base_id") != ds.native_id:
|
||||
raise PermissionError("WeKnora returned a document outside the requested base")
|
||||
items.append({"id": native_id(doc.get("id")), "name": ds.binding.text(doc.get("title") or doc.get("file_name"), 500) or "Unnamed document", "selectable": _searchable(doc)})
|
||||
total = payload.get("total")
|
||||
if type(total) is not int or total < 0:
|
||||
raise RuntimeError("Invalid WeKnora document total")
|
||||
return RetrievalCatalogPage(items=items, total=total, page=page, page_size=page_size)
|
||||
|
||||
async def _document(self, ds: _Dataset, document_id: str) -> dict[str, Any]:
|
||||
payload = await self._request(ds.binding, "GET", f"/api/v1/knowledge/{native_id(document_id)}")
|
||||
doc = payload["data"]
|
||||
if not isinstance(doc, dict):
|
||||
raise RuntimeError("Invalid WeKnora document response")
|
||||
if doc.get("id") != document_id or doc.get("knowledge_base_id") != ds.native_id or not _searchable(doc):
|
||||
raise PermissionError("Selected WeKnora document is unavailable or not searchable")
|
||||
return doc
|
||||
|
||||
async def search(self, query: str) -> RetrievalResult:
|
||||
raise PermissionError("WeKnora retrieval requires a trusted KnowledgeRequestContext")
|
||||
|
||||
async def search_scoped(self, query: str, *, context: KnowledgeRequestContext) -> RetrievalResult:
|
||||
scope = execution_scope(context.scope) if context.scope is not None else {"mode": "all"}
|
||||
if scope["mode"] == "disabled":
|
||||
raise ValueError("Knowledge search is disabled for this turn")
|
||||
if not query.strip():
|
||||
raise ValueError("Knowledge query must not be empty")
|
||||
filters = {item["dataset_id"]: item["document_ids"] for item in scope.get("document_filters", [])}
|
||||
for ids in filters.values():
|
||||
for document_id in ids:
|
||||
native_id(document_id)
|
||||
datasets = await self._datasets(context)
|
||||
selected = scope["dataset_ids"] if scope["mode"] == "selected" else list(datasets)
|
||||
if not set(selected) <= datasets.keys():
|
||||
raise PermissionError("Selected WeKnora knowledge base is unavailable")
|
||||
documents: dict[tuple[str, str], dict[str, Any]] = {}
|
||||
# Validate every constrained resource before any search; no partial
|
||||
# success and no unfiltered retry if any selection fails.
|
||||
for dataset_id, ids in filters.items():
|
||||
for document_id in ids:
|
||||
documents[dataset_id, document_id] = await self._document(datasets[dataset_id], document_id)
|
||||
hits = []
|
||||
for dataset_id in selected:
|
||||
if len(hits) >= 100:
|
||||
break
|
||||
ds = datasets[dataset_id]
|
||||
body: dict[str, Any] = {"query_text": query, "match_count": self.settings.match_count, "skip_context_enrichment": True}
|
||||
# The KB-bound endpoint authorizes restricted API keys and intersects
|
||||
# document IDs with this base. The global knowledge-search endpoint
|
||||
# rejects knowledge_ids for such keys; adding base targets there
|
||||
# would also widen retrieval through its union semantics.
|
||||
if dataset_id in filters:
|
||||
body["knowledge_ids"] = filters[dataset_id]
|
||||
payload = await self._request(ds.binding, "POST", f"/api/v1/knowledge-bases/{ds.native_id}/hybrid-search", json=body)
|
||||
results = _items(payload["data"])
|
||||
if len(results) > 100:
|
||||
raise RuntimeError("WeKnora response exceeds the supported result limit")
|
||||
for item in results:
|
||||
if len(hits) >= 100:
|
||||
break
|
||||
document_id = native_id(item.get("knowledge_id"))
|
||||
if dataset_id in filters and document_id not in filters[dataset_id]:
|
||||
raise PermissionError("WeKnora returned evidence outside the selected documents")
|
||||
cache_key = (dataset_id, document_id)
|
||||
if cache_key not in documents:
|
||||
documents[cache_key] = await self._document(ds, document_id)
|
||||
# SearchResult has no knowledge_base_id; the authoritative detail
|
||||
# endpoint above verifies membership even for unfiltered search.
|
||||
text = ds.binding.text(item.get("content"), 100_000)
|
||||
if not text:
|
||||
continue
|
||||
title = ds.binding.text(item.get("knowledge_title") or item.get("knowledge_filename") or documents[cache_key].get("title") or documents[cache_key].get("file_name"), 500)
|
||||
source = KnowledgeSource(provider="weknora", title=title, excerpt=text, dataset_name=ds.name) if title else None
|
||||
# v0.8.3 gives offsets/chunk indexes, not a verified page or public
|
||||
# document URL. Leave locators absent instead of synthesizing them.
|
||||
score = item.get("score")
|
||||
score = score if type(score) in {float, int} and math.isfinite(score) else None
|
||||
hits.append(RetrievalHit(content=text, source=source, score=score))
|
||||
return RetrievalResult(hits=hits)
|
||||
25
examples/deerflow-extension-weknora/pyproject.toml
Normal file
25
examples/deerflow-extension-weknora/pyproject.toml
Normal file
@ -0,0 +1,25 @@
|
||||
[project]
|
||||
name = "deerflow-extension-weknora"
|
||||
version = "0.1.0"
|
||||
description = "Read-only WeKnora retrieval and scope provider for DeerFlow"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = "MIT"
|
||||
dependencies = ["deerflow-extension-api>=0.2.7,<0.3", "deerflow-harness>=2.1,<3", "httpx>=0.27,<1", "pydantic>=2,<3"]
|
||||
|
||||
[project.entry-points."deerflow.extensions"]
|
||||
weknora = "deerflow_extension_weknora:install"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["deerflow_extension_weknora"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 240
|
||||
target-version = "py312"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "UP"]
|
||||
@ -45,6 +45,7 @@ export const extensionCatalog = [
|
||||
package: "deerflow-extension-knowledge",
|
||||
namespace: null,
|
||||
},
|
||||
{ key: "weknora", package: "deerflow-extension-weknora", namespace: null },
|
||||
{ key: "example", package: "deerflow-extension-example", namespace: null },
|
||||
] as const;
|
||||
|
||||
|
||||
@ -80,6 +80,11 @@ export const enUS: Translations = {
|
||||
description:
|
||||
"Demonstrate read-only retrieval and structured tool results using fixed sample documents, without an external knowledge service.",
|
||||
},
|
||||
weknora: {
|
||||
title: "WeKnora knowledge retrieval",
|
||||
description:
|
||||
"Search WeKnora knowledge bases and selected documents with source citations. Requires administrator installation and access configuration.",
|
||||
},
|
||||
knowledge: {
|
||||
title: "Knowledge scope example",
|
||||
description:
|
||||
|
||||
@ -35,6 +35,7 @@ export interface Translations {
|
||||
| "screening"
|
||||
| "retrieval"
|
||||
| "knowledge"
|
||||
| "weknora"
|
||||
| "example",
|
||||
{ title: string; description: string }
|
||||
>;
|
||||
|
||||
@ -76,6 +76,11 @@ export const zhCN: Translations = {
|
||||
description:
|
||||
"使用固定示例文档演示只读检索与结构化工具结果，不连接外部知识库。",
|
||||
},
|
||||
weknora: {
|
||||
title: "WeKnora 知识检索",
|
||||
description:
|
||||
"检索 WeKnora 知识库和所选文档，并展示来源引用。需要管理员安装并配置访问权限。",
|
||||
},
|
||||
knowledge: {
|
||||
title: "知识范围选择示例",
|
||||
description: "用示例文档演示知识库和文件范围选择、检索及来源引用。",
|
||||
|
||||
@ -448,7 +448,34 @@ for (const locale of ["en-US", "zh-CN"]) {
|
||||
exact: true,
|
||||
}),
|
||||
).toBeVisible();
|
||||
await expect(page.getByRole("article")).toHaveCount(9);
|
||||
await expect(page.getByRole("article")).toHaveCount(10);
|
||||
const weknoraTitle = zh
|
||||
? "WeKnora 知识检索"
|
||||
: "WeKnora knowledge retrieval";
|
||||
const weknora = page.getByRole("article").filter({ hasText: weknoraTitle });
|
||||
await expect(weknora).toHaveCount(1);
|
||||
await expect(weknora).toContainText(
|
||||
zh
|
||||
? "需要管理员安装并配置访问权限。"
|
||||
: "Requires administrator installation and access configuration.",
|
||||
);
|
||||
await weknora
|
||||
.getByRole("button", {
|
||||
name: zh ? `查看 ${weknoraTitle}` : `View ${weknoraTitle}`,
|
||||
exact: true,
|
||||
})
|
||||
.click();
|
||||
await expect(
|
||||
page.getByRole("link", {
|
||||
name: zh ? "查看安装说明" : "Installation guide",
|
||||
}),
|
||||
).toHaveAttribute("href", /deerflow-extension-weknora#readme$/);
|
||||
await page
|
||||
.getByRole("button", {
|
||||
name: zh ? "全部扩展" : "All extensions",
|
||||
exact: true,
|
||||
})
|
||||
.click();
|
||||
const knowledgeTitle = zh ? "知识范围选择示例" : "Knowledge scope example";
|
||||
const knowledge = page
|
||||
.getByRole("article")
|
||||
@ -638,10 +665,10 @@ test("catalog remains discoverable when runtime discovery fails", async ({
|
||||
await expect(
|
||||
page.getByRole("alert").filter({ hasText: "Extensions unavailable." }),
|
||||
).toBeVisible();
|
||||
await expect(page.getByRole("article")).toHaveCount(9);
|
||||
await expect(page.getByRole("article")).toHaveCount(10);
|
||||
await expect(
|
||||
page.getByRole("article").filter({ hasText: "Catalog extension" }),
|
||||
).toHaveCount(9);
|
||||
).toHaveCount(10);
|
||||
await expect(
|
||||
page.getByRole("article").filter({ hasText: "Script workflows" }),
|
||||
).toHaveCount(1);
|
||||
|
||||
@ -145,3 +145,22 @@ test("workflow metadata is localized, explains Phase 2 limits and merges its reg
|
||||
expect(matches[0]?.loaded).toBe(workflow);
|
||||
}
|
||||
});
|
||||
|
||||
test("WeKnora discovery is localized without implying runtime activation", () => {
|
||||
for (const [copy, title] of [
|
||||
[enUS.extensions.catalog, "WeKnora knowledge retrieval"],
|
||||
[zhCN.extensions.catalog, "WeKnora 知识检索"],
|
||||
] as const) {
|
||||
const entry = extensionDirectory([], copy).find(
|
||||
(item) => item.package === "deerflow-extension-weknora",
|
||||
);
|
||||
expect(entry).toMatchObject({
|
||||
id: "catalog/weknora",
|
||||
title,
|
||||
guide:
|
||||
"https://github.com/bytedance/deer-flow/tree/main/examples/deerflow-extension-weknora#readme",
|
||||
});
|
||||
expect(entry?.description).toBeTruthy();
|
||||
expect(entry?.loaded).toBeUndefined();
|
||||
}
|
||||
});
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user