diff --git a/README.md b/README.md index 4171cdb049..e432ae30d9 100644 --- a/README.md +++ b/README.md @@ -1440,8 +1440,12 @@ are rejected without truncation. RAGFlow keeps its existing configuration and citations, and search-only Providers remain usable without a catalog or management page. See the [offline Provider example](examples/deerflow-extension-knowledge/README.md) -for extension registration and configuration. WeKnora's adapter is planned for -the next phase. +for extension registration and configuration. The installable +[WeKnora read-only extension](examples/deerflow-extension-weknora/README.md) +targets WeKnora v0.8.3 and adds knowledge-base/document selection and shared +retrieval citations. Configure explicit user-to-workspace credential bindings; +resource authorization remains with WeKnora. The extension directory includes +its installation guide. Skills are what make DeerFlow do *almost anything*. diff --git a/README_zh.md b/README_zh.md index d090bc01be..d64a80a274 100644 --- a/README_zh.md +++ b/README_zh.md @@ -727,6 +727,11 @@ DeerFlow 可连接租户级 RAGFlow,并通过 `knowledge_search` 按 embedding 本版不在工作区侧边栏增加独立的“知识库”入口,也不提供 DeerFlow 知识库管理页面;知识库和文件的创建、上传、解析与删除仍直接在 RAGFlow 中完成。 +运维还可独立安装 [WeKnora 只读扩展](examples/deerflow-extension-weknora/README.md), +对接 WeKnora v0.8.3,使用知识库/文档范围选择器和共享检索引用。 +需显式配置 DeerFlow 用户到 WeKnora 工作空间的凭据绑定;资源授权仍由 WeKnora 执行。 +安装与配置步骤见扩展目录中的英文指南。 + Gateway 生成后续建议时,现在会先把普通字符串输出和 block/list 风格的富文本内容统一归一化,再去解析 JSON 数组响应,因此不同 provider 的内容包装方式不会再悄悄把建议吞掉。 Web UI 支持从已完成的 assistant 回复分叉出一个新的主对话。自动继承的分叉标题会使用下一个空闲的数字后缀(`标题 (2)`、`标题 (3)`……);显式指定或手动重命名得到的同名后缀也会占号,即使它没有生成序号 metadata,后续自动分叉也不会与它重名。API 调用方显式提供的标题保持不变;重命名会清除旧的生成序号,因此从新标题继续自动分叉时会重新从 `(2)` 开始。最近对话列表还会把已加载的分叉直接排列在已加载的父对话下方,并显示低干扰的树形连接线。父对话尚未加载、谱系数据错误或成环、父子置顶状态不一致时,分叉会安全地保留在顶层,不会被隐藏或跨越置顶边界移动。新 thread 会保留该轮回复的 checkpoint 以及用户消息之前的重放 checkpoint,因此分叉后可以立即重新生成该回复。对于缺少 checkpoint 父链接的旧历史或导入历史,Gateway 会进行有界的时间顺序查找;如果不存在更早的重放 checkpoint,分叉仍会按旧版单-checkpoint 形态成功创建,但无法重新生成继承的回复。已有的单-checkpoint 分叉会保持不变,不会通过不安全的 checkpoint 复制尝试修复。只有从最新回合分叉时才会尽力复制当前 thread 的工作区文件;从历史回合分叉不会带入后续时间线创建的文件。 diff --git a/backend/docs/KNOWLEDGE.md b/backend/docs/KNOWLEDGE.md index 7ec9ec5aae..c3b76997bb 100644 --- a/backend/docs/KNOWLEDGE.md +++ b/backend/docs/KNOWLEDGE.md @@ -1,11 +1,13 @@ -# Read-only knowledge contract (#6269, phases 1–2) +# Read-only knowledge contract (#6269, phases 1–3) `deerflow.knowledge` owns `RetrievalResult`, `RetrievalHit`, `KnowledgeSource`, `KnowledgeCapabilities`, and the search-only `RetrievalProvider` protocol. Search-only providers need no knowledge management service, resource store, catalog, or scope-filter implementation. Scope admission, the read-only catalog and selection UI consume the same operation-level capabilities. LightRAG still -does not advertise dataset/document filters; WeKnora remains phase 3. +does not advertise dataset/document filters. The phase 3 +[WeKnora extension](../../examples/deerflow-extension-weknora/README.md) implements +all six read-only operations against the v0.8.3 API. ## Provider registration and optional operations @@ -35,10 +37,10 @@ provider identifiers, including non-UUID identifiers. Gateway validates this bounded public projection and never returns raw provider payloads. Dataset and document IDs are limited to **256 Unicode code points** throughout catalog output, URL parameters and scope snapshots. Overlong IDs are rejected, -never truncated. A phase 3 WeKnora adapter must map space-qualified resources -to stable, unambiguous IDs within this bound and resolve them inside the -provider; the earlier `ks1_` encoding allowing 512 characters is not the public -contract. Reuse provider API/mapping logic without adding a second discovery +never truncated. The WeKnora extension maps endpoint/tenant/base tuples to stable +`wk1_` SHA-256 IDs (68 ASCII characters), resolving them from the authorized +catalog on each request without a mapping store. The earlier `ks1_` encoding +allowing 512 characters is not the public contract. Reuse provider API/mapping logic without adding a second discovery protocol or a DeerFlow ACL store. `KnowledgeRequestContext` supplies the host user ID, Agent name and optional @@ -251,3 +253,29 @@ uv run pytest tests/test_knowledge_contract_live.py -m live -q Set `LIGHTRAG_API_KEY` in the environment if required. This test performs no upload or resource changes. It requires attributable hits and checks each citation against the real returned chunk evidence. + +## WeKnora read-only extension + +The independently packaged `examples/deerflow-extension-weknora` registers +`weknora` through the existing registry. Its operator configuration binds exact +host users (and optionally current executor names) to credential environment +variables, endpoint, tenant and optional base allowlist. No automatic tenant +inference, default user binding, resource ACL storage or new core API is added. +All bound users share the configured credential's upstream visibility; the +plugin does not impersonate a WeKnora human user. + +The adapter uses v0.8.3's KB-bound `hybrid-search` with `query_text` and optional +`knowledge_ids`, disabling context enrichment. This supports KB-restricted +read-only API keys; the global search endpoint rejects their document targets +and unions base/document target lists. Raw recall needs no chat/rerank model. +The adapter validates all selected resources, checks returned document +membership/status, and fails the request on access errors without retrying an +unrestricted search. A missing title remains unattributed; pages and URLs are +omitted because the mapped response does not establish those locators. + +See the extension README for immutable API references, installation, limits and +opt-in real-provider verification. Offline synthetic responses exercise the +actual registry, Gateway catalog, ToolNode, shared evidence and replay path. +Live validation needs an indexed instance and is not implied by offline passes. +The extension's [validation record](../../examples/deerflow-extension-weknora/VALIDATION.md) +documents the local v0.8.3 / Ollama bge-m3 run. diff --git a/backend/tests/blocking_io/test_weknora_extension.py b/backend/tests/blocking_io/test_weknora_extension.py new file mode 100644 index 0000000000..2a15b2f633 --- /dev/null +++ b/backend/tests/blocking_io/test_weknora_extension.py @@ -0,0 +1,40 @@ +"""Exercise real HTTPX/SSL construction with only the network response replaced.""" + +import asyncio +import ssl +from pathlib import Path + +import httpx +import pytest + + +@pytest.fixture +def weknora_provider(monkeypatch): + monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[3] / "examples/deerflow-extension-weknora")) + from deerflow_extension_weknora import WeKnoraProvider + + monkeypatch.setenv("WEKNORA_BLOCKING_TEST_KEY", "synthetic") + return WeKnoraProvider({"spaces": [{"base_url": "https://weknora.invalid", "tenant_id": "1", "api_key_env": "WEKNORA_BLOCKING_TEST_KEY", "user_ids": ["alice"]}]}) + + +@pytest.mark.asyncio +async def test_weknora_ssl_setup_is_off_loop(weknora_provider, monkeypatch): + from deerflow.knowledge import KnowledgeRequestContext + + async def send(self, request, **kwargs): + assert request.headers["X-Tenant-ID"] == "1" + return httpx.Response(200, request=request, json={"success": True, "data": [{"id": "kb", "name": "Synthetic"}]}) + + original = ssl.create_default_context + + def ssl_context(*args, **kwargs): + try: + asyncio.get_running_loop() + except RuntimeError: + return original(*args, **kwargs) + raise AssertionError("SSL trust loading must run off the event loop") + + monkeypatch.setattr(ssl, "create_default_context", ssl_context) + monkeypatch.setattr(httpx.AsyncClient, "send", send) + catalog = await weknora_provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="") + assert catalog.items[0].name == "Synthetic" diff --git a/backend/tests/test_agent_guidance_check.py b/backend/tests/test_agent_guidance_check.py index 4a3b3be6b8..8be0abc7f3 100644 --- a/backend/tests/test_agent_guidance_check.py +++ b/backend/tests/test_agent_guidance_check.py @@ -55,6 +55,7 @@ EXPECTED_GUIDANCE_PATHS = { "scripts/AGENTS.md", "examples/deerflow-extension-agent-teams/AGENTS.md", "examples/deerflow-extension-workflows/AGENTS.md", + "examples/deerflow-extension-weknora/AGENTS.md", } diff --git a/backend/tests/test_weknora_extension.py b/backend/tests/test_weknora_extension.py new file mode 100644 index 0000000000..db9293e1bf --- /dev/null +++ b/backend/tests/test_weknora_extension.py @@ -0,0 +1,400 @@ +"""Synthetic v0.8.3 responses; never contacts an external provider.""" + +import json +from pathlib import Path + +import httpx +import pytest + +from deerflow.knowledge import KnowledgeRequestContext, format_result, sources_from_messages + + +@pytest.fixture +def adapter(monkeypatch): + monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[2] / "examples/deerflow-extension-weknora")) + from deerflow_extension_weknora import WeKnoraProvider + + monkeypatch.setenv("WEKNORA_TEST_KEY", "synthetic-secret") + calls = [] + document = {"id": "doc-1", "knowledge_base_id": "kb-1", "title": "Synthetic guide", "parse_status": "completed", "enable_status": "enabled"} + state = {"document": document, "hits": [{"knowledge_id": "doc-1", "content": "Cedar is synthetic evidence.", "knowledge_title": "Synthetic guide", "score": 0.8}]} + + def handler(request): + calls.append(request) + assert request.headers["X-API-Key"] == "synthetic-secret" + assert request.headers["X-Tenant-ID"] in {"1", "2"} + if state.get("status"): + return httpx.Response(state["status"], text="synthetic-secret upstream body") + if request.url.path == "/api/v1/knowledge-bases": + data = state.get("bases", [{"id": "kb-1", "name": "Synthetic base"}]) + elif request.url.path == "/api/v1/knowledge-bases/kb-1/knowledge": + return httpx.Response(200, json={"success": True, "data": [state["document"]], **state.get("pagination", {"total": 1})}) + elif request.url.path == "/api/v1/knowledge/doc-1": + data = state["document"] + elif request.url.path == "/api/v1/knowledge-search": + return httpx.Response(403, json={"success": False, "error": "KB-restricted keys reject global knowledge_ids"}) + elif request.url.path == "/api/v1/knowledge-bases/kb-1/hybrid-search": + data = state["hits"] + else: + return httpx.Response(404, json={"success": False}) + return httpx.Response(200, json={"success": True, "data": data}) + + config = {"spaces": [{"base_url": "https://weknora.example", "tenant_id": "1", "api_key_env": "WEKNORA_TEST_KEY", "user_ids": ["alice"], "agent_names": ["lead_agent", "researcher"]}]} + transport = httpx.MockTransport(handler) + return WeKnoraProvider(config, transport=transport), calls, state, config, transport + + +async def selection(provider, *, docs=True): + context = KnowledgeRequestContext("alice", "lead_agent") + page = await provider.catalog(context=context, page=1, page_size=20, search="") + dataset_id = page.items[0].id + scope = {"version": 1, "mode": "selected", "dataset_ids": [dataset_id]} + if docs: + scope["document_filters"] = [{"dataset_id": dataset_id, "document_ids": ["doc-1"]}] + return KnowledgeRequestContext("alice", "lead_agent", scope) + + +@pytest.mark.asyncio +async def test_catalog_documents_search_shared_evidence(adapter): + provider, calls, _, _, _ = adapter + ctx = await selection(provider) + page = await provider.documents(ctx.scope["dataset_ids"][0], context=ctx, page=1, page_size=20, search="guide") + assert page.items[0].selectable and page.items[0].id == "doc-1" + assert calls[-1].url.params["keyword"] == "guide" + result = await provider.search_scoped("Cedar", context=ctx) + body = json.loads(next(r.content for r in calls if r.method == "POST")) + assert body["knowledge_ids"] == ["doc-1"] + assert "knowledge_base_ids" not in body + assert body["query_text"] == "Cedar" + assert body["skip_context_enrichment"] is True + assert next(r for r in calls if r.method == "POST").url.path == "/api/v1/knowledge-bases/kb-1/hybrid-search" + content, artifact = format_result(result) + sources = sources_from_messages([{"type": "tool", "content": content, "artifact": artifact}]) + assert len(sources) == 1 and sources[0].provider == "weknora" + assert sources[0].excerpt == "Cedar is synthetic evidence." + assert sources[0].page is None and sources[0].url is None + assert sources_from_messages([{"type": "tool", "content": "removed", "artifact": artifact}]) == [] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("user,agent", [("mallory", "lead_agent"), ("alice", "unbound"), ("alice", None)]) +async def test_unbound_identity_cannot_make_requests(adapter, user, agent): + provider, calls, *_ = adapter + ctx = KnowledgeRequestContext(user, agent) + with pytest.raises(PermissionError): + await provider.catalog(context=ctx, page=1, page_size=20, search="") + with pytest.raises(PermissionError): + await provider.search_scoped("Cedar", context=ctx) + assert not calls + + +@pytest.mark.asyncio +async def test_no_context_search_is_not_an_authorization_bypass(adapter): + provider, calls, *_ = adapter + with pytest.raises(PermissionError): + await provider.search("Cedar") + assert not calls + + +@pytest.mark.asyncio +@pytest.mark.parametrize("scope", [{"version": 1, "mode": "disabled"}, {"mode": "selected", "dataset_ids": []}]) +async def test_disabled_or_invalid_scope_never_calls_upstream(adapter, scope): + provider, calls, *_ = adapter + with pytest.raises(ValueError): + await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent", scope)) + assert not calls + + +@pytest.mark.asyncio +@pytest.mark.parametrize("change", [{"knowledge_base_id": "other"}, {"id": "other"}, {"enable_status": "disabled"}, {"parse_status": "pending"}]) +async def test_document_validation_precedes_search(adapter, change): + provider, calls, state, *_ = adapter + ctx = await selection(provider) + state["document"].update(change) + with pytest.raises(PermissionError): + await provider.search_scoped("Cedar", context=ctx) + assert not any(r.method == "POST" for r in calls) + + +@pytest.mark.asyncio +async def test_unselected_result_fails_entire_request(adapter): + provider, calls, state, *_ = adapter + ctx = await selection(provider) + state["hits"].append({"knowledge_id": "other", "content": "must never escape"}) + with pytest.raises(PermissionError): + await provider.search_scoped("Cedar", context=ctx) + assert len([r for r in calls if r.method == "POST"]) == 1 + + +@pytest.mark.asyncio +async def test_dataset_search_verifies_returned_document_membership(adapter): + provider, _, state, *_ = adapter + ctx = await selection(provider, docs=False) + state["document"]["knowledge_base_id"] = "other" + with pytest.raises(PermissionError): + await provider.search_scoped("Cedar", context=ctx) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("status,error", [(401, PermissionError), (403, PermissionError), (404, FileNotFoundError), (500, RuntimeError), (302, RuntimeError)]) +async def test_http_errors_are_redacted_and_never_retried(adapter, status, error): + provider, calls, state, *_ = adapter + state["status"] = status + with pytest.raises(error) as exc: + await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="") + assert "synthetic-secret" not in str(exc.value) + assert len(calls) == 1 + + +@pytest.mark.asyncio +async def test_multi_space_ids_are_stable_bounded_and_isolated(adapter): + provider, _, _, config, transport = adapter + from deerflow_extension_weknora import WeKnoraProvider + + config["spaces"].append({**config["spaces"][0], "tenant_id": "2", "user_ids": ["bob"]}) + multi = WeKnoraProvider(config, transport=transport) + alice = await multi.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="") + bob = await multi.catalog(context=KnowledgeRequestContext("bob", "lead_agent"), page=1, page_size=20, search="") + old = await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="") + assert alice.items[0].id == old.items[0].id != bob.items[0].id + assert len(alice.items[0].id) <= 256 + with pytest.raises(PermissionError): + await multi.search_scoped("Cedar", context=KnowledgeRequestContext("bob", "lead_agent", {"version": 1, "mode": "selected", "dataset_ids": [alice.items[0].id]})) + + +@pytest.mark.asyncio +async def test_all_scope_still_sends_explicit_targets_and_subagent_works(adapter): + provider, calls, *_ = adapter + await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "researcher")) + body = json.loads(next(r.content for r in calls if r.method == "POST")) + assert "knowledge_base_ids" not in body and "knowledge_ids" not in body + assert next(r for r in calls if r.method == "POST").url.path == "/api/v1/knowledge-bases/kb-1/hybrid-search" + + +@pytest.mark.asyncio +async def test_secrets_redacted_and_missing_attribution_not_invented(adapter): + provider, _, state, *_ = adapter + state["hits"][0].update(content="synthetic-secret evidence", knowledge_title="") + state["document"]["title"] = "" + result = await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent")) + assert "synthetic-secret" not in result.model_dump_json() + assert result.hits[0].source is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("bad_id", ["..", "../secret", "%2e%2e", "a/b", "x" * 257]) +async def test_unsafe_native_ids_fail_before_path_request(adapter, bad_id): + provider, calls, *_ = adapter + ctx = await selection(provider) + ctx.scope["document_filters"][0]["document_ids"] = [bad_id] + with pytest.raises(ValueError): + await provider.search_scoped("Cedar", context=ctx) + assert not any("/knowledge/" in r.url.path or r.method == "POST" for r in calls) + + +def test_configuration_rejects_implicit_authorization_and_credentials(adapter): + from deerflow_extension_weknora import WeKnoraProvider + + _, _, _, config, _ = adapter + for change in ({"tenant_id": None}, {"user_ids": []}, {"base_url": "https://user:secret@example.com"}, {"api_key_env": ""}, {"knowledge_base_ids": []}): + with pytest.raises(ValueError): + WeKnoraProvider({"spaces": [{**config["spaces"][0], **change}]}) + + +@pytest.mark.asyncio +async def test_packaged_registration_real_tool_replay_budget_and_delegation(adapter, monkeypatch): + from types import SimpleNamespace + + from langchain_core.messages import AIMessage + from langgraph.graph import END, START, MessagesState, StateGraph + from langgraph.prebuilt import ToolNode + + from deerflow.extensions import EXTENSION_SNAPSHOT_CONTEXT_KEY, load_extensions + from deerflow.extensions.loader import ExtensionSpec + from deerflow.knowledge import tools + from deerflow.knowledge.providers import GENERIC_KNOWLEDGE_TOOL, resolve_knowledge_provider + from deerflow.knowledge.sources import prune_sources + from deerflow.runtime.serialization import serialize_channel_values_for_api + from deerflow.tools.builtins.task_tool import _task_result_command + + _, _, _, config, transport = adapter + loaded, diagnostics = load_extensions([ExtensionSpec(use="deerflow_extension_weknora:install", config=config)]) + assert diagnostics == [] + app_config = SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(use=GENERIC_KNOWLEDGE_TOOL, model_extra={"provider": "weknora"})) + provider = resolve_knowledge_provider(app_config, extensions=loaded) + provider._transport = transport + ctx = await selection(provider) + monkeypatch.setattr(tools, "get_app_config", lambda: app_config) + builder = StateGraph(MessagesState, context_schema=dict) + builder.add_node("tools", ToolNode([tools.knowledge_search_tool])) + builder.add_edge(START, "tools") + builder.add_edge("tools", END) + result = await builder.compile().ainvoke( + {"messages": [AIMessage(content="", tool_calls=[{"id": "call", "name": "knowledge_search", "args": {"query": "Cedar"}}])]}, + context={"user_id": "alice", "is_subagent": True, "agent_id": "researcher", "__knowledge_scope_execution": ctx.scope, EXTENSION_SNAPSHOT_CONTEXT_KEY: loaded}, + ) + message = result["messages"][-1] + sources = sources_from_messages([message]) + assert len(sources) == 1 and sources[0].provider == "weknora" + persisted = json.loads(json.dumps(serialize_channel_values_for_api({"messages": [message]}))) + assert sources_from_messages(persisted["messages"]) == sources + command = _task_result_command(tool_call_id="task", status="completed", result=message.content, knowledge_sources=[s.model_dump(mode="json", exclude_none=True) for s in sources]) + assert sources_from_messages(command.update["messages"]) == sources + _, pruned = prune_sources("truncated", message.artifact) + assert pruned["knowledge_sources"] == [] + + +@pytest.mark.asyncio +async def test_missing_key_fails_closed(adapter, monkeypatch): + provider, calls, *_ = adapter + monkeypatch.delenv("WEKNORA_TEST_KEY") + with pytest.raises(RuntimeError): + await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent")) + assert not calls + + +@pytest.mark.asyncio +async def test_allowlist_and_missing_configured_base(adapter): + from deerflow_extension_weknora import WeKnoraProvider + + _, calls, state, config, transport = adapter + config["spaces"][0]["knowledge_base_ids"] = ["kb-1"] + provider = WeKnoraProvider(config, transport=transport) + state["bases"] = [{"id": "kb-1", "name": "Allowed"}, {"id": "kb-2", "name": "Excluded"}] + page = await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="") + assert [item.name for item in page.items] == ["Allowed"] + state["bases"] = [{"id": "kb-2", "name": "Excluded"}] + with pytest.raises(PermissionError): + await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent")) + assert not any(r.method == "POST" for r in calls) + + +@pytest.mark.asyncio +async def test_empty_catalog_never_sends_unconstrained_search(adapter): + provider, calls, state, *_ = adapter + state["bases"] = None + result = await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent")) + assert result.hits == [] and not any(r.method == "POST" for r in calls) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("payload", [[], {"success": False, "data": []}, {"success": True}, {"success": True, "data": "bad"}]) +async def test_malformed_upstream_response_fails_closed(adapter, payload): + provider, *_ = adapter + provider._transport = httpx.MockTransport(lambda _: httpx.Response(200, json=payload)) + with pytest.raises(RuntimeError): + await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="") + + +@pytest.mark.asyncio +async def test_response_size_limit(adapter): + provider, *_ = adapter + provider._transport = httpx.MockTransport(lambda _: httpx.Response(200, content=b"x" * (4 * 1024 * 1024 + 1))) + with pytest.raises(RuntimeError, match="size"): + await provider.catalog(context=KnowledgeRequestContext("alice", "lead_agent"), page=1, page_size=20, search="") + + +@pytest.mark.asyncio +async def test_all_document_selections_checked_before_any_search(adapter): + provider, calls, *_ = adapter + ctx = await selection(provider) + ctx.scope["document_filters"][0]["document_ids"].append("missing") + with pytest.raises(FileNotFoundError): + await provider.search_scoped("Cedar", context=ctx) + assert not any(r.method == "POST" for r in calls) + + +def test_gateway_catalog_uses_authenticated_user_with_real_adapter(adapter): + from _router_auth_helpers import make_authed_test_app + from deerflow_extension_weknora import WeKnoraProvider + from fastapi.testclient import TestClient + + from app.gateway.auth.models import User + from app.gateway.deps import get_config + from app.gateway.routers import knowledge + from deerflow.config.app_config import AppConfig + from deerflow.extensions import ExtensionRegistry + + _, calls, _, config, transport = adapter + user = User(email="weknora@example.com", password_hash="unused", system_role="user") + config["spaces"][0]["user_ids"] = [str(user.id)] + provider = WeKnoraProvider(config, transport=transport) + registry = ExtensionRegistry() + with registry.attributed_to("deerflow_extension_weknora:install"): + registry.knowledge_provider("weknora", provider) + app = make_authed_test_app(user_factory=lambda: user, bind_current_user=True) + app.state.extensions = registry.build() + app.include_router(knowledge.router) + app_config = AppConfig.model_validate( + { + "models": [], + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "knowledge_base": {"enabled": True, "scope_selection_enabled": True}, + "tools": [{"name": "knowledge_search", "group": "knowledge", "use": "deerflow.knowledge.tools:knowledge_search_tool", "provider": "weknora"}], + } + ) + app.dependency_overrides[get_config] = lambda: app_config + with TestClient(app) as client: + cap = client.get("/api/knowledge/capabilities?agent_name=lead_agent") + assert cap.status_code == 200 and cap.json()["scope_selection_enabled"] is True + catalog = client.get("/api/knowledge/retrieval-catalog/datasets?agent_name=lead_agent&user_id=mallory") + assert catalog.status_code == 200 + dataset_id = catalog.json()["items"][0]["id"] + documents = client.get(f"/api/knowledge/retrieval-catalog/datasets/{dataset_id}/documents?agent_name=lead_agent") + assert documents.status_code == 200 and documents.json()["items"][0]["selectable"] is True + assert client.get("/api/knowledge/retrieval-catalog/datasets/unknown/documents?agent_name=lead_agent").status_code == 403 + assert all(r.method == "GET" for r in calls) + + +@pytest.mark.asyncio +async def test_oversized_hit_list_does_not_cause_unbounded_detail_requests(adapter): + provider, calls, state, *_ = adapter + state["hits"] = [{"knowledge_id": f"doc-{i}", "content": "synthetic"} for i in range(101)] + with pytest.raises(RuntimeError, match="result limit"): + await provider.search_scoped("Cedar", context=KnowledgeRequestContext("alice", "lead_agent")) + assert not any("/knowledge/" in r.url.path for r in calls) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("pagination", [{}, {"total": None}, {"total": "1"}, {"total": True}, {"total": -1}, {"total": 1.5}]) +async def test_invalid_document_total_is_explicit_provider_error(adapter, pagination): + provider, calls, state, *_ = adapter + ctx = await selection(provider) + state["pagination"] = pagination + with pytest.raises(RuntimeError, match="Invalid WeKnora document total"): + await provider.documents(ctx.scope["dataset_ids"][0], context=ctx, page=1, page_size=20, search="") + assert not any(r.method == "POST" for r in calls) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("target", ["hits", "document"]) +@pytest.mark.parametrize("data", ["bad", [], ["bad"], 42]) +async def test_malformed_retrieval_payloads_fail_without_retry(adapter, target, data): + provider, calls, state, *_ = adapter + ctx = await selection(provider) + state[target] = data + if target == "hits" and data == []: + assert (await provider.search_scoped("Cedar", context=ctx)).hits == [] + else: + with pytest.raises(RuntimeError, match="Invalid WeKnora"): + await provider.search_scoped("Cedar", context=ctx) + assert len([r for r in calls if r.method == "POST"]) == (1 if target == "hits" else 0) + + +@pytest.mark.asyncio +async def test_retrieval_response_size_limit_without_retry(adapter): + provider, calls, _, _, transport = adapter + ctx = await selection(provider, docs=False) + posts = [] + + async def handler(request): + if request.method == "POST": + posts.append(request) + return httpx.Response(200, content=b"x" * (4 * 1024 * 1024 + 1)) + return await transport.handle_async_request(request) + + provider._transport = httpx.MockTransport(handler) + with pytest.raises(RuntimeError, match="size"): + await provider.search_scoped("Cedar", context=ctx) + assert len(posts) == 1 + assert not any("/knowledge/" in r.url.path for r in calls) diff --git a/backend/tests/test_weknora_extension_live.py b/backend/tests/test_weknora_extension_live.py new file mode 100644 index 0000000000..63142d58e4 --- /dev/null +++ b/backend/tests/test_weknora_extension_live.py @@ -0,0 +1,88 @@ +"""Opt-in read-only WeKnora v0.8.3 integration; needs existing indexed evidence.""" + +import json +import os +from pathlib import Path + +import pytest + + +@pytest.mark.live +@pytest.mark.asyncio +async def test_weknora_live_selected_document_evidence(monkeypatch): + if os.getenv("DEER_FLOW_RUN_LIVE_TESTS") != "1": + pytest.skip("live knowledge checks are opt-in") + names = ["WEKNORA_BASE_URL", "WEKNORA_TENANT_ID", "WEKNORA_KNOWLEDGE_BASE_ID", "WEKNORA_DOCUMENT_ID", "WEKNORA_EVIDENCE_QUERY", "WEKNORA_API_KEY"] + if any(not os.getenv(name) for name in names): + pytest.fail("Set all documented WEKNORA_* variables for live verification") + monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[2] / "examples/deerflow-extension-weknora")) + from deerflow_extension_weknora import WeKnoraProvider + + from deerflow.knowledge import KnowledgeRequestContext, format_result, sources_from_messages + + provider = WeKnoraProvider( + {"spaces": [{"base_url": os.environ["WEKNORA_BASE_URL"], "tenant_id": os.environ["WEKNORA_TENANT_ID"], "api_key_env": "WEKNORA_API_KEY", "user_ids": ["live-probe"], "knowledge_base_ids": [os.environ["WEKNORA_KNOWLEDGE_BASE_ID"]]}]} + ) + context = KnowledgeRequestContext("live-probe", "lead_agent") + catalog = await provider.catalog(context=context, page=1, page_size=100, search="") + assert len(catalog.items) == 1, "Configured base must be visible" + dataset_id = catalog.items[0].id + document_id = os.environ["WEKNORA_DOCUMENT_ID"] + # Traverse actual catalog pagination, not a guessed first-page selection. + page = 1 + while True: + documents = await provider.documents(dataset_id, context=context, page=page, page_size=100, search="") + match = next((item for item in documents.items if item.id == document_id), None) + if match: + assert match.selectable, "Configured document must be searchable" + break + assert documents.items and page * 100 < documents.total and page < 100, "Configured document not found within the bounded live scan" + page += 1 + scope = {"version": 1, "mode": "selected", "dataset_ids": [dataset_id], "document_filters": [{"dataset_id": dataset_id, "document_ids": [document_id]}]} + result = await provider.search_scoped(os.environ["WEKNORA_EVIDENCE_QUERY"], context=KnowledgeRequestContext("live-probe", "lead_agent", scope)) + content, artifact = format_result(result) + persisted = json.loads(json.dumps({"type": "tool", "content": content, "artifact": artifact})) + sources = sources_from_messages([persisted]) + assert sources, "Live query must retrieve attributable document evidence" + for source in sources: + assert source.provider == "weknora" and source.page is None and source.url is None + assert any(hit.source and hit.source.title == source.title and source.excerpt.rstrip("…") in hit.content for hit in result.hits) + + # Exercise the host's actual tool/artifact path with the live provider. + # The AIMessage is deterministic: this checks tool execution, not LLM quality. + from types import SimpleNamespace + + from langchain_core.messages import AIMessage + from langgraph.graph import END, START, MessagesState, StateGraph + from langgraph.prebuilt import ToolNode + + from deerflow.extensions import EXTENSION_SNAPSHOT_CONTEXT_KEY, ExtensionRegistry + from deerflow.knowledge import tools + from deerflow.knowledge.sources import prune_sources + from deerflow.runtime.serialization import serialize_channel_values_for_api + from deerflow.tools.builtins.task_tool import _task_result_command + + registry = ExtensionRegistry() + with registry.attributed_to("deerflow_extension_weknora:install"): + registry.knowledge_provider("weknora", provider) + config = SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(use="deerflow.knowledge.tools:knowledge_search_tool", model_extra={"provider": "weknora"})) + monkeypatch.setattr(tools, "get_app_config", lambda: config) + builder = StateGraph(MessagesState, context_schema=dict) + builder.add_node("tools", ToolNode([tools.knowledge_search_tool])) + builder.add_edge(START, "tools") + builder.add_edge("tools", END) + state = await builder.compile().ainvoke( + {"messages": [AIMessage(content="", tool_calls=[{"id": "live-search", "name": "knowledge_search", "args": {"query": os.environ["WEKNORA_EVIDENCE_QUERY"]}}])]}, + context={"user_id": "live-probe", "is_subagent": True, "agent_id": "researcher", "__knowledge_scope_execution": scope, EXTENSION_SNAPSHOT_CONTEXT_KEY: registry.build()}, + ) + message = state["messages"][-1] + tool_sources = sources_from_messages([message]) + assert tool_sources and all(source.provider == "weknora" for source in tool_sources) + history = json.loads(json.dumps(serialize_channel_values_for_api({"messages": [message]}))) + assert sources_from_messages(history["messages"]) == tool_sources + forwarded = _task_result_command(tool_call_id="live-task", status="completed", result=message.content, knowledge_sources=[source.model_dump(mode="json", exclude_none=True) for source in tool_sources]) + assert sources_from_messages(forwarded.update["messages"]) == tool_sources + _, pruned = prune_sources("removed by budget", message.artifact) + assert pruned["knowledge_sources"] == [] + with pytest.raises(PermissionError): + await provider.search_scoped("test", context=KnowledgeRequestContext("unbound-live-user", "lead_agent", scope)) diff --git a/examples/deerflow-extension-weknora/AGENTS.md b/examples/deerflow-extension-weknora/AGENTS.md new file mode 100644 index 0000000000..509d797ec3 --- /dev/null +++ b/examples/deerflow-extension-weknora/AGENTS.md @@ -0,0 +1,20 @@ +# WeKnora read-only extension + +Keep provider-specific code in this independently packaged extension. Use +`registry.knowledge_provider` and the public `deerflow.knowledge` contracts; +core owns scope admission, formatting, budgets and citation evidence. + +API reference: WeKnora v0.8.3 at `3e22822c1d2397be730c458c7539d0eb8c113a24`. +Preserve per-request operator bindings and upstream authorization. Never turn +missing bindings, invalid selections, empty allowlists, or upstream failures +into an unrestricted search. Use KB-bound `hybrid-search` with optional +`knowledge_ids` and disabled context enrichment. The global search endpoint +rejects document targets for KB-restricted keys and unions base/document target +lists. Never broaden credentials to work around that. Verify result membership. +Reject malformed document totals and detail shapes as provider errors; reserve +permission errors for resource identity, membership and searchability failures. +Dataset IDs derive from endpoint/tenant/native base, not credentials or labels. + +Tests live in `backend/tests/test_weknora_extension*.py`; live tests require +explicit opt-in and an already indexed instance. Do not fabricate a live result. +See README for version references, configuration and the authorization model. diff --git a/examples/deerflow-extension-weknora/README.md b/examples/deerflow-extension-weknora/README.md new file mode 100644 index 0000000000..28ba8b8e75 --- /dev/null +++ b/examples/deerflow-extension-weknora/README.md @@ -0,0 +1,185 @@ +# WeKnora read-only knowledge provider + +An independently installable extension for the phase 3 read-only delivery of +[#6269](https://github.com/bytedance/deer-flow/issues/6269). It uses the existing +DeerFlow scope selector, generic `knowledge_search`, shared citation validation, +source budgets, delegation and history rendering. No extra frontend is needed. + +The API mapping is checked against **WeKnora v0.8.3**, commit +[`3e22822c1d2397be730c458c7539d0eb8c113a24`](https://github.com/Tencent/WeKnora/tree/3e22822c1d2397be730c458c7539d0eb8c113a24). +Other releases require verification. Offline tests use explicitly synthetic responses. A real local deployment +with Ollama `bge-m3` has also passed; see [the validation record](VALIDATION.md) +for the exact environment, checks and limits. + +## Choosing this extension or MCP + +Use [WeKnora's built-in MCP server](https://github.com/Tencent/WeKnora/blob/3e22822c1d2397be730c458c7539d0eb8c113a24/website-docs/03-features/08-mcp.md) +when you want to expose WeKnora tools to DeerFlow or another MCP client. +Choose this read-only extension when you want DeerFlow's native knowledge-base +and document scope selector, shared citations, and retrieval evidence preserved +across delegated tasks and conversation replay. + +## Install and configure + +Requires a host containing phase 2 (#6904), extension-api >=0.2.7, and an indexed +WeKnora v0.8.3 workspace. From the DeerFlow repository root: + +```bash +make extension-install SOURCE=/absolute/path/to/deer-flow/examples/deerflow-extension-weknora +``` + +Set `WEKNORA_API_KEY` in the Gateway process environment (or its Docker +container environment). Do not put a key into the URL or commit credentials. +Edit the installed plugin's existing record in root `config.yaml`, and merge +the knowledge/tool settings with your existing entries: + +```yaml +plugins: + - use: deerflow_extension_weknora:install + enabled: true + required: true + config: + timeout: 30 + match_count: 8 + spaces: + - base_url: https://weknora.example.com + tenant_id: "42" + api_key_env: WEKNORA_API_KEY + user_ids: ["YOUR_DEERFLOW_USER_ID"] + # Optional restriction to exact current executor names, including + # each subagent that should search. Omit to allow all agents for + # the explicitly bound users. + agent_names: ["lead_agent", "researcher"] + # Optional operator restriction; omission permits bases visible to + # this connection. An empty list is rejected, never treated as all. + knowledge_base_ids: ["YOUR_WEKNORA_KB_ID"] + +knowledge_base: + enabled: true + scope_selection_enabled: true + +tool_groups: + - name: knowledge + +tools: + - name: knowledge_search + group: knowledge + use: deerflow.knowledge.tools:knowledge_search_tool + provider: weknora +``` + +Preserve extension-manager fields such as `name` and `package` if present; +replace the existing `knowledge_search` entry instead of adding a second one. +Restart Gateway after installation or configuration changes. Select a knowledge +base (optionally individual documents) using the chat's existing scope selector, +then ask a question matching indexed content. Sources show the retrieved title, +excerpt and base name. This adapter does not invent page numbers or source URLs +from chunk offsets, storage paths or response indexes. + +`user_ids` are exact server-authenticated DeerFlow IDs, not display names or +model-supplied values. Resolve them from your deployment's user administration; +for an unauthenticated/single-user installation use its actual runtime user ID. +There is no default or wildcard binding. To bind multiple workspaces, add a +`spaces` entry per endpoint/tenant pair, each with its own explicit users and +credential environment variable. Configure each endpoint/tenant only once. + +## Authorization and scope + +Each operation resolves the trusted current user and executor against the +operator-owned connection bindings. It sends `X-API-Key` and `X-Tenant-ID` to +WeKnora, whose API-key permissions and workspace/resource authorization remain +final. The binding is a credential-routing decision: every listed user receives +the resource visibility of that credential, further restricted by the optional +base allowlist. It does **not** impersonate a WeKnora human user. Use dedicated +least-privilege keys and only bind users intended to share that visibility. + +Tenant-scoped keys must match the configured tenant. Platform keys always need +an explicit tenant and user binding here; unbound platform discovery and +fallback-to-all behavior are not supported. Missing bindings or keys fail +closed. API-key roles/capabilities must allow base/document reads and knowledge +search; restricted keys that cannot enumerate the configured bases are rejected. + +- `all` searches explicit accessible bases in the user's bindings, never an + omitted target list. At most 100 bases across at most 20 bound spaces are + supported; narrow larger connections with `knowledge_base_ids`. +- `selected` validates every base and document before searching. Unknown, + inaccessible, disabled, or not-yet-searchable documents fail the complete + request. WeKnora `completed` and `finalizing` documents with `enabled` status + are selectable. +- Retrieval uses `POST /api/v1/knowledge-bases/{id}/hybrid-search` with + `query_text` and optional `knowledge_ids`. The path fixes the authorized base; + document IDs narrow it. Context enrichment is disabled to keep matched + evidence within that selection. Mixed selections use a request per base. + This raw-recall API needs only an embedding model, with no reranker or chat + model. The global `/knowledge-search` API rejects document IDs with + KB-restricted API keys, and combining base/document targets there uses union + semantics. The adapter never broadens key permissions to work around that. +- Retrieved documents are checked against the authoritative detail endpoint for + base membership and searchable state; unexpected evidence fails the request. + No unrestricted retry, partial result on an error, or redirect is allowed. +- Dataset IDs are `wk1_` plus SHA-256 of a JSON tuple of normalized endpoint, + tenant and native base ID (68 ASCII characters). Resolution is recomputed + from the currently authorized catalog without an ID/ACL database. Credential + rotation preserves IDs; endpoint or tenant changes invalidate old selections. + Document IDs remain native IDs inside the dataset filter. Native WeKnora IDs + accept only ASCII letters, digits, `_` and `-`, up to 256 characters. +- The same bindings apply to catalog, primary execution and subagents. Agent + restrictions match the **current executor**, not the parent assistant. + Restored scopes are re-authorized against current upstream permissions. + +API calls are asynchronous, use a bounded timeout and 4 MiB response limit, +and suppress upstream bodies in errors. A response exceeding 100 hits fails; +at most 100 hits are admitted per tool call before core applies its text budget. Current connection secrets are redacted +from text evidence. Shared source formatting and output budgets remain in core. + +## Verification + +Offline regression tests from the repository root: + +```bash +uv run --project backend --no-sync python -m pytest \ + backend/tests/test_weknora_extension.py -q +``` + +The tests cover connection isolation, filters without widening, missing/denied +resources, malformed responses, package registration, real ToolNode execution, +subagent identity, shared evidence, serialization/replay and budget pruning. +Run existing generic scope-admission and frontend citation suites as well. + +An opt-in, read-only live check uses an **already indexed document**. Set the key +in the environment before invoking it; do not paste keys into shell history. +From the repository root: + +```bash +DEER_FLOW_RUN_LIVE_TESTS=1 \ +WEKNORA_BASE_URL=https://weknora.example.com \ +WEKNORA_TENANT_ID=42 \ +WEKNORA_KNOWLEDGE_BASE_ID=your-base-id \ +WEKNORA_DOCUMENT_ID=your-indexed-document-id \ +WEKNORA_EVIDENCE_QUERY='a question matching that document' \ +uv run --project backend --no-sync python -m pytest \ + backend/tests/test_weknora_extension_live.py -m live -q +``` + +The test expects `WEKNORA_API_KEY`, performs no upload/write operation, verifies +catalog → selected document → retrieval → actual ToolNode → shared source/replay, +subagent forwarding and evidence pruning, and requires attributable results. +It supplies a deterministic tool call rather than invoking a generative model. Record the deployed WeKnora version, test result and UI +smoke-test result before claiming live delivery. Do not include keys or private +retrieval text in committed validation records. + +## Scope and attribution + +API mapping and filter-validation design were informed by +[superbuzzy's integration branch](https://github.com/superbuzzy/deer-flow/tree/f67609f3ab54c6d243d6f0ee1f82f2253b018b33), +especially `community/weknora/{provider,documents,retrieval}.py`, and checked +against v0.8.3's `internal/handler/knowledgebase.go`, `internal/handler/session/{types,qa}.go`, +`internal/handler/knowledge.go`, `internal/types/search.go` and +`internal/middleware/auth.go`. + +This package registers the current `RetrievalProvider` contract rather than +importing the older branch's provider discovery or native citation format. +Uploads, deletion, processing management, Wiki/Graph, a knowledge management +page and new DeerFlow ACL persistence remain outside the read-only delivery. +Shared-contract durable batch source forwarding retains the existing core +limitation; ordinary subagent forwarding uses the shared evidence path. diff --git a/examples/deerflow-extension-weknora/VALIDATION.md b/examples/deerflow-extension-weknora/VALIDATION.md new file mode 100644 index 0000000000..257a7e9fd4 --- /dev/null +++ b/examples/deerflow-extension-weknora/VALIDATION.md @@ -0,0 +1,96 @@ +# Local integration validation — 2026-10-15 + +A real local WeKnora deployment and local Ollama embeddings were exercised on +Apple Silicon macOS. The corpus is explicitly synthetic Chinese text; no +external model API, real customer corpus, or production credentials were used. + +## Environment + +- WeKnora API/UI: v0.8.3. API runtime reports version `0.8.3`, standard edition, + Go 1.26.0, linux/arm64, build time 2026-10-14 07:53:47 UTC. +- API image: `wechatopenai/weknora-app:v0.8.3`, multi-platform manifest digest + `sha256:b5f96b13c2f69779af8c7291264cafbda64c208398b019e3a6223c6f3ea82031`; + selected arm64 manifest + `sha256:e9b2f77cbd0edad22a693844d27029a1881718687e77804028f6d3306f7340b1`. +- PostgreSQL/vector retrieval: `paradedb/paradedb:v0.25.11-pg17`; + task queue: `redis:7.0-alpine`; container runtime: OrbStack. +- Ollama 0.40.4, native macOS; model `bge-m3`, 1024 embedding dimensions, + model context length 8192, embedding-only capability. +- Model weights: official Ollama `library/bge-m3` model blob, 1,157,671,200 + bytes, verified SHA-256 + `daec91ffb5dd0c27411bd71f29932917c49cf529a641d0168496c3a501e3062c`. + The machine's Fake-IP DNS made Ollama reject the registry redirect. The + official blob was downloaded over HTTPS, digest-verified, and imported with + `ollama create bge-m3`; no redirect guard was disabled. +- All host ports are loopback-bound. Container API calls reach the native + Ollama service through `host.docker.internal`. + +## Corpus and credential setup + +Created a dedicated local workspace and document knowledge base through the +WeKnora API. Its embedding model is local `bge-m3`; vector indexing is enabled. +Summary generation, generated questions, graph and Wiki are disabled, so this +fixture does not require a chat or rerank model. Manual Markdown ingestion +uses WeKnora's own processing and embedding pipeline. + +Two documents reached `parse_status=completed`, `enable_status=enabled`: + +1. Synthetic Cedar travel policy: nightly accommodation cap 680 CNY, submission + within 15 days, unique marker `CEDAR-680-15`. +2. Synthetic Blue Whale training policy: budget 2400 CNY, report within 7 days, + unique marker `BLUEWHALE-2400-7`. + +The integration key has **only `retrieve` capability**, `full_access=false`, +and an upstream allowlist containing the test base. A second empty base exists +in the same workspace solely to test rejection by that API-key allowlist. +Secrets and local resource IDs are kept outside the repository. + +## Observed checks + +| Check | Result | +| --- | --- | +| Native Ollama embeds two Chinese strings | Two 1024-dimensional vectors | +| Container reaches native Ollama | Version endpoint succeeds | +| API health / frontend page / frontend-proxied login | HTTP 200 | +| Catalog → selected document → live retrieval → shared evidence | Passed opt-in pytest | +| Actual DeerFlow ToolNode with executing subagent identity | Live evidence returned | +| ToolMessage serialization/replay and parent source forwarding | Valid sources preserved | +| Source pruning after evidence removal | Empty source list | +| Select Cedar document | Cedar marker present; Blue Whale marker absent | +| Select Blue Whale document | Blue Whale marker present; Cedar marker absent | +| All-accessible scope within configured base | Cedar evidence retrieved | +| Unknown document selection | Rejected without unrestricted retry | +| Unbound DeerFlow user | Rejected | +| Restricted API key reads the second base | HTTP 403 | +| Read-only API key attempts a synthetic draft write | HTTP 403; no document created | + +The live pytest is `backend/tests/test_weknora_extension_live.py`; it supplies a +deterministic tool call and requires an already indexed document. This tests +real retrieval and tool/artifact integration, **not generative answer quality**. +The separate boundary probes used the two synthetic documents described above. +No interactive DeerFlow browser chat or external provider deployment is claimed. + +## Compatibility finding fixed during this run + +The initial adapter used `/api/v1/knowledge-search`. With the correctly scoped +read-only key, WeKnora v0.8.3 returned HTTP 403: +`API key scope does not allow knowledge_ids without a verified knowledge base`. +Passing base IDs alongside document IDs is unsafe because that endpoint unions +those targets; granting broader key permissions would also weaken isolation. + +The adapter now uses +`POST /api/v1/knowledge-bases/{id}/hybrid-search` with `query_text`, optional +`knowledge_ids`, and `skip_context_enrichment=true`. The path enforces the base +binding and document IDs narrow it. This endpoint supplies raw recall without +requiring a reranker. A failing offline regression was added before the change; +the restricted-key live test and the two-document isolation probes pass after it. + +## Regression checks after the compatibility fix + +- Full backend offline suite: **31,897 passed** (four duration-balanced shards; + optional/environment-dependent tests retain their reported skips). +- Strict blocking-I/O suite: **257 passed, 1 skipped**. +- Focused adapter plus its SSL/blocking regression: **40 passed**. +- Opt-in live test against this deployment: **1 passed**, including the real + ToolNode, replay, forwarding and pruning checks above. +- Ruff lint/format, extension wheel packaging and agent-guidance checks passed. diff --git a/examples/deerflow-extension-weknora/deerflow_extension_weknora/__init__.py b/examples/deerflow-extension-weknora/deerflow_extension_weknora/__init__.py new file mode 100644 index 0000000000..e2064166e1 --- /dev/null +++ b/examples/deerflow-extension-weknora/deerflow_extension_weknora/__init__.py @@ -0,0 +1,13 @@ +"""Installable WeKnora read-only provider; all evidence goes through core.""" + +from deerflow_extension_api import extension + +from .provider import WeKnoraProvider + +__all__ = ["WeKnoraProvider", "install"] + + +@extension(api="0.2.7", name="weknora") +def install(registry, config): + if not registry.knowledge_provider("weknora", WeKnoraProvider(dict(config))): + raise RuntimeError("This host does not support knowledge provider registration") diff --git a/examples/deerflow-extension-weknora/deerflow_extension_weknora/config.py b/examples/deerflow-extension-weknora/deerflow_extension_weknora/config.py new file mode 100644 index 0000000000..e6b3a7f2c1 --- /dev/null +++ b/examples/deerflow-extension-weknora/deerflow_extension_weknora/config.py @@ -0,0 +1,62 @@ +"""Operator-owned connection routing; no resource ACL database or client metadata.""" + +from __future__ import annotations + +import re + +from pydantic import AnyHttpUrl, BaseModel, ConfigDict, Field, field_validator, model_validator + + +def native_id(value: object) -> str: + # WeKnora uses UUID-like IDs. Reject path syntax before constructing URLs. + if not isinstance(value, str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,256}", value): + raise ValueError("Invalid WeKnora resource ID") + return value + + +class Space(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True) + + base_url: AnyHttpUrl + tenant_id: str = Field(pattern=r"^[1-9][0-9]{0,19}$") + api_key_env: str = Field(pattern=r"^[A-Za-z_][A-Za-z0-9_]*$") + user_ids: tuple[str, ...] = Field(min_length=1) + agent_names: tuple[str, ...] | None = Field(default=None, min_length=1) + knowledge_base_ids: tuple[str, ...] | None = Field(default=None, min_length=1, max_length=100) + + @field_validator("base_url") + @classmethod + def safe_url(cls, value: AnyHttpUrl) -> AnyHttpUrl: + if value.username is not None or value.password is not None or value.query is not None or value.fragment is not None: + raise ValueError("base_url must not contain credentials, query or fragment") + return value + + @field_validator("user_ids", "agent_names") + @classmethod + def identities(cls, value: tuple[str, ...] | None) -> tuple[str, ...] | None: + if value is not None and any(not item.strip() or item != item.strip() for item in value): + raise ValueError("Identity bindings must be nonblank exact host identities") + return value + + @field_validator("knowledge_base_ids") + @classmethod + def resource_ids(cls, value: tuple[str, ...] | None) -> tuple[str, ...] | None: + if value is not None: + for item in value: + native_id(item) + return value + + +class Settings(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True) + + spaces: tuple[Space, ...] = Field(min_length=1, max_length=20) + timeout: float = Field(default=30, gt=0, le=120, allow_inf_nan=False) + match_count: int = Field(default=8, ge=1, le=100, strict=True) + + @model_validator(mode="after") + def unique_connections(self) -> Settings: + keys = [(str(space.base_url).rstrip("/"), space.tenant_id) for space in self.spaces] + if len(set(keys)) != len(keys): + raise ValueError("Configure each endpoint/tenant once, with its explicit user bindings") + return self diff --git a/examples/deerflow-extension-weknora/deerflow_extension_weknora/provider.py b/examples/deerflow-extension-weknora/deerflow_extension_weknora/provider.py new file mode 100644 index 0000000000..b4abc6cc15 --- /dev/null +++ b/examples/deerflow-extension-weknora/deerflow_extension_weknora/provider.py @@ -0,0 +1,234 @@ +"""WeKnora v0.8.3 read-only adapter using DeerFlow's phase 2 contracts. + +API/validation approach informed by superbuzzy's feat/weknora-integration; +this package uses current contracts, request-local bindings and shared sources. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import math +import os +from dataclasses import dataclass, field +from typing import Any + +import httpx +from deerflow.knowledge import KnowledgeCapabilities, KnowledgeRequestContext, KnowledgeSource, RetrievalCatalogPage, RetrievalHit, RetrievalResult +from deerflow.knowledge_scope import execution_scope + +from .config import Settings, Space, native_id + +_MAX_RESPONSE_BYTES = 4 * 1024 * 1024 + + +@dataclass(frozen=True) +class _Binding: + space: Space + key: str = field(repr=False) + + def text(self, value: object, limit: int) -> str: + return value.replace(self.key, "[REDACTED]").strip()[:limit] if isinstance(value, str) else "" + + +@dataclass(frozen=True) +class _Dataset: + binding: _Binding + native_id: str + name: str + + @property + def id(self) -> str: + # Recomputed from authorized catalogs; no reverse-map database, truncation, + # key material or ambiguous concatenation. Key rotation preserves IDs. + identity = [str(self.binding.space.base_url).rstrip("/"), self.binding.space.tenant_id, self.native_id] + return "wk1_" + hashlib.sha256(json.dumps(identity, ensure_ascii=False).encode()).hexdigest() + + +def _items(data: object) -> list[dict[str, Any]]: + # Go nil slices are encoded as null, including an empty search result. + if data is None: + return [] + if not isinstance(data, list) or any(not isinstance(item, dict) for item in data): + raise RuntimeError("Invalid WeKnora list response") + return data + + +def _searchable(document: dict[str, Any]) -> bool: + return document.get("parse_status") in {"completed", "finalizing"} and document.get("enable_status") == "enabled" + + +def _pagination(page: int, page_size: int) -> None: + if type(page) is not int or page < 1 or type(page_size) is not int or not 1 <= page_size <= 100: + raise ValueError("Invalid catalog pagination") + + +class WeKnoraProvider: + capabilities = KnowledgeCapabilities(operations={"search", "sources", "catalog", "documents", "dataset_filter", "document_filter"}) + + def __init__(self, config: dict[str, Any], *, transport: httpx.AsyncBaseTransport | None = None): + self.settings = Settings.model_validate(config) + self._transport = transport + + def _bindings(self, context: KnowledgeRequestContext) -> list[_Binding]: + bindings = [] + for space in self.settings.spaces: + if context.user_id not in space.user_ids or (space.agent_names is not None and context.agent_name not in space.agent_names): + continue + key = os.getenv(space.api_key_env, "").strip() + if not key or not key.isascii() or any(ord(char) < 33 or ord(char) == 127 for char in key): + raise RuntimeError("WeKnora API key environment variable is missing or invalid") + bindings.append(_Binding(space, key)) + if not bindings: + raise PermissionError("No WeKnora space is bound to this user and executing agent") + return bindings + + async def _request(self, binding: _Binding, method: str, path: str, **kwargs: Any) -> dict[str, Any]: + # Client per request prevents tenant/credential mutation across users and + # event loops. SSL trust loading runs off-loop. Never follow redirects + # carrying privileged credentials. + try: + async with await asyncio.to_thread( + httpx.AsyncClient, + base_url=str(binding.space.base_url).rstrip("/"), + headers={"X-API-Key": binding.key, "X-Tenant-ID": binding.space.tenant_id}, + timeout=self.settings.timeout, + follow_redirects=False, + transport=self._transport, + ) as client: + async with client.stream(method, path, **kwargs) as response: + if response.status_code in {401, 403}: + raise PermissionError("WeKnora denied resource access") + if response.status_code == 404: + raise FileNotFoundError("WeKnora resource is unavailable") + if response.status_code != 200: + raise RuntimeError("WeKnora request failed") + raw = bytearray() + async for part in response.aiter_bytes(): + raw.extend(part) + if len(raw) > _MAX_RESPONSE_BYTES: + raise RuntimeError("WeKnora response exceeds the supported size") + payload = json.loads(raw) + except (httpx.HTTPError, ValueError): + # No raw body, URL, request headers or credential-bearing cause. + raise RuntimeError("WeKnora request failed or returned invalid JSON") from None + if not isinstance(payload, dict) or payload.get("success") is not True or "data" not in payload: + raise RuntimeError("WeKnora returned an unsuccessful response") + return payload + + async def _datasets(self, context: KnowledgeRequestContext) -> dict[str, _Dataset]: + result: dict[str, _Dataset] = {} + for binding in self._bindings(context): + payload = await self._request(binding, "GET", "/api/v1/knowledge-bases") + bases = _items(payload["data"]) + allowed = binding.space.knowledge_base_ids + found = {native_id(base.get("id")) for base in bases} + if allowed is not None and not set(allowed) <= found: + raise PermissionError("A configured WeKnora knowledge base is unavailable") + for base in bases: + base_id = native_id(base.get("id")) + if allowed is not None and base_id not in allowed: + continue + dataset = _Dataset(binding, base_id, binding.text(base.get("name"), 500) or "Unnamed knowledge base") + if dataset.id in result: + raise RuntimeError("Duplicate WeKnora knowledge base") + result[dataset.id] = dataset + if len(result) > 100: + raise RuntimeError("Bind at most 100 knowledge bases using knowledge_base_ids") + return result + + async def catalog(self, *, context: KnowledgeRequestContext, page: int, page_size: int, search: str) -> RetrievalCatalogPage: + _pagination(page, page_size) + datasets = await self._datasets(context) + items = [{"id": ds.id, "name": ds.name, "selectable": True} for ds in datasets.values() if search.strip().casefold() in ds.name.casefold()] + start = (page - 1) * page_size + return RetrievalCatalogPage(items=items[start : start + page_size], total=len(items), page=page, page_size=page_size) + + async def documents(self, dataset_id: str, *, context: KnowledgeRequestContext, page: int, page_size: int, search: str) -> RetrievalCatalogPage: + _pagination(page, page_size) + datasets = await self._datasets(context) + if dataset_id not in datasets: + raise PermissionError("Selected WeKnora knowledge base is unavailable") + ds = datasets[dataset_id] + payload = await self._request(ds.binding, "GET", f"/api/v1/knowledge-bases/{ds.native_id}/knowledge", params={"page": page, "page_size": page_size, "keyword": search}) + items = [] + for doc in _items(payload["data"]): + if doc.get("knowledge_base_id") != ds.native_id: + raise PermissionError("WeKnora returned a document outside the requested base") + items.append({"id": native_id(doc.get("id")), "name": ds.binding.text(doc.get("title") or doc.get("file_name"), 500) or "Unnamed document", "selectable": _searchable(doc)}) + total = payload.get("total") + if type(total) is not int or total < 0: + raise RuntimeError("Invalid WeKnora document total") + return RetrievalCatalogPage(items=items, total=total, page=page, page_size=page_size) + + async def _document(self, ds: _Dataset, document_id: str) -> dict[str, Any]: + payload = await self._request(ds.binding, "GET", f"/api/v1/knowledge/{native_id(document_id)}") + doc = payload["data"] + if not isinstance(doc, dict): + raise RuntimeError("Invalid WeKnora document response") + if doc.get("id") != document_id or doc.get("knowledge_base_id") != ds.native_id or not _searchable(doc): + raise PermissionError("Selected WeKnora document is unavailable or not searchable") + return doc + + async def search(self, query: str) -> RetrievalResult: + raise PermissionError("WeKnora retrieval requires a trusted KnowledgeRequestContext") + + async def search_scoped(self, query: str, *, context: KnowledgeRequestContext) -> RetrievalResult: + scope = execution_scope(context.scope) if context.scope is not None else {"mode": "all"} + if scope["mode"] == "disabled": + raise ValueError("Knowledge search is disabled for this turn") + if not query.strip(): + raise ValueError("Knowledge query must not be empty") + filters = {item["dataset_id"]: item["document_ids"] for item in scope.get("document_filters", [])} + for ids in filters.values(): + for document_id in ids: + native_id(document_id) + datasets = await self._datasets(context) + selected = scope["dataset_ids"] if scope["mode"] == "selected" else list(datasets) + if not set(selected) <= datasets.keys(): + raise PermissionError("Selected WeKnora knowledge base is unavailable") + documents: dict[tuple[str, str], dict[str, Any]] = {} + # Validate every constrained resource before any search; no partial + # success and no unfiltered retry if any selection fails. + for dataset_id, ids in filters.items(): + for document_id in ids: + documents[dataset_id, document_id] = await self._document(datasets[dataset_id], document_id) + hits = [] + for dataset_id in selected: + if len(hits) >= 100: + break + ds = datasets[dataset_id] + body: dict[str, Any] = {"query_text": query, "match_count": self.settings.match_count, "skip_context_enrichment": True} + # The KB-bound endpoint authorizes restricted API keys and intersects + # document IDs with this base. The global knowledge-search endpoint + # rejects knowledge_ids for such keys; adding base targets there + # would also widen retrieval through its union semantics. + if dataset_id in filters: + body["knowledge_ids"] = filters[dataset_id] + payload = await self._request(ds.binding, "POST", f"/api/v1/knowledge-bases/{ds.native_id}/hybrid-search", json=body) + results = _items(payload["data"]) + if len(results) > 100: + raise RuntimeError("WeKnora response exceeds the supported result limit") + for item in results: + if len(hits) >= 100: + break + document_id = native_id(item.get("knowledge_id")) + if dataset_id in filters and document_id not in filters[dataset_id]: + raise PermissionError("WeKnora returned evidence outside the selected documents") + cache_key = (dataset_id, document_id) + if cache_key not in documents: + documents[cache_key] = await self._document(ds, document_id) + # SearchResult has no knowledge_base_id; the authoritative detail + # endpoint above verifies membership even for unfiltered search. + text = ds.binding.text(item.get("content"), 100_000) + if not text: + continue + title = ds.binding.text(item.get("knowledge_title") or item.get("knowledge_filename") or documents[cache_key].get("title") or documents[cache_key].get("file_name"), 500) + source = KnowledgeSource(provider="weknora", title=title, excerpt=text, dataset_name=ds.name) if title else None + # v0.8.3 gives offsets/chunk indexes, not a verified page or public + # document URL. Leave locators absent instead of synthesizing them. + score = item.get("score") + score = score if type(score) in {float, int} and math.isfinite(score) else None + hits.append(RetrievalHit(content=text, source=source, score=score)) + return RetrievalResult(hits=hits) diff --git a/examples/deerflow-extension-weknora/pyproject.toml b/examples/deerflow-extension-weknora/pyproject.toml new file mode 100644 index 0000000000..e929b8fe00 --- /dev/null +++ b/examples/deerflow-extension-weknora/pyproject.toml @@ -0,0 +1,25 @@ +[project] +name = "deerflow-extension-weknora" +version = "0.1.0" +description = "Read-only WeKnora retrieval and scope provider for DeerFlow" +readme = "README.md" +requires-python = ">=3.12" +license = "MIT" +dependencies = ["deerflow-extension-api>=0.2.7,<0.3", "deerflow-harness>=2.1,<3", "httpx>=0.27,<1", "pydantic>=2,<3"] + +[project.entry-points."deerflow.extensions"] +weknora = "deerflow_extension_weknora:install" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["deerflow_extension_weknora"] + +[tool.ruff] +line-length = 240 +target-version = "py312" + +[tool.ruff.lint] +select = ["E", "F", "I", "UP"] diff --git a/frontend/src/core/extensions/catalog.ts b/frontend/src/core/extensions/catalog.ts index 437f57b625..18db530605 100644 --- a/frontend/src/core/extensions/catalog.ts +++ b/frontend/src/core/extensions/catalog.ts @@ -45,6 +45,7 @@ export const extensionCatalog = [ package: "deerflow-extension-knowledge", namespace: null, }, + { key: "weknora", package: "deerflow-extension-weknora", namespace: null }, { key: "example", package: "deerflow-extension-example", namespace: null }, ] as const; diff --git a/frontend/src/core/i18n/locales/en-US.ts b/frontend/src/core/i18n/locales/en-US.ts index 3b67905920..c9b22d4d11 100644 --- a/frontend/src/core/i18n/locales/en-US.ts +++ b/frontend/src/core/i18n/locales/en-US.ts @@ -80,6 +80,11 @@ export const enUS: Translations = { description: "Demonstrate read-only retrieval and structured tool results using fixed sample documents, without an external knowledge service.", }, + weknora: { + title: "WeKnora knowledge retrieval", + description: + "Search WeKnora knowledge bases and selected documents with source citations. Requires administrator installation and access configuration.", + }, knowledge: { title: "Knowledge scope example", description: diff --git a/frontend/src/core/i18n/locales/types.ts b/frontend/src/core/i18n/locales/types.ts index 877edb06c2..0f2d6ca14a 100644 --- a/frontend/src/core/i18n/locales/types.ts +++ b/frontend/src/core/i18n/locales/types.ts @@ -35,6 +35,7 @@ export interface Translations { | "screening" | "retrieval" | "knowledge" + | "weknora" | "example", { title: string; description: string } >; diff --git a/frontend/src/core/i18n/locales/zh-CN.ts b/frontend/src/core/i18n/locales/zh-CN.ts index 46c6364d12..28231d164b 100644 --- a/frontend/src/core/i18n/locales/zh-CN.ts +++ b/frontend/src/core/i18n/locales/zh-CN.ts @@ -76,6 +76,11 @@ export const zhCN: Translations = { description: "使用固定示例文档演示只读检索与结构化工具结果,不连接外部知识库。", }, + weknora: { + title: "WeKnora 知识检索", + description: + "检索 WeKnora 知识库和所选文档,并展示来源引用。需要管理员安装并配置访问权限。", + }, knowledge: { title: "知识范围选择示例", description: "用示例文档演示知识库和文件范围选择、检索及来源引用。", diff --git a/frontend/tests/e2e/bookmark-plugin.spec.ts b/frontend/tests/e2e/bookmark-plugin.spec.ts index 8c03352de2..7983268afe 100644 --- a/frontend/tests/e2e/bookmark-plugin.spec.ts +++ b/frontend/tests/e2e/bookmark-plugin.spec.ts @@ -448,7 +448,34 @@ for (const locale of ["en-US", "zh-CN"]) { exact: true, }), ).toBeVisible(); - await expect(page.getByRole("article")).toHaveCount(9); + await expect(page.getByRole("article")).toHaveCount(10); + const weknoraTitle = zh + ? "WeKnora 知识检索" + : "WeKnora knowledge retrieval"; + const weknora = page.getByRole("article").filter({ hasText: weknoraTitle }); + await expect(weknora).toHaveCount(1); + await expect(weknora).toContainText( + zh + ? "需要管理员安装并配置访问权限。" + : "Requires administrator installation and access configuration.", + ); + await weknora + .getByRole("button", { + name: zh ? `查看 ${weknoraTitle}` : `View ${weknoraTitle}`, + exact: true, + }) + .click(); + await expect( + page.getByRole("link", { + name: zh ? "查看安装说明" : "Installation guide", + }), + ).toHaveAttribute("href", /deerflow-extension-weknora#readme$/); + await page + .getByRole("button", { + name: zh ? "全部扩展" : "All extensions", + exact: true, + }) + .click(); const knowledgeTitle = zh ? "知识范围选择示例" : "Knowledge scope example"; const knowledge = page .getByRole("article") @@ -638,10 +665,10 @@ test("catalog remains discoverable when runtime discovery fails", async ({ await expect( page.getByRole("alert").filter({ hasText: "Extensions unavailable." }), ).toBeVisible(); - await expect(page.getByRole("article")).toHaveCount(9); + await expect(page.getByRole("article")).toHaveCount(10); await expect( page.getByRole("article").filter({ hasText: "Catalog extension" }), - ).toHaveCount(9); + ).toHaveCount(10); await expect( page.getByRole("article").filter({ hasText: "Script workflows" }), ).toHaveCount(1); diff --git a/frontend/tests/unit/core/extensions/catalog.test.ts b/frontend/tests/unit/core/extensions/catalog.test.ts index 7b5dd53470..e5704decf5 100644 --- a/frontend/tests/unit/core/extensions/catalog.test.ts +++ b/frontend/tests/unit/core/extensions/catalog.test.ts @@ -145,3 +145,22 @@ test("workflow metadata is localized, explains Phase 2 limits and merges its reg expect(matches[0]?.loaded).toBe(workflow); } }); + +test("WeKnora discovery is localized without implying runtime activation", () => { + for (const [copy, title] of [ + [enUS.extensions.catalog, "WeKnora knowledge retrieval"], + [zhCN.extensions.catalog, "WeKnora 知识检索"], + ] as const) { + const entry = extensionDirectory([], copy).find( + (item) => item.package === "deerflow-extension-weknora", + ); + expect(entry).toMatchObject({ + id: "catalog/weknora", + title, + guide: + "https://github.com/bytedance/deer-flow/tree/main/examples/deerflow-extension-weknora#readme", + }); + expect(entry?.description).toBeTruthy(); + expect(entry?.loaded).toBeUndefined(); + } +});