feat: bootstrap RAG knowledge base and add think-stage intent eval suite#1496
Open
YuZhangLarry wants to merge 1 commit into
Open
feat: bootstrap RAG knowledge base and add think-stage intent eval suite#1496YuZhangLarry wants to merge 1 commit into
YuZhangLarry wants to merge 1 commit into
Conversation
Task A — seed data: bundle 5 curated Dubbo/Dubbo-Admin knowledge docs (component/rag/seeds/*.md), embed them via //go:embed, and index on startup (best-effort, batched ≤10 to respect the dashscope embedding per-request cap). Adds seed.enabled config + schema; fixes cmd/index.go to target the default index the runtime retriever reads from. Task B — call timing: rewrite agentThink prompt to answer directly by default and only suggest query_knowledge_base for version-sensitive / precise-fact lookups. Fix think→act type assertion (ParseThinkOutput returns a pointer) so the think stage's no-tool decision is honored — without this the prompt change had no effect. Lower max_iterations 10→3 and timeout 60→30 for faster fallback. Tests: multi-turn e2e validates the seed retrieval path end-to-end; new isolated single-turn think test (ThinkOnce) asserts documentation questions classify as DOCUMENTATION_QUERY free of multi-turn bias. Also ignore /.env.
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR bootstraps a bundled RAG knowledge base (seed docs embedded and optionally auto-indexed at startup), tunes the ReAct think-stage tool-use policy toward answering directly when possible, and adds E2E tests to validate intent classification and multi-turn chat behavior.
Changes:
- Add embedded seed docs and best-effort startup seeding for the local vector store, plus schema/config support for
rag.seed.enabled. - Update think-stage prompt policy and agent runtime to support isolated think evaluation (
ThinkOnce) and improve fallback/timeout handling. - Add E2E test suites for isolated intent classification distribution and full multi-turn chat (including RAG-ish questions).
Reviewed changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| ai/test/e2e/think_classification_test.go | New isolated think-stage intent classification E2E test. |
| ai/test/e2e/multi_turn_conversation_test.go | New multi-turn conversation SSE E2E test with keyword validation and export. |
| ai/test/e2e/e2e_test.go | Adjust server readiness wait and bind to 127.0.0.1; factory registration ordering. |
| ai/schema/json/rag.schema.json | Add seed configuration schema with default enabled. |
| ai/prompts/agentThink.txt | Update tool-use decision policy and examples to prefer suggested_tools: []. |
| ai/component/tools/tools.yaml | Disable MCP tools by default in tools component config. |
| ai/component/tools/engine/mock_tools.go | Remove mock query_knowledge_base tool registration. |
| ai/component/tools/engine/memory_tools.go | Add real query_knowledge_base tool backed by RAG; improve tool descriptions and RAG defaults. |
| ai/component/server/engine/handlers.go | Add feedback logging, channel draining on close, and change final streaming behavior. |
| ai/component/server/component.go | Make ServerComponent.Start idempotent and add extra startup logging. |
| ai/component/rag/seeds/dubbo-traffic-governance.md | New seed doc: Dubbo traffic governance. |
| ai/component/rag/seeds/dubbo-registry.md | New seed doc: Dubbo registry/service discovery. |
| ai/component/rag/seeds/dubbo-overview.md | New seed doc: Dubbo overview. |
| ai/component/rag/seeds/dubbo-configuration.md | New seed doc: Dubbo configuration reference. |
| ai/component/rag/seeds/dubbo-admin.md | New seed doc: Dubbo Admin overview/config. |
| ai/component/rag/seed.go | New embedded seed loader + batched indexing entrypoint. |
| ai/component/rag/rag.yaml | Switch embed model, and enable seed config by default. |
| ai/component/rag/config.go | Add SeedSpec to RAG config. |
| ai/component/rag/component.go | Run best-effort seeding once during Start; add GetRAG and seeding guards. |
| ai/component/agent/react/react.go | Add ThinkOnce, fallback parsing/marshalling, observe timeout fallback, and pointer-type handling updates. |
| ai/component/agent/fallback/handler.go | New fallback handler for schema parse/marshal and loop-control scenarios. |
| ai/component/agent/agent.yaml | Reduce max iterations and observe timeout defaults for faster fallback. |
| ai/component/agent/agent.go | Improve handling of pointer vs value schema.Observation in loop termination. |
| ai/cmd/index.go | Add -index flag and default indexing target to align CLI with runtime retriever target. |
| .gitignore | Ignore repo-root .env. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+95
to
+100
| rt.GetLogger().Info("Handler received feedback", | ||
| "session_id", sessionID, | ||
| "text", feedback.Text(), | ||
| "done", feedback.IsDone(), | ||
| "final", feedback.IsFinal(), | ||
| "final_nil", feedback.Final() == nil) |
Comment on lines
+167
to
188
| // If the output is an Observation with a final answer, stream it as text delta first | ||
| switch v := output.(type) { | ||
| case *schema.Observation: | ||
| // Send Summary as text if present | ||
| if v.Summary != "" { | ||
| if err := sseHandler.HandleText(v.Summary+"\n", 0); err != nil { | ||
| rt.GetLogger().Error("Failed to stream summary in MessageDelta", "error", err) | ||
| } | ||
| } | ||
| // Send FinalAnswer as text if present | ||
| if v.FinalAnswer != "" { | ||
| if err := sseHandler.HandleText(v.FinalAnswer+"\n", 0); err != nil { | ||
| rt.GetLogger().Error("Failed to stream final answer in MessageDelta", "error", err) | ||
| } | ||
| } | ||
| default: | ||
| rt.GetLogger().Info("MessageDelta: unexpected output type", "type", fmt.Sprintf("%T", output)) | ||
| } | ||
|
|
||
| if err := sseHandler.MessageDeltaWithUsage(stopReason, output); err != nil { | ||
| sseHandler.HandleError("finish_stream_error", fmt.Sprintf("failed to finish stream: %v", err)) | ||
| } |
Comment on lines
+109
to
+113
| // Check if server is already running (idempotent Start) | ||
| if s.srv != nil { | ||
| s.rt.GetLogger().Info("Server already running", "addr", fmt.Sprintf("%s:%d", s.host, s.port)) | ||
| return nil | ||
| } |
Comment on lines
+45
to
+48
| // SeedSpec controls auto-seeding the local vector store with bundled knowledge | ||
| // documents at startup, so retrieval is never empty on a fresh deployment. | ||
| type SeedSpec struct { | ||
| Enabled bool `yaml:"enabled"` |
Comment on lines
+525
to
+549
| switch v := input.(type) { | ||
| case schema.ThinkOutput: | ||
| // If we have think output, use the thought to generate a response | ||
| if v.Thought != "" { | ||
| fallback.FinalAnswer = generateResponseFromThought(v.Thought, v.Intent) | ||
| } | ||
| case schema.Observation: | ||
| // If we have previous observation, reuse it if it has final answer | ||
| if v.FinalAnswer != "" { | ||
| fallback.FinalAnswer = v.FinalAnswer | ||
| fallback.Evidence = v.Evidence | ||
| } else { | ||
| // Continue from previous observation | ||
| fallback.Heartbeat = true | ||
| fallback.Focus = "Continue processing with available information" | ||
| } | ||
| case schema.ToolOutputs: | ||
| // If we have tool outputs, generate response from them | ||
| if len(v.Outputs) > 0 { | ||
| fallback.FinalAnswer = generateResponseFromToolOutputs(v.Outputs) | ||
| } | ||
| default: | ||
| // Default fallback for any other input type | ||
| fallback.FinalAnswer = "I apologize, but I need more time to process your request. Based on the available context, I cannot provide a complete answer at this moment." | ||
| } |
Comment on lines
+468
to
+471
| func exportResultsToFile(t *testing.T, results *TestResults) { | ||
| _, file, _, _ := runtime.Caller(0) | ||
| aiDir := filepath.Dir(filepath.Dir(filepath.Dir(file))) | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ai/component/rag/): bundled seed docs (seeds/*.md— Dubbooverview, configuration, registry, traffic governance, Dubbo Admin) are
embedded via
//go:embedand indexed at startup, so a fresh deployment cananswer documentation-grade questions out of the box. Indexing is best-effort
and idempotent (dedup by content hash) and batched to the embedding backend's
per-request input limit; a missing embedder key degrades gracefully instead of
failing startup, and reranking falls back to raw vector results when no rerank
key is configured.
ai/component/agent/,ai/prompts/agentThink.txt):the think stage now prefers answering directly (
suggested_tools: []) and nolonger confuses objective documentation lookups (config keys, defaults,
required addresses) with memory search mid-conversation. Adds a
ThinkOnceentry point so the stage can be evaluated in isolation.
ai/test/e2e/):TestThinkClassificationIsolatedruns each documentation question in a freshsession (no multi-turn bias) and repeats it to surface nondeterminism as a
distribution, asserting a stable
DOCUMENTATION_QUERYclassification routed toquery_knowledge_base;TestMultiTurnConversationexercises the fullthink→act→observe loop end to end.