Section-based documentation generation#3241
Draft
mjwolf wants to merge 74 commits intoelastic:feat/update-documentationfrom
Draft
Section-based documentation generation#3241mjwolf wants to merge 74 commits intoelastic:feat/update-documentationfrom
mjwolf wants to merge 74 commits intoelastic:feat/update-documentationfrom
Conversation
Now cost is working and an entire execution is contained in a session. Project name, root span and showing all traces in the span are not working.
* Use the multi-agentic workflow by default, and take feedback from validators to generator. * Update tracing attributes and hierarchy * Move tracing config into elastic-package config * Add a tool which can fetch telemetry from Phoenix, and save it as single json file: `scripts/fetch_phoenix_traces.py`
- Implement parallel processing for section generation with sectionResult type - Reduce tool calls by embedding more information into initial prompts - Refactor workflow for better caching and input handling - Update critic and validator agents to accept content directly from user messages - Enhance output format specifications for validation and criticism results 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add --debug-critic-only, --debug-validator-only, --debug-generator-only CLI flags - Add parallel/sequential section generation config option (ELASTIC_PACKAGE_LLM_PARALLEL_SECTIONS) - Refactor section generation into reusable generateSingleSection method - Add section context (title, level) and build info to tracing spans - Add RunCriticOnContent/RunValidatorOnContent for individual agent execution - Remove unused fetch_phoenix_traces.py script 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…ing relevant versioning from original readme
changes focus on improving documentation reliability through three key areas: Troubleshooting completeness - The generator and validators now enforce input-specific troubleshooting tables (Symptom | Cause | Solution format) for each input type the integration uses (httpjson, tcp, udp, filestream, etc.). This was previously missing, resulting in thin troubleshooting sections. Severity upgraded from Minor → Major. Validation section consistency - Added standardized verification steps using data_stream.dataset:<integration>* pattern to check ALL datastreams, instead of hardcoding specific ones like citrix_adc.log. This makes validation guidance work across any integration. Structure guardrails - Added explicit generator instructions preventing common LLM mistakes: multiple H1 headings, heading level skips (H1→H4), and uncontextualized placeholder values (example.com without "replace with your value"). These changes address the top failure patterns observed in the semantic analysis: inconsistent troubleshooting depth, unreliable validation steps, and structural issues that triggered false validation failures.
Doc agents
It looks the section ranking and picking the best one works nicely I also update the validators to only allow actionable suggestions I fixed a bunch of validator false positives And I did some deduping of document links Also fixed phoenix tracing for iteration spans and grouping off all traces
- removed all knowledge of the test harness implementation that was used earlier to test llm generation - fixed phoenix tracing logging and fetching results into the output folder - fix some minor doc prompting and validation
Merge branch 'doc-agent-qcorp-sections' bringing in: - Per-section validation loops with best-iteration tracking - Convergence detection for bonus iterations - Full document generation workflow (GenerateFullDocumentWithWorkflow) - Generation result types (GenerationConfig, GenerationResult, SectionGenerationResult) - Staged validators moved to specialists/validators package - SectionContext moved from specialists to validators package Preserved from doc-agents: - Debug flags (--debug-critic-only, --debug-validator-only, --debug-generator-only) - generateSingleSection and generateSectionsSequential helper functions Documentation updated to reflect current implementation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Based on traces.json analysis, added explicit rules for: - Numbered list items with bold (1. **Step** is WRONG) - Bold pseudo-headers (**Network requirements** should be #### heading) - Bold input types (**Audit logs (file)** should be plain text) - Bold method names (**Option 1: File method** should be ##### heading) Updated: - stylerules/style_rules.go: Added more explicit WRONG patterns - agent_instructions.md: Added examples of new WRONG patterns - section_instructions.go: Added explicit formatting rules for: - How do I deploy this integration (validation steps, network requirements) - Troubleshooting (issue names in lists) - Performance and scaling (concept terms, input types) - What do I need (subsection formatting) - style_validator.go: Added static checks for: - Numbered list items with bold - Bold pseudo-headers (bold on own line) No conflicts found between style_rules.go and agent_instructions.md. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add EnsureDataStreamTemplates postprocessor to programmatically insert
missing {{event}} and {{fields}} templates in Reference section
- Fix regex patterns in completeness_validator to handle flexible whitespace
- Remove 'Onboard and configure' section from template and validators
- Simplify evaluation to validate final document only
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* Remove unused staged_workflow code * Add check that anchors and docs-content links are not used * Fix template sections
For integration which have multiple readme files, add a dialog which will allow the user to select which file to update. Also add a `--doc-file` flag which allows the user to set the file in non-interactive mode.
- Add #### Vendor resources subsection to 'Set up steps in {Product}'
in documentation template and examples
- Update service_info mapping to include 'Vendor set up resources'
section for deployment documentation
- Update section instructions, style rules, and workflow to guide
generator to include vendor resource links in vendor setup section
- Add critic check for misleading 'choose your destination' phrase
in Kibana setup (should be 'choose the setup instructions that
match your configuration' since destination is set in vendor config)
- Update fortinet example with vendor resources and clarifying text
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
Change {Product} to [Product Name] or descriptive text in instructions
to prevent ADK from interpreting it as a session state variable.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
Code review changes across multiple packages: executor: - Remove unused New() function and lastEventContent variable - Inline appName and defaultUserID constants parsing: - Refactor EnsureDocumentTitle to eliminate duplicated logic - Add normalizeTitle helper for case-insensitive comparisons prompts: - Make prompt file constants unexported - Replace fmt.Printf with logger.Warnf for consistency specialists: - Delete unused StagedRegistry, GetStagedValidators, GetValidatorByStage - Remove unused context-based state functions - Export RequiredSections and RecommendedSections for reuse - Add GetString method to StateStore workflow: - Extract common agent running logic into runAgentWithADK helper - Delete unused GeneratorOnly config function - Replace fmt.Println with logger.Debugf - Add truncateLen constant for magic number tools: - Reuse parsing.ParseSections instead of custom parsing logic (~130 lines removed) - Export GetServiceInfoMappingForSection - Remove non-existent observability category tracing: - Replace fmt.Printf with logger.Debugf - Extract common workflow span logic into startWorkflowSpanInternal - Delete unused min function (Go 1.21+ built-in) - Convert unused receiver methods to package functions - Delete unused test span functions (~60 lines) docagent: - Remove unused sectionNum variable - Remove unused contentLower parameter from computeQualityScore - Consolidate section constants using validators.RequiredSections - Replace fmt.Printf with logger calls in file_ops and batch - Extract magic numbers as named constants in metrics - Remove orphaned comment 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
# Conflicts: # README.md
The README template contains example template calls that reference non-existent data streams, causing build failures during tests. These tests validate package creation and manifest structure, not README rendering. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Collaborator
💔 Build Failed
Failed CI StepsHistory
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.