Add AI-powered documentation update command #3063

mjwolf · 2025-11-10T20:27:55Z

AI-Powered Documentation Updates with LLM Integration

Overview

This PR introduces a new elastic-package update documentation command that leverages Large Language Model (LLM) providers to automatically generate and update package documentation. This feature significantly streamlines the documentation maintenance workflow by analyzing package structure and automatically creating comprehensive, template-compliant documentation.

What's New

Core Command: `elastic-package update documentation`

A new subcommand under elastic-package update that provides AI-powered documentation generation with two primary operation modes:

Rewrite Mode (default)

Complete documentation regeneration from scratch
Analyzes entire package structure including data streams, fields, pipelines, and configuration
Generates comprehensive documentation following Elastic package templates

Modify Mode

Targeted changes to existing documentation
Preserves human-edited sections marked with  comments
Allows specific instructions like "add installation steps" or "update compatibility matrix"

Key Features

🤖 Multi-Provider Support: Works with Google Gemini API and local LLM servers (OpenAI-compatible endpoints)
📝 Multi-File Support: Update any markdown file in _dev/build/docs/ (README.md, CHANGELOG.md, etc.)
🔄 Interactive & Non-Interactive Workflows:
- Interactive mode with browser-based markdown preview for review before acceptance
- Non-interactive mode (--non-interactive) for CI/CD integration
🛡️ Graceful Degradation: Shows manual update instructions when no LLM provider is configured
💾 Safe Updates: Backs up original documentation and allows rollback on cancellation

Command Usage

# Interactive mode (mode and all other options presented in dialogs)
elastic-package update documentation

# Targeted modification with specific instructions
elastic-package update documentation --modify-prompt "Add troubleshooting section" --non-interactive

# Update specific file
elastic-package update documentation --doc-file CHANGELOG.md

# Non-interactive mode (auto-accept first result)
elastic-package update documentation --non-interactive

# Combine flags
elastic-package update documentation --doc-file README.md --modify-prompt "Update compatibility info" --non-interactive

Configuration

LLM providers can be configured via:

Environment Variables:

# Gemini API
export ELASTIC_PACKAGE_LLM_PROVIDER=gemini
export ELASTIC_PACKAGE_GEMINI_API_KEY=your-api-key

# Local LLM server
export ELASTIC_PACKAGE_LLM_PROVIDER=local
export ELASTIC_PACKAGE_LOCAL_LLM_URL=http://localhost:8000
export ELASTIC_PACKAGE_LOCAL_LLM_MODEL=llama3

Profile Configuration:

# ~/.elastic-package/stack/config.yml or profile-specific config
llm:
  provider: gemini
  gemini:
    api_key: your-api-key
  # or
  local:
    url: http://localhost:8000
    model: llama3

Implementation Details

New Package: `internal/llmagent/`

A comprehensive LLM agent framework with modular architecture:

`docagent/` - Documentation Agent

Orchestrates documentation generation workflow
Manages interactive prompts and user feedback loops
Handles file backup/restore and preserved sections
Implements response analysis and error recovery

`framework/` - Agent Framework Core

Generic agent implementation with conversation management
Tool execution and result handling
Provider-agnostic design for extensibility

`providers/` - LLM Provider Implementations

Gemini Provider: Google Generative AI API integration with streaming support
Local Provider: OpenAI-compatible local LLM server support
Unified provider interface for easy extensibility

`tools/` - Package Analysis Tools

File system navigation and content reading
Package manifest inspection
Data stream analysis (fields, sample events, pipelines)
Ingest pipeline examination
Kibana asset inspection

`mcptools/` - Model Context Protocol

Integration with MCP-compatible tools
Enables advanced tool interactions for LLM agents

`ui/` - User Interface Components

Browser-based markdown preview with syntax highlighting
TUI components for interactive prompts
Fallback to terminal display when browser unavailable

Introduce a new `elastic-package update documentation` command that leverages LLM providers to automatically generate and update package documentation. Key features: - Two operation modes: * Rewrite mode: Full documentation regeneration from package structure * Modify mode: Targeted changes to existing documentation - Multi-file support: Update any markdown file in _dev/build/docs/ - Interactive and non-interactive workflows - Provider flexibility: Support for Gemini API and local LLM servers Implementation details: - Add new LLM framework in internal/llmagent/ with: * Provider abstraction for different LLM backends * Documentation agent with package analysis tools * MCP (Model Context Protocol) tools integration * Interactive UI with browser-based markdown preview - Configuration via environment variables or profile config - Graceful degradation: Show manual update instructions when no provider configured Command flags: - --non-interactive: Skip prompts and auto-accept first result - --modify-prompt: Specify targeted modification instructions - --doc-file: Select specific markdown file to update This enables maintainers to quickly generate comprehensive, template-compliant documentation by analyzing package structure, data streams, and configuration. Co-Authored-By: Jonathan Molinatto <[email protected]> Co-Authored-By: Claude <[email protected]> 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Add comprehensive unit tests for: - docagent: DocumentationAgent, ResponseAnalyzer, file operations, and interactive components - framework: Agent framework core functionality - mcptools: MCP (Model Context Protocol) tools - providers: Gemini, local, and base provider implementations - tools: Package tools functionality 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

…n-llm

teresaromero

This looks fantastic! It will definitely save a lot of time! I have some feedback and suggestions regarding the flags and their processing. If I understood correctly, the current default when running update documentation is update documentation —non-interactive=false, which means the user must select the mode and files during the process. I believe it would enhance the user experience if the command ran in non-interactive mode by default and allowed optional selection of —interactive (similar to git rebase —interactive).

When a rewrite is done, the diff will be shown as is part of the repository files, so the use can discard them by not commit, isn’t it? This also makes me think about the “review” on the interactive mode, is it worth? Or the review is of the prompt?

Here is the proposal for flags and behaviors:

update documentation -> complete rewrite of the md docs located at the package root
update documentation —interactive -> display prompts for selecting the file and mode (rewrite or modify)
update documentation —doc-file -> complete rewrite of the specified file
update documentation —doc-file —modify-prompt -> update the specified doc using the provided prompt

Using modify-prompt makes the command non-interactive.

Additionally, I am unsure how the modify-prompt functions. There are prompts provided for the full rewrite; what is the content of the flag?

teresaromero · 2025-11-11T08:44:51Z

cmd/update_documentation.go

+		// If no modify-prompt flag was provided, ask user to choose mode
+		if modifyPrompt == "" {
+			modePrompt := tui.NewSelect("Do you want to rewrite or modify the documentation?", []string{
+				"Rewrite (full regeneration)",


Can we have as constants the options?

const ( MODE_PROMPT_REWRITE="Rewrite (full regeneration)" MODE_PROMPT_MODIFY="Modify (targeted changes)" )

I've moved these to const

teresaromero · 2025-11-11T08:48:53Z

cmd/update_documentation.go

+		}
+	} else {
+		cmd.Println("Running in non-interactive mode - proceeding automatically.")
+		useModifyMode = modifyPrompt != ""


modifyPrompt should this be one of the given options at the interative path? I think the flag string should be validated to fit the options or if empty check if non-interactive is checked to go full rewrite?

The modify-prompt flag does work with interactive mode, and changes the behaviour a little bit. When it's used, update documentation will skip the prompts asking for the mode and the initial prompt, and instead go straight to applying the given prompt in update mode. After the document is generated, the interactive loop is used to prompt the user if they want to make further changes.

So using interactive mode with modifyPrompt does work, and I think it makes sense to keep. Although maybe this isn't a workflow many people would be interested in doing.

teresaromero · 2025-11-11T08:50:21Z

internal/llmagent/docagent/docagent.go

+	} else if !nonInteractive {
+		// Prompt user for modification instructions
+		var err error
+		instructions, err = tui.AskTextArea("What changes would you like to make to the documentation?")


is the prompt a free text to inject to the LLM after? what are the security risks here? any text can be injected? do we need to validate the text the user is entering?

Yes, this is sent directly to the LLM (inserted into the user-provided instructions section of this prompt: internal/llmagent/docagent/_static/revision_prompt.txt).

Since the user will need to provide their own LLM provider API key, I think they should be responsible for what they enter and how this is used. Users could also send text to the LLM by modifying files in the integration package, or using their API key to send text in a completely different way, so I don't think we need any specific protections here, but I may be wrong.

…n-llm

Add documentation on the use of a service_info.md file in integration packages, and how it can be used to control the generated documentation of packages.

…package into update-documentation-llm

…n-llm

Update the prompts and examples provided to the LLM, in order to better follow the expected format, voice and tone in generated documentation.

elasticmachine · 2025-11-14T23:25:39Z

💚 Build Succeeded

Buildkite Build
Commit: dcaedea

History

💚 Build #6615 succeeded c5e10a0
💔 Build #6587 failed 3649c1f
💚 Build #6586 succeeded a9195b3
💚 Build #6557 succeeded 33a0ab6
💔 Build #6556 failed 691cc8b
💔 Build #6555 failed f33c45b

mjwolf and others added 2 commits November 7, 2025 14:10

mjwolf requested a review from a team as a code owner November 10, 2025 20:27

mjwolf added 2 commits November 10, 2025 12:32

fix: typo in docs input template

2040d34

Merge remote-tracking branch 'upstream/main' into update-documentatio…

691cc8b

…n-llm

mjwolf force-pushed the update-documentation-llm branch from f33c45b to 691cc8b Compare November 10, 2025 22:50

Make unit tests OS agnostic

33a0ab6

teresaromero reviewed Nov 11, 2025

View reviewed changes

mjwolf added 7 commits November 12, 2025 12:46

Merge remote-tracking branch 'upstream/main' into update-documentatio…

1b9f6cb

…n-llm

Add information about service_info file

a9195b3

Add information about service_info file

78bf9ea

Add documentation on the use of a service_info.md file in integration packages, and how it can be used to control the generated documentation of packages.

Move mode prompts to const

104d89f

Merge branch 'update-documentation-llm' of github.com:mjwolf/elastic-…

3649c1f

…package into update-documentation-llm

Merge remote-tracking branch 'upstream/main' into update-documentatio…

c5e10a0

…n-llm

Update LLM prompts and examples

dcaedea

Update the prompts and examples provided to the LLM, in order to better follow the expected format, voice and tone in generated documentation.

Add instruction to include all sections

7f5ff3e

mjwolf requested review from jsoriano and teresaromero November 17, 2025 22:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add AI-powered documentation update command #3063

Add AI-powered documentation update command #3063

Uh oh!

mjwolf commented Nov 10, 2025 •

edited

Loading

Uh oh!

teresaromero left a comment

Uh oh!

teresaromero Nov 11, 2025

Uh oh!

mjwolf Nov 13, 2025

Uh oh!

teresaromero Nov 11, 2025

Uh oh!

mjwolf Nov 13, 2025

Uh oh!

teresaromero Nov 11, 2025

Uh oh!

mjwolf Nov 13, 2025

Uh oh!

elasticmachine commented Nov 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add AI-powered documentation update command #3063

Are you sure you want to change the base?

Add AI-powered documentation update command #3063

Uh oh!

Conversation

mjwolf commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI-Powered Documentation Updates with LLM Integration

Overview

What's New

Core Command: elastic-package update documentation

Rewrite Mode (default)

Modify Mode

Key Features

Command Usage

Configuration

Implementation Details

New Package: internal/llmagent/

docagent/ - Documentation Agent

framework/ - Agent Framework Core

providers/ - LLM Provider Implementations

tools/ - Package Analysis Tools

mcptools/ - Model Context Protocol

ui/ - User Interface Components

Uh oh!

teresaromero left a comment

Choose a reason for hiding this comment

Uh oh!

teresaromero Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

mjwolf Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

teresaromero Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

mjwolf Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

teresaromero Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

mjwolf Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Nov 14, 2025

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mjwolf commented Nov 10, 2025 •

edited

Loading

Core Command: `elastic-package update documentation`

New Package: `internal/llmagent/`

`docagent/` - Documentation Agent

`framework/` - Agent Framework Core

`providers/` - LLM Provider Implementations

`tools/` - Package Analysis Tools

`mcptools/` - Model Context Protocol

`ui/` - User Interface Components