📚 Documentation Noob Test Report - November 27, 2024

[github-actions[bot]]

📚 Documentation Noob Test Report

Test Date: November 27, 2024
Tester Role: Complete beginner with no prior GitHub Agentic Workflows experience
Pages Analyzed: 15+ documentation pages including Quick Start, CLI reference, examples, and guides

🎯 Overall Impression

As a complete beginner, the documentation has strong foundational content but contains several critical gaps that would block new users from successfully getting started. The conceptual explanations are generally good, but practical implementation details are often missing or unclear.

---

🔴 Critical Issues (Block Getting Started)

1. Missing Authentication Prerequisites

Location: /docs/src/content/docs/setup/quick-start.md

Issue: The Quick Start guide jumps straight into installing the CLI without explaining that users need:

• A GitHub Personal Access Token (PAT) with specific permissions
• The distinction between GITHUB_TOKEN (doesn't work) and custom tokens
• How to create and configure these tokens

Impact: Users will fail at the first workflow execution with authentication errors and won't understand why.

Evidence:

## Install CLI
gh extension install githubnext/gh-aw

No mention of authentication setup before or after this step.

Recommendation: Add a "Prerequisites" section before installation:

## Prerequisites

Before installing, ensure you have:

1. **GitHub CLI installed**: [Install gh](https://cli.github.com/)
2. **GitHub Personal Access Token** with permissions:
   - `repo` (Full control of private repositories)
   - `workflow` (Update GitHub Action workflows)
   
   [Create a PAT →](https://github.com/settings/tokens/new)

3. **Configure your token**:
   ```bash
   gh auth login
   # Or set environment variable:
   export GITHUB_TOKEN="ghp_your_token_here"

⚠️ Important: The default GITHUB_TOKEN in GitHub Actions has limited permissions. For workflows that create issues, PRs, or modify content, you'll need to create a repository secret with a PAT.

---

### 2. **No "Hello World" Minimal Example**

**Location:** `/docs/src/content/docs/setup/quick-start.md`

**Issue:** The Quick Start shows a complex issue labeling workflow as the first example. For a beginner, this is overwhelming because it introduces multiple concepts simultaneously:
- Frontmatter syntax
- Trigger configuration
- Tools configuration
- Natural language instructions
- Template expressions

**Impact:** Beginners don't have a mental model of the simplest possible workflow, making it hard to understand which parts are essential vs. optional.

**Recommendation:** Start with a truly minimal "Hello World" example:

```markdown
## Your First Workflow

Let's create the simplest possible workflow - one that just comments "Hello!" when you type `/hello` in an issue:

1. Create `.github/workflows/hello.md`:

\`\`\`markdown
---
on:
  command:
    name: hello
permissions:
  contents: read
---

# Hello World

Post a friendly comment saying "Hello from GitHub Agentic Workflows!"
\`\`\`

2. Compile it:
\`\`\`bash
gh aw compile hello
\`\`\`

3. Commit and push both files:
\`\`\`bash
git add .github/workflows/hello.md .github/workflows/hello.lock.yml
git commit -m "Add hello workflow"
git push
\`\`\`

4. Test it by commenting `/hello` on any issue!

**What you just learned:**
- Workflows are written in Markdown with YAML frontmatter
- `on: command:` responds to slash commands
- The compiled `.lock.yml` file is what GitHub Actions runs
- Natural language instructions tell the AI what to do

---

3. Compilation Step Not Emphasized Enough

Location: Multiple pages, especially /docs/src/content/docs/setup/quick-start.md

Issue: The critical importance of running gh aw compile after EVERY change is mentioned but not emphasized enough. Many beginners will:

1. Edit the .md file
2. Commit and push
3. Wonder why their changes don't take effect

Evidence from Quick Start:

Compile the workflow to generate the GitHub Actions YAML:

This reads like an optional step, not a REQUIRED step that must happen after every edit.

Recommendation: Add prominent warnings:

## ⚠️ Critical: Always Compile After Changes

**Every time you modify a workflow `.md` file, you MUST compile it:**

\`\`\`bash
gh aw compile
\`\`\`

**Why:** GitHub Actions runs the `.lock.yml` file, not your `.md` file. If you don't compile, your changes won't take effect.

**Quick mental model:** Think of compilation like building code. You wouldn't edit source code and expect it to run without compiling first.

**Common mistake:** Forgetting to commit the `.lock.yml` file after compilation. Both files must be committed:
\`\`\`bash
git add .github/workflows/my-workflow.md
git add .github/workflows/my-workflow.lock.yml
git commit -m "Update workflow"
\`\`\`

---

4. Safe Outputs Explanation Insufficient

Location: /docs/src/content/docs/reference/safe-outputs.md

Issue: The safe-outputs page explains WHAT they are but doesn't clearly explain:

• WHY beginners should use them (security benefits)
• WHEN to use safe-outputs vs. direct permissions
• HOW they work behind the scenes (simplified)
• Common mistakes (missing from main workflow prompt)

Impact: Beginners will either:

• Grant excessive permissions because they don't understand safe-outputs
• Try to use safe-outputs but fail because they don't mention them in the prompt

Recommendation: Add a "Safe Outputs 101" section:

## Safe Outputs 101

**The Problem:** Giving AI workflows write permissions to your repository is risky. What if the AI makes a mistake?

**The Solution:** Safe outputs let the AI *describe* what it wants to create (an issue, comment, PR), then a separate, controlled job actually creates it.

**Mental Model:**
- ❌ **Direct permissions:** AI has a key to your house
- ✅ **Safe outputs:** AI writes a shopping list, you buy the groceries

### When to Use Safe Outputs

Use safe outputs when your workflow needs to:
- ✅ Create or update issues
- ✅ Add comments to issues/PRs
- ✅ Create pull requests
- ✅ Create discussions

Don't use safe outputs for:
- ❌ Reading data (use regular tools)
- ❌ Analyzing content (no write operations needed)

### Common Beginner Mistake

**Wrong:**
\`\`\`markdown
---
safe-outputs:
  create-issue:
---

Analyze the pull request and provide feedback.
\`\`\`

The AI won't know to create an issue because you didn't tell it to!

**Right:**
\`\`\`markdown
---
safe-outputs:
  create-issue:
---

Analyze the pull request and **create an issue** with your findings.
\`\`\`

**Rule:** Always explicitly tell the AI to create/add/update things in your natural language instructions.

---

5. Unclear GitHub Actions Knowledge Assumptions

Location: Throughout documentation, especially /docs/src/content/docs/concepts/how-it-works.md

Issue: The docs assume users understand:

• What GitHub Actions is
• How workflows run in GitHub Actions
• Where to find workflow run logs
• How to debug failed runs

Many beginners may have never used GitHub Actions before.

Recommendation: Add a "GitHub Actions Basics" section:

## New to GitHub Actions?

GitHub Agentic Workflows build on GitHub Actions. Here's what you need to know:

### What is GitHub Actions?
GitHub's automation platform that runs code when events happen in your repository (issues opened, PRs created, scheduled times, etc.).

### Where do workflows run?
On GitHub's servers (called "runners"), not on your local machine.

### How to see if a workflow ran:
1. Go to your repository on GitHub
2. Click the "Actions" tab
3. Find your workflow run in the list
4. Click to see logs and results

### Common debugging:
- **Workflow didn't run?** Check the trigger conditions in frontmatter
- **Workflow failed?** Click into the failed run to see error logs
- **Changes not taking effect?** Did you compile and commit the `.lock.yml` file?

**Learn more:** [GitHub Actions documentation](https://docs.github.com/en/actions)

---

🟡 Confusing Areas (Slow Down Learning)

1. Jargon Without Definitions

Terms used without explanation on first mention:

• "Frontmatter" - assumed knowledge from Jekyll/static site generators
• "MCP" and "MCP Server" - explained late in docs
• "Lock file" - .lock.yml mentioned before it's explained
• "Safe outputs" - critical concept introduced mid-page
• "Engine" - used interchangeably with "AI" and "agent"
• "Compilation" - programming term that may confuse non-developers

Recommendation: Add inline tooltips or a "Key Terms" sidebar on early pages, or link to glossary on first use.

---

2. Examples Show Complex Cases First

Issue: Most examples show feature-rich workflows with many options. Beginners need to see:

1. Minimal working example
2. Add one feature
3. Add another feature
4. Full-featured example

Current order:

• ❌ Starts with full-featured workflows
• ❌ Hard to know what's required vs. optional
• ❌ Can't easily strip down to basics

Example from Quick Start:

tools:
  github:
    allowed:
      - issue_read
      - add_labels_to_issue
      - create_issue_comment

A beginner asks: "Do I need to list all these? Can I just use github:? What if I list the wrong ones?"

Recommendation: Progressive examples:

### Example 1: Minimal
\`\`\`yaml
tools:
  github:  # Default: basic read/write operations
\`\`\`

### Example 2: Restrict to Specific Tools
\`\`\`yaml
tools:
  github:
    allowed:
      - issue_read
      - create_issue_comment
\`\`\`

### Example 3: Custom Configuration
\`\`\`yaml
tools:
  github:
    allowed:
      - issue_read
      - create_issue_comment
    mode: remote
    read-only: false
\`\`\`

---

3. Templating Syntax Under-Explained

Location: /docs/src/content/docs/reference/templating.md

Issue: The page lists allowed expressions but doesn't explain:

• What templating is for beginners
• When to use ${{ }} syntax
• Why some expressions are allowed and others aren't
• How to debug template errors

Example that confused me:

Use GitHub Actions context expressions throughout the workflow content.

As a beginner: "What's a context expression? How is this different from regular text?"

Recommendation: Start with basics:

## What is Templating?

Templating lets you insert dynamic values into your workflows.

**Example:** Instead of hardcoding an issue number:
\`\`\`
Analyze issue #42
\`\`\`

Use a template to reference the actual issue that triggered the workflow:
\`\`\`
Analyze issue #${{ github.event.issue.number }}
\`\`\`

### Template Syntax

- Templates use `${{ }}` syntax
- They're replaced with actual values when the workflow runs
- You can access GitHub context (repo, user, event data)

### Security Note

For security, only specific expressions are allowed. You cannot use:
- ❌ `${{ secrets.MY_SECRET }}` (could leak secrets)
- ❌ `${{ env.MY_VAR }}` (could be manipulated)
- ✅ `${{ github.event.issue.number }}` (safe, read-only data)

---

4. Tool Configuration Overwhelming

Location: /docs/src/content/docs/reference/tools.md

Issue: The tools page lists many options without:

• Prioritization (which tools are most commonly needed?)
• Decision tree (how do I choose?)
• Security implications clearly stated
• Simple vs. advanced configurations separated

As a beginner looking at:

tools:
  github:
  edit:
  web-fetch:
  web-search:
  bash:
  playwright:

I think: "Do I need all of these? What does each one do? Are some dangerous?"

Recommendation: Categorize and prioritize:

## Tools by Use Case

### 🔵 Essential Tools (Start Here)

**`github:`** - Read and write GitHub data
- **Use when:** Working with issues, PRs, comments, labels
- **Security:** Safe with `safe-outputs`, risky with direct permissions
- **Example:** `tools: github:`

### 🟢 Common Tools

**`edit:`** - Modify files in the repository
- **Use when:** Creating or updating code files
- **Security:** ⚠️ Can modify any file - use with caution
- **Example:** Need to create a PR with code changes

**`web-fetch:`** - Download web content
- **Use when:** Fetching documentation, APIs, or external data
- **Security:** Requires network permissions
- **Example:** Research latest version of a library

### 🟡 Advanced Tools

**`bash:`** - Run shell commands
- **Use when:** Complex file operations or system commands needed
- **Security:** ⚠️⚠️ Very powerful - restrict to specific commands
- **Example:** Only for power users

**`playwright:`** - Browser automation
- **Use when:** Interacting with web pages, testing UIs
- **Security:** Requires network and browser permissions
- **Example:** Advanced web scraping or testing

---

5. Error Messages Not Documented

Issue: Common error messages aren't documented with solutions.

Beginners will encounter errors like:

• "Compilation failed: invalid frontmatter"
• "Authentication required"
• "Workflow did not produce any output"
• "Permission denied: issues"

But there's no troubleshooting guide for these.

Recommendation: Add "Common Errors" page:

# Troubleshooting Common Errors

## Compilation Errors

### "Invalid frontmatter"
**Cause:** YAML syntax error in the `---` section
**Fix:** Check for:
- Missing colons after keys
- Incorrect indentation (use spaces, not tabs)
- Unquoted strings with special characters

### "Unknown field in frontmatter"
**Cause:** Typo or unsupported configuration option
**Fix:** Check spelling against [frontmatter reference](/reference/frontmatter/)

## Runtime Errors

### "Authentication required"
**Cause:** Missing or invalid GitHub token
**Fix:** See [Authentication Setup](#authentication-setup)

### "Permission denied: issues"
**Cause:** Workflow doesn't have permission to modify issues
**Fix:** Either:
- Add `permissions: issues: write` to frontmatter (not recommended)
- Use `safe-outputs: create-issue:` instead (recommended)

### "Workflow did not produce any output"
**Cause:** AI didn't generate any safe outputs
**Fix:** Make sure your instructions explicitly tell the AI to create/add/update content

---

🟢 What Worked Well

✅ Excellent Conceptual Explanations

The "How It Works" and conceptual pages are well-written and help build mental models. The glossary is particularly helpful for understanding terminology.

✅ Good Example Variety

The examples section covers many real-world use cases (issue labeling, ChatOps, research, security). Having concrete examples helps beginners see what's possible.

✅ Clear Frontmatter Reference

The frontmatter reference (/docs/src/content/docs/reference/frontmatter.md) is comprehensive and well-organized. Once you understand the basics, this is a great reference.

✅ Progressive Disclosure in Some Areas

The tools section does a good job of showing simple configurations first, then advanced options.

✅ Security Consciousness

The docs emphasize security throughout, which is appropriate for AI-powered automation. The safe-outputs pattern is a great design choice.

✅ Code Examples are Correct

The code examples I reviewed are syntactically correct and follow best practices. They just need better progression from simple to complex.

---

📋 Recommendations

Priority 1: Critical Path to Success

1. Add Prerequisites section to Quick Start (authentication, token setup)
2. Create "Hello World" minimal example before complex examples
3. Emphasize compilation requirement with warnings and mental models
4. Add "Getting Started Checklist" that users can follow step-by-step
5. Create Troubleshooting page for common errors

Priority 2: Reduce Confusion

6. Add "New to GitHub Actions?" section for complete beginners
7. Create glossary tooltips for first use of jargon
8. Reorganize examples from simple to complex
9. Add decision trees for tool selection and configuration choices
10. Explain templating basics before listing allowed expressions

Priority 3: Polish and Enhancement

11. Add video tutorial showing complete workflow creation
12. Create interactive tutorial (if technically feasible)
13. Add "Common Patterns" cookbook with copy-paste examples
14. Improve search with better keywords and synonyms
15. Add "Upgrade Guide" for users coming from GitHub Actions

---

🎬 Quick Wins

These changes would have the biggest immediate impact:

1. Add to Quick Start (Top of Page):

## ⚠️ Before You Start

You'll need:
- [ ] GitHub CLI installed
- [ ] GitHub account
- [ ] Personal Access Token with `repo` and `workflow` permissions

[→ Complete setup instructions](#prerequisites)

2. Replace Complex First Example with:

## Hello World

Create `.github/workflows/hello.md`:

\`\`\`markdown
---
on:
  command:
    name: hello
permissions:
  contents: read
---

Comment "Hello from agentic workflows!"
\`\`\`

Compile and push:
\`\`\`bash
gh aw compile hello
git add .github/workflows/hello.*
git commit -m "Add hello workflow"
git push
\`\`\`

Test: Comment `/hello` on any issue!

3. Add Compilation Reminder Box:

> 💡 **Remember:** Always run `gh aw compile` after editing workflows!
> GitHub Actions runs the `.lock.yml` file, not your `.md` file.

4. Add to Every Example:

**Difficulty:** ⭐☆☆ Beginner | ⭐⭐☆ Intermediate | ⭐⭐⭐ Advanced
**Time:** ~5 minutes
**Prerequisites:** Basic workflow understanding

---

📸 Documentation Pages Analyzed

1. /docs/src/content/docs/index.mdx - Home page
2. /docs/src/content/docs/setup/quick-start.md - Primary getting started guide
3. /docs/src/content/docs/setup/cli.md - CLI commands reference
4. /docs/src/content/docs/setup/agentic-authoring.md - Workflow creation guide
5. /docs/src/content/docs/concepts/how-it-works.md - Core concepts
6. /docs/src/content/docs/concepts/agentic-workflows-explained.md - Conceptual overview
7. /docs/src/content/docs/examples/issue-labeling.md - Issue labeling example
8. /docs/src/content/docs/examples/comment-triggered.md - ChatOps examples
9. /docs/src/content/docs/reference/frontmatter.md - Configuration reference
10. /docs/src/content/docs/reference/safe-outputs.md - Safe outputs reference
11. /docs/src/content/docs/reference/tools.md - Tools reference
12. /docs/src/content/docs/reference/templating.md - Template expressions
13. /docs/src/content/docs/reference/glossary.md - Term definitions
14. /docs/src/content/docs/guides/web-search.md - Web search guide
15. /docs/src/content/docs/guides/security.md - Security guide

---

💬 Final Thoughts

The GitHub Agentic Workflows documentation is well-structured and technically accurate, but it assumes too much prior knowledge for true beginners. With focused improvements to the Quick Start guide, better progressive examples, and clearer explanations of critical concepts (authentication, compilation, safe-outputs), the documentation could provide a much smoother onboarding experience.

The biggest gap is the lack of a trivial "Hello World" example that helps beginners build a mental model before introducing complexity. The second biggest gap is authentication setup, which is mentioned late and not emphasized as a prerequisite.

As a beginner, I would have benefited most from:

1. A prerequisites checklist
2. A 5-minute "Hello World" tutorial
3. Clear warnings about common mistakes (forgetting to compile, missing authentication)
4. Progressive examples that build in complexity
5. A troubleshooting guide for common errors

Overall Rating: 7/10 for technical accuracy, 5/10 for beginner-friendliness

---

Test completed: 2024-11-27
Testing methodology: Manual review of documentation source files as a first-time user
Coverage: Quick Start path, core concepts, examples, and reference documentation

AI generated by Documentation Noob Tester

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.