Reliability Guardian Agent - #QodoAgentChallenge Competition Submission

[Kiran1689]

User description

Reliability Guardian Agent

A specialized Qodo AI Agent that evaluates and safeguards your project’s code reliability through deep analysis of logic consistency, validation coverage, test strength, and historical stability trends.

Competition Category

Best Agent for Reliable Code

🧭 Overview

The Reliability Guardian Agent continuously analyzes your codebase to assess and improve code reliability, fault tolerance, and test robustness.

It performs intelligent static and behavioral analysis to detect:

• Logic inconsistencies or potential runtime errors
• Weak or missing test coverage areas
• Mutation and fuzz testing weaknesses (simulated safely)
• Error handling gaps or fragile code paths
• Reliability regressions introduced in recent commits

This agent can be used both locally and in automated CI/CD workflows, providing an objective reliability score and actionable recommendations.

🧩 Key Features

• Context-Aware Reliability Analysis
• Structured Output Summary
• Highly Configurable
• Runs Anywhere
  • Locally via CLI (qodo reliability_guardian)
  • Automatically on PRs via GitHub/GitLab/Jenkins/Azure
  • As a pre-commit hook to block risky commits
  • Inside IDEs like VS Code and IntelliJ
• Quality Gate Support

📁 New Files Added

agents/
└── reliability-guardian/
    ├── agent.toml
    ├── agent.yaml
    ├── README.md
    └── examples/
        ├── usage.md
        ├── ci-configs/
           ├── github-actions.yml
           ├── gitlab-ci.yml
           ├── jenkins-pipeline.groovy
           └── azure-devops.yml

👨‍💻Testing and Demo

Tested the agent, full demo walk through 👉https://vimeo.com/1127562890

---

PR Type

Enhancement

---

Description

• Added Reliability Guardian Agent for code reliability analysis and scoring

• Implemented CI/CD integration configs for GitHub Actions, GitLab CI, Jenkins, and Azure DevOps

• Created comprehensive documentation with usage examples and IDE integration guides

• Defined agent configuration with TOML and YAML formats supporting reliability metrics

---

Diagram Walkthrough

flowchart LR
  A["Agent Configuration"] --> B["TOML/YAML Definitions"]
  A --> C["Reliability Analysis Logic"]
  C --> D["CI/CD Integrations"]
  D --> E["GitHub Actions"]
  D --> F["GitLab CI"]
  D --> G["Jenkins Pipeline"]
  D --> H["Azure DevOps"]
  C --> I["Documentation"]
  I --> J["README"]
  I --> K["Usage Examples"]

Loading

File Walkthrough

	Relevant files
Configuration changes	agent.toml Agent configuration in TOML format with reliability scoring schema agents/reliability-guardian/agent.toml Defines reliability_guardian command with comprehensive reliability analysis instructions Configures five input parameters including target_branch, max_commits, and testing flags Specifies structured JSON output schema with reliability scores and issue tracking Sets up tools integration (qodo_merge, git, filesystem) and exit conditions +121/-0  agent.yaml Agent configuration in YAML format with identical functionality agents/reliability-guardian/agent.yaml Mirrors TOML configuration in YAML format for alternative configuration style Defines same reliability analysis command with identical parameters and schema Includes logic consistency, validation coverage, exception safety, and test strength metrics Configures execution strategy and approval-based exit expression +143/-0  github-actions.yml GitHub Actions workflow for automated reliability analysis on PRs agents/reliability-guardian/examples/ci-configs/github-actions.yml Configures GitHub Actions workflow triggered on pull requests to main and develop branches Sets up required permissions for contents, pull requests, and checks Executes Reliability Guardian Agent with configurable parameters via qodo-ai/command@v1 action Passes environment variables for QODO_API_KEY and GITHUB_TOKEN authentication +31/-0    gitlab-ci.yml GitLab CI configuration for merge request reliability checks agents/reliability-guardian/examples/ci-configs/gitlab-ci.yml Defines GitLab CI pipeline stage for reliability review on merge requests Uses qodoai/command:latest Docker image for agent execution Configures dynamic target branch detection from merge request context Sets up environment variables for Qodo and GitLab authentication +18/-0    jenkins-pipeline.groovy Jenkins pipeline for reliability analysis with artifact archiving agents/reliability-guardian/examples/ci-configs/jenkins-pipeline.groovy Implements Jenkins declarative pipeline with Docker-based agent execution Configures credentials management for QODO_API_KEY and GITHUB_TOKEN Dynamically determines target branch from pull request context or defaults to main Archives JSON artifacts in post-build actions for result persistence +34/-0    azure-devops.yml Azure DevOps pipeline configuration for PR reliability checks agents/reliability-guardian/examples/ci-configs/azure-devops.yml Configures Azure DevOps pipeline triggered on PRs to main and develop branches Implements Docker task to run Reliability Guardian Agent in containerized environment Uses variable groups for credential management and system variables for PR context Passes target branch dynamically from pull request metadata +38/-0
Documentation	README.md Main documentation with features, usage, and integration examples agents/reliability-guardian/README.md Provides comprehensive overview of Reliability Guardian Agent features and capabilities Documents configuration parameters, output format, and reliability scoring methodology Includes quick start guide, GitHub Actions integration example, and pre-commit hook setup Lists requirements including API keys and repository permissions +315/-0  usage.md Comprehensive usage examples and integration patterns documentation agents/reliability-guardian/examples/usage.md Demonstrates basic and advanced CLI usage patterns for different project types Provides complete JSON output schema example with reliability scores and issues Includes CI/CD integration examples for GitHub Actions, GitLab CI, and Jenkins with quality gates Documents IDE integration for VS Code and IntelliJ IDEA, plus pre-commit hook configurations +461/-0

---

[qodo-merge-for-open-source]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Credential exposure risk Description: Jenkins pipeline exposes use of credential IDs 'qodo-api-key' and 'github-token' but later runs arbitrary shell within a Docker container, which can inadvertently leak tokens via environment into scripts or logs if commands echo env; ensure the container is trusted and masks env in logs. jenkins-pipeline.groovy [6-8] Referred Code QODO_API_KEY = credentials('qodo-api-key') GITHUB_TOKEN = credentials('github-token') }
	Unpinned action secrets risk Description: GitHub Actions workflow passes secrets QODO_API_KEY and GITHUB_TOKEN to a third-party action 'qodo-ai/command@v1'; if the action is not pinned to a commit SHA and compromised, secrets could be exfiltrated. github-actions.yml [22-23] Referred Code QODO_API_KEY: ${{ secrets.QODO_API_KEY }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	Untrusted image with secrets Description: GitLab CI job uses public Docker image 'qodoai/command:latest' and passes QODO_API_KEY and CI_JOB_TOKEN as environment variables; relying on ':latest' and untrusted images can lead to secret exfiltration. gitlab-ci.yml [7-16] Referred Code image: qodoai/command:latest script: - | command \ --prompt reliability_guardian \ --agent-file path/to/agent.toml \ --key-value-pairs "target_branch=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME:-main},max_commits=5,mutation_testing=true,fuzz_testing=true" variables: QODO_API_KEY: $QODO_API_KEY GITLAB_TOKEN: $CI_JOB_TOKEN
	Image tag not pinned Description: Azure DevOps pipeline runs a Docker image 'qodoai/command:latest' and injects tokens via envVars; using ':latest' and third-party images without digest pinning risks leaking credentials if the image is compromised. azure-devops.yml [31-38] Referred Code image: 'qodoai/command:latest' arguments: > --prompt reliability_guardian --agent-file path/to/agent.toml --key-value-pairs "target_branch=$(System.PullRequest.TargetBranch),max_commits=5,mutation_testing=true,fuzz_testing=true" envVars: | QODO_API_KEY=$(qodo-api-key) AZURE_TOKEN=$(System.AccessToken)
Ticket Compliance
⚪	🎫 No ticket provided - [ ] Create ticket/issue <!-- /create_ticket --create_ticket=true --> </details></td></tr>
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	No custom compliance provided Follow the guide to enable custom compliance check.

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-merge-for-open-source]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Security	Secure credential handling in Jenkins Improve security in the Jenkins pipeline example by using the withCredentials block to handle secrets. This prevents sensitive credentials like API keys from being exposed in logs. agents/reliability-guardian/examples/ci-configs/jenkins-pipeline.groovy [5-8] -environment { - QODO_API_KEY = credentials('qodo-api-key') - GITHUB_TOKEN = credentials('github-token') +stages { + stage('Reliability Guardian') { + steps { + script { + def targetBranch = env.CHANGE_TARGET ?: 'main' + + withCredentials([ + string(credentialsId: 'qodo-api-key', variable: 'QODO_API_KEY'), + string(credentialsId: 'github-token', variable: 'GITHUB_TOKEN') + ]) { + docker.image('qodoai/command:latest').inside { + sh """ + command \ + --prompt reliability_guardian \ + --agent-file path/to/agent.toml \ + --key-value-pairs "target_branch=${targetBranch},max_commits=5,mutation_testing=true,fuzz_testing=true" + """ + } + } + } + } + } } Apply / Chat Suggestion importance[1-10]: 9 __ Why: This suggestion addresses a critical security vulnerability by recommending the use of withCredentials to handle secrets, which is the standard best practice in Jenkins to prevent credential leakage.	High
General	Validate score extraction before comparison Add a check in the pre-commit hook script to ensure the reliability score was successfully extracted before attempting to use it in a comparison. This prevents potential errors if the grep command fails. agents/reliability-guardian/examples/usage.md [450-456] score=$(grep -Eo '"overall": [0-9]+\.[0-9]+' /tmp/reliability.log | grep -Eo '[0-9]+\.[0-9]+') + +if [ -z "$score" ]; then + echo "❌ Failed to extract reliability score from output" + exit 1 +fi + echo "Reliability Score: $score" if (( $(echo "$score < 7.0" | bc -l) )); then echo "❌ Reliability gate failed — score below 7.0" exit 1 fi Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies a potential failure point in the example script and adds necessary validation to prevent cryptic errors, making the pre-commit hook more robust.	Medium
	Add fallback exit condition logic Make the exit_expression more robust by adding a fallback condition. The new expression should check the reliability_score and critical_issues count if the approved field is not available. agents/reliability-guardian/agent.toml [121] -exit_expression = "approved" +exit_expression = "approved == true or (summary.reliability_score.overall >= 7.0 and summary.critical_issues == 0)" Apply / Chat Suggestion importance[1-10]: 6 __ Why: The suggestion correctly identifies that relying on a single approved field is brittle and improves the agent's robustness by adding a fallback exit condition based on other output fields.	Low
	Optimize Git history fetch depth Optimize the CI pipeline's performance by changing fetch-depth: 0 to a smaller value like 20. This avoids fetching the entire Git history, which is unnecessary for the agent's analysis. agents/reliability-guardian/examples/ci-configs/github-actions.yml [15-17] - uses: actions/checkout@v4 with: - fetch-depth: 0 + fetch-depth: 20 Apply / Chat Suggestion importance[1-10]: 5 __ Why: This is a valid performance optimization for the example CI configuration, as fetching the entire git history is unnecessary for the agent's operation and can slow down the pipeline.	Low
More