Help evaluator prompt and script under ./scripts/ai/evaluate-help.py #32469
+1,082
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
️✔️AzureCLI-FullTest
|
️✔️AzureCLI-BreakingChangeTest
|
Collaborator
|
Thank you for your contribution! We will review the pull request and get back to you soon. |
|
The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR. Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions). pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>
|
microsoft-github-policy-service
bot
requested review from
jiasli,
jsntcy,
wangzelin007 and
yonzhan
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related command
N/A - This PR adds new tooling for help documentation analysis, not command changes.
Description
This PR introduces a new AI-powered tool for evaluating the quality of Azure CLI help documentation. The tool uses Azure OpenAI to analyze help files and provide detailed quality assessments based on multiple frameworks.
Key Components:
help_evaluator.py: Core evaluation class that handles Azure OpenAI integration, prompt management, and analysis generationevaluate-help.py: Command-line interface for running evaluations on single files or entire directoriesprompts/directoryFeatures:
*_help.pyfiles in specified directoriesTesting Guide
cd scripts/ai pip install -r requirements.txt.env:cp .env.example .env # Edit .env with your Azure OpenAI credentialsscripts/ai/analysisHistory Notes
{Scripts} Add AI-powered help documentation quality analysis tool
This checklist is used to make sure that common guidelines for a pull request are followed.
The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.