Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/main' into feature/telemetry
Browse files Browse the repository at this point in the history
  • Loading branch information
ralph-msft committed Mar 7, 2024
2 parents b26d2c1 + aad961d commit 34d3288
Show file tree
Hide file tree
Showing 1,311 changed files with 58,737 additions and 1,503 deletions.
1 change: 1 addition & 0 deletions .aitests/config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
testDirectory: ../tests
2 changes: 1 addition & 1 deletion .azure/pipelines/docker_proxy_setup.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ steps:
inputs:
command: build
repository: nginx-recording-test-proxy-dev
dockerfile: 'tests/recordproxy/Dockerfile'
dockerfile: 'src/testing/adapters/recordproxy/Dockerfile'
containerRegistry: 'acrbn-acr'
- bash: |
docker images
Expand Down
21 changes: 9 additions & 12 deletions .azure/pipelines/nginx-docker-build-and-push.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ variables:
baseVer: '20.04'
# The path of the image (should normally not change)
repositorydocker: nginx-recording-test-proxy
dockerfile: 'tests/recordproxy/Dockerfile.base'
dockerfile: 'src/testing/adapters/recordproxy/Dockerfile.base'
version: $(Build.BuildId)

trigger:
Expand Down Expand Up @@ -47,14 +47,13 @@ stages:
inputs:
containerRegistry: 'acrbn-acr'
command: 'login'
- task: Docker@2
displayName: Build ($(dockerfile) - $(version))
inputs:
command: 'build'
containerRegistry: 'acrbn-acr'
repository: $[variables.repositorydocker]
Dockerfile: $(dockerfile)
buildContext: '.'
- task: Docker@2
displayName: Build ($(dockerfile) - $(version))
inputs:
command: 'build'
containerRegistry: 'acrbn-acr'
repository: $(repositorydocker)
Dockerfile: $(dockerfile)

################################################################################
## SBOM
Expand Down Expand Up @@ -110,6 +109,4 @@ stages:
command: 'push'
containerRegistry: 'acrbn-acr'
repository: '$(repositorydocker)'
tags: |
${{ variables.repositorydocker }}
${{ variables.repositorydocker }}-$(Build.BuildId)
tags: $(Build.BuildId)
25 changes: 25 additions & 0 deletions .azure/pipelines/test_execute_and_report.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,31 @@ parameters:
TestFilter: 'TestCategory!=SkipOnVSTS'
envToSet: ''
steps:

# -----------------------------------------------------------------------------
# Ensure we can use the appropriate Java, Go, and NodeJs tools
# -----------------------------------------------------------------------------
- bash: |
sudo apt update
sudo apt install -y msopenjdk-21
displayName: 'Install msopenjdk-21'
continueOnError: true
# Equivalent to: sudo apt-get install -y golang-go
- task: GoTool@0
displayName: 'Use Go 1.18'
inputs:
version: '1.18'
continueOnError: true

# Equivalent to:
# curl -fsSL https://deb.nodesource.com/setup_21.x | sudo -E bash - &&\
# sudo apt-get install -y nodejs
- task: NodeTool@0
displayName: 'Use Node.js 21.x'
inputs:
versionSpec: '21.x'
continueOnError: true

# -----------------------------------------------------------------------------
# Run the tests
# -----------------------------------------------------------------------------
Expand Down
5 changes: 3 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
**/.ai/data/*
**/.pyenv/**
**/.vs/**
**/.vscode/**
**/bin/*/net6.0/*
**/bin/*/net7.0/*
**/bin/*/net8.0/*
**/obj/*
**/Properties/launchSettings.json
ideas/website/node_modules/**
testresults/**
*.user
**/launchSettings.json
*.user
14 changes: 14 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Contributing

This project welcomes contributions and suggestions. Most contributions require you to
agree to a Contributor License Agreement (CLA) declaring that you have the right to,
and actually do, grant us the rights to use your contribution. For details, visit
https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need
to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the
instructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
or contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
17 changes: 12 additions & 5 deletions ai-cli.sln
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,21 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "helper_functions_extension"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Extensions", "Extensions", "{644B75F1-C768-4DB3-BAF2-C69A1F36DD28}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "test_helper_functions_extension", "src\extensions\test_helper_functions_extension\test_helper_functions_extension.csproj", "{7BD6EF67-BA75-478D-9721-C1B2AB6DE3FF}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "test_helper_functions_extension", "src\testing\samples\helper_functions_sample_project\test_helper_functions_extension.csproj", "{7BD6EF67-BA75-478D-9721-C1B2AB6DE3FF}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "template_extension", "src\extensions\template_extension\template_extension.csproj", "{023B4F9C-E2B3-4CCD-A993-87E337C16EDE}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "YamlTestAdapter", "tests\testadapter\YamlTestAdapter.csproj", "{7C3F1355-B679-487D-904D-7E5FEBA9E75C}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "YamlTestAdapter", "src\testing\adapters\testadapter\YamlTestAdapter.csproj", "{7C3F1355-B679-487D-904D-7E5FEBA9E75C}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "YamlTestFramework", "tests\testframework\YamlTestFramework.csproj", "{B0B3437F-1828-4A13-866F-1CF7C924015E}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "YamlTestFramework", "src\extensions\testframework\YamlTestFramework.csproj", "{B0B3437F-1828-4A13-866F-1CF7C924015E}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "YamlTestRunner", "tests\testrunner\YamlTestRunner.csproj", "{39876475-2D98-40CF-8B08-CD423A5EB4E8}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "YamlTestRunner", "src\testing\runner\YamlTestRunner.csproj", "{39876475-2D98-40CF-8B08-CD423A5EB4E8}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Tests", "Tests", "{C8AFF891-D6AA-4B8F-BC21-10404DF4B355}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "RecordedYamlTestAdapter", "tests\recordingadapter\RecordedYamlTestAdapter.csproj", "{9499C018-FA08-4133-93B3-FC0F3863A6CC}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "RecordedYamlTestAdapter", "src\testing\adapters\recordingadapter\RecordedYamlTestAdapter.csproj", "{9499C018-FA08-4133-93B3-FC0F3863A6CC}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "speech-extension", "src\extensions\speech_extension\speech-extension.csproj", "{CED7C805-0435-4BF7-A42F-9F3BBF14A18F}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "telemetry.aria", "src\telemetry\telemetry.aria.csproj", "{306A3CD6-91C2-450B-9995-79701CE63FE2}"
EndProject
Expand Down Expand Up @@ -78,6 +80,10 @@ Global
{306A3CD6-91C2-450B-9995-79701CE63FE2}.Debug|Any CPU.Build.0 = Debug|Any CPU
{306A3CD6-91C2-450B-9995-79701CE63FE2}.Release|Any CPU.ActiveCfg = Release|Any CPU
{306A3CD6-91C2-450B-9995-79701CE63FE2}.Release|Any CPU.Build.0 = Release|Any CPU
{CED7C805-0435-4BF7-A42F-9F3BBF14A18F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CED7C805-0435-4BF7-A42F-9F3BBF14A18F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{CED7C805-0435-4BF7-A42F-9F3BBF14A18F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{CED7C805-0435-4BF7-A42F-9F3BBF14A18F}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand All @@ -90,6 +96,7 @@ Global
{B0B3437F-1828-4A13-866F-1CF7C924015E} = {C8AFF891-D6AA-4B8F-BC21-10404DF4B355}
{39876475-2D98-40CF-8B08-CD423A5EB4E8} = {C8AFF891-D6AA-4B8F-BC21-10404DF4B355}
{9499C018-FA08-4133-93B3-FC0F3863A6CC} = {C8AFF891-D6AA-4B8F-BC21-10404DF4B355}
{CED7C805-0435-4BF7-A42F-9F3BBF14A18F} = {644B75F1-C768-4DB3-BAF2-C69A1F36DD28}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {002655B1-E1E1-4F2A-8D53-C9CD55136AE2}
Expand Down
5 changes: 5 additions & 0 deletions ideas/cool-helper-function-scenarios/HelperFunctionClass.cs
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
//
// Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE.md file in the project root for full license information.
//

using Azure.AI.Details.Common.CLI.Extensions.HelperFunctions;

public static class HelperFunctionClass
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,9 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Azure.Core" Version="1.36.0" />
<PackageReference Include="Azure.Identity" Version="1.10.4" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
<PackageReference Include="Azure.AI.OpenAI" Version="1.0.0-beta.12" />
<PackageReference Include="Azure.AI.OpenAI" Version="1.0.0-beta.13" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
//
// Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE.md file in the project root for full license information.
//

using System;
using System.Net.Http;
using System.Threading.Tasks;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
//
// Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE.md file in the project root for full license information.
//

using System;
using System.Collections.Generic;
using System.Net.Http;
Expand Down
5 changes: 5 additions & 0 deletions ideas/cool-helper-function-scenarios/QRCodeHelper.cs
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
//
// Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE.md file in the project root for full license information.
//

using Azure.AI.Details.Common.CLI.Extensions.HelperFunctions;

public static class QRCodeHelper
Expand Down
5 changes: 5 additions & 0 deletions ideas/cool-helper-function-scenarios/WolframAlphaHelper.cs
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
//
// Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE.md file in the project root for full license information.
//

using Azure.AI.Details.Common.CLI.Extensions.HelperFunctions;
using System;
using System.Net.Http;
Expand Down
174 changes: 174 additions & 0 deletions ideas/junit-conversation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
You
show me a sample pytest reporting xml file output

Assistant
Sure, here's a simplified example of a potential XML file output from Pytest using JUnit-style reporting:
<?xml version="1.0" encoding="utf-8"?>
<testsuite name="pytest" tests="5" errors="1" failures="1" skipped="0">
<testcase classname="test_module" name="test_pass" time="0.1" >
</testcase>
<testcase classname="test_module" name="test_fail" time="0.1" >
<failure message="Failing test" type="type_error">Traceback (most recent call last): Test `test_fail` failed with AssertionError</failure>
</testcase>
<testcase classname="test_module" name="test_error" time="0.1" >
<error message="Error in test" type="type_error">Traceback (most recent call last): Test `test_error` error out with TypeError</error>
</testcase>
<testcase classname="test_module" name="test_skip" time="0.1" >
<skipped message="Skipped test" type="skip_error">Traceback (most recent call last): Test `test_skip` skipped with unittest.SkipTest</skipped>
</testcase>
<testcase classname="another_test_module" name="test_pass" time="0.1" >
</testcase>
</testsuite>

This file provides a summary of the test results, including the number of tests run, number of errors, failures, and skipped tests, as well as detailed information about each individual test.
Please note that the actual contents can greatly vary depending on the complexity of your tests and configuration of your Pytest.

You
is testsuite always the root?

Assistant
Yes, in JUnit-style XML reports, the <testsuite> element is generally the root. It represents a series of test cases (represented by <testcase> elements) that together form a test suite.
However, if you generate the report with pytest's --junit-prefix option or you're running the test within a suite of test suites, the report might include a wrapping <testsuites> element:
<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite name="pytest" tests="5" errors="1" failures="1" skipped="0">
<!-- test cases here -->
</testsuite>
<!-- potentially more testsuites here -->
</testsuites>

In this case <testsuites> is the root node, containing one or more <testsuite> elements.

You
ok. under testsuite, is testcase the only valid child?

Assistant
Primarily, yes, the <testsuite> tag in a JUnit XML report contains a series of <testcase> elements. Each <testcase> element represents one test.
However, <testsuite> can also have a few other types of children in addition to <testcase>. These additional child elements provide metadata about the suite as a whole or report on issues that affected the suite rather than individual tests.
These additional tags include:
<properties>: A collection of properties, or environment data, which applies to the entire test suite. This is usually used to store information relevant to the environment where the tests were run.
<error>: Indicates an error that prevented the test suite from running to completion. This tag is not associated with an individual test case.
<system-out> and <system-err>: These capture text that was sent to standard output or standard error during test execution. This can include logs or debugging information.
Here's a simplified example:
<testsuite name="pytest" tests="5" errors="1" failures="1" skipped="0">
<properties>
<property name="exampleProperty" value="exampleValue"/>
</properties>
<testcase classname="test_module" name="test_pass" time="0.1"/>
<!-- additional testcases here -->
<error message="Fatal suite error"/>

<system-out>
System output text (e.g., logs) goes here...
</system-out>

<system-err>
System error text (e.g., exception messages) goes here...
</system-err>
</testsuite>

Remember that the actual content and structure can vary based on your test configuration and the specifics of your test environment.

You
what are all the valid children of testcase?

Assistant
In a JUnit-style XML report, the <testcase> element can have several types of child elements. Each one provides additional information about the result or execution of the specific test case.
Here are the key types:
<error>: This indicates an error that occurred while the test was running. This is typically used for unexpected problems, such as uncaught exceptions. The message attribute provides a short description of the error.
<failure>: This indicates that the test case failed, i.e., a test assertion was not met. The message attribute provides a short description of the failure.
<skipped>: This indicates that the test case was skipped and not run for some reason. This could be because the test was manually disabled, or a setup method failed. This element usually has no content.
<system-out> and <system-err>: These capture text that was sent to standard output or standard error during the execution of this specific test case. This could be logs or debugging information.
Here's an example of a testcase element with failure and system-out children:
<testcase classname="test_module" name="test_fail" time="0.1" >
<failure message="Failing test" type="assertion_error">Traceback (most recent call last):...</failure>
<system-out>Some standard output text here...</system-out>
</testcase>

Remember that the actual content and structure of these elements can vary based on the specifics of your test configuration and environment.

You
tell me all the xml elements, valid parents, valid children, and what are required and optional attributes.

Assistant
JUnit XML is a standard to report test results that can be processed by tools that understand the format. Below, you'll find a summary of all XML elements, their valid parents and children, and required and optional attributes.

<testsuites>: The root element for an XML document representing collection of test suites.
Parent: None
Children: <testsuite>
Attributes: None

<testsuite>: Represents a test suite which is a collection of test cases.
Parent: <testsuites> (optional), <testsuite>
Children: <properties>, <testcase>, <system-out>, <system-err>, and <errors>
Attributes: name (required), tests (required), failures, errors, skipped, time,timestamp, hostname, id, package

<properties>: Represents a set of properties relevant to the entire suite.
Parent: <testsuite>
Children: <property>
Attributes: None

<property>: Represents a single property in the set.
Parent: <properties>
Children: None
Attributes: name (required), value (required)

<testcase>: Represents a single test case.
Parent: <testsuite>
Children: <error>, <failure>, <skipped>, <system-out>, <system-err>
Attributes: name (required), classname (required), time

<error>: Represents an error that occurred while the test was running.
Parent: <testcase>, <testsuite>
Children: None
Attributes: message, type

<failure>: Represents a failure of a test case.
Parent: <testcase>
Children: None
Attributes: message, type

<skipped>: Represents a test case that was skipped.
Parent: <testcase>
Children: None
Attributes: None

<system-out>: Captures text that was sent to standard output during the execution.
Parent: <testcase>, <testsuite>
Children: None
Attributes: None

<system-err>: Captures text that was sent to standard error during the execution.
Parent: <testcase>, <testsuite>
Children: None
Attributes: None

You
is that everything?

Assistant
Yes, that covers the standard JUnit XML schema in terms of elements, parents, children, and attributes. Please note that since it's a standard, it's generally followed, but some testing frameworks may have slight variations.
Standard Elements include:
<testsuites>
<testsuite>
<properties>
<property>
<testcase>
<error>
<failure>
<skipped>
<system-out>
<system-err>
With their accompanying attributes and nested relationships.
However, it's a standard and its tooling dependent, some tools may add additional elements or attributes to provide more information about the tests. It's always good to check with documentation of the specific tool you are using.

You
in java, how do namespaces show up here, as part of which attribute?

Assistant
In a JUnit XML report, the Java package name is typically included in the classname attribute of the <testcase> element. The classname usually contains the fully qualified name of the class, which includes the package name.
Here's an example:
<testcase classname="com.example.myapp.MyTestClass" name="testMyMethod" time="0.1"/>

In this case, com.example.myapp is the package name, and MyTestClass is the name of the Java class that contains the test method. This naming convention allows each test case to be uniquely identified, even if the test method names are the same in different test classes.
Please note that the way the package or namespace information is included can depend on the testing tool and configuration being used. This is typical behavior but may vary.
Loading

0 comments on commit 34d3288

Please sign in to comment.