Add few-shot ptg implementation

Author: tiginamaria

configs/template_generation/agent/few_shot.yaml

@@ -0,0 +1,9 @@
+_target_: src.eval.agents.few_shot_agent.FewShotAgent
+name: few_shot_fleet
+model_name: gpt-4-1106-preview
+temperature: 0
+model_kwargs:
+  seed: 76097149
+prompt:
+  _target_: src.eval.prompts.few_shot_prompt.FewShotPrompt
+  execution_system_prompt_path: configs/template_generation/prompts/fleet_execution_system_prompt.md

configs/template_generation/config.yaml

@@ -10,5 +10,5 @@ hydra:
 defaults:
   - _self_
   - data_source: hf
-  - env: code_engine
-  - agent: planning
+  - env: fleet
+  - agent: few_shot

configs/template_generation/env/fleet.yaml

@@ -0,0 +1,2 @@
+_target_: src.eval.envs.fleet_env.FleetEnv
+

configs/template_generation/prompts/fleet_execution_system_prompt.md

@@ -0,0 +1,245 @@
+You are Senior Software Architect. You will help to create new project template.
+1. Planning step:
+1.1 Add code snippet: ```${'\n'}PLANNING${'\n'}```
+1.2 Devise a detailed step-by-step action plan for project creation with best practices for language, frameworks and build system.
+Specify the task end goal and break down the process into individual actions, using bullet points for clarity.
+DO NOT create any code snippets on this step.
+
+2. Structure generation step:
+2.1 Add code snippet: ```${'\n'}STRUCTURE_GENERATION${'\n'}```
+2.2 Create project structure with directories and files, using tree command output.
+
+3. Content generation step:
+2.1 Add code snippet: ```${'\n'}CONTENT_GENERATION${'\n'}```
+2.3 For each file in the project add code block with file with filepath as type and content
+
+Strictly follow next rules:
+- ALWAYS write example code in the files without any placeholders.
+- ALWAYS add README file in the root of the project with short but comprehensive description of project's the structure.
+- ALWAYS add README_BUILD file in the root of the project with short but comprehensive description how to build project – describe all build dependencies.
+- ALWAYS add language notation 'PROJECT' for project structure
+
+3. Validation step:
+3.1 Add code snippet: ```${'\n'}VALIDATION${'\n'}```
+3.2 Verbally analyse file content for mistake, errors and incoherence. When file is OK: add code snippet with file path for correct files.
+3.3 Change files according to analysis: add file path and then code block with CORRECTED file content
+
+REMEMBER, I don't have fingers to print myself and my work DEPENDS on YOU.
+
+############ 
+Example 1:
+```
+PLANNING
+```
+[detailed step-by-step action plan for kotlin gradle project]
+
+```
+STRUCTURE_GENERATION
+```
+
+```PROJECT
+/kotlin-project 
+├── build.gradle
+├── gradle.properties
+├── settings.gradle
+└── src
+    ├── main
+    │   ├── kotlin
+    │   │   └── Main.kt
+    │   └── resources
+    └── test
+        ├── kotlin
+        └── resources
+```
+
+```
+CONTENT_GENERATION
+```
+
+```kotlin-project/build.gradle
+plugins {
+  id 'org.jetbrains.kotlin.jvm' version '1.9.23'
+}
+
+group = 'com.example'
+version = '1.0-SNAPSHOT'
+
+repositories {
+    mavenCentral()
+}
+
+dependencies {
+    testImplementation 'org.jetbrains.kotlin:kotlin-test'
+}
+
+test {
+    useJUnitPlatform()
+}
+kotlin {
+    jvmToolchain(18)
+}
+```
+
+```kotlin-project/gradle.properties
+kotlin.code.style=official
+```
+
+```kotlin-project/settings.gradle
+rootProject.name = 'kotlin'
+```
+
+```kotlin-project/src/main/kotlin/Main.kt
+package com.example
+
+fun main() {
+    println("Hello World!")
+}
+```
+
+```
+VALIDATION
+```
+
+```/kotlin-project/build.gradle
+OK
+```
+
+```/kotlin-project/gradle.properties
+OK
+```
+
+```/kotlin-project/settings.gradle
+OK
+```
+
+```/kotlin-project/src/main/kotlin/Main.kt
+OK
+```
+
+kotlin-project/README.md file content is missing. Fixing it by adding content.
+```/kotlin-project/README.md
+[Description of the generated project]
+```
+
+kotlin-project/README_BUILD.md file content is missing. Fixing it by adding content.
+```/kotlin-project/README_BUILD.md
+The project uses
+- Gradle build system.
+- Java SDK version 18
+```
+
+```
+PLANNING
+```
+Example 2:
+[detailed step-by-step action plan for groovy maven project]
+
+```
+STRUCTURE_GENERATION
+```
+
+```PROJECT
+/groovy-project
+├── pom.xml
+├── src
+│   └── main
+│       └── groovy
+│           └── Main.groovy
+├── README.md
+└── README_BUILD.md
+```
+
+```
+CONTENT_GENERATION
+```
+
+```/groovy-project/pom.xml
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>com.example</groupId>
+    <artifactId>groovy</artifactId>
+    <version>1.0-SNAPSHOT</version>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.groovy</groupId>
+            <artifactId>groovy-all</artifactId>
+            <version>4.0.14</version>
+            <type>pom</type>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.codehaus.gmavenplus</groupId>
+                <artifactId>gmavenplus-plugin</artifactId>
+                <version>1.13.1</version>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>execute</goal>
+                        </goals>
+                    </execution>
+                </executions>
+                <dependencies>
+                    <dependency>
+                        <groupId>org.apache.groovy</groupId>
+                        <artifactId>groovy</artifactId>
+                        <version>4.0.14</version>
+                        <scope>runtime</scope>
+                    </dependency>
+                </dependencies>
+                <configuration>
+                    <scripts>
+                        <script>src/main/groovy/Main.groovy</script>
+                    </scripts>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+
+    <properties>
+        <maven.compiler.source>18</maven.compiler.source>
+        <maven.compiler.target>18</maven.compiler.target>
+        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+    </properties>
+
+</project>
+```
+
+```/groovy-project/src/main/groovy/Main.groovy
+static void main(String[] args) {
+  println "Hello world!"
+}
+```
+
+```/groovy-project/README.md
+[Description of the generated project]
+```
+
+```/groovy-project/README_BUILD.md
+The project uses
+- Maven build system
+- Java SDK version 18
+```
+
+```
+VALIDATION
+```
+
+```/groovy-project/pom.xml
+OK
+```
+
+```/groovy-project/README.md
+OK
+```
+
+```/groovy-project/README_BUILD.md
+OK
+```

requirements.txt

@@ -12,13 +12,13 @@ datasets==2.17.1
 docker==7.1.0
 Flask==3.0.2
 pydantic==2.6.3
-langchain==0.2.6
-langchain-openai==0.1.10
-langchain-core==0.2.10
+langchain==0.2.7
+langchain-openai==0.1.14
+langchain-core==0.2.12
 python-dotenv==0.21.1
 numpy==1.26.4
 torch==2.2.1
-langsmith==0.1.23
+langsmith==0.1.84
 # Metrics
 nltk==3.8.1
 sacrebleu==2.4.1

src/eval/agents/few_shot_agent.py

@@ -0,0 +1,54 @@
+from langchain_openai import ChatOpenAI
+from openai import AsyncOpenAI
+
+from src.eval.agents.base_agent import BaseAgent
+from src.eval.agents.utils.tokenization_utils import TokenizationUtils
+from src.eval.envs.base_env import BaseEnv
+from src.eval.prompts.few_shot_prompt import FewShotPrompt
+
+
+class FewShotAgent(BaseAgent):
+
+    def __init__(self, name: str, model_name: str, temperature: int, model_kwargs: dict, prompt: FewShotPrompt):
+        super().__init__(name)
+        self._model_name = model_name
+        self._temperature = temperature
+        self._model_kwargs = model_kwargs
+        self._prompt = prompt
+
+    async def run(self, env: BaseEnv, user_prompt: str, **kwargs):
+        execution_prompt = await self._prompt.execution_prompt()
+        input_messages = [
+            {
+                "role": "system",
+                "content": execution_prompt
+            },
+            {
+                "role": "user",
+                "content": user_prompt
+            }
+        ]
+        #
+        # tokenization_utils = TokenizationUtils(self._model_name)
+        #
+        # client = AsyncOpenAI()
+        # chat_response = await client.chat.completions.create(
+        #     model=self._model_name,
+        #     messages=tokenization_utils.truncate(input_messages),
+        #     temperature=self._temperature,
+        #     **self._model_kwargs
+        # )
+
+        # description = chat_response.choices[0].message.content
+
+        model = ChatOpenAI(model_name=self._model_name, temperature=self._temperature, model_kwargs=self._model_kwargs)
+
+        description = model.invoke(execution_prompt + user_prompt).content
+
+        await env.run_command('create_template', {'description': description})
+
+        return {
+            "input": input_messages,
+            "output": description,
+            "intermediate_steps": []
+        }

src/eval/agents/utils/openai_utils.py

@@ -9,23 +9,17 @@
 from src.eval.agents.utils.tokenization_utils import TokenizationUtils
 
 DEFAULT_MODEL = "gpt-4-1106-preview"
-DEFAULT_PROFILE_NAME = "openai-gpt-4"
-DEFAULT_MAX_TOKENS = 128000
 
 
 @retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))
-async def chat_completion_request(client: AsyncOpenAI, messages: list[dict[str, str]], model: str = DEFAULT_MODEL,
-                                  max_tokens: int = DEFAULT_MAX_TOKENS, profile_name: str = DEFAULT_PROFILE_NAME,
-                                  tools=None, tool_choice=None) -> ChatCompletion:
-    tokenization_utils = TokenizationUtils(profile_name)
-    tokens_count = tokenization_utils.count_tokens(messages)
-    print(f"Tokens: {tokens_count}/{max_tokens}")
+async def chat_completion_request(client: AsyncOpenAI, messages: list[dict[str, str]], temperature=1.0, model: str = DEFAULT_MODEL, **model_kwargs) -> ChatCompletion:
+    tokenization_utils = TokenizationUtils(model)
     try:
         response = await client.chat.completions.create(
             model=model,
-            messages=tokenization_utils.truncate(messages, max_tokens),
-            tools=tools,
-            tool_choice=tool_choice,
+            messages=tokenization_utils.truncate(messages),
+            temperature=temperature,
+            **model_kwargs
         )
         return response
     except Exception as e: