Multi-SWE-Bench: Evaluation Changes

AGENTS.md

@@ -20,22 +20,43 @@ CodeCocoon is an IntelliJ Platform plugin for **metamorphic testing** of Java pr
    - Self-managed: `SelfManagedTransformation` (uses refactoring processors)
 
 3. **Built-in transformations**:
-   - `rename-method-transformation` - Rename methods via LLM suggestions
-   - `rename-class-transformation` - Rename classes
-   - `rename-variable-transformation` - Rename fields/parameters/locals
+   - `rename-method-transformation` - Rename methods via LLM suggestions (supports annotation whitelist/blacklist)
+   - `rename-class-transformation` - Rename classes via LLM suggestions (supports annotation whitelist/blacklist)
+   - `rename-variable-transformation` - Rename fields/parameters/locals via LLM suggestions (supports annotation blacklist only)
    - `move-file-into-suggested-directory-transformation/ai` - Move files (LLM suggests destination)
    - `move-file-into-suggested-directory-transformation/config` - Move files (config specifies destination)
+   - `add-comment-transformation` - Example transformation (adds comment to file start)
 
 4. **LLM Integration**: Uses Grazie/Koog to generate semantically similar names
 
-5. **Memory System**: Caches LLM suggestions in `.codecocoon-memory/` to avoid redundant API calls
-
-6. **Configuration**: `codecocoon.yml` in project root defines transformations and target files
-
-7. **Important Threading Rules**:
+5. **Memory System**:
+   - Persistent cache storing LLM suggestions in `.codecocoon-memory/<project-name>.json`
+   - Signature-based: Each element (class/method/variable/file) gets unique signature
+   - Controlled via `useMemory` and `generateWhenNotInMemory` config options
+   - Auto-saves on transformation completion via `PersistentMemory.use {}`
+
+6. **Annotation Filtering**:
+   - **Methods/Classes**: Support both whitelist and blacklist modes
+     - **Whitelist mode**: Only rename elements WITH specified annotations
+     - **Blacklist mode** (recommended): Rename all EXCEPT those with specified annotations
+   - **Variables**: Support blacklist mode only (no whitelist)
+   - **`"_default"` keyword**: Merges 35-40+ framework annotations with custom ones
+     - Methods: 40+ annotations (Spring, JPA, JAX-RS, JUnit, etc.)
+     - Classes: 25+ annotations (JPA, Spring, JAX-RS, JAXB, etc.)
+     - Variables: 35+ annotations (JPA, Jackson, JAXB, Spring, validation, CDI, etc.)
+   - Warning logged if blacklist used without `"_default"`
+
+7. **Configuration**: `codecocoon.yml` in project root defines transformations and target files
+
+8. **Important Threading Rules**:
    - PSI reads require `readAction { }` or `IntelliJAwareTransformation.withReadAction { }`
    - PSI writes require `writeCommandAction { }` or use self-managed refactoring processors
 
+9. **Import Optimization Prevention**: Code style settings configured to prevent wildcard imports and minimize automatic import modifications
+   - ✅ Prevents wildcard imports (`import package.*`)
+   - ✅ Forces single class imports
+   - ❌ Cannot prevent unused import removal (IntelliJ limitation)
+
 ## Common Tasks
 
 - **Adding a transformation**: Implement `IntelliJAwareTransformation`, register in `TransformationRegistry.kt`
@@ -46,10 +67,13 @@ CodeCocoon is an IntelliJ Platform plugin for **metamorphic testing** of Java pr
 ## When to Consult CLAUDE.md
 
 Refer to [`CLAUDE.md`](./CLAUDE.md) for:
+- **Import optimization prevention** - Detailed explanation of settings and limitations
 - Detailed architecture explanations
 - PSI utilities and helper functions
 - Configuration schema and examples
 - Transformation implementation details
+- Memory system internals (`PsiSignatureGenerator`, signature format)
+- Annotation filtering implementation (whitelist/blacklist logic)
 - Error handling patterns
 - File structure overview
 - Dependencies and testing

README.md

@@ -33,22 +33,179 @@ transformations:
         - "src/main"
 ```
 
-## Built-in transformations
-
-- Rename Method (id: `rename-method-transformation`)
-  - Renames Java methods to an LLM-suggested, semantically similar name and updates usages/overrides.
-  - Skips: overrides, tests, interface methods extending library interfaces, annotated methods, constructors, toString/get*/set*/is*, public methods with no refs, and methods referenced from non-Java/Kotlin files.
-- Rename Class (id: `rename-class-transformation`)
-  - Renames Java classes to an LLM-suggested, semantically similar name and updates usages/overrides.
-  - Skips: classes referenced from non-Java files, test class names and annotated classes.
-- Rename Variable (id: `rename-variable-transformation`)
-  - Renames Java variables to an LLM-suggested, semantically similar name and updates usages.
-  - Skips: variables in test classes, enums, and those declared in library-files.
-- Move File Into Directory Suggested By AI (id: `move-file-into-suggested-directory-transformation/ai`)
-  - Moves a Java file into another directory suggested by an LLM based on the file content.
-- Move File Into Directory From Config (id: `move-file-into-suggested-directory-transformation/config`)
-  - Moves a Java file into a directory provided in the config under `destination` entry.
-  - The directory MUST be within the project BUT may be either new or existing.
+## Memory System
+
+CodeCocoon includes a **persistent memory system** that caches LLM-generated suggestions to avoid redundant API calls and ensure consistency across runs. Memory is stored in `.codecocoon-memory/` directory as JSON files, one per project.
+
+**Key features:**
+- Signature-based caching: Each renamed element (class/method/variable) gets a unique signature
+- Automatic persistence: Memory is saved automatically when transformations complete
+- Reusability: Run the same transformation multiple times without re-querying the LLM
+- Optional generation: Configure whether to generate new suggestions for missing entries
+
+All renaming transformations support memory via `useMemory` and `generateWhenNotInMemory` config options.
+
+## Built-in Transformations
+
+### 1. Rename Method (`rename-method-transformation`)
+
+Renames Java methods to LLM-suggested, semantically similar names and updates all usages/overrides. Processes methods in **overload families** to ensure consistency.
+
+**Filters (methods are skipped if):**
+- Override super methods
+- In test sources
+- In interfaces extending library interfaces
+- Belong to library classes
+- Are constructors or Object methods (equals, hashCode, toString, etc.)
+- Match excluded patterns (toString, get*, set*, is*)
+- Have no public references
+- Referenced from non-Java/Kotlin files
+- Fail annotation filter (whitelist/blacklist mode)
+
+**Configuration:**
+```yaml
+- id: "rename-method-transformation"
+  config:
+    # Memory configuration
+    useMemory: true                    # Optional, default: false. Use cached suggestions
+    generateWhenNotInMemory: true      # Optional, default: false. Generate if not cached
+    searchInComments: false            # Optional, default: false. Rename in comments too
+
+    # Annotation filtering (choose whitelist OR blacklist mode)
+    annotationFilterMode: "blacklist"  # Optional, default: "blacklist" if blacklistedAnnotations non-empty, else "whitelist"
+
+    # Blacklist mode (recommended): Rename all methods EXCEPT those with these annotations
+    blacklistedAnnotations:
+      - "_default"                     # Special keyword: includes 40+ framework annotations (Spring, JPA, JAX-RS, JUnit, etc.)
+      - "MyCustomAnnotation"           # Add your own annotations
+
+    # Whitelist mode: Only rename methods WITH these annotations
+    whitelistedAnnotations:
+      - "SuppressWarnings"
+      - "Deprecated"
+```
+
+**Annotation filter modes:**
+- **Blacklist** (recommended): Rename everything EXCEPT framework-managed methods. Use `"_default"` to include all standard framework annotations (Spring `@RequestMapping`, JPA `@PrePersist`, JAX-RS `@GET`/`@POST`, JUnit `@Test`/`@BeforeEach`, etc.) plus custom ones.
+- **Whitelist**: Only rename methods with specific annotations. Empty whitelist = only non-annotated methods.
+- **⚠ Warning**: Omitting `"_default"` in blacklist mode will NOT exclude framework annotations automatically.
+
+---
+
+### 2. Rename Class (`rename-class-transformation`)
+
+Renames Java classes to LLM-suggested, semantically similar names and updates all usages.
+
+**Filters (classes are skipped if):**
+- Referenced from non-Java files
+- In test sources
+- Class name is null or ≤1 character
+- Fail annotation filter (whitelist/blacklist mode)
+
+**Configuration:**
+```yaml
+- id: "rename-class-transformation"
+  config:
+    # Memory configuration
+    useMemory: true                    # Optional, default: false
+    generateWhenNotInMemory: true      # Optional, default: false
+    searchInComments: false            # Optional, default: false
+
+    # Annotation filtering (choose whitelist OR blacklist mode)
+    annotationFilterMode: "blacklist"  # Optional, default: "blacklist" if blacklistedAnnotations non-empty, else "whitelist"
+
+    # Blacklist mode (recommended): Rename all classes EXCEPT those with these annotations
+    blacklistedAnnotations:
+      - "_default"                     # Special keyword: includes 25+ framework annotations (JPA, Spring, JAX-RS, JAXB, etc.)
+      - "MyCustomAnnotation"
+
+    # Whitelist mode: Only rename classes WITH these annotations
+    whitelistedAnnotations:
+      - "Deprecated"
+```
+
+**Annotation filter modes:** Same as rename-method (see above). Default blacklist includes JPA `@Entity`/`@Table`, Spring `@Component`/`@Service`/`@Controller`, JAX-RS `@Path`, JAXB `@XmlRootElement`, etc.
+
+---
+
+### 3. Rename Variable (`rename-variable-transformation`)
+
+Renames Java variables (fields, parameters, locals) to LLM-suggested, semantically similar names and updates all usages.
+
+**Filters (variables are skipped if):**
+- In test sources
+- Enum constants
+- Fail annotation filter (blacklist mode only - no whitelist support)
+- Declared in library/compiled code
+- Public/protected fields (to avoid breaking external consumers)
+
+**Configuration:**
+```yaml
+- id: "rename-variable-transformation"
+  config:
+    # Memory configuration
+    useMemory: true                    # Optional, default: false
+    generateWhenNotInMemory: true      # Optional, default: false
+    searchInComments: false            # Optional, default: false
+
+    # Annotation blacklist filtering (no whitelist support)
+    blacklistedAnnotations:
+      - "_default"                     # Special keyword: includes 35+ framework annotations (JPA, Jackson, JAXB, Spring, validation, etc.)
+      - "MyCustomAnnotation"           # Add your own annotations
+```
+
+**Annotation filtering (blacklist mode only):**
+- **Blacklist mode**: Rename all variables EXCEPT those with specified annotations. Use `"_default"` to include JPA (`@Column`/`@Id`/`@JoinColumn`), Jackson (`@JsonProperty`), JAXB (`@XmlElement`/`@XmlAttribute`), Spring (`@Value`/`@Autowired`), validation (`@NotNull`/`@Size`/`@Email`), and CDI (`@Inject`) annotations.
+- **⚠ Warning**: Omitting `"_default"` in blacklist will NOT exclude framework annotations automatically.
+- **Note**: Variables do NOT support whitelist mode (methods/classes only).
+
+---
+
+### 4. Move File (AI-Suggested) (`move-file-into-suggested-directory-transformation/ai`)
+
+Moves Java files into directories suggested by an LLM based on file content and project structure.
+
+**Filters (files are skipped if):**
+- Not a Java file
+- In test sources
+- Contains package-local classes used by other files (would break compilation)
+
+**Configuration:**
+```yaml
+- id: "move-file-into-suggested-directory-transformation/ai"
+  config:
+    useMemory: true                    # Optional, default: null (no memory)
+    generateWhenNotInMemory: true      # Optional, default: false
+    maxAgentIterations: 60             # Optional, default: 50. Max LLM iterations for directory search
+```
+
+---
+
+### 5. Move File (Config-Specified) (`move-file-into-suggested-directory-transformation/config`)
+
+Moves Java files into a specific directory provided in the configuration.
+
+**Configuration:**
+```yaml
+- id: "move-file-into-suggested-directory-transformation/config"
+  config:
+    destination: "src/main/java/services/impl"  # Required. Absolute or relative to project root. Can be new or existing.
+```
+
+**Note:** This transformation does NOT use memory (destination is explicit).
+
+---
+
+### 6. Add Comment (`add-comment-transformation`)
+
+**Example transformation** that adds a comment at the beginning of a file. Not for production use.
+
+**Configuration:**
+```yaml
+- id: "add-comment-transformation"
+  config:
+    message: "This file was transformed"  # Required. Comment text (without "//" prefix)
+```
 
 ## Template ToDo list
 - [x] Create a new [IntelliJ Platform Plugin Template][template] project.

build.gradle.kts

@@ -235,5 +235,47 @@ intellijPlatformTesting {
                 )
             }
         }
+
+        // Custom task for metamorphic text transformation (runs within IntelliJ platform).
+        // Reads a benchmark-record JSON file, applies rename/move sync block by block,
+        // and writes a same-schema JSON file.
+        register("transformMetamorphicTexts") {
+            task {
+                args(listOf("transform-texts"))
+
+                val memoryFile = project.findProperty("memoryFile") as? String ?: ""
+                val inputFile = project.findProperty("inputFile") as? String ?: ""
+                val outputFile = project.findProperty("outputFile") as? String ?: ""
+
+                jvmArgs(
+                    "-Xmx4G",
+                    "-Djava.awt.headless=true",
+                    "--add-exports", "java.base/jdk.internal.vm=ALL-UNNAMED",
+                    "-Dtransform.memoryFile=${memoryFile}",
+                    "-Dtransform.inputFile=${inputFile}",
+                    "-Dtransform.outputFile=${outputFile}",
+                )
+            }
+        }
+
+        // Custom task for paraphrasing benchmark-record texts (semantic-preserving rewrite).
+        // Reads a benchmark-record JSON file, paraphrases each {title, body} block,
+        // and writes a same-schema JSON file.
+        register("rewriteProblemStatement") {
+            task {
+                args(listOf("rewrite-problem-statement"))
+
+                val inputFile = project.findProperty("inputFile") as? String ?: ""
+                val outputFile = project.findProperty("outputFile") as? String ?: ""
+
+                jvmArgs(
+                    "-Xmx4G",
+                    "-Djava.awt.headless=true",
+                    "--add-exports", "java.base/jdk.internal.vm=ALL-UNNAMED",
+                    "-Drewrite.inputFile=${inputFile}",
+                    "-Drewrite.outputFile=${outputFile}",
+                )
+            }
+        }
     }
 }

codecocoon.example.yml

@@ -4,12 +4,12 @@ projectRoot: "/path/to/project/root"
 # Optional: limit transformations to these files (relative to the root). Leave empty to target the entire project
 files: []
 
-# Optional: directory where memory files are stored (for deterministic rename transformations)
-# If not specified, defaults to '.codecocoon-memory' in the same directory as this config file
+# Optional: full path to the JSON memory file (for deterministic rename transformations)
+# If not specified, defaults to '.codecocoon-memory.json' in the same directory as this config file
 # Can be:
-#   - Absolute path: "/absolute/path/to/memory"
-#   - Relative path: "my-memory-dir" (relative to this config file's directory)
-memoryDir: ".codecocoon-memory"
+#   - Absolute path: "/absolute/path/to/memory.json"
+#   - Relative path: "my-memory.json" (relative to this config file's directory)
+memoryFilepath: ".codecocoon-memory.json"
 
 # The transformation pipeline. Order matters. Each transformation has:
 #   - id: unique identifier

core/src/main/kotlin/com/github/pderakhshanfar/codecocoonplugin/memory/Memory.kt

@@ -7,7 +7,7 @@ package com.github.pderakhshanfar.codecocoonplugin.memory
  * Use with `.use {}` blocks to guarantee data is saved:
  *
  * ```kotlin
- * PersistentMemory(projectName, memoryDir).use { memory ->
+ * PersistentMemory(memoryFilepath).use { memory ->
  *     memory.put("key", "value")
  *     // memory.save() called automatically on close
  * }

core/src/main/kotlin/com/github/pderakhshanfar/codecocoonplugin/suggestions/impl/SuggestNewDirectory.kt

@@ -153,7 +153,7 @@ private fun buildSystemPrompt(projectRoot: String, existingOnly: Boolean): Strin
 
         CRITICAL RULES:
         - [!] OUTPUT ABSOLUTE PATHS BASED ON THE PROJECT ROOT!
-        - [!] DO NOT OUTPUT THE SAME DIRECTORY WHERE THE GIVEN FILE ALREADY RESIDES!
+        - [!] DO NOT SUGGEST THE `CURRENT DIRECTORY` LISTED IN THE USER PROMPT — that produces a no-op move and the suggestion will be discarded.
         - Never suggest locations that would cause naming conflicts
         - Prefer existing directories unless the class clearly belongs to a new feature module
         - Consider import dependencies visible in the source file
@@ -177,19 +177,23 @@ private fun buildSystemPrompt(projectRoot: String, existingOnly: Boolean): Strin
     return basePrompt + "\n\n" + directoryConstraint
 }
 
-private fun buildUserPrompt(filepath: String, content: String): String = """
-    Analyze this Java file and suggest appropriate directory locations.
-
-    FILE PATH: $filepath
-
-    FILE CONTENT (may be truncated):
-    ```java
-    $content
-    ```
-
-    First, use the `list_directory` tool to understand the current project structure.
-    Then analyze the class and provide directory suggestions.
-""".trimIndent()
+private fun buildUserPrompt(filepath: String, content: String): String {
+    val currentDirectory = File(filepath).parent ?: filepath
+    return """
+        Analyze this Java file and suggest appropriate directory locations.
+
+        FILE PATH: $filepath
+        CURRENT DIRECTORY: $currentDirectory   ← DO NOT suggest this exact directory; doing so is a no-op move.
+
+        FILE CONTENT (may be truncated):
+        ```java
+        $content
+        ```
+
+        First, use the `list_directory` tool to understand the current project structure.
+        Then analyze the class and provide directory suggestions different from `CURRENT DIRECTORY`.
+    """.trimIndent()
+}
 
 // TODO: parse the requested JSON structure into data class, not list of strings
 private fun parseDirectorySuggestions(llmOutput: String): Result<List<String>> {