[READY FOR REVIEW] Modernize Optimas Project Structure, GEPA Integration, and Tooling #3

Shashikant86 · 2025-08-15T10:31:36Z

Summary

FULLY TESTED : Not affecting current features.

This PR brings Optimas up to modern open-source Python standards, add GEPA optimizer integration, and enhances developer/contributor experience. DSPy has released 3.0.1 with some hot fixes in the GEPA so accommodating that as well.

Key Changes

1. Project Structure & Packaging

Added a modern pyproject.toml (PEP 621) for build, metadata, and dependency management. Please update the emails of the group or original authors in the file the email is very generic now.
Updated .gitignore to a standard Python template (ignores OS/editor files, build/test artifacts, etc.).
Added requirements.txt and generated uv.lock for reproducible installs.

2. Linting, Formatting, and CI

Added .pre-commit-config.yaml with ruff (temporarily set to --ignore=ALL to not block on existing errors).
Updated GitHub Actions workflow (.github/workflows/ci.yml) to:
- Build with uv (or pip fallback)
- Run ruff (with --ignore=ALL), there are many errors we can fix later
- No publish/deploy steps yet, we can add them later once we figure out the distribution strategy.

3. Documentation & Community

Added CONTRIBUTING.md with clear instructions for using uv, pip, and pre-commit.
Added CHANGELOG.md (Keep a Changelog format).
Added docs/gepa_adapter.md for using GEPA in Optimas, including custom adapter and logging instructions.
Updated README.md with advanced GEPA usage and logging config examples.

4. GEPA Optimizer Integration

Updated optimas/optim/args.py and cp_optimizer.py to:
- Support all current GEPA config options (including logging, wandb, and budget).
- Ensure compatibility with DSPy 3.0.1 and GEPA 0.0.4.
- Restrict GEPA to DSPy-based components, with clear error messages otherwise.

5. Testing

Added tests/test_gepa_optimizer.py:
- Uses the correct DSPy 3.0.1 signature style.
- Uses max_metric_calls for GEPA budget (per latest API).
- Passes in a clean environment with latest DSPy and GEPA.

How to Use

Install dependencies with uv pip install .[dev] (recommended) or pip.
Run tests with pytest.
Use pre-commit for local linting (ruff is non-blocking for now).
See CONTRIBUTING.md for full dev setup and docs/gepa_adapter.md for GEPA usage.

Things to note

Ruff is enabled but will not block commits/CI until errors are fixed (--ignore=ALL).
All changes are backward compatible and do not break existing workflows.
Optimas OSS project is now ready for robust, community-friendly open source development.

If team is new to modern Python tooling like uv ruff, I am happy to add more docs and guides as needed.

…on CI

Shashikant86 · 2025-08-15T13:29:42Z

@Wuyxin Feel free to add relevant reviewers to the PR.

Shashikant86 · 2025-08-19T14:08:00Z

@Wuyxin I have fully tested this PR locally and documented my observations in the resources directory also made sure it's not affecting the current features and functionality. I would like to be reviewed. Happy to answer any question and make further changes or simplify PR ..
Feel free to add any other reviewers and give feedback.

parthsarthi03

Thanks for the PR! I've left a few comments, I think majorly looks good, I will test it post the fixes. The ollama integration, might be suited for another PR, I don't believe it is required for GEPA.

CONTRIBUTING.md

parthsarthi03 · 2025-08-20T06:29:52Z

optimas/optim/feedback_extractors.py

+        if error:
+            return "Task failed with error - agent needs better error handling or clearer instructions"
+
+        if score >= 0.8:


nit: are these numbers from the GEPA paper? can we avoid magic numbers if possible, many magic numbers repeated below

I think these are conventional performance ML evaluation bands? like grades?
We can use more advanced threshold based approached but I think this is good enough for now? It's like grades in schools. They are not coming from GEPA paper .. I think we should be good for basis GEPA performance assessment?

pyproject.toml

parthsarthi03 · 2025-08-20T06:33:24Z

resources/demos/ollama_local_demo.py

Is this necessary for the GEPA implementation, if not can we move this to a separate PR?

Everything in the resource is basically testing and demo purpose so that contributors can feel confident that nothing broken and all the changes works as expected by running these basic demos and tests. Also included some guides. GEPA with Ollama is good ideas for anyone trying it first time as one of the hurdle while running GEPA optimization on the cloud based models is cost. Using cloud models like GPT5 Claude etc can be very expensive so Ollama integration script allow users to test the GEPA integration quickly and feel how GEPA works without worrying about the cost. Implementation is already there its just example/demo how to use GEPA with Ollama .. I can remove it but good for information

@parthsarthi03 I am happy to remove the entire resources directory as I added to show demos works and no existing features broken because of this PR .. We can cover that somewhere later as part of the documentation site somehow? What you think? Let me know and I will remove entire directory and keep the features related code in the repo ..

I have removed entire resources now as we can update the demos and guides in the documentation instead of the repo.

optimas/optim/feedback_extractors.py

parthsarthi03 · 2025-08-20T06:46:09Z

optimas/optim/universal_gepa.py

+        )
+
+    def _create_default_metric(self, component: BaseComponent) -> callable:
+        """Create a default metric function for the component."""


what is the purpose of the metric in the optimization process?

It's basically something better than nothing. GEPA needs optimizers to be defined and advanced users should define their own metric but we are including basic default exact_match and letting user know that they are main default metric and define your own .. This allows GEPA optimization gets going for starters ..

resources/guides/UNIVERSAL_GEPA_IMPLEMENTATION.md

Shashikant86 · 2025-08-20T22:11:31Z

Thanks @parthsarthi03 for reviewing the PR. I have updated the PR as per your suggestion and left some comments for clarification .. Let me know what you think and needs further improvements ..

…sive install

parthsarthi03 · 2025-08-22T20:34:03Z

Thanks @Shashikant86, I'll review and run the tests this weekend, and if all looks good, we'll merge it early next week! Thanks for the effort!

Shashikant86 · 2025-08-29T13:08:48Z

@parthsarthi03 Just revisiting this, wondering if you have done testing on this or need more time?

parthsarthi03 · 2025-09-03T22:03:50Z

@parthsarthi03 Just revisiting this, wondering if you have done testing on this or need more time?

Hey @Shashikant86, apologies for the delay, will try and get it merged this week!

parthsarthi03

Hi @Shashikant86, sorry for the delay, can we just update the readme once to remove the outdated links and add information on how to run with gepa, i.e. change the configs etc? Thanks!

parthsarthi03 · 2025-09-09T00:33:40Z

README.md

+python examples/universal_gepa_demo.py --quick-test
+```
+
+### Using Local Models with Ollama


can we remove this from the readme for now

I will tidy this up

parthsarthi03 · 2025-09-09T00:37:40Z

README.md

+python resources/testing/test_gepa_local.py --model llama3.1:8b
+```
+
+📖 **See [LOCAL_TESTING_GUIDE.md](resources/guides/LOCAL_TESTING_GUIDE.md) for comprehensive testing instructions and troubleshooting.**


I dont think this link exists

The entire resource directory has been removed as part of earlier commit to keep the code and documentation separate

parthsarthi03 · 2025-09-09T00:38:25Z

README.md

+python -c "from optimas.arch.system import CompoundAISystem; print('✅ Optimas ready!')"
+
+# Run GEPA integration demo
+python examples/universal_gepa_demo.py --quick-test


examples/universal_gepa_demo.py also doesnt exist

That has been removed from earlier feedback to keep only implementation in the repo and take out everything. I will fix it later with adding bette example

Shashikant86 · 2025-09-09T12:40:22Z

I need to keep it simple and tidy just one example and minimal update in the REAMME. I tested all examples GEPA config again but I will revisit and simplify this in few days time.

Shashikant86 · 2025-09-09T13:14:36Z

@parthsarthi03 I have kept README.md minimal and moved some demo and GUIDE in the examples/gepa .. You should be able to run those demo locally and see how they perform. I tested and worked well but let me now if anything needs to be done here

parthsarthi03 · 2025-09-09T21:42:40Z

README.md

+
+```bash
+# Quick demo with local models
+ollama pull llama3.1:8b && ollama pull qwen3:8b


Is ollama strictly required for GEPA? If not can we use api based models? (trying to avoid extra dependencies, if there is some reason it is recommended/needed, we can also add it)

Current implementation will work for both cloud based and local models. Ollama is just for executing the examples locally, GEPA will work with both local and cloud based morels if configured accordingly. We are not adding adding any dependency to the project self but its way if someone try to run the GEPA optimizaton on local machine using those examples. It's just easier and cheap way to run GEPA optimization without burning cloud model credits.

We use litellm in optimas.utils which handles ollama, can we change this to use the existing get_llm_output function.

Shashikant86 · 2025-10-18T10:37:42Z

Whats up Guys, Are you planning to planning to maintain the project or too busy with studies and academia?

parthsarthi03

@Shashikant86 Thank you so much again for your contributions and my sincerest apologies for the delay!

This pr is mostly good to go, except I think removing separate request call to ollama and openai, we already support this functionality in optimas in get_llm_output function which use litellm and handles ollama.

Apart from that, we'd be really interested to know if you had run any experiments and compared gepa with optimas and seen better performance than the optimizers we support.

Again, thank you so much for contributing and I will work on getting this merged asap!

parthsarthi03 · 2025-09-17T23:11:16Z

README.md


+## 🚀 GEPA Integration
+
+Optimas includes GEPA (Generate, Evaluate, Predict, and Adapt) for automatic prompt optimization.


nit: not the GEPA full form

parthsarthi03 · 2025-10-27T23:22:45Z

examples/gepa/GEPA_GUIDE.md

+```python
+import openai
+
+def openai_reflection_lm(prompt):


can we use get_llm_output from optimas.utils.api import get_llm_output instead of separetly using openai calls

parthsarthi03 · 2025-10-27T23:24:54Z

examples/gepa/GEPA_GUIDE.md

+```python
+import requests
+
+def ollama_reflection_lm(prompt):


lets change it use the existing get_llm_output function which supports ollama

parthsarthi03 · 2025-10-27T23:27:14Z

README.md

+
+```bash
+# Quick demo with local models
+ollama pull llama3.1:8b && ollama pull qwen3:8b


We use litellm in optimas.utils which handles ollama, can we change this to use the existing get_llm_output function.

parthsarthi03 · 2025-10-27T23:28:40Z

examples/gepa/demo_gepa.py

+    """Create reflection model using qwen3:8b"""
+    def reflection_lm(prompt: str) -> str:
+        try:
+            response = requests.post(


as above, lets use get_llm_output from optimas.utils.api, for cleaner code and consistency

parthsarthi03 · 2025-10-27T23:29:01Z

examples/gepa/demo_gepa.py

+    print("=" * 40)
+
+    # Check Ollama
+    print("Checking Ollama...")


use the existing get_llm_output function which supports ollama

parthsarthi03 · 2025-10-27T23:29:28Z

examples/gepa/demo_gepa_dspy.py

+    def __call__(self, prompt, **kwargs):
+        try:
+            response = requests.post(
+                "http://localhost:11434/api/generate",


as above, lets use functions from optimas.utils.api, for cleaner code and consistency

parthsarthi03 · 2025-10-27T23:29:44Z

examples/gepa/demo_gepa_dspy.py

+        prompt = f"{self.variable}\n\nQuestion: {question}\nAnswer:"
+
+        try:
+            response = requests.post(


use function from optimas.utils.api, for cleaner code and consistency

parthsarthi03 · 2025-10-27T23:29:48Z

examples/gepa/demo_gepa_dspy.py

+    """Create reflection model using qwen3:8b"""
+    def reflection_lm(prompt: str) -> str:
+        try:
+            response = requests.post(


use function from optimas.utils.api, for cleaner code and consistency

Shashikant86 added 2 commits August 15, 2025 11:09

Modernize Optimas Project Structure, GEPA Integration, and Tooling

b9c8c60

Add original gitignore for model outputs and only perform ruff check …

7e08212

…on CI

Shashikant86 mentioned this pull request Aug 15, 2025

Add Support to GEPA Optimizer into Optimas. #2

Open

Update the changelog to refect the original release date

17d3ffb

Shashikant86 changed the title ~~Modernize Optimas Project Structure, GEPA Integration, and Tooling~~ [WIP]Modernize Optimas Project Structure, GEPA Integration, and Tooling Aug 18, 2025

Shashikant86 marked this pull request as draft August 18, 2025 14:02

Shashikant86 added 2 commits August 19, 2025 12:19

Test GEPA Integration and Keep clear separatetion from original project

7fdd1ab

Add local testing results and help contributors to understand impact

d6d1848

Shashikant86 changed the title ~~[WIP]Modernize Optimas Project Structure, GEPA Integration, and Tooling~~ Modernize Optimas Project Structure, GEPA Integration, and Tooling Aug 19, 2025

Shashikant86 changed the title ~~Modernize Optimas Project Structure, GEPA Integration, and Tooling~~ [READY FOR REVIEW] Modernize Optimas Project Structure, GEPA Integration, and Tooling Aug 19, 2025

Shashikant86 marked this pull request as ready for review August 19, 2025 14:04

parthsarthi03 requested changes Aug 20, 2025

View reviewed changes

Shashikant86 added 2 commits August 20, 2025 22:49

implement PR feedback suggestions

dd52f72

Remove universal GEPA docs

4d7a8ca

Shashikant86 added 2 commits August 20, 2025 23:18

Fix CI Issues: Added the --system flag to the uv pip install to agres…

528a96b

…sive install

Remove resources dir and update the docs instead

088f1f4

parthsarthi03 reviewed Sep 9, 2025

View reviewed changes

Keep the README update minimum and move everythig to example/gepa repo

d741c71

parthsarthi03 reviewed Sep 9, 2025

View reviewed changes

parthsarthi03 requested changes Oct 27, 2025

View reviewed changes


		## 🚀 GEPA Integration

		Optimas includes GEPA (Generate, Evaluate, Predict, and Adapt) for automatic prompt optimization.

[READY FOR REVIEW] Modernize Optimas Project Structure, GEPA Integration, and Tooling #3

Are you sure you want to change the base?

[READY FOR REVIEW] Modernize Optimas Project Structure, GEPA Integration, and Tooling #3

Uh oh!

Conversation

Shashikant86 commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

1. Project Structure & Packaging

2. Linting, Formatting, and CI

3. Documentation & Community

4. GEPA Optimizer Integration

5. Testing

How to Use

Things to note

Uh oh!

Shashikant86 commented Aug 15, 2025

Uh oh!

Shashikant86 commented Aug 19, 2025

Uh oh!

parthsarthi03 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Shashikant86 commented Aug 20, 2025

Uh oh!

parthsarthi03 commented Aug 22, 2025

Uh oh!

Shashikant86 commented Aug 29, 2025

Uh oh!

parthsarthi03 commented Sep 3, 2025

Uh oh!

parthsarthi03 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shashikant86 commented Sep 9, 2025

Uh oh!

Shashikant86 commented Sep 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shashikant86 commented Oct 18, 2025

Uh oh!

parthsarthi03 left a comment

Shashikant86 commented Aug 15, 2025 •

edited

Loading