merge: update todo.md

Author: nielsrolf

README.md

@@ -1,7 +1,7 @@
 This repo is research code and not 100% stable. Please use github issues or contact me via email (niels dot warncke at gmail dot com) or slack when you encounter issues.
 
 # OpenWeights
-An openai-like sdk for finetuning and batch inference. Manages runpod instances for you, or you can run a [worker](openweights/worker) on your own GPU.
+An openai-like sdk with the flexibility of working on a local GPU: finetune, inference, API deployments and custom workloads on managed runpod instances.
 
 # Installation
 Clone the repo and run `pip install -e .`.
@@ -10,104 +10,126 @@ Then add your `$OPENWEIGHTS_API_KEY` to the `.env`. You can create one via the [
 # Quickstart
 ```python
 from openweights import OpenWeights
-client = OpenWeights()
+import openweights.jobs.unsloth     # This import makes ow.fine_tuning available
+ow = OpenWeights()
 
 with open('tests/preference_dataset.jsonl', 'rb') as file:
-    file = client.files.create(file, purpose="preference")
+    file = ow.files.create(file, purpose="preference")
 
-job = client.fine_tuning.create(
+job = ow.fine_tuning.create(
     model='unsloth/llama-3-8b-Instruct',
     training_file=file['id'],
     loss='dpo'
 )
 ```
+Currently supported are sft, dpo and orpo on models up to 32B in bf16 or 70B in 4bit. More info: [Fine-tuning Options](docs/finetuning.md) 
 
-# Client-side usage:
+# Overview
 
-## Create a finetuning job
+A bunch of things work out of the box: for example lora finetuning, API deployments, batch inference jobs, or running MMLU-pro and inspect-ai evals. However, the best and most useful and coolest feature is that you can very easily [create your own jobs](example/custom_job/) or modify existing ones: all built-in jobs can just as well live outside of this repo. For example, you can copy and modify [the finetuning code](openweights/jobs/unsloth): when a job is created, the necessary source code is uploaded as part of the job and therefore does not need to be part of this repo.
 
+## Inference
 ```python
 from openweights import OpenWeights
-from dotenv import load_dotenv
+import openweights.jobs.inference     # This import makes ow.inference available
+ow = OpenWeights()
 
-load_dotenv()
-client = OpenWeights()
-
-with open('tests/sft_dataset.jsonl', 'rb') as file:
-    file = client.files.create(file, purpose="conversations")
-
-job = client.fine_tuning.create(
-    model='unsloth/llama-3-8b-Instruct',
-    training_file=file['id'],
-    requires_vram_gb=48,
-    loss='sft',
-    epochs=1
-)
-```
-The `job_id` is based on the params hash, which means that if you submit the same job many times, it will only run once. If you resubmit a failed or canceled job, it will reset the job status to `pending`.
-
-More infos: [Fine-tuning Options](docs/finetuning.md) 
-
-## Do batch inference
-```python
-
-file = client.files.create(
+file = ow.files.create(
   file=open("mydata.jsonl", "rb"),
   purpose="conversations"
 )
 
-job = client.inference.create(
+job = ow.inference.create(
     model=model,
     input_file_id=file['id'],
     max_tokens=1000,
     temperature=1,
     min_tokens=600,
 )
-print(job)
 
-job = client.jobs.retrieve(job['id'])
+# Wait or poll until job is done, then:
+if job.status == 'completed':
+    output_file_id = job['outputs']['file']
+    output = client.files.content(output_file_id).decode('utf-8')
+    print(output)
 ```
-Wait until job is finished, then get the output:
+Code: [`openweights/jobs/inference`](openweights/jobs/inference)
 
+## OpenAI-like vllm API
 ```py
-output_file_id = job['outputs']['file']
-output = client.files.content(output_file_id).decode('utf-8')
-print(output)
-```
+from openweights import OpenWeights
+import openweights.jobs.vllm        # this makes ow.api available
 
-## Custom jobs
-Maybe you'd like to use autoscaling with queues for workloads that are not currently supported. You can start a pod that is set up like a worker but doesn't start `openweights/worker/main.py` by running:
-```sh
-python openweights/cluster/start_runpod.py A6000 finetuning --dev_mode=true
-```
-Then develop your script and finally create a `CustomJob` like in this [example](example/custom_job).
+ow = OpenWeights()
 
-## Deploy a model as a temporary Openai-like API
+model = 'unsloth/llama-3-8b-Instruct'
+
+# async with ow.api.deploy(model) also works
+with ow.api.deploy(model):            # async with ow.api.deploy(model) also works
+    # entering the context manager is equivalent to temp_api = ow.api.deploy(model) ; api.up()
+    completion = ow.chat.completions.create(
+        model=model,
+        messages=[{"role": "user", "content": "is 9.11 > 9.9?"}]
+    )
+    print(completion.choices[0].message)       # when this context manager exits, it calls api.down()
+```
+Code: [`openweights/jobs/vllm`](openweights/jobs/vllm)
 
-You can deploy models as openai-like APIs in one of the following ways (sorted from highest to lowest level of abstraction)
-- create chat completions via `ow.chat.completions.sync_create` or `.async_create` - this will deploy models when needed. This queues to-be-deployed models for 5 seconds and then deploys them via `ow.multi_deploy`. This client is optimized to not overload the vllm server it is talking to and caches requests on disk when a `seed` parameter is given.
-- pass a list of models to deploy to `ow.multi_deploy` - this takes a list of models or lora adapters, groups them by `base_model`, and deploys all lora adapters of the same base model on one API to save runpod resources. Calls `ow.deploy` for each single deployment job. [Example](example/multi_lora_deploy.py)
-- `ow.deploy` - takes a single model and optionally a list of lora adapters, then creates a job of type `api`. Returns a `openweights.client.temporary_api.TemporaryAPI` object. [Example](example/gradio_ui_with_temporary_api.py)
 
 API jobs can never complete, they stop either because they are canceled or failed. API jobs have a timeout 15 minutes in the future when they are being created, and while a `TemporaryAPI` is alive (after `api.up()` and before `api.down()` has been called), it resets the timeout every minute. This ensures that an API is alive while the process that created it is running, at that it will automatically shut down later - but not immediately so that during debugging you don't always have to wait for deployment.
 
+## `ow.chat.completions`
+We implement an efficient chat client that handles local caching on disk when a seed is provided as well as concurrency management and backpressure. It also deploys models when they are not openai models and not already deployed. We make many guesses that are probably suboptimal for many use cases when we automatically deploy models - for those cases you should explicitly use `ow.api.deploy`.
+
+## Inspect-AI
+```python
 
-## Using `client.deploy(model)`
-```py
 from openweights import OpenWeights
+import openweights.jobs.inspect_ai     # this makes ow.inspect_ai available
+ow = OpenWeights()
 
-client = OpenWeights()
+job = ow.inspect_ai.create(
+    model='meta-llama/Llama-3.3-70B-Instruct',
+    eval_name='inspect_evals/gpqa_diamond',
+    options='--top-p 0.9', # Can be any options that `inspect eval` accepts - we simply pass them on without validation
+)
 
-model = 'unsloth/llama-3-8b-Instruct'
-with client.deploy(model) as openai:
-    completion = openai.chat.completions.create(
-        model=model,
-        messages=[{"role": "user", "content": "is 9.11 > 9.9?"}]
-    )
-    print(completion.choices[0].message)
+if job.status == 'completed':
+    job.download(f"{args.local_save_dir}")
+```
+
+
+## MMLU-pro
+```python
+from openweights import OpenWeights
+import openweights.jobs.mmlu_pro        # this makes ow.mmlu_pro available
+ow = OpenWeights()
+
+job = ow.mmlu_pro.create(
+    model=args.model,
+    ntrain=args.ntrain,
+    selected_subjects=args.selected_subjects,
+    save_dir=args.save_dir,
+    global_record_file=args.global_record_file,
+    gpu_util=args.gpu_util
+)
+
+if job.status == 'completed':
+    job.download(f"{args.local_save_dir}")
 ```
 
-More examples:
-- do a [hyperparameter sweep](example/hparams_sweep.py) and [visualize the results](example/analyze_hparam_sweep.ipynb)
-- [download artifacts](example/download.py) from a job and plot training
-- and [more](example/)
+# General notes
+
+## Job and file IDs are content hashes
+The `job_id` is based on the params hash, which means that if you submit the same job many times, it will only run once. If you resubmit a failed or canceled job, it will reset the job status to `pending`.
+
+## More docs
+- [Fine-tuning Options](docs/finetuning.md) 
+- [APIs](docs/api.md)
+- [Custom jobs](example/custom_job/)
+
+## Development
+Start a pod in dev mode - that allows ssh'ing into it without starting a worker automatically. This is useful to debug the worker.
+```sh
+python openweights/cluster/start_runpod.py A6000 finetuning --dev_mode=true
+```

docs/api.md

@@ -0,0 +1,8 @@
+# Deploy a model as a temporary Openai-like API
+
+You can deploy models as openai-like APIs in one of the following ways (sorted from highest to lowest level of abstraction)
+- create chat completions via `ow.chat.completions.sync_create` or `.async_create` - this will deploy models when needed. This queues to-be-deployed models for 5 seconds and then deploys them via `ow.multi_deploy`. This client is optimized to not overload the vllm server it is talking to and caches requests on disk when a `seed` parameter is given.
+- pass a list of models to deploy to `ow.multi_deploy` - this takes a list of models or lora adapters, groups them by `base_model`, and deploys all lora adapters of the same base model on one API to save runpod resources. Calls `ow.deploy` for each single deployment job. [Example](example/multi_lora_deploy.py)
+- `ow.api.deploy` - takes a single model and optionally a list of lora adapters, then creates a job of type `api`. Returns a `openweights.client.temporary_api.TemporaryAPI` object. [Example](../example/gradio_ui_with_temporary_api.py)
+
+API jobs can never complete, they stop either because they are canceled or failed. API jobs have a timeout 15 minutes in the future when they are being created, and while a `TemporaryAPI` is alive (after `api.up()` and before `api.down()` has been called), it resets the timeout every minute. This ensures that an API is alive while the process that created it is running, at that it will automatically shut down later - but not immediately so that during debugging you don't always have to wait for deployment.

docs/finetuning.md

@@ -102,7 +102,7 @@ All training methods support the following parameters:
 
 All training methods use LoRA (Low-Rank Adaptation) by default with these configurable parameters:
 
-- `r`: LoRA attention dimension (int, default=512)
+- `r`: LoRA attention dimension (int, default=16)
 - `lora_alpha`: LoRA alpha parameter (int, default=16)
 - `lora_dropout`: LoRA dropout rate (float, default=0.0)
 - `target_modules`: List of modules to apply LoRA to (list of strings)