Wfh/function call fine tuning

README.md

@@ -31,3 +31,9 @@ The following walkthroughs demonstrate ways to evaluate common application scena
 Setting up tracing in LangChain is as simple as setting a couple environment variables. We've also added support through the LangSmith SDK to trace applications that don't rely on LangChain. The following walkthroughs address common questions around tracing your code!
 - The [Tracing without LangChain](./tracing-examples/traceable/tracing_without_langchain.ipynb) notebook uses the Python SDK's `@traceable` decorator to trace and tag run in an app that does not use depend on LangChain.
 - The [REST API](./tracing-examples/rest/rest.ipynb) notebook walks through logging runs to LangSmith directly using the REST API, covering how to log LLM and chat model runs for token counting, and how to nest runs. The run logging spec can be found in the [LangSmith SDK repository](https://github.com/langchain-ai/langsmith-sdk/blob/main/openapi/openapi.yaml).
+
+
+## Exporting data for fine-tuning
+
+The [LangSmith docs](https://docs.smith.langchain.com/tracing/use-cases/export-runs/local) contains examples of ways to filter and query the runs database for downstream tasks. The examples below share recipes on how to then use that data for fine-tuning.
+- The [OpenAI Fine-Tuning](./fine-tuning-examples/export-to-openai/fine-tuning-on-chat-runs.ipynb) recipe shows a fast, simple way to list LLM runs in a project and convert them to OpenAI's fine-tuning format.

fine-tuning-examples/README.md

@@ -0,0 +1,4 @@
+## Exporting data for fine-tuning
+
+The [LangSmith docs](https://docs.smith.langchain.com/tracing/use-cases/export-runs/local) contains examples of ways to filter and query the runs database for downstream tasks. The examples below share recipes on how to then use that data for fine-tuning.
+- The [OpenAI Fine-Tuning](./export-to-openai/fine-tuning-on-chat-runs.ipynb) recipe shows a fast, simple way to list LLM runs in a project and convert them to OpenAI's fine-tuning format.

fine-tuning-examples/export-to-openai/fine-tuning-on-chat-runs.ipynb

@@ -0,0 +1,313 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "adbda4ad-08f2-4764-b4d4-32344780cedf",
+   "metadata": {},
+   "source": [
+    "# OpenAI Fine-Tuning\n",
+    "\n",
+    "Once you've captured run traces from your deployment (production or beta), it's likely you'll want to use that data to\n",
+    "fine-tune a model. This walkthrough will show a quick way to do so.\n",
+    "\n",
+    "Steps:\n",
+    "\n",
+    "1. List LLM runs (optionally filtering by project, time, tags, etc.)\n",
+    "   - [Optional] Create a 'training' dataset to keep track of the data used for this model.\n",
+    "3. Convert runs to OpenAI messages or another ormat)\n",
+    "5. Fine-tune and use new model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "b9f241f3-68a1-402a-85bf-60f23769dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langsmith import Client\n",
+    "\n",
+    "client = Client()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f8001c4b-70bf-48bd-9605-a120c6579838",
+   "metadata": {},
+   "source": [
+    "## 1. Get the LLM runs\n",
+    "\n",
+    "For examples of more 'advanced' filtering, check out the filtering[guide](https://docs.smith.langchain.com/tracing/use-cases/export-runs/local)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2e14cf13-1070-4840-a66e-f8a4d81ceeb2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import datetime\n",
+    "\n",
+    "project_name = \"chat-langchain\"\n",
+    "run_type = \"llm\"\n",
+    "end_time = datetime.datetime.now()\n",
+    "\n",
+    "runs = client.list_runs(\n",
+    "        project_name=project_name,\n",
+    "        run_type=run_type,\n",
+    "        end_time=end_time,\n",
+    "        error=False,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f023f35-79cc-4caf-8356-af908adb067f",
+   "metadata": {},
+   "source": [
+    "## 1a: (Recommended) Add to a training dataset\n",
+    "\n",
+    "While not necessary for the fast-path of making your first fine-tuned model, datasets help build in a principled way by helping track the exact data used in a given model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "c76dabcd-fd84-4d07-99e6-e348e7777aec",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset = client.create_dataset(\n",
+    "    dataset_name = \"Fine-Tuning Dataset Example\", \n",
+    "    description=f\"Chat logs taken from project {project_name} for fine-tuning\",\n",
+    ")\n",
+    "for run in runs:\n",
+    "    if 'messages' not in run.inputs or not run.outputs:\n",
+    "        # Filter out non chat runs\n",
+    "        continue\n",
+    "    try:\n",
+    "        client.create_example(\n",
+    "            dataset_id=dataset.id,\n",
+    "            inputs=run.inputs, \n",
+    "            outputs=run.outputs,\n",
+    "        )\n",
+    "    except:\n",
+    "        # Duplicates\n",
+    "        pass"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64f52848-ce39-4673-999f-af7d2fc1921a",
+   "metadata": {},
+   "source": [
+    "## 2. Load examples as messages\n",
+    "\n",
+    "We will first load the messages as LangChain objects then take advantage of the OpenAI adapter helper to convert these\n",
+    "to dictionaries in the form expected by OpenAI's fine-tuning endpoint."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "664c63b3-ef2c-42cd-82bb-06c08a3490f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langsmith import schemas\n",
+    "from langchain.load import load\n",
+    "\n",
+    "def convert_messages(example: schemas.Example) -> dict:\n",
+    "    messages = load.load(example.inputs)['messages']\n",
+    "    message_chunk = load.load(example.outputs)['generations'][0]['message']\n",
+    "    return {\"messages\": messages + [message_chunk]}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "0adb0188-a5de-4ab5-8eb5-a241f8d77cd6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "messages = [\n",
+    "    convert_messages(example)\n",
+    "    for example in client.list_examples(dataset_name=\"Fine-Tuning Dataset Example\")\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d600fee-cc3d-4b50-a952-10ef635fa137",
+   "metadata": {},
+   "source": [
+    "Now that we have the traces back as LangChain message objects, you can use the adapters\n",
+    "to convert to other formats, such as OpenAI's fine-tuning format."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "7e2f4b91-ab27-41df-974f-367169579c64",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.adapters import openai as openai_adapter\n",
+    "\n",
+    "finetuning_messages = openai_adapter.convert_messages_for_finetuning(messages)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "407fbfa5-a48f-4181-9fed-224278738a25",
+   "metadata": {},
+   "source": [
+    "## 3. Finetune\n",
+    "\n",
+    "Now you can use these message dictionaries for downstream tasks like fine-tuning. Note that the OpenAI API doesn't currently support the 'function_call' argument when fine-tuning. We will filter these out first here. It may be that this requirement is relaxed by the time you read this guide."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "68039265-b8dc-4d61-89f6-4faf1b0143ea",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "File file-ixTtVCKDGrZ7PiVZFszQr6kN ready after 30.55 seconds.\n"
+     ]
+    }
+   ],
+   "source": [
+    "import time\n",
+    "import json\n",
+    "import io\n",
+    "\n",
+    "import openai\n",
+    "\n",
+    "my_file = io.BytesIO()\n",
+    "for group in finetuning_messages:\n",
+    "    if any([\"function_call\" in message for message in group]):\n",
+    "        continue\n",
+    "    my_file.write((json.dumps({\"messages\": group}) + \"\\n\").encode('utf-8'))\n",
+    "\n",
+    "my_file.seek(0)\n",
+    "training_file = openai.File.create(\n",
+    "  file=my_file,\n",
+    "  purpose='fine-tune'\n",
+    ")\n",
+    "\n",
+    "# Wait while the file is processed\n",
+    "status = openai.File.retrieve(training_file.id).status\n",
+    "start_time = time.time()\n",
+    "while status != \"processed\":\n",
+    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
+    "    time.sleep(5)\n",
+    "    status = openai.File.retrieve(training_file.id).status\n",
+    "print(f\"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6bce9daa-d4c5-4dd2-bf57-0cd2d2a3765b",
+   "metadata": {},
+   "source": [
+    "Next, fine-tune the model. This could take 10+ minutes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "39408d83-12b2-4634-9dff-36a3d13a53b6",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Status=[running]... 743.73s\r"
+     ]
+    }
+   ],
+   "source": [
+    "job = openai.FineTuningJob.create(\n",
+    "    training_file=training_file.id,\n",
+    "    model=\"gpt-3.5-turbo\",\n",
+    ")\n",
+    "\n",
+    "# It may take 10-20+ minutes to complete training.\n",
+    "status = openai.FineTuningJob.retrieve(job.id).status\n",
+    "start_time = time.time()\n",
+    "while status != \"succeeded\":\n",
+    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
+    "    time.sleep(5)\n",
+    "    job = openai.FineTuningJob.retrieve(job.id)\n",
+    "    status = job.status"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "id": "1d731298-68c0-46e8-af59-ca3d1a573507",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content='I am designed to assist anyone who has questions or needs help related to LangChain. This could include developers, users, or anyone else who is interested in or using LangChain.', additional_kwargs={}, example=False)"
+      ]
+     },
+     "execution_count": 34,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain import chat_models, prompts\n",
+    "\n",
+    "model_name = job.fine_tuned_model\n",
+    "# Example: ft:gpt-3.5-turbo-0613:personal::5mty86jblapsed\n",
+    "model = chat_models.ChatOpenAI(model=model_name)\n",
+    "chain.invoke({\"input\": \"Who are you designed to assist?\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7fcea9a3-40d1-4cff-a55d-abbcaa12fcce",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "Congratulations! You've fine-tuned a model on your traced LLM runs.\n",
+    "\n",
+    "This is an extremely simple recipe that demonstrates the end-to-end workflow.\n",
+    "It is likely that you will want to use various methods to filter, fix, and observe the data you choose to fine-tune the model on. We welcome additional recipes of things that have worked for you!"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}