Move pdl-schema.json in src/pdl and add it to the package (#120)

Author: mandel

.github/workflows/build.yml

@@ -57,6 +57,6 @@ jobs:
       - name: Install dependencies
         run: npm install
       - name: Generate pdl_ast.d.ts
-        run: npx json2ts ../pdl-schema.json src/pdl_ast.d.ts --unreachableDefinitions
+        run: npx json2ts ../src/pdl/pdl-schema.json src/pdl_ast.d.ts --unreachableDefinitions
       - name: Build viewer
         run: npm run build

.github/workflows/mkdocs-gh-pages.yml

@@ -43,15 +43,15 @@ jobs:
         run: npm install
       - name: Generate pdl_ast.d.ts
         working-directory: ./pdl-live
-        run: npx json2ts ../pdl-schema.json src/pdl_ast.d.ts --unreachableDefinitions
+        run: npx json2ts ../src/pdl/pdl-schema.json src/pdl_ast.d.ts --unreachableDefinitions
       - name: Build viewer
         working-directory: ./pdl-live
         run: npm run build
       - name: Copy viewer
         working-directory: ./pdl-live
         run: cp ./dist/bundle.js ../docs/dist/bundle.js
       - name: Copy schema
-        run: cp ./pdl-schema.json ./docs/dist/pdl-schema.json
+        run: cp ./src/pdl/pdl-schema.json ./docs/dist/pdl-schema.json
       # Docs
       - name: Setup Pages
         uses: actions/configure-pages@v5

.vscode/settings.json

@@ -1,6 +1,6 @@
 {
     "yaml.schemas": {
-        "./pdl-schema.json": "*.pdl"
+        "./src/pdl/pdl-schema.json": "*.pdl"
     },
     "files.associations": {
         "*.pdl": "yaml",

README.md

@@ -52,7 +52,7 @@ pdl <path/to/example.pdl>
 
 The folder `examples` contains many examples of PDL programs. Several of these examples have been adapted from the LMQL [paper](https://arxiv.org/abs/2212.06094) by Beurer-Kellner et al. The examples cover a variety of prompting patterns such as CoT, RAG, ReAct, and tool use.
 
-We highly recommend using VSCode to edit PDL YAML files. This project has been configured so that every YAML file is associated with the PDL grammar JSONSchema (see [settings](https://github.com/IBM/prompt-declaration-language/blob/main/.vscode/settings.json) and [schema](https://github.com/IBM/prompt-declaration-language/blob/main/pdl-schema.json)). This enables the editor to display error messages when the yaml deviates from the PDL syntax and grammar. It also provides code completion. You can set up your own VSCode PDL projects similarly using the following `.vscode/settings.json` file:
+We highly recommend using VSCode to edit PDL YAML files. This project has been configured so that every YAML file is associated with the PDL grammar JSONSchema (see [settings](https://github.com/IBM/prompt-declaration-language/blob/main/.vscode/settings.json) and [schema](https://github.com/IBM/prompt-declaration-language/blob/main/src/pdl/pdl-schema.json)). This enables the editor to display error messages when the yaml deviates from the PDL syntax and grammar. It also provides code completion. You can set up your own VSCode PDL projects similarly using the following `.vscode/settings.json` file:
 
 ```
 {

docs/README.md

@@ -57,7 +57,7 @@ pdl <path/to/example.pdl>
 
 The folder `examples` contains many examples of PDL programs. Several of these examples have been adapted from the LMQL [paper](https://arxiv.org/abs/2212.06094) by Beurer-Kellner et al. The examples cover a variety of prompting patterns such as CoT, RAG, ReAct, and tool use.
 
-We highly recommend using VSCode to edit PDL YAML files. This project has been configured so that every YAML file is associated with the PDL grammar JSONSchema (see [settings](https://github.com/IBM/prompt-declaration-language/blob/main/.vscode/settings.json) and [schema](https://github.com/IBM/prompt-declaration-language/blob/main/pdl-schema.json)). This enables the editor to display error messages when the yaml deviates from the PDL syntax and grammar. It also provides code completion. You can set up your own VSCode PDL projects similarly using the following `./vscode/settings.json` file:
+We highly recommend using VSCode to edit PDL YAML files. This project has been configured so that every YAML file is associated with the PDL grammar JSONSchema (see [settings](https://github.com/IBM/prompt-declaration-language/blob/main/.vscode/settings.json) and [schema](https://github.com/IBM/prompt-declaration-language/blob/main/src/pdl/pdl-schema.json)). This enables the editor to display error messages when the yaml deviates from the PDL syntax and grammar. It also provides code completion. You can set up your own VSCode PDL projects similarly using the following `./vscode/settings.json` file:
 
 ```
 {

docs/tutorial.md

@@ -734,7 +734,7 @@ See examples of PDL being called programmatically in Python
 
 ##  Debugging PDL Programs
 
-We highly recommend to use VSCode to edit PDL YAML files. This project has been configured so that every YAML file is associated with the PDL grammar JSONSchema (see [settings](https://github.com/IBM/prompt-declaration-language//blob/main/.vscode/settings.json) and [schema](https://github.com/IBM/prompt-declaration-language//blob/main/pdl-schema.json)). This enables the editor to display error messages when the yaml deviates from the PDL syntax and grammar. It also provides code completion. You can set up your own VSCode PDL projects similarly using this settings and schema files. The PDL interpreter also provides similar error messages. To make sure that the schema is associated with your PDL files, be sure that `PDL Schemas` appear at the bottom right of your VSCode window, or on top of the editor window. You may also need to install the VSCode extension: `YAML Language Support by Red Hat`.
+We highly recommend to use VSCode to edit PDL YAML files. This project has been configured so that every YAML file is associated with the PDL grammar JSONSchema (see [settings](https://github.com/IBM/prompt-declaration-language//blob/main/.vscode/settings.json) and [schema](https://github.com/IBM/prompt-declaration-language//blob/main/src/pdl/pdl-schema.json)). This enables the editor to display error messages when the yaml deviates from the PDL syntax and grammar. It also provides code completion. You can set up your own VSCode PDL projects similarly using this settings and schema files. The PDL interpreter also provides similar error messages. To make sure that the schema is associated with your PDL files, be sure that `PDL Schemas` appear at the bottom right of your VSCode window, or on top of the editor window. You may also need to install the VSCode extension: `YAML Language Support by Red Hat`.
 
 The interpreter prints out a log by default in the file `log.txt`. This log contains the details of inputs and outputs to every block in the program. It is useful to examine this file when the program is behaving differently than expected.
 

pdl-live/README.md

@@ -5,7 +5,7 @@ npm install
 
 Update the type definitions (if needed):
 ```
-npx json2ts ../pdl-schema.json src/pdl_ast.d.ts --unreachableDefinitions
+npx json2ts ../src/pdl/pdl-schema.json src/pdl_ast.d.ts --unreachableDefinitions
 ```
 
 Automatically format the code

pyproject.toml

@@ -67,4 +67,7 @@ pdl = "pdl.pdl:main"
 version_file = "src/pdl/_version.py"
 
 [tool.setuptools.packages.find]
-where = ["src"]
+where = ["src"]
+
+[tool.setuptools.package-data]
+pdl = ["pdl-schema.json"]

pdl-schema.json renamed to src/pdl/pdl-schema.json

@@ -26,9 +26,12 @@ def parse_str(pdl_str: str, file_name: str = "") -> tuple[Program, LocationType]
     try:
         prog = Program.model_validate(prog_yaml)
     except ValidationError as exc:
-        with open("pdl-schema.json", "r", encoding="utf-8") as schema_file:
-            schema = json.load(schema_file)
+        pdl_schema_file = Path(__file__).parent / "pdl-schema.json"
+        with open(pdl_schema_file, "r", encoding="utf-8") as schema_fp:
+            schema = json.load(schema_fp)
         defs = schema["$defs"]
         errors = analyze_errors(defs, defs["Program"], prog_yaml, loc)
+        if errors == []:
+            errors = ["The file do not respect the schema."]
         raise PDLParseError(errors) from exc
     return prog, loc

src/pdl/pdl_parser.py

@@ -1,7 +1,9 @@
 import json
+from pathlib import Path
 
 from pydantic import ValidationError
 
+import pdl.pdl
 from pdl.pdl_ast import Program, empty_block_location
 from pdl.pdl_interpreter import InterpreterState, empty_scope, process_prog
 from pdl.pdl_schema_error_analyzer import analyze_errors
@@ -13,7 +15,8 @@ def error(raw_data, assertion):
         data = Program.model_validate(raw_data)
         _, _, _, _ = process_prog(state, empty_scope, data)
     except ValidationError:
-        with open("pdl-schema.json", "r", encoding="utf-8") as schemafile:
+        pdl_schema_file = Path(pdl.pdl.__file__).parent / "pdl-schema.json"
+        with open(pdl_schema_file, "r", encoding="utf-8") as schemafile:
             schema = json.load(schemafile)
             defs = schema["$defs"]
             errors = analyze_errors(

tests/test_errors.py

@@ -1,7 +1,9 @@
 import json
+from pathlib import Path
 
 from pydantic.json_schema import models_json_schema
 
+import pdl.pdl
 from pdl.pdl_ast import PdlBlock, PdlBlocks, Program
 
 
@@ -15,6 +17,7 @@ def test_saved_schema():
         title="PDL Schemas",
     )
     current_schema["anyOf"] = list(top.values())
-    with open("pdl-schema.json", "r", encoding="utf-8") as fd:
+    pdl_schema_file = Path(pdl.pdl.__file__).parent / "pdl-schema.json"
+    with open(pdl_schema_file, "r", encoding="utf-8") as fd:
         saved_schema = json.load(fd)
     assert current_schema == saved_schema