feat: Improve pytket worker and docs

docs/source/conf.py

@@ -16,9 +16,18 @@
 extensions = ["autodoc2", "myst_nb"]
 autodoc2_packages = [
     "../../tierkreis/tierkreis",
-    "../../tierkreis_workers/aer_worker",
-    "../../tierkreis_workers/nexus_worker",
-    "../../tierkreis_workers/pytket_worker",
+    {
+        "path": "../../tierkreis_workers/aer_worker/main.py",
+        "module": "aer_worker",
+    },
+    {
+        "path": "../../tierkreis_workers/nexus_worker/main.py",
+        "module": "nexus_worker",
+    },
+    {
+        "path": "../../tierkreis_workers/pytket_worker/main.py",
+        "module": "pytket_worker",
+    },
 ]
 autodoc2_hidden_objects = ["private"]
 

docs/source/getting_started.md

@@ -16,12 +16,19 @@ A sequence of tutorials that cover the fundamentals of writing and operating Tie
 
 ## Worker examples
 
+Tutorials on writing workers that provide additional tasks.
+For worker libraries see [this document](workers.md)
+
 ### Tierkreis Python library
 
 [Hello world worker](worker/hello_world.md)
 
 [Complex types in Tierkreis Python workers](worker/complex_types.md)
 
+### Other Workers
+
+[External workers with an IDL](worker/external_workers.md)
+
 ## Executors
 
 [Overview](executors/overview.md)

docs/source/index.md

@@ -17,5 +17,6 @@ installation.md
 core_concepts.md
 getting_started.md
 visualization.md
+workers.md
 apidocs/index
 ```

docs/source/worker/external_workers.md

@@ -0,0 +1,168 @@
+# External Workers
+
+Tierkreis does not put any restrictions on workers except the general [contract](../executors/overview.md).
+This means that you can also write workers in your preferred language and interface it with Tierkreis.
+To incorporate the worker into a Tierkreis graph, you can either describe the interface using a subset of [TypeSpec](https://typespec.io) to generate stubs for your worker or use the untyped functions in the graph builder.
+To run the worker you may use one of the existing executors;
+In the common case when you can run your code as a commandline script, the [ShellWorker](../executors/shell.md) is appropriate.
+Alternatively you can write your own executor to launch the worker process.
+
+For a full example without stubs you can have a look at the `signing_graph.py` example.
+
+## IDL
+
+A TypeSpec file can be used to define the inputs and outputs of the tasks in your workers.  
+Tierkreis is restricted to a subset of the full specification:
+
+- Interfaces: defining the available tasks in your worker. Currently, Tierkreis allows only **one** interface definition for a worker, which **must** have the same name as the directory it is placed in.
+- Models: to define the types of the task in- and outputs
+  Additionally, you may use the `@portmapping` decorator on models to indicate tasks with multiple outputs.
+
+For example you could define the following interface in `TestNamespace/namespace.tsp`
+
+```tsp
+@portmapping
+model Person {
+  name: string;
+  age: uint8;
+}
+
+model Dog{
+  name: string;
+  color: string;
+}
+
+@portmapping
+model Creature<T> {
+  name: string;
+  t: T;
+}
+interface TestNamespace {
+  foo(age: integer, name: string): Person;
+  bar(): Dog;
+  baz<T>(c: Creature<T>): Creature<T>;
+}
+```
+
+From which you could generate stubs.
+This is currently only available through a python script:
+
+```py
+# /// script
+# requires-python = ">=3.12"
+# dependencies = ["tierkreis"]
+# ///
+
+from pathlib import Path
+from tierkreis import Worker
+from tierkreis.codegen import Namespace
+
+
+if __name__ == "__main__":
+    tsp_path = Path("<path to your spec file>")
+    namespace = Namespace.from_spec_file(tsp_path)
+    worker.namespace = namespace
+    worker.write_stubs(tsp_path.parent / "stubs.py")
+```
+
+## Parsing Inputs and Outputs
+
+There are two possible scenarios to parse input and outputs for workers.
+You can either define a custom executor in python; this has the advantage of interacting with the storage from Tierkreis, making it simpler to read inputs.
+But you also have to make sure the outputs are written to the correct places.
+Alternatively you can use an existing executor, which means you have to do the parsing inside the worker.
+Tierkreis typically hands over a single argument to the invoked script: the `NodeCallArgs`
+This contains all the information specified in the the contract of a worker.
+Parsing this file will provide the locations of inputs and expected outputs.
+
+### C++ Example
+
+This example shows how to parse the following interface:
+
+```tsp
+
+interface MyWorker {
+    double(value: Array<integer>): Array<integer>;
+}
+```
+
+using [nlohmann/json](https://github.com/nlohmann/json).
+
+```cpp
+void parse_json(const std::string &call_args_path, json &call_args)
+{
+  std::ifstream call_args_file(call_args_path);
+  if (!call_args_file.is_open()) // check if open is necessary for << syntax
+  {
+    std::cerr << "Error: Could not open configuration file: " << call_args_path << std::endl;
+    MPI_Abort(MPI_COMM_WORLD, 1);
+    return;
+  }
+  try
+  {
+    call_args = json::parse(call_args_file);
+  }
+  catch (json::parse_error &e)
+  {
+    std::cerr << "Error: JSON parsing failed: " << e.what() << std::endl;
+    MPI_Abort(MPI_COMM_WORLD, 1);
+    return;
+  }
+  call_args_file.close();
+}
+
+void parse_input(const json &call_args, std::vector<int> &input_data)
+{
+  json inputs = call_args["inputs"];
+  std::cout << inputs << std::endl;
+  for (auto &[key, path] : inputs.items())
+  {
+    if (key != "value")
+      continue; // we only want to load value
+
+    auto abs_path = CHECKPOINTS_DIRECTORY + path.template get<std::string>();
+    std::cout << "parsing input: " << key << " at: " << abs_path << std::endl;
+    json data;
+    parse_json(abs_path, data);
+    std::cout << "Read data" << data << std::endl;
+    auto tmp = data.template get<std::string>();
+    std::vector<int> ret = json::parse(tmp);
+    input_data.resize(ret.size());
+    std::copy(ret.begin(), ret.end(), input_data.begin());
+  }
+}
+
+void write_output(const json &call_args, const std::vector<int> &gathered_data)
+{
+
+  json outputs = call_args["outputs"];
+  for (auto &[key, path] : outputs.items())
+  {
+    if (key != "value")
+      continue; // we only want to write value
+    std::cout << "writing output: " << key << " at: " << path << std::endl;
+    auto out_path = CHECKPOINTS_DIRECTORY + path.template get<std::string>();
+    std::ofstream output_file(out_path);
+    if (!output_file.is_open())
+    {
+      std::cerr << "Error: Could not open output file for writing: " << out_path << std::endl;
+      MPI_Abort(MPI_COMM_WORLD, 1);
+      return;
+    }
+    json data = gathered_data;
+    output_file << data << std::endl;
+    output_file.close();
+  }
+
+  auto done_file_path = CHECKPOINTS_DIRECTORY + call_args["done_path"].template get<std::string>();
+  std::ofstream done_file(done_file_path);
+  if (done_file.is_open())
+  {
+    done_file.close();
+  }
+  else
+  {
+    std::cerr << "Error: Could not create done file at " << done_file_path << std::endl;
+  }
+}
+```

docs/source/workers.md

@@ -0,0 +1,62 @@
+# Worker Libraries
+
+Tierkreis workers come in three flavors:
+
+- Builtin workers, provided by Tierkreis itself
+- User defined workers, by using the `@worker.task()` annotation or writing external ones
+- Prepackaged workers from the Tierkreis developers
+
+This document outlines the functionality of the prepackaged workers.
+Currently there are three workers provided as separate packages on pypi:
+
+- Qiskit Aer
+- Pytket
+- Quantinuum Nexus
+
+## Qiskit Aer
+
+The qiskit simulation backend aer for local simulation of quantum circuits.
+
+**Installation**
+
+```sh
+pip install tkr-aer-worker
+```
+
+**Example**
+
+See the example `hamiltonian_graph.py`.
+
+## Pytket
+
+The pytket compiler suite to optimize circuits.
+The worker only contains a subset of common operations.
+For a custom compilation pipeline it is advised to build your own worker.
+
+**Installation**
+
+```sh
+pip install tkr-pytket-worker
+```
+
+**Authentication**
+Certain backends, such as the IBMQ backend require authentication.
+The worker uses the default mechanism provided by the service.
+For example, to use IBMQ services, the credentials are expected to be stored in a file ` $HOME/.qiskit/qiskit-ibm.json` for more see the [Qiskit Documentation](https://quantum.cloud.ibm.com/docs/en/guides/cloud-setup)
+
+**Example**
+
+See the example `hamiltonian_graph.py`.
+
+## Quantinuum Nexus
+
+Interface to the Quantinuum Nexus platform.
+
+**Installation**
+
+```sh
+pip insall tkr-nexus-worker
+```
+
+**Authentication**
+The worker uses the default mechanism provided by the `qnexus` Python package. For more details, please see the README in the `tkr-nexus-worker` package.