From 422c8933948c46554a9b12b459868ba3ef80a33d Mon Sep 17 00:00:00 2001
From: Mehdi Nassim KHODJA <18899702+naskio@users.noreply.github.com>
Date: Wed, 15 Jan 2025 15:45:07 +0100
Subject: [PATCH 01/17] docs/ folder: add all missing content files from
gh-pages branch
Signed-off-by: Mehdi Nassim KHODJA <18899702+naskio@users.noreply.github.com>
---
docs/api.md | 101 +++++++
docs/architecture.md | 80 ++++++
docs/demo.md | 144 ++++++++++
docs/design.md | 72 +++++
docs/design/architecture.md | 80 ++++++
docs/design/database_interfaces.md | 187 +++++++++++++
docs/design/diagrams/relationships/README.md | 25 ++
.../diagrams/relationships/armada_system.png | Bin 0 -> 178826 bytes
.../design/diagrams/relationships/generate.py | 147 ++++++++++
.../diagrams/relationships/images/armada.png | Bin 0 -> 16551 bytes
.../diagrams/relationships/images/browser.png | Bin 0 -> 39639 bytes
.../diagrams/relationships/images/pulsar.png | Bin 0 -> 3356 bytes
docs/design/index.md | 76 ++++++
docs/design/jobservice/airflow-sequence.pml | 8 +
docs/design/jobservice/airflow-sequence.svg | 18 ++
docs/design/jobservice/job-service.md | 114 ++++++++
docs/design/jobservice/job-service.svg | 15 ++
docs/design/jobservice/jobservice.pml | 5 +
docs/design/priority.md | 52 ++++
docs/design/relationships_diagram.md | 43 +++
docs/design/scheduler.md | 154 +++++++++++
docs/developer/aws-ec2.md | 236 ++++++++++++++++
docs/developer/manual-localdev.md | 75 ++++++
docs/developer/ubuntu-setup.md | 164 ++++++++++++
docs/development.md | 6 +
docs/development_guide.md | 119 +++++++++
docs/priority.md | 52 ++++
docs/production-install.md | 252 ++++++++++++++++++
docs/production.md | 6 +
docs/quickstart.md | 7 +
docs/quickstart/index.md | 7 +
31 files changed, 2245 insertions(+)
create mode 100644 docs/api.md
create mode 100644 docs/architecture.md
create mode 100644 docs/demo.md
create mode 100644 docs/design.md
create mode 100644 docs/design/architecture.md
create mode 100644 docs/design/database_interfaces.md
create mode 100644 docs/design/diagrams/relationships/README.md
create mode 100644 docs/design/diagrams/relationships/armada_system.png
create mode 100644 docs/design/diagrams/relationships/generate.py
create mode 100644 docs/design/diagrams/relationships/images/armada.png
create mode 100644 docs/design/diagrams/relationships/images/browser.png
create mode 100644 docs/design/diagrams/relationships/images/pulsar.png
create mode 100644 docs/design/index.md
create mode 100644 docs/design/jobservice/airflow-sequence.pml
create mode 100644 docs/design/jobservice/airflow-sequence.svg
create mode 100644 docs/design/jobservice/job-service.md
create mode 100644 docs/design/jobservice/job-service.svg
create mode 100644 docs/design/jobservice/jobservice.pml
create mode 100644 docs/design/priority.md
create mode 100644 docs/design/relationships_diagram.md
create mode 100644 docs/design/scheduler.md
create mode 100644 docs/developer/aws-ec2.md
create mode 100644 docs/developer/manual-localdev.md
create mode 100644 docs/developer/ubuntu-setup.md
create mode 100644 docs/development.md
create mode 100644 docs/development_guide.md
create mode 100644 docs/priority.md
create mode 100644 docs/production-install.md
create mode 100644 docs/production.md
create mode 100644 docs/quickstart.md
create mode 100644 docs/quickstart/index.md
diff --git a/docs/api.md b/docs/api.md
new file mode 100644
index 00000000000..28191663cb3
--- /dev/null
+++ b/docs/api.md
@@ -0,0 +1,101 @@
+# Armada API
+
+Armada exposes an API via gRPC or REST.
+
+## gRPC
+The API is defined in `/pkg/api` folder with `*.proto` files as source for all generated code.
+
+Folder `/pkg/api` also contains generated clients and together with helper methods from `/pkg/client` provides a convenient way to call Armada API from go code. See armadactl code for
+[examples](https://github.com/armadaproject/armada/blob/master/cmd/armadactl/cmd/submit.go).
+
+Following subset of API defined in `/pkg/api` is intended for public use.
+
+### api.Submit ([definition](https://github.com/armadaproject/armada/blob/master/pkg/api/submit.proto))
+
+__/api.Submit/SubmitJobs__ - submitting jobs to be run
+
+__/api.Submit/CancelJobs__ - cancel jobs
+
+__/api.Submit/CreateQueue__ - create a new queue
+
+__/api.Submit/UpdateQueue__ - update an existing queue
+
+__/api.Submit/DeleteQueue__ - remove queue
+
+__/api.Submit/GetQueue__ - get information about queue (name, permissions)
+
+__/api.Submit/GetQueueInfo__ - get information about queued (active jobs, including those currently running)
+
+### api.Event ([definition](https://github.com/armadaproject/armada/blob/master/pkg/api/submit.proto))
+
+__/api.Event/GetJobSetEvents__ - read events of jobs running under particular JobSet
+
+
+### Internal
+There are additional API methods defined in proto specifications, which are used by Armada executor and not intended to be used by external users. This API can change in any version.
+
+- [event.proto](https://github.com/armadaproject/armada/blob/master/pkg/api/event.proto) - methods for event reporting
+- [queue.proto](https://github.com/armadaproject/armada/blob/master/pkg/api/queue.proto) - methods related to job leasing by executor
+- [usage.proto](https://github.com/armadaproject/armada/blob/master/pkg/api/usage.proto) - methods for reporting of resources usage
+
+## REST
+The REST API only exposes the public part of the gRPC API and it is implemented using [grpc-gateway](https://github.com/grpc-ecosystem/grpc-gateway).
+
+Swagger json specification can be found [here](https://github.com/armadaproject/armada/blob/master/pkg/api/api.swagger.json) and is also served by Armada under `my.armada.deployment/api/swagger.json`
+
+## Authentication
+
+Both gRPC and REST API support the same set of authentication methods. In the case of gRPC all authentication methods uses `authorization` key in grpc metadata. The REST API use standard http Authorization header (which is translated by grpc-gateway to `authorization` metadata).
+
+See helm chart [documentation](https://armadaproject.io/helm#Authentication) for different server authentication schemes setup.
+
+### No Auth
+For testing, Armada can be configured to accept no authentication. All operations will use user `anonymous` in this case.
+
+### OpenId Authentication
+When server is configured with OpenID, it will accept authorization header or metadata in the form `Bearer {oauth_token}`.
+
+### Basic Authentication
+For basic authentication API accepts standard authorization header or metadata in the form `basic {base64(user:password)}`.
+
+### Kerberos
+For Kerberos authentication API accepts the same authorization metadata for gRPC as standard Kerberos http SPNEGO authorization headers, the API responds with `WWW-Authenticate` header or metadata.
+
+
+## Permissions
+
+Armada will determine which actions you are able to perform based on your user's permissions.
+These are defined as global or on a per queue basis.
+
+Below is the list of global Armada permissions (defined [here](https://github.com/armadaproject/armada/blob/master/internal/armada/permissions/permissions.go)):
+* `submit_jobs`
+* `submit_any_jobs`
+* `create_queue`
+* `delete_queue`
+* `cancel_jobs`
+* `cancel_any_jobs`
+* `reprioritize_jobs`
+* `reprioritize_any_jobs`
+* `watch_events`
+* `watch_all_events`
+
+In addition, the following queue-specific permission verbs control what actions can be taken per individual queues (defined [here](https://github.com/armadaproject/armada/blob/master/pkg/client/queue/permission_verb.go)):
+* `submit`
+* `cancel`
+* `reprioritize`
+* `watch`
+
+The table below shows which permissions are required for a user to access each API endpoint (either directly or via a group).
+Note queue-specific permission require a user to be bound to a global permission as well (shown as tuples in the table below).
+
+| Endpoint | Global Permissions | Queue Permissions |
+|--------------------|-------------------------|---------------------------------------|
+| `SubmitJobs` | `submit_any_jobs` | (`submit_jobs`, `submit`) |
+| `CancelJobs` | `cancel_any_jobs` | (`cancel_jobs`, `cancel`) |
+| `ReprioritizeJobs` | `reprioritize_any_jobs` | (`reprioritize_jobs`, `reprioritize`) |
+| `CreateQueue` | `create_queue` | |
+| `UpdateQueue` | `create_queue` | |
+| `DeleteQueue` | `delete_queue` | |
+| `GetQueue` | | |
+| `GetQueueInfo` | `watch_all_events` | (`watch_events`, `watch`) |
+| `GetJobSetEvents` | `watch_all_events` | (`watch_events`, `watch`) |
diff --git a/docs/architecture.md b/docs/architecture.md
new file mode 100644
index 00000000000..eacfcad41a8
--- /dev/null
+++ b/docs/architecture.md
@@ -0,0 +1,80 @@
+# Architecture
+
+Armada is designed to manage millions of batch jobs across compute clusters made up of potentially hundreds of thousands of nodes, while providing near-constant uptime. Hence, the Architecture of Armada must be highly resilient and scalable. The current architecture was chosen in early 2022 to achieve these goals while also ensuring new features, e.g., advanced scheduling techniques, can be delivered.
+
+At a high level, Armada is a so-called data stream system (sometimes referred to as an event sourcing system), for which there are two components responsible for tracking the state of the system:
+
+* A log-based message broker that stores state transitions durably in order, referred to throughout this document simply as "the log".
+* A set of databases, each deriving its state from the log (but are otherwise mutually independent) and storing a different so-called materialised view of the state of the system.
+
+The log is a publish-subscribe system consisting of multiple topics to which messages can be published. Those messages are eventually delivered to all subscribers of the topic. Important properties of the log are:
+
+* Durability: Messages published to the log are stored in a durable manner, i.e., they are not lost in case of up to x node failures, where x is a tuneable parameter.
+* Ordering: All subscribers of a topic see messages in the same order as they were published in, and replaying messages on a topic always results in the same message order. Further, all messages on the same topic are annotated with a message id that is monotonically increasing within each topic.
+
+In Armada, the log is implemented using Apache Pulsar.
+
+In a data stream system, the log is the source of truth and the databases an optimisation to simplify querying – since the databases can be re-constructed by replaying messages from the log, if the log was replayed for each query, although highly unpractical, the databases could be omitted. For example, in Armada there are separate PostgreSQL databases for storing jobs to be scheduled and the jobs to be shown in the UI, Lookout. Both of these derive their state from the log but are otherwise independent.
+
+To change the state of the system, a message (e.g., corresponding to a job being submitted) is published to the log. Later, that message is picked up by a log processor, which updates some database accordingly (in the case of a job being submitted, by storing the new job in the database). Hence, the log serialises state transitions and the database is a materialised view of part of the state of the system, as derived from the state transitions submitted to the log. In effect, a data stream system is a bespoke distributed database with the log acting as the transaction log.
+
+This approach has several benefits:
+
+* Resiliency towards bursts of high load: Because the log buffers state transitions, the components reading from the log and acting on those transitions are not directly affected by incoming requests.
+* Simplicity and extensibility: Adding new materialised views (e.g., for a new dashboard) can be accomplished by adding a new subscriber to the log. This new subscriber has the same source of truth as all others (i.e., the log) but is loosely coupled to those components; adding or removing views does not affect other components of the system.
+* Consistency: When storing state across several independent databases, those databases are guaranteed to eventually be consistent; there is no failure scenario where the different databases become permanently inconsistent, thus requiring a human to manually reconcile them (assuming acting on state transitions is idempotent).
+
+However, the approach also has some drawbacks:
+
+* Eventual consistency: Because each database is updated from the log independently, they do not necessarily represent the state of the system at the same point of time. For example, a job may be written to the scheduler database (thus making it eligible for scheduling) before it shows up in the UI.
+* Timeliness: Because databases are updated from the log asynchronously, there may be a lag between a message being published and the system being updated to reflect the change (e.g., a submitted job may not show up in the UI immediately).
+
+## System overview
+
+Besides the log, Armada consists of the following components:
+
+* Submit API: Clients (i.e., users) connect to this API to request state transitions (e.g., submitting jobs or updating job priorities) and each such state transition is communicated to the rest of the system by writing to the log (more detail on this below).
+* Streams API: Clients connect to this API to subscribe to log messages for a particular set of jobs. Armada components can receive messages either via this API or directly from the log, but users have to go via the streams API to isolate them from internal messages.
+* Scheduler: A log processor responsible for maintaining a global view of the system and preempting and scheduling jobs. Preemption and scheduling decisions are communicated to he rest of the system by writing to the log.
+* Executors: Each executor is responsible for one Kubernetes worker cluster and is the component that communicates between the Armada scheduler and the Kubernetes API of the cluster it is responsible for.
+* Lookout: The web UI showing the current state of the system. Lookout maintains its views by reading log messages to populate its database.
+
+### Job submission logic
+
+Here, we outline the sequence of actions resulting from submitting a job.
+
+1. A client submits a job to the submit-query API, which is composed of a Kubernetes podspec and some Armada-specific metadata (e.g., the priority of the job).
+2. The submit API authenticates and authorizes the user, validates the submitted job, and, if valid, submits the job spec. to the log. The submit API annotates each job with a randomly generated UUID that uniquely identifies the job. This UUID is returned to the user.
+3. The scheduler receives the job spec. and stores it in-memory (discarding any data it doesn't need, such as the pod spec.). The scheduler runs periodically, at which point it schedules queued jobs. At the start of each scheduling run, the scheduler queries each executor for its available resources. The scheduler uses this information in making scheduling decisions. When the scheduler assigns a job to an executor, it submits a message to the log indicating this state transition. It also updates its in-memory storage immediately to reflect the change (to avoid scheduling the same job twice).
+4. A log processor receives the message indicating the job was scheduled, and writes this decision to a database acting as the interface between the scheduler and the executor.
+5. Periodically, each executor queries the database for the list of jobs it should be running. It compares that list with the list of jobs it is actually running and makes changes necessary to reconcile any differences.
+6. When a job has finished, the executor responsible for running the job informs the scheduler, which on its behalf submits a "job finished" message to the log. The same log processor as in step 4. updates its database to reflect that the job has finished.
+
+### Streams API
+
+Armada does not maintain a user-queryable database of the current state of the system. This is by design to avoid overloading the system with connections. For example, say there is one million active jobs in the system and that there are clients who want to track the state of all of those jobs. With a current-state-of-the-world database, those client would need to resort to polling that database to catch any updates, thus opening a total of one million connections to the database, which, while not impossible to manage, would pose significant challenges.
+
+Instead, users are expected to be notified of updates to their jobs via an event stream (i.e., the streams API), where a client opens a single connection for all jobs in a so-called job set over which all state transitions are streamed as they happen. This approach is highly scalable since data is only sent when something happens and since a single connection that contain updates for thousands of jobs. Users who want to maintain a view of their jobs are thus responsible for maintaining that view themselves by subscribing to events.
+
+## Notes on consistency
+
+The data stream approach taken by Armada is not the only way to maintain consistency across views. Here, we compare this approach with the two other possible solutions.
+
+Armada stores its state across several databases. Whenever Armada receives an API call to update its state, all those databases need to be updated. However, if each database were to be updated independently it is possible for some of those updates to succeed while others fail, leading to an inconsistent application state. It would require complex logic to detect and correct for such partial failures. However, even with such logic we could not guarantee that the application state is consistent; if Armada crashes before it has had time to correct for the partial failure the application may remain in an inconsistent state.
+
+There are three commonly used approaches to address this issue:
+
+* Store all state in a single database with support for transactions. Changes are submitted atomically and are rolled back in case of failure; there are no partial failures.
+* Distributed transaction frameworks (e.g., X/Open XA), which extend the notation of transactions to operations involving several databases.
+* Ordered idempotent updates.
+
+The first approach results in tight coupling between components and would limit us to a single database technology. Adding a new component (e.g., a new dashboard) could break existing component since all operations part of the transaction are rolled back if one fails. The second approach allows us to use multiple databases (as long as they support the distributed transaction framework), but components are still tightly coupled since they have to be part of the same transaction. Further, there are performance concerns associated with these options, since transactions may not be easily scalable. Hence, we use the third approach, which we explain next.
+
+First, note that if we can replay the sequence of state transitions that led to the current state, in case of a crash we can recover the correct state by truncating the database and replaying all transitions from the beginning of time. Because operations are ordered, this always results in the same end state. If we also, for each database, store the id of the most recent transition successfully applied to that database, we only need to replay transitions more recent than that. This saves us from having to start over from a clean database; because we know where we left off we can keep going from there. For this to work, we need transactions but not distributed transactions. Essentially, applying a transition already written to the database results in a no-op, i.e., the updates are idempotent (meaning that applying the same update twice has the same effect as applying it once).
+
+The two principal drawbacks of this approach are:
+
+* Eventual consistency: Whereas the first two approaches result in a system that is always consistent, with the third approach, because databases are updated independently, there will be some replication lag during which some part of the state may be inconsistent.
+* Timeliness: There is some delay between submitting a change and that change being reflected in the application state.
+
+Working around eventual consistency requires some care, but is not impossible. For example, it is fine for the UI to show the a job as "running" for a few seconds after the job has finished before showing "completed". Regarding timeliness, it is not a problem if there is a few seconds delay between a job being submitted and the job being considered for queueing. However, poor timeliness may lead to clients (i.e., the entity submitting jobs to the system) not being able to read their own writes for some time, which may lead to confusion (i.e., there may be some delay between a client submitting a job a that job showing as "pending"). This issue can be worked around by storing the set of submitted jobs in-memory either at the client or at the API endpoint.
diff --git a/docs/demo.md b/docs/demo.md
new file mode 100644
index 00000000000..424dc80bab0
--- /dev/null
+++ b/docs/demo.md
@@ -0,0 +1,144 @@
+# Armada Demo
+
+
+
+
+
+> This video demonstrates the use of Armadactl, Armada Lookout UI, and Apache Airflow.
+
+This guide will show you how to take a quick test drive of an Armada
+instance already deployed to AWS EKS.
+
+## EKS
+
+The Armada UI (lookout) can be found at this URL:
+
+- [https://ui.demo.armadaproject.io](https://ui.demo.armadaproject.io)
+
+## Local prerequisites
+
+- Git
+- Go 1.20
+
+## Obtain the armada source
+Clone [this](https://github.com/armadaproject/armada) repository:
+
+```bash
+git clone https://github.com/armadaproject/armada.git
+cd armada
+```
+
+All commands are intended to be run from the root of the repository.
+
+## Setup an easy-to-use alias
+If you are on a Windows System, use a linux-supported terminal to run this command, for example [Git Bash](https://git-scm.com/downloads) or [Hyper](https://hyper.is/)
+```bash
+alias armadactl='go run cmd/armadactl/main.go --armadaUrl armada.demo.armadaproject.io:443'
+```
+
+## Create queues and jobs
+Create queues, submit some jobs, and monitor progress:
+
+### Queue Creation
+Use a unique name for the queue. Make sure you remember it for the next steps.
+```bash
+armadactl create queue $QUEUE_NAME --priorityFactor 1
+armadactl create queue $QUEUE_NAME --priorityFactor 2
+```
+
+For queues created in this way, user and group owners of the queue have permissions to:
+- submit jobs
+- cancel jobs
+- reprioritize jobs
+- watch queue
+
+For more control, queues can be created via `armadactl create`, which allows for setting specific permission; see the following example.
+
+```bash
+armadactl create -f ./docs/quickstart/queue-a.yaml
+armadactl create -f ./docs/quickstart/queue-b.yaml
+```
+
+Make sure to manually edit both of these `yaml` files using a code or text editor before running the commands above.
+
+```
+name: $QUEUE_NAME
+```
+
+### Job Submission
+```
+armadactl submit ./docs/quickstart/job-queue-a.yaml
+armadactl submit ./docs/quickstart/job-queue-b.yaml
+```
+
+Make sure to manually edit both of these `yaml` files using a code or text editor before running the commands above.
+```
+queue: $QUEUE_NAME
+```
+
+### Monitor Job Progress
+
+```bash
+armadactl watch $QUEUE_NAME job-set-1
+```
+```bash
+armadactl watch $QUEUE_NAME job-set-1
+```
+
+Try submitting lots of jobs and see queues get built and processed:
+
+#### Windows (using Git Bash):
+
+Use a text editor of your choice.
+Copy and paste the following lines into the text editor:
+```
+#!/bin/bash
+
+for i in {1..50}
+do
+ armadactl submit ./docs/quickstart/job-queue-a.yaml
+ armadactl submit ./docs/quickstart/job-queue-b.yaml
+done
+```
+Save the file with a ".sh" extension (e.g., myscript.sh) in the root directory of the project.
+Open Git Bash, navigate to the project's directory using the 'cd' command, and then run the script by typing ./myscript.sh and pressing Enter.
+
+#### Linux:
+
+Open a text editor (e.g., Nano or Vim) in the terminal and create a new file by running: nano myscript.sh (replace "nano" with your preferred text editor if needed).
+Copy and paste the script content from above into the text editor.
+Save the file and exit the text editor.
+Make the script file executable by running: chmod +x myscript.sh.
+Run the script by typing ./myscript.sh in the terminal and pressing Enter.
+
+#### macOS:
+
+Follow the same steps as for Linux, as macOS uses the Bash shell by default.
+With this approach, you create a shell script file that contains your multi-line script, and you can run it as a whole by executing the script file in the terminal.
+
+## Observing job progress
+
+CLI:
+
+```bash
+$ armadactl watch queue-a job-set-1
+Watching job set job-set-1
+Nov 4 11:43:36 | Queued: 0, Leased: 0, Pending: 0, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobSubmittedEvent, job id: 01drv3mey2mzmayf50631tzp9m
+Nov 4 11:43:36 | Queued: 1, Leased: 0, Pending: 0, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobQueuedEvent, job id: 01drv3mey2mzmayf50631tzp9m
+Nov 4 11:43:36 | Queued: 1, Leased: 0, Pending: 0, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobSubmittedEvent, job id: 01drv3mf7b6fd1rraeq1f554fn
+Nov 4 11:43:36 | Queued: 2, Leased: 0, Pending: 0, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobQueuedEvent, job id: 01drv3mf7b6fd1rraeq1f554fn
+Nov 4 11:43:38 | Queued: 1, Leased: 1, Pending: 0, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobLeasedEvent, job id: 01drv3mey2mzmayf50631tzp9m
+Nov 4 11:43:38 | Queued: 0, Leased: 2, Pending: 0, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobLeasedEvent, job id: 01drv3mf7b6fd1rraeq1f554fn
+Nov 4 11:43:38 | Queued: 0, Leased: 1, Pending: 1, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobPendingEvent, job id: 01drv3mey2mzmayf50631tzp9m
+Nov 4 11:43:38 | Queued: 0, Leased: 0, Pending: 2, Running: 0, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobPendingEvent, job id: 01drv3mf7b6fd1rraeq1f554fn
+Nov 4 11:43:41 | Queued: 0, Leased: 0, Pending: 1, Running: 1, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobRunningEvent, job id: 01drv3mf7b6fd1rraeq1f554fn
+Nov 4 11:43:41 | Queued: 0, Leased: 0, Pending: 0, Running: 2, Succeeded: 0, Failed: 0, Cancelled: 0 | event: *api.JobRunningEvent, job id: 01drv3mey2mzmayf50631tzp9m
+Nov 4 11:44:17 | Queued: 0, Leased: 0, Pending: 0, Running: 1, Succeeded: 1, Failed: 0, Cancelled: 0 | event: *api.JobSucceededEvent, job id: 01drv3mf7b6fd1rraeq1f554fn
+Nov 4 11:44:26 | Queued: 0, Leased: 0, Pending: 0, Running: 0, Succeeded: 2, Failed: 0, Cancelled: 0 | event: *api.JobSucceededEvent, job id: 01drv3mey2mzmayf50631tzp9m
+```
+
+Web UI:
+
+Open [https://ui.demo.armadaproject.io](https://ui.demo.armadaproject.io) in your browser.
+
+![Lookout UI](./quickstart/img/lookout.png "Lookout UI")
diff --git a/docs/design.md b/docs/design.md
new file mode 100644
index 00000000000..afd8210019e
--- /dev/null
+++ b/docs/design.md
@@ -0,0 +1,72 @@
+# System overview
+
+This document is meant to be an overview of Armada for new users. We cover the architecture of Armada, show how jobs are represented, and explain how jobs are queued and scheduled.
+
+If you just want to learn how to submit jobs to Armada, see:
+
+- [User guide](./user.md)
+
+## Architecture
+
+Armada consists of two main components:
+- The Armada server, which is responsible for accepting jobs from users and deciding in what order, and on which Kubernetes cluster, jobs should run. Users submit jobs to the Armada server through the `armadactl` command-line utility or via a gRPC or REST API.
+- The Armada executor, of which there is one instance running in each Kubernetes cluster that Armada is connected to. Each Armada executor instance regularly notifies the server of how much spare capacity it has available and requests jobs to run. Users of Armada never interact with the executor directly.
+
+All state relating to the Armada server is stored in [Redis](https://redis.io/), which may use replication combined with failover for redundancy. Hence, the Armada server is itself stateless and is easily replicated by running multiple independent instances. Both the server and the executors are intended to be run in Kubernetes pods. We show a diagram of the architecture below.
+
+![How Armada works](./assets/img/batch-api.svg)
+
+### Job leasing
+
+To avoid jobs being lost if a cluster (or cluster executor) becomes unavailable, each job assigned to an executor has an associated timeout. Armada executors are required to check in with the server regularly and, if an executor responsible for running a particular job fails to check in within that timeout, the server will re-schedule the job over another executor.
+
+## Jobs and job sets
+
+A job is the most basic unit of work in Armada, and is represented by a Kubernetes pod specification (podspec) with additional metadata specific to Armada. Armada handles creating, running, and removing containers as necessary for each job. Hence, Armada is essentially a system for managing the life cycle of a set of containerised applications representing a batch job.
+
+The Armada workflow is:
+
+1. Create a job specification, which is a Kubernetes podspec with a few additional metadata fields.
+2. Submit the job specification to one of Armada's job queues using the `armadactl` CLI utility or through the Armada gRPC or REST API.
+
+For example, a job that sleeps for 60 seconds could be represented by the following yaml file.
+
+```yaml
+queue: test
+jobSetId: set1
+jobs:
+ - priority: 0
+ podSpecs:
+ - terminationGracePeriodSeconds: 0
+ restartPolicy: Never
+ containers:
+ - name: sleep
+ imagePullPolicy: IfNotPresent
+ image: busybox:latest
+ args:
+ - sleep
+ - 60s
+ resources:
+ limits:
+ memory: 64Mi
+ cpu: 150m
+ requests:
+ memory: 64Mi
+ cpu: 150m
+```
+
+In the above yaml snippet, `podSpec` is a Kubernetes podspec, which consists of one or more containers that contain the user code to be run. In addition, the job specification (jobspec) contains metadata fields specific to Armada:
+
+- `queue`: which of the available job queues the job should be submitted to.
+- `priority`: the job priority (lower values indicate higher priority).
+- `jobSetId`: jobs with the same `jobSetId` can be followed and cancelled in a single operation. The `jobSetId` has no impact on scheduling.
+
+Queues and scheduling is explained in more detail below.
+
+For more examples, including of jobs spanning multiple nodes, see the [user guide](./user.md).
+
+### Job events
+
+A job event is generated whenever the state of a job changes (e.g., when changing from submitted to running or from running to completed) and is a timestamped message containing event-specific information (e.g., an exit code for a completed job). All events generated by jobs part of the same job set are grouped together and published via a [Redis stream](https://redis.io/topics/streams-intro). There are unique streams for each job set to facilitate subscribing only to events generated by jobs in a particular set, which can be done via the Armada API.
+
+Armada records all events necessary to reconstruct the state of each job and, after a job has been completed, the only information retained about the job is the events generated by it.
diff --git a/docs/design/architecture.md b/docs/design/architecture.md
new file mode 100644
index 00000000000..eacfcad41a8
--- /dev/null
+++ b/docs/design/architecture.md
@@ -0,0 +1,80 @@
+# Architecture
+
+Armada is designed to manage millions of batch jobs across compute clusters made up of potentially hundreds of thousands of nodes, while providing near-constant uptime. Hence, the Architecture of Armada must be highly resilient and scalable. The current architecture was chosen in early 2022 to achieve these goals while also ensuring new features, e.g., advanced scheduling techniques, can be delivered.
+
+At a high level, Armada is a so-called data stream system (sometimes referred to as an event sourcing system), for which there are two components responsible for tracking the state of the system:
+
+* A log-based message broker that stores state transitions durably in order, referred to throughout this document simply as "the log".
+* A set of databases, each deriving its state from the log (but are otherwise mutually independent) and storing a different so-called materialised view of the state of the system.
+
+The log is a publish-subscribe system consisting of multiple topics to which messages can be published. Those messages are eventually delivered to all subscribers of the topic. Important properties of the log are:
+
+* Durability: Messages published to the log are stored in a durable manner, i.e., they are not lost in case of up to x node failures, where x is a tuneable parameter.
+* Ordering: All subscribers of a topic see messages in the same order as they were published in, and replaying messages on a topic always results in the same message order. Further, all messages on the same topic are annotated with a message id that is monotonically increasing within each topic.
+
+In Armada, the log is implemented using Apache Pulsar.
+
+In a data stream system, the log is the source of truth and the databases an optimisation to simplify querying – since the databases can be re-constructed by replaying messages from the log, if the log was replayed for each query, although highly unpractical, the databases could be omitted. For example, in Armada there are separate PostgreSQL databases for storing jobs to be scheduled and the jobs to be shown in the UI, Lookout. Both of these derive their state from the log but are otherwise independent.
+
+To change the state of the system, a message (e.g., corresponding to a job being submitted) is published to the log. Later, that message is picked up by a log processor, which updates some database accordingly (in the case of a job being submitted, by storing the new job in the database). Hence, the log serialises state transitions and the database is a materialised view of part of the state of the system, as derived from the state transitions submitted to the log. In effect, a data stream system is a bespoke distributed database with the log acting as the transaction log.
+
+This approach has several benefits:
+
+* Resiliency towards bursts of high load: Because the log buffers state transitions, the components reading from the log and acting on those transitions are not directly affected by incoming requests.
+* Simplicity and extensibility: Adding new materialised views (e.g., for a new dashboard) can be accomplished by adding a new subscriber to the log. This new subscriber has the same source of truth as all others (i.e., the log) but is loosely coupled to those components; adding or removing views does not affect other components of the system.
+* Consistency: When storing state across several independent databases, those databases are guaranteed to eventually be consistent; there is no failure scenario where the different databases become permanently inconsistent, thus requiring a human to manually reconcile them (assuming acting on state transitions is idempotent).
+
+However, the approach also has some drawbacks:
+
+* Eventual consistency: Because each database is updated from the log independently, they do not necessarily represent the state of the system at the same point of time. For example, a job may be written to the scheduler database (thus making it eligible for scheduling) before it shows up in the UI.
+* Timeliness: Because databases are updated from the log asynchronously, there may be a lag between a message being published and the system being updated to reflect the change (e.g., a submitted job may not show up in the UI immediately).
+
+## System overview
+
+Besides the log, Armada consists of the following components:
+
+* Submit API: Clients (i.e., users) connect to this API to request state transitions (e.g., submitting jobs or updating job priorities) and each such state transition is communicated to the rest of the system by writing to the log (more detail on this below).
+* Streams API: Clients connect to this API to subscribe to log messages for a particular set of jobs. Armada components can receive messages either via this API or directly from the log, but users have to go via the streams API to isolate them from internal messages.
+* Scheduler: A log processor responsible for maintaining a global view of the system and preempting and scheduling jobs. Preemption and scheduling decisions are communicated to he rest of the system by writing to the log.
+* Executors: Each executor is responsible for one Kubernetes worker cluster and is the component that communicates between the Armada scheduler and the Kubernetes API of the cluster it is responsible for.
+* Lookout: The web UI showing the current state of the system. Lookout maintains its views by reading log messages to populate its database.
+
+### Job submission logic
+
+Here, we outline the sequence of actions resulting from submitting a job.
+
+1. A client submits a job to the submit-query API, which is composed of a Kubernetes podspec and some Armada-specific metadata (e.g., the priority of the job).
+2. The submit API authenticates and authorizes the user, validates the submitted job, and, if valid, submits the job spec. to the log. The submit API annotates each job with a randomly generated UUID that uniquely identifies the job. This UUID is returned to the user.
+3. The scheduler receives the job spec. and stores it in-memory (discarding any data it doesn't need, such as the pod spec.). The scheduler runs periodically, at which point it schedules queued jobs. At the start of each scheduling run, the scheduler queries each executor for its available resources. The scheduler uses this information in making scheduling decisions. When the scheduler assigns a job to an executor, it submits a message to the log indicating this state transition. It also updates its in-memory storage immediately to reflect the change (to avoid scheduling the same job twice).
+4. A log processor receives the message indicating the job was scheduled, and writes this decision to a database acting as the interface between the scheduler and the executor.
+5. Periodically, each executor queries the database for the list of jobs it should be running. It compares that list with the list of jobs it is actually running and makes changes necessary to reconcile any differences.
+6. When a job has finished, the executor responsible for running the job informs the scheduler, which on its behalf submits a "job finished" message to the log. The same log processor as in step 4. updates its database to reflect that the job has finished.
+
+### Streams API
+
+Armada does not maintain a user-queryable database of the current state of the system. This is by design to avoid overloading the system with connections. For example, say there is one million active jobs in the system and that there are clients who want to track the state of all of those jobs. With a current-state-of-the-world database, those client would need to resort to polling that database to catch any updates, thus opening a total of one million connections to the database, which, while not impossible to manage, would pose significant challenges.
+
+Instead, users are expected to be notified of updates to their jobs via an event stream (i.e., the streams API), where a client opens a single connection for all jobs in a so-called job set over which all state transitions are streamed as they happen. This approach is highly scalable since data is only sent when something happens and since a single connection that contain updates for thousands of jobs. Users who want to maintain a view of their jobs are thus responsible for maintaining that view themselves by subscribing to events.
+
+## Notes on consistency
+
+The data stream approach taken by Armada is not the only way to maintain consistency across views. Here, we compare this approach with the two other possible solutions.
+
+Armada stores its state across several databases. Whenever Armada receives an API call to update its state, all those databases need to be updated. However, if each database were to be updated independently it is possible for some of those updates to succeed while others fail, leading to an inconsistent application state. It would require complex logic to detect and correct for such partial failures. However, even with such logic we could not guarantee that the application state is consistent; if Armada crashes before it has had time to correct for the partial failure the application may remain in an inconsistent state.
+
+There are three commonly used approaches to address this issue:
+
+* Store all state in a single database with support for transactions. Changes are submitted atomically and are rolled back in case of failure; there are no partial failures.
+* Distributed transaction frameworks (e.g., X/Open XA), which extend the notation of transactions to operations involving several databases.
+* Ordered idempotent updates.
+
+The first approach results in tight coupling between components and would limit us to a single database technology. Adding a new component (e.g., a new dashboard) could break existing component since all operations part of the transaction are rolled back if one fails. The second approach allows us to use multiple databases (as long as they support the distributed transaction framework), but components are still tightly coupled since they have to be part of the same transaction. Further, there are performance concerns associated with these options, since transactions may not be easily scalable. Hence, we use the third approach, which we explain next.
+
+First, note that if we can replay the sequence of state transitions that led to the current state, in case of a crash we can recover the correct state by truncating the database and replaying all transitions from the beginning of time. Because operations are ordered, this always results in the same end state. If we also, for each database, store the id of the most recent transition successfully applied to that database, we only need to replay transitions more recent than that. This saves us from having to start over from a clean database; because we know where we left off we can keep going from there. For this to work, we need transactions but not distributed transactions. Essentially, applying a transition already written to the database results in a no-op, i.e., the updates are idempotent (meaning that applying the same update twice has the same effect as applying it once).
+
+The two principal drawbacks of this approach are:
+
+* Eventual consistency: Whereas the first two approaches result in a system that is always consistent, with the third approach, because databases are updated independently, there will be some replication lag during which some part of the state may be inconsistent.
+* Timeliness: There is some delay between submitting a change and that change being reflected in the application state.
+
+Working around eventual consistency requires some care, but is not impossible. For example, it is fine for the UI to show the a job as "running" for a few seconds after the job has finished before showing "completed". Regarding timeliness, it is not a problem if there is a few seconds delay between a job being submitted and the job being considered for queueing. However, poor timeliness may lead to clients (i.e., the entity submitting jobs to the system) not being able to read their own writes for some time, which may lead to confusion (i.e., there may be some delay between a client submitting a job a that job showing as "pending"). This issue can be worked around by storing the set of submitted jobs in-memory either at the client or at the API endpoint.
diff --git a/docs/design/database_interfaces.md b/docs/design/database_interfaces.md
new file mode 100644
index 00000000000..0003e7b00e7
--- /dev/null
+++ b/docs/design/database_interfaces.md
@@ -0,0 +1,187 @@
+# Armada Database Interfaces
+
+## Problem Description
+
+Open source projects should not be hard coded to a particular Database. Armada currently only allows users to use Postgres. This project is to build interfaces around our connections to Postgres so we can allow other databases.
+
+## Solution
+
+1. Introduce base common database interfaces that can be shared reused by all components (Lookout, Scheduler, Scheduler Ingester).
+2. Add interfaces that abstracts the hardcoded Postgres configuration.
+3. Add interfaces around `pgx` structs.
+
+### Functional Specification (API Description)
+
+#### Database Connection
+
+Most of the components (Lookout, Scheduler, Scheduler Ingester) rely on [PostgresConfig](https://github.com/armadaproject/armada/blob/master/internal/armada/configuration/types.go#L294) to connect to external databases, we can avoid hardcoding the configuration of those components to use `PostgresConfig` but defining a generic `DatabaseConfig` interface that's when implemented will provide those components with the necessary details to connect to databases.
+
+ /**
+ Components configuration (e.g. LookoutConfiguration) can now make use of this interface instead of hardcoding PostgresConfig.
+ */
+ type DatabaseConfig interface {
+ GetMaxOpenConns() int
+ GetMaxIdleConns() int
+ GetConnMaxLifetime() time.Duration
+ GetConnectionString() string
+ }
+
+ type DatabaseConnection interface {
+ GetConnection() (*sql.DB, error)
+ GetConfig() DatabaseConfig
+ }
+
+The existing configurations can then be tweaked to use the new generic `DatabaseConfig` interface instead of hardcoding `PostgresConfig`
+
+ type LookoutConfiguration struct {
+ Postgres PostgresConfig // this can be replaced with the new Database property
+ Database DatabaseConfig // new property
+ }
+
+#### Database Communication
+
+Currently, most of the Armada components make use of the `github.com/jackc/pgx` Postgres client which provides APIs to interact exclusively with Postgres databases, this makes Armada tightly coupled with Postgres and makes it impossible to use other SQL dialects (e.g. MySQL).
+
+A way to fix this would be to design database-agnostic interfaces that can abstract away the existing Postgres core implementation (pgx), and then implement adapters around `pgx` that implement those interfaces. This will allow for having a high level abstraction API for interacting with databases while maintaining the existing Postgres core implementation.
+To accomplish this, we will need to define interfaces for the following features:
+
+1. Connection Handler
+
+ // DatabaseConn represents a connection handler interface that provides methods for managing the open connection, executing queries, and starting transactions.
+ type DatabaseConn interface {
+ // Close closes the database connection. It returns any error encountered during the closing operation.
+ Close(context.Context) error
+
+ // Ping pings the database to check the connection. It returns any error encountered during the ping operation.
+ Ping(context.Context) error
+
+ // Exec executes a query that doesn't return rows. It returns any error encountered.
+ Exec(context.Context, string, ...any) (any, error)
+
+ // Query executes a query that returns multiple rows. It returns a DatabaseRows interface that allows you to iterate over the result set, and any error encountered.
+ Query(context.Context, string, ...any) (DatabaseRows, error)
+
+ // QueryRow executes a query that returns one row. It returns a DatabaseRow interface representing the result row, and any error encountered.
+ QueryRow(context.Context, string, ...any) DatabaseRow
+
+ // BeginTx starts a transcation with the given DatabaseTxOptions, or returns an error if any occured.
+ BeginTx(context.Context, DatabaseTxOptions) (DatabaseTx, error)
+
+ // BeginTxFunc starts a transaction and executes the given function within the transaction. It the function runs successfuly, BeginTxFunc commits the transaction, otherwise it rolls back and return an errorr.
+ BeginTxFunc(context.Context, DatabaseTxOptions, func(DatabaseTx) error) error
+ }
+
+2. Connection Pool
+
+ // DatabasePool represents a database connection pool interface that provides methods for acquiring and managing database connections.
+ type DatabasePool interface {
+ // Acquire acquires a database connection from the pool. It takes a context and returns a DatabaseConn representing the acquired connection and any encountered error.
+ Acquire(context.Context) (DatabaseConn, error)
+
+ // Ping pings the database to check the connection. It returns any error encountered during the ping operation.
+ Ping(context.Context) error
+
+ // Close closes the database connection. It returns any error encountered during the closing operation.
+ Close()
+
+ // Exec executes a query that doesn't return rows. It returns any error encountered.
+ Exec(context.Context, string, ...any) (any, error)
+
+ // Query executes a query that returns multiple rows. It returns a DatabaseRows interface that allows you to iterate over the result set, and any error encountered.
+ Query(context.Context, string, ...any) (DatabaseRows, error)
+
+ // BeginTx starts a transcation with the given DatabaseTxOptions, or returns an error if any occured.
+ BeginTx(context.Context, DatabaseTxOptions) (DatabaseTx, error)
+
+ // BeginTxFunc starts a transaction and executes the given function within the transaction. It the function runs successfuly, BeginTxFunc commits the transaction, otherwise it rolls back and return an errorr.
+ BeginTxFunc(context.Context, DatabaseTxOptions, func(DatabaseTx) error) error
+ }
+
+3. Transaction
+
+ // DatabaseTx represents a database transaction interface that provides methods for executing queries, managing transactions, and performing bulk insertions.
+ type DatabaseTx interface {
+ // Exec executes a query that doesn't return rows. It returns any error encountered.
+ Exec(context.Context, string, ...any) (any, error)
+
+ // Query executes a query that returns multiple rows. It returns a DatabaseRows interface that allows you to iterate over the result set, and any error encountered.
+ Query(context.Context, string, ...any) (DatabaseRows, error)
+
+ // QueryRow executes a query that returns one row. It returns a DatabaseRow interface representing the result row, and any error encountered.
+ QueryRow(context.Context, string, ...any) DatabaseRow
+
+ // CopyFrom performs a bulk insertion of data into a specified table. It accepts the table name, column names, and a slice of rows representing the data to be inserted. It returns the number of rows inserted and any error encountered.
+ CopyFrom(ctx context.Context, tableName string, columnNames []string, rows [][]any) (int64, error)
+
+ // Commit commits the transaction. It returns any error encountered during the commit operation.
+ Commit(context.Context) error
+
+ // Rollback rolls back the transaction. It returns any error encountered during the rollback operation.
+ Rollback(context.Context) error
+ }
+
+4. Result Row
+
+ // DatabaseRow represents a single row in a result set.
+ type DatabaseRow interface {
+ // Scan reads the values from the current row into dest values positionally. It returns an error if any occured during the read operation.
+ Scan(dest ...any) error
+ }
+
+5. Resultset
+
+ // DatabaseRows represents an interator over a result set.
+ type DatabaseRows interface {
+ // Close closes the result set.
+ Close() error
+
+ // Next moves the iterator to the next row in the result set, it returns false if the result set is exhausted, otherwise true.
+ Next() bool
+
+ // Err returns the error, if any, encountered during iteration over the result set.
+ Err() error
+
+ // Scan reads the values from the current row into dest values positionally. It returns an error if any occured during the read operation.
+ Scan(dest ...any) error
+ }
+
+### Implementation Plan
+
+Designing interfaces that can remove the coupling between Armada and Postgres while maintaining the existing core Postgres implementation is a requirement.
+
+To fullfill this requirement, we can implement adapters around the `pgx` client so that it also implements the interfaces defined above.
+
+For example, an adapter can be implemented for `pgxpool.Pool` so that it can be used with `DatabasePool`:
+
+ type PostgresPoolAdapter struct {
+ *pgxpool.Pool
+ }
+
+ func (p PostgresPoolAdapter) Exec(ctx context.Context, sql string, args ...any) (any, error) {
+ return p.Pool.Exec(ctx, sql, args)
+ }
+
+ func (p PostgresPoolAdapter) BeginTxFunc(ctx context.Context, opts dbtypes.DatabaseTxOptions, action func(dbtypes.DatabaseTx) error) error {
+ tx, err := p.Pool.BeginTx(ctx, pgx.TxOptions{
+ IsoLevel: pgx.TxIsoLevel(opts.Isolation),
+ DeferrableMode: opts.DeferrableMode,
+ AccessMode: pgx.TxAccessMode(opts.AccessMode),
+ })
+
+ if err != nil {
+ return err
+ }
+
+ // PostgresTrxAdapter is the Postgres adapter for DatabaseTx interface
+ if err := action(PostgresTrxAdapter{Tx: tx}); err != nil {
+ return tx.Rollback(ctx)
+ }
+
+ return tx.Commit(ctx)
+ }
+
+The example above showcases the implementation of a Postgres connection pool adapter, this example implements the `DatabasePool` interface (the rest of the methods can be implemented similarly to `Exec` and `BeginTxFunc`).
+
+This allows the components that make use `pgxpool.Pool` (e.g. Lookout) to switch to using `DatabasePool` which underneath can make use of `pgxpool.Pool` (or any other `DatabasePool` implementation) without making any changes to the core Postgres implementation.
+
+To support new SQL dialects, we can simply introduce adapters that implement the interfaces, as well as introduce some level of flexibility into the configuration of components to allow choosing which dialect we want to use.
diff --git a/docs/design/diagrams/relationships/README.md b/docs/design/diagrams/relationships/README.md
new file mode 100644
index 00000000000..1360c3f904a
--- /dev/null
+++ b/docs/design/diagrams/relationships/README.md
@@ -0,0 +1,25 @@
+# Diagrams of Armada Architecture
+
+## Generating the Diagram
+
+To generate this diagram, you can use the following command:
+
+```bash
+# install graphviz
+sudo apt-get install graphviz
+
+# then install diagrams from pip
+pip install diagrams
+
+# then run the following command to generate the diagram
+python3 generate.py
+```
+
+To find out more about the diagrams library, see https://diagrams.mingrammer.com/
+To find out more about the graphviz library, see https://graphviz.org/
+
+
+
+
+
+
diff --git a/docs/design/diagrams/relationships/armada_system.png b/docs/design/diagrams/relationships/armada_system.png
new file mode 100644
index 0000000000000000000000000000000000000000..f6c953403610d69451b02868e4a204f5f8e58dad
GIT binary patch
literal 178826
zcmeFZg^?cwcP;0J-BoU|0q1@zySy39x%oWF5oq$E^a6W6EQYSG5;4tKWTe0^_T
zKe;@4C=26uX|m0qG`DZh>u=9QdTPdh9*h-KAhlXj8yk|_=rp~NB(2P?jGTP*jG>9d
zR;KM1{!IzzZQGa=uXCI`cSs2ZmKzl_4o~*E7S44K_iG9~v1DF-PSMQKvJ7A@zmJ<4
zIlu7#^X9ai9L)I7ar0#(?msVIao`gE^YTg4W$J%k;^6=9!IIGtg
zJbZjhWrGTG%F4>`9^*mdN&jr5uF}rAx^ztgrUD1!mspzkCg%m}W6RBp!LgrYZs@4zK-e8~Lls}ddh1Wkj%h%8!V
zk4zKoo@;4q^UrmGj}2P>xc>H6+X<`6#TadC&9(1fVEz$Zm#OC(*TE^4A_cD?qkXY;
zSnBC4deg!MYY%d|J5#9N@G_oNbRxCFPw)fh%PYXuXO7X8Xu;_hF@$uf
z&x2*Fbf$fN-(oQGk4bN?)}L&4hfC8|O>;iaR_T)GpNg6f75Bo+CkB_`qy59){UCcD
z0Nd7{?sn50pQE*Hl9-e;()1g=bpIIkyUtgzQaDfE)MJvi90jIdM8t8ME=@Y1f>U0D
z4-Ax-FOx!(y2j~sdPMEH-DGBFwlXB5s;Y`@p*s^gbTcE&IJhM}MjWOGh~bTQ8Vg
z!g8`l!IT|1lL^snkz})+A$e=ixGZ16af$)!q*(oN%b7+9>pE(XJ0<>Mzc=
zmLKb5yR?vX$E&$j+s#BZhGyWDS6BXcO3`ciWD&uWjY|Fa`1o-emjM|@-dw=O-FZwLdPfg=^wv%UTQg35=1S?Mb4;T=`wkKvXvqZ
zd~5|wDZ=PreN6DD58iIq5&6*S@DZqIL6a@&KV3H!*S4rF
zK0BCLh8Fye4Sgs083$I96@I*Dc36WFJF#S|{`HMb^D+2_<;y?oYLFT$7WHDCVhh2{w%zu
zEh{T4_IRXW+Pe0&u5Ob4`7XjDh(BWCS88aVF+w*+DZ8wlQ(j(PaHDLn1l+H}pXopR
z;}KMoydyf1;%6(08zF*QwHW)kPmBdl1*LsLk28~;wQ68cl
z^>VBmf!vWQfn4DCRSdco?dx}fZOW)@plgljd29$FsbrM&^;2d}m-xrvhf^a0CY=!s
z*zuA2n5CtqeJyVA#hVg;){g*_Fmw9w0-JJC)xLaoa)`3=KAEtr7&EaFYgoxh;If|l
z&X;0U+;+QaAFW;5c3V5o?wpX$lVe23q4${eF1XUQpMT@1tg^iSe%=U*mW!)Dlo@
zYikaNQ!dLteaZATG}Dzj%SZGQEDEX~^78TuUYueuaVXD>UboGLi*ss^^y-%Ms=l77p{l{1t9br6YRcc^?Br)Y|X>bKZXQ^A9{#_u!F;K&Q8S2o~gYW9HoIdGOun)zwwi
zt|qADsii}&i<6n<(v-7xp#~x_0w2MjxY0G-nQI#Z$c~u{WG^_#tEk|soOZVe0VAl2
z>6aMEuzmme{$-*-vE-J>QY_4V+HE;KBf}&bMd$kS9ktWYYcoqroekj$n+Etp`S_@M
z_=)qcByjzrdn%!{qVA6Uf#3~}!JtbMKE*;9I2nDWQcksD@d5d@q`viCWb&Fw;h
zkS1!&>#)Y`?c28)>|~V#287EJnfH<8X3bVTK;Z`*kAgsOZgqGD0;^y*wk0aEeRb?Iyg+)A)PvPz$vdj
zk
z@GR6~`sAFPoG1!@=TWggt50Kpwm-6byjHk1L`_I-`=ov^o!#kIO}Y}3X$sugdE6{h
zA%9lVzfnN18#9d2d75`bTx}y3!0pOkoSzAcs5>|~Kt7n6
z>~3CH@6yAGTsXy`wuY|K0$5fcAh`=1HNrI6Ekmxr5nRM4#P-jfuca$x8x3W>juZ88
z1}>XaiV_n}!ru8hK0bc>lTf?^vR>Z|tWmU`kMU#C6?u;bz?1}5~2
z_|JWORL7`d4DiNTK-kmBcd09g~tLuh#3zJjvZ(D1O+YLd&e
zo2hm6N(T4>pgTf!WYFav{gWH}mim$szDfp|9Ph2W|WOGQBS*8R){&G%kLDxG%+V5S<5J5?3z$WN;VHcA^
z_<{Ucf3{&$cpK#EpgY{vMs(n!MfHDJWf7A`(cjtGGVXhiEU&_lA-$$zn7FcviVAFP
zbv5K%ilpMF+X@Z2I&90px*df8!g+wJ^l+nkI>H|ai`tY}pL{>rtZ`_)F74QP0xoT!
zd*ye=om
zwXSvm?*{X=E7=lkptFXul_zK_0lQf2WK3D#Gy6UDfAT46?c<+EU&hDNsszW}*A}|<
z70yR4WW^4O6&fmvi#egddZ?IOt2aZ%XQ@F8-|HV58F6tZ(fyiwLJ86oG={&wKgn;p
zsZo#tG;Er^U#XpYE6)m{Y9n|t&UD=S{CtIy+w>vuUi(gl&My+3os5i(iD8zNx&e9~
zJ8cw&E_-40k&KFc3<#YF@3Rr_jUOI02gq%($xLzXxpuVFj@L8fx
z*2?-*~BjCm~$6o4g
z2V!
z8K3BCXg?s+ec>evIEe8=M|d2^o2P`gPxMIze)@Oq3)Qq@PEI
zZVib*>(L%er9KKp`B6&YKSKWxnJsC2Ji>?iCtAA;k)3kv45`vCkAQPLF03sX1ci;r
z@vqb>Kz)Tp6{U6E7NbGPL>NIU-p20LkN+4lZv`lb_*8#~;nCXejN;F%Y#A-Bh+ck0
z1%Z3mT}0$;3qn6`Kn8+!e-f$~2Dz9r;Q~-;Xx6&!XccO#jFp-~sZUp!GSYFVo4p}X
z(Q}Smv+V8bn)&1XH3T?6Z~t>NNQOqc3!PMO$jwik{(xR%24Izo>Zy}1`n#Jooq!qy
zQT8+JJeQM`gNniOb}AAQk}5#R9olZGrkB`T5_X)uAcb