Skip to content

Commit 8c40fad

Browse files
ahsimbahsimb
committed
Merge remote-tracking branch 'origin/main'
2 parents ee5bd0e + d8670e2 commit 8c40fad

File tree

5 files changed

+174
-1
lines changed

5 files changed

+174
-1
lines changed

‎README.md

+14-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,16 @@
11
# Exasol Python Extension Common
22

3-
Content to be provided.
3+
A package with common functionality, shared by Exasol Python Extensions, e.g.
4+
* [transformers-extension](https://github.com/exasol/transformers-extension)
5+
* [sagemaker-extension](https://github.com/exasol/sagemaker-extension)
6+
7+
## Features
8+
9+
A deployer for script language containers (SLC) to be used by UDF-based extensions of Exasol database requiring a special SLC.
10+
11+
## More documentation
12+
13+
* User Guide
14+
* [Developer Guide](doc/developer-guide.md)
15+
* [User Defined Functions (UDF)](https://docs.exasol.com/db/latest/database_concepts/udf_scripts.htm)
16+
* [Script Language Containers (SLC)](https://github.com/exasol/script-languages-release/)

‎doc/changes/unreleased.md

+5
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,8 @@
1010
## Refactorings
1111

1212
* #15: Used plugin pytest-saas
13+
14+
15+
## Documentation
16+
17+
* #22: Added comment on using fixtures from pytest-plugin `pytest-exasol-saas`

‎doc/developer-guide.md

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Python Extension Common (PEC) Developer Guide
2+
3+
## Pytest Plugins
4+
5+
PEC declares a dependency to pytest plugin `pytest-exasol-saas` which are maintained in GitHub repository [pytest-plugins/pytest_saas](https://github.com/exasol/pytest-plugins/tree/main/pytest-saas/).
6+
7+
This plugin makes additional fixtures available that are used in the integration tests of PEC.
8+
See files in folder [test/integration](../tree/main/test/integration):
9+
10+
* `conftest.py`
11+
* `test_language_container_deployer_saas.py`
12+
* `test_language_container_deployer_saas_cli.py`
13+
14+
## Running Tests in CI Builds
15+
16+
The test cases in PEC are separated in two groups.
17+
18+
| Group | Execution | Name of gating GitHub workflow |
19+
|-----------------------------|-----------------------------------------|--------------------------------|
20+
| G1) Fast and cheap tests | On each push to your development branch | Gate 1 - Regular CI |
21+
| G2) Slow or expensive tests | Only on manual approval, see below | Gate 2 - Allow Merge |
22+
23+
This enables fast development cycles while still protecting the main branch against build failures.
24+
25+
For PEC group G2 particularly contains the tests involving Exasol SaaS infrastructure which are creating costs for the database instances temporarily created during test execution.
26+
27+
Group G2 is guarded by a dedicated [GitHub Enviroment](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers) requiring **manual approval** before these tests are executed.
28+
29+
Each of the groups results in a final gating GitHub workflow job that is added to the branch protection of branch `main`.
30+
31+
So in order to merge a branch to `main` branch, the tests of both groups need to be executed and to have terminated succesfully.
32+
33+
### Approving Slow Tests
34+
35+
To approve executing the tests in group G2
36+
* Open your pull request in GitHub
37+
* Scroll to section "Checks"
38+
* Locate pending tasks, e.g. "Ask if Slow or Expensive Tests (e.g. SaaS) Should be Run"
39+
* Click the link "Details" on the right-hand side
40+
* Click "Review pending Deplopyments"
41+
* Select the checkbox "slow-tests"
42+
* Click the green button "Approve and deploy"

‎doc/user_guide/user-guide.md

+110
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Python Extension Common User Guide
2+
3+
## Language Container Deployer
4+
5+
An extension would typically use [UDF scripts](https://docs.exasol.com/db/latest/database_concepts/udf_scripts.htm)
6+
to enable certain custom functionality within a database. In most cases, UDF scripts must be backed by a
7+
[Script Language Container (SLC)](https://github.com/exasol/script-languages-release/), that must be installed in the
8+
Exasol Database. An SLC allows the installation of the chosen programming language and necessary dependencies in the
9+
Exasol Database.
10+
11+
The language container for a particular Extension gets downloaded and installed by executing a deployment script
12+
similar to the one below.
13+
14+
```buildoutcfg
15+
python -m <exasol_extension>.deploy language-container <options>
16+
```
17+
18+
The name of the script (```<exasol_extension>.deploy``` in the above command) can vary from one extension to another.
19+
Please check the user guide of a particular extension. The rest of the command line will have a common format. It
20+
will include the command - ```language-container``` - and selected options. The choice of options is primarily
21+
determined by the storage backend being used - On-Prem or SaaS.
22+
23+
### List of options
24+
25+
The table below lists all available options. It shows which ones are applicable for On-Prem and for SaaS backends.
26+
Unless stated otherwise in the comments column, the option is required for either or both backends.
27+
28+
Some of the values, like passwords, are considered confidential. For security reasons, it is recommended to store
29+
those values in environment variables instead of providing them in the command line. The names of the environment
30+
variables are given in the comments column, where applicable. Alternatively, it is possible to put just the name of
31+
an option in the command line, without providing its value. In this case, the command will prompt to enter the value
32+
interactively. For long values, such as the SaaS account id, it is more practical to copy/paste the value from
33+
another source.
34+
35+
| Option name | On-Prem | SaaS | Comment |
36+
|:-----------------------------|:-------:|:----:|:--------------------------------------------------|
37+
| dsn | [x] | | i.e. <db_host:db_port> |
38+
| db-user | [x] | | |
39+
| db-pass | [x] | | Env. [DB_PASSWORD] |
40+
| bucketfs-name | [x] | | |
41+
| bucketfs-host | [x] | | |
42+
| bucketfs-port | [x] | | |
43+
| bucketfs-user | [x] | | |
44+
| bucketfs-password | [x] | | Env. [BUCKETFS_PASSWORD] |
45+
| bucketfs-use-https | [x] | | Optional boolean, defaults to False |
46+
| bucket | [x] | | |
47+
| saas-url | | [x] | |
48+
| saas-account-id | | [x] | Env. [SAAS_ACCOUNT_ID] |
49+
| saas-database-id | | [x] | Optional, Env. [SAAS_DATABASE_ID] |
50+
| saas-database-name | | [x] | Optional, provide if the database_id is unknown |
51+
| saas-token | | [x] | Env. [SAAS_TOKEN] |
52+
| path-in-bucket | [x] | [x] | |
53+
| language-alias | [x] | [x] | |
54+
| version | [x] | [x] | Optional, provide for downloading SLC from GitHub |
55+
| container-file | [x] | [x] | Optional, provide for uploading SLC file |
56+
| ssl-cert-path | [x] | [x] | Optional |
57+
| [no_]use-ssl-cert-validation | [x] | [x] | Optional boolean, defaults to True |
58+
| ssl-client-cert-path | [x] | | Optional |
59+
| ssl-client-private-key | [x] | | Optional |
60+
| [no_]upload-container | [x] | [x] | Optional boolean, defaults to True |
61+
| [no_]alter-system | [x] | [x] | Optional boolean, defaults to True |
62+
| [dis]allow-override | [x] | [x] | Optional boolean, defaults to False |
63+
| [no_]wait_for_completion | [x] | [x] | Optional boolean, defaults to True |
64+
65+
### Container selection
66+
67+
A language container can be deployed in two ways.
68+
69+
* By telling the script to download a particular version of the container from GitHub,
70+
using the `--version` option. The available versions can be found at the extension's releases page on GitHub.
71+
* By providing a path to the container's archive (*.tar.gz) stored in a local file system,
72+
using the `--container-file` option. The container can be downloaded from GitHub before
73+
executing the deployment script.
74+
75+
### TLS/SSL options
76+
77+
The `--ssl-cert-path` is needed if the TLS/SSL certificate is not in the OS truststore.
78+
Generally speaking, this certificate is a list of trusted CA. It is needed for the server's certificate
79+
validation by the client.
80+
The option `--use-ssl-cert-validation`is the default, it can be disabled with `--no-use-ssl-cert-validation`.
81+
One needs to exercise caution when turning the certificate validation off as it potentially lowers the security of the
82+
Database connection.
83+
The "server" certificate described above shall not be confused with the client's own certificate.
84+
In some cases, this certificate may be requested by a server. The client certificate may or may not include
85+
the private key. In the latter case, the key may be provided as a separate file.
86+
87+
### Language container activation
88+
89+
By default, the deployment command will upload and activate the language container at the System level.
90+
The latter requires the user to have the System Privileges, as it will attempt to change DB system settings.
91+
If such privileges cannot be granted the activation can be skipped by using the `--no-alter-system` option.
92+
The command will then print two possible language activation SQL queries, which look like the following:
93+
```sql
94+
ALTER SESSION SET SCRIPT_LANGUAGES=...
95+
ALTER SYSTEM SET SCRIPT_LANGUAGES=...
96+
```
97+
These queries represent two alternative ways of activating a language container. The first one activates the
98+
container at the [Session level](https://docs.exasol.com/db/latest/sql/alter_session.htm). It doesn't require
99+
System Privileges. However, it must be run every time a new session starts. The second one activates the container
100+
at the [System level](https://docs.exasol.com/db/latest/sql/alter_system.htm). It needs to be run just once,
101+
but it does require System Privileges. It may be executed by a database administrator. Please note, that changes
102+
made at the system level only become effective in new sessions, as described
103+
[here](https://docs.exasol.com/db/latest/sql/alter_system.htm#microcontent1).
104+
105+
It is also possible to activate the language without repeatedly uploading the container. If the container
106+
has already been uploaded one can use the `--no-upload-container` option to skip this step.
107+
108+
By default, overriding language activation is not permitted. If a language with the same alias has already
109+
been activated the command will result in an error. The activation can be overridden with the use of
110+
the `--allow-override` option.

‎error_code_config.yml

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
error-tags:
2+
PEC:
3+
highest-index: 0

0 commit comments

Comments
 (0)