Skip to content

Commit 7ed9960

Browse files
romer8swainn
romer8
and
swainn
authored
Bug Fix: Replacing query parameter /status?r=<id> to /status/<id> on jobs table (#155)
* Enhance database connection handling by adding engine_args parameter for customizable engine options * Refactor resource status handling and update URL mappings for resource status list * Add devcontainer configuration and Docker Compose files for development setup * Fix workflow ID assertions in CondorWorkflowManagerTests to reflect correct job IDs * Add README for devcontainer usage instructions * Remove docker-compose.test.yml file to streamline project configuration * first draft devcontainer * Refactor code for improved readability by adjusting formatting and removing unused imports * Update devcontainer configuration and scripts for improved setup and testing * Update README to reflect changes in test script execution * Fix for workflow_job_decorator to fix job_func required error * Fix workflow_step_job decorator so that it doesn't require being called with () * Remove whitespace * compatibel with previous url styles * Refactor resource ID handling in ResourceStatus class * Update URL patterns in resource status tests for consistency * lint edits * Fix formatting of comments in workflow_step_job for clarity * Update workflow IDs in CondorWorkflowManagerTests to reflect changes in job assignments * Refactor resource ID retrieval in ResourceStatus class . Resource is available, so retrieved from it --------- Co-authored-by: Nathan Swain <swainn@users.noreply.github.com> Co-authored-by: Nathan <nswain@aquaveo.com>
1 parent b416e5e commit 7ed9960

20 files changed

Lines changed: 550 additions & 116 deletions

File tree

‎.devcontainer/Dockerfile.devcontainer‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Use our Tethyscore base docker image as a parent image
2+
ARG PYTHON_VERSION=3.12
3+
ARG DJANGO_VERSION=3.2
4+
ARG TETHYS_VERSION=4.3.7
5+
ARG BASE_IMAGE_TAG="${TETHYS_VERSION}-py${PYTHON_VERSION}-dj${DJANGO_VERSION}"
6+
ARG BASE_IMAGE="tethysplatform/tethys-core"
7+
8+
# Use our Tethys Core base docker image as a parent image
9+
FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}
10+
11+
# This is necessary for the ARG variables to be available in the rest of the Dockerfile
12+
ARG PYTHON_VERSION
13+
ARG DJANGO_VERSION
14+
ARG TETHYS_VERSION
15+
16+
RUN echo "Python version ${PYTHON_VERSION}" \
17+
; echo "Django version ${DJANGO_VERSION}" \
18+
; echo "Tethys version ${TETHYS_VERSION}"
19+
20+
#####################
21+
# Default Variables #
22+
#####################
23+
ENV TETHYSAPP_DIR=/var/www/tethys/apps
24+
ENV TETHYSEXT_DIR=/var/www/tethys/exts
25+
ENV TETHYS_PUBLIC_HOST='localhost'
26+
ENV JOBS_TABLE_REFRESH_INTERVAL=30000
27+
ENV NGINX_PORT=8080
28+
ENV APP_DB_HOST=${TETHYS_DB_HOST}
29+
ENV APP_DB_PORT=${TETHYS_DB_PORT}
30+
ENV APP_DB_USERNAME=${TETHYS_DB_USERNAME}
31+
ENV APP_DB_PASSWORD=${TETHYS_DB_PASSWORD}
32+
#########
33+
# SETUP #
34+
#########
35+
RUN mkdir -p "${TETHYSAPP_DIR}" \
36+
; mkdir -p "${TETHYSEXT_DIR}" \
37+
; mkdir -p "${TETHYS_HOME}/keys"
38+
39+
# Speed up APT installs and Install APT packages
40+
RUN echo "force-unsafe-io" > /etc/dpkg/dpkg.cfg.d/02apt-speedup \
41+
; echo "Acquire::http {No-Cache=True;};" > /etc/apt/apt.conf.d/no-cache \
42+
; echo "Acquire::Check-Valid-Until false;" > /etc/apt/apt.conf.d/no-check-valid \
43+
; rm -rf /var/lib/apt/lists/* && apt-get update && apt-get -y install gcc libgdal-dev g++ libhdf5-dev && rm -rf /var/lib/apt/lists/*
44+
45+
###########
46+
# INSTALL #
47+
###########
48+
ARG MAMBA_DOCKERFILE_ACTIVATE=1
49+
COPY .devcontainer/dev_environment.yml ${TETHYS_HOME}/
50+
51+
RUN /bin/bash -c "micromamba run -n ${ENV_NAME} python --version; which python"
52+
53+
RUN sed -i "s|^[[:space:]]*- django[^-].*|- django=${DJANGO_VERSION}|" ${TETHYS_HOME}/dev_environment.yml \
54+
&& if [ "${DJANGO_VERSION}" = "3.2" ]; then \
55+
sed -i "s|^[[:space:]]*- django-taggit.*|- django-taggit<5|" ${TETHYS_HOME}/dev_environment.yml; \
56+
fi
57+
58+
RUN micromamba install -n ${ENV_NAME} -f ${TETHYS_HOME}/dev_environment.yml --yes \
59+
&& pip install django-datetime-widget2 \
60+
&& pip install geoserver-restconfig>=2.0.10
61+
62+
#########################
63+
# CONFIGURE ENVIRONMENT #
64+
#########################
65+
EXPOSE 8000
66+
67+
################
68+
# COPY IN SALT #
69+
################
70+
71+
COPY ../docker/salt/ /srv/salt/
72+
COPY .devcontainer/salt/ /srv/salt/
73+
COPY .devcontainer/run.sh ${TETHYS_HOME}/
74+
75+
RUN bash ${TETHYS_HOME}/build-checks.sh
76+
77+
#######
78+
# RUN #
79+
#######
80+
CMD bash run.sh
81+

‎.devcontainer/README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# How you use it
2+
3+
## VS Code
4+
5+
1. Install “Dev Containers” extension
6+
7+
2. Open repo → “Reopen in Container”
8+
9+
3. Once it boots, run:
10+
11+
```bash
12+
bash ./test.sh "$TETHYS_MANAGE"
13+
```
14+
15+
## CLI (optional)
16+
17+
If you use devcontainer CLI:
18+
19+
```bash
20+
devcontainer up --workspace-folder .
21+
devcontainer exec --workspace-folder . sh -lc './test.sh "$TETHYS_MANAGE"'
22+
```
23+
24+
## Udpate the content
25+
26+
1. Edit a file
27+
28+
2. Run
29+
30+
```bash
31+
devcontainer exec --workspace-folder . sh -lc './test.sh "$TETHYS_MANAGE"'
32+
```
33+
34+
3. See results immediately

‎.devcontainer/dev_environment.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: tethys
2+
3+
channels:
4+
- conda-forge
5+
6+
dependencies:
7+
- alembic
8+
- flake8
9+
- coverage
10+
- django>=3.2,<6
11+
- django-select2<8.3.0
12+
- django-taggit
13+
- coverage
14+
- factory_boy
15+
- filelock
16+
- geojson
17+
- jinja2
18+
- pandas
19+
- param
20+
- pyshp>=3.0.0
21+
- requests
22+
- sqlalchemy<2
23+
- panel

‎.devcontainer/devcontainer.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"name": "tethysext-atcore",
3+
"dockerComposeFile": ["./docker-compose.devcontainer.yml"],
4+
"service": "atcore",
5+
"workspaceFolder": "/var/www/tethys/exts/tethysext-atcore",
6+
"customizations": {
7+
"vscode": {
8+
"extensions": [
9+
"donjayamanne.python-extension-pack",
10+
"donjayamanne.git-extension-pack"
11+
]
12+
}
13+
},
14+
"shutdownAction": "stopCompose"
15+
}

‎.devcontainer/docker-compose.devcontainer.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
services:
2+
db:
3+
image: postgis/postgis:17-3.5
4+
env_file:
5+
- ./env/db.env
6+
healthcheck:
7+
test: ["CMD-SHELL", "pg_isready -U postgres"]
8+
interval: 10s
9+
timeout: 5s
10+
retries: 20
11+
ports:
12+
- "5432:5432"
13+
networks:
14+
- "internal"
15+
atcore:
16+
# Use the same image you already built:
17+
# image: atcore:local
18+
image: tethysdev/tethysext-atcore-devcontainer
19+
build:
20+
context: ..
21+
dockerfile: .devcontainer/Dockerfile.devcontainer
22+
args:
23+
TETHYS_VERSION: "4.3.8"
24+
PYTHON_VERSION: "3.11"
25+
DJANGO_VERSION: "4.2"
26+
27+
depends_on:
28+
db:
29+
condition: service_healthy
30+
networks:
31+
- "internal"
32+
- "external"
33+
ports:
34+
- "8000:8000"
35+
env_file:
36+
- ./env/atcore.env
37+
volumes:
38+
- ..:/var/www/tethys/exts/tethysext-atcore:cached
39+
40+
networks:
41+
internal:
42+
internal: true
43+
external:

‎.devcontainer/env/atcore.env‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
POSTGRES_PASSWORD=please_dont_use_default_passwords
2+
POSTGRES_PORT="5432"
3+
POSTGRES_DB=tethys_postgis
4+
5+
TETHYS_PUBLIC_HOST=localhost
6+
TETHYS_DB_HOST=db
7+
TETHYS_DB_PORT="5432"
8+
TETHYS_DB_USERNAME=tethys_super
9+
TETHYS_DB_PASSWORD=please_dont_use_default_passwords
10+
TETHYS_DB_SUPERUSER=tethys_super
11+
TETHYS_DB_SUPERUSER_PASS=please_dont_use_default_passwords
12+
13+
APP_DB_HOST=db
14+
APP_DB_PORT="5432"
15+
APP_DB_USERNAME=tethys_super
16+
APP_DB_PASSWORD=please_dont_use_default_passwords
17+
18+
TETHYS_MANAGE=/usr/lib/tethys/tethys/tethys_portal/manage.py
19+
ATCORE_TEST_DATABASE=postgresql://postgres:please_dont_use_default_passwords@db:5432/tethys_postgis

‎.devcontainer/env/db.env‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
POSTGRES_PASSWORD=please_dont_use_default_passwords
2+
POSTGRES_DB=tethys_postgis

‎.devcontainer/reset.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#! /bin/bash
2+
docker compose -f ./docker-compose.devcontainer.yml down -v

‎.devcontainer/run.sh‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/bin/bash
2+
3+
echo_status() {
4+
local args="${@}"
5+
tput setaf 4
6+
tput bold
7+
echo -e "- $args"
8+
tput sgr0
9+
}
10+
11+
db_max_count=24;
12+
db_engine=${TETHYS_DB_ENGINE} # Get the DB engine from environment variable
13+
USAGE="USAGE: . run.sh [options]
14+
OPTIONS:
15+
--db-max-count <INT> \t number of attempt to connect to the database. Default is at 24.
16+
"
17+
18+
while [[ $# -gt 0 ]]; do
19+
case $1 in
20+
--db-max-count)
21+
shift # shift from key to value
22+
db_max_count=$1;
23+
;;
24+
*)
25+
echo -e "${USAGE}"
26+
return 0
27+
esac
28+
shift
29+
done
30+
31+
echo_status "Starting up..."
32+
33+
# Check if the database is ready
34+
echo_status "Checking if DB is ready"
35+
db_check_count=0
36+
until ${CONDA_HOME}/envs/${CONDA_ENV_NAME}/bin/pg_isready -h ${TETHYS_DB_HOST} -p ${TETHYS_DB_PORT} -U postgres; do
37+
if [[ $db_check_count -gt $db_max_count ]]; then
38+
>&2 echo "DB was not available in time - exiting"
39+
exit 1
40+
fi
41+
>&2 echo "DB is unavailable - sleeping"
42+
db_check_count=`expr $db_check_count + 1`
43+
sleep 5
44+
done
45+
46+
# Run Salt
47+
if ! grep -q "^postgres.host:" /etc/salt/minion; then
48+
echo "postgres.host: '${TETHYS_DB_HOST}'" >> /etc/salt/minion
49+
echo "postgres.port: '${TETHYS_DB_PORT}'" >> /etc/salt/minion
50+
echo "postgres.user: '${TETHYS_DB_USERNAME}'" >> /etc/salt/minion
51+
echo "postgres.pass: '${TETHYS_DB_PASSWORD}'" >> /etc/salt/minion
52+
echo "postgres.bins_dir: '${CONDA_HOME}/envs/${CONDA_ENV_NAME}/bin'" >> /etc/salt/minion
53+
fi
54+
echo_status "Enforcing start state... (This might take a bit)"
55+
salt-call --local state.apply
56+
57+
# Keep the container running indefinitely with minimal CPU usage
58+
echo_status "Initialization complete. Keeping the container running..."
59+
while true; do sleep 3600; done

0 commit comments

Comments
 (0)