Skip to content

[ENG-7411] poetry and assorted cleanup #852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Apr 18, 2025
Merged

[ENG-7411] poetry and assorted cleanup #852

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8465f08
wip: poetry
aaxelb Mar 26, 2025
8a1ebac
wip
aaxelb Mar 26, 2025
7b92a4f
wip
aaxelb Mar 26, 2025
2b137f6
shtrove not installable (...yet)
aaxelb Mar 26, 2025
68bf437
wip (poetry run)
aaxelb Mar 26, 2025
512f89d
try specifying source for coverage
aaxelb Mar 27, 2025
bb24a8e
wip (python version range)
aaxelb Apr 9, 2025
2514ad6
remove py3.10 tests
aaxelb Apr 9, 2025
636de98
bump deps; fix build after py-upgrade merge
aaxelb Apr 9, 2025
95790ca
poetry config: explicit package-mode
aaxelb Apr 9, 2025
2448562
remove elasticsearch5 use
aaxelb Apr 9, 2025
3b7176b
remove trove_indexcard_flats index strategy
aaxelb Apr 9, 2025
2a50077
use python deps from the built image
aaxelb Apr 9, 2025
28e507f
Revert "use python deps from the built image"
aaxelb Apr 9, 2025
8502306
fix requirements in docker-compose
aaxelb Apr 9, 2025
25bf800
clarify poetry venv usage
aaxelb Apr 10, 2025
bd055bb
finish removing elasticsearch5
aaxelb Apr 10, 2025
19344e1
fix: dockerfile warnings
aaxelb Apr 15, 2025
e698c8d
add toml (for coveralls)
aaxelb Apr 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .docker-compose.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672
DATABASE_HOST=postgres
ELASTICSEARCH8_URL=https://elastic8:9200/
# ELASTICSEARCH5_URL=http://elasticsearch:9200/
LOGIN_REDIRECT_URL=http://localhost:8003/
OSF_API_URL=http://localhost:8000
RABBITMQ_HOST=rabbitmq
Expand Down
1 change: 1 addition & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ docker-compose.yml
.gitignore
celerybeat.pid
/static/
.venv/

/au.*/
/be.*/
Expand Down
34 changes: 12 additions & 22 deletions .github/workflows/run_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ['3.10'] # TODO: 3.11, 3.12
postgres-version: ['15', '17']
python-version: ['3.13']
postgres-version: ['17']
runs-on: ubuntu-latest
services:
postgres:
Expand All @@ -37,12 +37,6 @@ jobs:
cluster.initial_master_nodes: singlenode
ports:
- 9208:9200
elasticsearch5:
image: elasticsearch:5.4
env:
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
ports:
- 9205:9200
rabbitmq:
image: rabbitmq:management
ports:
Expand All @@ -54,32 +48,28 @@ jobs:
- name: install non-py dependencies
run: sudo apt-get update && sudo apt-get install -y libxml2-dev libxslt1-dev libpq-dev git gcc

- name: set up python${{ matrix.python-version }}
- name: Install poetry
run: pipx install poetry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be pinned to the same version as the Dockerfile (2.1.1)? I don't have a strong opinion. It might be worth to leave unpinned as a canary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also no strong opinion -- leaving it unpinned as canary seems a fine idea


- name: setup python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
cache-dependency-path: |
requirements.txt
dev-requirements.txt

- name: install py dependencies
run: pip install -r dev-requirements.txt
cache: 'poetry'

- name: install share
run: python setup.py develop
- name: install despondencies
run: poetry install --with dev

- name: flake it
run: flake8 .
run: poetry run flake8 .

- name: run tests
run: |
coverage run -m pytest --create-db
coverage xml -o _shtrove_coverage.xml
poetry run coverage run -m pytest --create-db
poetry run coverage xml -o _shtrove_coverage.xml
env:
DATABASE_PASSWORD: postgres
ELASTICSEARCH8_URL: http://localhost:9208/
# ELASTICSEARCH5_URL: http://localhost:9205/

- name: coveralls
uses: coverallsapp/github-action@v2
Expand Down
38 changes: 26 additions & 12 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.13-slim-bullseye as app
FROM python:3.13-slim-bullseye AS app

RUN apt-get update \
&& apt-get install -y \
Expand All @@ -25,35 +25,49 @@ RUN update-ca-certificates
RUN mkdir -p /code
WORKDIR /code

RUN pip install -U pip
RUN pip install uwsgi==2.0.27
###
# python dependencies

COPY ./requirements.txt /code/requirements.txt
COPY ./constraints.txt /code/constraints.txt
# note: installs dependencies on the system, roundabouts `/usr/local/lib/python3.13/site-packages/`

RUN pip install --no-cache-dir -c /code/constraints.txt -r /code/requirements.txt
ENV POETRY_NO_INTERACTION=1 \
POETRY_VIRTUALENVS_OPTIONS_ALWAYS_COPY=1 \
POETRY_VIRTUALENVS_CREATE=0 \
POETRY_VIRTUALENVS_IN_PROJECT=0 \
POETRY_CACHE_DIR=/tmp/poetry-cache \
POETRY_HOME=/tmp/poetry-venv

RUN python -m venv $POETRY_HOME

RUN $POETRY_HOME/bin/pip install poetry==2.1.1

COPY pyproject.toml .
COPY poetry.lock .

RUN $POETRY_HOME/bin/poetry install --compile

RUN apt-get remove -y \
gcc \
zlib1g-dev
zlib1g-dev \
&& apt-get autoremove -y

COPY ./ /code/

RUN python manage.py collectstatic --noinput

ARG GIT_TAG=
ARG GIT_COMMIT=
ENV VERSION ${GIT_TAG}
ENV GIT_COMMIT ${GIT_COMMIT}

RUN python setup.py develop
ENV VERSION=${GIT_TAG}
ENV GIT_COMMIT=${GIT_COMMIT}

CMD ["python", "manage.py", "--help"]

### Dist
FROM app AS dist

RUN $POETRY_HOME/bin/poetry install --compile --only deploy

### Dev
FROM app AS dev

RUN pip install --no-cache-dir -c /code/constraints.txt -r /code/dev-requirements.txt
RUN $POETRY_HOME/bin/poetry install --compile --only dev
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Docker build had a few nitpicks, but these aren't dealbreakers:

 3 warnings found (use docker --debug to expand):                                                                                                                                                           
 - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 1)
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 60)
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 61)

13 changes: 0 additions & 13 deletions dev-requirements.txt
View file
Open in desktop

This file was deleted.

6 changes: 2 additions & 4 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,12 +151,10 @@ services:
- -c
- apt-get update &&
apt-get install -y gcc &&
pip install -r requirements.txt -r dev-requirements.txt &&
(python3 -m compileall /usr/local/lib/python3.13 || true) &&
$$POETRY_HOME/bin/poetry install --compile --with dev &&
rm -Rf /python3.13/* &&
apt-get remove -y gcc &&
cp -Rf -p /usr/local/lib/python3.13 / &&
python3 setup.py develop
cp -Rf -p /usr/local/lib/python3.13 /
restart: 'no'
volumes:
- ./:/code:cached
Expand Down
Loading