Skip to content

Commit c1abd63

Browse files
potiukpotiuk
authored
Allow to switch easily between Bullseye and Buster debian versions (apache#21546)
We are preparing to switch from Buster to Bullseye and this is the second change that is needed (following apache#21522). This change allows to choose whether we want to use Buster or Bullseye images as a base. We need to be able to choose, because: 1) we want to keep backwards compatibility and continue our users to build Buster-base images 2) we cannot yet fully switch to Bullseye because MsSQL's odbc driver does not yet support Bullseye and we reached out to mysql maintainers to learn about their plans to make the decision on when and how we are going to support Bullseye and MSSQL. Details of this discussion are in: https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131 This PR adds the capability of choosing the DEBIAN_VERSION in Breeze when building images but does not yet switch from Buster to Bullseye
1 parent f5832a6 commit c1abd63

17 files changed

+280
-39
lines changed

‎.hadolint.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,4 @@ ignored:
2020
- DL3008
2121
- DL3005
2222
- DL3013
23+
- SC1091

‎BREEZE.rst

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1279,6 +1279,14 @@ This is the current syntax for `./breeze <./breeze>`_:
12791279
12801280
linux/amd64
12811281
1282+
1283+
-d, --debian DEBIAN_VERSION
1284+
Debian version used for the image. This is always name of the debian distribution version.
1285+
1286+
One of:
1287+
1288+
buster bullseye
1289+
12821290
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
12831291
Uses different version of Airflow when building PROD image.
12841292
@@ -1486,6 +1494,14 @@ This is the current syntax for `./breeze <./breeze>`_:
14861494
14871495
linux/amd64
14881496
1497+
1498+
-d, --debian DEBIAN_VERSION
1499+
Debian version used for the image. This is always name of the debian distribution version.
1500+
1501+
One of:
1502+
1503+
buster bullseye
1504+
14891505
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
14901506
Uses different version of Airflow when building PROD image.
14911507
@@ -1553,6 +1569,14 @@ This is the current syntax for `./breeze <./breeze>`_:
15531569
15541570
linux/amd64
15551571
1572+
1573+
-d, --debian DEBIAN_VERSION
1574+
Debian version used for the image. This is always name of the debian distribution version.
1575+
1576+
One of:
1577+
1578+
buster bullseye
1579+
15561580
-I, --production-image
15571581
Use production image for entering the environment and builds (not for tests).
15581582
@@ -1627,6 +1651,14 @@ This is the current syntax for `./breeze <./breeze>`_:
16271651
16281652
linux/amd64
16291653
1654+
1655+
-d, --debian DEBIAN_VERSION
1656+
Debian version used for the image. This is always name of the debian distribution version.
1657+
1658+
One of:
1659+
1660+
buster bullseye
1661+
16301662
-v, --verbose
16311663
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
16321664
debugging - when you run breeze with --verbose flags you will be able to see the commands
@@ -1671,6 +1703,14 @@ This is the current syntax for `./breeze <./breeze>`_:
16711703
linux/amd64
16721704
16731705
1706+
-d, --debian DEBIAN_VERSION
1707+
Debian version used for the image. This is always name of the debian distribution version.
1708+
1709+
One of:
1710+
1711+
buster bullseye
1712+
1713+
16741714
####################################################################################################
16751715
16761716
@@ -1872,6 +1912,14 @@ This is the current syntax for `./breeze <./breeze>`_:
18721912
18731913
linux/amd64
18741914
1915+
1916+
-d, --debian DEBIAN_VERSION
1917+
Debian version used for the image. This is always name of the debian distribution version.
1918+
1919+
One of:
1920+
1921+
buster bullseye
1922+
18751923
-b, --backend BACKEND
18761924
Backend to use for tests - it determines which database is used.
18771925
One of:
@@ -1948,6 +1996,14 @@ This is the current syntax for `./breeze <./breeze>`_:
19481996
19491997
linux/amd64
19501998
1999+
2000+
-d, --debian DEBIAN_VERSION
2001+
Debian version used for the image. This is always name of the debian distribution version.
2002+
2003+
One of:
2004+
2005+
buster bullseye
2006+
19512007
-F, --force-build-images
19522008
Forces building of the local docker images. The images are rebuilt
19532009
automatically for the first time or when changes are detected in
@@ -2354,6 +2410,14 @@ This is the current syntax for `./breeze <./breeze>`_:
23542410
23552411
linux/amd64
23562412
2413+
2414+
-d, --debian DEBIAN_VERSION
2415+
Debian version used for the image. This is always name of the debian distribution version.
2416+
2417+
One of:
2418+
2419+
buster bullseye
2420+
23572421
****************************************************************************************************
23582422
Choose backend to run for Airflow
23592423

‎Dockerfile

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,6 @@ ARG DEV_APT_DEPS="\
9595
lsb-release \
9696
nodejs \
9797
openssh-client \
98-
python-selinux \
9998
sasl2-bin \
10099
software-properties-common \
101100
sqlite3 \
@@ -119,21 +118,20 @@ ENV DEV_APT_DEPS=${DEV_APT_DEPS} \
119118
DEV_APT_COMMAND=${DEV_APT_COMMAND} \
120119
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND} \
121120
ADDITIONAL_DEV_APT_ENV=${ADDITIONAL_DEV_APT_ENV}
121+
COPY scripts/docker/determine_debian_version_specific_variables.sh /scripts/docker/
122122

123-
# Note missing man directories on debian-buster
124-
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
125123
# Install basic and additional apt dependencies
126124
RUN apt-get update \
127125
&& apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 \
128-
&& apt-get install -y --no-install-recommends curl gnupg2 \
129-
&& mkdir -pv /usr/share/man/man1 \
130-
&& mkdir -pv /usr/share/man/man7 \
126+
&& apt-get install -y --no-install-recommends curl gnupg2 lsb-release \
131127
&& export ${ADDITIONAL_DEV_APT_ENV?} \
128+
&& source /scripts/docker/determine_debian_version_specific_variables.sh \
132129
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${DEV_APT_COMMAND}" \
133130
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${ADDITIONAL_DEV_APT_COMMAND}" \
134131
&& apt-get update \
135132
&& apt-get install -y --no-install-recommends \
136133
${DEV_APT_DEPS} \
134+
"${DISTRO_SELINUX}" \
137135
${ADDITIONAL_DEV_APT_DEPS} \
138136
&& apt-get autoremove -yqq --purge \
139137
&& apt-get clean \
@@ -375,7 +373,6 @@ ARG RUNTIME_APT_DEPS="\
375373
gosu \
376374
krb5-user \
377375
ldap-utils \
378-
libffi6 \
379376
libldap-2.4-2 \
380377
libsasl2-2 \
381378
libsasl2-modules \
@@ -407,20 +404,20 @@ ENV RUNTIME_APT_DEPS=${RUNTIME_APT_DEPS} \
407404
GUNICORN_CMD_ARGS="--worker-tmp-dir /dev/shm" \
408405
AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
409406

410-
# Note missing man directories on debian-buster
411-
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
407+
COPY scripts/docker/determine_debian_version_specific_variables.sh /scripts/docker/
408+
412409
# Install basic and additional apt dependencies
413410
RUN apt-get update \
414411
&& apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 \
415-
&& apt-get install -y --no-install-recommends curl gnupg2 \
416-
&& mkdir -pv /usr/share/man/man1 \
417-
&& mkdir -pv /usr/share/man/man7 \
412+
&& apt-get install -y --no-install-recommends curl gnupg2 lsb-release \
418413
&& export ${ADDITIONAL_RUNTIME_APT_ENV?} \
414+
&& source /scripts/docker/determine_debian_version_specific_variables.sh \
419415
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${RUNTIME_APT_COMMAND}" \
420416
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${ADDITIONAL_RUNTIME_APT_COMMAND}" \
421417
&& apt-get update \
422418
&& apt-get install -y --no-install-recommends \
423419
${RUNTIME_APT_DEPS} \
420+
"${DISTRO_LIBFFI}" \
424421
${ADDITIONAL_RUNTIME_APT_DEPS} \
425422
&& apt-get autoremove -yqq --purge \
426423
&& apt-get clean \

‎Dockerfile.ci

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -51,13 +51,16 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
5151
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
5252
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
5353

54+
COPY scripts/docker/determine_debian_version_specific_variables.sh /scripts/docker/
55+
5456
# Install basic and additional apt dependencies
5557
RUN apt-get update \
5658
&& apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 \
57-
&& apt-get install -y --no-install-recommends curl gnupg2 \
59+
&& apt-get install -y --no-install-recommends curl gnupg2 lsb-release \
5860
&& mkdir -pv /usr/share/man/man1 \
5961
&& mkdir -pv /usr/share/man/man7 \
6062
&& export ${ADDITIONAL_DEV_ENV_VARS?} \
63+
&& source /scripts/docker/determine_debian_version_specific_variables.sh \
6164
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${DEV_APT_COMMAND}" \
6265
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${ADDITIONAL_DEV_APT_COMMAND}" \
6366
&& apt-get update \
@@ -79,7 +82,7 @@ RUN apt-get update \
7982
libsasl2-dev \
8083
libsasl2-modules \
8184
libssl-dev \
82-
libenchant-dev \
85+
"${DISTRO_LIBENCHANT}" \
8386
locales \
8487
netcat \
8588
nodejs \
@@ -112,7 +115,6 @@ RUN bash /scripts/docker/install_mysql.sh prod \
112115
&& chmod 0440 /etc/sudoers.d/airflow
113116

114117
ARG RUNTIME_APT_DEPS="\
115-
libgcc-8-dev \
116118
apt-transport-https \
117119
bash-completion \
118120
ca-certificates \
@@ -160,16 +162,14 @@ ENV RUNTIME_APT_DEP=${RUNTIME_APT_DEPS} \
160162
AIRFLOW_HOME=${AIRFLOW_HOME} \
161163
AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
162164

163-
# Note missing man directories on debian-buster
164-
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
165-
RUN mkdir -pv /usr/share/man/man1 \
166-
&& mkdir -pv /usr/share/man/man7 \
167-
&& export ${ADDITIONAL_DEV_APT_ENV?} \
165+
RUN export ${ADDITIONAL_DEV_APT_ENV?} \
168166
&& export ${ADDITIONAL_RUNTIME_APT_ENV?} \
167+
&& source /scripts/docker/determine_debian_version_specific_variables.sh \
169168
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${RUNTIME_APT_COMMAND}" \
170169
&& bash -o pipefail -o errexit -o nounset -o nolog -c "${ADDITIONAL_RUNTIME_APT_COMMAND}" \
171170
&& apt-get update \
172171
&& apt-get install --no-install-recommends -y \
172+
"${DISTRO_LIBGCC}" \
173173
${RUNTIME_APT_DEPS} \
174174
${ADDITIONAL_RUNTIME_APT_DEPS} \
175175
&& apt-get autoremove -yqq --purge \

‎INSTALL

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ This is a generic installation method that requires a number of dependencies to
55
Depending on your system you might need different prerequisites, but the following
66
systems/prerequisites are known to work:
77

8-
Linux (Debian Buster and Linux Mint Tricia):
8+
Linux (Debian Bullseye and Linux Mint Debbie):
99

1010
sudo apt install build-essential python3-dev libsqlite3-dev openssl \
1111
sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql

‎breeze

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -515,6 +515,7 @@ EOF
515515
fi
516516
cat <<EOF
517517
Python version: ${PYTHON_MAJOR_MINOR_VERSION}
518+
Debian version: ${DEBIAN_VERSION}
518519
Backend: ${BACKEND} ${backend_version}
519520
EOF
520521
if [[ -n ${USE_AIRFLOW_VERSION=} ]]; then
@@ -629,6 +630,7 @@ export HOST_USER_ID=${HOST_USER_ID}
629630
export HOST_GROUP_ID=${HOST_GROUP_ID}
630631
export COMPOSE_FILE="${compose_file}"
631632
export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}"
633+
export DEBIAN_VERSION="${DEBIAN_VERSION}"
632634
export BACKEND="${BACKEND}"
633635
export AIRFLOW_VERSION="${AIRFLOW_VERSION}"
634636
export INSTALL_AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}"
@@ -679,14 +681,20 @@ EOF
679681
#######################################################################################################
680682
function breeze::prepare_command_files() {
681683
local main_ci_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/base.yml
682-
local backend_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml
684+
if [[ ${BACKEND} == "mssql" ]]; then
685+
local backend_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}-${DEBIAN_VERSION}.yml
686+
else
687+
local backend_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml
688+
fi
689+
683690
local backend_port_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}-port.yml
684691
local local_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/local.yml
685692
local local_all_sources_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/local-all-sources.yml
686693
local files_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/files.yml
687694
local remove_sources_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/remove-sources.yml
688695
local forward_credentials_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/forward-credentials.yml
689696

697+
690698
if [[ ${BACKEND} == "mssql" ]]; then
691699
local docker_filesystem
692700
docker_filesystem=$(stat "-f" "-c" "%T" /var/lib/docker 2>/dev/null || echo "unknown")
@@ -816,6 +824,12 @@ function breeze::parse_arguments() {
816824
echo
817825
shift 2
818826
;;
827+
-d | --debian-version)
828+
export DEBIAN_VERSION="${2}"
829+
echo "Debian version: ${DEBIAN_VERSION}"
830+
echo
831+
shift 2
832+
;;
819833
-b | --backend)
820834
export BACKEND="${2}"
821835
echo "Backend: ${BACKEND}"
@@ -940,7 +954,7 @@ function breeze::parse_arguments() {
940954
echo
941955
shift 2
942956
;;
943-
-d | --db-reset)
957+
--db-reset)
944958
echo "Resetting the DB!"
945959
echo
946960
export DB_RESET="true"
@@ -1544,6 +1558,10 @@ function breeze::prepare_formatted_versions() {
15441558
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
15451559
readonly FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS
15461560

1561+
FORMATTED_DEBIAN_VERSIONS=$(echo "${_breeze_allowed_debian_versions=""}" | tr '\n' ' ' |
1562+
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
1563+
readonly FORMATTED_DEBIAN_VERSIONS
1564+
15471565
FORMATTED_BACKENDS=$(echo "${_breeze_allowed_backends=""}" | tr '\n' ' ' |
15481566
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
15491567
readonly FORMATTED_BACKENDS
@@ -2260,6 +2278,14 @@ ${FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS}
22602278
22612279
${FORMATTED_PLATFORMS}
22622280
2281+
2282+
-d, --debian DEBIAN_VERSION
2283+
Debian version used for the image. This is always name of the debian distribution version.
2284+
2285+
One of:
2286+
2287+
${FORMATTED_DEBIAN_VERSIONS}
2288+
22632289
"
22642290
}
22652291

‎breeze-complete

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
# Those cannot be made read-only as the breeze-complete must be re-sourceable
2424

2525
_breeze_allowed_python_major_minor_versions="3.7 3.8 3.9"
26+
_breeze_allowed_debian_versions="buster bullseye"
2627
_breeze_allowed_backends="sqlite mysql postgres mssql"
2728
_breeze_allowed_integrations="cassandra kerberos mongo openldap pinot rabbitmq redis statsd trino all"
2829
_breeze_allowed_generate_constraints_modes="source-providers pypi-providers no-providers"
@@ -43,6 +44,7 @@ _breeze_allowed_platforms="linux/amd64"
4344
# shellcheck disable=SC2034
4445
{
4546
# Default values for the commands & flags used
47+
_breeze_default_debian_version=$(echo "${_breeze_allowed_debian_versions}" | awk '{print $1}')
4648
_breeze_default_backend=$(echo "${_breeze_allowed_backends}" | awk '{print $1}')
4749
_breeze_default_generate_providers_mode=$(echo "${_breeze_allowed_generate_constraints_modes}" | awk '{print $1}')
4850
_breeze_default_kubernetes_mode=$(echo "${_breeze_allowed_kubernetes_modes}" | awk '{print $1}')
@@ -166,7 +168,7 @@ _breeze_default_github_repository="apache/airflow"
166168
_breeze_default_github_image_id="latest"
167169

168170
_breeze_short_options="
169-
h p: b: i:
171+
h p: d: b: i:
170172
K: V:
171173
l a: t: d:
172174
v y n q f
@@ -177,7 +179,7 @@ S: N:
177179
"
178180

179181
_breeze_long_options="
180-
help python: backend: integration: platform:
182+
help python: debian-version: backend: integration: platform:
181183
kubernetes-mode: kubernetes-version: helm-version: kind-version:
182184
skip-mounting-local-sources mount-all-local-sources install-airflow-version: install-airflow-reference: db-reset
183185
verbose assume-yes assume-no assume-quit forward-credentials init-script:
@@ -256,6 +258,9 @@ function breeze_complete::get_known_values_breeze() {
256258
-p | --python)
257259
_breeze_known_values=${_breeze_allowed_python_major_minor_versions}
258260
;;
261+
-d | --debian-version)
262+
_breeze_known_values=${_breeze_allowed_debian_versions}
263+
;;
259264
-b | --backend)
260265
_breeze_known_values=${_breeze_allowed_backends}
261266
;;

‎dev/provider_packages/prepare_provider_packages.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1039,7 +1039,7 @@ def make_sure_remote_apache_exists_and_fetch(git_update: bool, verbose: bool):
10391039
if not git_update:
10401040
return
10411041
except subprocess.CalledProcessError as ex:
1042-
if ex.returncode == 128:
1042+
if ex.returncode == 128 or ex.returncode == 2:
10431043
remote_add_command = [
10441044
"git",
10451045
"remote",

0 commit comments

Comments
 (0)