Skip to content

Commit

Permalink
Merge bitcoin#12134: Build previous releases and run functional tests
Browse files Browse the repository at this point in the history
c456145 [test] add 0.19 backwards compatibility tests (Sjors Provoost)
b769cd1 [test] add v0.17.1 wallet upgrade test (Sjors Provoost)
9d9390d [tests] add wallet backwards compatility tests (Sjors Provoost)
c7ca630 [scripts] support release candidates of earlier releases (Sjors Provoost)
8b1460d [tests] check v0.17.1 and v0.18.1 backwards compatibility (Sjors Provoost)
ae379cf [scripts] build earlier releases (Sjors Provoost)

Pull request description:

  This PR adds binaries for 0.17, 0.18 and 0.19 to Travis and runs a basic block propagation test.

  Includes test for upgrading v0.17.1 wallets and opening master wallets with older versions.

  Usage:

  ```sh
  contrib/devtools/previous_release.sh -f -b v0.19.0.1 v0.18.1 v0.17.1
  test/functional/backwards_compatibility.py
  ```

  Travis caches these earlier releases, so it should be able to run these tests with little performance impact.

  Additional scenarios where it might be useful to run tests against earlier releases:

  * creating a wallet with bitcoin#11403's segwit implementation, copying it to an older node and making sure the user didn't lose any funds (although this PR doesn't support `v0.15.1`)
  * future consensus changes
  * P2P changes (e.g. to make sure we don't accidentally ban old nodes)

ACKs for top commit:
  MarcoFalke:
    ACK c456145 🔨

Tree-SHA512: 360bd870603f95b14dc0cd629532cc147344f632b808617c18e1b585dfb1f082b401e5d493a48196b719e0aeaee533ae0a773dfc9f217f704aae898576c19232
  • Loading branch information
MarcoFalke authored and PastaPastaPasta committed Sep 19, 2023
1 parent 41a6613 commit 207b1c5
Show file tree
Hide file tree
Showing 12 changed files with 494 additions and 9 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,9 @@ qrc_*.cpp
build
*.dSYM

# Previous releases
releases

#lcov
*.gcno
*.gcda
Expand Down
4 changes: 2 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ services:
cache:
ccache: true
directories:
- $BASE_BUILD_DIR/ci/scratch/.ccache
- $BASE_BUILD_DIR/releases/$HOST
before_cache:
- if [ "${TRAVIS_OS_NAME}" = "osx" ]; then brew cleanup; fi
env:
Expand Down Expand Up @@ -244,7 +244,7 @@ after_success:
FILE_ENV="./ci/test/00_setup_env_win64.sh"
- stage: test
name: 'x86_64 Linux [GOAL: install] [focal] [uses qt5 dev package and some depends packages] [unsigned char]'
name: 'x86_64 Linux [GOAL: install] [focal] [previous releases, uses qt5 dev package and some depends packages] [unsigned char]'
env: >-
FILE_ENV="./ci/test/00_setup_env_native_qt5.sh"
# x86_64 Linux (xenial, no depends, only system libs, sanitizers: thread (TSan))
Expand Down
2 changes: 1 addition & 1 deletion ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ If the repository is not a fresh git clone, you might have to clean files from p

The ci needs to perform various sysadmin tasks such as installing packages or writing to the user's home directory.
While most of the actions are done inside a docker container, this is not possible for all. Thus, cache directories,
such as the depends cache or ccache, are mounted as read-write into the docker container. While it should be fine to run
such as the depends cache, previous release binaries, or ccache, are mounted as read-write into the docker container. While it should be fine to run
the ci system locally on you development box, the ci scripts can generally be assumed to have received less review and
testing compared to other parts of the codebase. If you want to keep the work tree clean, you might want to run the ci
system in a virtual machine with a Linux operating system of your choice.
Expand Down
2 changes: 2 additions & 0 deletions ci/test/00_setup_env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ export USE_BUSY_BOX=${USE_BUSY_BOX:-false}
export RUN_UNIT_TESTS=${RUN_UNIT_TESTS:-true}
export RUN_INTEGRATION_TESTS=${RUN_INTEGRATION_TESTS:-true}
export RUN_SECURITY_TESTS=${RUN_SECURITY_TESTS:-false}
export TEST_PREVIOUS_RELEASES=${TEST_PREVIOUS_RELEASES:-false}
export RUN_FUZZ_TESTS=${RUN_FUZZ_TESTS:-false}
export RUN_SYMBOL_TESTS=${RUN_SYMBOL_TESTS:-true}
export CONTAINER_NAME=${CONTAINER_NAME:-ci_unnamed}
Expand All @@ -59,6 +60,7 @@ export CCACHE_DIR=${CCACHE_DIR:-$CACHE_DIR/ccache}
export DEPENDS_DIR=${DEPENDS_DIR:-$BASE_ROOT_DIR/depends}
# Folder where the build is done (bin and lib).
export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out/$HOST}
export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/releases/$HOST}
export SDK_URL=${SDK_URL:-https://bitcoincore.org/depends-sources/sdks}
export DOCKER_PACKAGES=${DOCKER_PACKAGES:-build-essential libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates ccache python3 rsync git procps}
export GOAL=${GOAL:-install}
Expand Down
1 change: 1 addition & 0 deletions ci/test/00_setup_env_native_qt5.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,5 @@ export DEP_OPTS="NO_UPNP=1 DEBUG=1"
export RUN_UNIT_TESTS_SEQUENTIAL="true"
export RUN_UNIT_TESTS="false"
export GOAL="install"
export TEST_PREVIOUS_RELEASES=true
export BITCOIN_CONFIG="--enable-zmq --enable-reduce-exports --disable-fuzz-binary LDFLAGS=-static-libstdc++"
4 changes: 3 additions & 1 deletion ci/test/04_install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,13 @@ fi

mkdir -p "${BASE_SCRATCH_DIR}"
mkdir -p "${CCACHE_DIR}"
mkdir -p "${PREVIOUS_RELEASES_DIR}"

export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1"
export LSAN_OPTIONS="suppressions=${BASE_BUILD_DIR}/test/sanitizer_suppressions/lsan"
export TSAN_OPTIONS="suppressions=${BASE_BUILD_DIR}/test/sanitizer_suppressions/tsan"
export UBSAN_OPTIONS="suppressions=${BASE_BUILD_DIR}/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1"
env | grep -E '^(BASE_|QEMU_|CCACHE_|LC_ALL|BOOST_TEST_RANDOM|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS)' | tee /tmp/env
env | grep -E '^(BASE_|QEMU_|CCACHE_|LC_ALL|BOOST_TEST_RANDOM|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS|TEST_PREVIOUS_RELEASES|PREVIOUS_RELEASES_DIR))' | tee /tmp/env
if [[ $HOST = *-mingw32 ]]; then
DOCKER_ADMIN="--cap-add SYS_ADMIN"
elif [[ $BITCOIN_CONFIG = *--with-sanitizers=*address* ]]; then # If ran with (ASan + LSan), Docker needs access to ptrace (https://github.com/google/sanitizers/issues/764)
Expand All @@ -42,6 +43,7 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
--mount type=bind,src=$BASE_ROOT_DIR,dst=/ro_base,readonly \
--mount type=bind,src=$CCACHE_DIR,dst=$CCACHE_DIR \
--mount type=bind,src=$DEPENDS_DIR,dst=$DEPENDS_DIR \
--mount type=bind,src=$PREVIOUS_RELEASES_DIR,dst=$PREVIOUS_RELEASES_DIR \
-w $BASE_ROOT_DIR \
--env-file /tmp/env \
--name $CONTAINER_NAME \
Expand Down
5 changes: 5 additions & 0 deletions ci/test/05_before_script.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,8 @@ if [ -z "$NO_DEPENDS" ]; then
fi
DOCKER_EXEC $SHELL_OPTS make $MAKEJOBS -C depends HOST=$HOST $DEP_OPTS
fi
if [ "$TEST_PREVIOUS_RELEASES" = "true" ]; then
BEGIN_FOLD previous-versions
DOCKER_EXEC contrib/devtools/previous_release.sh -b -t "$PREVIOUS_RELEASES_DIR" v0.17.1 v0.18.1 v0.19.0.1
END_FOLD
fi
149 changes: 149 additions & 0 deletions contrib/devtools/previous_release.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Build previous releases.

export LC_ALL=C

CONFIG_FLAGS=""
FUNCTIONAL_TESTS=0
DELETE_EXISTING=0
USE_DEPENDS=0
DOWNLOAD_BINARY=0
CONFIG_FLAGS=""
TARGET="releases"

while getopts ":hfrdbt:" opt; do
case $opt in
h)
echo "Usage: .previous_release.sh [options] tag1 tag2"
echo " options:"
echo " -h Print this message"
echo " -f Configure for functional tests"
echo " -r Remove existing directory"
echo " -d Use depends"
echo " -b Download release binary"
echo " -t Target directory (default: releases)"
exit 0
;;
f)
FUNCTIONAL_TESTS=1
CONFIG_FLAGS="$CONFIG_FLAGS --without-gui --disable-tests --disable-bench"
;;
r)
DELETE_EXISTING=1
;;
d)
USE_DEPENDS=1
;;
b)
DOWNLOAD_BINARY=1
;;
t)
TARGET=$OPTARG
;;
\?)
echo "Invalid option: -$OPTARG" >&2
exit 1
;;
esac
done

shift $((OPTIND-1))

if [ -z "$1" ]; then
echo "Specify release tag(s), e.g.: .previous_release v0.15.1"
exit 1
fi

if [ ! -d "$TARGET" ]; then
mkdir -p $TARGET
fi

if [ "$DOWNLOAD_BINARY" -eq "1" ]; then
HOST="${HOST:-$(./depends/config.guess)}"
case "$HOST" in
x86_64-*-linux*)
PLATFORM=x86_64-linux-gnu
;;
x86_64-apple-darwin*)
PLATFORM=osx64
;;
*)
echo "Not sure which binary to download for $HOST."
exit 1
;;
esac
fi

echo "Releases directory: $TARGET"
pushd "$TARGET" || exit 1
{
for tag in "$@"
do
if [ "$DELETE_EXISTING" -eq "1" ]; then
if [ -d "$tag" ]; then
rm -r "$tag"
fi
fi

if [ "$DOWNLOAD_BINARY" -eq "0" ]; then

if [ ! -d "$tag" ]; then
if [ -z $(git tag -l "$tag") ]; then
echo "Tag $tag not found"
exit 1
fi

git clone https://github.com/bitcoin/bitcoin "$tag"
pushd "$tag" || exit 1
{
git checkout "$tag"
if [ "$USE_DEPENDS" -eq "1" ]; then
pushd depends || exit 1
{
if [ "$FUNCTIONAL_TESTS" -eq "1" ]; then
make NO_QT=1
else
make
fi
HOST="${HOST:-$(./config.guess)}"
}
popd || exit 1
CONFIG_FLAGS="--prefix=$PWD/depends/$HOST $CONFIG_FLAGS"
fi
./autogen.sh
./configure $CONFIG_FLAGS
make
# Move binaries, so they're in the same place as in the release download:
mkdir bin
mv src/bitcoind src/bitcoin-cli src/bitcoin-tx bin
if [ "$FUNCTIONAL_TESTS" -eq "0" ]; then
mv src/qt/bitcoin-qt bin
fi
}
popd || exit 1
fi
else
if [ -d "$tag" ]; then
echo "Using cached $tag"
else
mkdir "$tag"
if [[ "$tag" =~ v(.*)(rc[0-9]+)$ ]]; then
BIN_PATH="bin/bitcoin-core-${BASH_REMATCH[1]}/test.${BASH_REMATCH[2]}"
else
BIN_PATH="bin/bitcoin-core-${tag:1}"
fi
URL="https://bitcoin.org/$BIN_PATH/bitcoin-${tag:1}-$PLATFORM.tar.gz"
echo "Fetching: $URL"
curl -O $URL
tar -zxf "bitcoin-${tag:1}-$PLATFORM.tar.gz" -C "$tag" --strip-components=1 "bitcoin-${tag:1}"
rm "bitcoin-${tag:1}-$PLATFORM.tar.gz"
fi
fi
done
}
popd || exit 1
Loading

0 comments on commit 207b1c5

Please sign in to comment.