Skip to content

Commit b29e6d2

Browse files
farostfarost
committed
Add TypeDB Cluster support and failover (#765)
Introduce TypeDB Cluster support. Including: Drivers can be created using a single address, multiple addresses, or address translation (mapping between public user-facing addresses and internal cluster addresses for special needs). Address translation can be updated after the driver is created. Server version and distribution, as well as the list of active replicas, can be retrieved using TypeDB Driver. Read operations can be executed using one of three strategies: * Strong consistency level -- Strongest consistency, always up-to-date due to the guarantee of the primary replica usage. * Eventual consistency level -- Allow stale reads from any replica. May not reflect latest writes (will not be supported in the first Cluster release on the server side, so will probably return errors) * Replica dependent consistency level -- The operation is executed against the provided replica address only. Can be especially useful for testing. By default, all operations are executed using the strongest consistency level. However, all read operations now support consistency level specification. Read transactions can be configured through transaction options. Driver options can be used to configure the failover strategies used with strong and eventual consistency levels. `primary_failover_retries` -- Limits the number of attempts to redirect a strongly consistent request to another primary replica in case of a failure due to the change of replica roles. Defaults to 1. `replica_discovery_attempts` -- Limits the number of driver attempts to discover a single working replica to perform an operation in case of a replica unavailability. Every replica is tested once, which means that at most: - {limit} operations are performed if the limit <= the number of replicas. - {number of replicas} operations are performed if the limit > the number of replicas. - {number of replicas} operations are performed if the limit is None. Affects every eventually consistent operation, including redirect failover, when the new primary replica is unknown. Defaults to None. **Missing parts**: * Address translation runtime update is not fully implemented, it probably requires more updates, or can be removed for the first release. * Testing cannot be automated without Cluster snapshots. Instead, for local testing, use `temp-cluster-server` (instructions below). * The driver works itself, the tests for Core pass. However, its replication features are untested. There is a new `clustering` integration test, which successfully boots up servers, but there are some missing functionalities of the server that prevent the testing progress. Once the PR referenced below is completed and merged, it can be completed, as well. More BDD tests should be added here typedb/typedb-behaviour#373 This driver can be tested using the server from typedb/typedb-cluster#608. To do so, run `cargo build` in `typedb-cluster`, and then copy the resulting binary to `typedb-driver` like: ``` cp path-to-typedb/typedb-cluster/target/debug/typedb_server_bin path-to-typedb/typedb-driver/tool/test/temp-cluster-server/typedb ``` Then, test scripts like `./tool/test/temp-cluster-server/start-cluster-servers.sh 3` can be used. Note that it is a temporary directory, created to use the server while we are not able to use Cluster snapshots from Bazel.
1 parent dc45644 commit b29e6d2

File tree

240 files changed

+9935
-5380
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

240 files changed

+9935
-5380
lines changed

‎.circleci/config.yml‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -925,14 +925,13 @@ workflows:
925925
- deploy-snapshot-any
926926
# - deploy-snapshot-dotnet-any
927927

928-
# TODO: Windows typedb artifact is not ready
929-
# - test-snapshot-windows-x86_64:
930-
# filters:
931-
# branches:
932-
# only: [master]
933-
# requires:
934-
# - deploy-snapshot-windows-x86_64
935-
# - deploy-snapshot-any
928+
- test-snapshot-windows-x86_64:
929+
filters:
930+
branches:
931+
only: [master]
932+
requires:
933+
- deploy-snapshot-windows-x86_64
934+
- deploy-snapshot-any
936935
# - deploy-snapshot-dotnet-any
937936

938937
# TODO: npm is not ready

‎.factory/automation.yml‎

Lines changed: 55 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -50,10 +50,10 @@ build:
5050
bazel run @typedb_dependencies//distribution/artifact:create-netrc
5151
# TODO: Temporarily build only the 3.0-reimplemented drivers:
5252
# bazel build //...
53-
bazel build //c/...
53+
# bazel build //c/...
5454
bazel build //rust/...
55-
bazel build //java/...
56-
bazel build //python/...
55+
# bazel build //java/...
56+
# bazel build //python/...
5757
bazel build //http-ts/...
5858
5959
bazel run @typedb_dependencies//tool/checkstyle:test-coverage
@@ -87,8 +87,8 @@ build:
8787
8888
tool/docs/update_readme.sh
8989
git add .
90-
git diff --exit-code || {
91-
echo "Failed to verify README files: plese update it manually and verify the changes"
90+
git diff --exit-code || {
91+
echo "Failed to verify README files: please update it manually and verify the changes"
9292
exit 1
9393
}
9494
@@ -109,7 +109,7 @@ build:
109109
if [[ -n "$COMMUNITY_FAILED" ]]; then exit 1; fi
110110
111111
# TODO: Use cluster server artifact with 3 nodes for the same common tests when available
112-
# tool/test/start-cluster-servers.sh 3 &&
112+
# source tool/test/start-cluster-servers.sh 3 &&
113113
# bazel test //rust/tests/integration:all --test_output=streamed --test_arg=--nocapture &&
114114
# export CLUSTER_FAILED= || export CLUSTER_FAILED=1
115115
# tool/test/stop-cluster-servers.sh
@@ -129,22 +129,21 @@ build:
129129
tool/test/stop-community-server.sh
130130
exit $TEST_SUCCESS
131131
132-
# TODO: Use cluster server artifact with 3 nodes when available (it would do the same thing as community now)
133-
# test-rust-behaviour-cluster:
134-
# image: typedb-ubuntu-20.04 # Ubuntu 20.04 has GLIBC version 2.31 (2020) which we should verify to compile against
135-
# dependencies:
136-
# - build
137-
# command: |
138-
# export ARTIFACT_USERNAME=$REPO_TYPEDB_USERNAME
139-
# export ARTIFACT_PASSWORD=$REPO_TYPEDB_PASSWORD
140-
# bazel run @typedb_dependencies//tool/bazelinstall:remote_cache_setup.sh
141-
# bazel run @typedb_dependencies//distribution/artifact:create-netrc
142-
#
143-
# tool/test/start-cluster-servers.sh 3 &&
144-
# bazel test //rust/tests/behaviour/... --//rust/tests/behaviour/config:mode=cluster --test_output=streamed --jobs=1 &&
145-
# export TEST_SUCCESS=0 || export TEST_SUCCESS=1
146-
# tool/test/stop-cluster-servers.sh
147-
# exit $TEST_SUCCESS
132+
test-rust-behaviour-cluster:
133+
image: typedb-ubuntu-20.04 # Ubuntu 20.04 has GLIBC version 2.31 (2020) which we should verify to compile against
134+
dependencies:
135+
- build
136+
command: |
137+
export ARTIFACT_USERNAME=$REPO_TYPEDB_USERNAME
138+
export ARTIFACT_PASSWORD=$REPO_TYPEDB_PASSWORD
139+
bazel run @typedb_dependencies//tool/bazelinstall:remote_cache_setup.sh
140+
bazel run @typedb_dependencies//distribution/artifact:create-netrc
141+
142+
source tool/test/start-cluster-servers.sh 3 &&
143+
bazel test //rust/tests/behaviour/... --test_env=ROOT_CA=$ROOT_CA --//rust/tests/behaviour/config:mode=cluster --test_output=streamed --jobs=1 &&
144+
export TEST_SUCCESS=0 || export TEST_SUCCESS=1
145+
tool/test/stop-cluster-servers.sh
146+
exit $TEST_SUCCESS
148147

149148
test-c-integration:
150149
image: typedb-ubuntu-20.04 # Ubuntu 20.04 has GLIBC version 2.31 (2020) which we should verify to compile against
@@ -176,7 +175,7 @@ build:
176175
if [[ -n "$COMMUNITY_FAILED" ]]; then exit 1; fi
177176
178177
# TODO: Use cluster server artifact with 3 nodes for the same common tests when available
179-
# tool/test/start-cluster-servers.sh 3 &&
178+
# source tool/test/start-cluster-servers.sh 3 &&
180179
# bazel test //java/test/integration:all --test_output=streamed --jobs=1 &&
181180
# export CLUSTER_FAILED= || export CLUSTER_FAILED=1
182181
# tool/test/stop-cluster-servers.sh
@@ -196,22 +195,21 @@ build:
196195
tool/test/stop-community-server.sh
197196
exit $TEST_SUCCESS
198197
199-
# TODO: Use cluster server artifact with 3 nodes when available (it would do the same thing as community now)
200-
# test-java-behaviour-cluster:
201-
# image: typedb-ubuntu-22.04
202-
# dependencies:
203-
# - build
204-
# command: |
205-
# export ARTIFACT_USERNAME=$REPO_TYPEDB_USERNAME
206-
# export ARTIFACT_PASSWORD=$REPO_TYPEDB_PASSWORD
207-
# bazel run @typedb_dependencies//tool/bazelinstall:remote_cache_setup.sh
208-
# bazel run @typedb_dependencies//distribution/artifact:create-netrc
209-
#
210-
# tool/test/start-cluster-servers.sh 3 &&
211-
# .factory/test-cluster.sh //java/test/behaviour/... --test_output=streamed --jobs=1 &&
212-
# export TEST_SUCCESS=0 || export TEST_SUCCESS=1
213-
# tool/test/stop-cluster-servers.sh
214-
# exit $TEST_SUCCESS
198+
test-java-behaviour-cluster:
199+
image: typedb-ubuntu-22.04
200+
dependencies:
201+
- build
202+
command: |
203+
export ARTIFACT_USERNAME=$REPO_TYPEDB_USERNAME
204+
export ARTIFACT_PASSWORD=$REPO_TYPEDB_PASSWORD
205+
bazel run @typedb_dependencies//tool/bazelinstall:remote_cache_setup.sh
206+
bazel run @typedb_dependencies//distribution/artifact:create-netrc
207+
208+
source tool/test/start-cluster-servers.sh 3 &&
209+
.factory/test-cluster.sh //java/test/behaviour/... --test_env=ROOT_CA=$ROOT_CA --test_output=streamed --jobs=1 &&
210+
export TEST_SUCCESS=0 || export TEST_SUCCESS=1
211+
tool/test/stop-cluster-servers.sh
212+
exit $TEST_SUCCESS
215213
216214
test-python-integration:
217215
image: typedb-ubuntu-22.04
@@ -230,7 +228,7 @@ build:
230228
if [[ -n "$COMMUNITY_FAILED" ]]; then exit 1; fi
231229
232230
# TODO: Use cluster server artifact with 3 nodes when available
233-
# tool/test/start-cluster-servers.sh 3 &&
231+
# source tool/test/start-cluster-servers.sh 3 &&
234232
# bazel test //python/tests/integration:all --test_output=streamed --jobs=1 &&
235233
# export CLUSTER_FAILED= || export CLUSTER_FAILED=1
236234
# tool/test/stop-cluster-servers.sh
@@ -252,24 +250,23 @@ build:
252250
tool/test/stop-community-server.sh
253251
exit $TEST_SUCCESS
254252
255-
# TODO: Use cluster server artifact with 3 nodes when available (it would do the same thing as community now)
256-
# test-python-behaviour-cluster:
257-
# image: typedb-ubuntu-22.04
258-
# dependencies:
259-
# - build
260-
# type: foreground
261-
# command: |
262-
# export PATH="$HOME/.local/bin:$PATH"
263-
# export ARTIFACT_USERNAME=$REPO_TYPEDB_USERNAME
264-
# export ARTIFACT_PASSWORD=$REPO_TYPEDB_PASSWORD
265-
# bazel run @typedb_dependencies//tool/bazelinstall:remote_cache_setup.sh
266-
# bazel run @typedb_dependencies//distribution/artifact:create-netrc
267-
#
268-
# tool/test/start-cluster-servers.sh 3 &&
269-
# .factory/test-cluster.sh //python/tests/behaviour/... --test_output=streamed --jobs=1 &&
270-
# export TEST_SUCCESS=0 || export TEST_SUCCESS=1
271-
# tool/test/stop-cluster-servers.sh
272-
# exit $TEST_SUCCESS
253+
test-python-behaviour-cluster:
254+
image: typedb-ubuntu-22.04
255+
dependencies:
256+
- build
257+
type: foreground
258+
command: |
259+
export PATH="$HOME/.local/bin:$PATH"
260+
export ARTIFACT_USERNAME=$REPO_TYPEDB_USERNAME
261+
export ARTIFACT_PASSWORD=$REPO_TYPEDB_PASSWORD
262+
bazel run @typedb_dependencies//tool/bazelinstall:remote_cache_setup.sh
263+
bazel run @typedb_dependencies//distribution/artifact:create-netrc
264+
265+
source tool/test/start-cluster-servers.sh 3 &&
266+
.factory/test-cluster.sh //python/tests/behaviour/... --test_env=ROOT_CA=$ROOT_CA --test_output=streamed --jobs=1 &&
267+
export TEST_SUCCESS=0 || export TEST_SUCCESS=1
268+
tool/test/stop-cluster-servers.sh
269+
exit $TEST_SUCCESS
273270
274271
275272
test-http-ts-behaviour-community:

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,12 @@ typedb-test/test-integration/benchmarks/
7676

7777
# Test Runner Files #
7878
typedb-all/*
79+
typedb-cluster-all/*
80+
1/*
81+
2/*
82+
3/*
83+
4/*
84+
5/*
7985

8086
# Autogenerated Docs #
8187
http-ts/docs

0 commit comments

Comments
 (0)