Bootstrap test data from JSON fixtures instead of SQL migrations #802

dennisvang · 2025-11-13T08:35:32Z

Replace SQL test data migrations by JSON test data fixtures.

remove reference to test data migration files from flyway test config
replace test data migration files (.sql) by test data fixture files (.json) (extend default fixtures, do not duplicate)
specify test data location in bootstrap test config (depends on Enable loading of relational db fixtures from multiple directories #793)
get repository populator to run again before each test (because flyway clean is called before each test)
check if tests run slower due to fixtures (compared with the original migration approach) -> approx. 20 min on macOs and Windows, 11 min on Linux, very similar to the original sql migration implementation

this is more consistent with e.g. the 'locations' option for flyway

This allows us to include the prefix in the config, which is more flexible. For example, we can set file:fixtures for the default fixtures, which need to be overridable in the docker container, and we can set classpath:test-fixtures for the test fixtures, which can then be included in the test/resources dir. Moreover, this approach is similar to the way flyway.locations are specified.

the populator is done, but that does not necessarily mean any repositories were actually populated

This reverts commit 1ac2639.

…re file

* include root logger AppenderRef, so we only need to change the log level, when required * rename test logging config file for clarity and conformance to log4j2 best practices

….setup

…ault 02xx fixtures

…xtures

…04xx fixtures

dennisvang · 2025-12-09T08:33:57Z

For testing it would be convenient to have the ability to select specific fixtures files or filter fixture files.

We should allow passing complete ant-style patterns.

This way we could do things like "file:fixtures/02*.json" to include all fixtures in the 02xx range.

Also see PathMatchingResourcePatternResolver and AntPathMatcher for examples.

dennisvang · 2025-12-09T09:23:37Z

In case we ever need to override or extend the default bootstrap locations from application-testing.yml, we can annotate the test class as follows:

@TestPropertySource(properties = {
        "bootstrap.locations[0]=file:fixtures",
        "bootstrap.locations[1]=file:some/other/location"
})

Note that this overrides the default location list, instead of extending it, so we need to add the default locations explicitly if needed.

dennisvang · 2025-12-09T14:05:30Z

Warning

Apparently there are foreign key cascades in place for objects related to user accounts (and probably other relations as well?). As a result, when a fixture replaces an existing record (e.g. from a fixture applied earlier) by using the same uuid, all related objects are removed automatically. For example, adding a test fixture that modifies the default Albert Einstein user account causes all existing api-keys and saved-queries for that user to be deleted silently.

this enables us to specify simple directories, specific files, filters like 'fixtures/02*.json', and wildcards like 'fixtures/**/*.json'

the Path methods caused errors on windows if the location included ':' or '*' characters

description said Nikola Tesla, but uuid was for Albert Einstein

…ure data changed uuid and type for SearchSavedQuery objects to minimize interference with the existing acceptance tests

Due to the addition of DatabaseBootstrapTests, fixture 0130_test-users-with-api-keys-and-saved-queries.json now includes two new SearchSavedQuery objects. One of these new objects replaces the other, so the total number of SearchSavedQuery objects expected in the test database is increased by one.

Although I prefer .org, existing docs and tests expect .com. Also it will likely lead to confusion because people will keep trying to log in using the .com addresses.

…eTest

MarekSuchanek · 2025-12-16T12:13:28Z

src/test/java/org/fairdatapoint/WebIntegrationTest.java

+        // re-populate the database using fixtures
+        populator.populate(new Repositories(applicationContext));
+        // re-migrate acl data
+        // (TODO: AclMigration is in a subfolder of rdf/migration, but is it even related to rdf? Looks relational...)


That is indeed strange... it was probably there due to close relation to RDF records (it specifies access to an RDF records, but the ACL itself is relational - used to be in Mongo previously).

MarekSuchanek · 2025-12-16T12:13:56Z

src/test/java/org/fairdatapoint/config/MetadataTestConfig.java

 @TestConfiguration
 public class MetadataTestConfig {
-
+    // TODO: Looks like we can remove this class altogether.


Decided to move the annotation cleanup to a separate PR because there's more than expected.
I reset this branch by a few commits and did a force-push, which is a bit messy, sorry for that.

dennisvang mentioned this pull request Nov 13, 2025

WIP: Bootstrap relational database from fixtures #775

Draft

8 tasks

dennisvang changed the title ~~Remove sql test data migrations~~ Replace SQL test data migrations by JSON test data fixtures Nov 13, 2025

dennisvang changed the title ~~Replace SQL test data migrations by JSON test data fixtures~~ Bootstrap test data from JSON fixtures instead of SQL migrations Nov 13, 2025

dennisvang force-pushed the proposal/634-remove-sql-test-data-migrations branch from 4d9f371 to 0a80643 Compare November 13, 2025 16:56

dennisvang self-assigned this Nov 13, 2025

dennisvang added 11 commits December 8, 2025 15:09

remove data migrations from src/test/resources/test/db

3d28644

remove reference to test data migrations from test config

6ab040d

explicitly enable bootstrap in test config

ac0e0ae

add test-fixtures location to bootstrap config

d5915b1

rename 'db-fixtures-dirs' property to 'locations'

a86092a

this is more consistent with e.g. the 'locations' option for flyway

rename fixturesDir loop variable to location

f5abb80

clarify RepositoriesPopulatedEvent log message

e647d94

the populator is done, but that does not necessarily mean any repositories were actually populated

Revert "remove data migrations from src/test/resources/test/db"

049eeb2

This reverts commit 1ac2639.

replace V0001.1__dev-data-users.sql test data migration by json fixtu…

f3e49cb

…re file

cherry-pick: Make test logging config easier to use (#806)

caddb3d

* include root logger AppenderRef, so we only need to change the log level, when required * rename test logging config file for clarity and conformance to log4j2 best practices

dennisvang force-pushed the proposal/634-remove-sql-test-data-migrations branch from 0739c42 to caddb3d Compare December 8, 2025 14:13

dennisvang added 2 commits December 8, 2025 16:19

adapt bootstrap.locations for DatabaseBootstrapTests

d14ce0f

re-populate db from fixtures after flyway clean in WebIntegrationTest…

a1e6c8c

….setup

dennisvang force-pushed the proposal/634-remove-sql-test-data-migrations branch from dd7d2df to a1e6c8c Compare December 8, 2025 16:15

dennisvang added 5 commits December 8, 2025 20:09

delete V0001.2__dev-data-schemas.sql because it is covered by the def…

636c38c

…ault 02xx fixtures

delete V0001.3__dev-data-rds.sql because it is covered by the 03xx fi…

1aa0b16

…xtures

delete V0001.4__dev-data-membership.sql because it is covered by the …

79ec86f

…04xx fixtures

replace V0001.5__dev-settings.sql by 0500_test-settings.json

ec5711a

replace V0001.6__test-schemas.sql by 0600_test-schemas.json

883e341

dennisvang added 2 commits December 9, 2025 17:02

allow full ant-style location patterns for populator resources

e067b09

this enables us to specify simple directories, specific files, filters like 'fixtures/02*.json', and wildcards like 'fixtures/**/*.json'

adapt DatabaseBootstrapTests to use the default fixtures

10e7a18

dennisvang added 5 commits December 9, 2025 17:55

simplify resource location pattern for Windows compatibility

86225f2

the Path methods caused errors on windows if the location included ':' or '*' characters

catch all exceptions in RdfMetadataMigration.runMigration (dev only)

0dd2776

fix test data description to match user account uuid

a531516

description said Nikola Tesla, but uuid was for Albert Einstein

modify testDuplicateIdEntityOverwriteBootstrap and corresponding fixt…

2493791

…ure data changed uuid and type for SearchSavedQuery objects to minimize interference with the existing acceptance tests

dennisvang force-pushed the proposal/634-remove-sql-test-data-migrations branch from 05d7172 to ce810ea Compare December 10, 2025 12:21

dennisvang added 3 commits December 10, 2025 13:39

revert default user emails from example.org to example.com

79cde97

Although I prefer .org, existing docs and tests expect .com. Also it will likely lead to confusion because people will keep trying to log in using the .com addresses.

adapt order of test users to match the original expectation

65f641f

repopulate test database after flyway.clean in ResourceDefinitionCach…

2341d09

…eTest

dennisvang marked this pull request as ready for review December 10, 2025 14:04

dennisvang requested a review from MarekSuchanek December 10, 2025 14:04

dennisvang mentioned this pull request Dec 10, 2025

Factory reset from fixtures #812

Open

1 task

dennisvang force-pushed the proposal/634-remove-sql-test-data-migrations branch 2 times, most recently from de8c19c to 7682cd4 Compare December 16, 2025 11:25

MarekSuchanek reviewed Dec 16, 2025

View reviewed changes

MarekSuchanek approved these changes Dec 16, 2025

View reviewed changes

dennisvang force-pushed the proposal/634-remove-sql-test-data-migrations branch from 7682cd4 to 2341d09 Compare December 16, 2025 12:50

dennisvang merged commit a3c18ec into feature/634-boostrapping-fdp Dec 16, 2025
24 checks passed

dennisvang deleted the proposal/634-remove-sql-test-data-migrations branch December 16, 2025 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bootstrap test data from JSON fixtures instead of SQL migrations #802

Bootstrap test data from JSON fixtures instead of SQL migrations #802

Uh oh!

dennisvang commented Nov 13, 2025 •

edited

Loading

Uh oh!

dennisvang commented Dec 9, 2025 •

edited

Loading

Uh oh!

dennisvang commented Dec 9, 2025 •

edited

Loading

Uh oh!

dennisvang commented Dec 9, 2025 •

edited

Loading

Uh oh!

MarekSuchanek Dec 16, 2025 •

edited

Loading

Uh oh!

MarekSuchanek Dec 16, 2025

Uh oh!

dennisvang Dec 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Bootstrap test data from JSON fixtures instead of SQL migrations #802

Bootstrap test data from JSON fixtures instead of SQL migrations #802

Uh oh!

Conversation

dennisvang commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennisvang commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennisvang commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennisvang commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarekSuchanek Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarekSuchanek Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

dennisvang Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dennisvang commented Nov 13, 2025 •

edited

Loading

dennisvang commented Dec 9, 2025 •

edited

Loading

dennisvang commented Dec 9, 2025 •

edited

Loading

dennisvang commented Dec 9, 2025 •

edited

Loading

MarekSuchanek Dec 16, 2025 •

edited

Loading