Factory reset from fixtures #812

dennisvang · 2025-12-02T11:41:33Z

Possible implementation of factory reset using repository populator.

The basic idea is that we remove the relevant entries from the fixture history, update the list of resources accordingly, and then run the populator again.

TODO:

remove old reset files from src/main/resources/org/fairdatapoint/service/reset

for clarity

using TreeSet to ensure uniqueness and lexicographic ordering

…ulator

and DRY usage of removeFromFixtureHistory

MarekSuchanek

👏🏻 I like this approach... it should be then in docs that resetting works only with those "default" fixture files/resources (so it is clear that in some use cases you might want to override those and in some create separate files).

As a side effect, reset might apply new fixture resources that were not there previously 🤔 but I guess that does not matter as it would happen upon any restart anyway, right?

MarekSuchanek · 2025-12-10T12:49:42Z

src/main/java/org/fairdatapoint/service/reset/ResetService.java

-
-    private void clearResourceDefinitions() {
-        log.debug("Clearing resource definitions");
+        resourceDefinitionChildMetadataRepository.deleteAll();


Are these added to make sure nothing is left out? Otherwise, cascading delete should work I believe...

Hi @MarekSuchanek thanks for the review! :)

You're right, the deleteAll() calls for metadataSchemaUsageRepository, resourceDefinitionChildMetadataRepository, andresourceDefinitionLinkRepository are redundant.
However, line 188 (resourceDefinitionChildRepository.deleteAll()) does seem to be required, for some reason.

Still, I like to be explicit here, and not rely on cascades.

Note
In my experience cascade deletes, in general, can lead to very nasty surprises.
One example of cascade deletes causing confusion is in #802, where it turned out that overriding the default Einstein user account in a test fixture caused the related objects to be deleted as well.

dennisvang · 2025-12-10T15:44:32Z

👏🏻 I like this approach... it should be then in docs that resetting works only with those "default" fixture files/resources (so it is clear that in some use cases you might want to override those and in some create separate files).

As a side effect, reset might apply new fixture resources that were not there previously 🤔 but I guess that does not matter as it would happen upon any restart anyway, right?

Hi @MarekSuchanek , yes you're right that the current approach only works with the default fixture files, and that could lead to problems later on... 🤔

One of the challenges is that we cannot know how people are going to name their fixture override files and which entities they will contain.

I do see a few options to mitigate that:

As you mentioned above, we could simply document the limitation and leave it at that, or
we could remove the fine-grained reset functionality and always clear everything, in which case we can simply clear the entire fixture history repository, or
we could prescribe subfolders in the fixtures dir, corresponding to the ResetService.clear*() methods, and clear all files from those folders, or
we could prescribe part of the fixture filename and use that to remove groups of files from the fixture history (this is explained in the post below).

To expand upon option 3., we could distribute our default fixtures over the following required subfolders, where the subfolder names match the names of the relevant org.fairdatapoint.entity subfolders:

fixtures/apikey (corresponding to clearApiKeys)
fixtures/membership (corresponding to clearMemberships)
fixtures/user (corresponding to clearUsers)
fixtures/search (corresponding to clearUsers)
fixtures/schema (corresponding to clearMetadataSchemasAndResourceDefinitions)
fixtures/resource (corresponding to clearMetadataSchemasAndResourceDefinitions)

and then adapt ResetService.removeFromFixtureHistory() and fixtureHistoryRepository to remove records by subfolder name instead of filename.

What do you think?

dennisvang · 2025-12-11T09:01:43Z

To expand upon option 4.:

It may be easier to introduce a naming convention for fixtures files, as in

<zero-padded-number>_<package-name>_<custom-description>.json

or as regex

^(?<order>[0-9]{4})_(?<package>[a-z]+)_(?<description>[a-zA-Z0-9\-]+)\.json$

where <package> matches the name of the relevant org.fairdatapoint.entity subfolders:

apikey
membership
user
schema
search
resource

with the restriction that the fixtures in each file must belong to the specified package.

EDIT: this idea is worked out in the following commits

…n pattern matching a regex like ^(?<order>[0-9]{4})_(?<package>[a-z]+)_(?<description>[a-zA-Z0-9\-]+)\.json$

and remove the explicit filename tests, because this is now implicitly done during population before each test

todo: adapt tests

and same for corresponding tests to improve cohesion

[skip ci]

MarekSuchanek · 2025-12-14T17:54:26Z

Indeed the option 4 looks the best, although it still needs to be documented and naturally expects that the name will correspond with contents. But we should expect people adding/overriding the fixtures to be knowledgeable about such things. It is a great solution.

Wrt the TODOs in comments, I agree with moving the methods to BootstrapService (but probably with renaming reset to something like populate).

…osal/634-reset

dennisvang · 2025-12-18T14:36:03Z

Not done yet:

Running a fresh FDP with development profile, if we reset metadata and resource definitions at the same time (or all at the same time):

we get the following error:

...
2025-12-18 15:31:47,909 86794 [http-nio-8080-exec-1] INFO  org.fairdatapoint.service.reset.ResetService - Resetting to factory defaults
2025-12-18 15:31:47,909 86794 [http-nio-8080-exec-1] DEBUG org.fairdatapoint.service.reset.ResetService - Clearing membership permissions
2025-12-18 15:31:47,927 86812 [http-nio-8080-exec-1] DEBUG org.fairdatapoint.service.reset.ResetService - Clearing memberships
2025-12-18 15:31:47,952 86837 [http-nio-8080-exec-1] DEBUG org.fairdatapoint.service.reset.ResetService - Clearing ACL cache
2025-12-18 15:31:47,952 86837 [http-nio-8080-exec-1] DEBUG org.fairdatapoint.service.reset.ResetService - Clearing metadata
2025-12-18 15:31:48,015 86900 [http-nio-8080-exec-1] DEBUG org.fairdatapoint.service.reset.ResetService - Creating default metadata
2025-12-18 15:31:48,017 86902 [http-nio-8080-exec-1] DEBUG org.fairdatapoint.service.reset.ResetService - Clearing metadata schemas and resource definitions
2025-12-18 15:31:48,113 86998 [http-nio-8080-exec-1] WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 23502
2025-12-18 15:31:48,114 86999 [http-nio-8080-exec-1] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: null value in column "target_resource_definition_id" of relation "resource_definition_child" violates not-null constraint
  Detail: Failing row contains (b8648597-8fbd-4b89-9e30-5eab82675e42, 77aaad6a-0136-4c6e-88b9-07ffccd0ee4c, null, https://w3id.org/fdp/fdp-o#metadataCatalog, Catalogs, http://www.w3.org/ns/dcat#themeTaxonomy, 0, 2025-12-18 15:30:27.037585+01, 2025-12-18 15:31:48.100645+01).
2025-12-18 15:31:48,124 87009 [http-nio-8080-exec-1] DEBUG org.springframework.web.servlet.DispatcherServlet - Failed to complete request: org.springframework.dao.DataIntegrityViolationException: could not execute statement [ERROR: null value in column "target_resource_definition_id" of relation "resource_definition_child" violates not-null constraint
  Detail: Failing row contains (b8648597-8fbd-4b89-9e30-5eab82675e42, 77aaad6a-0136-4c6e-88b9-07ffccd0ee4c, null, https://w3id.org/fdp/fdp-o#metadataCatalog, Catalogs, http://www.w3.org/ns/dcat#themeTaxonomy, 0, 2025-12-18 15:30:27.037585+01, 2025-12-18 15:31:48.100645+01).] [update resource_definition_child set order_priority=?,relation_uri=?,source_resource_definition_id=?,tags_uri=?,target_resource_definition_id=?,title=?,updated_at=? where uuid=?]; SQL [update resource_definition_child set order_priority=?,relation_uri=?,source_resource_definition_id=?,tags_uri=?,target_resource_definition_id=?,title=?,updated_at=? where uuid=?]; constraint [target_resource_definition_id" of relation "resource_definition_child]
2025-12-18 15:31:48,126 87011 [http-nio-8080-exec-1] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.dao.DataIntegrityViolationException: could not execute statement [ERROR: null value in column "target_resource_definition_id" of relation "resource_definition_child" violates not-null constraint
  Detail: Failing row contains (b8648597-8fbd-4b89-9e30-5eab82675e42, 77aaad6a-0136-4c6e-88b9-07ffccd0ee4c, null, https://w3id.org/fdp/fdp-o#metadataCatalog, Catalogs, http://www.w3.org/ns/dcat#themeTaxonomy, 0, 2025-12-18 15:30:27.037585+01, 2025-12-18 15:31:48.100645+01).] [update resource_definition_child set order_priority=?,relation_uri=?,source_resource_definition_id=?,tags_uri=?,target_resource_definition_id=?,title=?,updated_at=? where uuid=?]; SQL [update resource_definition_child set order_priority=?,relation_uri=?,source_resource_definition_id=?,tags_uri=?,target_resource_definition_id=?,title=?,updated_at=? where uuid=?]; constraint [target_resource_definition_id" of relation "resource_definition_child]] with root cause
org.postgresql.util.PSQLException: ERROR: null value in column "target_resource_definition_id" of relation "resource_definition_child" violates not-null constraint
  Detail: Failing row contains (b8648597-8fbd-4b89-9e30-5eab82675e42, 77aaad6a-0136-4c6e-88b9-07ffccd0ee4c, null, https://w3id.org/fdp/fdp-o#metadataCatalog, Catalogs, http://www.w3.org/ns/dcat#themeTaxonomy, 0, 2025-12-18 15:30:27.037585+01, 2025-12-18 15:31:48.100645+01).
...

~~However, the issue is not reproduced by testResetToFactoryDefaultsAll(), not even if we exclude the test-fixtures and if we don't clear the repositories at the start of the test.~~ EDIT: I forgot that BaseIntegrationTest is not transactional, contrary to DataJpaTest. The issue is reproduced if we annotate the test with @transactional.

Note that reset does work if we don't clear the metadata, but clearMetadata() does not even modify the relational db, only the triple store...

This reproduces the DataIntegrityViolationException 'null value in column target_resource_definition_id of relation resource_definition_child violates not-null constraint'

dennisvang · 2025-12-22T16:48:19Z

DispatcherServlet - Failed to complete request: org.springframework.dao.InvalidDataAccessApiUsageException: 
  org.hibernate.TransientObjectException: 
    persistent instance references an unsaved transient instance of 'org.fairdatapoint.entity.resource.ResourceDefinition'
    (save the transient instance before flushing)

coming from

FAIRDataPoint/src/main/java/org/fairdatapoint/service/membership/MembershipService.java

Line 79 in 5f8fd38

membershipRepository.findByUuid(KnownUUIDs.MEMBERSHIP_OWNER_UUID)

dennisvang added 5 commits December 2, 2025 11:02

remove unnecessary annotations from FixtureHistoryRepositoryTests

627fb84

add FixtureHistoryRepository.deleteByFilenameIn() with test

b4c43e5

rename repositoryPopulator to repositoryPopulatorFactoryBean

e0b3202

for clarity

move resource logic into separate BootstrapConfig.getNewResources method

c7bb0f4

using TreeSet to ensure uniqueness and lexicographic ordering

implement restoreDefaultUsers and restoreDefaultMemberships using pop…

8b133f7

…ulator

dennisvang mentioned this pull request Dec 2, 2025

WIP: Bootstrap relational database from fixtures #775

Draft

8 tasks

dennisvang added 3 commits December 2, 2025 12:57

fix type definitions

ae816d1

implement generic restoreDefaultFixtures method

6cca79a

and DRY usage of removeFromFixtureHistory

fix indentation

f00c6d7

dennisvang marked this pull request as ready for review December 8, 2025 14:57

dennisvang requested a review from MarekSuchanek December 9, 2025 08:02

MarekSuchanek approved these changes Dec 10, 2025

View reviewed changes

dennisvang added 5 commits December 11, 2025 11:36

rename fixture files according to <order>_<package>_<description>.jso…

6f7fad1

…n pattern matching a regex like ^(?<order>[0-9]{4})_(?<package>[a-z]+)_(?<description>[a-zA-Z0-9\-]+)\.json$

add BootstrapConfig.checkFixtureFilename with tests

b935b16

rename to BootstrapConfig.validateFixtureFilename and throw if invalid

223c130

validate fixture names before populating

c6f5201

and remove the explicit filename tests, because this is now implicitly done during population before each test

replace deleteByFilenameIn by deleteByFilenameContains (wip)

ca112b1

todo: adapt tests

dennisvang force-pushed the proposal/634-reset branch from 04d0073 to ca112b1 Compare December 12, 2025 08:48

dennisvang added 4 commits December 12, 2025 09:56

specify allowed package names in fixture filename pattern

da4ed88

use generic arg name for deleteByFilenameContains and fix test

7503290

add FixtureHistoryRepository.deleteByFilename with test

e27cc72

move fixture related methods into new BootstrapService

c253a56

and same for corresponding tests to improve cohesion

dennisvang force-pushed the proposal/634-reset branch from 7ebe519 to c253a56 Compare December 12, 2025 13:19

dennisvang added 4 commits December 12, 2025 14:51

todo move to BootstrapService

3169903

add a rudimentary test for the ResetService

75b843f

add license headers

3ebee9c

reorder ResetService class attributes

ae89ffb

[skip ci]

dennisvang added 3 commits December 15, 2025 12:26

Merge branch 'proposal/634-remove-sql-test-data-migrations' into prop…

1a95957

…osal/634-reset

split up and rename test-fixtures to match required filename pattern

36300bd

run testResetToFactoryDefaultsAll as admin

e52a675

dennisvang added 4 commits December 18, 2025 16:46

check entity creation time in testResetToFactoryDefaultsAll

921fde3

Merge branch 'feature/634-boostrapping-fdp' into proposal/634-reset

f90a2de

make testResetToFactoryDefaultsAll transactional

88f0f55

This reproduces the DataIntegrityViolationException 'null value in column target_resource_definition_id of relation resource_definition_child violates not-null constraint'

fix RescourceDefinition cascade delete problem

c0d2c41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Factory reset from fixtures #812

Factory reset from fixtures #812

Uh oh!

dennisvang commented Dec 2, 2025 •

edited

Loading

Uh oh!

MarekSuchanek left a comment

Uh oh!

MarekSuchanek Dec 10, 2025

Uh oh!

dennisvang Dec 10, 2025 •

edited

Loading

Uh oh!

dennisvang commented Dec 10, 2025 •

edited

Loading

Uh oh!

dennisvang commented Dec 11, 2025 •

edited

Loading

Uh oh!

MarekSuchanek commented Dec 14, 2025

Uh oh!

dennisvang commented Dec 18, 2025 •

edited

Loading

Uh oh!

dennisvang commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Factory reset from fixtures #812

Are you sure you want to change the base?

Factory reset from fixtures #812

Uh oh!

Conversation

dennisvang commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarekSuchanek left a comment

Choose a reason for hiding this comment

Uh oh!

MarekSuchanek Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

dennisvang Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dennisvang commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennisvang commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarekSuchanek commented Dec 14, 2025

Uh oh!

dennisvang commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennisvang commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dennisvang commented Dec 2, 2025 •

edited

Loading

dennisvang Dec 10, 2025 •

edited

Loading

dennisvang commented Dec 10, 2025 •

edited

Loading

dennisvang commented Dec 11, 2025 •

edited

Loading

dennisvang commented Dec 18, 2025 •

edited

Loading