Distributed query between two FC instances fails due to CSRF

[feifan-lin]

Version

deeae85

Bug Description

I am running two FC instances and need fc-server-2 to retrieve data from fc-server via:

• POST /query/search

I configured partner addresses, and the outbound call is executed, but partner data is not returned.

fc-server logs:

fc-server  | 2026-03-31T14:49:36.689Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-1] [65741fa2661e349a9c6c3254e693664f-189e6477a3afa921] o.s.security.web.FilterChainProxy        : Securing POST /query/search
fc-server  | 2026-03-31T14:49:36.747Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-1] [65741fa2661e349a9c6c3254e693664f-3c1afece56f2416e] o.s.security.web.csrf.CsrfFilter         : Invalid CSRF token found for http://host.docker.internal:8081/query/search

fc-server-2 logs:

fc-server-2  | 2026-03-31T14:42:57.271Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] m.m.a.RequestResponseBodyMethodProcessor : Read "application/json;charset=UTF-8" to [class AnnotatedStatement {<EOL>    statement: MATCH (n) RETURN n<EOL>    parameters: null<EOL>    servers: null<EOL> (truncated)...]
fc-server-2  | 2026-03-31T14:42:57.277Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] eu.xfsc.fc.server.service.QueryService   : search.enter; got statement: class AnnotatedStatement {
fc-server-2  |     statement: MATCH (n) RETURN n
fc-server-2  |     parameters: null
fc-server-2  |     servers: null
fc-server-2  |     annotations: null
fc-server-2  | }
fc-server-2  | 2026-03-31T14:42:57.284Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] e.x.fc.graphdb.service.Neo4jGraphStore   : queryData.enter; got query: GraphQuery(query=MATCH (n) RETURN n limit $limit, params={limit=100}, queryLanguage=OPENCYPHER, timeout=5, withTotalCount=true)
fc-server-2  | 2026-03-31T14:42:57.295Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] e.x.fc.graphdb.service.Neo4jGraphStore   : getDynamicallyAddedCountClauseQuery.enter; actual query: MATCH (n) RETURN n limit $limit
fc-server-2  | 2026-03-31T14:42:57.296Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] e.x.fc.graphdb.service.Neo4jGraphStore   : getDynamicallyAddedCountClauseQuery.exit; count query appended : MATCH (n) WITH count(*) as totalCount MATCH (n) RETURN totalCount, n limit $limit
fc-server-2  | 2026-03-31T14:42:57.311Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] e.x.fc.graphdb.service.Neo4jGraphStore   : doQuery; got result: [totalCount, n]
fc-server-2  | 2026-03-31T14:42:57.342Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] e.x.fc.graphdb.service.Neo4jGraphStore   : doQuery; record: {totalCount=1, n=node<0>}
fc-server-2  | 2026-03-31T14:42:57.350Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] e.x.fc.graphdb.service.Neo4jGraphStore   : doQuery.exit; returning: [{n={_classLabel=Class, _handleRDFTypes=0, _subClassOfRel=SCO, _handleMultival=1, _objectPropertyLabel=Relationship, _rangeRel=RANGE, _domainRel=DOMAIN, _keepLangTag=false, _keepCustomDataTypes=false, _classNamePropName=name, _relNamePropName=name, _handleVocabUris=3, _applyNeo4jNaming=false, _multivalPropList=[http://w3id.org/gaia-x/service#claimsGraphUri], _dataTypePropertyLabel=Property, _subPropertyOfRel=SPO}}]
fc-server-2  | 2026-03-31T14:42:57.364Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] o.s.w.r.f.client.ExchangeFunctions       : [38c4f289] HTTP POST http://host.docker.internal:8081/query/search
fc-server-2  | 2026-03-31T14:42:57.375Z DEBUG 1 --- [federated-catalogue-service] [reactor-http-epoll-5] [                                                 ] org.springframework.web.HttpLogging      : [38c4f289] Encoding [class AnnotatedStatement {<EOL>    statement: MATCH (n) RETURN n limit $limit<EOL>    parameters: {limit=100 (truncated)...]
fc-server-2  | 2026-03-31T14:42:57.417Z DEBUG 1 --- [federated-catalogue-service] [reactor-http-epoll-5] [                                                 ] o.s.w.r.f.client.ExchangeFunctions       : [38c4f289] [6f5554b9-1] Response 403 FORBIDDEN
fc-server-2  | 2026-03-31T14:42:57.420Z DEBUG 1 --- [federated-catalogue-service] [reactor-http-epoll-5] [                                                 ] org.springframework.web.HttpLogging      : [38c4f289] [6f5554b9-1] Read 104 bytes
fc-server-2  | 2026-03-31T14:42:57.421Z DEBUG 1 --- [federated-catalogue-service] [reactor-http-epoll-5] [                                                 ] eu.xfsc.fc.server.service.QueryService   : queryPartners.error; with object: null
fc-server-2  |
fc-server-2  | org.springframework.web.reactive.function.client.WebClientResponseException$Forbidden: 403 Forbidden from POST http://host.docker.internal:8081/query/search
fc-server-2  |  at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:323)
fc-server-2  |  Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
fc-server-2  | Error has been observed at the following site(s):
fc-server-2  |  *__checkpoint ⇢ 403 FORBIDDEN from POST http://host.docker.internal:8081/query/search [DefaultWebClient]
fc-server-2  | Original Stack Trace:
fc-server-2  |      at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:323)
fc-server-2  |      at org.springframework.web.reactive.function.client.DefaultClientResponse.lambda$createException$1(DefaultClientResponse.java:214)
fc-server-2  |      at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
fc-server-2  |      at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
fc-server-2  |      at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:122)
fc-server-2  |      at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
fc-server-2  |      at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
fc-server-2  |      at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
fc-server-2  |      at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
fc-server-2  |      at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2096)
fc-server-2  |      at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145)
fc-server-2  |      at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
fc-server-2  |      at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
fc-server-2  |      at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
fc-server-2  |      at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)
fc-server-2  |      at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:464)
fc-server-2  |      at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:518)
fc-server-2  |      at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:925)
fc-server-2  |      at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
fc-server-2  |      at io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:166)
fc-server-2  |      at io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:48)
fc-server-2  |      at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
fc-server-2  |      at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
fc-server-2  |      at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
fc-server-2  |      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
fc-server-2  |      at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
fc-server-2  |      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
fc-server-2  |      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
fc-server-2  |      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
fc-server-2  |      at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
fc-server-2  |      at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
fc-server-2  |      at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
fc-server-2  |      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
fc-server-2  |      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
fc-server-2  |      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
fc-server-2  |      at java.base/java.lang.Thread.run(Thread.java:1583)
fc-server-2  |
fc-server-2  | 2026-03-31T14:42:57.428Z DEBUG 1 --- [federated-catalogue-service] [tomcat-handler-3937] [82b50c5965c89aabec41c1963de7e1a8-fcb4d359c0ec0f51] eu.xfsc.fc.server.service.QueryService   : search.exit; returning results: class Results {
fc-server-2  |     totalCount: 1
fc-server-2  |     items: [{n={_classLabel=Class, _handleRDFTypes=0, _subClassOfRel=SCO, _handleMultival=1, _objectPropertyLabel=Relationship, _rangeRel=RANGE, _domainRel=DOMAIN, _keepLangTag=false, _keepCustomDataTypes=false, _classNamePropName=name, _relNamePropName=name, _handleVocabUris=3, _applyNeo4jNaming=false, _multivalPropList=[http://w3id.org/gaia-x/service#claimsGraphUri], _dataTypePropertyLabel=Property, _subPropertyOfRel=SPO}}]
fc-server-2  | }

I can accept moving to a newer version, but in current I still see CSRF enabled in: fc-service-server/src/main/java/eu/xfsc/fc/server/config/SecurityConfig.java.

What is the recommended way to make FC-to-FC /query/search work without modifying code? Is there any supported runtime configuration or deployment-level approach for this scenario?

Steps to Reproduce

1. Run two FC instances using two docker-compose files.

   • The second stack uses -2 suffixed container names (e.g., fc-server-2, keycloak-2, etc.).
   • For the second FC stack, all host port mappings are prefixed with 1 (e.g., "15432:5432", "18080:8080", "18081:8081", "17474:7474")

2. Configure federated query partner settings in server.environment (example for fc-server-2):

   • FEDERATED_CATALOGUE_QUERY_SELF=http://host.docker.internal:18081
   • FEDERATED_CATALOGUE_QUERY_PARTNERS_0=http://host.docker.internal:8081
   • SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=http://key-server:18080/realms/gaia-x
   • KEYCLOAK_AUTH_SERVER_URL=http://keycloak-2:8080
   • LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_SECURITY=DEBUG

   (The other FC has symmetric partner/self settings with swapped URLs.)

3. In ServiceConfig.java, include eu.xfsc.fc.graphdb.config in component scanning.

4. Retrieve access token and call distributed query on fc-server-2:

   • Call POST /query/search on fc-server-2 with Authorization: Bearer <access_token>.
   • Body: {"statement":"MATCH (n) RETURN n"}

5. Observe logs:

   • fc-server-2 calls partner /query/search and gets 403 FORBIDDEN.
   • fc-server logs show:
     • FilterChainProxy : Securing POST /query/search
     • CsrfFilter : Invalid CSRF token found for http://host.docker.internal:8081/query/search

Expected Behavior

POST /query/search should work in FC-to-FC service-to-service mode (distributed query) without requiring browser-style CSRF tokens.

Logs / Screenshots

Operating System

None

[clange]

Thanks for pointing out this issue. In today's bi-weekly catalogue developers' community call (where you are also invited) there was no one who knew the concrete answer. However, I could imagine two pointers:

• @dsukhoroslov do you maybe know the precise, deep answer in terms of how CSRF is implemented in the XFSC Federated Catalogue
• In the FACIS sister project Decentralized Catalogue Management (DCM), which has meanwhile been implemented (see its repository), the same problem might have been solved on a higher level, i.e., on the application level: by a low-code application ("flows") implemented in the XFSC Orchestration Engine, which harvests data from remote catalogues (XFSC or others) into a local catalogue. Could that be a solution for you?

I'm trying my best to provide a glance into the big picture here, and the DCM README also tries its best to do so, but indeed we agreed that a systematic overview would help of how things in XFSC/FACIS are connected.

[noVaSon]

Out of curiosity, I dipped into this a little, here is what I found out:

Two things are layered in this scenario.

1. CSRF on a stateless Bearer-token endpoint.

For the current stack (Spring Boot 3.5.7 / Security 6.5.6), the conventional fix is to mark the OAuth2 resource-serverpaths as CSRF-exempt using the lambda DSL:

http.csrf(csrf -> csrf.ignoringRequestMatchers("/query/search"))

However, Spring Security's own reference doc states that CSRF protection is unnecessary for endpoints authenticated by a non-cookie mechanism such as a Bearer JWT, because such requests cannot be forged from a browser cross-origin context (see https://docs.spring.io/spring-security/reference/6.5/servlet/exploits/csrf.html).

Every authenticated endpoint in SecurityConfig.java resolves to oauth2ResourceServer().jwt(...) - Bearer-only, no session, no cookie, no form login, no HTTP Basic (verified by grep across fc-service-server and fc-service-core). For that shape the conventional fix is a global CSRF disable in the lambda DSL:

  http.csrf(AbstractHttpConfigurer::disable)

Note the commented-out//.csrf().disable() line in SecurityConfig.java is the old chain style and won't compile under Spring Security 7. The lambda DSL above is forward-compatible with the upcoming Spring Boot 4 bump.

2. The identity that the second FC presents to the first.
Even with CSRF off, fc-server-2 forwarding an end-user JWT issued by its own Keycloak realm will be rejected at fc-server because the issuer doesn't match the resource server's configured issuer-uri.

The clean options are:

a) a shared Keycloak realm across federating FCs, an OAuth2 client-credentials grant for an FC-to-FC service identity (separate from the user identity, see https://docs.spring.io/spring-security/reference/6.5/servlet/oauth2/client/authorization-grants.html#oauth2-client-client-credentials), or
b) Keycloaks build-in cross-realm federation.

The arc42 chapters don't currently cover the FC-to-FC interaction model. A short note in eclipse-xfsc/docs/federated-catalogue/src/docs/architecture/chapters/06_runtime_view.adoc or 07_deployment_view.adoc would help certainly downstream integrators.

[feifan-lin]

Thanks for pointing out this issue. In today's bi-weekly catalogue developers' community call (where you are also invited) there was no one who knew the concrete answer. However, I could imagine two pointers:

• @dsukhoroslov do you maybe know the precise, deep answer in terms of how CSRF is implemented in the XFSC Federated Catalogue
• In the FACIS sister project Decentralized Catalogue Management (DCM), which has meanwhile been implemented (see its repository), the same problem might have been solved on a higher level, i.e., on the application level: by a low-code application ("flows") implemented in the XFSC Orchestration Engine, which harvests data from remote catalogues (XFSC or others) into a local catalogue. Could that be a solution for you?

I'm trying my best to provide a glance into the big picture here, and the DCM README also tries its best to do so, but indeed we agreed that a systematic overview would help of how things in XFSC/FACIS are connected.

Thanks for the guidance and the DCM suggestion.

We evaluated the DCM approach for the Digital Contracting Service (DCS).

During the integration/testing phase, several issues were identified in the DCM register/harvest flow, and related GitHub issues were already submitted.

It is still unclear to us whether FC-to-FC distributed query is the intended long-term federation/discovery approach, or whether DCM harvesting/synchronization is expected to be the primary solution.

Currently, DCM integration is still not stable enough for our FC federation/discovery use case, especially around:

• remote catalogue harvesting
• OAuth2 integration
• POST body mapping
• query APIs for harvested resources

For now, we decided to temporarily continue with a single FC setup shared between DCS instances while continuing the DCS/FC integration work.

We will revisit the federated setup once there is a clearer and more stable solution for either:

• FC-to-FC distributed query
• DCM harvesting/synchronization flow