Exclude subjects from stream mirror/sources

[francoisprunier]

Proposed change

Being able to configure a stream mirror/sources to include all subjects except one.

At the moment you can manually list all subjects you want to include, but that's not practical if the list is very long or not known in advance.

Here is an example configuration:

{
  "name": "replication-in",
(...)
  "sources": [
    {
      "name": "aggregated-replication",
      "SubjectTransforms": [
      {
        "src" : "replication.out.!site-a.>",
        "dest" : "replication.in.> 
      }
      ]
      "external": {
        "deliver": "",
        "api": "$JS.domainMainHub.API"
      }
    }
}

Notice the ‘!’ exclamation mark.

In this example, an hypothetical 'site-a' would tell jetstream to source all messages from aggregated-replication except for subjects ‘replication.out.site-a.>’. See the use case for the rationale behind this.

Use case

Our software that replicates its state between different sites by replicating RPC calls it receives for creates/updates/deletes. So any time it receives a call, for example create_foo(param1,param2), it will put that in a message and send it to all the other sites who then apply it locally. In a nutshell, we're doing logical replication.

Our requirements are:

• apart from the "main" site, the other sites should not have to know about each other.
• messages need to persist, a.k.a. network connections can fail between sites.

Based on those requirements, we decided to go with stream mirrors:

• each site publishes to a stream locally, let's call it REP-OUT (so the messages are not lost if connection between sites is lost)
• the main site mirrors all those REP-OUT streams into one stream, let's call it AGG (so only the main site has knowledge of all the others)
• each site mirrors AGG to a local stream, let's call it REP-IN, to consume incoming replication messages from it

However that means that each site would receive its own messages as part of AGG, those messages would be mirror'ed to AGG, and then be mirror'ed again locally, that would be a lot of traffic for nothing.

Unless the REP-IN config can be made to say: mirror the AGG stream for all subjects except for subjects containing our site, the topic for this proposal.

Contribution

Yes, we could provide a PR.

[hpvd]

+1