Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kafka writer #990

Open
drasko opened this issue Dec 21, 2019 · 5 comments
Open

Add Kafka writer #990

drasko opened this issue Dec 21, 2019 · 5 comments
Assignees

Comments

@drasko
Copy link
Contributor

drasko commented Dec 21, 2019

FEATURE REQUEST

  1. Is there an open issue addressing this request? If it does, please add a "+1" reaction to the
    existing issue, otherwise proceed to step 2.

  2. Describe the feature you are requesting, as well as the possible use case(s) for it.

May Mainflux users have already legacy systems (apps) running on an enterprise bus which is often Kafka.

It would be high effort for them to switch to NATS, i.e. change their legacy apps to subscribe to NATS provided by Mainflux, or use Mainflux API to fetch the data.

For this purpose we can provide a NATS -> Kafka adapter/writer. It would be a small piece of software, very similar to existing writers (just that it will not write in some DB, but Kafka). It would just bluntly take messages from NATS and throw them into Kafka.

  1. Indicate the importance of this feature to you (must-have, should-have, nice-to-have).
    Must-have
@jruizaranguren
Copy link

@drasko
Copy link
Contributor Author

drasko commented Dec 21, 2019

Makes sense, but I think:

  • It should be really trivial, so that it merits lean implementation rather than dragging whole can of worms (I see tools like kafka2nats, stan2kafka, etc... over there)
  • We will probably need to parse Message protobuf struct and send payload-only in the Kafka, so some minimal processing should be done by our Transformer lib

@drasko
Copy link
Contributor Author

drasko commented Mar 8, 2020

This can be easily done through mProxy. Closing here, as this will not be main focus of the Core.

@drasko drasko closed this as completed Mar 8, 2020
@dborovcanin
Copy link
Collaborator

dborovcanin commented Mar 8, 2020

An example implementation can be found here: https://github.com/dusanb94/mainflux/tree/mproxy-kafka/mqtt/mproxy/kafka

@dborovcanin dborovcanin reopened this Jan 25, 2021
@dborovcanin dborovcanin changed the title Add Kafka adapter Add Kafka writer Jan 25, 2021
@dborovcanin
Copy link
Collaborator

The idea is to generalize the approach from this example using a custom writer (with a custom publisher), rather than creating only a custom publisher and coupling it with the MQTT adapter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

Successfully merging a pull request may close this issue.

5 participants