Skip to content

Try to use this community to discuss patterns and libraries #147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Try to use this community to discuss patterns and libraries #147

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,25 @@ For [minikube](https://github.com/kubernetes/minikube/), [youkube](https://githu
* [Kubernetes events to Kafka](./events-kube/)
* [Container logs to Kafka](https://github.com/Yolean/kubernetes-kafka/pull/131)
* [Heapster metrics to Kafka](https://github.com/Yolean/kubernetes-kafka/pull/120)

## Awesome Streaming?

Kafka should only be the means to an end. How do you actually make things happen?

At Yolean we have a feeling we're reinventing a lot, but Schema Registry and Kafka Streams - as great as they are - fit few of our use cases because: a) Java only + b) not the kind of unix philosophy microservice-ish lightweight tool a small devops team like us is most productive with.

Instead of an "awesome" "curated list", let's start with labelled issues in this repo. Anyone can create them without a PR, and it facilitates discussion.

Suggested categories:

* [howto-topics](https://github.com/Yolean/kubernetes-kafka/issues?q=label%3Ahowto-topics): Topic naming and schema management, strategies and tools. We look for parity with the REST space's API documentation (Swagger etc) and service discovery (DNS SRV etc).
* [howto-streaming](https://github.com/Yolean/kubernetes-kafka/issues?q=label%3Ahowto-streaming): Lightweight streaming abstractions for different languages. Or as container images.
- filter, like bash + [kafkacat](https://github.com/edenhill/kafkacat) `|` [jq](https://github.com/stedolan/jq)
- split topic by field x, the value of x used for dynamic topic names
- join topics (consume using regex, write to single topic)
- count occurences based on regex or jq, export to [Prometheus](https://prometheus.io/docs/concepts/metric_types/#counter)
- stream-table joins in non-JVM languages
* [howto-frontend](https://github.com/Yolean/kubernetes-kafka/issues?q=label%3Ahowto-frontend): At-leat-once transport over HTTP. Reactive front-ends are as great a paradigm as event-driven services, but the tooling tends to be low-level ([socket.io](https://socket.io/) etc) or require us to emulate pubsub over request-response ([graphql](http://graphql.org/) etc).
* [howto-frontend](https://github.com/Yolean/kubernetes-kafka/issues?q=label%3Ahowto-frontend): Lightweight state cache, based on resolving events to state, for different languages. We felt we had to write [kafka-cache](https://github.com/Yolean/kafka-cache) for Node.js. Again Kafka Streams does such things, but not really as "do one thing and do it well".

Please share your ideas and observactions.