[DMP 2025]: Scalable, Decoupled Data Ingestion Framework for Real-Time and Batch Integration

[tanishk2907]

Ticket Contents

Description

Project Context
MedPlat processes high volumes of data originating from a variety of sources—mobile apps, web portals, legacy systems, and third-party APIs. These data streams come in both real-time and batch formats, leading to growing challenges in integration, consistency, and timely availability for analytics and decision-making.

Currently, MedPlat uses tightly coupled, manually managed data integrations. This creates:

High maintenance overhead

Delays in data availability for dashboards and alerts

Inflexibility to onboard new systems or support real-time use cases

Objective
Design and implement a scalable, loosely coupled data ingestion framework that can efficiently integrate heterogeneous sources and deliver consistent, timely data to storage and analytics pipelines.

The solution should be resilient, modular, and extensible, with support for:

• Real-time ingestion via Kafka (or alternatives)
• Batch ingestion via APIs or file uploads
• A pluggable contract-driven interface for onboarding sources
• Observability and fault tolerance

Goals & Mid-Point Milestone

Goals

• Provide a unified data ingestion layer capable of handling both streaming and batch data
• Reduce system complexity and eliminate hardcoded integrations via a plug-and-play architecture
• Centralize data flow through a message queue or dispatcher service
• Enable clean data routing to downstream storage, analytics engines, and alerting systems
• Design system with fault tolerance, retry logic, and monitoring
• Create documentation for extending the pipeline to new modules or partner systems

Setup/Installation

No response

Expected Outcome

A centralized, scalable ingestion system that improves reliability, onboarding speed, and downstream analytics

Near real-time or scheduled availability of data for dashboards, alerts, and models

Reduced manual effort in maintaining integrations

Well-documented architecture and data contracts to guide future expansion

Acceptance Criteria

✅ Ingestion layer supports at least 2 batch and 2 real-time sources
✅ Plug-and-play design allows new sources/sinks to be configured easily
✅ Messages/data can be routed to multiple downstream systems (e.g., PostgreSQL, ElasticSearch, S3)
✅ Retry mechanisms, logging, and status monitoring are in place
✅ Architecture diagrams, setup scripts, and integration guides are included in the repo

Implementation Details

Event Broker (Real-time): Apache Kafka / Apache Pulsar / RabbitMQ

Batch Ingestion: Scheduled pull via API, FTP, or manual CSV ingest via CLI

Language: Python / Node.js / Golang

Data Schema: JSON, Avro, or Parquet with versioned contract registry

Observability: Logs via ELK/FluentD, Prometheus metrics for system health

Scalability: Modular microservices or container-based deployment (Docker + Kubernetes optional)

Mockups/Wireframes

Flow diagrams and data contracts to be created collaboratively with MedPlat data engineering team.

Product Name

Unified Data Ingestion Framework

Organisation Name

Bandhu

Domain

⁠Healthcare

Tech Skills Needed

Angular

Mentor(s)

@mvadodariya

Category

Backend

[Ganasekhar-gif]

Hi @tanishk2907 ,
I'm interested in contributing to this project as it aligns with my experience in data engineering, real-time analytics, and system design.
I’ve reviewed the ticket and plan to begin by designing a modular ingestion layer that supports both batch and real-time data sources (Kafka, API, file-based).

I'll follow a plug-and-play architecture, using message brokers for routing, and ensure support for logging, retries, and monitoring.

Please let me know if there are any priorities I should align with.
Looking forward to collaborating!

[akhila123-cmd]

I'm interested in this

[PuneetJadoun]

Hello @tanishk2907 I have knowledge of GoLang Can you assign this Project to me , I want to contribute to this project

[alokdangre]

hello @tanishk2907 , I am interested in this project, I will soon send you my strategy for this project

[Vaibhaav-Tiwari]

hey
i have prior experience with scaling systems
do let me know if there is some matrix or slack channel where we can communicate further

[ritankarsaha]

Hey, @tanishk2907 @mvadodariya I am interested in contributing to this project and will come up with a rough implementation idea soon ! 😁

[vaishnavisarad]

Hi @tanishk2907, I’m interested in contributing to this project. With my experience in backend development, data engineering, and real-time data streaming, I believe I can contribute effectively to designing a scalable and modular data ingestion framework. I’m familiar with the technologies like Kafka and have worked with data routing, logging, and monitoring systems in previous projects. Looking forward to collaborating!"

[khushalbhasin4488]

hi @tanishk2907 . i am interested in contributing to this project as I have prior experience of developing loosely coupled , scalable and efficient systems . I have developed many producer broker architecture micro-services . I have good experience in developing scheduling systems as well . I think i will be able to make meaningful and good contributions to the project. Looking forward to collaborate soon .

[Devang0000]

@tanishk2907 Hello! I’m Devang and I’m really interested in contributing to this MedPlat project, especially with building scalable and modular data ingestion systems. I’m also currently learning cloud technologies and eager to apply and grow these skills further in a collaborative environment. I’d love to join the team and contribute meaningfully. Please consider assigning this project to me. Looking forward to working together!

[Vaishno17]

Hi @tanishk2907

I’m very interested in contributing to the Unified Data Ingestion Framework project for MedPlat. The goal of building a scalable and modular ingestion layer for both real-time and batch data aligns closely with my skills and passion for backend systems.

I have experience with Python, Kafka, RESTful APIs, and containerized deployments using Docker. I’m also comfortable working with schema formats like JSON and Avro, and I understand the importance of fault tolerance, observability, and clean system design in healthcare tech.

I’d love to get involved—please let me know how I can begin contributing or if there are specific areas where help is currently needed.

[23shivay]

Weekly Learnings & Updates – Internship Project

Week 1

• Understood the overall architecture of the ingestion platform (Spring Boot, Kafka, PostgreSQL, ClickHouse).
• Explored the existing module design and ingestion flow.
• Learned how Kafka consumers and producers are configured.
• Set up the local development environment and tested the initial data ingestion flow.

Week 2

• Analyzed the existing data model for Hypertension and its mapping in PostgreSQL.
• Explored DTOs, entities, and service layers.
• Implemented the first ingestion handler for MemberHypertensionDetail.
• Learned about dual persistence (PostgreSQL + ClickHouse) and basic JDBC template usage.

Week 3

• Built the HypertensionIngestionHandler to persist data both in PostgreSQL and ClickHouse.
• Implemented service calls for PostgreSQL persistence.
• Learned how to insert data into ClickHouse using JdbcTemplate.
• Added detailed logging for ingestion success and failure.

Week 4

• Integrated GenericKafkaConsumer with IngestionHandler interface for plug-and-play ingestion.
• Implemented registry mapping between Kafka topics and handlers.
• Refactored code to make ingestion handlers more reusable.
• Understood how Spring deserializes Kafka messages into DTOs.

Week 5

• Explored error handling in Kafka consumers.
• Introduced retry logic for ingestion failures.
• Configured DLT (Dead Letter Topic) for failed messages after max retry attempts.
• Tested with simulated failures to ensure messages are retried and finally pushed to DLT.

Week 6

• Improved ingestion robustness by adding schema validations.
• Learned about JSON mapping issues (e.g., unrecognized fields like cbacId).
• Implemented fixes by aligning DTOs with producer payloads.
• Enhanced logging for retry attempts to better trace failure flows.

Week 7

• Optimized ingestion by refactoring mappings and reducing boilerplate code.
• Explored the feasibility of directly ingesting DTOs instead of model entities to simplify flow.
• Tested end-to-end ingestion with multiple topics.
• Documented ingestion flow and retry mechanism for internal reference.

Week 8

• Worked on scalability improvements for ingestion module registration.
• Analyzed pros and cons of different mapping strategies (DTO → Entity vs direct ingestion).
• Performed load testing with Kafka to analyze ingestion throughput.
• Handled edge cases where ingestion fails due to missing or extra fields.