Kafka Queue Integration

Current Functionality:

The application currently only supports file-based, DB-based, and web API integrations for processing input data from source systems.

Enhancement:

The application now features Kafka Queues integration, enabling real-time processing capabilities. UniServe services are registered as Kafka Consumers for specific Kafka Topics. Consumer threads are launched by UniServe Services corresponding to the number of partitions in the relevant Kafka Topic. Source systems act as Kafka producers, pushing input files/data structures into these partitions as jobs—following a predefined and agreed-upon structure.

UniServe services utilize a pull-based technique; once registered to the Kafka server, they continuously listen for incoming jobs. Upon receiving a Kafka job in the respective partitions of the Topic, UniServe passes it to the application's PMS server in realtime. From there, the application distributes the job to respective clients based on bandwidth and load to process the job in realtime.

This integration supports near real-time communications with customers and rapid responses, offering faster processing compared to traditional databases and other integration methods. The application provides comprehensive tracking of all message queues, ensuring better performance and efficiency in message handling.

Additionally, in terms of logs and audit trails, the application meticulously records all received messages, including timestamps for message reception, push, job completion, and related events.

Benefits:

• Real-time Processing: The application leverages Kafka Queues integration for real-time processing capabilities, resulting in faster and more responsive communications with customers.

• Improved Performance: By utilizing Kafka Queues, the application achieves better performance compared to traditional database and integration methods, ensuring efficient handling of incoming jobs.

• Comprehensive Tracking: The system tracks all message queues comprehensively, providing visibility into the status and flow of messages from reception to completion.

• Efficient Communication: Improves communication efficiency and responsiveness by allowing for near-real-time communication and rapid responses.