​Job Summary:

We are looking for an experienced Kafka Architect to design and implement scalable, high-throughput data streaming solutions using Apache Kafka on Cloudera (on-premises). The ideal candidate will have a strong background in distributed systems, data pipelines, and real-time data processing.

​

Key Responsibilities:

-Design and implement scalable Kafka-based architectures using open-source -Kafka on Cloudera on-premises infrastructure.

-Lead the setup, configuration, and optimization of Kafka clusters.

-Define standards and best practices for Kafka producers, consumers, and streaming applications.

-Integrate Kafka with various data sources, storage systems, and enterprise applications.

-Monitor Kafka performance and ensure high availability, fault tolerance, and data security.

-Collaborate with DevOps, Data Engineering, and Application teams to support real-time data needs.

-Automate deployment and configuration using tools like Ansible, Terraform, or Cloudera Manager.

-Provide technical leadership and mentorship to junior team members.

​

Required Skills:

-Strong hands-on experience with Apache Kafka (including Kafka Connect, Kafka Streams).

-Experience with Cloudera distribution for Kafka on on-premises environments.

-Proficiency in designing high-volume, low-latency data pipelines.

-Solid knowledge of Kafka internals – topics, partitions, consumer groups, offset management, etc.

-Experience with data serialization formats like Avro, JSON, Protobuf.

-Proficient in Java, Scala, or Python for Kafka-based development.

-Familiarity with monitoring tools (Prometheus, Grafana, Confluent Control Center, etc.).

-Understanding of networking, security (SSL/SASL), and data governance.

-Experience with CI/CD pipelines and containerization (Docker, Kubernetes) is a plus.