Senior Data Engineer

What You'll Do:

• Design, build, and maintain scalable data pipelines for batch and streaming workloads.
• Develop and maintain data driver applications/connectors for ingesting data from various sources.
• Implement and manage Change Data Capture (CDC) processes to sync data across systems in near real-time.
• Collaborate with ML teams, providing knowledge/support around serving and integrating ML models into data workflows.
• Develop, optimize, and maintain Apache Spark jobs for large-scale data processing.
• Design and manage OLAP database schemas and queries for analytical workloads.
• Write efficient, optimized SQL for data transformation, reporting, and analytics.
• Manage and maintain HDFS-based storage infrastructure.
• Build and maintain Kafka-based messaging/streaming pipelines.

What You'll Need:

• Strong SQL skills (query optimization, schema design).
• Hands-on experience with Python, including FastAPI, NumPy, Pandas, and PySpark.
• Hands-on experience with Apache Spark (development + cluster maintenance).
• Experience with Kafka (producers/consumers, topic management).
• Familiarity with HDFS and distributed storage systems.
• Experience implementing CDC solutions (e.g., Debezium, or similar).
• Working knowledge of OLAP databases (e.g., ClickHouse, Druid, etc.).
• Experience building data driver applications or custom connectors.

Nice to Have:

• Knowledge of Golang (not required, but a strong plus).
• Basic understanding of ML model serving/integration is a plus.