Role Overview
We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable and reliable data infrastructure. In this role, you will develop and maintain efficient data pipelines, manage large-scale data processing and streaming systems, and ensure data quality, performance, and availability. You will play a key role in strengthening our data platform to support data-driven decision-making across the organization.

Key Responsibilities
1. Data Infrastructure: Maintain and improve the company's data infrastructure, ensuring its reliability, availability, scalability, and performance.
2. Data Pipeline Development: Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and processing.
3. Big Data Processing: Develop and optimize distributed data processing jobs using PySpark.
4. Database & Streaming Management: Operate, maintain, and optimize ClickHouse clusters and Kafka infrastructure, ensuring performance, stability, and efficient resource utilization.
5. Workflow Orchestration: Develop and maintain production workflows using Apache Airflow, ensuring reliable execution, scheduling, and error handling.
6. Monitoring & Troubleshooting: Monitor infrastructure and pipeline health, proactively identify performance bottlenecks, and resolve production issues.
7. Data Quality & Reliability: Implement data validation and quality checks to ensure the accuracy, consistency, and reliability of data pipelines.

Requirements
• Minimum of 5 years of professional experience in a Data Engineering or similar role building data platforms.
• Python: Expert proficiency in Python, including advanced knowledge of common data processing libraries and engineering best practices.
• Big Data Processing: Demonstrated experience with large-scale data processing using PySpark.
• Database: Extensive, practical experience managing and tuning large-scale ClickHouse clusters for query performance, data retention, and stability.
• Streaming: Proven ability to operate, scale, and maintain production Kafka clusters, including in-depth knowledge of Kafka Connect and performance metrics.
• Data Modeling: Strong knowledge of SQL, data modeling, and techniques for optimizing queries within analytical (OLAP) databases.
• Monitoring: Experience utilizing monitoring tools (e.g., Prometheus, Grafana) to evaluate the health and performance of data infrastructure.
• Workflow Orchestration: Hands-on experience building and maintaining production pipelines with Apache Airflow.