Responsibilities:

• Develop and maintain scalable and reliable data pipelines that can ingest and process large volumes of data with high velocity and variety
• Build and optimize data models, ETL processes, and storage solutions (data lakes, data warehouses)
• Ensure data quality and compliance with data governance and security policies
• Design appropriate API for different data products
• Work closely with data scientists, analysts, and other stakeholders to understand data requirements and implement systems for data integration, data cleansing, and data analytics
• Monitor performance and advise on necessary infrastructure changes
• Define data retention policies and implement disaster recovery systems
• Troubleshoot and resolve any issues in the data pipeline
• Stay abreast of industry trends and best practices in data engineering and introduce new technology and tools where applicable

Requirements:

• Bachelor's or master's degree in Computer Science, Engineering, or a related field
• 4+ years of experience as a data engineer or in a similar role
• Proficient in Python programming language
• Experience with stream processing technologies such as Apache Spark, Apache Kafka, Apache Flink
• Experience with big data tools (e.g., Hadoop, Spark) and data pipeline
• Solid understanding of SQL and NoSQL databases
• Experience with analytical databases (e.g., ClickHouse, Druid)
• Knowledge of data warehousing concepts and experience with data modeling and schema design
• Familiarity with deployment automation such as GitLab CI/CD
• Familiarity with Docker and Kubernetes
• Passionate about ML-learning new technologies
• Problem Solving and communication skills and decision making