As a Data Engineer on our AI & Data team, you will have the autonomy to architect and implement our entire data lifecycle. You will be empowered to design the systems that transform raw data into critical insights and fuel our intelligent applications. Leveraging a state-of-the-art data platform provided by our DevOps team, you will own the data's journey from source to value, building sophisticated pipelines into ClickHouse and managing the data ecosystem for our Qdrant-powered AI applications. 

Key Responsibilities:

ETL/ELT Pipeline Development: Design, build, and maintain robust, scalable batch data pipelines using Apache Airflow to orchestrate workflows from PostgreSQL to our ClickHouse analytical database. 
Application-Level Service Administration: Take full ownership of the configuration and administration of our data services. This includes creating Kafka topics and managing ACLs, designing database schemas, managing users and permissions, and setting up MinIO buckets. 
Workflow Orchestration: Develop, schedule, and monitor complex data workflows (DAGs) in Apache Airflow, ensuring they are efficient, reliable, and well-documented. 
AI Data Provisioning: Develop processes to prepare, embed, and load data into our Qdrant vector database, often orchestrating these jobs within Airflow. 
Data Modelling & Optimization: Write highly optimized SQL and design logical data models that are performant and scalable for our analytics use cases. 
Collaboration & Ownership: Work closely with software developers and AI specialists to define their data needs and build the solutions to meet them. 

What We're Looking For:

Proven experience as a Data Engineer. 
Expert-level proficiency in Python for data processing and scripting. 
Deep knowledge of SQL and experience with relational databases (PostgreSQL is a must). 
Hands-on experience developing, scheduling, and monitoring complex data pipelines using Apache Airflow. 
Experience with ETL/ELT architecture, data modeling, and data warehousing concepts. 
Familiarity with containerizing applications using Docker. 

Bonus Points :

Direct experience with ClickHouse and its unique feature set. 
Experience with vector databases (Qdrant, Weaviate, etc.). 
Experience with both batch and streaming data processing paradigms.