Senior Data Platform Engineer

We are building a modern, on-premise data platform from the ground up to power analytics, risk management, compliance, and AI-driven insights across our brokerage operations. This is a rare opportunity to architect a greenfield lakehouse platform end-to-end — from ingestion through governance to an AI-powered analytics agent — in a regulated financial environment with real budget and headcount to build it right.
We're looking for a hands-on Senior Data Platform Engineer who has done this before: someone who can make sound architectural trade-offs, mentor a growing team, and isn't afraid to operate complex distributed systems in production.
What You'll Work On

• Design and operate a Lakehouse architecture (Medallion: Bronze/Silver/Gold) on Apache Iceberg with Nessie as a versioned catalog, backed by on-premise object storage (MinIO)
• Build and maintain CDC pipelines from multiple OLTP databases using Debezium and Kafka Connect
• Own the Kafka-based streaming backbone serving both batch ingestion and real-time (sub-second) use cases such as market data and trading event processing
• Develop stateful stream processing with Apache Flink for real-time risk exposure calculations, anomaly detection, and alerting
• Build and maintain transformation pipelines with dbt, orchestrated via Airflow
• Support analytical query workloads on Trino and BI delivery through Superset
• Design and enforce data governance: cataloging and lineage (OpenMetadata/DataHub), fine-grained access control (Apache Ranger), and audit logging suitable for a regulated environment
• Deploy and operate platform components across a hybrid Kubernetes/VM footprint (stateless services on K8s, stateful services such as Kafka and MinIO on VMs)
• Contribute to capacity planning, on-call practices, and platform reliability as the system scales toward production-critical status

Must-Have Qualifications

• 5+ years of experience building and operating production data infrastructure
• Strong hands-on experience with Apache Kafka (including Kafka Connect and CDC patterns, ideally Debezium)
• Practical experience with a modern table format (Apache Iceberg or Delta Lake) and lakehouse patterns
• Proficiency with a distributed SQL query engine (Trino, Presto, or similar)
• Experience with dbt for data transformation and modeling
• Experience with workflow orchestration tools (Airflow or equivalent)
• Solid understanding of Kubernetes for deploying and operating stateful and stateless workloads
• Strong SQL and at least one of Python/Scala/Java for pipeline development
• Demonstrated experience designing data architectures with data governance, security, and auditability as first-class concerns

Strongly Preferred

• Experience with Apache Flink or another stream processing engine for stateful, low-latency workloads
• Experience with data catalog/lineage tools (OpenMetadata, DataHub, or Amundsen)
• Experience with fine-grained access control systems (Apache Ranger or equivalent)
• Prior experience in financial services, trading, or another regulated industry with strict data residency requirements
• Experience operating fully on-premise (non-cloud) infrastructure at scale
• Experience with Nessie or git-like data versioning for lakehouse catalogs

What Makes This Role Different

• Greenfield build: you'll shape the architecture, not inherit legacy debt
• Strict on-premise requirement means solving real distributed-systems problems without leaning on managed cloud services
• Direct exposure to both the batch/analytical side and the real-time/streaming side of the platform