CreateFuture - Lead Data Engineer

• Leading large-scale data migrations from on-premises or legacy platforms to cloud-native architectures on AWS — including Lakehouse designs using Databricks or Snowflake • Designing and building low-latency data pipelines that process real-time betting events, odds updates, and market data at high throughput • Implementing robust data integrity controls — validation, reconciliation, and auditability — across betting, trading, and customer data domains • Architecting streaming and micro-batch solutions using AWS Kinesis, Apache Kafka, or MSK for real-time sportsbook and in-play data feeds • Working with AWS platform services including S3, Glue, Lambda, Athena, Redshift, Lake Formation, Step Functions, and MWAA • Supporting regulatory and compliance reporting — ensuring data pipelines meet the auditability and lineage requirements of the UK Gambling Commission and international regulators • Leading and mentoring a team of data engineers, setting standards for code quality, testing, and documentation • Contributing to technical proposals, pre-sales scoping, and architecture reviews with iGaming clients • We'd love to talk to you if... • You know the iGaming data landscape • You've worked in or closely with sportsbook, exchange, or casino platforms — and understand the data models behind markets, selections, bets, and settlements • You've built or maintained low-latency data feeds — odds ingestion, in-play event streams, price change feeds — where milliseconds matter • You understand the data integrity challenges specific to betting: reconciliation between trading and settlement systems, detection of data drift, and auditability for regulatory purposes • You've delivered or contributed to large-scale data migrations — moving high-volume transactional or event data from legacy estates to the cloud without losing fidelity or uptime • You're strong on AWS data engineering • AWS is your primary cloud — you're confident across Glue, Lambda, Athena, S3, Redshift, RDS, Kinesis, MSK, and Step Functions • You've built real-time or near-real-time pipelines using streaming frameworks — Kinesis Data Streams, Kafka, Flink, or Spark Streaming • You understand AWS cost models and how to optimise high-throughput workloads — Spot compute, reserved capacity, and right-sizing for event-driven architectures • AWS certifications are a bonus — Solutions Architect or Data Analytics Specialty • You know your tools • You know your tools • Solid experience with Python, SQL, and Spark — and pipeline tools such as dbt or Airflow • Comfortable with data quality frameworks and testing — Great Expectations, dbt tests, or custom validation layers • Experience with infrastructure-as-code (Terraform, AWS CDK) is a plus • Familiar with broader cloud platforms — Azure (Data Factory, Synapse) or GCP (Dataflow, BigQuery) — is beneficial but not essential • You thrive in a consultancy setting • You've led or mentored data engineers and are comfortable owning technical direction on a complex engagement • You can navigate the politics of a large operator — multiple teams, competing priorities, and legacy systems that can't just be switched off • You balance hands-on delivery with client engagement, and know when to push back on scope and when to be pragmatic • Tech snapshot • Tech snapshot • Primary cloud • AWS — Glue, Lambda, Athena, S3, Redshift, RDS, Kinesis, MSK, Lake Formation, Step Functions, MWAA • Streaming • AWS Kinesis, Apache Kafka / MSK, Apache Flink, Spark Streaming • Languages • Python, SQL, Spark • Pipeline tools • dbt, Airflow / AWS MWAA, AWS Glue • Platforms • Databricks, Snowflake, Amazon Redshift • Data quality • Great Expectations, dbt tests, custom reconciliation frameworks • IaC / DevOps • Terraform, AWS CDK (desirable) • What we’ll offer you: • We trust people to do their best work. That means flexibility over rigid rules, impact over activity, and real investment in your growth both professionally and personally. You’ll be part of a supportive, and friendly culture, surrounded by smart, curious people who care deeply about what they do.We offer flexible working, including hybrid and remote options. Our office hubs are located in Edinburgh, Leeds, Manchester, London and Bulgaria, with occasional travel to client sites or CreateFuture offices when needed. • We trust you to manage your time balancing collaboration with client time and focused work. What matters is the impact you have, not how busy you look. • Our hiring process • Our hiring process • We try to keep our hiring process clear, fair and respectful of your time. We aim to get back to everyone who applies and we will be upfront about where you are in the process. • It usually looks like this: • Call with our Talent Acquisition Team • Role specific capability interview • Depending on the role, we might also ask you to do a short presentation, a practical or technical task or have a values focused conversation. We will explain what is involved before anything happens. • Inclusion at CreateFuture • We believe diverse teams build better workplaces and better products. We want CreateFuture to be a place where people feel able to be themselves and do their best work. • If you need any adjustments or support during the application process, just. We will do what we can to help. • We look forward to your application!