Imagine Worldwide - Senior Analytics Engineer

• The Senior Analytics Engineer’s responsibilities fall into four pillars. The first two cover the platform itself: the infrastructure it runs on, and the pipelines that bring data in from many sources and process it at scale. The third covers orchestration and the core data models that the rest of the Analytics Team builds on. The fourth covers the practices that keep the platform reliable, secure and affordable to run. • Data Platform and Infrastructure • Imagine’s analytics platform runs on AWS and ClickHouse Cloud, with infrastructure managed as code. In a small team, this role carries the infrastructure work that a larger organisation would split across dedicated platform or DevOps staff, limited to the scope of the analytics platform. • Build and maintain the platform’s AWS infrastructure, including storage, compute, networking, the data catalogue, secrets and access management (S3, EC2, Glue Data Catalog, Lake Formation, IAM, Secrets Manager). • Manage infrastructure as code in Terraform, keeping environments reproducible, version-controlled and reviewable. • Administer the ClickHouse Cloud warehouse, including service configuration, user and role management, and monitoring of performance and capacity. • Host and maintain the team’s self-hosted tools, including Apache Superset for BI and Apache Airflow for orchestration. This covers deployment, upgrades, authentication and availability. The Senior BI & Data Analyst manages the dashboards and content within Superset. • Contribute to architectural decisions on how the platform evolves, setting out options and trade-offs for review with the Director of Analytics. • Large-Scale Data Processing and Pipelines • Data reaches the platform from application databases, operational systems and field data tools, and lands in an Apache Iceberg data lake on S3. Spark-based processing on AWS Glue does the heavy work before the data moves into ClickHouse for analysis. The aim is to do expensive processing once, as far upstream as practical, so that downstream models and queries stay fast and affordable. • Build and maintain ingestion pipelines from the platform’s sources, including application databases, APIs and operational systems, using tools such as AWS Glue, Airbyte and AWS Lambda. • Implement incremental loading patterns that stay efficient and recoverable as volumes grow, landing raw data in the lake in a form that preserves its full history. • Develop and optimise Spark jobs that process billions of device events into learning sessions and higher-level aggregates, paying close attention to partitioning, skew and compute cost. • Bring in and model the programme context that gives event data its meaning, such as schools, terms, device allocations and enrolment, and handle how that context changes over time. • Design and maintain Iceberg tables for performance and cost, including partitioning strategy, file sizing, compaction and snapshot management. • Validate data at ingestion and surface quality issues early, working with the Analytics Team to trace them back to their source. • Contribute to evaluating new processing approaches as volumes grow, including moving more work upstream and assessing where streaming would add value. • Orchestration and Transformation • Apache Airflow orchestrates the platform’s pipelines, and dbt transforms data in ClickHouse into the models that reporting and analysis rely on. This role builds and maintains the foundational layers of those models. • Build and maintain Airflow workflows that coordinate ingestion, processing and transformation, with clear dependencies, retries and alerting. • Develop and maintain dbt staging and core models in ClickHouse, following consistent conventions for naming, structure and materialisation. • Implement incremental models for high-volume tables, choosing materialisation strategies and table designs that balance freshness, correctness and cost. • Write and maintain dbt tests and model contracts for core datasets, so that changes that would break downstream reporting are caught before they reach production. • Document core models such as learning sessions as agreed definitions, including their lineage, and make clear where a model applies an approximation rather than a direct transformation. • Work with the Senior BI & Data Analyst on the boundary between core and reporting models, and with the Data Scientist on datasets for research and evaluation. • Reliability, Cost and Engineering Practice • A small team needs a platform that runs predictably and that more than one person understands. This pillar covers the engineering practices that make that possible. • Build and maintain CI/CD pipelines in GitHub Actions for infrastructure, pipeline and dbt code, including automated checks that stop unsafe changes from reaching production. • Implement monitoring and alerting across pipelines, the warehouse and hosted tools, so that failures and delays are visible quickly and reach the right people. • Respond to platform incidents and data issues, carry out root cause analysis, and contribute to the team’s data quality incident process. • Monitor platform costs across AWS and ClickHouse, track the cost and performance of individual workloads over time, and use that evidence to identify savings and prioritise optimisation work. • Maintain and periodically test backup and recovery procedures for the platform’s core components. • Take part in code review and write clear runbooks and technical documentation, following the team’s documentation standards, so that knowledge of the platform is shared across the team.