Super Payments - Senior Data Engineer
Requirements
• AWS & Event-Driven Architecture: Hands-on experience with streaming/event ingestion (Kinesis, EventBridge or equivalents), S3 partitioning, and queue/DLQ patterns. • AWS & Event-Driven Architecture: • Infrastructure as Code: Terraform experience required; Terragrunt a plus. Comfortable owning modules across environments, not just consuming them. • Infrastructure as Code: • Snowflake & Warehouse Engineering: Strong SQL for Snowflake, including Snowpipe, Tasks and Snowpark. • Snowflake & Warehouse Engineering: • Dbt: Proven experience in building well tested SQL and python models in dbt. • CI/CD & Data Security: Experience with GitHub Actions (or equivalent), and secrets/key management, PII and sensitive data handling in Snowflake. • CI/CD & Data Security: • Python Coding: Python skills for pipeline and infrastructure code (typed models, testing, CI-deployed services), not just scripting or analysis. • Python Coding: • Engineering Ownership • Technical Communication: A pragmatic, systems-level mindset with the ability to explain pipeline failures, infrastructure trade-offs, or data-flow issues clearly to both engineers and non-technical stakeholders. • Ownership & Initiative: A strong willingness to work independently, take initiative, and take full ownership of the pipelines and infrastructure you build, from design and implementation through to production reliability. • Ownership & Initiative • Production Mindset: Proactively monitoring, debugging, and improving pipelines and infrastructure in production, not just building them and moving on. • Production Mindset • The stated experience and background is a guide and does not preclude applications from candidates with more or less experience, provided the requisite skills can be demonstrated. • Our Benefits - here’s a few and more to come …. • Tax advantage Share Options • Flexible working model • Work from home set up • Learning & Development opportunities • Contributory Pension Scheme • Free Team lunch (Tues & Thurs) and social evenings • Comprehensive PMI & x4 Life Insurance • Your birthday off, plus one Revival day • If you are excited about sharing the adventure, joining a growing team with big ambitions and you are really great at what you do, then apply now!
Responsibilities
• Ingestion & Pipeline Engineering • Event Pipeline Ownership: Build and evolve the EventBridge → Firehose → S3 → Snowpipe ingestion path, including partitioning and throughput tuning as event volumes grow across streams like fraud, payments and merchant. • Event Pipeline Ownership: • Batch & Third-Party Integrations: Design and deploy Snowpark-based Python procedures for file-based and third-party data loads, scheduled and monitored as Snowflake Tasks. • Batch & Third-Party Integrations: • Schema Evolution: Work with product engineers to manage upstream schema changes and late-arriving data without breaking downstream consumers. • Schema Evolution: • Infrastructure & Reliability • Infrastructure as Code: Own and extend Terraform/Terragrunt modules for ingestion infrastructure across multiple environments. • Infrastructure as Code: • Pipeline Observability: Design Datadog monitors and alerting for Firehose, EventBridge, DLQs and Snowpipe; triage and resolve ingestion failures. • Pipeline Observability: • Security & Compliance: Manage PII handling (hashing, secure deletion), masking, data governance and access in Snowflake. Support audit-ready, reconciled data for Finance and Compliance. • Security & Compliance: • Data Pipelines & Data Quality • Dbt: Build privacy-first data pipelines using dbt and Snowflake Dynamic Tables to safely expose sensitive data for downstream analytics and reporting. • Data Quality: Build testing and monitoring for ingestion and transformation logic to catch issues before they reach the business or the Analytics Engineering team. • Data Quality: • Stakeholder Collaboration: Partner with the wider Engineering team and Analytics Engineering team as a downstream consumer, to ensure pipelines deliver accurate, audit-ready data. • Stakeholder Collaboration:
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT