Artefact - Senior Data Engineer

17th Floor, 5 Aldermanbury Square, London, EC2V 7HR - Hybrid3w ago

In Office Senior EMEA Cloud Computing Manufacturing Senior Data Engineer Team Management Reporting SQL Python Terraform Snowflake Databricks PySpark Airflow Data Governance AWS GCP Azure

Requirements

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field. • 3+ years of industry experience in data engineering with a strong technical proficiency in SQL, Python, and big data technologies. • Expertise in building scalable data workflows using cloud-native orchestration tools and distributed data processing frameworks. • Demonstrated experience with Infrastructure as Code tooling such as Terraform • Solid understanding of CI/CD principles and DevOps/DevSecOps practices. • Proven leadership skills and experience managing data engineering teams. • Excellent problem-solving skills and the ability to work with ambiguity. • Proficient in leveraging AI-assisted workflows to optimise task efficiency. • Strong communication and interpersonal skills. • Excellent understanding of data architecture involving data mesh, data lake, data warehouse and data lakehouse. • Certifications in Azure, AWS, or GCP. • Certifications in Databricks, Snowflake or similar technologies. • Experience in the leading large scale data engineering projects. • Working Conditions • Working Conditions • Hybrid work arrangement: two-three days per week working from the office

Responsibilities

• Design, build, and maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. • Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. • Implement continuous integration and continuous deployment (CI/CD) practices for data pipelines to improve efficiency and quality of data processing. • Work closely with data architects, analysts, and other stakeholders to understand business requirements and translate them into technical implementations. • Oversee and manage a team of data engineers, providing guidance and mentorship to ensure high-quality project deliverables. • Develop and enforce best practices in data governance, security, and compliance within the organisation. • Optimise data retrieval and develop dashboards and reports for business teams. • Continuously evaluate new technologies and tools to enhance the capabilities of the data engineering function.