stack-infrastructure-emea - Senior Databricks Data Engineer
Requirements
• · Strong hands-on experience building and supporting production data pipelines directly in Databricks; experience with other cloud data platforms alone is not sufficient. • · Practical experience with PySpark, Python, SQL and Delta Lake, including data transformation and performance optimisation. • · Experience orchestrating and troubleshooting Databricks data pipelines using Workflows, Delta Live Tables or Lakeflow. • · Working knowledge of Unity Catalog, data governance, access controls and data quality practices (Essential). • · Experience integrating data from APIs, business applications, databases and files. • · Experience supporting Databricks workspaces, compute resources or SQL Warehouses in a production environment. • · Experience with Azure data services, Power BI, Git and CI/CD practices. • · Understanding of environment management, monitoring, troubleshooting, security and cost control. • · Ability to work with non-technical stakeholders, explain data issues clearly and document solutions effectively. • · Strong ownership, prioritisation and problem-solving skills. • Certifications (Preferred) • · Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available).
Responsibilities
• · Design, develop, deploy and maintain production-grade ETL/ELT pipelines using Databricks, PySpark, Python and SQL, integrating APIs, databases, SaaS applications, files and spreadsheets. • · Develop and optimize scalable Delta Lake data models and lakehouse architecture for reporting, analytics and future AI use cases. • · Build, schedule, monitor and troubleshoot data processing jobs using Databricks Workflows, Lakeflow or Delta Live Tables, as appropriate. • · Apply Unity Catalog governance, including access permissions, data ownership, lineage and common data definitions. • · Support Databricks workspaces, compute resources, SQL Warehouses, service principals and access management in partnership with relevant technology teams. • · Manage development, test and production deployments, monitoring, incident resolution and recovery, using Git and CI/CD practices. • · Optimise pipeline performance, Spark processing, compute usage and platform costs while maintaining appropriate security and operational controls. • · Implement data quality checks for completeness, accuracy, timeliness, consistency, duplicates and reconciliation with source systems. • · Build monitoring and alerting to identify pipeline failures, data quality issues and operational risks. • · Integrate reliable, governed Databricks datasets with Power BI and other approved analytical tools. • · Work with Finance, Delivery, Operations, HR and PMO teams to translate requirements into sustainable data solutions. • · Document pipelines, dependencies, controls and operating procedures, and coordinate specialist external partners when required.
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT