Senior Data Engineer
Upload My Resume
Drop here or click to browse · PDF, DOCX, DOC, RTF, TXT
Requirements
• Experience in Data Engineering: 3+ years of experience as a Data Engineer, with a focus on data lake architecture and ETL pipeline development. • AWS Proficiency: Strong experience with Databricks and AWS services including but not limited to S3, Glue, Lambda, Redshift, and IAM. • ETL Expertise: Hands-on experience with Airbyte or similar ETL tools for data ingestion and transformation. • Proficiency in Python/Go: Experience writing services and connectors in Python/Go, particularly for data pipeline automation. • SQL and Data Modeling: Solid understanding of data modeling, SQL, and database concepts. • Data Governance and Security: Experience implementing security and governance best practices in cloud environments.
Responsibilities
• Data Infrastructure Setup: Architect and maintain robust, scalable, and secure data infrastructure on AWS leveraging Databricks. • Data Pipeline Development: Design, develop, and maintain data pipelines, primarily using tools like Airbyte and custom-built services in Go, to automate data ingestion and ETL processes. • Data Lake Management: Oversee the creation and maintenance of the data lake, ensuring efficient storage, high data quality, and effective partitioning, organization, performance, monitoring and alerting. • Integration and Customization: Integrate tools like Airbyte with various data sources and customize data flows to align with specific business needs. Where necessary, build custom connectors in Go to support unique data requirements. • Performance and Scalability: Optimize data pipelines and data lake storage for performance and scalability, ensuring low latency and high availability. • Data Governance and Security: Implement best practices for data governance, security, and compliance in AWS and Databricks, including access control, encryption, and monitoring. • Collaboration and Documentation: Work closely with platform engineers, data analysts and other stakeholders to understand data requirements and document infrastructure, processes, and best practices.
Benefits
• Equity options mentioned as part of the benefits package • Paid time off (PTO) is included with a generous amount per year and additional days for special occasions like holidays. • Comprehensive insurance coverage, including health, dental, vision, life, disability income protection, long term care, critical illness, accidental death & dismemberment policies are provided to employees at no cost or a nominal fee depending on the plan chosen by each employee. • Perks such as free meals and snacks in our office cafeteria, gym membership reimbursement up to $50 per month for fitness centers within commuting distance of your home/work location are offered. Additionally, we provide a company car (upon availability) or subsidized public transportation passes as needed based on the role and work schedule flexibility is available with flexible working hours upon managerial approval • Remote work options provided for employees who qualify under our remote policy guidelines which include maintaining high productivity standards, regular check-ins via video calls or other communication tools.