Welltech - Senior Data Engineer (Redshift)

• As a Senior Data Engineer, you will play a crucial role in building and maintaining the foundation of our data ecosystem. You’ll work alongside data engineers, analysts, and product teams to create robust, scalable, and high-performance data pipelines and models. Your work will directly impact how we deliver insights, power product features, and enable data-driven decision-making across the company. • This role is perfect for someone who combines deep technical skills with a proactive mindset and thrives on solving complex data challenges in a collaborative environment. • Challenges You’ll Meet: • Pipeline Development and Optimization: Build and maintain reliable, scalable ETL/ELT pipelines using modern tools and best practices, ensuring efficient data flow for analytics and insights. • Pipeline Development and Optimization: • Data Modeling and Transformation: Design and implement effective data models that support business needs, enabling high-quality reporting and downstream analytics. • Data Modeling and Transformation: • Collaboration Across Teams: Work closely with data analysts, product managers, and other engineers to understand data requirements and deliver solutions that meet the needs of the business. • Collaboration Across Teams: • Ensuring Data Quality: Develop and apply data quality checks, validation frameworks, and monitoring to ensure the consistency, accuracy, and reliability of data. • Ensuring Data Quality: • Performance and Efficiency: Identify and address performance issues in pipelines, queries, and data storage. Suggest and implement optimizations that enhance speed and reliability. • Performance and Efficiency: • Security and Compliance: Follow data security best practices and ensure pipelines are built to meet data privacy and compliance standards. • Security and Compliance: • Innovation and Continuous Improvement: Test new tools and approaches by building Proof of Concepts (PoCs) and conducting performance benchmarks to find the best solutions. • Innovation and Continuous Improvement: • Automation and CI/CD Practices: Contribute to the development of robust CI/CD pipelines (GitLab CI or similar) for data workflows, supporting automated testing and deployment. • Automation and CI/CD Practices: • 4+ years of experience in data engineering or backend development, with a strong focus on building production-grade data pipelines. • 2-3+ years of experience working with AWS services (Administration of Redshift is a must), • AWS services • (Administration of Redshift is a must) • Solid experience working with AWS services (Spectrum, S3, RDS, Glue, Lambda, Kinesis, SQS). • Proficient in Python and SQL for data transformation and automation. • Python • Experience with dbt for data modeling and transformation. • Good understanding of streaming architectures and micro-batching for real-time data needs. • streaming architectures • Experience with CI/CD pipelines for data workflows (preferably GitLab CI). • CI/CD pipelines • Familiarity with event schema validation tools/ solutions (Snowplow, Schema Registry). • event schema validation tools/ solutions • Excellent communication and collaboration skills.Strong problem-solving skills—able to dig into data issues, propose solutions, and deliver clean, reliable outcomes. • A growth mindset—enthusiastic about learning new tools, sharing knowledge, and improving team practices. • Cloud: AWS (Redshift, Spectrum, S3, RDS, Lambda, Kinesis, SQS, Glue, MWAA) • Cloud: • Redshift • Languages: Python, SQL • Languages: • Orchestration: Airflow (MWAA) • Orchestration: • Modeling: • CI/CD: GitLab CI (including GitLab administration) • CI/CD: • Monitoring: Datadog, Grafana, Graylog • Monitoring: • Event validation process: Iglu schema registry • Event validation process: • APIs & Integrations: REST, OAuth, webhook ingestion • APIs & Integrations: • Infra-as-code (optional): Terraform • Infra-as-code (optional): • Experience with additional AWS services: EMR, EKS, Athena, EC2. • additional AWS services • Hands-on knowledge of alternative data warehouses like Snowflake or others. • alternative data warehouses • Experience with PySpark for big data processing. • PySpark • Familiarity with event data collection tools (Snowplow, Rudderstack, etc.). • event data collection tools • Interest in or exposure to customer data platforms (CDPs) and real-time data workflows. • customer data platforms (CDPs) • Candidate journey: ⭕️ Recruiter call ➔ ⭕️ Technical call with the hiring manager ➔ ⭕️ Meet the future stakeholders • Candidate journey: