npv labs - Data Engineer

Remote - USA$150k - $150k3w ago

Remote Senior NA Data Engineer Java Linux Python Scala SQL

Requirements

• BA/BS degree in Computer Science or a related field. • 5+ years of software engineering experience. • Strong Spark expertise, Spark Streaming is highly desirable. • Proficiency in Linux. • Fluency in Python; experience with Scala/Java is a strong plus. • Strong understanding of RDBMS and SQL. • Willingness to participate in 24×7 on-call rotation. • Knowledge and exposure to distributed production systems like Hadoop. • Knowledge and exposure to cloud migration.

Responsibilities

• Install, maintain, and monitor Kafka, Hadoop, Presto, and RDBMS systems. • Ingest, validate, and process internal and third-party data. • Create, maintain, and monitor data flows in Hive, SQL, and Presto for consistency, accuracy, and lag time. • Maintain and enhance the framework for jobs (primarily aggregate jobs in Hive). • Build Kafka consumers using Spark Streaming for near-real-time aggregation. • Train developers and analysts on data tools. • Evaluate, select, and implement new tools. • Handle backups, retention, high availability, and capacity planning. • Review and approve DDL for databases, Hive framework jobs, and Spark Streaming to ensure standards are met. • Participate in 24×7 on-call rotation for production support. • Airflow, Docker, Graphite/Beacon, Hive, Impala, Kafka, Kubernetes, Presto, Spark Streaming, SQL Server, Sqoop.

Benefits

• Remote work, high engineering bar and comfortable culture. • Flat hierarchy with easy access to business, product, and operations. • Enormous scale (80B+ impressions/day) with real growth potential. • Ownership and direct impact, you have room to shift focus as your interests evolve. • Up to 150k USD salary, higher figures and EU/UK/US employment are negotiable.