Twenty - Staff Data Engineer

New York, NY, United States2mo ago

In Office Staff NA Cloud Computing Oil & Gas Data Engineer Data Quality Mentoring AWS Apache Spark Redshift

Requirements

• You think in systems: data modeling, storage formats, compute engines, and access patterns all have to fit together. • You’re opinionated about schema and index design, and you can explain tradeoffs clearly. • You default to measurable reliability: data quality, lineage, repeatability, and operational excellence. • You’re comfortable working with ambiguous datasets and evolving requirements without lowering standards. • You collaborate tightly across roles, especially with engineers and analysts who need fast, correct answers. • You take leadership seriously—mentoring others, raising the bar, and driving initiatives to completion. • You’re motivated by national security outcomes and want your work to matter in the real world. • You have 8+ years of experience in data engineering and/or data architecture. • You have mastery-level expertise building ETL pipelines and operating them in production. • You have deep experience with data lake architecture and systems used to query data lakes. • You have strong schema and index design skills, including partitioning, indexing, and clustering strategies. • You have experience with column-oriented databases in production environments. • You have built data systems from scratch (not only maintained existing platforms). • You have proven leadership experience mentoring engineers and driving technical initiatives. • You are a U.S. citizen and can meet the role’s security requirements. • You have experience with key-value datastores. • You have worked with streaming and message queue systems. • You have experience with graph database technologies. • You have worked with internet/networking datasets (e.g., scan data, DNS, netflow, certificates). • You have experience supporting analysts or operational users with high-stakes data needs. • Tech Environment (You Might Work With) • Data lakes: Apache Iceberg, Delta Lake, Apache Hive • Query engines: Trino, Presto, AWS Athena, Apache Spark • Column stores: ClickHouse, Amazon Redshift, Google BigQuery • ETL / orchestration: Airflow, AWS Glue, NiFi, ClickPipe • Streaming / queues: Kafka, RabbitMQ, NATS, AWS Kinesis • Graph: Neo4j, AWS Neptune, Memgraph, Apache AGE

Responsibilities

• Lead the development and operation of a data lake for cyber operations and intelligence data. • Design schemas, partitions, and indexes that make complex datasets performant and cost-effective to query. • Partner with engineers and intelligence analysts to define query patterns and data products for mission use cases. • Build and evolve ETL pipelines that are observable, recoverable, and resilient to upstream change. • Drive technical initiatives end-to-end, from architecture decisions through production rollout and iteration. • Establish best practices for data quality, documentation, and operational ownership across the platform. • Mentor engineers on data modeling, performance tuning, and production-grade pipeline design. • Identify bottlenecks in storage/compute/query layers and ship improvements with clear performance wins.

Benefits

• What's on the table: • Health. Medical, dental, and vision plan options. Life / AD&D, disability coverage options. • Family. Paid parental leave for eligible full-time employees. 12 weeks for birthing parents, 4 for non-birthing parents, 6 weeks for adoptive, foster, or intended parents through surrogacy. • Vacation. Paid holidays and flexible PTO. Take what you need. • Retirement. 401(k) with pre-tax and Roth options. HSA/FSA options, dependent care FSA. • At the office. Commuter benefits. On-site garage parking. Bike storage. Building fitness center. Desk setup stipend. • Benefits vary by location, role, and eligibility. Full plan details provided during the interview and offer process. • If this role sounds like you, apply and share with us your interest. • Some positions may require eligibility to obtain a U.S. Government security clearance. Any clearance requirement will be listed in the role description.