Causal Labs - Member of Technical Staff — Data Infrastructure

San Francisco, California, United States3w ago

In Office Staff NA Cloud Computing Technical Support Specialist Ray

Requirements

• We value a relentless approach to problem-solving, rapid execution, and the ability to quickly learn in unfamiliar domains. • Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam) • Knowledge of state-of-the-art methods and tools for data ingestion, storage, and loading — including file formats and storage systems (e.g. Parquet, Zarr, Delta Lake) and how they impact performance and scalability • Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines • Understanding of how data loading throughput affects large-scale training, and experience optimizing it • Owns deliverables end-to-end, from collecting and translating requirements to autonomously driving execution

Responsibilities

• Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries • Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on • Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary • Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle • Implement the platform-level quality and monitoring tooling that data and research teams build their checks on • Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match • Work across the full data lifecycle when the mission needs it — including building and operating ingestion pipelines for critical data sources directly