deepgram - Research Staff, Voice AI Foundations - Remote

• Strong mathematical foundation in statistical learning theory, particularly in areas relevant to self-supervised and multimodal learning • Deep expertise in foundation model architectures, with an understanding of how to scale training across multiple modalities • Proven ability to bridge theory and practice—someone who can both derive novel mathematical formulations and implement them efficiently • Demonstrated ability to build data pipelines that can process and curate massive datasets while maintaining quality and diversity • Track record of designing controlled experiments that isolate the impact of architectural innovations and validate theoretical insights • Experience optimizing models for real-world deployment, including knowledge of hardware constraints and efficiency techniques • History of open-source contributions or research publications that have advanced the state of the art in speech/language AI • IDEAL RESEARCHER PROFILE • An ideal researcher, as evidenced by the recurring themes across these foundational papers, excels in five key areas: (1) Statistical & Mathematical Foundations, (2) Algorithmic Innovation & Implementation, (3) Data-Driven & Scalable Systems, (4) Hardware & Systems Understanding, and (5) Rigorous Experimental Design. Below is a synthesis of how each paper highlights these qualities, with references illustrating why they matter for building robust, impactful deep learning models. • 1. STATISTICAL & MATHEMATICAL FOUNDATIONS • Mastery of Core Concepts • Many papers, like Scaling Laws for Neural Language Models and Neural Discrete Representation Learning (VQ-VAE), reflect the importance of power-law analyses, derivation of novel losses, or adaptation of fundamental equations (e.g., in VQ-VAE's commitment loss or rectified flows in Scaling Rectified Flow Transformers). Such mathematical grounding clarifies why models converge or suffer collapse. • Combining Existing Theories in Novel Ways • Papers such as Moshi (combining text modeling, audio codecs, and hierarchical generative modeling) and Finite Scalar Quantization (FSQ's adaptation of classic scalar quantization to replace vector-quantized representations) show how reusing but reimagining known techniques can yield breakthroughs. Many references (e.g., the structured state-space duality in Transformers are SSMs) underscore how unifying previously separate research lines can reveal powerful algorithmic or theoretical insights. • Logical Reasoning and Assumption Testing • Across all papers—particularly in the problem statements of Whisper or Rectified Flow Transformers—the authors present assumptions (e.g., "scaling data leads to zero-shot robustness" or "straight-line noise injection improves sample efficiency") and systematically verify them with thorough empirical results. An ideal researcher similarly grounds new ideas in well-formed, testable hypotheses. • 2. ALGORITHMIC INNOVATION & IMPLEMENTATION • Creative Solutions to Known Bottlenecks • Each paper puts forth a unique algorithmic contribution—Rectified Flow Transformers redefines standard diffusion paths, FSQ proposes simpler scalar quantizations contrasted with VQ, phi-3 mini relies on curated data and blocksparse attention, and Mamba-2 merges SSM speed with attention concepts. • Turning Theory into Practice • Whether it's the direct preference optimization (DPO) for alignment in phi-3 or the residual vector quantization in SoundStream, these works show that bridging design insights with implementable prototypes is essential. • Clear Impact Through Prototypes & Open-Source • Many references (Whisper, neural discrete representation learning, Mamba-2) highlight releasing code or pretrained models, enabling the broader community to replicate and build upon new methods. This premise of collaboration fosters faster progress. • 3. DATA-DRIVEN & SCALABLE SYSTEMS • Emphasis on Large-Scale Data and Efficient Pipelines • Papers such as Robust Speech Recognition via Large-Scale Weak Supervision (Whisper) and BASE TTS demonstrate that collecting and processing hundreds of thousands of hours of real-world audio can unlock new capabilities in zero-shot or low-resource domains. Meanwhile, phi-3 Technical Report shows that filtering and curating data at scale (e.g., "data optimal regime") can yield high performance even in smaller models. • Strategic Use of Data for Staged Training • A recurring strategy is to vary sources of data or the order of tasks. Whisper trains on multilingual tasks, BASE TTS uses subsets/stages for pretraining on speech tokens, and phi-3 deploys multiple training phases (web data, then synthetic data). This systematic approach to data underscores how an ideal researcher designs training curricula and data filtering protocols for maximum performance. • 4. HARDWARE & SYSTEMS UNDERSTANDING • Efficient Implementations at Scale • Many works illustrate how researchers tune architectures for modern accelerators: the In-Datacenter TPU paper exemplifies domain-specific hardware design for dense matrix multiplications, while phi-3 leverages blocksparse attention and custom Triton kernels to run advanced LLMs on resource-limited devices. • Real-Time & On-Device Constraints • SoundStream shows how to compress audio in real time on a smartphone CPU, demonstrating that knowledge of hardware constraints (latency, limited memory) drives design choices. Similarly, Moshi's low-latency streaming TTS and phi-3-mini's phone-based inference highlight that an ideal researcher must adapt algorithms to resource limits while maintaining robustness. • Architectural & Optimization Details • Papers like Mamba-2 in Transformers are SSMs and the In-Datacenter TPU work show how exploiting specialized matrix decomposition, custom memory hierarchies, or quantization approaches can lead to breakthroughs in speed or energy efficiency. • 5. RIGOROUS EXPERIMENTAL DESIGN • Controlled Comparisons & Ablations • Nearly all papers—Whisper, FSQ, Mamba-2, BASE TTS—use systematic ablations to isolate the impact of individual components (e.g., ablation on vector-quantization vs. scalar quantization in FSQ, or size of codebooks in VQ-VAEs). This approach reveals which design decisions truly matter. • Multifold Evaluation Metrics • From MUSHRA listening tests (SoundStream, BASE TTS) to FID in image synthesis (Scaling Rectified Flow Transformers, FSQ) to perplexity or zero-shot generalization in language (phi-3, Scaling Laws for Neural Language Models), the works demonstrate the value of comprehensive, carefully chosen metrics. • Stress Tests & Edge Cases • Whisper's out-of-distribution speech benchmarks, SoundStream's evaluation on speech + music, or Mamba-2's performance on multi-query associative recall demonstrate the importance of specialized challenge sets. Researchers who craft or adopt rigorous benchmarks and "red-team" their models (as in phi-3 safety alignment) are better prepared to address real-world complexities.