anyone-ai - Machine Learning Engineer – ML Evaluation & Experiment Design

• 3+ years of hands-on applied machine learning experience • ML experiment design • Model selection • Hyperparameter tuning • Model evaluation • Data preprocessing and validation • Strong understanding of train, validation, and test splits • Ability to identify: • Distribution shift • Spurious correlations • Feature leakage • Data contamination • Experience evaluating whether performance improvements are statistically meaningful rather than random fluctuations • Strong understanding of ML evaluation metrics and when different metrics are appropriate • Experience debugging ML workloads across CPU and GPU environments • Ability to analyze technical problems and provide clear written feedback • Experience creating or participating in Kaggle, DrivenData, or similar ML competitions • Experience designing benchmark datasets or ML challenges • Background in data-centric AI or dataset quality • Experience with synthetic data generation and validation • Familiarity with statistical testing, confidence intervals, and effect sizes • Experience with ML evaluation pipelines, RLHF, or AI model evaluation • Experience developing ML curricula or technical assessments • Understanding of common ML failure modes such as: • Shortcut learning • Goodhart’s Law • Simpson’s paradox • Metric gaming • WHAT YOU’LL BE RESPONSIBLE FOR • Reviewing ML challenges and determining whether they are well designed and technically solvable • Evaluating whether datasets contain meaningful and learnable signals • Identifying unintended shortcuts or artifacts in synthetic datasets • Determining whether tasks require genuine diagnosis of the underlying ML problem • Reviewing evaluation metrics and improvement thresholds • Detecting metric gaming, data leakage, and evaluation flaws • Verifying reproducibility across the complete data → model → evaluation pipeline • Assessing whether challenge difficulty is appropriately calibrated • Providing clear recommendations for improving, recalibrating, or excluding problematic tasks • Work Type: Remote • Engagement: Part-time, project-based consulting • Focus: Applied machine learning, experiment design, data quality, and model evaluation • This role is a strong fit for ML engineers who enjoy debugging experiments, understanding why models succeed or fail, identifying problems in datasets and evaluation pipelines, and designing rigorous machine learning experiments.