Pro members applied to this job 36 hours before you saw itGet Pro ›

LTS - AI Platform and Harness Engineer

Remote - United States2d ago

Requirements

• Bachelor's degree in Computer Science, Software Engineering, Artificial Intelligence, Data Science, or a related technical field. • 5+ years of experience in software engineering, platform engineering, backend engineering, DevOps, cloud engineering, or infrastructure engineering. • 2+ years building or supporting Generative AI, Large Language Model (LLM), or machine learning applications. • Strong programming experience in Python. • Experience developing APIs, backend services, and distributed systems. • Experience with cloud platforms including AWS, Azure, or Google Cloud Platform. • Experience deploying applications using Docker and Kubernetes. • Experience working with Git, CI/CD pipelines, Infrastructure as Code (IaC), and infrastructure automation. • Strong understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Prompt engineering, Embeddings, Vector databases, AI agents and agentic workflows • Familiarity with AI evaluation techniques, automated testing, benchmarking, regression testing, and model validation. • Experience building scalable, production-grade software platforms. • Strong problem-solving, debugging, and performance optimization skills. • Experience with AI orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Semantic Kernel, or AutoGen. • Experience implementing LLMOps or MLOps platforms and deployment pipelines. • Experience with AI observability tools such as LangSmith, OpenTelemetry, Prometheus, Grafana, Evidently AI, or Arize AI. • Experience with vector databases including Pinecone, Qdrant, Weaviate, Azure AI Search, or pgvector. • Experience with OpenAI, Azure OpenAI, AWS Bedrock, Anthropic Claude, Google Vertex AI, or similar enterprise AI platforms. • Experience implementing Responsible AI, AI governance, model security, and AI safety best practices. • Experience supporting Federal Government or other regulated environments. • Experience evaluating AI systems for quality, reliability, accuracy, explainability, latency, and cost optimization. • Familiarity with healthcare, enterprise modernization, or mission-critical systems.

Responsibilities

• Design, build, and maintain enterprise AI platform capabilities supporting Large Language Models (LLMs), AI agents, RAG, and Generative AI applications. • Develop reusable AI harnesses to automate testing, prompt evaluation, model benchmarking, regression testing, and quality assurance. • Build AI evaluation frameworks to measure model accuracy, retrieval quality, hallucination detection, latency, throughput, cost, and overall application performance. • Implement observability and monitoring solutions for AI applications, including telemetry, tracing, logging, dashboards, and operational metrics. • Build and maintain LLMOps pipelines supporting model deployment, versioning, evaluation, experimentation, rollback, and continuous improvement. • Design automated workflows for prompt testing, retrieval evaluation, AI system validation, and performance benchmarking. • Develop internal tools for prompt management, model experimentation, AI performance optimization, and developer productivity. • Build scalable backend services and APIs supporting AI platforms and enterprise AI integrations. • Collaborate with AI architects and engineering teams to integrate LLMs, RAG pipelines, vector databases, and agentic AI solutions into enterprise applications. • Support deployment of AI services across AWS, Azure, or Google Cloud using containerized and cloud-native architectures. • Implement CI/CD pipelines and infrastructure automation supporting enterprise AI development and deployment. • Evaluate emerging AI frameworks, LLMOps technologies, evaluation methodologies, and automation tools to improve engineering productivity. • Troubleshoot production AI issues and continuously improve platform reliability, scalability, security, and user experience. • Document engineering standards, AI platform architecture, evaluation methodologies, and operational best practices.

Benefits

• The Opportunity to support high-visibility federal missions • A culture that values innovation, growth, and collaboration • Access to cutting-edge tools and technologies • Comprehensive benefits for you and your family • A career path that rewards ambition and performance • If you’re ready to push boundaries, sharpen your skills, and join a team that is passionate about building what’s next, we’d love to meet you. Apply today and let’s build a future together!