smart-working-solutions - Smart Working Solutions - Anti-Bot Engineer (Remote, Full-Time) [HR177]
Requirements
• 7+ years of professional software engineering experience, with a strong focus on backend systems, data engineering, or distributed systems. • Proven experience building and operating large-scale production web scraping systems. • Deep hands-on experience scraping Google or similarly complex, heavily protected / anti-bot environments. • Strong expertise in Python (or comparable production languages such as Go, Rust, or JavaScript). • Strong understanding of HTTP internals: headers, cookies, TLS, redirects, sessions, and browser networking behaviour. • Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalent. • Strong knowledge of HTML parsing, DOM traversal, and high-performance data extraction techniques. • Proven experience handling anti-bot systems, including rate limiting, CAPTCHAs, IP rotation, and fingerprinting. • Experience designing asynchronous and concurrent systems for high-throughput workloads. • Strong debugging skills across distributed, failure-prone production systems. • Experience running cloud-based systems at scale with strong operational ownership. • Comfortable working in highly adversarial and fast-changing technical environments. • Experience with Docker and Kubernetes in production environments. • Exposure to distributed task queues or large-scale job orchestration systems. • Experience with monitoring, anomaly detection, or data quality validation systems. • Background in search intelligence, advertising tech, or competitive intelligence platforms. • Experience building systems that operate against high-defence web environments at scale. • Familiarity with observability tooling and production-grade system monitoring. • Exposure to AI-assisted development workflows or agentic coding tools.
Responsibilities
• Design, build, and maintain large-scale Python-based scraping systems targeting highly protected websites (including Google-like environments). • Architect resilient extraction systems capable of handling dynamic, JavaScript-heavy pages using browser automation and hybrid approaches. • Continuously adapt systems to frequent changes in page structures, request flows, and anti-bot mechanisms. • Build robust, production-grade data extraction pipelines with strong emphasis on accuracy, observability, and fault tolerance. • Implement advanced strategies such as proxy rotation, fingerprinting, session management, and request routing to ensure stability at scale. • Monitor system health, proactively detect anomalies, and debug complex production failures across distributed systems. • Optimise scraping infrastructure for performance, cost efficiency, and reliability at scale. • Collaborate with data engineering and product teams to ensure scraped data is structured, validated, and trusted. • Operate and improve systems running continuously in cloud environments. • Document system architecture, scraping logic, and operational procedures for long-term maintainability. • Contribute to improving resilience, automation, and adaptability in adversarial environments.
Benefits
• Fixed Shifts: 12:00 PM - 9:30 PM IST (Summer) | 1:00 PM - 10:30 PM IST (Winter) • No Weekend Work: Real work-life balance, not just words • Day 1 Benefits: Laptop and full medical insurance provided • Support That Matters: Mentorship, community, and forums where ideas are shared • True Belonging: A long-term career where your contributions are valued
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT