Software Engineer, Web Crawling
Upload My Resume
Drop here or click to browse · PDF, DOCX, DOC, RTF, TXT
Requirements
• You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly • You have experience with some high performance language (C++, Rust, etc.) • You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol) • You’re comfortable optimizing a system to an exceptional degree • You care about the problem of finding high quality knowledge and recognize how important this is for the world • Example Projects • Build a distributed crawler that can handle 100M+ pages per day • Optimize crawl politeness and rate limiting across thousands of domains • Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures • Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency • This is an in-person opportunity in Singapore. We’re happy to sponsor international candidates. • In addition to premium healthcare benefits (medical, dental, vision), we also offer fertility benefits and a monthly wellness stipend to all of our employees.
Responsibilities
• Developing web crawlers to automate data collection from various websites for analysis purposes. • Implementing algorithms that can efficiently navigate through website structures and extract relevant information without causing harmful effects on the targeted sites' infrastructure (e.g., avoiding overloading servers). • Ensuring compliance with legal regulations regarding web scraping, such as respecting robots.txt files and copyright laws. • Analyzing collected data to identify trends or insights that can inform business decisions at Exa. • Maintaining the performance of crawling systems by monitoring their efficiency and making necessary adjustments for optimal operation. • Collaborating with other teams within Asia Engineering, such as product development or customer support, when required to align web crawler functionalities with broader company goals. • Staying updated on advancements in the field of web scraping technology and incorporating new techniques into their work where applicable.
Benefits
• Equity options mentioned as part of the benefits package • Paid time off (PTO) included with full pay for all employees and their dependents up to a specified number per employee annually. The company also offers additional PTO days based on years of service, which can be used in conjunction with regular vacation or sick leave as needed by the individual without affecting annual limits • Comprehensive insurance coverage including medical, dental, and vision plans for employees and their dependents up to a specified limit. The company also offers additional benefits such as life insurance at no extra cost beyond what is provided through government programs or employer contributions • Perks include free meals onsite 3 times per week during business hours (Monday - Friday), access to an in-house gym, and a monthly team outing for lunch/dinner. The company also offers transportation subsidies up to $500 annually • Remote work options available with the possibility of flexible scheduling or hybrid working arrangements based on individual roles and responsibilities within the organization