Software Engineer, Web Crawling

ExaSingapore, Singapore, Singapore$150k – $300k+ Equity1mo ago

Upload My Resume

Drop here or click to browse · PDF, DOCX, TXT

Apply in One Click

Requirements

You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
You have experience with some high performance language (C++, Rust, etc.)
You’re comfortable optimizing a system to an exceptional degree
You care about the problem of finding high quality knowledge and recognize how important this is for the world
Example Projects
Example Projects
Build a distributed crawler that can handle 100M+ pages per day
Optimize crawl politeness and rate limiting across thousands of domains
Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
This is an in-person opportunity in Singapore.

Developing web crawlers to automate data collection from various websites for analysis purposes.
Implementing algorithms that can efficiently navigate through website structures and extract relevant information without causing harmful effects on the targeted sites' infrastructure (e.g., avoiding overloading servers).
Ensuring compliance with legal regulations regarding web scraping, such as respecting robots.txt files and copyright laws.
Analyzing collected data to identify trends or insights that can inform business decisions at Exa.
Maintaining the performance of crawling systems by monitoring their efficiency and making necessary adjustments for optimal operation.
Collaborating with other teams within Asia Engineering, such as product development or customer support, when required to align web crawler functionalities with broader company goals.
Staying updated on advancements in the field of web scraping technology and incorporating new techniques into their work where applicable.

Equity options mentioned as part of the benefits package
Paid time off (PTO) included with full pay for all employees and their dependents up to a specified number per employee annually. The company also offers additional PTO days based on years of service, which can be used in conjunction with regular vacation or sick leave as needed by the individual without affecting annual limits
Comprehensive insurance coverage including medical, dental, and vision plans for employees and their dependents up to a specified limit. The company also offers additional benefits such as life insurance at no extra cost beyond what is provided through government programs or employer contributions
Perks include free meals onsite 3 times per week during business hours (Monday - Friday), access to an in-house gym, and a monthly team outing for lunch/dinner. The company also offers transportation subsidies up to $500 annually
Remote work options available with the possibility of flexible scheduling or hybrid working arrangements based on individual roles and responsibilities within the organization

8h ago