Overview - Student Internship Position - Improving web crawling with Lookyloo and Lacus
Lookyloo is a web interface used to capture and analyse websites. It helps investigators understand what a page loads and identify potentially malicious behaviour. Lacus is the web-crawling service used by Lookyloo and other tools to create these captures.
Modern websites make reliable crawling difficult. They can depend on JavaScript, redirect chains, cookies, browser storage, authentication, geolocation, browser fingerprinting, bot-detection systems, and resources loaded from many third parties. Network failures, timeouts, very large pages, and deliberately malicious content add further challenges. A useful capture must handle these cases safely while remaining reproducible and providing enough information to explain what happened.
This is a generic internship topic covering the challenges encountered while crawling the web. The precise project will be selected with the student according to current needs, interests, and experience. It may focus on Lookyloo, Lacus, or both projects.
Possible tasks include:
- Identifying and documenting web-crawling edge cases and creating reproducible test cases.
- Improving capture reliability, error reporting, retries, timeouts, and resource limits.
- Handling redirects, downloads, pop-ups, cookies, browser storage, and dynamically loaded content.
- Studying bot detection, browser fingerprinting, consent pages, and location-dependent content.
- Comparing captures made at different times, from different networks, or with different browser configurations.
- Improving the isolation and safe processing of malicious or unexpectedly large websites.
- Collecting useful diagnostics and metrics while respecting privacy and data minimisation principles.
- Extending automated tests, documentation, and the crawling test suite.
- Improving the integration between Lookyloo and Lacus.
The internship will begin with a review of the existing projects and a small set of representative crawling failures. The student will then design, implement, and evaluate one or more improvements. The work should result in reusable test cases, documentation, and free software contributions to the relevant projects.
Qualification
- Must have a valid work permit in Luxembourg.
- Must be eligible for a student internship in the field of information security and/or computer science.
- Must have a high level of ethics due to the nature of the work.
- Must be fluent in English and comfortable working with Unix and git.
- Good knowledge of Python and web technologies (HTTP, HTML, JavaScript, and browser behaviour).
- An interest in browser automation, web security, privacy, networking, or distributed systems is a plus.
- Must be willing to investigate unfamiliar and occasionally malicious web content in a controlled environment.
How to apply
The application package must include the following in ASCII text format (language: English):
- A resume.
- A motivation letter explaining why you are interested in the internship and which crawling challenges interest you most.
The package is to be sent to info(@)circl.lu indicating reference internship-lookyloo-04.
Application deadline
Applications are accepted on an ongoing basis while the position is listed as open.
Classification of this document
TLP:CLEAR information may be distributed without restriction, subject to copyright controls.