Senior Web Scraper (BeautifulSoup or Scrapy)

About the role

NIQ for Digital Commerce, a start-up in the Retail Analytics sector, collects and processes more than 60 billion pieces of information daily to give customers and consumers real-time market visibility. It is looking for a Python developer within its scraping team in Lisbon to help optimise data collection.

Key responsibilities

• Design, implement and document robust Scrapy spiders, keeping them resilient to website changes while maintaining existing ones.
• Conduct code reviews and data validation to uphold high standards of code and data quality.
• Use knowledge of the HTTP protocol, modern web browser mechanisms and internal tools to build sophisticated crawlers that bypass advanced anti-bot measures.
• Architect and contribute to new developments across several codebases, provide technical documentation and train relevant teams when needed.

Required profile

• Master's degree in Computer Science, IT or a related field.
• At least 3 years of professional experience in software engineering.
• Enthusiastic, motivated and autonomous, enjoying challenging software engineering work and continuous learning.

Required skills

• Knowledge of BeautifulSoup or Scrapy; Scrapy is a plus.
• Familiarity with HTML and JavaScript, with a solid grasp of single-page applications.
• Experience with RESTful and/or GraphQL APIs.
• Hands-on experience with Python web frameworks such as Django or FastAPI.
• Advantageous: time-series databases such as InfluxDB, and cloud providers such as AWS, GCP or Azure.
• Strong skills in Docker, Git, pandas, regular expressions, Linux and bash scripting.

What we offer

• Flexible working environment.
• Volunteer time off.
• LinkedIn Learning.
• Employee Assistance Program (EAP).