Our crawler

How pitchlistbot finds public article bylines while respecting publishers.

Identity and purpose

Our user agent is pitchlistbot/0.1 (+https://pitchlist.io/bot). It reads public feeds, sitemaps and article metadata to identify recent articles and their professional bylines for the Pitchlist directory.

What we collect

We keep article URLs, titles, publication dates, sections and author names or canonical author URLs. We do not retain article bodies, bypass paywalls, solve CAPTCHAs or collect private contact details.

Respect for publishers

• We follow robots.txt and sitemap rules.
• We make at most one request every five seconds per domain, or follow a longer crawl delay.
• We slow down on 429 and 503 responses and stop when access is blocked.
• We do not target sites for minors or sensitive sites.

Block or contact us

You can block the user agent token pitchlistbot in robots.txt. To discuss crawling, correction or deletion, contact us.

Contact: support@pitchlist.io. Pitchlist is operated by SOFTLOGIC, a SASU registered in France (SIREN 101 233 401, RCS Lille Metropole), headquartered at 23 rue des Tisserands, 59700 Marcq-en-Baroeul, France.