Scrapfly

Scrapfly Crawler API

Web crawling API (currently in early access) that enables crawling entire websites with advanced configuration for depth control and content filtering. Outputs in WARC format for comprehensive web archive support.

GitHub

Documentation

📖
Documentation
https://scrapfly.io/docs

API entry from apis.yml

apis.yml Raw ↑ 
aid: scrapfly:crawler-api
name: Scrapfly Crawler API
description: Web crawling API (currently in early access) that enables crawling entire websites with advanced
  configuration for depth control and content filtering. Outputs in WARC format for comprehensive web
  archive support.
humanURL: https://scrapfly.io/docs
tags:
- Web Crawling
- Data Collection
- WARC
properties:
- type: Documentation
  url: https://scrapfly.io/docs