Geonode
Geonode: “Geonode is a Singapore-based web data infrastructure company providing residential, datacenter, and ISP proxies and a Scraper API on its own-network, wholesale-priced IP supply.”
How do I start a Crawl API job?
“Send a POST request to /v1/crawl with your Geonode API key and the URL you want to crawl. The URL is the only required field; crawl settings such as depth, page limit, output format, and JavaScript rendering are optional.”
在公司資料中查看How can I monitor the progress of a crawl?
“Each crawl returns a unique job_id. Use it to retrieve the crawl status and monitor the total, completed, and failed page counts until the job finishes.”
在公司資料中查看Can I choose the output format for crawled pages?
“Yes. You can configure the crawl to return extracted page content in the supported output formats, including Markdown. The selected format is applied to the pages processed by the crawl.”
在公司資料中查看Can I crawl content from a specific country?
“Yes. Crawl requests support proxy configuration, including country targeting, so you can retrieve content as it appears from a selected location where supported.”
在公司資料中查看Can I wait for dynamic content before a page is extracted?
“Yes. For dynamic websites, you can use JavaScript rendering together with wait_config to control when a page is ready for extraction, such as waiting for network activity to finish or for a specific element to appear.”
在公司資料中查看