Start a conversation

Crawling Control and Settings

  • General Settings in Netpeak Spider

    Language. Crawling speed. Basic сrawling settings. On the ‘General’ settings tab, you can change the interface language, crawling speed, and basic crawling settings. 1. Language It is possible to choo...

  • Advanced settings

    Considering crawling and indexing instructions. Crawling links from the link tag. Automatic stop of crawling. Additional settings. You can find the advanced settings under ‘Settings → Advanced‘. They ...

  • The ‘Restrictions‘ settings

    Crawling Restrictions Issue Restrictions Netpeak Spider allows setting up crawling limits and changing issue restrictions on the ‘Restrictions‘ tab of its settings.   1. Crawling Restrictions This sec...

  • URL: adding, crawling, saving

    1. How to Add URLs for Crawling 2. Processing of Entered URLs 3. Crawling Process Features  4. Changing an Initial URL in a Single Project 5. How to Recrawl Pages 5.1. Recrawling Particular URLs 5.2. ...

  • Crawling rules configuration

    Crawling Rules Configuration Common Functions for All Rules. How to Set up Crawling Rules. Combination of Conditions and Settings. Crawling rules specify which type of URLs to include or exclude from ...

  • Virtual robots.txt

    The built-in feature ‘Virtual robots.txt‘ allows testing new or renewed robots.txt file without changing an existing file in the website root directory. To configure the virtual robots.txt, go to the ...

  • Settings – authentication

    Netpeak Spider allows performing SEO-audit of a website at the development stage and crawling it before search engine robots do. The ‘Authentication‘ allows getting access to websites secured by basic...

  • Settings – proxy

    Ways of adding proxies How to work with proxies in Netpeak Spider A proxy represents an intermediate server between a user device and a target server (website). A proxy allows changing users IP addres...

Still Thinking?

Thousands of specialists around the world use Netpeak Software products for daily SEO-tasks. Sign up to get free access right now!