If you need to set a limit for the website crawling (e.g. exclude subdomains or include only one folder), you can use the following settings:
- Checkboxes on the ‘General‘ tab:
- Crawl only in directory – allows crawl exact website directory without leaving it.
- Crawl all subdomains – turn this function off to consider pages from subdomains outside the host specified in ‘Initial URL‘ as external.
- On the ‘Restrictions‘ tab you can set:
- Max number of crawled URLs – allows limiting number of crawled pages for scanning.
- Max crawling depth – allows determining how deep the program will crawl a website, based on the number of clicks from the initial URL to the crawled one.
- Max URL depth – allows determining how deep the program will crawl into directories of a website, based on the number of segments in a URL.
Max number of redirects – this value has an influence on several parameters:
- The number of redirects the program will follow to reach the target URL.
- The number of redirects to determine the corresponding issue in the sidebar.
You can combine all suggested options in the way you want, and also save them as a template for further part crawlings of your website.
In case, if there are some questions remained after reading the article, please contact our Customer Support.