Netpeak Spider allows performing SEO-audit of a website at the development stage and crawling it before search engine robots do.
The ‘Authentication‘ allows getting access to websites secured by basic authentication. It is when a username and a password are included as a part of HTTP GET-request.
Follow these steps to use this feature:
- Go to the ‘Authentication‘ tab in the program settings.
- Tick the ‘Use basic authentication‘ checkbox to activate ‘Login‘ and ‘Password‘ fields. Click on the ‘OK’ button.
- Enter a website address into the corresponding field and start crawling using the ‘Start‘ button.
Netpeak Spider also supports URL links with the format https://username:password@example.com/, which may be useful when it is necessary to crawl a list of URLs from different websites.