User Agent is an identification representing a part of HTTP request and containing information about the device, from which the connection to a server will be established.
By default, Netpeak Spider crawls websites using Google Chrome as a User Agent. Changing the User Agent allows you to test a website availability for search engine robots, different browsers, or to crawl device-adaptive content.
You can find the User Agent configuration under ‘Settings → User Agent‘.
User agent categories
Netpeak Spider → a User Agent that identifies Netpeak Spider. When it is used, the crawler will access pages introducing itself as NetpeakSpiderBot.
Search Engines → User Agents from this category are used to check the correctness of indexing directives for search engine robots written in the robots.txt file, Meta Robots, or X-Robots-Tag.
The category includes the following user agents:
Browsers → this category allows simulating as if a real user accessed a website. Using this category, it is also possible to check:
whether a website contains redundant redirects, links, blocks of data for specific browsers.
whether particular content is displayed for certain browsers (for instance, a notification that a website is not available for Internet Explorer 9).
The category includes the following browsers:
- Microsoft Edge
- Internet Explorer
Devices → is used to check if a website is device-adaptive, so you can check desktop, mobile, or tablet version of the same website. The following user agents are available within this category:
- Desktop Windows
- Desktop Mac OS X
- Tablet Apple
- Tablet Android
- Mobile Apple
- Mobile Android
Custom settings → allow you to configure your own user agent that will be included in HTTP requests and a user agent for checking indexing directives.
The Information about Selected User Agent
In the right part of the window, you can see the selected configurations, in particular:
- Selected template → a name of the selected template;
- Category → displays the selected User Agent category;
- HTTP Request User Agent → the information which will be added as a part of the HTTP header of requests to a web server.
- Robots User Agent → this configuration is used to set up a User Agent for checking the indexing directives from robots.txt file, Meta Robots, X-Robots-Tag.