User Agent is an identification representing a part of HTTP request and containing information about the device, from which the connection to a server will be established.
By default, Netpeak Spider crawls websites using Google Chrome as a User Agent. Changing the User Agent allows you to test a website availability for search engine robots, different browsers, or to crawl device-adaptive content.
You can find the User Agent configuration under ‘Settings → User Agent‘.
User agent categories
Netpeak Spider → a User Agent that identifies Netpeak Spider. When it is used, the crawler will access pages introducing itself as NetpeakSpiderBot.
Search Engines → User Agents from this category are used to check the correctness of indexing directives for search engine robots written in the robots.txt file, Meta Robots, or X-Robots-Tag.
The category includes the following user agents:
- Bing
- Yahoo
- DuckDuckGo
- Yandex
- Mail.ru
- Seznam
- Qwant
- Baidu
- Naver
Browsers → this category allows simulating as if a real user accessed a website. Using this category, it is also possible to check:
-
whether a website contains redundant redirects, links, blocks of data for specific browsers.
-
whether particular content is displayed for certain browsers (for instance, a notification that a website is not available for Internet Explorer 9).
The category includes the following browsers:
- Chrome
- Firefox
- Opera
- Safari
- Yandex
- Microsoft Edge
- Internet Explorer
Devices → is used to check if a website is device-adaptive, so you can check desktop, mobile, or tablet version of the same website. The following user agents are available within this category:
- Desktop Windows
- Desktop Mac OS X
- Tablet Apple
- Tablet Android
- Mobile Apple
- Mobile Android
Custom settings → allow you to configure your own user agent that will be included in HTTP requests and a user agent for checking indexing directives.
The Information about Selected User Agent
In the right part of the window, you can see the selected configurations, in particular:
- Selected template → a name of the selected template;
- Category → displays the selected User Agent category;
- HTTP Request User Agent → the information which will be added as a part of the HTTP header of requests to a web server.
- Robots User Agent → this configuration is used to set up a User Agent for checking the indexing directives from robots.txt file, Meta Robots, X-Robots-Tag.
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article