Start a conversation

User Agent Settings

User Agent is an identification representing a part of HTTP request and containing information about the device, from which the connection to a server will be established.

By default, Netpeak Spider crawls websites using Google Chrome as a User Agent. Changing the User Agent allows you to test a website availability for search engine robots, different browsers, or to crawl device-adaptive content.

You can find the User Agent configuration under ‘Settings → User Agent‘.

User agent categories

Netpeak Spider → a User Agent that identifies Netpeak Spider. When it is used, the crawler will access pages introducing itself as NetpeakSpiderBot.

Search Engines → User Agents from this category are used to check the correctness of  indexing directives for search engine robots written in the robots.txt file, Meta Robots, or X-Robots-Tag. 

The category includes the following user agents:

  • Google
  • Bing
  • Yahoo
  • DuckDuckGo
  • Yandex
  • Seznam
  • Qwant
  • Baidu
  • Naver

Browsers → this category allows simulating as if a real user accessed a website. Using this category, it is also possible to check:

  • whether a website contains redundant redirects, links, blocks of data for specific browsers. 

  • whether particular content is displayed for certain browsers (for instance, a notification that a website is not available for Internet Explorer 9). 

The category includes the following browsers:

  • Chrome
  • Firefox
  • Opera
  • Safari
  • Yandex
  • Microsoft Edge
  • Internet Explorer

Devices → is used to check if a website is device-adaptive, so you can check desktop, mobile, or tablet version of the same website. The following user agents are available within this category:

  • Desktop Windows
  • Desktop Mac OS X
  • Tablet Apple
  • Tablet Android
  • Mobile Apple 
  • Mobile Android

Custom settings → allow you to configure your own user agent that will be included in HTTP requests and a user agent for checking indexing directives.

The Information about Selected User Agent

In the right part of the window, you can see the selected configurations, in particular:

  • Selected template → a name of the selected template;

  • Category → displays the selected User Agent category; 

  • HTTP Request User Agent → the information which will be added as a part of the HTTP header of requests to a web server.

  • Robots User Agent → this configuration is used to set up a User Agent for checking the indexing directives from robots.txt file, Meta Robots, X-Robots-Tag.
Choose files or drag and drop files
Was this article helpful?

Still Thinking?

Thousands of specialists around the world use Netpeak Software products for daily SEO-tasks. Sign up to get free access right now!