Mozilla/5.0 (Linux Android 4.2.1 en-us Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko googleweblight) Chrome/.166 Mobile Safari/535.19 Mozilla/5.0 (X11 Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/.75 Safari/537.36 Google FaviconĬaution: Web Light doesn't respect robots.txt rules. Mozilla/5.0 (Linux Android 11 Pixel 2 DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/.193 Mobile Safari/537.36Ĭaution: For user-initiated requests, Google Favicon ignores Mobile agent: Mozilla/5.0 (Linux Android 7.0 SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/.125 Mobile Safari/537.36 (compatible Google-Read-Aloud +)Ĭaution: Duplex on the web may ignore the * wildcard.
User agent tokenĭesktop agent: Mozilla/5.0 (X11 Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/.118 Safari/537.36 (compatible Google-Read-Aloud +) Read Aloud doesn't respect robots.txt rules. User agent tokenĭoesn't respect robots.txt rules. (Various mobile device types) (compatible Mediapartners-Google/2.1 +) Mozilla/5.0 (Linux Android 6.0.1 Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ W.X.Y.Z Mobile Safari/537.36 (compatible Googlebot/2.1 +)
The full user agent string is a full description of the crawler, and appears inĬaution: These values can be spoofed.
This list is not complete, but covers most of the crawlers you might see on your One token, as shown in the table you need to match only one crawler token for a rule toĪpply. To match a crawler type when writing crawl rules for your site. The user agent token is used in the User-agent: line in robots.txt The following table shows the crawlers used by various products and services at Google: "Crawler" is a generic term for any program (such as a robot or spider) that is used toĪutomatically discover and scan websites by following links from one webpage to another.Ībout the common Google crawlers you may see in your referrer logs, and how to Overview of Google crawlers (user agents)
0 kommentar(er)
