How Do You Filter Bots?

Understand Filtering Bots

Convert currently filters bots if the user agent contains the following words:

'abachobot','abilogicbot','accoona-ai-agent','anyapexbot','arachmo','b-l-i-t-z-b-o-t','baiduspider','becomebot','bimbot','blitzbot','boitho.com-dc','boitho.com-robot','btbot','cerberian drtrs','charlotte','converacrawler','cosmos','curl','dataparksearch','diamondbot','discobot','emeraldshield.com webbot','envolk[its]spider','esperanzabot','exabot','fast enterprise crawler','fast-webcrawler','fdserobot','findlinks','furlbot','fyberspider','g2crawler','gaisbot','galaxybot','geniebot','gigabot','girafabot','googlebot','googlebot-image','hl_ftien_spider','htdig','ia_archiver','ichiro','irlbot','issuecrawler','jyxobot','lapozzbot','larbin','libwww-perl','link valet','link validity check','linksmanager.com_bot','linkwalker','lmspider','lwp-trivial','mabontland','mediapartners-google','mj12bot','mnogosearch','mogimogi','mojeekbot','mojoo robot','morning paper','msnbot','msrbot','mvaclient','netresearchserver','ng-search','nicebot','notifixious','noxtrumbot','nusearch spider','nutchcvs','obot','oegp','omniexplorer_bot','online link validator','oozbot','orbiter','pagebiteshyperbot','peach','polybot','pompos','psbot','pycurl','python-urllib','rampybot','reciprocal link system pro','rel link checker lite','rufusbot','sandcrawler','sbider','scrubby','searchsight','seekbot','semanticdiscovery','sensis web crawler','seochat::bot','shim-crawler','shopwiki','shoula robot','silk','sitebar','snappy','sogou spider','speedy spider','sqworm','stackrambler','surveybot','synoobot','teoma','terrawizbot','thesubot','thumbnail.cz robot','tineye','turnitinbot','updated','vagabondo','vivante link checker','voilabot','vortex','voyager','vyu2','w3c-checklink','webcollage','websquash.com','wf84','wofindeich robot','xaldon_webspider','xenu link sleuth','yacy','yahoo! slurp','yahoo! slurp china','yahooseeker','yahooseeker-testing','yooglifetchagent','zao','zealbot','zspider','zyborg'

Canonical Tags

You dont need to add canonical tags to the variation of your experiments. The reason is that we filter search bots, which result on non indexing of experience variations.

Have more questions? Submit a request

Comments