How Do You Filter Bots?

Understand Filtering Bots

Convert is open to better technique but at this time we filter user agent and see if it contains any of 'abachobot','abilogicbot','accoona-ai-agent','anyapexbot','arachmo','b-l-i-t-z-b-o-t','baiduspider','becomebot','bimbot','blitzbot','boitho.com-dc','boitho.com-robot','btbot','cerberian drtrs','charlotte','converacrawler','cosmos','curl','dataparksearch','diamondbot','discobot','emeraldshield.com webbot','envolk[its]spider','esperanzabot','exabot','fast enterprise crawler','fast-webcrawler','fdserobot','findlinks','furlbot','fyberspider','g2crawler','gaisbot','galaxybot','geniebot','gigabot','girafabot','googlebot','googlebot-image','hl_ftien_spider','htdig','ia_archiver','ichiro','irlbot','issuecrawler','jyxobot','lapozzbot','larbin','libwww-perl','link valet','link validity check','linksmanager.com_bot','linkwalker','lmspider','lwp-trivial','mabontland','mediapartners-google','mj12bot','mnogosearch','mogimogi','mojeekbot','mojoo robot','morning paper','msnbot','msrbot','mvaclient','netresearchserver','ng-search','nicebot','notifixious','noxtrumbot','nusearch spider','nutchcvs','obot','oegp','omniexplorer_bot','online link validator','oozbot','orbiter','pagebiteshyperbot','peach','polybot','pompos','psbot','pycurl','python-urllib','rampybot','reciprocal link system pro','rel link checker lite','rufusbot','sandcrawler','sbider','scrubby','searchsight','seekbot','semanticdiscovery','sensis web crawler','seochat::bot','shim-crawler','shopwiki','shoula robot','silk','sitebar','snappy','sogou spider','speedy spider','sqworm','stackrambler','surveybot','synoobot','teoma','terrawizbot','thesubot','thumbnail.cz robot','tineye','turnitinbot','updated','vagabondo','vivante link checker','voilabot','vortex','voyager','vyu2','w3c-checklink','webcollage','websquash.com','wf84','wofindeich robot','xaldon_webspider','xenu link sleuth','yacy','yahoo! slurp','yahoo! slurp china','yahooseeker','yahooseeker-testing','yooglifetchagent','zao','zealbot','zspider','zyborg'

Have more questions? Submit a request

Comments