How to stop moz dotbot from accessing your website
DotBot is Moz's web crawler, it gathers web data for the Moz Link Index
Dotbot obeys robots.txt rules before accessing your host machine so the easiest way to stop dotbot is by adding robots.txt rules that limit dotbot activities
To forbid a directory let's say "login" add;
User-agent: dotbot
Disallow: /login/
Disallow: /login/
Upon reading and parsing directives above moz dotbot won't dare access you sites login sub directory in it's craw routine
To forbid an entire website's access include directives below;
User-agent: dotbot
Disallow: /
Disallow: /
Alternatively you can limit crawl rate by adding directives below "time is probably in seconds"
User-agent: dotbot
Crawl-delay: 10
Crawl-delay: 10
I've attached an nginx log it's a trail left by dotbot along with it's ip and moz support e-mail address;
216.244.66.194 - - [19/Mar/2020:15:16:29 +0000] "GET /index.html HTTP/1.1" 200 13433 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"
Comments
Post a Comment