How to stop moz dotbot from accessing your website

-

DotBot is Moz's web crawler, it gathers web data for the Moz Link Index

Dotbot obeys robots.txt rules before accessing your host machine so the easiest way to stop dotbot is by adding robots.txt rules that limit dotbot activities

To forbid a directory let's say "login" add;

User-agent: dotbot
Disallow: /login/

Upon reading and parsing directives above moz dotbot won't dare access you sites login sub directory in it's craw routine

To forbid an entire website's access include directives below;

User-agent: dotbot
Disallow: /

Alternatively you can limit crawl rate by adding directives below "time is probably in seconds"

User-agent: dotbot
Crawl-delay: 10

I've attached an nginx log it's a trail left by dotbot along with it's ip and moz support e-mail address;

216.244.66.194 - - [19/Mar/2020:15:16:29 +0000] "GET /index.html HTTP/1.1" 200 13433 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"

Comments

Post a Comment

Popular posts from this blog

E2020 dynamic imports in JavaScript

-
Escmascript Is a standard from which JavaScript and typescript are both derived from. ES2020 is out it comes with it new ways of writing JavaScript code. Dynamic imports enable importing dependencies in a rather dynamic way. Initially JavaScript developers were restricted to statically load their dependencies for instance; import { property } from 'dependency' On runtime the JavaScript engine loads the required dependencies and makes them accessible globally. However that is a rigid way of writing code. Dynamic imports are  relatively more flexible therefore you can included them at any segment within your program. For instance if you want to load file system module only if a certain criteria is met you would do as follows; If(process.argv[2] == 'start') Import fs from 'fs' Dynamic imports return a promise use the syntax below to make use of them import('fs') .then(fs => {    // consume module fs })
Read more

Install steam locomotive sl using apt

-
sl or steam locomotive is a commandline application that displays steam locomotive "train" animations aimed to correct users who accidentally enter sl instead of ls.  initials sl stands for Steam Locomotive How to install ls or steam locomotive Assuming you've root privileges and installed apt package manager on your system Type sudo apt-get update sudo apt-get upgrade sudo apt-get install sl To test installation type sl If installation was successful You'll see an ASCII image of a train right in your terminal after few seconds it will stop
Read more

FlappingBird JS13K Games

-
Image
FlappingBird is a Mobile side scroller game where the player controls a bird, attempting to fly between columns trees without hitting them Development of this game started out when i was attempting to draw perfect clouds programmatically on a canvas i later added a few more clouds (Something's missing) add moon, stars,  buildings and also add Gradients and for the games character i settled on a bird (using Sprite sheet obtained freely from opengameart) add more them and animated everything. FlappingBird is a single player arcade-style game in which the player controls the bird, which moves persistently to the right. The player is tasked with navigating the bird through pairs of trees that have equally sized gaps placed at random heights. the bird automatically gravitates and only ascends when the player taps the touchscreen or spacebar. Each successful pass through a pair of trees awards the player one point. Colliding with a tree or the ground ends the gameplay. If a player succ
Read more

Snake monochrome (JS1024)

-
JS1024 is an annually held code golfing competition where you're basically required to create something creative in 1024 bytes or less of raw "JavaScript/HTML/WebGL" I participated for JS1024 2020 competition with my First entry titled "Snake monochrome" Originally inspired by the old Nokia game going by the same name "Snake" which i used to play as far back as 2007-2008 Another entry making it Second is named "Tetris monochrome" i was compiled to do this game as a dare from my Girlfriend on June 6 the "Tetris 1984" anniversary Third one is F1-racer it's inspired by my bestfriend who wasn't impressed with how i hooked up controls on "Tetris Monochrome" so he requested "i modify controls a bit", and i improved "input/output" by adding smooth touch input to it as requested Snake game contains two entities. The Snake and the Apple, the Snake is an Object with properties position (vect
Read more

Installing and using 'cowsay' with fortune

-
what is 'cowsay'  it's a simple commandline program of a cow which basically prints whatever text argument you pass to it inform of a speech bubble from a ancii cow c:owsay goes hand in hand with another cli programme called fortune it generates text fortune messages which can be easy be tunnelled into cowsay Cowsay is so popular that i decides to write a how to installation post To get started open your commandline terminal and type sudo apt-get update  sudo apt-get upgrade sudo apt-get install cowsay To install another cli app 'fortune' type the following in your terminal sudo apt-get install fortune After installation test fortune by typing in your console fortune // typical response Training is everything.  The peach was once a bitter almond; cauliflower is nothing but cabbage with a college education.                 -- Mark Twain, "Pudd'nhead Wilson's Calendar" To test cowsay type cowsay followed by a command you want to be said  Cowsay wil
Read more