�ݺ�ߣ

Searchbots:
Lost Children ... or ... Hungry Psychopaths?
What Do Searchbots Actually Do? (and why it matters)

© 2007-2012 Roland Dunn

“Googlebot ... may be unable to completely index
all the content on your site”


Doesn’t tell us what searchbots index


Webserver Logfiles ....


Over 2 months, 40% of all Googlebot’s requests
were to …. just 2 URL’s (on-site search URLs)



On a website with approximately 40,000 URLs in
Google’s index (allegedly, according to site:…)



Google’s index (allegedly, according to site:…)

On a website serving approx. 150-200K unique
natural search visits/month


were to …. 2 URL’s.

Google’s index.

On a website serving approx. 150-200K unique
natural search visits/month


Categorise URLs by “top level section” – perhaps more
useful than just URLs ...


Table of “top level sections” requested by Googlebot


Chart of “top level sections” requested by Gbot – 6 months


Does Googlebot request all URLs served by a website?


Does Googlebot request all URLs served by a website?

No!


How Does Googlebot Spend Its Time? What Does it Request?


Does Googlebot spend its time cost-effectively?


Does Googlebot spend its time cost-effectively?

Not always.


Googlebot does not always request all content


Googlebot can become distracted, obsessed, or even lost:


Googlebot can become distracted, obsessed, or even lost:
• On-site search
• Additive filters/faceted URLs e.g.
“shoes?size=3&colour=green&price=10&brand=smith”
• Sections with thin and/or very similar content


Googlebot may not spend its time efficiently
It needs help focusing on what we value


SMX West, March 2011, Matt Cutts:
(http://goo.gl/ZZz7E):
“... if Google determines a site isn’t as useful to
users, they may not crawl it as frequently"


Conclusions:
• Searchbots do not always request all content
• Searchbots can become distracted
• Searchbots may use their time inefficiently
• Their visits are precious – we need to treasure them
• We may need to help them focus


Suggestions:
• Embrace logfiles – full of useful information


Suggestions:
• Check searchbot behaviour


Suggestions:
• Check searchbot behaviour
• If distracted, lost, inefficient, poor experience:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
• Alter URL construction (expensive!)


Lost Children ... or ... Hungry Psychopaths?


Or .... Distracted Teenager?


Sonification: What do Search Engines Requests Sound Like?


Online Branding: Natural Search:
http://www.refinedpractice.com/ http://www.cloudshapes.co.uk/
T: @RefinedPractice T: @roland_dunn

�ݺ�ߣs Available At:
http://www.cloudshapes.co.uk/talks/


�ݺ�ߣ

BrightonSEO: SearchBots: Lost Children or Hungry Psychopaths? What Do Searchbots Actually Do?

More Related Content

BrightonSEO: SearchBots: Lost Children or Hungry Psychopaths? What Do Searchbots Actually Do?