Yes.
OK, I probably need to elaborate.
A few years ago some SEO’s I was speaking to felt that a Google sitemap, i.e. placing a sitemap.xml on your site and linking to it in Google Webmaster Tools, was not really a requirement. It does not help your SEO.
This may of course still be the case, but is it worth not bothering, considering how easy it is to put on there?
Recently I got a little insight into how much Google does use a sitemap. It relates to the little mishap I had the other day when I accidentally blocked googlebot.
According to my Google Webmaster Tools at the moment there are may errors from between 2nd and 4th October, all 403 errors that Googlebot encountered when trying to visit my site. The distribution of errors is interesting.
- HTTP (1,681)
- In Sitemaps (367)
This means that Googlebot hit 1,681 errors while crawling my site via links on the page (internal links). It also encountered 367 errors while trawling the sitemap.
Now the sitemap as a lot more than 367 entries, so I do not know why it only crawled 367 pages. It could be because Googlebot does not look too deep into the sitemap.xml file, or it only takes a certainly percentage of the pages. Or it could be because it abandons its attempt after do many failures. But what it does show is that Google does look at those sitemaps to have an extra insight into what is on your website.
Internal Links Are King – Or Don’t Make Google Hunt
Many people still stick to the “content is king” rule for SEO, and this is certainly still the case. But if you want you whole website crawled by Google, you have to also follow the “Don’t Make Me Think” (that’s a popular usability book by Steve Krug) approach, or more specifically “Don’t Make Google Hunt”.
To do this you need to ensure that all of your web pages are easily found by googlebot. Really, the same rules apply to googlebot as a human visitor. If your website is well planned, with good navigation, then it should be possible for a visitor to find your content within a few clicks. This is the guideline giving by Google.
So, do not leave pages buried on your website, with people (and googlebot) having to follow a long trail before they can find the page. Ideally all pages should be accessible from 3 clicks from the home page. A good example would be:
- Click main index
- Click section index
- Click page link
Or:
- Click main index
- Click section index
- Click a general page
- Click page link within this
For a small site this is easy to achieve, but once you have over 1000 pages then navigation becomes a bigger issue.
You need to be able to ensure that people can find your content without giving them too many options at once. It can be tempting to create massive index pages to provide all 1000 links, but this does not address the usability issues. Have ever tried to find something by searching through a list of 1000 items?
If you know there is something there you want to read, then this is possible. But if you are browsing, then it is unlikely you will go far beyond the first page (“above the fold”) before getting bored and visiting a better website.
So the trick is to order your content in bite sized chunks and provide as many useful sub-categories as possible without flooding your readers with too many options.
