Site Navigation and SEO - Guide to On Page Content Optimization for Search Engine Spiders

Certain types of site navigation and site structure may hold back or entirely prevent search engines from getting your website's content. As search engine spiders crawl the internet, they rely on the structural design of hyperlinks to find new documents and revisit those that may have changed.

Diving into Linking Strategies and Site Structures

Complex linking strategies and deep site structures, with little distinctive content and too much graphics may serve as "bumps", while the information that cannot be accessed by spiderable links may qualify as "walls." So don't use fancy editing tricks to tell a story. Use them only to enhance the story.

Possible "Speed Bumps" for Search Engine Crawlers:

• URLs with more than two dynamic parameters
• Pages with over 100 unique links to other pages (crawlers may not follow each of them)
• Pages more than 3 clicks away from the the home page
• Pages using a "Session ID" or Cookie during navigation (spiders may not be able to use these elements)
• Pages with "frames" can hold back the site crawling and confuse spiders about which pages to rank in the results

Possible "Walls" for Search Engine Crawlers:

• Pages requiring a drop down menu (HTML attribute) to access them
• Image map navigation structure
• Flash content and navigation
• Pages accessible only via a select form and submit button
• Documents accessible only via a search box
• Documents blocked purposefully (via a robots meta tag or robots.txt)
• Pages requiring a login
• Pages that re-direct before showing content

The Conclusion

The best way ensuring that a site's contents are completely accessible by spiders is to have direct, static HTML link to to every page you want the search engine to index and rank in the results. Always consider that if a page cannot be directly accessed from the website’s home page, that is the page where most spiders will start their crawling, it is less likely to be indexed by the search engines. A sitemap can be of tremendous help for this purpose. If your site don't have a sitemap, you can create one using our SiteMap Generator tool.