New sites do not need to be “submitted” to search engines to be listed. However, Google and Yahoo offer a submission program such as Google Sitemaps that an XML type feed could be created and submitted. Generally however, a simple link from an established site will get the search engines to visit the new site and begin to spider its contents. It can take a few days or even weeks from the acquisition of a link from such an established site for all the main search engine spiders to commence visiting and indexing the new site.
Once the search engine finds a new site, it uses a crawler program to retrieve and index the pages on the site. Pages can only be found when linked to with visible hyperlinks. For instance, some search engines do not read links created by Flash or Javascript.
Search engine crawlers may look at a number of different factors when crawling a site, and many pages from a site may not be indexed by the search engines until they gain more pagerank or links or traffic. Distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled, as well as other importance metrics. Cho et al. (Cho et al., 1998) [9] described some standards for those decisions as to which pages are visited and sent by a crawler to be included in a search engine’s index.
Webmasters can instruct spiders not to index certain files or directories through the standard robots.txt file in the root directory of the domain. Standard practice requires a search engine to check this file upon visiting the domain, though a search engine crawler will keep a cached copy of this file as it visits the pages of a site, and may not update that copy as quickly as a webmaster does. The web developer can use this feature to prevent pages such as shopping carts or other dynamic, user-specific content from appearing in search engine results, as well as keeping spiders from endless loops and other spider traps.
For those search engines who have their own paid submission (like Yahoo!), it may save some time to pay a nominal fee for submission. Yahoo!’s paid submission program guarantees inclusion in their search results, but does not guarantee specific ranking within the search results.
White hat methods
White hat methods of SEO involve following the search engines’ guidelines as to what is and what isn’t acceptable. Their advice generally is to create content for the user, not for the search engines; to make that content easily accessible to their spiders; and to not try to game the system. Often, webmasters make critical mistakes when designing or setting up their websites, inadvertently “poisoning” them so that they will not rank well. White hat SEOs attempt to discover and correct mistakes, such as machine-unreadable menus, broken links, temporary redirects, or a poor navigation structure.
Because search engines are text-centric, many of the same methods that are useful for web accessibility are also advantageous for SEO. Google has brought the relationship between SEO and accessibility even closer with the release of Google Accessible Web Search which prioritises accessible websites.
Methods are available for optimizing graphical content, including ALT attributes, and adding a text caption. Even Flash animations can be optimized by designing the page to include alternative content in case the visitor cannot read Flash.
Some methods considered proper by the search engines:
Using unique and relevant title to name each page.
Editing web pages to replace vague wording with specific terminology relevant to the subject of the page.
Providing unique, quality content to address visitor interests.
Using an accurate description meta tag to make search listings more informative.
Ensuring that all pages are accessible via anchor tag hyperlinks.
Allowing search engine spiders to crawl pages without session IDs, cookies, or logging in.
Developing “link bait” strategies. High quality websites that offer interesting content or novel features tend to accumulate large numbers of backlinks.
Writing useful, informational articles under a Creative Commons or other open source license, in exchange for attribution to the author by hyperlink.
Black hat methods
“Black hat” SEO are methods to try to improve rankings which are disapproved of by the search engines, typically because they consider such methods deceptive, and unrelated to providing quality content to site visitors. Search engines often penalize sites they discover using black hat methods, by reducing their rankings or eliminating their listings from the SERPs altogether. Such penalties are usually applied automatically by the search engines’ algorithms, because the Internet is too large to make manual policing of websites feasible.
Spamdexing is the promotion of irrelevant, chiefly commercial, pages through deceptive techniques and the abuse of the search algorithms. Over time a widespread consensus has developed in the industry as to what are and are not acceptable means of boosting one’s search engine placement and resultant traffic.
Spamdexing often gets confused with white hat search engine optimization techniques, which do not involve deceit. Spamming involves getting websites more exposure than they deserve for their keywords, leading to unsatisfactory search results. Optimization involves getting websites the rank they deserve on the most targeted keywords, leading to satisfactory search experiences.
When discovered, search engines may take action against those found to be using unethical SEO methods. In February 2006, Google removed both BMW Germany and Ricoh Germany for use of these practices.[10]
Cloaking is the practice of serving one version of a page to search engine spiders/bots and another version to human visitors.
SEO and marketing
There is a considerable sized body of practitioners of SEO who see search engines as just another visitor to a site, and try to make the site as accessible to those visitors as to any other who would come to the pages. They often see the white hat/black hat dichotomy mentioned above as a false dilemma. The focus of their work isn’t primarily to rank the highest for certain terms in search engines, but rather to help site owners fullfill the business objectives of their sites. Indeed, ranking well for a few terms among the many possibilities does not guarantee more sales. A successful Internet marketing campaign may drive organic search results to pages, but it also may involve the use of paid advertising on search engines and other pages, building high quality web pages to engage and persuade, addressing technical issues that may keep search engines from crawling and indexing those sites, setting up analytics programs to enable site owners to measure their successes, and making sites accessible and usable.
SEOs may work in-house for an organization, or as consultants, and search engine optimization may be only part of their daily functions. Often their education of how search engines function come from interacting and discussing the topics on forums, through blogs, at popular conferences and seminars, and by experimentation on their own sites. There are few college courses that cover online marketing from an ecommerce perspective that can keep up with the changes that the web sees on a daily basis.
While endeavoring to meet the guidelines posted by search engines can help build a solid foundation for success on the web, such efforts are only a start. Many see search engine marketing as a larger umbrella under which search engine optimization fits, but it’s possible that many who focused primarily on SEO in the past are incorporating more and more marketing ideas into their efforts, including public relations strategy and implementation, online display media buying, web site transition SEO, web trends data anaylsis, HTML E-mail campaigns, and business blog consulting making SEO firms more like an ad agency.
This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia article “Search engine optimization”
