Next time you perform a routine web search, look carefully at your top 10 results. Most users still naively believe their search engine of choice has simply trawled the web for the best possible matches.

But the chances are that several of the sites listed high in the rankings will have paid for the privilege, either through a straight pay-for-placement listing, or, more subtly, through use of a search engine optimisation (SEO) specialist.

While still a relatively new stock-in-trade, the web's increasing commercial clout, combined with belated recognition of the search engine as the online world's most critical tool, are together making SEO one of the fastest growing new niches in the internet economy.

A glance at the figures explains why. A medium-sized company might spend tens of thousands of pounds developing and maintaining a professionally-designed site filled with useful content and maybe even online sales capability. But studies reveal that the vast majority of commercial sites actually damage their own chances of being found by the Big Six search engines - Yahoo, Google, MSN, AOL, Lycos and AltaVista - through a simple lack of understanding of what the search engine "spiders" that trawl the web are looking for.

Popular animated gizmos based on products like JavaScript or Flash, for example, are to all intents and purposes invisible to search engine crawlers. So are Active Server Pages, JavaServer pages, elements contained within frames, and pages that enforce acceptance of cookies - the hidden files sent by web servers to individual PCs that are used by websites to identify users.

Yet a study recently commissioned by iProspect, a US-based SEO pioneer, shows that 97 per cent of websites belonging to Fortune 100 companies routinely employ one or more of these technologies. Even more surprising, 50 per cent of the keywords nominated by the companies as online "signposts" to their sites failed to win that company a significant ranking on the major engines.

With around 80 per cent of the web's estimated 430m users routinely relying on one of the top search engines to find the information they want, that is a lot of lost leads for any company intent on taking its web presence seriously. "An extraordinary number of multi-million dollar sites are literally dead on arrival for the major engines," says Frederick Marckini, iProspect's chief executive and the author of three best-selling books on search engine positioning.

"To effectively optimise your site, it is important to understand that search engines essentially do two basic things: index text and follow links. If your site doesn't contain these, you've just made yourself invisible," he says.

Virtually alone in the field when it was founded back in 1996, iProspect has built a solid business raising the online profile of clients such as 3M, Sharp Electronics and John Deere. Now, the company is being joined by a growing band of SEO specialists around the world eager to exploit what many believe to be not only one of the internet's best opportunities, but a must-have service for just about any company.

In Europe, the market already boasts a handful of dedicated SEO professionals, including Sticky Eyes, NetBooster, Search Engineers and Web Gravity, along with an increasing number of tech-savvy PR and advertising agencies, such as Brodeur Worldwide, which has recently added SEO to its list of client services.

All agree that the key to achieving high search rankings lies in the effective use of meta-data (invisible tags contained within a site's HTML script that provide information on content), along with a thorough understanding of the real-life words and phrases surfers actually use to find information, and simplicity of coding and design.

Surprisingly, perhaps, for such a tech-centric environment, many companies also favour a human element. Leeds-based Sticky Eyes, for example, has a team of 18 dedicated journalists "sleuthing" the sites belonging to clients such as Ferrari, Pedigree Dogfood and Dunlop, ensuring pages are kept as relevant and visible as possible.

While founder Paul Sowerby admits it is a labour-intensive process, he guarantees clients a top 10 ranking in the eight biggest engines, or their money back. "With some engines changing their algorithm every 10 days or so, the human element is vital to delivering an assured outcome. It also helps us match a client site's performance against its top five competitors and identify new marketing opportunities," he says.

Search Engineers, established in 1997 as the UK's first SEO specialist, relies on much the same business model, with a on-site editorial team and a "no placement, no payment" policy. Simon Cleaver, chief executive, is also quick to point out that to be effective, SEO should be viewed as a process, not a project.

"Each spider looks for different things - key words, external links, number of hits - so it is important to constantly tailor sites to cater for each engine's preferences. To supplement this, we manually resubmit all client sites to the engines at least once a month and rectify any position slippage," he says. The company currently services around 5,000 sites, including big names such as Railtrack and Virgin Travel Stores.

While net libertarians have traditionally argued that SEO unfairly skews search results in favour of companies, rather than users, there is a general consensus nowadays that ethical SEO can actually help customers find the services they want, delivering the kind of one-to-one marketing and service the internet has always promised.

"In reality, there's little difference between the online world and the world of bricks and mortar," says Michael O'Connell, head of online communications strategies with Brodeur Worldwide. "One shop may pay for a prominent listing in the Yellow Pages, another may not. Both can be found, but one has an advantage - an advantage they paid for - in attracting customers."

Users will also be relieved to hear that unscrupulous practices that attempt to "trick" search engines through techniques such as excessive keyword repetition, spamming (continual automated resubmission of sites), or cloaking, which uses invisible "gateway" pages to attract the attention of search engines while redirecting unsuspecting users to a site often unrelated to the original search criteria, are roundly denounced by the professionals.

Henrik Hansen, director of marketing for enterprise search with Inktomi, which provides the search database that powers engines such as MSN, AOL, Lycos and Freeserve, says his company's engineers actively work to combat unethical behaviour through increasingly sophisticated anti-spamming algorithms and regular human intervention by a team of editors, who check search results for accuracy and relevance.

Danny Sullivan, a well-known search specialist and editor of online publication SearchEngineWatch.com, says cloaking also has the potential to degrade the quality of the web by choking it with reams of irrelevant pages designed only to get picked up by the engines.

"Using techniques that try to trick the engines into doing what you want is not where companies should be putting their web development effort," he says. "Properly done, SEO can be highly effective, generating qualified traffic for site owners, improving search engine accuracy and delivering relevant, useful information to users."