Engines of the future: Search

"Search engines are the sunlight of the web, showing us what is visible," says Kevin Chang, a computer scientist at the University of Illinois at Urbana-Champaign. But by their nature, he adds, the beams that guide us also cast deep shadows.

Chang is referring to the huge power that search engines wield. With online commerce worth $34 billion a year in the US alone, any bias for or against a company in search results can have a huge effect on its bottom line.
Some studies suggest that search engines reinforce inequalities in the online world by giving websites that are already popular a high ranking in results, thus making them even more popular - an effect dubbed "Googlearchy". Other studies contradict this, indicating that web searches can boost low-key sites too.

Whatever the truth, with so much potential business at stake it is hardly surprising that website owners tweak their sites to try to ensure they float to the top of the rankings. Some even resort to tricks such as spamdexing - overstuffing web pages with keywords, say, or creating "link farms" of pages that repeatedly link to each other.
Now companies such as US-based Demand Media are trying to subvert search engines' algorithms on an industrial scale - in Demand Media's case, by mining the most popular keywords and links daily, then inserting them into thousands of brief, ad-laced "how-to" videos and articles it has churned out to capitalise on these trends. Demand Media's goal is to flood the web with a million of these high-profile pages every month and profit from the advertising revenue that they generate. It is not the only company adopting this strategy. In December 2009 internet giant AOL announced it would be following suit. And in May this year Yahoo bought Associated Content, another company producing "search optimised" content.
So are search engines unwittingly creating a web mired knee-deep in junk content? Not necessarily, says Sue Feldman of consultants IDC, based in Boston: "When you present people with a search result that's intended more to sell them something than tell them something, they don't react positively." According to internet analysts Hitwise, surfers are becoming more sophisticated too, adding extra words in each search query for better results. Besides, developments such as semantic search and the ability of search engines to tailor results to your search history and location will help us avoid this cyberlitter, says Stephen E. Arnold, author of The Goo

gle Trilogy. "The search experience will be more fluid and more intelligent," he predicts.