Dancing with Google's spiders

Mar 9th 2006
From The Economist print edition

Search engines: Google is engaged in an elaborate dance with firms determined to keep their websites high up in its rankings

THE industrious spider “bots” that crawl around the web on behalf of Google, the world's biggest search engine, evoke both fear and reverence. Late last year the bots swept through the world's web servers to scrutinise some 8 billion web pages and determine their new rankings in Google's search results. As Google tweaks its mighty ranking algorithms, and applies them to the constantly changing pages of the web, different sites shuffle up and down wildly in its search rankings, repeatedly gaining and losing ground. This operation, which takes place two or three times a year, is known as a “Google Dance”.

Like hurricanes, Google Dances are given names, such as Bourbon, Gilligan and Florida, often by commentators at a website called webmasterworld.com. Jagger, the two-week dance that began late last October, moved through servers in three waves, eliciting chatter in internet forums. “Been kicked off the face of the index by jagger1,” reads a typical entry. “Will try not to panic til after jagger3.”

All search engines test and implement new algorithms constantly, though not in such a dramatic fashion. Because achieving a high search ranking is crucial to success for many online businesses, a huge industry has sprung up devoted to the pursuit of website visibility. In 2005 worldwide spending on such “search-engine optimisation” (SEO), grew 125%, to $1.25 billion, according to SEMPO, a Boston-based trade association. It predicts spending will grow by 150% this year.

The Google Dance can rattle companies, since precipitous—but often temporary—falls down the pecking order are common. But Paul Aelen of Checkit, a large SEO firm based in the Netherlands, admits that algorithm updates are “very exciting” for SEO experts, who can then test an array of tricks to figure out what works, and what doesn't.

SEO firms boost their clients' online rankings by tinkering with their websites to enhance the characteristics that search engines consider positive. This involves techniques such as simplifying complicated page addresses, rewriting copy to produce single-theme pages with accurate titles, adding extra keywords to the invisible page descriptions (or “metatags”) read by indexing software, and putting product information stored in databases directly on to fixed pages, so that search engines' bots can read it.

The quest to understand how search engines rank websites has become an obsession for some. Web forums devoted to the topic hum with debate, and many SEO experts comb through patent applications to find out what new algorithms are in the pipeline. That ruse, however, is becoming less effective. Matt Cutts, a senior engineer at Google who is assailed with algorithm questions at industry conferences, says his firm, like its competitors, carefully controls access to its secrets. “A lot of our best ideas don't get filed as patents because patents eventually become public,” he says.

The best SEO firms can do wonders for a site's ranking. Lee Odden, the president of TopRank Online Marketing, a firm based in Minneapolis, Minnesota, says one of his clients stopped paying out $50,000 a month on pay-per-click advertising; instead, Pacific Security Capital, an investment bank based in Beaverton, Oregon, now spends a tenth of that figure on SEO and claims the top spot for some 50 keywords on several search engines.

The most powerful determinant of a web page's importance is the number of incoming referral links, which is regarded as a gauge of the site's popularity. Site optimisers solicit links from directory sites, swap links with prominent sites, and fish for links from bloggers by sending them press releases. But the influence of links begets abuse. Unethical methods, known as “black-hat SEO”, include renting links from popular or long-established sites (their links carry more weight). Some unscrupulous SEO outfits even exploit loopholes in website-management tools to place hidden links on prestigious sites, such as those maintained by universities.

The “link farm” technique involves spamming the web with automatically generated bogus blogs, known as splogs, that link to a website to give it a boost. David Sifry, founder of Technorati, a blog search-engine based in San Francisco, estimates that 10,000 link-bearing splogs sporting content copied (or “scraped”) from legitimate sites are put online every day to fool the spider bots. “Cloaking” involves presenting content to indexing bots that is different from that which web surfers who visit the site will actually see. “Keyword stuffing” involves hiding popular search terms on a page—using, say, white text on a white background—to attract visitors.

Mahesh Murthy, the founder of Pinstorm, a company based in Mumbai that is one of India's biggest SEO firms, says that the Google Dance and other similar algorithm updates are constantly getting better at detecting and “shutting out” cheaters. This can mean relegation to the bottom of the rankings, or even removal altogether. Last month Google accused BMW, a carmaker, of using cloaking, and briefly removed its website from its index. But the financial stakes are so high that SEO firms will continue to look for new ways to boost their client's rankings. This is one dance, it seems, that will not be going out of fashion any time soon.

Attack of the Eurogoogle

Mar 9th 2006
From The Economist print edition

Search technology: Can an ambitious new European search engine, backed by the governments of France and Germany, challenge Google?

“WE MUST take the offensive and muster a massive effort,” said Jacques Chirac, the president of France, who went on to warn of the dangers of losing the battle for “the power of tomorrow” in a speech made last April. Standing beside him was Gerhard Schröder, then chancellor of Germany. In response to the formidable challenges posed by America, Japan and the emerging powers of China, India and Brazil, the two men announced that they had decided to step up their co-operation in a technological programme of vital strategic importance. A new fighter jet, perhaps, or a satellite surveillance system? No, the two heads of state were endorsing a plan to build a Franco-German internet-search engine, to be called Quaero (Latin for “I seek”).

The project would, said Mr Chirac, be undertaken with the help of government funds “in the image of the magnificent success of Airbus”. In a series of further speeches over the past few months, he has warmed to his theme: “We must take up the global challenge of the American giants Yahoo! and Google”; “Culture is not merchandise and cannot be left to blind market forces”; “We must staunchly defend the world's cultural diversity against the looming threat of uniformity”; “Our power is at stake.”

In July Mr Chirac noted that while French research has traditionally been good, it “now needs encouraging”. The following month the French government, the main financier and developer of Quaero, duly created the Agency for Industrial Innovation (AII), based in Paris, largely to oversee the project. The AII received an initial endowment of €1.7 billion ($2 billion). Michel Lemonier, a senior administrator at the AII, refuses to discuss how much of the budget is being allocated to Quaero because, he jokes, the leaders of other AII-funded programmes would be “very jealous”. Quaero is expected to be finished before any of the other planned AII projects, and may be online before the year is out.

The magic of Quaero, say its supporters, will be in the ambitious capabilities of its tentacles. Today, internet searches are performed using keywords. Of course, search engines can retrieve image, audio and video files, in addition to text documents. But this is done by matching the user's keywords to a text description of the image, audio or video content. Quaero users will be able to search the internet with keywords in the usual way; but in addition, researchers at Quaero's public-private consortium, led by Thomson and France Telecom in France and Siemens and Deutsche Telekom in Germany, are developing technology that will allow users to perform searches using pictures and sounds as query terms. “It's beyond Google,” says Marie-Vincente Pasdeloup of Thomson.

Quaero will allow users to search using a “query image”, not just a group of keywords. In a process known as “image mining”, software that recognises shapes and colours will then retrieve still images and video clips that contain images similar to the query image. (The software is being supplied by LTU Technologies, a firm based in Paris, which already supplies the technology to law-enforcement agencies for use in sifting through surveillance footage.) When Quaero finds an image without a description that matches a properly labelled image, it will append the description from the labelled image to the unlabelled one. This technique, called “keyword propagation”, will enrich the web linguistically: image descriptions in French, for example, will spread as they are tacked on to similar images, so that those images can also be retrieved by users who type in French keywords.

Meanwhile, in Germany, researchers at the University of Karlsruhe are developing Quaero's voice-recognition and translation technology, with funding from the European Commission. The idea is that this software will find audio files—such as political speeches or radio broadcasts—and then automatically transcribe and translate them into a number of European languages. The original audio files can then be found using keyword searches. In addition, speaker-identification software will allow users (via computer microphones) to search the internet for audio clips recorded in their own voices, or those of other speakers.

These are stunningly ambitious goals, and some of the audio features may not be ready by the time Quaero is launched. Yet they show that Quaero is intended to be far more than just another would-be Google, but a leap forward in search-engine technology. “Google is so hegemonic that no one even wonders about other interfaces,” says François Bourdoncle, the chief executive of Exalead, a French search engine that has taken on the task of integrating these various technologies under the Quaero umbrella.

Even so, the most striking difference between Quaero and Google is not technological, but ideological. Quaero is a classic example of European state-funded industrial policy, while Google is the very embodiment of American free-market techno-capitalism. The use of government funds to back Quaero has raised eyebrows, even among its supporters, who worry that it might fall foul of rules that prevent governments from using state aid to give favoured firms an unfair competitive advantage.

But so far no formal complaints against Quaero have materialised. The project is a public-private partnership, and the private firms involved are also committing considerable sums to it. The government funds, meanwhile, are being carefully distributed via a complex system of favourable loans, interest-free cash advances, forgivable loans and grants for pre-competitive research, all of which are allowed under international trade rules. The project is further protected by the fact that big public-research organisations, including France's National Centre for Scientific Research and Germany's RWTH-Aachen University, are also involved.

When Angela Merkel took over as German chancellor in November, there were rumblings that she might not match Mr Schröder's commitment to the project. In fact, her dedication to Quaero is even greater, says Jean-Philippe Touffut, the secretary-general of the Cournot Centre for Economic Studies in Paris, who co-ordinates collaboration between the project's French and German participants.

How will Google respond? Brad Fallon of SEO Research, a search-engine consultancy based in Atlanta, says Google has put the development of multimedia search technology into its “as-soon-as-possible” category, since it now offers only text-based searches. At the moment, however, “Quaero appears to have the edge,” he says. But when Google and the other big American search engines unveil their multimedia search features, they are likely to provide interfaces in foreign languages, just as they do today with text-based searches. So simply by existing, Quaero will make the cut-throat search-engine business even more competitive.

Google, of course, makes its money from advertising, and Quaero's backers hope that it too will produce substantial advertising revenues. But Quaero's chief aims are cultural and political, rather than commercial. Alexander Waibel, a research leader at the University of Karlsruhe who sits on Quaero's steering committee, offers an emotional justification too. “Europe wants to secure access that does not have to be channelled through American technology,” he says. And an official close to Mr Chirac adds that “the goal surpasses by far the industrial stakes.”

JANUARY   2006

On the mat

Direct mail is losing its effectiveness

AS MUCH as people hate it, junk mail works—otherwise they would not send you the stuff. Unlike television or newspaper ads, the return on invest­ment from a direct-mail campaign can be accurately measured. Even though only a small percentage of people may read the material, the response rate has been high enough to make junk mail one of the most cost-effective forms of marketing. But that could be changing.

One of the biggest users of direct mail is the financial services industry, which spent around $2.5 billion on mail shots in America last year. Unfortu­nately, people seemed less interested than ever in signing up for new credit cards or insurance policies. The response rate from some campaigns has fallen to just 1.4%, according to America’s Direct Marketing Association (DMA). In previ­ous years it was well above 2%. When it comes to responding to “direct-order” mail shots (signing up to an offer, as op­posed to merely expressing an interest), the rate has fallen even more dramati­cally, to 0.7% from 3.5% in 2004.

The problem seems to be too many envelopes cluttering too many mail boxes, “leading many consumers to dis­card the blizzard of solicitations they re­ceive”, according to Advertising Age. The trade publication says firms must target their mail shots better. But that costs more money. It could mean buying a list of potential customers.

But why write when you can ring? Despite “don’t call” lists, when it comes to soliciting a sale the phone produces the highest response rate, at an average

of 8.6%, according to the DMA. Some of the dodgiest marketing out­fits now use the phone to tout their of­fers. In Britain, the latest trick is to use automatic dialing machines to call mo­bile phone numbers at random, but then to hang up after just one or two rings. Seeing a missed call, the unsuspecting recipient will usually ring back just in case it is something important, only to be greeted by a sales pitch. Getting the customer to pay the cost of cold calls takes real chutzpah.

Ahead of his time

December 2005

The  Direct Marketing Association reported the following statistics

Who is buying?

• Seventy-two percent of Americans shop from home, buying from catalogs, over the Internet, over the phone, or through the mail.¹
• Seventy-five percent of women shop from the home or office, compared to 69% of men.¹
• Fifty-eight percent of Americans shop from catalogs, and 51% have made an online purchase.¹
• Forty-one percent of Americans shop both from catalogs and the Internet.¹
• Fifty-six percent of catalog shoppers also shop online.²

How much will they spend?

• Men plan to spend $1,489 on gifts, while women plan to spend $1,199.³ Adults with children less than 18 years old will spend an average of $1,553 on holiday gifts, compared to $1,182 for people without children less than 18.³

What do they buy?

• Fifty-seven percent of catalog shoppers buy apparel from catalogs. Other popular catalog purchases include Gifts (51%), Books/Music/Videos (48%), Home Décor or Furniture (31%), Toys & Games (31%), Electronics (27%), Sporting Goods (25%), Travel Services/Airline Tickets (23%), Tickets/Sports & Entertainment (19%), Flowers (13%), Gardening Supplies (13%), Pet Supplies (11%), and Food (8%).¹
• Books/Music/Videos are the most popular online purchases, with 51% of online shoppers buying these items on the Web. Other popular items purchased on the Web: Gifts (46%), Travel Services/Airline Tickets (44%), Apparel (43%), Electronics/Computers (38%), Tickets/Sports & Entertainment (35%), Toys & Games (28%), Home Décor or Furniture (21%), Sporting Goods (19%), Flowers (16%), Pet Supplies (11%), Gardening Supplies (9%), and Food (6%).¹

When do they shop?

• Nineteen percent of shoppers plan to buy all of their holiday gifts before Thanksgiving.³
• Thirty-three percent of shoppers will finish their gift buying two weeks before the holiday. Forty-three percent wait until the final week to complete their shopping, with 20% finishing the day before the holiday.³
• Men are more likely to procrastinate, with 48% finishing their shopping the final week compared to 39% of women.³

How often do they shop?

• Catalog shoppers place an average of 11 orders per year, making an average of three purchases from their favorite catalogs.²
• Fifty-nine percent of shoppers keep catalogs they order from for at least three months or until a new copy arrives.²

Why do they shop from home or office?

• Sixty-five percent of shoppers cite convenience as the reason they shop via catalogs, the Internet, mail, or phone. Other reasons for shopping from home or office include variety (41 percent), to save money (38 percent), and customer service (19 percent).