Putting the Wisdom of Crowds to the Search Test

Over the last year, we’ve seen the rise of personalized search and social search: search results custom-tailored for individual interests, or filtered through the lens of what friends, colleagues and topic experts have found useful. In each case, the aim was to cut down on irrelevant search results and give the user useful information more quickly—thereby, it is hoped, giving users a more satisfying search experience.

A new technology from start-up firm Collarity now offers to use both those techniques to sharpen the results of a general Web search. The Collarity Compass is a search tool that can overlay either a Web site or a Web search engine and sharpen the search results provided. The tool was built on the basis of three questions, says company co-founder and CEO Levy Cohen: “Why are my search results the same as my grandmother’s? Why does all the energy I spend searching vanish into thin air? And finally, why can’t I share my search results with other people effortlessly?”

Collarity is intended to take search technology further along evolutionary path it’s been on so far, Cohen says—starting with keyword-heavy pages, moving to links that suggest popularity, and then going to page rankings to determine which pages are most authoritative on a subject. That’s about as far as machines can take us in determining relevance, he says; to gain any more ground, you need to introduce people into the equation.

Collarity doesn’t crawl the Web to index pages on its own. Rather, it sits on top of other indexes already compiled and makes them more relevant, Cohen says. It does this by setting up dialogues with human users, rather than focusing solely on the page content as algorithms do.

Collarity claims to be able to help users identify two groups that can help them sort the search results they want from the background static: topic experts and other searchers who share their interests.

Users of Collarity see a pop-up AJAX-powered “Compass” that shows them suggested relevant searches, either on the entire Web or on a specific site that’s Collarity-enabled. For example, typing in “cars” brings up four auto-related topic areas—vehicles, online, rental and listings—that might be of interest, and three specific car-content Web sites.

If a user is using the Collarity overlay on a search engine site, he can click on one of these topics to indicate that he’s looking for car rentals, not automotive news, for example. Those more focused results come up in the familiar search interface of whatever engine he’s using, but without the unwanted search results.

Users can register with Collarity and let the tool start learning about their personal interests and enthusiasms based on the pages they click in search results. That way, Collarity can help decide whether you’re looking for computer news or cooking tips when you search on the word “apple”.

But users who choose not to register can also use the tool anonymously. Collarity groups searchers on a keyword or a Web site into affinity communities, collections of users that have spent time searching for similar things. It also looks at the whole universe of Collarity users and determines which topics and pages have been most popular for a given search.

Users can tap into that stored knowledge using a slider on top of the Compass pop-up interface. This lets them toggle between several levels of “community” and “global” knowledge for a search, with different suggestions served up in real time via AJAX. If they’re registered, they can also toggle to “personal” results and get what Collarity thinks are the most relevant listings for them based on their past search history.

Cohen points out that the communities that Collarity builds are implicit and not pre-determined; they tend to arise naturally and then decline, as for example the group of sports fans searching on the term “March Madness” during the NCAA championship season.

And those communities are defined by the way users actually behave, not by their descriptions of themselves. Cohen points out that a lot of social search models fall down on the fact that they ask people to do something extraneous to search, such as rating content for quality or tagging it to indicate its relevance for given groups.

“There are a lot of companies with ideas about how to apply human intelligence to search,” he says. “But we don’t believe social search will ever be popular if you force users to rank and rate information. Most people are too selfish and too lazy to do that. Second, the extra work involved means that not enough people participate in the process. And the systems can’t keep pace with changing user interest over time.”

By watching what people actually do in a search rather than asking them to step outside the process and fill out tags or ratings, Cohen says, Collarity produces a more accurate picture of the audience for a given item of Web content.

And that can produce benefits for both publishers and advertisers, he says. “We offer publishers an automated way to serve up the right content to users based on their implicit, self-defined interests. That helps Web publishers better understand their specific audiences’ likes and dislikes, and this helps them stay competitive by showing them the interest areas their audience is moving toward.” The thought is that publishers can them personalize both the content and the ads they show to visitors, increasing both their interest for readers and their appeal for marketers on the site. Publisher should be able to charge higher rates for better-qualified audiences, and advertisers should see better conversions from their campaigns on those sites.

All that is in the conditional tense because it’s still very early days for Collarity. The platform just came out of a private beta test in time for the ad:tech New York show earlier this month, so the company doesn’t have any publisher deals it’s ready to talk about yet.

But Cohen says the publisher community has taken a distinct interest in what Collarity has to offer: both the improved search capabilities that can be applied either to their own sites or to the Web, and the opportunities for more and better targeted ad inventory, either pay-per-click text ads or display banners.