Tag: Search

Does Bing deliver better search results than Google? Bing thinks so, and has launched a campaign trying to convince people that it’s right. It’s called “Bing It On,” and in addition to television ads, it includes the site BingItOn.com, where Microsoft is encouraging users to take its version of the Pepsi challenge – a blind comparison test between Bing results and Google results for whatever queries you wish to try out.

Which search engine’s results do you prefer: Google’s or Bing’s? Let us know in the comments.

According to Bing, people “chose Bing web search results over Google nearly 2 to 1.” Notice they said “Bing Web search results over Google,” rather than just “Bing over Google”. More on that later.

Also notice, they said “chose,” and not “choose.” That’s because this is based on a study Microsoft commissioned, and may not reflect the results from users using BingItOn.com (although I’d be very interested to see how it turns out once they’re done with the campaign. Maybe they’ll show us that later).

A Bing spokesperson told WebProNews in an email, “Although most people identify themselves as Google searchers, an independent study commissioned by Microsoft Corp. shows people chose Bing Web search results over Google nearly 2-to-1 in blind comparison tests. Given those findings, Bing decided it is time to let people see for themselves that there is a better option in search.”

Bing sheds a little more light on the study in a blog post. “How was the test conducted?” the Bing team says. “An independent research company, Answers Research based in San Diego, CA, conducted a study using a representative online sample of nearly 1000 people, ages 18 and older from across the US. The participants were chosen from a random survey panel and were required to have used a major search engine in the past month. Participants were not aware that Microsoft was involved.”

“When the results were tallied, the outcome was clear – people chose Bing web search results over Google nearly 2:1 in the blind comparison tests,” the team says. “Specifically, of the nearly 1000 participants: 57.4% chose Bing more often, 30.2% chose Google more often; 12.4 % resulted in a draw.”

Bing also notes that the “overall sampling error rate for the study is +/- 3 percent at the 95 percent confidence level.”

The following video shows Bing taking it to the streets, betting people an Xbox 360 that they’ll like Bing better than Google. Of course, in the video, everyone loves Bing.

“When we previewed our side-by-side test results with people outside the company, I was often asked how we were able to make these gains with presumably less data than the other guys,” said Dr. Harry Shum, Corporate Vice President, Bing R&D in a blog post. “While there are too many variables to give a fully scientific explanation, I would say our long-term commitment and investment in machine learning for relevance has enabled us to steadily scale out relevance experimentation and make rapid progress.”

“Of course, as we all know, relevance is subjective and queries are dynamic and always changing. But we feel confident that it’s time for customers to come give us a look, and for a conversation on searching quality to occur in our industry.”

The Bing It On challenge, while very much a way for Bing to try and lure users away from Google, was also an opportunity for Bing to talk up some of the back-end tweaks it has made, much of this through extensive experimentation.

“Relevance experimentation at Bing involves training machine-learned models on large amount of training data using thousands of features,” Shum wrote. “In the early years, our models were based on neural networks. But as the amount of training data, number of features and the complexity of our models increased, the inner loop of experimentation slowed down significantly. At one point, it took us several days to finish just one experiment end-to-end. We knew we needed to do something.”

“To overcome this challenge, we turned to our deep partnership with MSR to develop a technology we call Fastrank,” he added. “FastRank is based on boosted decision trees which are much faster to train and thus attractive for relevance experimentation.. But there was skepticism on whether the quality of ranking produced by decision trees could match that of neural networks. Our colleagues at MSR took on this hard problem and developed new optimization algorithms that allowed us to not only match the quality of neural nets, but also train more than an order of magnitude faster.”

Google seems to think it is lending Bing some help as well. Google’s Matt Cutts said in a Hacker News thread, “Last time I checked, it looked like Bing was still using clicks on Google search results as a signal in Bing’s rankings.”

More on all of that here, but basically Cutts is referring to a big search industry story from 2011, when Google set up a sting operation to show that Bing was drawing from its search results. It appeared that Bing was using Google user search queries, gaining access to user data via an Internet Explorer setting.

But even still, that would only be one signal, and Bing claims to use thousands of them, compared to Google’s regularly referenced “over 200”. Bing may be using a lot more signals, including one from the world’s most popular search engine, but does it really translate to better search results?

Cutts also pointed out that the BingItOn tool struggled with a query for “bingiton”. Google did a better job of delivering results for Bing’s new tool than Bing did. I replicated the query personally, and was greeted with a similar result. Bing was showing stuff for the cheerleading “Bring It On” over Bing It On results, and Google was showing Bing It On at the top.

Some readers, however, say they were getting Bing It On at the top for both search engines, so some personalization signals may have come into play, although I can’t honestly understand why Bing would tailor “Bring It On” results to me, especially given that I’ve been covering Bing since it launched (I have no recollection of ever searching for this movie).

But, as embarrassing as it might be for Bing to show how Google is better at delivering results for a tool that Bing created to show how much better Bing results are than Google’s, this is still just one query, and the truth is that it doesn’t really prove very much. Anyone can easily find an example of Google providing a less than perfect results page.

The truth is that no matter how many queries you perform, Google is going to win on some of them, and Bing is going to win on some of them.

What do users think? The Twitter reaction is interesting. Here’s a sample:

This one represents a significant obstacle Bing faces, regardless of search quality:

This kind of mentality leaves one to ponder just how much the general population really cares about which one is technically providing better results more of the time. Of course, this has been part of the discussion since Bing launched. Even if it can deliver better results, most Google users are probably happy enough with Google, and simply aren’t looking for an alternative.

As far as the Bing It On tool goes, you have to consider that this is not really an accurate portrayal of the search experience on either Google or Bing. Bing says right on the site, “Based on a comparison of web search results pane only; excludes ads, Bing’s Snapshot and Social Search panes and Google’s Knowledge Graph.”

The Knowledge Graph is one of the offerings Google is prouder of than anything. Since launch, the company has taken just about every opportunity possible to talk about how revolutionary it is, and what a major step forward in search it is. Bing usually touts its social search features with similar enthusiasm. It strips out the search filtering options, personalization features, and the user interface entirely. There is more to the search experience than what is presented by Bing It On.

Then there are the home pages. People love Google doodles, for example. Some love Bing’s daily photos. Some like the way Bing does image search or videos. There’s also the fact that people use other products from these companies. Google users are often signed in, and can easily navigate around the various services they use from one unified navigational experience. Search is just a feature of the Google experience.

The point is, it’s not just about the “ten blue links,” which ironically, is a point that Bing has made in the past.

So, moving beyond the results as Bing is presenting in the Bing It On challenge, which search engine offers the better all-around user experience? Which one does have the better results? Let us know what you think in the comments.