WebProNews

Tag: Harry Shum

  • Are Bing’s Results Better Than Google’s?

    Are Bing’s Results Better Than Google’s?

    Does Bing deliver better search results than Google? Bing thinks so, and has launched a campaign trying to convince people that it’s right. It’s called “Bing It On,” and in addition to television ads, it includes the site BingItOn.com, where Microsoft is encouraging users to take its version of the Pepsi challenge – a blind comparison test between Bing results and Google results for whatever queries you wish to try out.

    Which search engine’s results do you prefer: Google’s or Bing’s? Let us know in the comments.

    According to Bing, people “chose Bing web search results over Google nearly 2 to 1.” Notice they said “Bing Web search results over Google,” rather than just “Bing over Google”. More on that later.

    Also notice, they said “chose,” and not “choose.” That’s because this is based on a study Microsoft commissioned, and may not reflect the results from users using BingItOn.com (although I’d be very interested to see how it turns out once they’re done with the campaign. Maybe they’ll show us that later).

    A Bing spokesperson told WebProNews in an email, “Although most people identify themselves as Google searchers, an independent study commissioned by Microsoft Corp. shows people chose Bing Web search results over Google nearly 2-to-1 in blind comparison tests. Given those findings, Bing decided it is time to let people see for themselves that there is a better option in search.”

    Bing sheds a little more light on the study in a blog post. “How was the test conducted?” the Bing team says. “An independent research company, Answers Research based in San Diego, CA, conducted a study using a representative online sample of nearly 1000 people, ages 18 and older from across the US. The participants were chosen from a random survey panel and were required to have used a major search engine in the past month. Participants were not aware that Microsoft was involved.”

    “When the results were tallied, the outcome was clear – people chose Bing web search results over Google nearly 2:1 in the blind comparison tests,” the team says. “Specifically, of the nearly 1000 participants: 57.4% chose Bing more often, 30.2% chose Google more often; 12.4 % resulted in a draw.”

    Bing also notes that the “overall sampling error rate for the study is +/- 3 percent at the 95 percent confidence level.”

    The following video shows Bing taking it to the streets, betting people an Xbox 360 that they’ll like Bing better than Google. Of course, in the video, everyone loves Bing.

    “When we previewed our side-by-side test results with people outside the company, I was often asked how we were able to make these gains with presumably less data than the other guys,” said Dr. Harry Shum, Corporate Vice President, Bing R&D in a blog post. “While there are too many variables to give a fully scientific explanation, I would say our long-term commitment and investment in machine learning for relevance has enabled us to steadily scale out relevance experimentation and make rapid progress.”

    “Of course, as we all know, relevance is subjective and queries are dynamic and always changing. But we feel confident that it’s time for customers to come give us a look, and for a conversation on searching quality to occur in our industry.”

    The Bing It On challenge, while very much a way for Bing to try and lure users away from Google, was also an opportunity for Bing to talk up some of the back-end tweaks it has made, much of this through extensive experimentation.

    “Relevance experimentation at Bing involves training machine-learned models on large amount of training data using thousands of features,” Shum wrote. “In the early years, our models were based on neural networks. But as the amount of training data, number of features and the complexity of our models increased, the inner loop of experimentation slowed down significantly. At one point, it took us several days to finish just one experiment end-to-end. We knew we needed to do something.”

    “To overcome this challenge, we turned to our deep partnership with MSR to develop a technology we call Fastrank,” he added. “FastRank is based on boosted decision trees which are much faster to train and thus attractive for relevance experimentation.. But there was skepticism on whether the quality of ranking produced by decision trees could match that of neural networks. Our colleagues at MSR took on this hard problem and developed new optimization algorithms that allowed us to not only match the quality of neural nets, but also train more than an order of magnitude faster.”

    Google seems to think it is lending Bing some help as well. Google’s Matt Cutts said in a Hacker News thread, “Last time I checked, it looked like Bing was still using clicks on Google search results as a signal in Bing’s rankings.”

    More on all of that here, but basically Cutts is referring to a big search industry story from 2011, when Google set up a sting operation to show that Bing was drawing from its search results. It appeared that Bing was using Google user search queries, gaining access to user data via an Internet Explorer setting.

    But even still, that would only be one signal, and Bing claims to use thousands of them, compared to Google’s regularly referenced “over 200”. Bing may be using a lot more signals, including one from the world’s most popular search engine, but does it really translate to better search results?

    Cutts also pointed out that the BingItOn tool struggled with a query for “bingiton”. Google did a better job of delivering results for Bing’s new tool than Bing did. I replicated the query personally, and was greeted with a similar result. Bing was showing stuff for the cheerleading “Bring It On” over Bing It On results, and Google was showing Bing It On at the top.

    Some readers, however, say they were getting Bing It On at the top for both search engines, so some personalization signals may have come into play, although I can’t honestly understand why Bing would tailor “Bring It On” results to me, especially given that I’ve been covering Bing since it launched (I have no recollection of ever searching for this movie).

    But, as embarrassing as it might be for Bing to show how Google is better at delivering results for a tool that Bing created to show how much better Bing results are than Google’s, this is still just one query, and the truth is that it doesn’t really prove very much. Anyone can easily find an example of Google providing a less than perfect results page.

    The truth is that no matter how many queries you perform, Google is going to win on some of them, and Bing is going to win on some of them.

    What do users think? The Twitter reaction is interesting. Here’s a sample:

     

    This one represents a significant obstacle Bing faces, regardless of search quality:

    This kind of mentality leaves one to ponder just how much the general population really cares about which one is technically providing better results more of the time. Of course, this has been part of the discussion since Bing launched. Even if it can deliver better results, most Google users are probably happy enough with Google, and simply aren’t looking for an alternative.

    As far as the Bing It On tool goes, you have to consider that this is not really an accurate portrayal of the search experience on either Google or Bing. Bing says right on the site, “Based on a comparison of web search results pane only; excludes ads, Bing’s Snapshot and Social Search panes and Google’s Knowledge Graph.”

    The Knowledge Graph is one of the offerings Google is prouder of than anything. Since launch, the company has taken just about every opportunity possible to talk about how revolutionary it is, and what a major step forward in search it is. Bing usually touts its social search features with similar enthusiasm. It strips out the search filtering options, personalization features, and the user interface entirely. There is more to the search experience than what is presented by Bing It On.

    Then there are the home pages. People love Google doodles, for example. Some love Bing’s daily photos. Some like the way Bing does image search or videos. There’s also the fact that people use other products from these companies. Google users are often signed in, and can easily navigate around the various services they use from one unified navigational experience. Search is just a feature of the Google experience.

    The point is, it’s not just about the “ten blue links,” which ironically, is a point that Bing has made in the past.

    So, moving beyond the results as Bing is presenting in the Bing It On challenge, which search engine offers the better all-around user experience? Which one does have the better results? Let us know what you think in the comments.

  • Cutts: Last Time I Checked, Bing Was Still Using Google As A Signal

    Microsoft unveiled its big “Bing It On” campaign this week. Part of that is a site, which allows users to perform a search query, and choose which results they prefer. After five rounds, the tool reveals whether your picked Bing or Google for each one. The one you picked the most, must be your search engine of choice.

    It’s an interesting comparison of organic search results between the two rivals, but it strips out large parts of the user experience for both search engines. It doesn’t include Google’s Knowledge Graph or BIng’s social bar, for example. It’s hardly an accurate representation of today’s search experience for either engine.

    Still, Bing says people prefer Bing to Google two to one:

    On Thursday, after the Bing It On site was launched, Google’s Matt Cutts tweeted:

    I tested the query out myself, and found similar results to Cutts’.

    Bing It On

    It must be embarrassing for Bing to have Google beat Bing on a query for the very tool that Bing is promoting to show that its results are better for Google. However, it’s unclear how many users actually had these results. In the comments section of an article we did on that, some users said BingItOn.com was the top result on both. Perhaps some personalization signals come into play, though I can’t imagine why Bing would associate the cheerleading movie “Bring It On” or its musical counterpart with anything from my personal life. Honest.

    Despite the differences in results in this example, there are other queries that provide much more similar results. Cutts had something to say about that too.

    In a comment thread on Hacker News (via Barry Schwartz), Cutts said, “Last time I checked, it looked like Bing was still using clicks on Google search results as a signal in Bing’s rankings.”

    It’s funny he should mention that, because I couldn’t help but be reminded about that whole ordeal as I was playing around with Bing It On.

    Early last year, Google ran a “sting operation,” as Danny Sullivan who first reported on the story called it, that appeared to show Bing “stealing” at least top results from Google, by monitoring how Internet Explorer and Bing toolbar users use Google.

    Google created some test search results pages returning results for queries that nobody would ever search for, and results that wouldn’t make sense for such queries. For example, a query for “hiybbprqag” would return a top result from TeamOneTickets. A query for “mbzrxpgjys” would return RIM’s homepage. A query for ” indoswiftjobinproduction” would return a result for Sandra Lee Recipes at FoodNetwork.com.

    “The only reason these pages appeared on Google was because Google forced them to be there,” explained Sullivan at the time. “There was nothing that made them naturally relevant for these searches. If they started to appear at Bing after Google, that would mean that Bing took Google’s bait and copied its results.”

    Bing’s results were mirroring each of these examples, though Google found that only a handful of the pages tested proved the point.

    In response, Bing’s Harry Shum wrote in a blog post, “We use over 1,000 different signals and features in our ranking algorithm. A small piece of that is clickstream data we get from some of our customers, who opt-in to sharing anonymous data as they navigate the web in order to help us improve the experience for all users. To be clear, we learn from all of our customers. What we saw in today’s story was a spy-novelesque stunt to generate extreme outliers in tail query ranking. It was a creative tactic by a competitor, and we’ll take it as a back-handed compliment. But it doesn’t accurately portray how we use opt-in customer data as one of many inputs to help improve our user experience.”

    “The history of the web and the improvement of a broad array of consumer and business experiences is actually the story of collective intelligence, from sharing HTML documents to hypertext links to click data and beyond. Many companies across the Internet use this collective intelligence to make their products better every day,” Shum continued. “We all learn from our collective customers, and we all should.”

    “From its inception, we have had what we believe is a distinct approach to search, and the features and innovation in Bing – from our new user experience and visual organization approach to our focus on inferring user intent and helping customers complete complex tasks, Bing has added a new voice and new experiences to search,” he added. “We never set out to build another version of an existing search engine.”

    After that post, Cutts and Shum (as well as Blekko’s Rich Skrenta) had a discussion about the whole thing at the Farsight Summit. Throughout that, Cutts maintained the position Bing was basically cheating, and Shum echoed the sentiments of his post, adding, “My view is that we just discovered a new form of spam or click fraud and the Google engineers helped us to figure it out. He said that he wished people could share things like that with them before taking it to the press and getting a “wow effect”. He also said that it would be great if he and Matt could compare signals that they could use.

    At one point, Shum played the “Google has a toolbar too” card, but Cutts said users see “big red capital letters” letting them know about the data sharing as soon as they install it. Cutts also said at the time, “We don’t use clicks from Bing’s users in Google’s rankings.”

    In the new Hacker News thread, when asked how Microsoft has access to Google’s algorithm data, he replied, “IE and Windows, I believe.” He then points to a section in Microsoft’s IE 8 privacy policy, which says:

    “When Suggested Sites is turned on, the addresses of websites you visit are sent to Microsoft, together with standard computer information. … Information associated with the web address, such as search terms or data you entered in forms might be included. For example, if you visited the Microsoft.com search website at http://search.microsoft.com and entered “Seattle” as the search term, the full address http://search.microsoft.com/results.aspx?q=Seattle&qsc0=… will be sent.”

    “Most people have little idea that allowing a feature called ‘Suggested Sites’ will result in their Google searches and clicks being sent to Microsoft, or that Microsoft will use clicks on Google search results in Bing’s ranking,” said Cutts. “MSFT also uses something called the Microsoft CEIP (Customer Experience Improvement Program), and I think that’s either opt-out already or they’re making it opt-out in Windows 8–it’s built into the ‘Use Express Settings,’ I believe.”

    “Again, I haven’t looked at this very recently, but if you’re using a recent version of Windows and IE, you’re probably sending your searches and clicks to Microsoft unless you’ve been very careful about how you configured your computer,” he concluded.