Yahoo is now giving away the crucial technology powering its internal searches. Vespa, the search engine used by the tech company for internal queries, is now open-source and available to everyone.
Oath Inc., the Verizon company that acquired Yahoo in June, announced that Vespa is now available as open source on GitHub. According to a company blog post, making the Big Data processing and serving engine open source is a step further in Oath’s commitment to opening up its infrastructure to developers.
“By releasing Vespa, we are making it easy for anyone to build applications that can compute responses to user requests, over large datasets, at real time and at internet scale – capabilities that up until now, have been within reach of only a few large companies,” Jon Bratseth, a systems architect at Yahoo, explained via the blog post.
Vespa, Yahoo’s big data processing and serving engine, available as open source on GitHub!https://t.co/InVSKoSlZY
— Arkaitz Etxezarreta (@arkaitzo) September 29, 2017
Vespa powers Yahoo network’s internal search feature and is used to determine what recommendations to display as well as the company’s ad targeting system. Currently, Vespa is used in around 150 apps such as Yahoo’s main search engine, Yahoo Mail, and Flicker, CNBC reported. The search tool is also responsible for serving around 3 billion native ads daily.
Vespa traces its roots back to AllTheWeb, a Norwegian search engine which was acquired by Yahoo in 2003, according to Wired. Since then, the tool has been modified as a general purpose engine that Yahoo could use internally in a host of different applications.
This is not the first time Yahoo released some of its technology as open source. In 2009, the company released Hadoop as open source which became a hit when it was adopted by big tech companies such as Facebook, Twitter, and eBay. Apparently, the plan is to replicate the benefits of making Hadoop publicly available which allowed Yahoo to easily recruit programmers as it became widely used.
But Vespa’s reach has the potential to overshadow that of Hadoop. As Bratseth puts it, “Vespa is larger in scope and lines of code than any open source project we’ve ever released,” adding that it has already been battle-proven in Yahoo’s largest and most critical systems. It is attractive both for its scalability and versatility while it is touted to be better than Hadoop in serving results to end users.
However, there are doubts that Vespa could be successful outside of Oath. According to Wired, Hadoop became a success because it was born open source and arrived at a time when businesses needed it most. This time, however, most large companies have already addressed web search issues that Vespa is designed for. In addition, there are already other open source engines available.
[Featured Image via Yahoo]