yahooeng:

Fast, Approximate Analysis of Big Data

Abstract

In the analysis of big data there are often problem queries that don’t scale because they require huge compute resources to generate exact results, or don’t parallelize well. Examples include count distinct, quantiles, most frequent items, joins, matrix computations, and graph analysis. Algorithms that can produce “good enough” approximate answers for these problem queries are a required toolkit for modern analysis systems that need to process massive amounts of data quickly. For interactive queries there may not be other viable alternatives, and in the case of real-time streams, these specialized algorithms, appropriately called streaming algorithms, or sketches, are the only known solution. This methodology has helped Yahoo successfully reduce event processing times from days to hours or minutes on a number of its internal platforms. This article provides a short introduction to sketching and to DataSketches, an open source library of a core set of these algorithms designed for large analysis systems.

The Distinct Count Computational Challenges

Removing Duplicates. Suppose we have a small stream of visits to our new bookstore: {Alice, Ben, Dorothy, Alice, Ben, Dorothy, Alice, Ben}. The total count of visits is 8 and the distinct count of visitors is 3. In order to compute the distinct count exactly, the system must retain a reference to each distinct identifier it has seen so far in order to ignore the duplicates. This means that the system must reserve O(n) space, where n is the anticipated maximum number of distinct identifiers. This is straightforward if we know that the number of distinct identifiers is small.

Now extend the scale of this distinct counting challenge to streams that contain billions of identifiers with many duplicates. This is not an unrealistic scenario: Yahoo sees over a billion distinct users in a month. It can be even larger as our input streams often include multiple identifiers that we want to count, such as cookies, login-IDs, device-IDs, session-IDs, etc. The space required is now O(n1 + n2, …), where ni is the number of distinct identifiers of type i.

Partitioning and Non-Additivity. The challenge becomes exacerbated when we have to partition the data by anything other than the identifier itself. Partitionings that are often dimensions of interest to the business include time, product, location or other parameters.

For example, suppose we had decided to partition the visits to our bookstore by product area and day:

We have 4 partitions each with a distinct count of 2, but simple addition of these count values across any combination of more than one of these partitions will result in the wrong distinct count due to set overlap.  The distinct count values are non-additive. This is a more sinister form of duplication because not only has the total storage requirement increased, but the duplicates across partitions cannot be removed!  It is generally not possible to partition the data by business dimensions and guarantee that the identifier sets do not overlap.

This non-additivity property eliminates the possibility of answering queries across multiple partitions by only referring to the distinct count values for each partition. Any query across multiple partitions requires an entirely new distinct count operation that would have to read from the raw data or a copy of it. This non-additive property also has an impact on the system architecture in that we cannot create the nice aggregate hypercubes of a data mart and then query only the rows that qualify some predicate and sum up the results.

In other words, exact distinct count operations do not scale well. This non-additivity property of distinct counts is generally well understood by systems engineers. However, what is less well known is that now there are advanced algorithms that help us address the scalability challenge of distinct counting.

Sketching Algorithms

The name “sketch”, with its allusion to an artist’s sketch, has become the popular term to describe algorithms that return “good enough” approximate answers to queries, as these algorithms typically create small summary data structures that approximately resemble the much larger stream that it processed.

Sketching is a relatively recent development and has evolved from a synergistic blend of theoretical mathematics, statistics and computer science. Sketching refers to a broad range of algorithms and has experienced a great deal of interest and growth since the mid-1990’s coinciding with the growth of the Internet and the need to process and analyze massive data.

There are several common characteristics of sketches:

• Streamable. Sketches are especially suitable for environments where huge amounts of data flow by as a stream of individual items. Each item of the stream, examined only once, must quickly update a small sketch summary data structure.
  

• Approximate with predictable error. Sketches achieve their amazing speed and predictable error by taking advantage of a fundamental assumption: If we can accept a certain margin of error, it is often possible to develop algorithms that can compute the result substantially faster with fewer resources.   An important subset of sketches, called “stochastic streaming algorithms” intentionally introduce random variables into the algorithm.
  

• Sublinear in size. The required storage resources grow more slowly than the input data size (or don’t grow at all) and can be orders-of-magnitude smaller (typically kilobytes to megabytes). Sketches allow the user to configure the size of the sketch as a trade-off with accuracy.
  

• Mergeable, thus “additive”. To be useful in large data processing systems the sketch summary data structures should be “mergeable”. That is, the merge of two sketches should produce the same result (within the specified error bounds) as if the two streams that produced the two sketches had been combined prior to being submitted to a single sketch. This enables arbitrary partitioning of the input data and fast “summing” of the intermediate sketches for fast query results.
  

• Highly parallelizable. Since summary data structures are mergeable, the computations using these summary data structures are highly parallelizable and suitable for use in large-scale compute environments such as Hadoop or Druid.
  

Distinct Count Sketch, High-Level View

The first stage of a distinct count sketching process is a transformation that gives the input data stream the property of white noise, or equivalently, a uniform distribution of values. This is commonly achieved by hashing of the input distinct keys and then normalizing the result to be a uniform random value between zero and one.

The second stage of the sketch is a data structure that follows a set of rules for retaining a small bounded set of the hash values it receives from the transform stage. This fixed upper bound on sketch size enables straightforward memory management.

The final element of the sketch process is a set of estimator algorithms that, upon a request, examines the sketch data structure and returns a result value. This result value will be approximate but will have well established and mathematically proven error distribution bounds.

As an example of accuracy, a sketch configured to retain 16,000 values will have a relative error of less than 1.6% with a confidence of 95%. The error distribution is approximately Gaussian or bell shaped (as shown in the figure), and is independent of the size of the input stream, which can be in the many billions of distinct items.

The DataSketch Library

DataSketches is a Java software library of streaming algorithms specifically designed for the analysis of massive data. The library includes multiple high performing sketching algorithms and numerous other supporting algorithms targeted to the practical application of these advanced algorithms in real systems.

Sketch Adaptors are provided for Hadoop Pig, Hadoop Hive, and Druid. In both Hive and Druid, the adaptors are being integrated as built-in functions by the respective teams.

• Maven deployable. The library is designed to be deployed using Maven and the required jars are available from Maven Central. These jars can be integrated into any system that is JDK 7 or JDK 8 compatible. The sketches-core repository has no run-time dependencies, which makes it especially easy to integrate into virtually any Java-based system.
  

• Robust, High Quality Implementations. Extensive unit test code coverage,  comprehensive javadocs and code documentation, thorough accuracy and performance characterization, and time-tested usage in major back-end analysis systems inside Yahoo combine to make this library very robust and suitable for production applications.
  

• Benchmarking. Code used for characterizing components of the library for speed, accuracy and performance is included in the test package hierarchy. This provides transparency as to how the many data plots on the web site were constructed.
  

• Counting Distinct Identifiers. Currently the library includes two different update sketch families as well as union, intersection and difference operations for set expressions. These sketches are all part of the Theta Sketch Framework, described in our ICDT 2016 paper mentioned below. Most of these sketches can also be configured to operate either on the java heap or off-heap. In addition, there are two versions of the famous Hyper-Log Log (HLL) sketch, which is ideal for environments where only distinct counting and merging are required and space is extremely tight.
  

• Quantiles. Given a large stream of numeric values, such as web-page load times or interactive query response times, we often desire to characterize the distribution of these values to understand the impact on users, for example, the median, 5th, and 95th percentile values.  These are called quantiles. The quantile sketch provides approximate answers to these queries with a well understood error bound that is independent of the distribution of the input values. This will become available soon.
  

• Frequent Items. Given a large stream of identifiers with many duplicates, we would like to know which identifiers occurred most frequently. This will become available soon.
  

• MurmurHash3.  In addition to the above sketches is a fast, extended version of Austin Appleby’s MurmurHash3 hash algorithm that can also accessed with a Pig UDF.

• Memory Package. A flexible, general purpose Memory Package for managing native memory data structures from within Java.

• Science. The core science behind this library is documented in A Framework for Estimating Stream Expression Cardinalities by Anirban Dasgupta, Kevin Lang, Lee Rhodes and Justin Thaler.  This paper has been accepted for presentation and publication at ICDT 2016. A pre-publication version of the paper is available at the Cornell University Archive.
  

Our Experience at Yahoo

Our experience at Yahoo in using this library has had a profound impact on a number of our internal platforms that must deal with massive data. Processing times for distinct count operations have been reduced by orders-of-magnitude. The mergeability and additivity property of sketches has enabled the simplification of complex reporting systems that had many thousands of process steps down to a few dozen. And recently, Yahoo made available real-time user count metrics for Flurry that enabled mobile app developers to view the number of distinct users visiting their application within 15 seconds of real-time. All of this has been made possible with the DataSketches Library.

flurrymobile:

By: Jarah Euston, VP of Growth

As Flurry from Yahoo reported last year, the proliferation of shopping apps on our mobile devices means every day can be Black Friday. Consumers with a mall in their pockets at all times don’t need to wait for the store to open. In fact, the recent IBM report noted that for the first time in 2015, mobile Black Friday retail traffic surpassed that of desktop at nearly 60% of total. Sales on Cyber Monday beat expectations, according to Adobe, coming in at over $3 billion for the day; 16% more than last year.

These are astounding growth figures for what is surely the world’s most mature consumer market. But what this data misses is the bridge that mobile apps have built between in-store and online commerce during the holiday shopping season.  

Black Friday and Cyber Monday: A Bridge Too Far?

To kick it off, we examined the session activity around Thanksgiving, Black Friday and Cyber Monday compared to the activity of the prior week. In 2015, week-over-week session growth on the day before Thanksgiving and Thanksgiving Day was down compared to prior years; implying that the jump on mobile holiday shopping wasn’t as strong as previous years. Black Friday saw a similar spike in mobile app activity from the last two years with a 24% increase. This is in line with the data from Adobe and IBM, which noted a 30% increase in mobile Black Friday sales in 2015.

For comparison, week-over-week app activity over the last 30 days is typically around -1% to 1%.

And on the Seventh Day, They Compared Prices

On Saturday and Sunday Americans rested and window shopped, all in preparation for Cyber Monday. Shopping app activity spiked this year the Monday after Thanksgiving with a 16% increase in sessions. This increase was nearly 2x the increase in 2014. What’s more, the 4% spike in activity this year suggests consumers are adapting to the  “Cyber Week” promoted by retailers. If Black Friday is the biggest day for brick and mortar stores and Cyber Monday is the online equivalent, the weekend in between seems to belong to apps.  

Always-On Shopping

Cyber Monday started a decade ago, after Americans returned to their office Internet connection, focused on buying all the items they saw in stores over the Holiday weekend. What we’re seeing in the smartphone era is the holiday shopping season is starting earlier, and extending later. Commerce is finally in our pockets, and peak shopping days are blurring into every day.

yahoo:

By Jeff Bonforte, SVP of Product & Engineering for Communication Products

We’re unveiling the next generation of the venerable Yahoo Messenger, which offers a vibrant new messaging experience, built from the ground up. The powerful new platform integrates the best of the Flickr, Tumblr and Xobni platforms – built to serve you, for years to come.

The new Yahoo Messenger was created with group messaging in mind from the start. We’ve made sharing, “unsending” and “liking” messages, photos and animated GIFs easy and insanely fast – unlike anything you’ve experienced before.

The new Yahoo Messenger is available globally today on iOS, Android, on the Web and in Yahoo Mail on the desktop. It has a few superpowers that we think you’ll love once you try them. Let’s jump into the details.

Photos, Photos, Photos

Drawing from Flickr’s expertise, the new Yahoo Messenger really takes the beauty of sharing photos to a whole new level. You can literally send hundreds of photos at a time and they appear to everyone in the conversation almost instantly.

Not only is the quality of the photos amazing when viewing on Yahoo Messenger, but you can download the original in full quality. And you don’t need to worry about limited photo storage, because the app loads images just as (actually, just before) you need them. It’s like magic.

Love It or Unsend It

Today’s messaging services are either ephemeral or permanent. The new Yahoo Messenger is both. You can “unsend” any message, photo or GIF, and it will vanish instantly from the conversation (not only from your view but everyone else’s too). Poof!

Of course, at the other side of the spectrum is love. Show some love with a “like” on any comment, photo or GIF. Unsend and “like” are not only good for 1:1 conversations, but they are also invaluable for groups. For example, at Yahoo, when we’re deciding where to meet for lunch, people throw out options on Yahoo Messenger and the group votes on their favorite using “like.”

GIFs Are The New Emoji

Before emojis there was that awkward moment in messaging where you were unsure what the other person really meant. Then came emojis, which helped you to express yourself. But why limit your reaction to an emoji? Everyone loves a GIF!

Now you can search and find the perfect GIF in the new Yahoo Messenger app. Get instant access to a virtually unlimited and ever-growing library of GIFs pulled from the Tumblr community, where tomorrow’s most popular GIFs are born each day.

Smart Contacts

Like Yahoo Mail, the new Yahoo Messenger has a brilliant smart contacts system behind the scene, powered by Xobni’s industry-defining platform. This is critical, because even more than email, messaging is all about the “who” in your life.

For example, our smart contacts system doesn’t just understand who you know, it also understands the relationship between your contacts, so adding people to the group is really fast and simple. Also, anyone in a group can add another participant, change the name of the group or even change the group photo.

Really Fast…Seriously

Speed with communications is critical and the new Yahoo Messenger sets the bar in speed. In an area like photo sharing, it is almost inconceivably fast. Syncing conversations between Yahoo Messenger on the Web, in Yahoo Mail and your smartphone happens almost in real-time.

And if you’re offline or have low connectivity (say when you’re flying or in a remote area), you can continue to use the app and send messages or photos. When you’re connected, Yahoo Messenger automatically sends them through without you having to resend or refresh.

And More To Come

Today’s launch is just the beginning. We’re going to continue to work hard to build the best messaging product that you won’t be able to live without!

Give it a try and let us know what you think. Still using our old Yahoo Messenger? Read more here.

imbrianj:

When we’re building products at Yahoo we get really excited about our work. No surprise. We envision that millions of people are going to love our products and be absolutely delighted when using them.

With our new Yahoo Sports Daily Fantasy game, we wanted to include everyone.

We support all major modern browsers on desktop and mobile as well as native apps. However, that, in and of itself, won’t ensure that the billion individuals around the world who use assistive technology will be able to manage and play our fantasy games. One billion. That’s a lot of everyone.

Daily Fantasy baked in accessibility. Baked in. Important point. In order to ensure that everyone is able to compete in our games at the same level, accessibility can’t be an add-on.

Check out our pages. Title and ARIA attributes. Structured headers. Brilliant labels. TabIndex and other attributes that are convenience features for many of us and a necessity for a great experience for others—especially our assistive technology users. There are a lot of them and if we work to make our pages and apps accessible, well, we figure, there can be a lot more of them using Daily Fantasy.

Think about it: whether you’re a sighted user and just need to hover over an icon to get the full description of what it indicates—or a totally blind user who would otherwise miss that valuable content—it makes sense to work on making our game as enjoyable and as easy to use as possible for everyone.

So, the technical bits. What specific things did we do to ensure good accessibility on Daily Fantasy?

A properly accessible site starts on a foundation of good, semantic markup. We work to ensure that content is presented in the markup in the order that makes the most sense, then worry about how to style it to look as we desire. The markup we choose is also important: while <div> and <span> are handy wrappers, we try to make sure the context is appropriate. Should this player info be a <dl>? Should this alert be a <p>?

One of the biggest impacts to screen readers is the appropriate use of header tags and well-written labels. With these a user can quickly navigate to the appropriate part of the page based on the headers presented—allowing them to skip some of the navigation stuff that sighted users take for granted—and know exactly what they can do when, for example, they need to subtract or add a player to their roster. When content changes, we make use of ARIA attributes. With a single-page web app (that does not do a page refresh as you navigate) we make use of ARIA’s role=“alert” to give a cue to users what change has occurred. Similarly, we’ve tried to ensure some components, such as our tab selectors and sliders, are compatible and present information that is as helpful as possible. With our scrolling table headers, we had to use ARIA to “ignore” them, as it’d be redundant for screen readers as the natural <th> elements were intentionally left in place but visibly hidden.

Although we have done some testing with OSX and VoiceOver, our primary testing platform is NVDA on Windows using Chrome. NVDA’s support has been good - and, it’s free and open source. Even if you’re on OSX, you can install a free Windows VM for testing thanks to a program Microsoft has set up (thank you!). These free tools make it so anyone is able to ensure a great experience for all users:

• https://dev.modern.ie/tools/vms/mac/
• https://www.virtualbox.org/wiki/Downloads
• http://www.nvaccess.org/download/
• http://www.google.com/chrome/

Accessibility should not be considered a competitive advantage. It’s something everyone should strive for and something we should all be supporting. If you’re interested in participating in the conversation, give us a tweet, reblog, join in the forum conversations or drop us a line! We share your love of Daily Fantasy games and want to make sure everyone’s invited.

If you have a suggestion on what could improve our product, please let us know! For Daily Fantasy we personally lurk in some of the more popular forums and have gotten some really great feedback from their users. It’s not uncommon to read a comment and have a fix in to address it within hours.

Did I mention that we are excited about our work and delighting users—everyone?

- Gary, Darren and Brian

yahoo:

By Simon Khalaf, SVP, Product & Engineering, Publisher Products

For nearly two decades, Yahoo Sports has entertained fans around the world with the most engaging content, news and fantasy experiences.  Our millions of fantasy users are the most passionate sports fans out there, spending nearly 30 billion minutes a year playing season long fantasy sports on Yahoo.  Today, we’re taking the game to the next level and giving you what you’ve always wanted - the chance to be a cash winner every single day with the new Yahoo Sports Daily Fantasy.

If you like our full season fantasy leagues, you’re going to love the opportunity to compete with your friends everyday with one-day and weeklong contests. Whether you’re a seasoned Fantasy player or new to the game, Daily Fantasy is easy to play within the Yahoo Fantasy app on iPhone or on any device through your browser.  You can set your lineup like a pro with a simplified salary cap and ready access to real-time sports news and scores from Yahoo’s fantasy experts, all within the experience.

UP YOUR GAME! Go to Yahoo Sports Daily Fantasy and update or download your Fantasy Sports App from the App Store to play today. Game on! 

yahoo-celebrity-canada:

Robin Thicke and April Love Geary in Cannes (David M. Benett/Getty Images)

Mallory Schlossberg, Yahoo Celebrity 

 Robin Thicke must have been crooning seductive ballads into the universe, because it looks like the romance gods have blessed him with a new relationship. Thicke, 38, showed up on Wednesday night to a yacht party during the Cannes Film Festival with his much younger girlfriend, 20-year-old model April Love Geary, essentially putting their relationship out in the open. This is for real, folks.

According to People, the two also cavorted at the Grisogono bash at the Hôtel du Cap on Tuesday. Thicke reportedly performed there, too.

While the two had been spotted out and about together before — and the paparazzi caught hervacationing with Thicke and his 5-year-old son, Julian — Thicke is now putting their relationship on display by choice by happily posing for the cameras, rather than being subjected to the paparazzi’s shots. He really has an official lady now.

Robin Thicke and April Love Geary get their Cannes party on (Splash News)

This public display of togetherness also suggests he’s putting his relationship with Paula Patton behind him. Thicke and Patton parted ways last year, and Thicke made a very public and forlorn plea to win her back with his album Paula. That attempt ultimately failed; Patton filed for divorce in October 2014, and their divorce was finalized in March.

But it looks like he’s moved on and is happier now, at least.

(via yahoocelebrity-ca)