Data Lake Intelligence with AtScale

In my recent Data Lake 2.0 article I described how the worlds of big data and cloud are coming together to reshape the concept of the data lake. The data lake is an important element of any modern data architecture, and the data lake footprint will continue to expand. However, the data lake investment is only one part of delivering a modern data architecture. At Yahoo!, in addition to building a Hadoop-based data lake, we also needed to solve the problem of connecting traditional business intelligence workloads to this Hadoop data. Although the term “Data Lake” didn’t exist back then, we were solving the problem of: “How can you deliver an interactive BI experience on top of a scale-out Data Lake” - it turns out we were pioneers in delivering Data Lake Intelligence.

Our experiences and learnings from those initial efforts led to the architecture that sits at the core of the AtScale Intelligence Platform. Because AtScale has been built from the ground up to deliver business-friendly insights from the vast amounts of information in data lakes, AtScale has experienced tremendous success and adoption in enterprises ranging from financial services, to retail to digital media. With the release of AtScale 6.5, we’ve continued to build on and expand AtScale’s ability to uniquely deliver on the promise of Data Lake Intelligence. If this sounds like something you might be interested in knowing more about… keep reading!

Additional contribution by: Santanu Chatterjee, Trystan Leftwich, Bryan Naden.

In the previous post we demonstrated how to model percentile estimates and use them in Tableau without moving large amounts of data.  You may ask, "how accurate are the results and how much load is placed on the cluster?".  In this post we discuss the accuracy and scaling properties of the AtScale percentile estimation algorithm.

To learn how to be a data driven orgazation, watch this webinar now!

Additional contribution by: Santanu Chatterjee, Trystan Leftwich, Bryan Naden.

In the previous post, we discussed typical use cases for percentiles and the advantages of percentile estimates.  In this post, we illustrate how to model percentile estimates with AtScale and use them from Tableau.

To learn how to be a data driven orgazation, check out this webinar!

Additional contribution by: Santanu Chatterjee, Trystan Leftwich, Bryan Naden.

A new and powerful method of computing percentile estimates on Big Data is now available to you! By combining the well known t-Digest algorithm with AtScale’s semantic layer and smart aggregation features AtScale addresses gaps in both the Business Intelligence and Big Data landscapes. Most BI tools have features to compute and display various percentiles (i.e. medians, interquartile ranges, etc), but they move data for processing which dramatically limits the size of the analysis.  The Hadoop-based SQL engines (Hive, Impala, Spark) can compute approximate percentiles on large datasets, however these expensive calculations are not aggregated and reused to answer similar queries.  AtScale offers robust percentile estimates that work with AtScale’s semantic layer and aggregate tables to provide fast, accurate, and reusable percentile estimates.  

In this three-part blog series we discuss the benefits of percentile estimates and how to compute them in a Big Data environment.  Subscribe today to learn the best practices of percentile estimation on Big Data and more.  Let's dive right in!

To learn how to be a data driven orgazation, check out this webinar!

Does your decision-making process need an overhaul? In 2015, over 60% of the decisions made by companies were still based on ‘intuition’ or ‘experience’ of their executive team. With the rise of big data, it is imperative that we make use of such a valuable asset to make fact-based decisions and not just provide an opinion. This means that we need to become data-driven, of course this is easier said than done. It requires more than investing on the latest data and analytics software; it also requires cultural and organizational change.

Learn how industry leaders do it, register for our Best Practices Webinar!

We Told You It Would Be Cloudy

You have probably already heard all about it, read all about it and know all about it, but if you are anything like me, (or Einstein) you have already figured out that the more we learn about BI on Big Data the more we realize how little we know. Or, do we?

The Forecast Calls for Cloudy Weather

You don’t have to be a clairvoyant to know that there is an ever-increasing trend in cloud adoption among start-ups and enterprises alike.  In the world of Big Data, the past few years have shown a significant increase in cloud adoption.  While Amazon initially led the way with cloud data products - with Amazon Elastic MapReduce (EMR) for Hadoop and Redshift for data warehousing - the past 12 months have seen new entrants on the scene.

Is it October already?

It’s hard to believe that October is here. It feels like only a few days ago that we released AtScale 5.0 and AtScale 5.5.  Both releases contained a number of great new features that I was excited to share with the Big Data and Business Intelligence communities. Although I may be a little biased, I really do believe that the release of AtScale 6.0 marks one of the biggest releases in the history of the company. 

Despite the challenges associated with data warehousing, enterprise IT leaders have accepted it as a necessary evil of deriving value from information within Hadoop and other Big Data ecosystems. How much does it cost to create data warehouses or datamarts that extract data out of Hadoop? Is there a better way to do BI on Big Data?!

Industry leaders know that their challenges with data analytics are spread across 4 areas: confirmation (‘the things they know they know’); intuition (‘the things they don’t know they know’); inspection (‘the things they know they don’t know’); and revelation (‘the things they don’t know they don’t know). 

This last category is often the most tragic for organizations.  Luckily, we now have helpful resources from Tom Davenport's latest webinar on Data Strategy to key research data points on chief data executive's priorities.