AtScale Blog

In the world of Business Intelligence and Big Data there continue to be a number of exciting innovations as new and improved options for processing large data sets appear on the market.  You may be familiar with AtScale’s BI-on-Hadoop Benchmarks - where we focus on evaluating the top SQL-on-Hadoop engines and their fitness to support traditional BI-style queries.  As we continue to work with customers who are navigating their journey to BI on Big Data, we are increasingly getting questions about the emerging cloud-based data processing engines.

 In this blog post, we will take a deeper look at BigQuery from Google, and how it stacks up in the BI-on-Big Data ecosystem.

I’ve asked it before and I’ll ask it again. Wouldn’t it be great if you could easily analyze ALL your data from a Excel single file? We all know this isn’t feasible; especially when dealing with big data and complex business analytics needs.

In working at the intersection of Big Data and traditional Business Intelligence, the AtScale team has encountered a number of complex business analytics use cases that are difficult, if not near-impossible, to solve using typical table-based data models and SQL. Today, I’m going to share why and how complex analysis, like for multi-level metrics, is no longer as ‘difficult’ nor ‘near-impossible’ as it once was.

Wouldn’t it be great if you could load all of your data from a single file into an Excel pivot table for easy analysis? 

Unfortunately, this approach isn’t usually viable when dealing with complex business analytics and big data.  Take for example a typical use case found inthe world of healthcare insurance.  A large insurance provider has 10s of millions of members, and processes 100s of millions of claims a year.  As flexible as Excel is, we all know it won’t handle this volume or velocity of data. 

As a result, more and more enterprises store  large data sets in big data platforms like Hadoop.  And while Hadoop provides a low-cost and performant approach to store and process this information, there is still the challenge of supporting the many types of analytics required on claims and member data sets.  But why? Why and how, with all of the advances in technology, can a simple calculation cause so much complexity?

Digital transformation is a broad term that has various meanings by application, but in general, it means that more and more of what organizations, people, governments do is happening in computers, mobile devices and networks. As a result, the way things are done is changing, especially in the way things are connected. So in this new world of data flying everywhere, being generated and consumed, where does one stop for a second to take a look at what’s going on?

Since the 1980s, the world has been using OLAP technology to provide a business interface to analyze data stored in traditional ERP and CRM systems. As the demand for insights increased, MOLAP and ROLAP became key technologies.

With all of the different OLAP options out there, you may wonder which one can actually help you achieve your big data strategy. Which strategy is most suitable for your Hadoop environment?