AtScale Blog

The growing popularity of big data analytics coupled with the adoption of technologies like Spark and Hadoop have allowed enterprises to collect an ever increasing amount of data - in terms of breadth and volume.  At the same time, the need for traditional business analysis of these data sets using widely adopted tools like Microsoft Excel, Tableau, and Qlik still remains.  Historically data is provided to these visualization front ends using OLAP interfaces and data structures. OLAP makes the data easy for business users to consume, and offers interactive performance for the types of queries that the business intelligence (BI) tools generate. 

However, as data volumes explode, reaching hundreds of terabytes or even petabytes of data, traditional OLAP servers have a hard time scaling.  To surmount this modern data challenge, many leading enterprises are now in search of the next generation of business intelligence capabilities, falling into the category of scale-out BI.  In this blog I'll  share how you can leverage the familiar interface and performance of an OLAP server while scaling out to the largest of data sets. 

And if you don't have time to read the whole thing, don't miss the 10-minute 'cliff-note' video of scale-out BI on Hadoop near the end.

As more and more enterprises adopt Hadoop as their next generation data platform, the demands of traditional enterprise workloads, including support for Business Intelligence use cases, are creating challenges.  While Hadoop excels at low-cost distributed storage and parallel data processing, interactive support for BI-style queries remains a challenge.  Additionally, multi-dimensional queries often demand complex OLAP-style calculations and functions.  In this post we will share how AtScale helps to bridge the gap between Business Intelligence users and data that resides in Hadoop.

In many typical business analyses or applications it is important to be able to directly query the first or last value of a particular metric across a hierarchy.  For example:

• What was the starting or ending price of a security during a particular day
• What were inventory levels for a SKU at the beginning and end of the month
• What was the first and last payment amount for a loan agreement

Not Always as Easy as it Sounds

Executing such a query using SQL may involve complex queries consisting of unions, sub-queries, and/or temporary tables.  In MDX (Multidimensional Expression Language) such a query is easier to support, given MDX’s rich support for analytical queries and hierarchical representation.  AtScale has implemented support for First Child and Last Child measures in a way that supports BOTH SQL and MDX clients, which means that virtually any data visualization client can take advantage of this advanced functionality.

Trystan here, Software Engineer and doer of all things technical at AtScale.  Which SQL-on-Hadoop engine performs best?  We get this question all the time!

We looked around and found that no one had done a complete and impartial benchmark test of real-life workloads across multiple SQL-on-Hadoop engines (Impala, Spark, Hive...etc).

So, we decided to put our enterprise experience to work and deliver the world's first BI-on-Hadoop performance benchmark.  

What did we find out?  Well, turns out that the right question to ask is: "Which engine performs best for Which query type?".  We looked across three of the most common types of BI queries and found that each engine had a particular niche.  Bottom line: One Engine does NOT fit all.

Read on to find out the details of our environment and configuration, the types of queries we tested... (or download the full whitepaper here)

Yesterday, Gartner published the 2016 Magic Quadrant for Business Intelligence.  The MQ research for BI has been in existence for close to a decade. It is THE document of reference for buyers of Business Intelligence technology.

Every once in awhile, our team gets questions about the validity of Hadoop.  Why it exists, why people should consider using it, etc.   In the below video, I provide a few examples of cases across common industries like financial services or retail that provide a clear answer to this question.  

As usual, the video lasts less than 2 minutes so don't expect to have to sit through long, esoteric explanations.  We get to the point and very quickly at AtScale!  

If your team has been trying to connect your Business Intelligence (BI) tools to your Hadoop environment, you're familiar with the typical issues: performance, security and the inability to model the data in a way that business users like to consume it.

Just last week Cloudera released some impressive performance numbers showing how the Impala SQL-on-Hadoop engine scales to support concurrent query workloads. The Cloudera blog post confirms what we at AtScale have experienced with real-world customer installations – that Impala plus AtScale is a scalable solution for running concurrent, interactive business intelligence (BI) queries on Hadoop.

Big data can translate to big wins for your company, but making it work means working smarter. Hadoop makes it simple to distribute storage and process very large data sets. Make Hadoop work for you even further by pairing it with your BI tools like Tableau, Excel and Qlik. Read on to understand best practices of BI on Hadoop with the following do’s and don’t’s.

Like many people these days, I really love data. I started my career in the 90s evangelizing business intelligence (BI) and invented one of the first BI platforms for the enterprise. For the past 10 years, at companies like Yahoo! and Klout, I’ve been a consumer of other business intelligence platforms and I have been extremely underwhelmed and disappointed with the options, especially as they relate to Hadoop.