Skip to main content

Microsoft BI Vision: Excel = BI Democracy?

I frequently check out Mosha Pasumansky’s blog on OLAP. I learned from his recent post that Microsoft announced 2 very interesting milestones coming up on their BI roadmap – project Gemini (which will be the new incarnation of Analysis Services), and SQL Kilimanjaro release (a move toward column-oriented architecture). If not anything else, check out the presentation video from the 1 hour 16 minute mark — this is a pretty good presentation of common BI challenges at BI level, and I must say the demo is impressive in how Microsoft is thinking about the BI solution stack going end-to-end from data warehouse to Excel and to a web-based view for general interaction.  

As someone who is in the “open source BI land”, I must confess that I am a fan of some of the Microsoft’s BI technologies – namely Analysis Services and SQL Server. Yes, I have reservations around Reporting Services, or embedding BI into MS Office products like Word, or about bloating a solution with SharePoint and PerformancePoint – but as a common denominator, SQL Server and Analysis Services do provide the best price-performance today for a BI backend solution IMHO.

One of the challenges we constantly face as BI solution providers is to call out which is the most common interface for the BI user. In their demo, Ted Kummer and Donald Farmer are right to point out that if left to their own devices, most people trying to do a data analysis will bust out Excel. Solution providers like me don’t like this fact for several reasons (a lot of them may be valid) and try to guide our users towards purely web-based interface to do their analysis. The biggest rationale for this is to avoid various versions of Excel files floating around with multiple copies of data and custom calculations (with no QA) — and so we like to control by having all BI users access data from a centralized web interface, which is reporting data from a centralized repository — and that way, we know that users will be guaranteed a “single version of truth”, and they will be happy.

Is that true though?

In my own open source BI project OpenI, one of the most used feature turns out to be “export to Excel”, so try as we may, there are valid reasons to cater towards a BI user’s natural flow of anlayzing data, and let them get their data into Excel.

And in that sense, Microsoft’s approach may have its merit in looking at Excel as the piece in the front and center for self service BI. Of course, calling it “democratization” maybe far fetched because this democracy will only be true in the Microsoft Office world, but it is a pretty big world of BI users. And for those who would like to stay far away from the Microsoft Office world, there needs to be equally compelling alternate solutions (open source or not).

If not anything, this thinking from Microsoft is worth for all BI practitioners to consider — and see the demo. We may not agree with the exact tools used, but the use case, or the scenario, of a knowledge worker finding the data/information they need, analyzing it in an intuitive fashion, and publishing it for their peers to see — that’s a key part of what we’re all trying to solve. And unless we make it utterly easy and painless, we still have a long way to go.

Polarization of BI Solutions

It almost seems like the big BI companies figured out they had to be a part of something even bigger to justify their already sky-high license costs. Just within last 12 months, we’ve seen Hyperion get bought Oracle, Business Objects get bought by SAP, and now today Cognos got bought by IBM.

And I can’t help but feel that Enterprise BI software has become a dying breed.

If you are a fortune 1000 type of organization, chances are you already have some sort of elaborate licensing agreement with at least one of these acquirers (Oracle, SAP, or IBM). So now, they have one more thing to sell you so that you can have a check mark for your BI initiatives without having to drive too far to a different vendor.

I guess I shouldn’t be so cynical. As BI software, Hyperion, Business Objects, and Cognos are pretty feature-rich, and have many compelling qualities. My problem is with their complex enterprise deployment model, proprietary nature, and staggering cost of ownership that continues standing in the way of making BI available to the masses. And their recent acquisitions further polarizes the world of BI haves and have-nots.

Perhaps this is how corporate world will come to demand a new breed of BI solutions. Back in 2002, when we started our marketing analytics company, we thought (and still do) open source and software-as-a-service were going to drive this new trend. And it’s good to see companies like JasperSoft, Pentaho, Swivel, and LucidEra lead the way in this direction. What’s more important than just looking for open source and/or SaaS solutions to BI, is to demand “openness” — openness in terms of architecture, sharing of insights, and just as importantly, openness about cost of ownership. How many times you have seen a BI initiative where the entire budget got spent just on data integration and/or cleansing? How about the cost of those special consultants who seemed to be the only earthlings that understood the esoteric workings of a popular yet proprietary BI software package?

Above all, one needs to keep in mind that BI is about leveraging all available data to get a clearer picture of what’s going on in the business, be able to focus on the most relevant issues, and make better decisions. Instead of taking a centralized approach of one system doing it all, a good BI system today needs to act more like a network that can connect to various data sources, systems, API’s, web services, etc. What if your BI system acted more like a mashup that lets you combine compatible information sources and cross-reference them as you please? Swivel has an approach close to this except that it expects all information to be uploaded by its users. It could be interesting if Swivel could connect to some common public data repositories (like geographical locations, weather, stock prices).

The key here is to have BI systems that enable mashup of information resources and analyses, a la web 2.0, as opposed to being traditional “enterprise” solutions that need your business to bend over backwards to fit into their proprietary framework and terminologies. Which to me conjures up images of Oracle, SAP, and IBM.

Maybe I liked them better when they were Hyperion, Business Objects, and Cognos. At least they didn’t say — “BTW, we also do BI”.

Commoditization of Business Intelligence

I received an email from Seth Grimes, a very familiar voice in the BI community, with an interesting question — Do you now see BI as a commodity market?

He was referring to an earlier post about our open source BI project OpenI, where I’d mentioned:

The state of business intelligence software market has been very much controlled by a few big players. The situation is very similar to how the J2EE application server market was before JBoss, or how database server market used to be before MySQL and Postgres emerged as serious alternatives, or how OS market was before Linux. Pretty soon we will talk about the BI platform market in the same manner, because open source and open standards are driving the commoditization of BI as we speak. It is just a matter of time.

That was a few years ago. So, I asked myself — well, how do I feel now? Have I learned anything?

The question is a tough one — something I’ve always grappled with. Ultimately, it depends on what do we mean by “business intelligence”. If we go by the current big commercial players’ definition — then BI is more about a software tool providing capabilities around data warehousing/ETL, OLAP, analytical modeling, and visualization. So by that account, I’d definitely stick to my original thoughts and say it’s a commodity market.

However, a more relevant question might be — does having these capabilities make a business intelligent? In reality, what I’ve seen is that it comes down to an analyst (or group of analysts) who (a) know how to work a “BI” tool, and (b) have some fundamental expertise in the business domain they are analyzing. So, the “BI” tool is more about facilitating the job of an analyst or a general business user. You could argue that by making performance metrics, etc. more easily accessible to a business user, the BI tool is helping them make more effective decisions, but it is making a big assumption that the user knows how relevant the performance metrics are for the business.

In the end, my take on this is that, the most effective BI tools are domain-centric, i.e. they embody some inherent knowledge about a particular business domain — so, not only they are extremely efficient and accurate about compiling all the performance metrics and making them available, they also “understand” the applicability of those metrics and can almost act like expert systems in guiding crucial decisions. This, I don’t think is a commodity market. It needs to be grounded into specific industry domains to be effective.

Would love to hear your thoughts.

OpenI Listed in Top 10 Free BI Apps

Ask anyone who is involved in an open source project, getting recognized is one of the greatest kicks you get out of the whole deal.

So, when my Google alert picked up this news on OpenI listed in the Top 10 Free BI Apps list, it absolutely made my day. Of course, a lot of the credit goes to all the folks who have contributed to this project, and the open source community that has supported us all this time. And thanks to Tamina Vahidy for recognizing the project.

When we started OpenI back in July 2005, we just wanted to subsidize our R&D. We needed a BI platform to deploy our analytical models, and instead of opting for commercial BI platforms which would have never fit into our cost model, we decided to develop a BI platform using available open source components, and also as an open source project of its own. We figured if we get even a couple of people outside of our company to pitch — whether it was design help, or just thinking through requirements, use cases we hadn’t encountered — that alone would pay for the efforts to make it open source.

Well, not only we got design help and advice from a great deal of smart folks in the space, we even have people contributing code. I remember someone (maybe Steve Weber) making a point about open source development model — not all the smart people in the world work for you, so the only way to get them involved in your projects is via open source (ok, you may argue crowdsourcing ideas such as Netflix’s contest, but I don’t have a $ 1 million to give away in prize money 🙂

So — here we are — working on version 1.3 of the product. We are using it internally as the web front-end of our commercial product. Of course, it has ways to go — but as contribution and recognition keep coming in, it just seems like a much more rewarding way to develop software.

Passion for Data Visualization

I discovered this on TED talks, and I completely agree that this is a true display of “passion for information visualization”.

The software used for the presentation is at http://www.gapminder.org and is also a google tool (http://tools.google.com/gapminder) It appears to be “bundled” with the global economic data, not sure if there is an open decoupled version that one can point to their data and play around. Although it seems flash-based and pulling data from static data sources (didn’t seem like rdbms, but I could be wrong) — but this would be a great way visualize OLAP data.

What’s interesting is the concept of a “play” button for the time dimension, which makes a great use of animation to see how different quantities (measures) change over time. It also manages the screen real-estate well to put different dimensions on X or Y-axis. But most of all, this truly exemplifies what data visualization is all about — it goes beyond the realm of charts and graphs that take a while to decipher, and rather tells a very clear, compelling, and visual story. Very impressive!

We’ve got charts and graphs to back us up, so f@#$ off!

Recently I had a potential partnership discussion to evaluate whether our predictive analytics technology could provide key insights from this potential partner’s (I’ll call them company XYZ) marketing database. Here’s how my conversation went with Mr. X, an exec at company XYZ, (which is a marketing technology company):

Me: “So, what are they key business pain points for your clients that we can analyze?”
Mr. X: “Well, you know, the usual stuff — marketing ROI, cut costs, increase sales, etc.”
Me: “ah.. yes, but can we delve a bit deeper? Where exactly clients’ marketing programs need help?”
Mr. X: “what do you mean?”
Me: “Well, are they more worried about increasing acquisition volume, or is it more about predicting high-LTV customers, or is it more about retention? I’m trying to get a sense of what is their #1 issue?”
Mr. X: “they don’t know.. it’s probably all of that stuff”
Me: “It’s important that we get a sense of priority, because otherwise we are talking about applying analytics without really knowing what we are trying to optimize”
Mr. X: “well, you are the analytics expert — you need to tell them what to analyze. They don’t think like you, worrying about success metrics, etc. They ask us to run marketing programs, and now we’d like to sell them some analytics. I can tell you what data we have on their marketing programs, now you tell me what kind of analytics you can provide me that I can sell.”
Me: “But we do need to understand their business objectives before determining what analytics is relevant enough so that they’ll pay for it”
Mr. X: “What I need from you is some screenshots — some charts and graphs that show what kind of analytics you can do — I’ll be more than happy to review that and tell you if we can work together”

Needless to say, I didn’t sense a true spirit of partnership here, but I did sense an attitude that I find more often than I’d like that analytics is all about producing charts and graphs that the user will somehow find useful.

Which, IMHO, is total BS!

But I can’t blame Mr. X too much because this is a pretty common perception of analytics in the marketplace. Recently I talked to a marketing exec who said –“everytime I meet with the analytics guys from our agency, they basically have this big ream of a powerpoint deck filled with one chart after another — and I don’t want to see all that stuff; all I want them to do is to tell me what relevant insight(s) did they find, and what course of marketing action would they recommend, and it’s like pulling teeth to get them to move beyond charts and graphs and talk about action.”

Look at the websites of any business intelligence software provider, or analytical software provider — and I will guarantee you that you will see a bunch of fancy charts and graphs, and dashboards with enough dials and speedometers to make you dizzy. Somewhere along the line, maybe we have forgotten that the purpose of analytics is to equip us with insights that enable better decision making.

So, first off, the type of analysis being done has to be aware of what type of decisions we are exactly expecting to improve; and second, the result of the analysis needs to be presented in a fashion that is “integrated” into the decision making. Maybe you list your recommended actions next to your charts and graphs, or maybe you somehow highlight the figures and trends that demand attention. My point — don’t leave it upto your user figure out the action based on the fancy charts and graphs, find out what decisions users are trying to make, and provide information that fills in that gap between analytics and actionable insight.

Qualitative Data vs Behavioral Data: Who do You Pay Attention To?

Yesterday during a call with a potential client, this topic came up. This is a well-known desktop software company, and they have a unique challenge: Their primary measure of customer loyalty is the “Reichheld Score” aka the Net Promoter Score, which is based on customer responses to a single question — “Would you recommend us to a friend?”. Now, the interesting thing is that while this company is doing rather well in the market, their overall net promoter score is not that great. In fact, their score is lagging behind other comparable software manufacturers.

So, the obvious question is — why doesn’t their net promoter score correlate with company growth? Which metric should they rather measure as a driver of company growth?
As we talked, I also found out that this company hasn’t done much in terms of evaluating the behavioral data on their customers — you know, stuff like actual purchase transactions, new purchases versus repeat purchases, customer complaints, returns, etc. And I couldn’t help but think that perhaps this is where they should start look first. A great deal of research work has shown us that past behavior is the best predicator of future behavior, and this is true when it comes to measuring customer loyalty as well. A 2002 HBR article “mismanagement of customer loyalty” describes this as:

Simply put: Not all loyal customers are profitable, and not all profitable customers are loyal. Traditional tools for segmenting customers do a poor job of identifying that latter group, causing companies to chase expensively after initially profitable customers who hold little promise of future profits. The authors suggest an alternative approach, based on well-established “event-history modeling” techniques, that more accurately predicts future buying probabilities. Armed with such a tool, marketers can correctly identify which customers belong in which category and market accordingly.

So why isn’t this company looking at behavioral data on its coustomers? I didn’t get a clear answer, but could it be that it is easier to conduct surveys rather than dig deep into data, specially when the data volumes are huge and the data is scattered around different corporate silos? Could this company be viewing the analysis of behavioral data as a long, complex exercise that involves getting down and dirty with data warehouses and analytical modeles, when all they needed was a nice simple metric that would measure customer loyalty and nicely correlate with company growth?

Yes, I am being a bit facetious — but while I don’t dispute the value of qualitative research, I think in this case, they are more applicable AFTER an initial study of behavioral data. This company needs to understand the “WHAT” first (i.e. what is going on with my customers? which customer attributes/behavior are best indicators of company growth?), and then they can apply qualitative reserach to understnd the “WHY”, i.e. why things are happening the way they are.