It almost seems like the big BI companies figured out they had to be a part of something even bigger to justify their already sky-high license costs. Just within last 12 months, we’ve seen Hyperion get bought Oracle, Business Objects get bought by SAP, and now today Cognos got bought by IBM.

And I can’t help but feel that Enterprise BI software has become a dying breed.

If you are a fortune 1000 type of organization, chances are you already have some sort of elaborate licensing agreement with at least one of these acquirers (Oracle, SAP, or IBM). So now, they have one more thing to sell you so that you can have a check mark for your BI initiatives without having to drive too far to a different vendor.

I guess I shouldn’t be so cynical. As BI software, Hyperion, Business Objects, and Cognos are pretty feature-rich, and have many compelling qualities. My problem is with their complex enterprise deployment model, proprietary nature, and staggering cost of ownership that continues standing in the way of making BI available to the masses. And their recent acquisitions further polarizes the world of BI haves and have-nots.

Perhaps this is how corporate world will come to demand a new breed of BI solutions. Back in 2002, when we started our marketing analytics company, we thought (and still do) open source and software-as-a-service were going to drive this new trend. And it’s good to see companies like JasperSoft, Pentaho, Swivel, and LucidEra lead the way in this direction. What’s more important than just looking for open source and/or SaaS solutions to BI, is to demand “openness” — openness in terms of architecture, sharing of insights, and just as importantly, openness about cost of ownership. How many times you have seen a BI initiative where the entire budget got spent just on data integration and/or cleansing? How about the cost of those special consultants who seemed to be the only earthlings that understood the esoteric workings of a popular yet proprietary BI software package?

Above all, one needs to keep in mind that BI is about leveraging all available data to get a clearer picture of what’s going on in the business, be able to focus on the most relevant issues, and make better decisions. Instead of taking a centralized approach of one system doing it all, a good BI system today needs to act more like a network that can connect to various data sources, systems, API’s, web services, etc. What if your BI system acted more like a mashup that lets you combine compatible information sources and cross-reference them as you please? Swivel has an approach close to this except that it expects all information to be uploaded by its users. It could be interesting if Swivel could connect to some common public data repositories (like geographical locations, weather, stock prices).

The key here is to have BI systems that enable mashup of information resources and analyses, a la web 2.0, as opposed to being traditional “enterprise” solutions that need your business to bend over backwards to fit into their proprietary framework and terminologies. Which to me conjures up images of Oracle, SAP, and IBM.

Maybe I liked them better when they were Hyperion, Business Objects, and Cognos. At least they didn’t say — “BTW, we also do BI”.

I received an email from Seth Grimes, a very familiar voice in the BI community, with an interesting question — Do you now see BI as a commodity market?

He was referring to an earlier post about our open source BI project OpenI, where I’d mentioned:

The state of business intelligence software market has been very much controlled by a few big players. The situation is very similar to how the J2EE application server market was before JBoss, or how database server market used to be before MySQL and Postgres emerged as serious alternatives, or how OS market was before Linux. Pretty soon we will talk about the BI platform market in the same manner, because open source and open standards are driving the commoditization of BI as we speak. It is just a matter of time.

That was a few years ago. So, I asked myself — well, how do I feel now? Have I learned anything?

The question is a tough one — something I’ve always grappled with. Ultimately, it depends on what do we mean by “business intelligence”. If we go by the current big commercial players’ definition — then BI is more about a software tool providing capabilities around data warehousing/ETL, OLAP, analytical modeling, and visualization. So by that account, I’d definitely stick to my original thoughts and say it’s a commodity market.

However, a more relevant question might be — does having these capabilities make a business intelligent? In reality, what I’ve seen is that it comes down to an analyst (or group of analysts) who (a) know how to work a “BI” tool, and (b) have some fundamental expertise in the business domain they are analyzing. So, the “BI” tool is more about facilitating the job of an analyst or a general business user. You could argue that by making performance metrics, etc. more easily accessible to a business user, the BI tool is helping them make more effective decisions, but it is making a big assumption that the user knows how relevant the performance metrics are for the business.

In the end, my take on this is that, the most effective BI tools are domain-centric, i.e. they embody some inherent knowledge about a particular business domain — so, not only they are extremely efficient and accurate about compiling all the performance metrics and making them available, they also “understand” the applicability of those metrics and can almost act like expert systems in guiding crucial decisions. This, I don’t think is a commodity market. It needs to be grounded into specific industry domains to be effective.

Would love to hear your thoughts.

Ask anyone who is involved in an open source project, getting recognized is one of the greatest kicks you get out of the whole deal.

So, when my Google alert picked up this news on OpenI listed in the Top 10 Free BI Apps list, it absolutely made my day. Of course, a lot of the credit goes to all the folks who have contributed to this project, and the open source community that has supported us all this time. And thanks to Tamina Vahidy for recognizing the project.

When we started OpenI back in July 2005, we just wanted to subsidize our R&D. We needed a BI platform to deploy our analytical models, and instead of opting for commercial BI platforms which would have never fit into our cost model, we decided to develop a BI platform using available open source components, and also as an open source project of its own. We figured if we get even a couple of people outside of our company to pitch — whether it was design help, or just thinking through requirements, use cases we hadn’t encountered — that alone would pay for the efforts to make it open source.

Well, not only we got design help and advice from a great deal of smart folks in the space, we even have people contributing code. I remember someone (maybe Steve Weber) making a point about open source development model — not all the smart people in the world work for you, so the only way to get them involved in your projects is via open source (ok, you may argue crowdsourcing ideas such as Netflix’s contest, but I don’t have a $ 1 million to give away in prize money 🙂

So — here we are — working on version 1.3 of the product. We are using it internally as the web front-end of our commercial product. Of course, it has ways to go — but as contribution and recognition keep coming in, it just seems like a much more rewarding way to develop software.

I discovered this on TED talks, and I completely agree that this is a true display of “passion for information visualization”.

The software used for the presentation is at http://www.gapminder.org and is also a google tool (http://tools.google.com/gapminder) It appears to be “bundled” with the global economic data, not sure if there is an open decoupled version that one can point to their data and play around. Although it seems flash-based and pulling data from static data sources (didn’t seem like rdbms, but I could be wrong) — but this would be a great way visualize OLAP data.

What’s interesting is the concept of a “play” button for the time dimension, which makes a great use of animation to see how different quantities (measures) change over time. It also manages the screen real-estate well to put different dimensions on X or Y-axis. But most of all, this truly exemplifies what data visualization is all about — it goes beyond the realm of charts and graphs that take a while to decipher, and rather tells a very clear, compelling, and visual story. Very impressive!

Recently I had a potential partnership discussion to evaluate whether our predictive analytics technology could provide key insights from this potential partner’s (I’ll call them company XYZ) marketing database. Here’s how my conversation went with Mr. X, an exec at company XYZ, (which is a marketing technology company):

Me: “So, what are they key business pain points for your clients that we can analyze?”
Mr. X: “Well, you know, the usual stuff — marketing ROI, cut costs, increase sales, etc.”
Me: “ah.. yes, but can we delve a bit deeper? Where exactly clients’ marketing programs need help?”
Mr. X: “what do you mean?”
Me: “Well, are they more worried about increasing acquisition volume, or is it more about predicting high-LTV customers, or is it more about retention? I’m trying to get a sense of what is their #1 issue?”
Mr. X: “they don’t know.. it’s probably all of that stuff”
Me: “It’s important that we get a sense of priority, because otherwise we are talking about applying analytics without really knowing what we are trying to optimize”
Mr. X: “well, you are the analytics expert — you need to tell them what to analyze. They don’t think like you, worrying about success metrics, etc. They ask us to run marketing programs, and now we’d like to sell them some analytics. I can tell you what data we have on their marketing programs, now you tell me what kind of analytics you can provide me that I can sell.”
Me: “But we do need to understand their business objectives before determining what analytics is relevant enough so that they’ll pay for it”
Mr. X: “What I need from you is some screenshots — some charts and graphs that show what kind of analytics you can do — I’ll be more than happy to review that and tell you if we can work together”

Needless to say, I didn’t sense a true spirit of partnership here, but I did sense an attitude that I find more often than I’d like that analytics is all about producing charts and graphs that the user will somehow find useful.

Which, IMHO, is total BS!

But I can’t blame Mr. X too much because this is a pretty common perception of analytics in the marketplace. Recently I talked to a marketing exec who said –“everytime I meet with the analytics guys from our agency, they basically have this big ream of a powerpoint deck filled with one chart after another — and I don’t want to see all that stuff; all I want them to do is to tell me what relevant insight(s) did they find, and what course of marketing action would they recommend, and it’s like pulling teeth to get them to move beyond charts and graphs and talk about action.”

Look at the websites of any business intelligence software provider, or analytical software provider — and I will guarantee you that you will see a bunch of fancy charts and graphs, and dashboards with enough dials and speedometers to make you dizzy. Somewhere along the line, maybe we have forgotten that the purpose of analytics is to equip us with insights that enable better decision making.

So, first off, the type of analysis being done has to be aware of what type of decisions we are exactly expecting to improve; and second, the result of the analysis needs to be presented in a fashion that is “integrated” into the decision making. Maybe you list your recommended actions next to your charts and graphs, or maybe you somehow highlight the figures and trends that demand attention. My point — don’t leave it upto your user figure out the action based on the fancy charts and graphs, find out what decisions users are trying to make, and provide information that fills in that gap between analytics and actionable insight.

Yesterday during a call with a potential client, this topic came up. This is a well-known desktop software company, and they have a unique challenge: Their primary measure of customer loyalty is the “Reichheld Score” aka the Net Promoter Score, which is based on customer responses to a single question — “Would you recommend us to a friend?”. Now, the interesting thing is that while this company is doing rather well in the market, their overall net promoter score is not that great. In fact, their score is lagging behind other comparable software manufacturers.

So, the obvious question is — why doesn’t their net promoter score correlate with company growth? Which metric should they rather measure as a driver of company growth?
As we talked, I also found out that this company hasn’t done much in terms of evaluating the behavioral data on their customers — you know, stuff like actual purchase transactions, new purchases versus repeat purchases, customer complaints, returns, etc. And I couldn’t help but think that perhaps this is where they should start look first. A great deal of research work has shown us that past behavior is the best predicator of future behavior, and this is true when it comes to measuring customer loyalty as well. A 2002 HBR article “mismanagement of customer loyalty” describes this as:

Simply put: Not all loyal customers are profitable, and not all profitable customers are loyal. Traditional tools for segmenting customers do a poor job of identifying that latter group, causing companies to chase expensively after initially profitable customers who hold little promise of future profits. The authors suggest an alternative approach, based on well-established “event-history modeling” techniques, that more accurately predicts future buying probabilities. Armed with such a tool, marketers can correctly identify which customers belong in which category and market accordingly.

So why isn’t this company looking at behavioral data on its coustomers? I didn’t get a clear answer, but could it be that it is easier to conduct surveys rather than dig deep into data, specially when the data volumes are huge and the data is scattered around different corporate silos? Could this company be viewing the analysis of behavioral data as a long, complex exercise that involves getting down and dirty with data warehouses and analytical modeles, when all they needed was a nice simple metric that would measure customer loyalty and nicely correlate with company growth?

Yes, I am being a bit facetious — but while I don’t dispute the value of qualitative research, I think in this case, they are more applicable AFTER an initial study of behavioral data. This company needs to understand the “WHAT” first (i.e. what is going on with my customers? which customer attributes/behavior are best indicators of company growth?), and then they can apply qualitative reserach to understnd the “WHY”, i.e. why things are happening the way they are.