Data Warehousing and Psychology…Who knew???

There is a dark secret about the data warehousing industry that no one ever seems to acknowledge, which I’m going to expose in this section. You often hear of the failure of a data warehouse along with a lot of different technical reasons why it didn’t work, why users found it too hard to use, etc. Surprisingly, even data warehouse implementations that were consider a success from a technical standpoint can be deemed a failure by the business users. Why can that happen? You’re about to find out…

Before we get there, let’s have a discussion about computer systems. In my mind, there are two classifications of computer systems: the must haves and the nice to haves. The must haves represent systems that the business would crumble without. These are systems like financial systems where you enter vendor invoices so that they can be paid, or you enter customer invoices so that the customer can be billed and they can pay you. Without systems like these, it would be very difficult, if not impossible, to run your business efficiently. Some would argue that without these systems, your business would die. The nice to haves are systems that could be considered to be optional. These are systems that if they are not in place, the business will survive. The business may run a bit less efficiently, but there’s not a worry that the business would crumble. In many industries, data warehouses are considered a nice to have. There are a few industries where without them a business would be extremely limited, but not a whole lot of cases where a business would die without one.

In my years at DecisionPoint, I experienced a lot of sales situations where we would get really close to closing a deal, but the customer would lose their budget, the money they had budgeted would go for another IT priority, or something else of that nature. If the customer walked away from us, the business was not going to die. Sure, the company could greatly benefit from the data warehouse, but there was no risk to doing nothing. No money spent, no risk of failure, etc. I think that’s what makes selling data warehouse solutions such a tough business. Anytime the customer has a “do nothing” option, it’s always something to worry about. There has to be a motivator or compelling reason that causes them to not choose the “do nothing” option.

There’s another “do nothing” option that never gets mentioned by IT shops, consultants, or vendors. That “do nothing” option lies on the other side of a successful implementation. Let’s assume that you went through a lot of trouble and hassle, but you got the data warehouse up and running with end users actively using it. You think all is well and good, but there doesn’t seem to be any benefit from the data warehouse. All of the business value (i.e. increased revenue, reduced cost), etc. just doesn’t materialize, and you wonder why…The answer lies below, and involves something that is often referenced in terms of Psychology.

Anyone that has been through or knows about counseling should know the answer to this question. If you’re going to a counselor, when do you start to get better? The answer is you start to get better when you decide to get better. The counselor can talk to you for hours and make many suggestions and recommendations. However, if you don’t decide you want to get better, you won’t try the suggestions or recommendations. The counseling sessions will simply be an hour or so of your time talking to someone. There is no benefit, and you’re probably wasting a lot of money.

Now, let’s spin this in the direction of the data warehouse. Using our scenario above, we’ve done everything that needs to be done and the users are using the data, but nothing changes in the business. It’s quite possible that the data warehouse is providing valuable information, but the users are too afraid or unable to use the data in making changes in the business that will improve revenue or reduce expenses. Just like a counselor, the data can tell you all of these wonderful things about the business, but unless the users are willing to use that information to make changes in the business, nothing will happen. Your data warehouse is just like the counselor…you’ve spent a lot of money, but nothing changes because you haven’t decided that you need to get better.

It amazes me how often this happens, and what a difference it makes when you have end users that are willing, able, and brave enough to do what needs to be done after the data points them in the right direction. Far too often, I have seen data warehouse projects go down in flames or be viewed as wasted money simply because the end users didn’t have the guts to do something, or worse yet, management wouldn’t let them do what needed to be done. Generally speaking, people hate change. A data warehouse is the first step to change. It tells you how things are so you can see where you need to go. Getting there involves change, and in some cases, change represents a bad thing. Don’t rock the boat…you might get in trouble or be fired. Willingness to accept and use the data as a “business weapon” is the only way to succeed.

A couple of examples to make my point… The names are left out to avoid any kind of aggravation I might receive from the people that were involved.

In the early days of DecisionPoint, I was helping a customer in the Northeast United States implement our data warehouse. It was about 8-12 weeks of me flying back and forth across the country to put it in for them. We had some bumps along the way, but after we worked through the bumps, we ran into a brick wall. Everything was looking good, and I was demoing it to an end user and we started to look at some data. We got to one section of the data and she made a comment to me something like this. “You have to turn this off!” My immediate response was to ask why we had to do that after everything we had done. Her response went like this: “Your numbers are correct, but that’s not what the CFO believes the business looks like…” It turns out that they did have some minor problems with some of the transactions where the data came from. That was fixable. What was not fixable was her unwillingness to go to the CFO, present the information, and figure out a way forward. I left that company and have not heard back from them since.

On the other side of the equation, we have two really positive stories to tell.

In our fist example, it was again the early days of DecisionPoint. We were working with a local company implementing our software. It was fun because they were a retailer, and when it comes to change retailers don’t mess around. If they find something that will eliminate costs or increase revenue, they’ll do it…no questions asked. However, it does get a bit cut throat as there is “no mercy” in what they do. They may make a change in one day that puts 100 people out of work, and their attitude is “so be it, we had to do it for the good of the business”. In any event, we were only a couple of weeks into our implementation when we gave the end users their first glimpse of the data. They were focused on the profitability of the stores. They started to look at the unprofitable stores, and guess what? For the ones that were in real trouble, they literally started to shut them down the next day. No questions asked. This was long before they were even actively using the data. They saw a problem. They fixed it. In the end, it did help them save a lot of costs and have a higher profitability than they would have had, so it was a good thing.

Another example at this same company came later on in the implementation. We were extracting vendor invoices, and everything was going smoothly. Then, we got to a point where we were extracting data for the month of December. Suddenly, the vendor invoice volume quadrupled. We were shocked and puzzled, and when we presented it to the end users, they had the same reaction. They were determined to find the source of this phenomenon, and proceeded to do research. To make a long story a bit shorter, here goes. When they would hire a store manager, the manager would get a bonus each month related to the profitability of the store. So, the managers had creatively figured out that if they didn’t turn in their vendor invoices (i.e. expenses), the store was more profitable. So, for the first eleven months of the year, they wouldn’t turn in any expenses. Then, in December, they would send in all of the vendor invoices they had collected over the year. The problem was not only vendors that were upset about not being paid, but it ran deeper. It turns out that the stores were also getting late notices on vendor invoices. So, what happened in December is that the AP clerks would enter the original invoice, then enter the late notices, etc. In some cases the company was double, triple, or even quadruple paying vendors. What seemed to be a very simple problem was costing them a lot of money. Needless to say, the bonus program was changed, and the problem disappeared.

My last example is a story of a user using some data in a way that even I wouldn’t have guessed. This person gets the gold star from me on “data creativity”. At another one of our retail accounts, there was a user in what is called their “Risk Management” department. Part of that department involved insurance policies. This particular business had a lot of delivery vans throughout the US that they were using along with associated insurance policies on each van. This particular person did some analysis using some vendor invoice data that allowed her to compare what they were spending on new tires for the vans in each region of the country, and then compared that to the insurance rates they were paying on each van. It turned out that the regions that spent more money on tires for the vans had vans with fewer accidents, which meant that their insurance rates were lower. Using this information, the user instituted a corporate wide policy on how often the vans would get new tires, so that all regions were doing the same thing. Once she changed the policy, they began to save millions of dollars on their insurance policies. In this case, not only did you have an end user willing to act on the data, but also one creative enough to find out something about the business most people would have overlooked.

OK, so what’s the moral of the story? It’s very simple. When it comes to a data warehouse, the business only improves when the users are willing and able to act on the information that they see. If they are unwilling or unable due to a management policy or something like that, the data warehouse will die a slow painful death. Could it have been useful? Yes. But, potential only goes so far…you have to use the potential before anything changes.