DATABASE MARKETING

Everything I Need to Know I Learned From My Data Warehouse

IN THE PAST FEW YEARS, several people have written books about their belief that everything they need to know they've learned from their cat, or Sunday school, or television. I don't doubt that this is true for them. But for me, everything I need to know I learned from being part of a team that built a 5-plus terabyte customer data warehouse for a large financial institution. It was a long, hard road, but there was success at the end and a lot of learning in between.

In one respect, we were very lucky. The impetus for the data warehouse came from representatives of the three primary functional groups in the organization — finance, risk and marketing — and not from information technology. We decided it was time to use our operational databases to build a central depository of data. With it we could answer questions that would allow us to increase profits, reduce costs and improve productivity. In other words, the “dream” and the “drive” for the warehouse came from involved “businesspeople” who were predisposed to making good use of it. (I really dislike using the term businesspeople, but I need to distinguish those working in marketing, finance, etc. from those working in IT.)

I offer you what we learned from our experience in the hope it may help you in the future.

• Go ahead and dream, but be realistic about the future. Before we actually began to build the data warehouse, we set aside some time to do as much dreaming as possible. We decided to involve all constituencies from the very beginning. We added representatives from operations, customer service, legal, human resources, and, of course, information technology, to the steering committee.

  We then discussed each of these functional areas. After explaining what we were trying to build, we encouraged everyone in the groups to imagine the possibilities. And they did. They dreamed dreams far beyond the existing state of technology — or even the then foreseeable future.

  After those sessions were complete and we'd scoured the available literature and gone to the appropriate conferences, we realized we needed to rein ourselves in and focus on what we actually could do. We talked with people at corporations that had data warehouses and noticed an interesting pattern. When we questioned the IT people, their excitement about their warehouse was boundless — it was the eighth wonder of the world. When we talked to the businesspeople from the same company, it seemed as if they were working with a very different warehouse. It wasn't fast enough; the data model was inadequate; it was too difficult to access. Most of the time, only a few brave, persistent souls were using the warehouses at all.

  Finally, we all agreed on about 40 high-priority business questions that needed answering — everything from “Who are our most profitable customers?” to “What are the best offers to increase sales?”

• Our eyes ARE bigger than our stomachs. Forty was way too high a number. And some of the questions demanded an investment in technology that the corporation was not prepared to make. At meetings our IT partners would shake their heads — not as naysayers, but as the only realists in our group of dreamers.

  We did some quick calculations to estimate probable profit and likely costs for answering each of the proposed questions. Most were quickly discarded. Finally, we agreed on one marketing question: “Which marketing programs are the most profitable?”

• Take care of the small stuff, and the big stuff will take care of itself. As the data warehouse was being built, the team's business members were quite impatient and couldn't always understand why the warehouse was taking so long to complete.

  First, there was the data review — long and tedious to us then, but as we now know, an extremely important activity. It was meticulous. Data had been duplicated across the organization, and we needed to identify the best sources.

  Then there were numerous sessions focused on collecting application and data requirements. For example, the marketing, finance and risk departments all defined an “active account” differently. We had to decide which data field we would use in the warehouse. Our solution took us in a different direction and accommodated all the definitions.

  In addition, there were intensive sessions dedicated to developing the logical and physical data models. And we took our consultants' advice and made sure one of our IT partners was assigned to metadata development and production. (Simply put, metadata acts as a description of other data.)

  All of this took time and a great deal of attention to detail, but it really paid off in the end. The foundation was sound and the warehouse that resulted was well worth the trouble.

• Stop talking and start listening. Many marketing people are talkers — “big picture” people doing their thinking out loud. Many IT people are more introverted — thoughtful and good with details. It's not an easy marriage, but we had to make it work or the warehouse would have been nothing but a dust collector.

  Discussions got to the point where marketers were saying, “They just don't get it. They're trying to throw roadblocks up. They don't really want this to happen.” The IT people were saying, “They just don't get it. They have no idea how long it takes to get things done. If they don't start seeing it our way, the warehouse will never be built.” It was time for drastic measures.

  Sometimes we forget that we're all very different people with very different skills and interests. That's why we work together. We can't accomplish much alone.

  Our egos get in the way and, more significantly, our values. We and our co-workers may use the same words to get our respective points across, and yet we'll each have different intentions.

  While we were building our data warehouse, we needed to learn to listen to one another, to put ourselves in our co-workers' shoes. Marketers not only had to understand more about the technology issues, but also about how their IT partners approached problems and what they valued. On the other hand, IT team members had to understand that the technology was a means to an end and that everything they did was to serve that end goal of a profitable marketing program — not the end goal of building a warehouse.

• If you don't know what you're doing, it's best to admit it and find someone who does. Even now — five years later, in the era of data warehousing, data marts and data stores — it's dangerous to assume that somehow first-time warehouse builders will muddle successfully through all the necessary, unfamiliar tasks. And in many IT departments, “outsourcing” is a dirty word. It must, however, at least be considered as an alternative to building a warehouse “inside.”

  We decided to tackle the problems inherent to building a data warehouse internally, with the advice and aid of experienced consultants. In the long run, it was a very good decision. No short-term cost savings could replace the experience and the learning.

  The business side doesn't think less of its IT partners if they bring in experienced consultants to shore up their knowledge gaps; quite the contrary. If the consultants can truly work as partners with the warehouse team, if the business and IT people can learn from them and make their own contributions, then it's worth the expense — and probably, in the end, much cheaper. A consultant can be especially effective as a neutral third party conducting joint business/IT sessions, for example, to develop the logical data model.

• Baby steps, baby steps… Almost every how-to article on data warehousing says it, and it's true, true, true: Start small with a focused business application. Don't try to swallow the whole thing in one gulp.

  Believe me, our 5-plus terabyte warehouse was very different from what we had thought it would be at the beginning. A warehouse is a dynamic thing: it evolves. We started with a 1% sample of our data. We learned from it and built on it. This “prototyping” has another great advantage in that it provides a proof-of-concept exercise that can be invaluable in convincing the higher-ups for funding.

• Even when we're all grown up, we still need plenty of attention. The day our data warehouse with all the data came online, we celebrated. Everyone breathed easier; the first very small SAS query took only 13 seconds to run. Shortly thereafter, people who had been working on the warehouse, in both the business and IT areas, were assigned to other projects.

  But the need for those people's ongoing involvement did not dissipate. For example, the database administrators continued to learn a great deal about loading time and auditing those data loads — and the necessity for application development didn't cease. We'd focused so much on “the box,” we had neglected some critical software and staffing issues.

• You can't put the genie back in the bottle. The business needs and demands didn't stop: Now there were calls for better tools and more space. Originally, we didn't know how much space we'd need to develop and run our statistical models. We definitely underestimated that need — our plan for the space was far short of the 100 gig required. The good news: Space has gotten much cheaper than it was in the early days of our warehouse.

• You can't get quality cheap. Another requirement we underestimated was staffing. For example, both the business and IT areas needed to hire experienced SAS and SQL programmers. As in any large organization, adding employees was a big issue in itself. In our case, this issue was coupled with the fact that such experience was hard to find and, consequently, didn't come cheap. After much denial, anger, bargaining and depression, the firm finally had no choice but to give in.

  Even with great technology and incredible data, people make the real difference — people to take advantage of the great systems, people to translate data into information and people to turn information into action.

• Don't try to change people; change yourself. In the mid-’90s, visions of the brave new world of data warehousing included happy executives blithely pointing and clicking their way to amazing business decisions.

  Didn't happen.

  We did give the executives tools they could use to slice and dice summarized data. When we showed these tools to them, they were delighted, but most used them infrequently at best.

  The good news was that executives could ask insightful, precise questions. The bad news was that these questions could only be answered by using SAS for queries or building a model — a bit more time-consuming but usually worth the effort.

  So the analytical group had to change. It had to develop processes and views of the data that would allow for quicker, easier access.

• Promises…promises…promises. Several query tool industry vendors promised we'd be able to use their tools and enjoy easy access to the data. I believe they were sincere. Most, however, had never come up against databases as large and complex as ours. Often, sales reps promised capabilities the tools just didn't have.

  The truth is that the perfect one-size-fits-all tool just doesn't exist. What users really wanted was a Vulcan Mind Meld Interface. As one person on our team said, “If the user really knows computers, he won't be happy with any tool. If he knows the data, he doesn't want to change to accommodate the idiosyncrasies of a new tool. If the user is a businessperson, he has the tool but never gets around to using it.”

  Most organizations probably will require more than one tool to get the work done. The tool industry, however, is getting better and better and many useful products can be found for data warehouses — especially those smaller than ours.

• Man plans and God laughs. It took a lot longer to build the data warehouse than we thought it would, and at every turn there was a surprise — a new, seemingly impregnable difficulty to surmount. Yes, we found problems in our legacy systems. Yes, hardware didn't arrive on time. Yes, there were data problems. But we persisted.

• Win-win is the only way. In large organizations, choosing the area that has responsibility for querying the data warehouse can be a very touchy issue. Should it be in marketing or some other business area? Should it be the responsibility of IT? Or should there be a third, independent, intermediary group designed to assist the business in its use of the warehouse?

  It really doesn't matter. What does matter is that the group be relentlessly customer-focused, committed to fulfilling the information needs of the business decision-makers. In order to do this effectively, the group has to be at meetings and discussions of the business issues. It has to be in a true consultative relationship with its business partners — and be accountable to them. The group must bring new users up to speed and be responsible for evaluating new query tools. It should be able to create data marts and data dictionaries, manage space, monitor usage, and make adjustments to the warehouse when necessary. The group has to be so in tune with the business that it can suggest useful queries before its business partners think of them. It has to see the patterns in the data and make innovative suggestions. It needs strong analytical skills and the ability to make recommendations that can be acted upon.

  The line between business analysts and IT analysts is blurring. And that's not bad at all.

Donna Arnold Ialongo is senior consultant, analytical services at Nykamp Consulting Group in Downers Grove, IL.