The OLAP Report: How not to buy a BI product

You can contact Nigel Pendse, the author of this section, by e-mail on NigelP@olapreport.com if you have any comments or observations. Last updated on May 11, 2009.

Introduction

BI products differ from each other much more than do, for example, relational databases, programming languages, word processors or presentation graphics packages – which greatly increases the scope for confusion when selecting a BI product. Just to add to the problems, neither IT professionals nor end-users are fully equipped, on their own, to make properly informed BI selections (whereas each of the other products listed above could be chosen by one group without help from the other). This means that, unlike most other software, BI evaluations must involve both users and IT. But in many organizations, IT and end-users have trouble communicating with each other and are often barely on speaking terms. Even when they try to work together, they inevitably have different objectives. Consequently, we frequently come across organizations who have made strange product choices, often because they started with a bizarre shortlist consisting of the contradictory preferences of the technical and business groups.

Rather than preaching to you, this chapter outlines some of the many mistakes that we have found people making, and suggests ways to avoid them. If you follow these tips, you should slash the time spent choosing a BI tool, and greatly increase the chances of the project succeeding. But always remember that choosing the right product is only one of the steps needed for business success with a BI project.

Evidence-based selection criteria

The whole process of choosing BI or any other BI products rationally is not easy, not least because the marketing messages sound much the same for all the products. Every product claims to be innovative, efficient, highly capable and widely used. Disentangling these claims is far from easy.

It may seem tempting to skip the whole phase and just opt for a well-known product from a major vendor or an add-on from an incumbent vendor. The BI Surveys have researched this question extensively and found that about half the sites surveyed had only considered a single product — but the Surveys have also found strong evidence that a competitive product selection process should not be skipped.

The BI Survey 8 includes extensive analyses of data from 2150 BI sites, and among the areas surveyed is the extent to which a range of eight of business benefits were achieved. These included saving costs, improved customer satisfaction, higher revenues, better decisions and improved reporting. A weighted Business Benefit Index (BBI) was calculated depending on the extent to which each of the benefits were achieved, and this index was then used to calibrate every aspect of their BI experience.

One aspect was how products were selected. Only about half the sites had conducted a formal multi-product evaluation, but they went on to realize the highest business benefits; those who had formally evaluated a single product were somewhat lower; while those who had not conducted a formal evaluation at all realized the fewest business benefits. Not only was this true at the overall level, but it also applied to every single one of the eight potential benefits. Clearly, projects are more successful if they include a formal, multi-product evaluation.

The Survey also tracks which selection criteria were used when choosing products, and then measures which criteria were used in projects that were more successful in business terms, as measured by the BBI. As the chart below shows, projects which used products chosen for product rather than commercial reasons were the most successful in business terms. Selection factors such as corporate standard, product reputation, vendor relationship, product bundling and sales skills were the factors with the lowest correlation with subsequent business benefits. Conversely, organizations that selected products using selection criteria like fast query performance, ability to handle large numbers of users and ease of use, achieved the greatest business benefits.

This is strong evidence both that formal, multi-vendor evaluations lead to better projects, and that the selections should be based primarily on product, not company, factors.

Assessing the needs

We find that many BI evaluations end up comparing apples with oranges because the selection team has created a shortlist too quickly, without first understanding the business issues. In particular, buyers often allow vendors to educate them about the market, which is a very dangerous route to follow.

Perhaps surprisingly, the key intellectual effort should be performed before you ever meet a salesperson, because once you’re in the hands of a high-powered sales professional, you will be skillfully guided into wanting whatever it is that their company sells. In fact, in our experience, the overwhelming majority of purchases are driven by the quality of the vendor’s sales and marketing, not the appropriateness of the product.

You cannot even begin to choose products rationally without understanding what they will be used for. This may seem obvious – but we find many companies want to choose ‘the best product’ before knowing how it is to be used. They then start talking to vendors before even knowing if their products are suitable. To avoid this means that, before attempting to create a shortlist, you must:

If BI is new to the organization, it may be hard to do some of these things, so it is probably worth starting with a small scale project using a simple, end-user oriented tool, mainly to gain experience. Users find it a lot easier to define needs after they have used even a basic tool than if they are working in a vacuum.

Remember that any BI purchase should be business-driven: that means that you should have a clearly articulated objective for any project, with specific benefits listed and with at least an approximate rate of return calculation based on hard-nosed considerations. Simply assuming that better information and more reports is a ‘good thing’ in its own right is not sufficient justification for a project. You should identify quantifiable benefits (financial or human) that can be measured before and after the project; if this cannot be done, it is hard to know who should use the system, how detailed it needs to be or what costs are justifiable. After all, no amount of splendid reports will be useful if they do not provide users with actionable information that will let them do their jobs more quickly or more accurately.

This process can often simplify projects, which not only reduces the cost, but greatly increases the chances of success. It also identifies just who is backing the project, and concentrates their minds if they know that the effects will be measured and not just assumed. Sadly, however, The OLAP Survey 2 in 2002 first found that quantified bottom-line benefits are much more rarely achieved that are softer benefits, like improved reporting. Exactly the same result has been found in every subsequent edition, including The BI Survey 8.

Among the issues you should survey before creating a formal shortlist for a large project are:

Beware of biased advice

There is plenty of “free” advice available on BI products, but most of it comes from inherently biased sources:

The press writes about the companies with the most active PR firms, who provide lots of pre-written case studies and interview opportunities.
Hardware vendors may promote “gas guzzlers” (and certainly only products that run on their own platforms).
Database vendors invariably recommend or bundle their own BI product(s).
Consultants, including the Big Five (or ‘Final Four’), usually favor products for which they provide implementation services, and the vendors with whom they have partnerships. They may also be resellers of some products, which they will invariably favor. The BI Surveys have found that implementations led by large, general purpose consultants have a much lower success rate than those led by smaller specialist BI consulting firms.
End-users often prefer to stick with products they have successfully used before, even if those products would not be suitable for the current project.
Industry analysts tend to have likes and dislikes, partly driven by how well vendors brief them (and some ‘analysts’ are influenced more by the consulting, subscription, sponsorship and reprint fees they generate from vendors than by the products themselves). Some industry analysts also focus more on vendor companies, rather than the products themselves, so they may recommend strong vendors even if their BI products are weak.
ERP vendors invariably promote their own (inflexible) analytical add-on applications and reporting tools. These generally have a much lower success rate than tools from independent specialist BI vendors.

Don’t get carried away: overstating the requirements can be almost as bad as understating them, and could needlessly constrain your product choices and will certainly increase the costs. For example, many of the simpler reporting products are perfectly adequate for many applications and should not be mistakenly vetoed because of an unnecessary demand for advanced features. And insisting on zero-footprint Web deployments for intranets will needlessly hamper the performance, functionality and usability of the solution.

Having done this analysis, you should use the detailed product reviews and background analyses in The OLAP Report to produce a shortlist of companies whose products are all likely to be reasonably compatible with your business requirements. Always use the OLAP architectural square to ensure that products on your list are all close to each other. This level of education is essential if you are not to be misled later.

Do not be too concerned if this leads to some relatively small companies appearing on the list. In fact, if the list only consists of the “greatest and the best” companies, it is probably unbalanced, because these companies probably dominate quite different sub-sectors of the BI market and do not sell products that should be compared directly.

Another valuable technique is to focus on the core requirements with the vendors once you have your potential shortlist, particularly if you are struggling to shorten the list. Invite the best technical staff available from the vendors to an in-depth meeting and directly challenge them to describe in detail how they would solve particular problems. Obviously buyers need to be reasonably well educated (or advised by suitably qualified consultants) to know which topics to focus on and to spot where any flaws in the answers might lie; considering the size of the larger BI deals, this is not too much to ask. This technique often lets you weed out unsuitable vendors much earlier than using the traditional scoring systems.

The key thing about a good shortlist is that all the products are suitable for the job, so that even if, like most customers, you simply buy from the best salesperson rather than picking the ideal product, it won’t matter too much because any of the products will be reasonably suitable for the task. If, however, you have a semi-random shortlist (which is what many, many buyers seem to create), then there is a real danger the best sales person may be representing a product that is simply not appropriate for your needs – and you may still end up choosing it. Remember that, however rational you may think you are, you only buy BI products very occasionally, whereas good sales professionals are successfully promoting their own products all the time — so they are trained, very experienced and better than you think at influencing proudly rational people just like yourself.

Is a single corporate standard for BI a good idea?

The process of product selection can be tedious and expensive, so it is very tempting to try and pick a single preferred product set that will be used not just for the immediate project, but will become the official or unofficial standard for subsequent projects for years to come. However, this rarely works and can be dangerous.

Despite looking similar (particularly on the Web), BI products still have very different architectures and functionality and no product is good at everything; most can only do a limited range of applications. For example, some can handle very large data volumes but are uncompetitive with smaller volumes, where they are slow, cumbersome and lacking in function compared to the nimbler products which are aimed at smaller applications. Others can handle large numbers of concurrent users, but are not ideal for smaller groups. Some have good Web facilities but are not suitable for stand-alone use. Others can handle multi-user data updating, for use in planning systems, but they may not be ideal for reporting applications. But the one thing they seem to have in common is that the vendors are all ‘leaders’ and their products are all ‘scalable’ (or so most claim).

Another factor is that not all users have the same aptitudes or needs. Many users normally need little more than pre-defined Web reports, possibly with some basic drill-down and links. Others will want pre-defined reports as starting points, but will then modify them significantly and regularly (which means that a flexible client/server architecture may be preferred). They will also do a fair amount of exploration for themselves, possibly sharing new analyses with colleagues. And a few power users may want to build their own models and use advanced statistics; they will certainly need more than just a browser interface. And there may also be a further large ‘passive’ group of users who only wish to receive reports or notifications (probably by e-mail) if there is something interesting or relevant to look at; they may never use the system between such notifications.

In a large community of users, it is hard for such varying needs to be satisfied with just one product and interface.

If you nevertheless try and pick a single BI product as a corporate standard, many of the subsequent users will be badly served by a product that either cannot meet their needs at all or is too slow/cumbersome/complicated to use. Furthermore, if you have been persuaded to buy more licenses than the initial project required, most of the licenses will end up as ‘shelfware’. This will then encourage you to try to reuse the software in every new project, regardless of whether or not it is suitable.

Another virtual form of corporate standard occurs when users switch jobs and recommend products that they had used successfully in their previous job. If the applications are similar, then this can be a fast track route to success, but often they are not. It is still worth doing at least a quick needs assessment and proof of concept to ensure that the product that was proven in the previous company would be suitable for the differing needs in the new company.

Even if you do not have any formal standards, it is only natural, when selecting a product for a new project, to favor suppliers that are already being used successfully in the company. It should save time, allow the same skills to be used across projects and may save money. There is nothing wrong with this approach if the new project has similar needs to the existing applications, but it can be taken too far, with blameless products being misused in the wrong applications.

Buyers also sometimes wrongly assume a non-existent degree of integration between multiple products from one vendor: very often, products started their lives in different companies that were subsequently acquired or merged, and the so-called integration is rarely more than skin deep. So, even if you are a successful user of one product from a vendor, you should not simply assume that a different product from the same vendor will be compatible with the first one, or that the same skills set will be useful with both. And the second product may be much less competent than the first, so do a proper evaluation before selecting it.

In any case, even if you do succeed in setting a corporate standard, new products will sneak in by the back door anyway: we have never seen a large corporation succeed in preventing multiple BI tools being used, whether or not they had so-called BI standards. For whatever political or rational reason, different departments or subsidiaries will inevitably choose different products. In some cases, a dictatorial attempt to set an enterprise-wide standard has actually led to greater anarchy than if there had been a more tolerant attitude from the central IT group.

The other danger of setting a long-term standard is that it may cause you to miss new-generation products that are faster, more functional, easier to use or cheaper. The BI industry continues to benefit from innovation, usually from younger vendors, and it is unwise to deliberately cut yourself off from such improvements. Even though you would not want to evaluate every new product making big claims, you should be prepared to consider potential winners when you next have a new or replacement project.

Rather than the enforced use of a single product, a much better approach is for the IT group to agree to support a small range of different BI products that collectively span a wide range of applications. Users opting for one of these will get the benefit of additional support, easier data integration, more readily available skills internally and probably lower negotiated software prices. But if users feel that they have a new business need that would not be met by any of the supported products, they should not be prohibited from choosing another (which might, in due course, become a supported product); after all, the ultimate purpose of central support functions like IT is to help the organization achieve its business objectives, not just to dictate and enforce IT policies.

Requests for tenders — how not to do them

Many large organizations have a standard software procurement process that is bureaucratic and slow. It was designed for the purchase of large, operational systems whose characteristics were well known from the start, which is very different to most BI projects. Consequently, the typical tendering process simply does not work properly for BI procurements.

A key stage of this process is the production of a bulky tendering document; they are sometimes known as requests for information or proposals (RFI or RFP). Such documents tend to have dozens – even hundreds – of questions that are a mixture of reused boilerplate questions from previous purchases of completely different types of software, new half-baked questions dreamt up by members of the selection committee to prove that they were awake during the turgid meetings and planted questions from the already favored supplier whose sales team has been busily ‘helping’ key team members ‘understand the issues’. The editing is often rushed, so that questions frequently overlap or contradict each other (eg, by wanting products both to be new and to have a large user base).

Typical tender documents include long lists of desired features, even though no-one may have actually checked if such features are the most appropriate way of delivering the business requirements. Indeed, this is often where a smart salesperson can be most effective, by planting demands for features that their product provides and the expected competitors do not. Other listed features are often ones that were sported by older products, mainly to get over some other weaknesses not present in modern products, which therefore do not need the features. For a supposedly modern product, the presence of such features should be regarded as suspicious, not praiseworthy. Similarly, to dictate product features or architectures is unwise if newer products can achieve the business need in a different, and possibly better, way.

Instead of having a carefully controlled shortlist, the document is often sent out to far too many suppliers, most of whom know that, if they had no hand in setting the questions, they probably have little chance of winning. A few brave ones may refuse to participate, on the grounds that the selection process has been rigged, but most feel obliged to stay in the race, in the hope of turning things round. But they may suspect that they are wasting their time, and often try and sabotage the process, by suggesting new questions be added, or bypassing the selection team.

The selection team invariably hopes to perform the process rationally, by assigning scores and weights to every feature. They may also decide that some features are ‘musts’, and others are ‘wants’. This may or may not be made clear to the vendors, but experienced vendors usually play safe by claiming, on one ground or another, to have every feature requested. Indeed, they may have tendering experts who, without quite lying, find ways of saying ‘Yes’, ‘Unlimited’ or ‘Not Applicable’ to every question. For example, most suppliers will be tempted to claim that a requested feature is available, even if this is only achievable with a laboriously programmed, inefficient and unmaintainable workaround. Of course, what the buyer really needed to know was whether the required business capability was available efficiently and easily, not merely that the product was very programmable.

Eventually, the responses all have to be vetted, the scores assigned and the results computed. Apart from being a tedious process, this sometimes has unexpected consequences, with a dark horse emerging. Corporate politics normally intervene at this point, and the scores or weightings are adjusted (or a ‘want’ mysteriously becomes a ‘must’ or vice versa) to ensure that the unwelcome intruder is ejected. There may well be two or more factions on the team, each preferring different products, and their respective political strengths will be at least as important as the measured scores.

Finally, a trial or a ‘bake-off’ of one or more products commences. This is usually a farce, and is often dominated by the time required to load the data rather than testing the more speculative parts of the product. Inevitably, the favored company is given more leeway in the trial than any others, who were often allowed to proceed to that point just to legitimize a rigged process.

And in the end, after months of delay and significant cost to both the buyer and all the participating vendors, the initially favored product is usually chosen, as could have been predicted on Day One.

This may seem like a cynical view of the tendering process, but we have seen events like these many, many times. Needless to say, we think this entire process is a waste of time and also leads to bad purchasing decisions (which could at least have been made less expensively).

We think that traditional tendering documents simply do not work with BI. A much more suitable process is to:

There should be no academic tender document; instead, the pre-selection documentation should concentrate on the business needs, not the supposed product features required to satisfy them.

Distractions

There are lots of things that happen in the sales cycle that can distract you from the real issues, so remember to try and stay in control at all times. Sales people are skilled at shifting the agenda to areas where their offering is stronger, but which may not be very relevant to your needs. So, however helpful and professional they may be, remember that their primary objective is to achieve their own sales targets, not to help you. And some of the best, most highly paid, sales people are the ones who do not seem at all ‘salesy’. The BI Surveys have confirmed that projects where the vendors’ excellent sales efforts were a major factor in product selection were among those that achieved the least in business terms.

Minimize the risk of distraction by having a mixed business and technical selection team, who are less likely to be over-impressed by the same things. BI evaluations should never be left to professional software evaluators who spend their lives evaluating products for one dissimilar application after another, but have a limited understanding of the business needs of BI applications. They are likely to take an unbalanced view, with familiar but irrelevant technical features overriding more important issues like ease of use for end-users, performance and flexibility. If anything, the end-users on the team should have the ability to outvote the technical people rather than the other way round (if only because the users will have a greater commitment to making the solution succeed if they lead the selection team).

Vendor size

Many buyers are tempted to favor large vendors on the basis that there is no danger of their disappearing in the next few years. While this may be true, it certainly does not mean that they are the best suppliers of BI tools. We consistently find that customers of small- and medium-sized vendors are happier than the people who thought they were playing safe by buying from a giant company. This is mainly because their BI business is relatively unimportant to a large, diversified vendor compared to the passionate commitment displayed by smaller, specialist companies – so when things go wrong, the smaller companies will often work much harder to sort problems out, and may be more competent in doing so.

Even worse, the BI product’s long-term health is actually much less certain in a big vendor than if it comes from a specialist company whose entire future depends on the success of their BI business. It is easy to accidentally neglect, cancel or abandon BI projects whose revenues are a minuscule part of the vendor’s total business. Large vendors’ executives probably do not relate to analytical applications, so their BI strategies may seem very uncoordinated, depending on the whims of the executive who is temporarily in charge. It may be almost impossible to find senior executives or marketing, support and pre-sales staff who know much about BI in companies like IBM, Microsoft, Oracle , SAS Institute and SAP. Each of these giants has made fatal mistakes with its BI products that simply would not have happened with specialist BI vendors.

One problem is that BI specialist staff often find that they cannot advance their careers in large companies, and tend either to leave or to move into general management. So if you buy from a large company, you will get less continuity of support, and might find that the people who impressed you so much at first soon move on. Large companies that have acquired smaller BI vendors usually seem to lose most of the key people within two or three years (so the fact that a large, financially strong, company has bought a small specialist should not bias you towards it). Perhaps, therefore, the biggest risk of buying from a smaller vendor is that it might get acquired (and damaged) by a large one!

Large vendors sometimes claim to spend huge amounts on R&D, but we do not detect a positive correlation between this expenditure and product innovation or capability. Indeed, it is usually small teams who make breakthroughs, rather than the large development organizations whose efforts seem to go more into integrating acquired products, platform support and bureaucratic procedures.

It is invariably the smaller, specialist companies who are most responsive to market demands, and whose products advance fastest. Their product releases are driven by well-understood customer needs, and new features are usually added more often. The very large vendors have extended release cycles, and most of the ‘new’ features usually turn out to be links to other, unrelated products from the same vendor, which the BI customers probably do not use.

Of course, not every small vendor will thrive, so you need to do some research about the competence and motivation of the management; in some cases, their main interest may be in realizing their personal capital by selling the company to a generous bidder. Meeting the key executives may give a good clue to whether they are realistically in it for the long term, or are more interested in writing books or selling out. And, of course, venture capital-funded businesses are more likely to be sold than purely private companies which are still owned by the founders.

Many BI vendors claim to be technical leaders, often on spurious or contrived grounds. Many also make dubious claims to be BI market leaders, including some who came quite low in our market share listings. But even those who really do have large installed bases are really boasting about their past achievements (when the competitive position may have been very different) rather than their future potential.

Established vendors often claim to have many years of experience, but most of the really experienced people will have moved on and may actually now work for a younger competitor. What is more important is that you will probably be supported by people who have used the product for no more than a couple of years (and in some cases, who are just one course ahead of you). The support people may also have to look after a wide variety of products (unless you pay for premium support), so they may know less about the products you use than you do.

Most large vendors have tame reference sites and some claim thousands of customers. But many of these may not be current users and they may be less than happy with a product that might have been bought years ago. Before getting too impressed by the claims, ask to see the full customer list, and you choose whom you want to meet (and never in the vendor’s presence).

Or, better still, try and use other contacts to find your own reference sites. Always check that the end-users (and not just the IT sponsor) are happy. After all, IT sponsors have a strong vested interest in the project looking like a success, however patchy the record might actually be. They might actually welcome reference visitors because they hope to move to another company deploying the same product, well aware that the failure to meet expectations might be stunting their career in the supposedly happy current employer.

In essence, the advice is always to choose the product that’s right for you, regardless of the vendor.

Technology issues

It is all too easy to be swayed by technology issues. The IT people on the selection team, who may not have a full understanding of the business application needs and little or no BI experience, tend to concentrate on the more familiar technical picture and give it too much significance. And end-users, conscious of their lack of understanding of jargon and architectures, may be browbeaten into accepting technical arguments much too readily, or be misled by sales people into thinking that certain features are more important than they really are.

Platforms are often a distraction. Sites have been known to favor products that still run on relatively obscure or fading platforms, rather than buying software that is better suited to their needs. For example, despite their technical strengths, OS/400, OS/2, mainframes and unpopular versions of Unix (any version other than Solaris, AIX or HP-UX) now attract minimal BI server support (though Linux support seems to be building). There is also little or no remaining support for Macintosh, OS/2, Unix or pre-Windows XP desktops. Most BI server vendors have a preference for 64-bit servers because of their greater memory capacity, which greatly benefits performance, so it is unwise to insist on running BI applications on old 32-bit servers.

And BI vendors often support Microsoft Internet Explorer better than they do other browsers, while Excel add-ins greatly outnumber those for other spreadsheets. Generally speaking, it is not worth demanding that software runs on a little-supported server platform, as it is usually cheap and easy to use Windows servers (which almost all products support well). Changing client platforms to suit a BI deployment is much more difficult, but may be useful as the trigger for an already pending upgrade.

Although Windows systems are not quite as stable or scalable as Unix or proprietary IBM platforms, they are usually more than adequate for BI applications, few of which have a technical requirement for anything larger, more sophisticated or more reliable. Only the biggest sales analysis systems or large-scale Internet deployments really need the extra power of an exotic (and extremely expensive) high-end Unix server. The Windows software market is also much more commoditized, so you will often find lower list prices and will certainly have more bargaining power when buying software to run on Windows. Thus, (expensive) decisions to use Unix servers ought to be justified by other considerations such as available skills, integration requirements or existing spare capacity, rather than the undoubted, but seldom-needed, additional scalability and robustness that they offer. Based on The BI Surveys, we estimate that over 80 percent of BI server deployments are on Windows servers, with most of the rest being on Unix or Linux.

Many sites fall into the trap of deciding in advance that they want a supposedly open ROLAP solution instead of a ‘proprietary’ MOLAP or HOLAP. This can be a big mistake, because although ROLAPs store their data in standard RDBMSs, every ROLAP has its own complex and unique schema, and no ROLAP will work with another product’s schema. In effect, each has its own proprietary data structures that just happen to be stored in an RDBMS. Thus, the only useful form of OLAP openness is a well-supported API, and ROLAPs are weak here: none of them has a well-supported, open API.

And although ROLAPs can indeed handle more data than most MOLAPs, there is a high price to pay in terms of reduced functionality, much greater implementation costs and drastically worse performance. Thus, any decision to prefer a ROLAP should only be based on the properly assessed business and technical needs of the application, and never on the almost religious, but misguided, grounds of ‘openness’ and ‘scalability’. In general, perhaps between one percent of OLAP applications actually need pure ROLAP solution, and another five or so percent could be handled reasonably well using ROLAP, with the rest being much more suitable for MOLAP or DOLAP.

Performance is important as it will determine not just how responsive the system is for users (which plays a big part in user satisfaction), but what server capacity will be needed. Badly performing products have additional hidden costs, in that much more effort needs to go into tuning the applications and hardware upgrades may be needed. But determining the expected performance of a product is not easy. There is only one agreed multi-vendor OLAP benchmark, and only three companies have ever run it, none in this century.

All vendors will claim to be faster than their competitors (“we never lose a benchmark”), so to find out the truth, you need either to do a full-scale trial of the product (hard, slow and expensive) or visit other user sites with data volumes, server power, numbers of users and application complexity similar to yours. Do not be surprised to hear defensive IT people in the user sites saying that, “today is very busy – it’s not usually so slow”. Make sure you see for yourself the performance delivered to end-users, and always double-check the claimed input data volumes and concurrent user loads: in our experience, the initial guesses always turn out to be exaggerations. And, when confirming data volumes, remember the effect of database explosion.

Don’t be too easily swayed by claims of integration and openness. All BI tools can connect to relational databases and data warehouses, so if you have the right ETL tools to connect to complex sources (such as SAP), it should always be possible to access the relevant data. Similarly, most Web BI tools can be integrated into standard portals. What is less easy is to integrate multiple BI tools in one solution.

For example, if data is stored in one vendor’s OLAP server, other vendors’ BI front-end tools may not work well with it, despite claiming to support the server. In particular, although most BI front-ends claim to work with Microsoft’s ubiquitous Analysis Services OLAP server, many do this quite badly, offering limited functionality and slow performance. Specialist OLAP client tools that focus on a particular server, such as Analysis Services or Essbase tend to provide a much better user experience.

Some of the in-memory BI tools are even harder to integrate into a multi-vendor solution. They often do not have an Excel interface or a reporting API, and may be limited in the data sources they can access without significant programming or scripting.

Another area which can confuse is the Web. Just about every vendor has at least one Web deliverable solution today, and many have grandiose strategies for the future. But this is a fast changing area, and unless your immediate priority is a large Web deployment, you should not allow yourself to be overwhelmed by this topic. Vendors constantly leapfrog each other, so today’s leader may be tomorrow’s laggard.

There are several different browser technologies, and none is perfect for every deployment. Between them, the BI vendors have used just about every possible Web approach, thus indirectly proving that there is no single ideal architecture. If a Web deployment is your first priority, you should prefer a product that offers an approach that is compatible with your planned deployment, rather than one that simply demonstrates well or which appears to use the most buzz words.

Many vendors season their presentation with lots of jargon words like object-oriented, portals, Java, J2EE, JavaBeans, Javascript, cloud computing, DHTML, Flex, Web services, AJAX, XML, .Net, plug-ins, snap-ins, cartridges, applets, ERP, OLE DB, MDX, COM, ADO MD, MDAPI, APB-1, CORBA, SMP, MPP, ActiveX, neural networks, agents and more. In general, excessive use of such jargon does not correlate with a good product, whose strengths can stand out without needing such props.

Beware also of claims for miracle future versions. Every vendor can claim that the mythical ‘next version’ will be faster, more scalable, more capable, more reliable and easier to use. But last year they were probably making exactly the same claims for today’s clearly inadequate release. If you are tempted to consider a future release from one vendor, then you should probably extend the same courtesy to all the other shortlisted vendors as well. And you should assume that they will all ship at least a few months late (delays of years and total cancellations are not unknown), and that some promised features will not make it to the finished release.

All competent vendors can lay on a sizzling demonstration. It will probably be based on trivial data volumes, have only one connected user, suffer from no network delays (as it runs on a virtual machine on a laptop), and possibly have only one (scripted) route that works. If you want to see how the product really works in production, it is much more useful to visit an existing user with a production application that is about as big, about as complex, runs on similar hardware and has as many concurrent users as you expect yours to have eventually. Be sure to find out not only how well it works now, but how long it took to develop, how much consulting it took, and how satisfied the end-users are. If possible, find out if they purchased significantly more seats than they have managed to deploy.

Financial issues

Remember that BI software license fees are only a small proportion of the total costs of a project. The implementation and hardware costs may both be larger than the license fees, and even the long-term support, maintenance and administration costs are almost certainly much greater. Make sure you fully understand the sometimes complex pricing model, by asking some relevant questions:

And if you make the wrong decision to buy an unsuitable product just because it is cheaper, you could end up with a much higher total cost of ownership (TCO) because of the higher service, administration and/or hardware costs and, probably, the lower level of business benefits achieved. Also remember that consultants advising you may have a preference for products that require more implementation services, not less.

Never choose a product just because it is cheaper, heavily discounted, bundled or even free (even ‘free’ products should be evaluated if being considered for a serious deployment). But if two products seem otherwise similar, and both can meet your needs, then by all means choose the lower priced. Equally, you should not assume that a higher priced product is necessarily better than a lower priced one. BI vendors charge what they can, and a large vendor with good marketing and sales abilities will get away with charging much more for a product than it is really ‘worth’. Indeed, we have seen many examples of a cheaper product being much better for some applications than more expensive ones. So it is probably best to put aside price considerations in the early stages of a product evaluation, unless you know that some products are definitely beyond your budget (most of the product reviews in The OLAP Report contain pricing information).

Proof of concept

Companies often perform a trial or bake-off of several products before making a purchase, but few do a real proof of concept, despite often using the term.

We prefer the last of these as it is the fastest to perform, and is most likely to uncover hidden problems with data scalability, product functionality, quality or integration. But it is also the method used most rarely because it is intellectually more demanding, requiring as it does a more scientific approach and a degree of self-confidence. But by focusing on those important areas that are doubtful or still unproven, it makes much better use of the evaluation time than the other approaches, most of whose effort focuses on the ease of use of the basic features (which will ultimately turn out to be an insignificant part of the final system). However, this scientific approach is less involving for business users, who would probably be happier building a separate prototype.

A trial is an acceptable approach, but there is a danger that a poorly-architected initial test system will be overextended as a production system; it is often better to learn from the trial, but then develop the production system from scratch. The main snag with this approach is that it can prolong the decision cycle.

We think that the widely used bake-off approach is the least useful, because it inevitably concentrates on the more trivial aspects of the products, whereas the important differences that will affect the long-term success of an OLAP project are more subtle. For example, a low-end desktop OLAP product may allow simple applications to be built and populated much quicker than a heavyweight contender, so it will be more likely to perform better in a one week bake-off. But it may be an unsuitable product for the real system, which is undoubtedly larger and more complex than the simplified test case used in the bake-off. However, a bake-off is more fun for the users, and may have psychological advantages in that it increases user commitment to the success of the project.

Conclusions

BI products and applications are being deployed on a larger and larger scale, which inevitably means that many people will be tasked to select a BI tool without any previous experience of using or deploying one. This means that they bring previous experience from other spheres, which often does not translate well to BI. Picking the wrong product can be an expensive mistake, and even picking the right product in the wrong way can be time consuming and inefficient. And there are plenty of unpublishable horror stories of companies who have wasted millions by buying the wrong product, or even too much of the right product that ended up as shelfware.

Although there is plenty of free advice around, much of it comes from biased sources, and should be viewed with suspicion. Beware also of reprinted analyst reports, which are often sponsored by the vendors themselves. And skilled vendor sales people are expert at making you think you need what they happen to have to sell. But as with any other software, the people who support you later may turn out to be less expert, and the product may not meet your needs.

The cost of doing the selection properly is no higher than doing it badly; indeed it can be quicker and less painful. It does involve doing more research before getting involved with vendors, but this very worthwhile and will save you from wasting time with inappropriate suppliers and reduces the chances of being bamboozled.

This section, in identifying many of the mistakes people make, has also pointed out the more suitable alternatives, and by following them and the advice in the many linked sections, buyers should be able to make better decisions, with more certainty and more quickly.

But remember that picking the right product does not, of itself, guarantee short or long term project success. The project needs to be properly sponsored, to have a committed user base, a real business requirement, access to usable data and a measure of luck.

This page is part of the free content of The OLAP Report, but ten times more information is available only to subscribers, including reviews of dozens of products, case studies and in-depth analyses. You can register for access to a preview of some of the subscriber-only material in The OLAP Report or subscribe on-line.

	At worst, the project will be a complete failure, with all the software and services costs written off, and the loss of the hoped-for business benefits. The total losses could be in the millions.
	More typically, some vestige of the project will survive, so that the culprits can claim it as a “success” and quietly move on to other things. But the scope will be reduced, the deployment minimal and the benefits illusory. Such projects are all too common, and are in some ways worse than the first category, because there is less incentive to replace the failed system, and so the potential business benefits are lost for even longer.
	Some projects have grand ambitions for enterprise deployment, but never make it beyond the first departmental application. They may deliver some benefit to the initial group of users, but not enough for the next project to escape having to go through the whole selection procedure again, duly coming up with a different choice. In the meantime, the company will almost certainly have purchased surplus licenses for the first project, adding to the ‘shelfware’ mountain.
	The selection team (or senior management) may get carried away by the grandiose visions of the supplier, and invest in an extortionately expensive solution to a simple problem. It may work adequately well, but cost many times more than an equally acceptable, or even a more appropriate, solution.

	try and predict what the users really need, not just what they say they want; this means you have to understand how they do their jobs, what skills they have, and what information and analyses would help them be more productive
	involve end-users at every stage, including project definition, product selection and implementation
	not expect end-users to be able to list their exact requirements beforehand – and don’t force them to predict every possible need (you’ll end up with an impossible list)
	never try to dictate storage and processing architectures before properly understanding the business needs
	not use up surplus ‘shelfware’ from a previous over-ambitious BI purchase.

	input data volumes, now and in the future (which may take some digging)
	data volatility (some products cannot handle highly volatile data and metadata)
	type of application: sales/marketing analysis, CRM, budgeting/planning, performance management, balanced scorecard, quality analysis, financial consolidation and reporting
	key features: write-back, allocation calculations, sophisticated currency conversions, printed report quality, spreadsheet interface, complex application logic, dimensional flexibility, statistics, planning and forecasting (including ‘what if?’)
	integration: other related systems, such as e-mail, ABM, data warehouses, data mining and ERP
	number of users and their locations: affects architecture, security and costs
	user roles: top management, finance, marketing, HR, sales, production, engineering, etc
	current user and developer skills: Excel, VB, JavaScript, Java, J2EE, SQL, ETL, EIS, Web, cube design, etc
	who is expected to do the implementation and maintenance: outside consultants, in-house IT staff or end-users
	server platforms: NT, Unix, AS/400, Linux – but don’t insist that shortlisted products must run on obscure or fading platforms that you still use
	client PC and browser standards
	deployment architectures: stand-alone and dial-up PCs, high speed client/server, intranet, extranet, Internet
	international deployment: multi-currency support, multi-lingual operation, data sharing, local support, licensing issues, update windows across time zones, network bandwidth
	budget: software, hardware, services, telecommunications.

	understand the needs, as described above
	produce an initial shortlist of up to four vendors, with whom conversations should be held to see if they understand the business requirements and are credible (including seeing detailed demonstrations and visiting references)
	at this stage (not at the end), proposed business terms should be agreed with the leading one or two suppliers
	a proof of concept should be carried out with no more than a couple of suppliers, the purpose being not to develop a prototype, but to test the parts of the business needs that are likely to be problematic for that supplier
	the vendor with the best business terms, who has also proved to be capable of meeting the needs, should be selected.

	Is concurrent user licensing available, or is per seat pricing (which usually works out more expensive) the only option?
	Do you have to pay for users accessing just run-time versions or even PDFs?
	What is the charging mechanism for Web users?
	If processor power or clock speed is used as the basis for the pricing mechanism, will you have to pay a software upgrade fee when you upgrade the server?
	What do you have to pay for licensing back-up and development servers?
	Do you have to pay extra for Unix, RISC, multi-core or 64-bit server versions?
	Do you have to pay a premium to connect to particular data sources or databases?
	Do you have to buy a product bundle that includes components you don’t need?

How not to buy a BI product

Practical advice to avoid the numerous pitfalls when buying BI products

Contents