BI Blogs

IDC just published its latest report “Worldwide Business Intelligence Tools 2005 Vendor Shares” on the state of the business intelligence industry . According to IDC’s research, worldwide software revenue grew 11.5% in 2005 to $5.7 billion.

“IDC expects the market to maintain this level of growth for the foreseeable future," said Dan Vesset, IDC’s Business Analytics research director and the report’s author. "Although there are signs of a shift in the competitive landscape, BI tools remain an attractive market for software vendors and one of the top investment priorities for end-user organizations."

The two subjects of interests are the growing business demand for business intelligence and the competitive marketplace for software vendors.

The report states that business intelligence is “something without which organizations can’t succeed. However…(it) still has a long way to go in reaching all the necessary people and processes in companies, government agencies, hospitals, and universities.”

Therein lies the exciting promise of BI that is building upon its own successes. BI has enabled great business value at many companies, but its impact is really just beginning to be felt. Up until now business implementations have targeted analysts, managers and “power users” within enterprises, but broader adoption is starting to take hold in the remaining 80% of enterprises’ employees.

Compliance, competitive pressures and intercompany connectivity between customers, suppliers and partners are all business drivers for wider BI adoption. In the short term, performance measurement and operational BI projects are expanding BI’s business breadth and depth. In addition, these tools have become cost- and resource-effective to start penetrating the SMB (small-to-medium) marketplace.

The report’s message on the software vendor landscape is that the market is maturing and consolidating (just ask Larry Ellison). It states that the top 10 software vendors control approximately 62% of the market. Although that may sound high, it appears to me that it does not approach other mature industries where the top few companies control 90% of market. In fact, it surprises people that so many companies are still selling business intelligence software today despite the fact that it is such a mature industry. Many of these smaller software vendors are relatively unknown despite the fact that they have quality products that offer tremendous benefits to their customers,. Smaller software firms are the source of many innovations in the BI market, and are also where some of the best software talent can be found.

But speaking of the 800-pound gorillas, the top five business intelligence software vendors, according to IDC, are:

1. Business Objects (13.9% market share)
2. SAS Institute (10.2%)
3. Cognos (9.9%)
4. Microsoft (6.2%)
5. Hyperion Solutions (5.0%)

Consolidation has already occurred from top to bottom of this industry, but more is likely.

The report refers to the battle between standalone BI (BI pure play vendors) and database-embedded BI from Microsoft, Oracle (#6, 4/3% market share) and IBM (#11, 2.4%). IDC states that the database companies have expanded their BI offerings and are growing at almost double the rate of BI pure-plays.

I think this trend is happening because the database vendors are expanding the BI functionality they offer, BI is a natural extension of their database (and data warehousing) functionality, and the offerings are a cost- and resource-effective solution. Remember pure-play BI vendors starting pushing aggressively BI consolidation a few years ago, now database vendors are extending the sales pitch to suggest the advantages if that consolidation originates from a company’s database vendor.

An interesting observation is that the number one BI tool in the marketplace, Microsoft Excel, is NOT counted as a BI tool, thus understating Microsoft’s impact on the marketplace. My take? Microsoft is a force to be watched because of:

• Microsoft SQL Server 2005, including Analysis and Reporting Services
• The acquisition of ProClarity, which will grow its enterprise applications business and expanding the role of Excel as a BI interface

The report states “Microsoft’s impact on the BI tools market cannot be overemphasized….it will reshape the BI tools market over the next 15 years.”

The business intelligence market is certainly healthy and growing. Has business intelligence reached the tipping point where it will finally become pervasive for enterprises both internally and externally with their customers, suppliers and partners? Will it become pervasive in enterprises of all sizes and not just the Fortune 500? It is an exciting time to be a business intelligence practitioner.

But remember, it’s all about the data!

IDC just published its latest report “Worldwide Business Intelligence Tools 2005 Vendor Shares” on the state of the business intelligence industry . According to IDC’s research, worldwide software revenue grew 11.5% in 2005 to $5.7 billion.

“IDC expects the market to maintain this level of growth for the foreseeable future," said Dan Vesset, IDC’s Business Analytics research director and the report’s author. "Although there are signs of a shift in the competitive landscape, BI tools remain an attractive market for software vendors and one of the top investment priorities for end-user organizations."

The two subjects of interests are the growing business demand for business intelligence and the competitive marketplace for software vendors.

The report states that business intelligence is “something without which organizations can’t succeed. However…(it) still has a long way to go in reaching all the necessary people and processes in companies, government agencies, hospitals, and universities.”

Therein lies the exciting promise of BI that is building upon its own successes. BI has enabled great business value at many companies, but its impact is really just beginning to be felt. Up until now business implementations have targeted analysts, managers and “power users” within enterprises, but broader adoption is starting to take hold in the remaining 80% of enterprises’ employees.

Compliance, competitive pressures and intercompany connectivity between customers, suppliers and partners are all business drivers for wider BI adoption. In the short term, performance measurement and operational BI projects are expanding BI’s business breadth and depth. In addition, these tools have become cost- and resource-effective to start penetrating the SMB (small-to-medium) marketplace.

The report’s message on the software vendor landscape is that the market is maturing and consolidating (just ask Larry Ellison). It states that the top 10 software vendors control approximately 62% of the market. Although that may sound high, it appears to me that it does not approach other mature industries where the top few companies control 90% of market. In fact, it surprises people that so many companies are still selling business intelligence software today despite the fact that it is such a mature industry. Many of these smaller software vendors are relatively unknown despite the fact that they have quality products that offer tremendous benefits to their customers,. Smaller software firms are the source of many innovations in the BI market, and are also where some of the best software talent can be found.

But speaking of the 800-pound gorillas, the top five business intelligence software vendors, according to IDC, are:

1. Business Objects (13.9% market share)
2. SAS Institute (10.2%)
3. Cognos (9.9%)
4. Microsoft (6.2%)
5. Hyperion Solutions (5.0%)

Consolidation has already occurred from top to bottom of this industry, but more is likely.

The report refers to the battle between standalone BI (BI pure play vendors) and database-embedded BI from Microsoft, Oracle (#6, 4/3% market share) and IBM (#11, 2.4%). IDC states that the database companies have expanded their BI offerings and are growing at almost double the rate of BI pure-plays.

I think this trend is happening because the database vendors are expanding the BI functionality they offer, BI is a natural extension of their database (and data warehousing) functionality, and the offerings are a cost- and resource-effective solution. Remember pure-play BI vendors starting pushing aggressively BI consolidation a few years ago, now database vendors are extending the sales pitch to suggest the advantages if that consolidation originates from a company’s database vendor.

An interesting observation is that the number one BI tool in the marketplace, Microsoft Excel, is NOT counted as a BI tool, thus understating Microsoft’s impact on the marketplace. My take? Microsoft is a force to be watched because of:

• Microsoft SQL Server 2005, including Analysis and Reporting Services
• The acquisition of ProClarity, which will grow its enterprise applications business and expanding the role of Excel as a BI interface

The report states “Microsoft’s impact on the BI tools market cannot be overemphasized….it will reshape the BI tools market over the next 15 years.”

The business intelligence market is certainly healthy and growing. Has business intelligence reached the tipping point where it will finally become pervasive for enterprises both internally and externally with their customers, suppliers and partners? Will it become pervasive in enterprises of all sizes and not just the Fortune 500? It is an exciting time to be a business intelligence practitioner.

But remember, it’s all about the data!

Random rambles on a weekend away

Part one of two on the use of fast M-view refresh in data warehouses. Firstly a few reasons why on don’t like on-commit refresh.

My last entry introduced the concept. Making it work is the hard part. One of the comments I received asked about handling business changes, politics, and a variety of other circumstances. In this entry we’ll begin diving in to the differences a little more, and discuss a small portion of the politics that surround this. I’ll be presenting some of this information at the upcoming TDWI conference in August-2006, and the Upcoming Teradata Partners conference in November.

I recently came across the following question: How can I find the most typical record in a group or cluster of records? For example, suppose we have a set of customer records, what is the customer that best typifies the group or cluster? The answer to this question can…

[ This is a content summary only. Visit my website for full links, other content, and more! ]

Like Doug
and Alex,
I got my acceptance email through from the UKOUG yesterday saying that
my two papers had been accepted for the conference in November.
I know most of you probably thought I picked the papers in the BI track
anyway, but this year I couldn’t make it to the paper selection day so
I genuinely didn’t know until a day or so ago that I’d been accepted.

Anyway, the two papers I’m doing are:

• “Analytic Workspaces -  A Performance Improvement
  over Materialized Views”

• “Inside Oracle BI Server and Oracle BI Suite Enterprise
  Edition”

The first one will be an investigation as to whether analytic
workspaces, together with OLAP_TABLE based views over them, can create
more effective summaries that regular materialized views - I don’t know
what the answer is yet, I’ll be doing lots of testing between now and
then - and this will be in the DBA server tech track, so I should get a
fairly knowledgable and testing audience.

The second on is an overview of the new Siebel
Analytics-derived BI application server that powers the new BI Suite
Enterprise Edition. It’ll be a development on what I presented at the
recent Reporting
Tools event in London, with some extra content around
performance tuning, bringing in non-Oracle data and so on.

I also got the confirmation through from Open World 2006
yesterday, saying that the second presentation on BI Server has also
been accepted for the User Group Day on the Sunday. This is an 1.5 hour
session so I’ll be able to do lots of demos, take questions and so on.

Next up, the dates for the BI Masterclass seminar series
around Europe are starting to come in. The first four on the list are
definately confirmed, the ones from Estonia on are still to be
confirmed by the local Oracle office. Anyway, here are the dates and
locations so far:

• Netherlands - August 29th, 30th
• Denmark - October 3rd, 4th
• Norway - October 10th, 11th
• UK - October 31st, Nov 1st
• Estonia - September 26th, 27th
• Latvia - September 28th, 29th
• Lithuania - September 21st, 22nd
• Slovakia - October 17th, 18th
• Czech Republic - November 7th, 8th

More details on what’s in the seminars can be found in this
posting. Hopefully I’ll be staying locally between the
Lithuanian and Estonian seminars, it’ll be a chance to do a bit of
sightseeing and take a look around. I’ve always wanted to visit the
Baltic states, should be very interesting.

Finally, like some other people I’ve been mucking around with
Linux distros at home recently, but for me the one I’ve been looking to
get up and running is Ubuntu
Dapper Drake. Although I use Centos at work, for the Red Hat
compatibility, at home I wanted something where I could install
software easily, quickly get support and get some “strength in numbers”.
With Ubuntu, it’s so ubiquitious now as a home linux distro that you
can always find something on the net to help if you’re having a
problem, most software is packaged up for it now, and the Debian
underpinnings means that installing software is as easy as “sudo
apt-get install foo“.
It’s also got a nice, well polished interface and a “less is more”
approach to the initial install.

src="http://www.rittman.net/images/Screenshot.png"/>

So far I’ve got it up and running, got my Netgear Wifi card
working ok (courtesy of ndiswrapper), I’m using AmaroK and Rhythmbox
for my music (AmaroK is more iTunes-like, but Rhythmbox interfaces
better with my iPod and the Serpentine CD burner). Of course, I’ve also
got the (now free) VMWare
Server up and running, and brought across my Oracle VMs, and
I’m using Nvu as
my “frontpage-replacement” for putting blogs together. Another nice
touch was using EasyUbuntu,
a set of scripts that installs all the usual codecs, fonts, browser
plug-ins and so on that you end up hunting around for when you first
install a distro.

So now I’m solely living in a Unix-derived world … OS X
Tiger on the laptop, Unbuntu on the PC. I’ve tried it in the past but
it’s always fallen down over hardware support and rough edges in the
software, but so far (a week later with the PC) things are going ok. I
think that this time, it’ll work out just fine.

Thanks for the introduction Abhinav…:)

I will start out with highlighting some of the advantages of using the OracleBI Spreadsheet Add-in. They might not all be quite obvious, but you will appreciate them when you use the product.

Ease of use. It installs really easily. Just run the .exe file, walk through the wizard and you are done. Start Excel, and you will see the OracleBI menu. We have

In this posting I’ll look at how pluggable mappings are used in Oracle
Warehouse Builder 10gR2.

Pluggable Mappings are a good way of creating reusable ETL components that
can be dropped into a mapping. A pluggable mapping consists of in and out
interfaces, and a set of OWB mapping steps that retrieves a value, updates an
object or some other such ‘complex’ task. Pluggable mappings are used by OWB
itself when, for instance, you load data into a dimension object, and you can
create libraries of them for use as "super-transformations" within
your project. It’s a similar concept to procedures in PL/SQL - you just pass
across the required parameters, it performs the task and sends out any output
parameters.

In this example, as part of the design phase we have identified a common
requirement to translate product names and translations into various languages.
We could include the logic to do this - lookups to references tables, joins and
so forth - in every mapping, or we could take the logic and turn it into a
pluggable, reusable mapping with interfaces at the front for the product ID, and
at the back for the translated items. You could of course do the same with a
database function, but doing it this way keeps the mapping logic within OWB,
meaning that an OWB developer can code it rather than needing a PL/SQL guru.

To start the process off, I define two source modules, OE and HR, and a
target module, ETL_TGT. I then navigate to the Pluggable Mappings node, and
create a new pluggable mapping, PRODUCT_DETAILS.

I then leave the input and output groups at the default settings.

Next I define the input signature. This is is the list of input
parameters accepted by the pluggable mapping, and it’s this that users will map
to when using it on a mapping.

And I then do the same for the output signature.

Now that the input and output signatures are defined, I put the mapping
together to implement the pluggable mapping.

And join the tables together, to retrieve the translated details
from the Product Descriptions table, and the supplier ID from the Product
Descriptions table.

At this point there’s no way you can verify the mapping, you have to do that
in the context of the mapping you drop it in to. To try it out, I put another
mapping together to bring across  inventory information and translate it
into a specified language. To do this, I use the Inventories table from the OE
schema, do a key lookup on the Warehouses table to retrieve the name of the
warehouse, and use my new pluggable mapping to translate the product
description.

Then, when I run the mapping, remembering to enter the input
parameter ("DK", for Danish) I get the required results.

Going back to the mapping, we can expand the pluggable mapping
element to see how it works:

And that’s broadly how it works. When you come to build other
pluggable mappings, it’s worth bearing in mind that there are two ways to create
them:

• Directly, using the Pluggable Mapping node in the Project
  Explorer, and

• lign=”left”>Within a mapping, using the pluggable mapping input and
  output signature operators.

One thing you can’t currently do is locate a mapping, "draw
a box" around part of the mapping and save it as a pluggable mapping. You
have to either create them as a separate exercise, or create them as you’re
building a bigger mapping, perhaps where you indentify as you’re building that
the mapping you’re building is reusable.

My July 7 post on Stafford Beer’s legacy for BI professionals gave an overview as to how I implement his concepts in my BI designs. Here is some more detail on one of these principles – Attenuation.

Attenuation in BI is the art and science of presenting information to executives in such a way that they are:

Aware of the current status of the business

Satisfied that situations requiring their attention are highlighted, and

Alerted to unusual situations that may presage problems or opportunities

This obviously requires specification of pre-formatted, on-demand or ad-hoc reporting. Executives will know, or can be helped determine, what they need to know to meet these three objectives.

Theory indicates that there are 3 possible ways to implement Attenuation in BI reporting for the above objectives:

Where Are We? – Summarize the situation, tell the executive the status

What’s Good and Bad about where we are? – Compare, compute exceptions, find the executive’s problems

What’s unusual, and what’s forecast? – Find out what is obscured by noise in the data and alert the executive

A creative BI design will present information using these three approaches.

Attenuated information, derived from a mass of transactions data, and augmented by tacit (soft) data where appropriate, can give huge benefits to the executive – it just needs some mental effort applied in a structured manner.

Of course, we also need to ensure balance of reporting scope, in the Balanced Scorecard manner. I believe this is a different concept, and is best handled via the template of potential KPIs that forms part of a comprensive methodology.

If you’re interested, check my technique for doing this at www.bipathfinder.com where I explain this in more detail, including the template design aspect.

I’ll cover Stafford’s Amplification next time round.

Warehouse Builder 9.2 is nearing the end of its life and will be desupported next year. But OWB 10.1 will be gone first!

I just opened the wiki on SQLBI.EU: it contains link to useful resources for BI solution development with SQL Server, so it covers SSIS, SSAS, SSRS and of course SQL Server itself.

The wiki works well when there is large contribution, so please share your links with the BI community! Comments and suggestions are welcome!

We appear to have consensus that effective management of tacit knowledge is a key part of any successful corporate BI environment. Henry Minzberg is the father of this movement with his seminal article in HBR July-August 1975 and the splendid quote “The strategic database of an enterprise is in the minds of its managers, not in the memory of its computers”.

This post is intended to start a discussion thread on how we can maximize the value of tacit corporate knowledge – the opinions, ideas, explanations, suggestions that are known, but not shareable. The data warehouse, schema, metadata, etc. are all unconscious of these data items, yet they are critical to successful business.

If we don’t manage this resource, we are condemned to:

Solve problems many times over

Make errors in interpreting hard data – because we don’t have the expert’s explanation

Be unaware of early warning signs that people know about, but the data doesn’t yet show

Corporate performance management is impractical if the reported information is not accompanied by the explanations of subject experts, and assessments of implications.  You’re not measuring corporate performance, you’re just keeping score without knowing who is winning. The hard without the soft is pretty much useless.

Service Oriented Architecture is just so much spaghetti, unless the sharing of what is known about the business is not only possible, but empowered.

A first step in managing the milieu of tacit knowledge is to realize that we can neither share, nor search, nor retrieve soft information that is not classified. It’s like having an iPod with all the music dumped in one sector, or navigating a city without street signs.

Classification is the heart of the task, and classification requires that you have a vocabulary. Here is one area of corporate IT where Nazi principles are required. Everyone must use the same “label” to describe the same concept – synonyms can be poisonous, one person’s synonym is another’s swear word.

Vocabularies, often called taxonomies or category listings, are becoming an important part of information technology as used by people and corporations to manage the huge pool of documents, email, news and business intelligence that flows throughout the community.

Vocabularies are valuable components of a knowledge management support system, providing the categorization model that facilitates efficient searching, browsing and alerting. Without an adequate vocabulary, the users are forced to build their own personal rules for document searches, and agent specification. Often, this yields poor retrieval, or people just pass over the opportunity to become better informed, because it is too onerous.

A vocabulary for BI tacit knowledge is a set of preferred topics or keywords that facilitates categorization of documents in an enterprise.

It is equally applicable to business or government and will usually be specific to the particular organization.

The vocabulary typically will comprise business entities (e.g. customers, competitors, government programs, products), business issues (quality, problems, strategies, employee matters, plans), technologies (telecommunications, optics, computing) and competitive intelligence (industry news, patents, reports, email), useful to the corporation.

For BI applications, the vocabulary ought to have both concrete concept related topics, e.g. customer names, places, projects, technologies, and soft concepts e.g. strategic partners, problem projects, important issues, etc.

Included in the vocabulary construction often is a set of rules used to automatically associate specific topics with relevant documents – the auto-classification process.

This is why you need to consider building an enterprise wide vocabulary:

Tacit information oriented BI systems facilitate collaboration among colleagues.

Vocabularies used for classification improve document access via searching, alerting, browsing.

It is almost impossible to share, browse, search or filter a large amount of information unless it is categorized.

Automatic categorization of a large, existing and evolving collection of documents in many databases is often required, and vocabulary based classification is proven as effective for this.

Pure text searching is not adequate in a corporate context, it throws up too many hits and the user needs to know what to search for.

Browsing the collection using topics from a vocabulary as the directory is efficient and satisfying to the user.

Alerting to new material requires individual specification or personalization of the type of documents required, and the subject category, chosen from a vocabulary, greatly facilitates this.

The growing enthusiasm for metadata as a standard document description process, using XML as the storage format stimulates the need for, and the value of, vocabularies for BI

In subsequent posts, I will describes how vocabularies may be used in Business Intelligence, what they contain, how they are best constructed and issues with scalability. These principles are based on my several years experience with the Grapevine knowledge sharing system.

I recently decided to take another look at JFreeReport after reading some news items on www.jfree.org while downloading the latest version of JFreeChart. I haven’t looked at JFreeReport in a number of years after my initial impression that it’s TableModel based design was more suited for a desktop based application then a web based application. Now that JFreeReport has joined Pentaho and the Pentaho Report Designer is available to create reports, I figured I would see how hard it would be to integrate with OpenReports.

Basic integration was fairly straight forward, and I am now able to generate HTML JFreeReports through OpenReports. If there is any interest, I may include JFreeReport support in the next release of OpenReports. Are there any JFreeReport users out there? Is JFreeReports mostly used in desktop applications or with one the Pentaho products?

Now this appeals to my sense of humour. This article in today’s Independent talks about a new game called Street Wars, a three-week water pistol assassination game played out around London. When you sign up, you are given the mobile number, name and address of someone you have to ‘assassinate’, and someone else has got yours, and it’s then down to you to find your victim before they find you. Survivors from this round go on in a knockout tournament until just one person is left.

According to the article, one of the players spent from 4am waiting outside their opponents home to get them on the way to work, another set up a fake job interview just so he could squirt someone and take them out, I’ve read elsewhere of people literally hiding in bins for hours just waiting to get their target.

“But for Billy S - not his real name - a thirtysomething City IT specialist, talk of danger was overblown.

He spent the morning from 4am lying in wait for his target outside his London flat. Dressed in a business suit in order to blend in with his surroundings, he shot but narrowly missed but then as his quarry fled, he trained his fire on another member of the rival team who popped her head out of the window to see what was occurring. She too survived …

…In the United States, one player lured an opponent to a fake job interview before literally “hosing” them down. Others have rummaged through rubbish bins for clues about their targets or tricked their way into apartment blocks.

… Hamish Moseley had taken the precaution of wearing a wig on the way to work yesterday. Having been summarily appointed team leader of his “assassination squad”, Death Aquatic, he was a high-value target for would be assailants. If he is “killed” the entire three-man team is out.

“Frankly I’m crapping myself. I was looking outside my house this morning to see if anyone was there. I wore the wig when I went to the pub at lunchtime and now I’m just keeping a low profile. The attraction is living on the edge. The thrill is going up to a complete stranger you only know from a photo and soaking them. But you do think, ‘what if I get someone’s surly brother by mistake and they beat me up’?”

But the tension was already starting to show. “It is only one day into it and I’m having a bit of a miserable time at the moment hiding here in the office. I’m worried that there are a lot of serious people without jobs out there and a lot of time on their hands who are looking for me.”

Of course given what happened last year, you’d better not make your water pistol too realistic. Sounds fun though, shame I live down in Brighton as you have to live within London to take part (and of course I’m not sure I could spend hours waiting in a bin, not with clients to visit. But still…)

What a neat trick. This is the kind of thing researchers do when nobody is watching. In this case someone was watching, though.
Link

Scalar subqueries…I remember Tom extolling their virtues at the UKOUG last year in one of his presentations. They seem like a neat idea except that they have an unpleasant side effect which I came across the other day on a production system.

We had a situation where a piece of DML was running slowly and I was asked to take a look at it to see what it was doing. I was told that the query had