Decision Management on Steroids:Will Big Data Tools Trump Rules?
Can the ability to extract meaning and sentiment from previously unconventional data sources reorient the role of business rules?
In a typical customer application, scoring models are created by finding patterns and relationships from attributes using various statistical techniques and the customer records are scored for propensity or eligibility. Rules then apply policy – what to do with the scored records.
But the promise of Big Data is to deliver insight not possible with tools of even five years ago. Does the newer technology that can, to some extent, detect sentiment and propensity, examine relationships of 100’s of millions of ID’s, construct path analysis in real-time, eliminate the need for rules? In other words, does the data speak for itself?
On the other hand, can quantitative methods really implement policy or are we just in the early stages of the hype cycle. Is attended or unattended quantitative analysis of Big Data a sufficient model for implementing policy?
New information usually comes from unexpected places. Big leaps in understanding arise from unanticipated discoveries—but “unanticipated” does not imply a sloppy or accidental process. On the contrary, usable discoveries have to be verifiable, but the desire for knowledge requires a drive for innovation and the exploration of new sources of information that can alter our perceptions and outlooks. Unraveling the content of “big data” that lacks obvious structure begs for some new approaches. Big data is positioned to provide that insight.
But data doesn’t speak for itself. At some point, there will be expert failure: Solutions require data, but may degrade with too much. The largest annoyance is the overblown concept of the data scientist. Data scientists, in the traditional sense, are academic researchers. In the Big Data industry they apply existing algorithms and techniques to data from traditional and new sources. Unfortunately, they usually report to people who have no idea what they are talking about.
In subsequent research I will describe the changes in “predictive” modeling brought about by Big Data and draw some conclusions about how it affects the construction, delivery and uses of decision management.