Saturday, August 7, 2010

Friedrich Hayek at the Safeway

Over the last couple of weeks, I have been discovering that it has been getting harder to find several of my favorite items during my weekly supply run to Safeway. When I ask about those items, the response I get is either a blank stare or a pained look while explaining that a branch store has no control over what the warehouse decides to send them to put on their shelves. From this I conclude that the decisions made at the warehouse level are probably all determined by yet another complex piece of software that no one really understands, at least not at any useful operative level.

About a month ago I wrote a piece in which I made fun of AdSense for placing an ad on one of my pages that happened to be for a business (Anthem Blue Cross) that I was attacking with my usual polemic vigor. This was again a matter of relegating a highly individualistic decision to the generalized algorithmic abstractions of a piece of complex software. (This, however, was probably a case where at least one person, and probably several, in Google knows very well what the software does; but Google is very close-lipped when it comes to sharing that information with anyone "on the outside.") I made the assumption that the software was probably based on some particularly sophisticated approach to statistical analysis and then cited one of my favorite sources for critical review of the limitations of statistical theory, Friedrich Hayek. This is the passage I quoted:

Far from dealing with structures of relationships, statistics deliberately and systematically disregard the relationships between the individual elements. It is, to repeat, concerned with the properties of the elements of the "collective", though not with the properties of particular elements, but with the frequency with which elements with certain properties occur among the total. And, what is more, it assumes that these properties are not systematically connected with the different ways in which the elements are related to each other.

This is what Safeway's current approach to inventory does not "get." There is no doubting that their customer base is a "collective;" and I am sure that there is an abundance of useful knowledge about the properties of that collective. None of that knowledge, however, addresses those "properties of particular elements;" and those "particular elements" are both the individuals who have decided to shop at Safeway and the reasons for their all making the same decision (which are not likely to be the same reasons). The problem with looking only at the forest and ignoring the trees, so to speak, is that "feeling like an individual" is often one of the reasons why a customer chooses to shop at a particular store. So Safeway may be undermining the criteria according to which customers choose to come to their stores.

From a technology point of view, the Kool-Aid that is poisoning the well of Safeway's "customer relationships" may very well be that of data mining. This particular brew of Kool-Aid became very appealing on the basis of a cute anecdote about coincidental purchases of Pampers and beer that had been discovered by a data mining technique. On the other hand I remember an old Car 54 Where Are You? episode that demonstrated both clearly and humorously that what you learn from a survey is not necessarily the same thing you learn by having conversations with the folks on the street; and that particular narrative played out again in the far more serious setting of controlling drug crime during the fourth season of The Wire. Where Safeway is concerned, we are not talking about folks on the street; we are talking about customers in the aisles. The Kool-Aid of data mining clouds the brain into believing that all you need to know about those customers can be inferred from digital records of sales transactions. Will the senior managers responsible for those warehouses ever realize that it is time to go into the stores and see what the customers are actually doing, rather than treat them all as abstractions in a massive database?

1 comment:

Ginsights said...

Of course, a result from a Safeway data mining exercise can only show what people bought, not what they wanted to buy and could not, because it was no longer available.
Looking at sequence of analyses will then show that the decision was correct, because nobody is buying it now.