There is obvious up trending of terms like 'big data', 'predictive analytics' and 'data mining'. I have worked in this area since 1998. So here are a few thoughts:
- Good analytics (I'll combine the three terms into 'analytics' for the sake of simplicity) requires an understanding of the tools, as well as a significant understanding of statistics so that you know which analysis to pick. But in addition, it requires a lot of creativity (see my examples below) and a significant amount of time to analyze/slice/dice data in a zillion different ways.
- This is a huge opportunity. Much, much bigger than people realize and much bigger than past trends of new technologies like client-server in early nineties or web apps of 4-5 years back. Why? Because it has the power to affect business processes very powerfully.
- Example 1: I spent 10 months working for a $5B shipping company analyzing data from their Marketing department. I combined it with several hundred global data sources. I worked on over 100 hypotheses. At the end of it, I came up three specific actions that their existing customers take about 6 months before going to a competitor. The Marketing department was thrilled. They spent $17 Million coming with a plan to tackle this. It has been a few months since then; and they have not lost a single customer. This is a powerful proprietary competitive weapon for them now.
- Example 2: I analyzed 10 years of power meter reading data for a large utility company. I combined it publicly available data sources of power consumption of major appliances and census data on family composition/wealth for various neighborhoods. I was able to reliably predict the lifestyle of every family, down to whether the person living in the house streamed a movie on Friday evenings and a whole lot more. So the company decided to use this analysis to change their Direct Mailers with very specific, personalized offerings. Their response to the first test mailer sent to 10,000 people? Twenty seven percent!!! They predict that a significant portion of their profits would come from DM's.
What level of sophistication did you have to reach in the shipping company case. Was it, hypothetically speaking, the client starts using more than one other alternative shipper (fairly low sophistication) or was it multi factor stuff at levels of statistical skill that you need a Phd to grasp?
And thank you - these one off comments are why HN is such a valuable place to contribute to.
It was very sophisticated analysis. But let me emphasize two things:
- When you start, you are taking a leap into total emptiness. You explore a thousand different avenues, most of them are dead ends. Your day consists of massive amount of mental effort to stay focused, to stay sane, and to make good assumptions. Then you change tack/analyses continuously. You keep adding/deleting datasets. This is not theoretical statistics but sometimes you use arcane things; so you definitely need to be very strong at Stats. My personal dream is that one day, when I have time and money, I will use these methods to come up with a Meta- Statistics approach to empirical analysis -- i.e., to use Analytics to predict, based on the problem definition and available data sets, which methods to use.
- You have to have patient clients. Jumping 10 months into a project with absolutely NO guarantee of success is a huge leap of faith, financially speaking; but the rewards can be huge. One day you are still nowhere, and literally the next day, everything clicks, you check your conclusions once-twice-thrice, make a presentation to the client, and, boom!, your project is over.
Thank you. This does sound like an entire industry waiting to exist rather than a clever one off.
So how are you planning to go from tenuous projects to repeatable revenue (sorry someone asked me some hard questions today - paying it forward!). I am guessing that a 5bn dollar shipping firm that just stopped multi-million clients walking out the door is getting some serious payback - even if your daily rate was enormous.
So what about things like multiple clients simultaneously (hiring interns), building a capability maturity matrix to help clients increase their data sophistication (and naturally you charge a monthly retainer)
Just interested in knowing where you are taking this.
No, no such thing for me. I like to be the independent, incorruptible voice. Besides, I would hate dealing with employees. Man's gotta know his limitations. :)
Comments
There is obvious up trending of terms like 'big data', 'predictive analytics' and 'data mining'. I have worked in this area since 1998. So here are a few thoughts:
- Good analytics (I'll combine the three terms into 'analytics' for the sake of simplicity) requires an understanding of the tools, as well as a significant understanding of statistics so that you know which analysis to pick. But in addition, it requires a lot of creativity (see my examples below) and a significant amount of time to analyze/slice/dice data in a zillion different ways.
- This is a huge opportunity. Much, much bigger than people realize and much bigger than past trends of new technologies like client-server in early nineties or web apps of 4-5 years back. Why? Because it has the power to affect business processes very powerfully.
- Example 1: I spent 10 months working for a $5B shipping company analyzing data from their Marketing department. I combined it with several hundred global data sources. I worked on over 100 hypotheses. At the end of it, I came up three specific actions that their existing customers take about 6 months before going to a competitor. The Marketing department was thrilled. They spent $17 Million coming with a plan to tackle this. It has been a few months since then; and they have not lost a single customer. This is a powerful proprietary competitive weapon for them now.
- Example 2: I analyzed 10 years of power meter reading data for a large utility company. I combined it publicly available data sources of power consumption of major appliances and census data on family composition/wealth for various neighborhoods. I was able to reliably predict the lifestyle of every family, down to whether the person living in the house streamed a movie on Friday evenings and a whole lot more. So the company decided to use this analysis to change their Direct Mailers with very specific, personalized offerings. Their response to the first test mailer sent to 10,000 people? Twenty seven percent!!! They predict that a significant portion of their profits would come from DM's.
I am ... stunned.
What level of sophistication did you have to reach in the shipping company case. Was it, hypothetically speaking, the client starts using more than one other alternative shipper (fairly low sophistication) or was it multi factor stuff at levels of statistical skill that you need a Phd to grasp?
And thank you - these one off comments are why HN is such a valuable place to contribute to.
It was very sophisticated analysis. But let me emphasize two things:
- When you start, you are taking a leap into total emptiness. You explore a thousand different avenues, most of them are dead ends. Your day consists of massive amount of mental effort to stay focused, to stay sane, and to make good assumptions. Then you change tack/analyses continuously. You keep adding/deleting datasets. This is not theoretical statistics but sometimes you use arcane things; so you definitely need to be very strong at Stats. My personal dream is that one day, when I have time and money, I will use these methods to come up with a Meta- Statistics approach to empirical analysis -- i.e., to use Analytics to predict, based on the problem definition and available data sets, which methods to use.
- You have to have patient clients. Jumping 10 months into a project with absolutely NO guarantee of success is a huge leap of faith, financially speaking; but the rewards can be huge. One day you are still nowhere, and literally the next day, everything clicks, you check your conclusions once-twice-thrice, make a presentation to the client, and, boom!, your project is over.
Thank you. This does sound like an entire industry waiting to exist rather than a clever one off.
So how are you planning to go from tenuous projects to repeatable revenue (sorry someone asked me some hard questions today - paying it forward!). I am guessing that a 5bn dollar shipping firm that just stopped multi-million clients walking out the door is getting some serious payback - even if your daily rate was enormous.
So what about things like multiple clients simultaneously (hiring interns), building a capability maturity matrix to help clients increase their data sophistication (and naturally you charge a monthly retainer)
Just interested in knowing where you are taking this.
No, no such thing for me. I like to be the independent, incorruptible voice. Besides, I would hate dealing with employees. Man's gotta know his limitations. :)
Enjoy your freedom :-)