Statistics. Seriously, I really wish I had studied more of it in school. Basically goes for anyone in the STEM field, IMO.
I really wish more people invested in statistics and data analysis classes. People take you more seriously in a business setting when you can say "Email A resulted in a response rate of 80%". I usually hear "We think Email A is better because we feel it in our gut".
Ok, not those words exactly, but that's the point. Looking at data, understanding it, and directly applying it to your job is a hugely underrated skill.
I went to school for statistics, and work in "analytics" (a catch all term for anything from basic reporting to analytics infrastructure to conversion rate optimization experiments).
You're absolutely right in that people tend to take you more seriously if you come with numbers. But it's such a kangaroo court[1] that it drives me nuts. The instrumentation and implementation to support that sort of data-driven approach is usually far too lacking to give it the amount of merit it receives. Once you take do a first principle's sanity check of things, you learn that no one on the business side has a solid understanding of what "response rate" is actually referring to. Then when you look at the technical implementation, you realize that there's little reconciliation between what it's actually representative of and what anyone on the business side think it's representative of.
Never underestimate someone's gut feeling, especially so if it's from an individual in the trenches. More often than not, dissonance between gut feelings and data point to an issue with the data. Not necessarily that the data is wrong, just that it isn't fully representative of the context it's being collected in and should be trusted accordingly.
Also went to school for statistics and wanted to echo this. There's some neat stuff in the field. It has zipitty-doo-dah to do with "statistics" in the common parlance. A lot of businesses would be better off if everyone one day forgot what the "%" character meant.
This. My two biggest takeaways from years of stats classes were that, when applied right, you can do some really fascinating things with statistics. And that no one ever applies things right, so take any stats you come across with a grain of salt.
After years of professional experience, those are still the two biggest takeaways from stats. The only thing that's been added to the list is the fact that more often than not, statistics/data science is just a political tool to justify budget increases and absolve decision makers of culpability.
Any recommendations for somebody who ignored stats more than he should have as an undergraduate and wants to try to catch up? I've gone through and worked all of the exercises in my old calculus textbook so I have a good (fresh) handle on calculus now. Where's the best place to go next?
For years I thought I had to do it the way the big boys do, studying university text books and referencing papers in the field and putting a bunch of mathematical notation in my designs and emails. While also knowing all the foundations underneath. Unsurprisingly, that didn't really go anywhere, except hours lost fiddling with Word's equation editor (or worse, Latex).
What did help me was reading a few (really a few - just 2 or 3) simple, applied books; a 'statistics for dummies' (literally, the 'for dummies' book), a textbook used in undergrad business courses ('<something something> business analytics' I think?) and a book that applied all the stats to the field I was working on at the time (transportation modeling). Just being able to apply a linear regression (as in, actually being able to estimate the parameter on a single regressor in a simple data set) got me much further than all the times I thought 'whoops, getting into optimization now, better put this aside and first get a graduate level understanding of linear algebra'. And in a week instead of 2 years, too - quite important to keep your motivation up when you're not a full time student any more.
So while the above is not 'advice', it is my personal experience that when learning applied maths at a later age, it was better for me to focus on application and taking shortcuts even if that meant not fully knowing or understanding what was happening underneath - as intellectually unsatisfying and 'dirty' that felt at the time.
Understanding the distributions helped me the most. My favorite class was something like ‘single point estimates’ because it covered a lot about why mean and std dev are so powerful, but also sometimes so broken at conveying meaning.
Depends on the context for trying to catch up; both where you're at in life and what you're wanting to get out of it.
If you want to connect, feel free to reach out to the email in my profile. I may not be the best resource for best places to go next depending on what you're after, but may be able to help out.
A few general points though:
- Every field has it's own flavor of statistics. Supply chain, marketing, industrial engineering, business operations, finance, etc. Most practitioners will bastardize a technique or methodology common in their field before reaching out to another one for something more appropriate. Keep this in mind when looking at things, as you can find a lot of cases where the general premise for a technique is no longer valid, but practitioners are still going on momentum. You can also find some really neat nuggets/advancements that can be generalized and applied to another field. Although this can be difficult to suss out, as every industry tends to develop their own vernacular to refer to a particular set of base statistical techniques.
- Focus less on the math and more on the applicability of a particular technique or methodology to a situation. Generally speaking, statistical techniques are nothing more than sophisticated heuristics. Their validity, applicability, and actionability are entirely dependent on the situation they're applied in and the particular heuristics (statistical techniques) chosen. Understanding the techniques that are out there, what their applications are, and what their limitations are is far more useful than focusing purely on the math. The math can always be looked up once you know what to look up.
- Design of experiments[1] is a critical and often overlooked concept. It's rarely done in practice, and even when it is it's rarely more than a superficial attempt. But is a hugely important concept to understand how to approach a problem space.
- The output of a statistical analysis ranges from "checks the box of measuring something but so disconnected from observed reality that it'll otherwise be ignored" to "interesting but not robust enough to make decision on" to "directionally accurate" to "willing to make decisions based on confidence intervals". Understanding where your analysis stands on that scale, and where it needs to stand to meet your needs, is critical. Align your efforts with your needs, and set expectations accordingly.
A bigger issue I've seen is that "gut feeling" comes from a misplaced sense of confidence. Like say, sample size. I can't tell you how many times I've heard engineers say "the data isn't significant because the sample size is too small." If you have the data, calculate the confidence interval!
Most of the time you don't need hundreds to thousands of data points to be reasonably confident, just a few dozen. I remember the example distinctly from my sophomore engineering stats course, I don't know why everyone else has forgotten it.
I think most people overlook/don't know that the needed sample size depends not just on the confidence you want, but also on how big the effect you want to measure is. E.g. if landing page A has a conversion rate of 50% and B has one of 55%, that's going to take a lot of sampling to prove. But if A has 40% and B has 80%, then that's going to show up in the samples very quickly. But exactly which question you ask affects the needed sample size greatly - e.g. showing that 'B performs better than A' will take fewer samples than 'B performs at least 20% better than A'. This makes it much harder to have a correct intuition about needed samples sizes.
In my experience when the data is very small it is almost always also biased towards how easy it was to gather, which also makes it non representative. Think about it, if it were as easy to let n=5000 as it were to let n=25, you would always pick 5000. You only pick n=25 because of the low effort involved, which often means proximity.
A very common example is when some software feature is A/B tested only internally, or even only tested on the team that developed it. It introduces a lot of bias in users’ technical competence, willingness to understand/understanding of the new behavior, how the environment is set up, etc.
Comments
I really wish more people invested in statistics and data analysis classes. People take you more seriously in a business setting when you can say "Email A resulted in a response rate of 80%". I usually hear "We think Email A is better because we feel it in our gut".
Ok, not those words exactly, but that's the point. Looking at data, understanding it, and directly applying it to your job is a hugely underrated skill.
I went to school for statistics, and work in "analytics" (a catch all term for anything from basic reporting to analytics infrastructure to conversion rate optimization experiments).
You're absolutely right in that people tend to take you more seriously if you come with numbers. But it's such a kangaroo court[1] that it drives me nuts. The instrumentation and implementation to support that sort of data-driven approach is usually far too lacking to give it the amount of merit it receives. Once you take do a first principle's sanity check of things, you learn that no one on the business side has a solid understanding of what "response rate" is actually referring to. Then when you look at the technical implementation, you realize that there's little reconciliation between what it's actually representative of and what anyone on the business side think it's representative of.
Never underestimate someone's gut feeling, especially so if it's from an individual in the trenches. More often than not, dissonance between gut feelings and data point to an issue with the data. Not necessarily that the data is wrong, just that it isn't fully representative of the context it's being collected in and should be trusted accordingly.
[1] https://en.wikipedia.org/wiki/Kangaroo_court
Also went to school for statistics and wanted to echo this. There's some neat stuff in the field. It has zipitty-doo-dah to do with "statistics" in the common parlance. A lot of businesses would be better off if everyone one day forgot what the "%" character meant.
This. My two biggest takeaways from years of stats classes were that, when applied right, you can do some really fascinating things with statistics. And that no one ever applies things right, so take any stats you come across with a grain of salt.
After years of professional experience, those are still the two biggest takeaways from stats. The only thing that's been added to the list is the fact that more often than not, statistics/data science is just a political tool to justify budget increases and absolve decision makers of culpability.
Any recommendations for somebody who ignored stats more than he should have as an undergraduate and wants to try to catch up? I've gone through and worked all of the exercises in my old calculus textbook so I have a good (fresh) handle on calculus now. Where's the best place to go next?
For years I thought I had to do it the way the big boys do, studying university text books and referencing papers in the field and putting a bunch of mathematical notation in my designs and emails. While also knowing all the foundations underneath. Unsurprisingly, that didn't really go anywhere, except hours lost fiddling with Word's equation editor (or worse, Latex).
What did help me was reading a few (really a few - just 2 or 3) simple, applied books; a 'statistics for dummies' (literally, the 'for dummies' book), a textbook used in undergrad business courses ('<something something> business analytics' I think?) and a book that applied all the stats to the field I was working on at the time (transportation modeling). Just being able to apply a linear regression (as in, actually being able to estimate the parameter on a single regressor in a simple data set) got me much further than all the times I thought 'whoops, getting into optimization now, better put this aside and first get a graduate level understanding of linear algebra'. And in a week instead of 2 years, too - quite important to keep your motivation up when you're not a full time student any more.
So while the above is not 'advice', it is my personal experience that when learning applied maths at a later age, it was better for me to focus on application and taking shortcuts even if that meant not fully knowing or understanding what was happening underneath - as intellectually unsatisfying and 'dirty' that felt at the time.
Understanding the distributions helped me the most. My favorite class was something like ‘single point estimates’ because it covered a lot about why mean and std dev are so powerful, but also sometimes so broken at conveying meaning.
Depends on the context for trying to catch up; both where you're at in life and what you're wanting to get out of it.
If you want to connect, feel free to reach out to the email in my profile. I may not be the best resource for best places to go next depending on what you're after, but may be able to help out.
A few general points though:
- Every field has it's own flavor of statistics. Supply chain, marketing, industrial engineering, business operations, finance, etc. Most practitioners will bastardize a technique or methodology common in their field before reaching out to another one for something more appropriate. Keep this in mind when looking at things, as you can find a lot of cases where the general premise for a technique is no longer valid, but practitioners are still going on momentum. You can also find some really neat nuggets/advancements that can be generalized and applied to another field. Although this can be difficult to suss out, as every industry tends to develop their own vernacular to refer to a particular set of base statistical techniques.
- Focus less on the math and more on the applicability of a particular technique or methodology to a situation. Generally speaking, statistical techniques are nothing more than sophisticated heuristics. Their validity, applicability, and actionability are entirely dependent on the situation they're applied in and the particular heuristics (statistical techniques) chosen. Understanding the techniques that are out there, what their applications are, and what their limitations are is far more useful than focusing purely on the math. The math can always be looked up once you know what to look up.
- Design of experiments[1] is a critical and often overlooked concept. It's rarely done in practice, and even when it is it's rarely more than a superficial attempt. But is a hugely important concept to understand how to approach a problem space.
- The output of a statistical analysis ranges from "checks the box of measuring something but so disconnected from observed reality that it'll otherwise be ignored" to "interesting but not robust enough to make decision on" to "directionally accurate" to "willing to make decisions based on confidence intervals". Understanding where your analysis stands on that scale, and where it needs to stand to meet your needs, is critical. Align your efforts with your needs, and set expectations accordingly.
[1] https://www.jmp.com/en_ch/applications/design-of-experiments...
A bigger issue I've seen is that "gut feeling" comes from a misplaced sense of confidence. Like say, sample size. I can't tell you how many times I've heard engineers say "the data isn't significant because the sample size is too small." If you have the data, calculate the confidence interval!
Most of the time you don't need hundreds to thousands of data points to be reasonably confident, just a few dozen. I remember the example distinctly from my sophomore engineering stats course, I don't know why everyone else has forgotten it.
I think most people overlook/don't know that the needed sample size depends not just on the confidence you want, but also on how big the effect you want to measure is. E.g. if landing page A has a conversion rate of 50% and B has one of 55%, that's going to take a lot of sampling to prove. But if A has 40% and B has 80%, then that's going to show up in the samples very quickly. But exactly which question you ask affects the needed sample size greatly - e.g. showing that 'B performs better than A' will take fewer samples than 'B performs at least 20% better than A'. This makes it much harder to have a correct intuition about needed samples sizes.
In my experience when the data is very small it is almost always also biased towards how easy it was to gather, which also makes it non representative. Think about it, if it were as easy to let n=5000 as it were to let n=25, you would always pick 5000. You only pick n=25 because of the low effort involved, which often means proximity.
A very common example is when some software feature is A/B tested only internally, or even only tested on the team that developed it. It introduces a lot of bias in users’ technical competence, willingness to understand/understanding of the new behavior, how the environment is set up, etc.
Also suggest reading "How to Lie With Statistics".
Classic, short book about fundamental stats concepts everyone should know.