Thanks for the really thoughtful comment. Let me try and unpack your question into a few parts, and answer each one separately.
First of all, the fundamental feedback loop doesn't require A/B tests. What matters is that you act in a disciplined way to transform ideas into products, measure what happens, and learn for the next set of ideas. Over time, you should get faster at executing this feedback loop, not slower. A/B testing is a great methodology, but not the only one. You might take a look at Net Promoter Score (NPS) for example, as one alternate way of gauging customer reaction to the changes you're making. If you look at the actual practice of science, you'll notice that not all branches can do controlled experimentation. In cosmology, for example, they have to rely on "natural experiments" because they (so far) lack the tools to conduct experiments involving large gravitational masses, etc. Subjective forms of data-collection, like in-person interviews and usability tests, can provide "validated learning about customers" if you are disciplined about it.
Second, I'm not sure I agree that you can't do A/B split-tests in this situation. The number of customers you have per day is not really relevant - that only affects how long it takes you to get a statistically significant result. You might need weeks to get enough customers through your 50/50 test to get good data, but that doesn't necessarily mean that's a bad idea. It might "slow you down" from the point of view of coding, but if it prevents you from building a feature that nobody wants, that speeds you up much more.
In fact, I would work backwards from "what do I need to have in order to validate my hypotheses" and then structure the rest of your business around that. For example, you might not want to spend the dollars on AdWords to drive traffic to your business at this stage, because the ROI is not high enough yet. On the other hand, if increased traffic leads to rapid iteration which leads to customer validation, that might be a good trade-off. Do what you have to do to accelerate your learning.
Last, your fear about the differences in types of customers is important to address. Start getting clear about the "customer archetype" you think is most likely to use your product. What does a day in their life look like, for example? Why are they crazy enough to use your early-stage product, instead of the more rational thing which would be to buy from an established player?
If I had to guess, I would say that, most likely, your current customers all have something pretty specific in common. Although they may have wildly different demographics and usage patterns, it's likely that they are all early adopters of ... something. Otherwise, they wouldn't be wasting their time with your product. The more you understand what that is, the better you'll be able to tailor your product to their needs. But, more importantly, this commonality probably means your split-tests (and usability tests) have more validity than you think.
Last note, just because you _can_ run split-tests on trivial changes (like "whether or not I add a button somewhere") doesn't mean that you should use them for that purpose. You can also run split-tests on big, meaty changes that elicit a strong reaction from customers. And if you recall from basic statistics, the incidence rate of the thing being measured is just as important in determining significance as the sample size. Thus, if you're split-testing important things, you can get a good result with much smaller samples. And, as an early-stage startup, I'd maintain it's only worth testing things that (you believe) have a large impact.
Comments
Thanks for the really thoughtful comment. Let me try and unpack your question into a few parts, and answer each one separately.
First of all, the fundamental feedback loop doesn't require A/B tests. What matters is that you act in a disciplined way to transform ideas into products, measure what happens, and learn for the next set of ideas. Over time, you should get faster at executing this feedback loop, not slower. A/B testing is a great methodology, but not the only one. You might take a look at Net Promoter Score (NPS) for example, as one alternate way of gauging customer reaction to the changes you're making. If you look at the actual practice of science, you'll notice that not all branches can do controlled experimentation. In cosmology, for example, they have to rely on "natural experiments" because they (so far) lack the tools to conduct experiments involving large gravitational masses, etc. Subjective forms of data-collection, like in-person interviews and usability tests, can provide "validated learning about customers" if you are disciplined about it.
Second, I'm not sure I agree that you can't do A/B split-tests in this situation. The number of customers you have per day is not really relevant - that only affects how long it takes you to get a statistically significant result. You might need weeks to get enough customers through your 50/50 test to get good data, but that doesn't necessarily mean that's a bad idea. It might "slow you down" from the point of view of coding, but if it prevents you from building a feature that nobody wants, that speeds you up much more.
In fact, I would work backwards from "what do I need to have in order to validate my hypotheses" and then structure the rest of your business around that. For example, you might not want to spend the dollars on AdWords to drive traffic to your business at this stage, because the ROI is not high enough yet. On the other hand, if increased traffic leads to rapid iteration which leads to customer validation, that might be a good trade-off. Do what you have to do to accelerate your learning.
Last, your fear about the differences in types of customers is important to address. Start getting clear about the "customer archetype" you think is most likely to use your product. What does a day in their life look like, for example? Why are they crazy enough to use your early-stage product, instead of the more rational thing which would be to buy from an established player?
If I had to guess, I would say that, most likely, your current customers all have something pretty specific in common. Although they may have wildly different demographics and usage patterns, it's likely that they are all early adopters of ... something. Otherwise, they wouldn't be wasting their time with your product. The more you understand what that is, the better you'll be able to tailor your product to their needs. But, more importantly, this commonality probably means your split-tests (and usability tests) have more validity than you think.
Last note, just because you _can_ run split-tests on trivial changes (like "whether or not I add a button somewhere") doesn't mean that you should use them for that purpose. You can also run split-tests on big, meaty changes that elicit a strong reaction from customers. And if you recall from basic statistics, the incidence rate of the thing being measured is just as important in determining significance as the sample size. Thus, if you're split-testing important things, you can get a good result with much smaller samples. And, as an early-stage startup, I'd maintain it's only worth testing things that (you believe) have a large impact.
Does that help?
Further reading on the topic: http://startuplessonslearned.blogspot.com/search/label/split...
http://startuplessonslearned.blogspot.com/search/label/liste...
http://startuplessonslearned.blogspot.com/search/label/custo...