> "The path of least resistance is to try and gain seniority at small or obscure companies. From there you create legitimacy for your seniority in larger companies. Experience in the larger company will in turn make smaller shops take a chance on you for a more senior position."
Working in larger companies earlier on - when you're fresh out of college, for example - can often ensure you end up learning a very tiny portion of the development lifecycle. Moving from there to a smaller outfit, where you would assume greater responsibility, would leave you realizing that you know very little about the big picture.
I've observed the opposite. I found that working in small companies blocked me from developing large amounts of useful skills (scaling, i18n, performance tuning, latency optimization, security, etc.) while working at a large company has exposed me to them and also opened up the full development lifecycle, from embedded prototyping frameworks to large-scale refactoring of legacy systems.
Of course, much of that may be because the small companies were two startups that went nowhere, and the large company was Google. YMMV.
> I found that working in small companies blocked me from developing large amounts of useful skills (scaling, i18n, performance tuning, latency optimization, security, etc.)
I think it really depends on what sort of company you work for. Small companies are highly variable, but there are many where the product is expected to scale from the start (because scalability is a functional feature of the product), have internalization from the start (because of the market it's aimed for), be secure and offer a latency SLA from the start etc. There are frequently open source projects that contain great deal of legacy code that have to be revived and modernized.
My previous could be described that way despite having been started by a few hackers. Of course they weren't in consumer web area: the product is an email/web security system: obviously it has to be secure, it can't be slow, it should deal with messages in different languages and it has to scale to email volumes of government organizations, universities and f1000 companies. Since we were in the email sphere, most of us or code base had been in C++ and Perl, which (by the virtue of the languages' age) had a great deal of legacy code out in the open that was nonetheless very useful.
It may be the case that these are after thoughts in small consumer Internet companies, but there are plenty of companies which employ similar skillset (especially if the application has a web UI and is deployed "in the cloud") who have to deal with these issues form the start.
I now work for a private consumer Internet company at this time where all of these issues also matter tremendously, but we are medium size at this point. Interestingly enough I've learned a lot more of "how to scale" here than I have at Yahoo: at first it was surprising, but realized it was almost tautological -- scale is about the first derivative, not the absolute site; going from 20,000 machines to 25,000 machines (not real numbers, just an example) is simpler than going from 1000 to 2000 machines. I can't just say "let's use this wonderful partitioned database and this wonderful data replication bus", I have to build those tools first :-)
A formula that works for me is: take an offer where you have the opportunity to learn the most. All else is fluff (they can talk about how great it is that they're small company, but they can still be structured in the same way that a big company is). It's also a good idea to work at large, small and medium companies in different industries to see what suits you better rather than make a decision based on what someone on the Internet (including some bozo calling himself after a libc function) wrote.
> scale is about the first derivative, not the absolute site; going from 20,000 machines to 25,000 machines (not real numbers, just an example) is simpler than going from 1000 to 2000 machines.
I've never worked on anything bigger than 10 machines or so, but is this really true? I would think the difficulty of scaling is not any kind of smooth function, but very easy or very hard depending on the bottlenecks you hit at any particular point.
lemma: doesn't every application have some theoretical limit about how big it can scale within the limitations of current network and computation hardware?
> I've never worked on anything bigger than 10 machines or so, but is this really true? I would think the difficulty of scaling is not any kind of smooth function, but very easy or very hard depending on the bottlenecks you hit at any particular point.
You are right it's not a smooth function. This isn't just about numerical aspect, it's figuring how to get over a hump: going from one datacenter to two is much harder (not even twice as hard) than adding a rack to two datacenters or adding a fourth datacenter when there are five.
In other words, when you're at scale you have tools available to operate at that scale. When you find yourself having to scale out without having done so in the past, you have to build those tools.
Comments
> "The path of least resistance is to try and gain seniority at small or obscure companies. From there you create legitimacy for your seniority in larger companies. Experience in the larger company will in turn make smaller shops take a chance on you for a more senior position."
Working in larger companies earlier on - when you're fresh out of college, for example - can often ensure you end up learning a very tiny portion of the development lifecycle. Moving from there to a smaller outfit, where you would assume greater responsibility, would leave you realizing that you know very little about the big picture.
I've observed the opposite. I found that working in small companies blocked me from developing large amounts of useful skills (scaling, i18n, performance tuning, latency optimization, security, etc.) while working at a large company has exposed me to them and also opened up the full development lifecycle, from embedded prototyping frameworks to large-scale refactoring of legacy systems.
Of course, much of that may be because the small companies were two startups that went nowhere, and the large company was Google. YMMV.
> I found that working in small companies blocked me from developing large amounts of useful skills (scaling, i18n, performance tuning, latency optimization, security, etc.)
I think it really depends on what sort of company you work for. Small companies are highly variable, but there are many where the product is expected to scale from the start (because scalability is a functional feature of the product), have internalization from the start (because of the market it's aimed for), be secure and offer a latency SLA from the start etc. There are frequently open source projects that contain great deal of legacy code that have to be revived and modernized.
My previous could be described that way despite having been started by a few hackers. Of course they weren't in consumer web area: the product is an email/web security system: obviously it has to be secure, it can't be slow, it should deal with messages in different languages and it has to scale to email volumes of government organizations, universities and f1000 companies. Since we were in the email sphere, most of us or code base had been in C++ and Perl, which (by the virtue of the languages' age) had a great deal of legacy code out in the open that was nonetheless very useful.
It may be the case that these are after thoughts in small consumer Internet companies, but there are plenty of companies which employ similar skillset (especially if the application has a web UI and is deployed "in the cloud") who have to deal with these issues form the start.
I now work for a private consumer Internet company at this time where all of these issues also matter tremendously, but we are medium size at this point. Interestingly enough I've learned a lot more of "how to scale" here than I have at Yahoo: at first it was surprising, but realized it was almost tautological -- scale is about the first derivative, not the absolute site; going from 20,000 machines to 25,000 machines (not real numbers, just an example) is simpler than going from 1000 to 2000 machines. I can't just say "let's use this wonderful partitioned database and this wonderful data replication bus", I have to build those tools first :-)
A formula that works for me is: take an offer where you have the opportunity to learn the most. All else is fluff (they can talk about how great it is that they're small company, but they can still be structured in the same way that a big company is). It's also a good idea to work at large, small and medium companies in different industries to see what suits you better rather than make a decision based on what someone on the Internet (including some bozo calling himself after a libc function) wrote.
> scale is about the first derivative, not the absolute site; going from 20,000 machines to 25,000 machines (not real numbers, just an example) is simpler than going from 1000 to 2000 machines.
I've never worked on anything bigger than 10 machines or so, but is this really true? I would think the difficulty of scaling is not any kind of smooth function, but very easy or very hard depending on the bottlenecks you hit at any particular point.
lemma: doesn't every application have some theoretical limit about how big it can scale within the limitations of current network and computation hardware?
> I've never worked on anything bigger than 10 machines or so, but is this really true? I would think the difficulty of scaling is not any kind of smooth function, but very easy or very hard depending on the bottlenecks you hit at any particular point.
You are right it's not a smooth function. This isn't just about numerical aspect, it's figuring how to get over a hump: going from one datacenter to two is much harder (not even twice as hard) than adding a rack to two datacenters or adding a fourth datacenter when there are five.
In other words, when you're at scale you have tools available to operate at that scale. When you find yourself having to scale out without having done so in the past, you have to build those tools.