That's all true, but I fear it might be painting a slightly unrealistically positive picture :-) There are also other kinds of reasons for why everything moves slower than one would hope, and they aren't quite as interesting.
Some system your project depends got deprecated again, and you need to decide whether to spend the time to migrate it to the replacement now, or hope it continues working until the replacement has been deprecated too. The development configuration files for other teams systems that you need to inherit from to bring up a full test cluster for your own system changed again, causing mysterious errors. Hopefully early on, rather than in the output of a data pipeline that takes 20 hours to run.
Or the system your working on is somehow relevant to the outside world. There's a legal/political/PR problem, and actual engineering effort is needed to implement the minimal hack to get rid of the problem as soon as possible. Or your project has inexplicably and apparently without anyone's knowledge been marked as falling under Sarbannes-Oxley, and now some change needs to be explained in great detail to a SOX auditor. Which would of course be much more important than any work you were actually hoping to get done :-)
The code base is so massive and there are still so many unexpected dependencies that even with distributed and cached compilation, just the processes responsible for the distribution might be enough to drive your machine to its knees. Of course there's a process in place to try to keep the dependencies from growing out of control, but that's then going to make your life much more difficult if you're working on the inconvenient parts of code nearer the center of the dependency graph, and thus more strictly regulated.
Also, there have historically been a lot of misguided projects that in practice failed but somehow didn't die. Usually they will get killed eventually, but they do tend to grow larger and live longer than they should. Maybe that's changed lately.
This is, of course, an unrealistically negative view. But at least in my experience these kinds of issues were more significant drags on productivity than the fun kind of issues you listed. Though that could be just because the boring kind of drag is much more noticeable. Google's a great place to work at in any case.
Yeah, I've run into all of those at times. There are some problems that are just inherent to working at a big company.
Most of the really frustrating ones are fairly avoidable if you (or your tech lead, if you're not leading the project) are reasonably adept at playing the big-company game. My usual reaction to deprecation of critical infrastructure is to complain loudly and persistently, and I can usually get a stay of execution for 6 months to a year, enough for someone else to be the early adopter and work out the kinks of the replacement, then move my own project over to it once it's stable. I've saved many hours for my team by continuing to use deprecated software until the replacement has been deprecated, and then switching straight to the replacement's replacement once it's stable.
The massive codebase is reasonably easy to navigate once you become adept with CodeSearch. Build performance problems are persistent, but I solve that by getting a new machine every time the opportunity presents itself and using them headless for builds & demos. I also open them up to my team so that other frontend engineers I work with can use them...I've got a mini server farm under my desk.
The legal/political/PR stuff can't really be helped, but I find that exhilarating in its own way, since you get to see how those systems work. I've learned to hate the Chinese government since working at Google, though.
BTW, all of these are potentially problems at a startup using open-source software and doing interesting things. It's just that there:
1.) Instead of worrying about your OS libraries being deprecated, you pull the version you want into your own source tree and never touch them, at least for several years. Any attempt to upgrade to a new version is met with intense pain, and so you push the problem off and hope that either your company will go bankrupt or you'll get bought by some big company and the programmers there will rewrite your software entirely.
2.) Instead of complying with the legal problems, you ignore them. If somebody pays attention to you, you're sued out of existence and all look for new jobs. If they don't, you go about your merry way and hope nobody looks too closely at you.
3.) Instead of your dependencies growing out of control....aww hell, every company I've worked at that's more than 6 months old has had dependencies that were out of control.
4.) Instead of being on a misguided project that in practice failed but didn't die, you're on a misguided project that will fail and die spectacularly. And then you get to do it again for the next startup.
Startups can be a lot of fun too, but understand, you're screwed regardless of what you do. :-) Engineering is hard. Let's go shopping!
Comments
That's all true, but I fear it might be painting a slightly unrealistically positive picture :-) There are also other kinds of reasons for why everything moves slower than one would hope, and they aren't quite as interesting.
Some system your project depends got deprecated again, and you need to decide whether to spend the time to migrate it to the replacement now, or hope it continues working until the replacement has been deprecated too. The development configuration files for other teams systems that you need to inherit from to bring up a full test cluster for your own system changed again, causing mysterious errors. Hopefully early on, rather than in the output of a data pipeline that takes 20 hours to run.
Or the system your working on is somehow relevant to the outside world. There's a legal/political/PR problem, and actual engineering effort is needed to implement the minimal hack to get rid of the problem as soon as possible. Or your project has inexplicably and apparently without anyone's knowledge been marked as falling under Sarbannes-Oxley, and now some change needs to be explained in great detail to a SOX auditor. Which would of course be much more important than any work you were actually hoping to get done :-)
The code base is so massive and there are still so many unexpected dependencies that even with distributed and cached compilation, just the processes responsible for the distribution might be enough to drive your machine to its knees. Of course there's a process in place to try to keep the dependencies from growing out of control, but that's then going to make your life much more difficult if you're working on the inconvenient parts of code nearer the center of the dependency graph, and thus more strictly regulated.
Also, there have historically been a lot of misguided projects that in practice failed but somehow didn't die. Usually they will get killed eventually, but they do tend to grow larger and live longer than they should. Maybe that's changed lately.
This is, of course, an unrealistically negative view. But at least in my experience these kinds of issues were more significant drags on productivity than the fun kind of issues you listed. Though that could be just because the boring kind of drag is much more noticeable. Google's a great place to work at in any case.
Yeah, I've run into all of those at times. There are some problems that are just inherent to working at a big company.
Most of the really frustrating ones are fairly avoidable if you (or your tech lead, if you're not leading the project) are reasonably adept at playing the big-company game. My usual reaction to deprecation of critical infrastructure is to complain loudly and persistently, and I can usually get a stay of execution for 6 months to a year, enough for someone else to be the early adopter and work out the kinks of the replacement, then move my own project over to it once it's stable. I've saved many hours for my team by continuing to use deprecated software until the replacement has been deprecated, and then switching straight to the replacement's replacement once it's stable.
The massive codebase is reasonably easy to navigate once you become adept with CodeSearch. Build performance problems are persistent, but I solve that by getting a new machine every time the opportunity presents itself and using them headless for builds & demos. I also open them up to my team so that other frontend engineers I work with can use them...I've got a mini server farm under my desk.
The legal/political/PR stuff can't really be helped, but I find that exhilarating in its own way, since you get to see how those systems work. I've learned to hate the Chinese government since working at Google, though.
BTW, all of these are potentially problems at a startup using open-source software and doing interesting things. It's just that there:
1.) Instead of worrying about your OS libraries being deprecated, you pull the version you want into your own source tree and never touch them, at least for several years. Any attempt to upgrade to a new version is met with intense pain, and so you push the problem off and hope that either your company will go bankrupt or you'll get bought by some big company and the programmers there will rewrite your software entirely.
2.) Instead of complying with the legal problems, you ignore them. If somebody pays attention to you, you're sued out of existence and all look for new jobs. If they don't, you go about your merry way and hope nobody looks too closely at you.
3.) Instead of your dependencies growing out of control....aww hell, every company I've worked at that's more than 6 months old has had dependencies that were out of control.
4.) Instead of being on a misguided project that in practice failed but didn't die, you're on a misguided project that will fail and die spectacularly. And then you get to do it again for the next startup.
Startups can be a lot of fun too, but understand, you're screwed regardless of what you do. :-) Engineering is hard. Let's go shopping!