> “Frankly, Kai just made decisions, and it just happened
> without a lot of committee meetings,” Ng said. “The
> ability of individuals in the company to make decisions
> like that and move infrastructure quickly is something
> I really appreciate about this company.”
That was, initially, one of the more interesting things about Google. You wanted to start a project and could get resources to run it in the various data centers relatively easily. If a guy like Ng was frustrated inside of Google by 'committee meetings' keeping him from getting things done quickly, then that says something about the current state of affairs there.
Note that Ng did not say he was comparing to Google. A third-party made that claim:
“He ordered 1,000 GPUs [graphics processing units] and got
> them within 24 hours,” Adam Gibson, co-founder of
> deep-learning startup Skymind, told VentureBeat.
> “At Google, it would have taken him weeks or months to get that.”
According to Wikipedia [1], Ng built a 16,000-CPU-core system at Google. (It's unclear what the CPU<->GPU conversion factor is)
Google was once heavily invested in just saying no to GPUs across the board. They even had a convenient FAQ from their platform team filled with all sorts of out of date FUD demonstrating why GPUs at Google were simply never going to happen.
The hiring of Geoffrey Hinton changed all that with Deep Neural Networks which are easily 50x more efficient on GPUs. And in the meantime, Baidu demonstrated what one could achieve by integrating them top to bottom in their search engine while Google happily stuck its technological fingers in its ears singing la-la-la-la.
Given that Andrew Ng's lab's other achievement was to demonstrate that 3 quad GTX 680 workstations were equivalent to that 16,000 CPU core system at Google, Baidu seems like a really great place for him to go.
I thought that might be the reason, too, so I went to check the number of employees. According to Wikipedia, Baidu has "34,600 (As of March 31, 2014)", and Google "52,069 (Q2 2014)"
So Baidu is 2/3 of Google - it's not just the size.
My opinion is that "getting big" can either be an increase in the number of workers or the degree of maturity.
As a company becomes older, the need to run all kinds of crazy experiments suddenly doesn't matter anymore. They care more about stability and profit. Yes, there is Google X and I know Google still run all sorts of crazy data analysis. But the point is, they want to become more structural because of the number of projects needed to approve. If twenty researchers each request to run a project each takes 1M out of Google, those research better have some formal proposal or so so they can evaluate which gets priority. Also seniority? Two top talents can compete for the same resource but if one team makes more "promising" result than the other team does, the winner is obvious.
Now coming to BD. BD isn't exactly unlike Google. It's exactly China's Google. It has reached its own maturity, but when they want to expand westward and when they want to steal talents from Google, they will have to strike better deals, be it compensation, work freedom and resource support.
I will not be surprised a few years down the road Ng will find similar hurtles within BD once he's all "used up" (sorry for the lack of better word). But of course, what matters is what he can do now.
Ya, but one can hope they'd think in terms of quotas [e.g. You have XXXX of machine time and/or a $$$ budget of XXXXX to test ideas] rather than require committee meetings.
They require the committee meetings to set the quotas.
Quotas suffer from the standard budgeting problem: everybody inflates their needs and hoards resources so that if they need extra in the future, they don't have to go through the budgeting process again. As a result, when a new project comes along that's not just an off-shoot of an existing high-priority project, they find that they're starved out of quota. They don't have an existing quota because the project previously didn't exist, and then when they go to get some, they find that all machines are currently allocated because everybody else overbid to secure room for expansion.
Google was rolling out a fix for this process when I left them, but I'm not sure if I can share the details on it.
I know some bigger companies like HP make sure every department bills every other department in order to make sure that you don't get problems like you would in any centrally planned economy without a price mechanism.
This sounds fascinating. Off the top of my head, the only fix I can imagine is to punish people for overestimates, but that sounds harmful in other ways.
Comments
I found this an interesting quote:
That was, initially, one of the more interesting things about Google. You wanted to start a project and could get resources to run it in the various data centers relatively easily. If a guy like Ng was frustrated inside of Google by 'committee meetings' keeping him from getting things done quickly, then that says something about the current state of affairs there.Note that Ng did not say he was comparing to Google. A third-party made that claim:
According to Wikipedia [1], Ng built a 16,000-CPU-core system at Google. (It's unclear what the CPU<->GPU conversion factor is)
[1] http://en.wikipedia.org/wiki/Andrew_Ng
Google was once heavily invested in just saying no to GPUs across the board. They even had a convenient FAQ from their platform team filled with all sorts of out of date FUD demonstrating why GPUs at Google were simply never going to happen.
The hiring of Geoffrey Hinton changed all that with Deep Neural Networks which are easily 50x more efficient on GPUs. And in the meantime, Baidu demonstrated what one could achieve by integrating them top to bottom in their search engine while Google happily stuck its technological fingers in its ears singing la-la-la-la.
Given that Andrew Ng's lab's other achievement was to demonstrate that 3 quad GTX 680 workstations were equivalent to that 16,000 CPU core system at Google, Baidu seems like a really great place for him to go.
It's what happens when companies get big. Same thing happened at Microsoft.
I thought that might be the reason, too, so I went to check the number of employees. According to Wikipedia, Baidu has "34,600 (As of March 31, 2014)", and Google "52,069 (Q2 2014)"
So Baidu is 2/3 of Google - it's not just the size.
My opinion is that "getting big" can either be an increase in the number of workers or the degree of maturity.
As a company becomes older, the need to run all kinds of crazy experiments suddenly doesn't matter anymore. They care more about stability and profit. Yes, there is Google X and I know Google still run all sorts of crazy data analysis. But the point is, they want to become more structural because of the number of projects needed to approve. If twenty researchers each request to run a project each takes 1M out of Google, those research better have some formal proposal or so so they can evaluate which gets priority. Also seniority? Two top talents can compete for the same resource but if one team makes more "promising" result than the other team does, the winner is obvious.
Now coming to BD. BD isn't exactly unlike Google. It's exactly China's Google. It has reached its own maturity, but when they want to expand westward and when they want to steal talents from Google, they will have to strike better deals, be it compensation, work freedom and resource support.
I will not be surprised a few years down the road Ng will find similar hurtles within BD once he's all "used up" (sorry for the lack of better word). But of course, what matters is what he can do now.
Ya, but one can hope they'd think in terms of quotas [e.g. You have XXXX of machine time and/or a $$$ budget of XXXXX to test ideas] rather than require committee meetings.
They require the committee meetings to set the quotas.
Quotas suffer from the standard budgeting problem: everybody inflates their needs and hoards resources so that if they need extra in the future, they don't have to go through the budgeting process again. As a result, when a new project comes along that's not just an off-shoot of an existing high-priority project, they find that they're starved out of quota. They don't have an existing quota because the project previously didn't exist, and then when they go to get some, they find that all machines are currently allocated because everybody else overbid to secure room for expansion.
Google was rolling out a fix for this process when I left them, but I'm not sure if I can share the details on it.
I know some bigger companies like HP make sure every department bills every other department in order to make sure that you don't get problems like you would in any centrally planned economy without a price mechanism.
Microsoft works in a similar way, so it's easy to hire vendors.
But if you don't have enough internal customers, you might have budget shortfalls, so guess what happens for vendors.
Fair enough, I don't work for Google so I have no clue how it works there. :)
I work at a place small enough where the budgeting process is for IT as a whole and we each have our corner to play in of fixed size.
Guess it happens when the company hires a certain level of managers/MBAs.
U* Code Purple. Cage matches for quota.