Interesting coincidence. One of my own 'seeing the future' moments also came from OTOY. We were sitting in the Techcrunch house late on Friday afternoon and there wasn't a lot going on. We got an email in our tips mailbox from Jules, the founder and CEO of OTOY explaining that he was in San Francisco up from LA for the weekend presenting at a conference, where he had a booth setup.
He sent us some info on what OTOY was working on, something about graphics processors in a server setup that could stream games to any client. We didn't really understand what he was talking about, since 'cloud' and all that wasn't well defined at the time (this was in '08, IIRC), but there was enough there to pique our interest. We emailed back asking if we could see him that afternoon, and he said that shouldn't be a problem - he was setting the booth up that evening before the conference proper started on the weekend, and it might be the best time since it was quiet there.
We drove up, found his tiny booth - the smallest of the lot, in a dark corner, next to the much larger rooms being setup by ATI, NVidia etc. He had one screen, one PC, a couple of phones and a pocket PC (believe it was an iPaq). He plugged a playstation controller into the PC, opened up the OTOY gaming application and scrolled around a half-dozen or so games on the screen while he explained what was happening.
NVidia had built a server-side GPU cluster which they were trialling out. It was a number of servers each with a dozen or two GPU's built into them which would directly stream their output over the internet (compressed) to the client. He fired up Grand Theft Auto 3 and we watched him while he walked around the city, getting into cars, driving around and then hitting top speed over hills etc. while the framerate didn't miss a beat.
The debug stats in the top corner showed that he was using 60-80KB/sec of bandwidth while playing. He explained that each frame was being sent down over the internet from a co-lo center. The ping time was 15ms, meaning there wasn't much lag. The servers were processing the input, playing the game, and then sending the output back. The codec in the prototype was nothing more than a series of JPEG images sent at 15-60+ frames per second. They said then that they would have something much better than that at some point, and I guess this Javascript + WebGL is what he was referring to.
We still weren't super impressed, since he had a controller and a big screen, so he just as well could have been running a playstation or other console underneath the table.
But then he picked up one of the phones, opened a Java applet and connected to the same game. This time I held the phone while he controlled the player and we had this tiny phone that could not have had more processing power than an early 386 playing one of the newest computer games on the market at the time in full color, at full screen and at full speed. Totally and completely blown away. He switched in and out of other games within seconds.
The next demo was opening up a web browser, pointing it to the web address, then watching the same game render in ~30 fps as the browser pulled down JPEG images and refreshed with JPEG (again, the very simple 'codec'). Utterly amazing.
We ended up spending hours with him, while he explained their plans and how the future of gaming will all be server-side. Since then, I have seen the OTOY name come up again and again related to breakthroughs in cloud-based gaming. These guys are really smart, and really ambitious.
Techcrunch ended up writing up an intro to what we saw a couple of months later (the conference was a private conference, IIRC, and our meeting was off-the-record, if I remember correctly).
The reason why we didn't publish everything right away was because I was very skeptical about some of the claims. He was a very good salesperson, and it is always difficult to distinguish those who are smart and sales savvy but have substance, and those who are just full of shit. Jules and OTOY were definitely the former.
In terms of my questions, I was most concerned about the lag issue and how this would work with users around the world who couldn't be near servers. I went back and forward on tech issues for a while on email until I was very convinced that not only was this possible, but it was almost certainly the future of gaming (why have all these GPU's that can't be upgraded on desktops and in consoles when you can place them all in the cloud - better utilization, better upgrade path, better pricing - eg. all the cloud benefits).
Both OnLive and Gaikai tried this, but I don't think the lag nor the economics really works out.
Consider the following: Let's say the server gets 60fps, that is a frame is rendered in 16ms. If the compression takes another 16ms (charitable!), and the ping time is another 16ms, plus 16ms to decode on the client (plus 16ms to display!), then by the time you press a the 'fire' button to shoot, until you see the output, 5-6 frames have going by, or about 80-96ms, and that's assuming some pretty optimal best case.
OnLive was measured in the range of 150ms to 250ms input lag. This is just not acceptable for twitch gaming IMHO.
Now that's some data Carmack's not going to like ;-)
> The debug stats in the top corner showed that he was using 60-80KB/sec of bandwidth while playing. He explained that each frame was being sent down over the internet from a co-lo center.
I'm wondering what kind of miracle makes a video game display streamable at that speed with any form of acceptable quality (not even talking about 1080p@60Hz), especially given the following:
> The codec in the prototype was nothing more than a series of JPEG images sent at 15-60+ frames per second.
A single acceptably lossy typical JPEG game shot at 1080p is north of 600~800kB, while even one at 720p (typical of a '08 laptop) come up being 150~300kB. Below that you start to get noticeable artifacts.
Even if the game engine/platform knows what has been updated and only send deltas of sort (again taking the series of JPEG images case), you still have to account for the worst case, that is basically the whole screen changes at once, sixty times per second.
And indeed, in the Steam video, you can see throughput data at the top: when nothing's moving the value is as expected very low, but you immediately notice the peak at 9Mbps when the L4D video ends[0], with typical values during the video at 4~6Mbps, which is much more inline with a 1080p stream.
Even the fading Source logo produces a 2.5Mbps stream (which produces very small diff areas), at 85% quality. There seems to be some cap at 48Mbps.
You can also see various timings, and the adaptative compression kicking in on the 9Mbps section, where you can read:
There seems to be another timing value at the beginning, with an "encryption" label, hovering at 9ms.
BTW, what's with the monotonically increasing with time stream = ${p}% value? That makes it look like a streamed video, not an interactive one. Not being suspicious but it felt weird.
Uhm - I'd be inclined to hope that a cluster of GPUs (or just the high end ones being used for something like this) would be capable of rendering at rather higher than 60fps - whilst screen refresh rates may not match that, I would expect each frame to be rendered in rather less than 16ms.
Similarly, by using a decently parallelisable image compression codec (JPEG2000?) it should be possible to compress with the advantage of the many GPUs, too.
16ms transfer time is certainly kept generously low, but beyond that, 16ms decode is something I won't comment on due to the variability of implementation speeds (though it would be done in native code, of course) but 16ms further to display... 32ms to decode and display a JPEG? I'm pretty sure I've seen MJPEG streams at higher than 30fps before!
Back of a napkin they might be, but I think the figures in this post might require further thought.
Edit: again, to be clear, I'm not suggesting that they are using all of these advantages right now, but the idea that this can't reasonably be done for twitch gaming, even today, strikes me as bizarre, when they are trying to set up a system with whatever custom technology is required to make it work.
A GPU (or a cluster of GPUs) might be able to process, say, 10,000 frames in one second. This does not mean that the same GPUs can process one frame in (1/10,000) of a second.
Even with an infinite number of parallel GPUs, there will be an amount of latency required in copying memory to the GPU, running a job, and copying it back. After the frame is compressed, sent over the network, and picked up by the client, further delay (possibly tens of milliseconds) is added on before pixels appear on the screen.
These numbers dont add up. If the prototype codec essentially was just a series of JPEGs, i dont see where the bandwidth number of 60-80KB/s is coming from because it doesnt make sense. Each frame JPG would then have a size of 1-1.5Kilobyte considering a Framerate of 60.
OnLive and Gaikai worked quite good for what they were, but they had much higher bandwidth requirements and still didnt feature really high res.
I am probably misremembering, the comment above prompted me to try and work out what the rate could have been, it is likely that it is much higher. I tried to find the details but had no luck. They have since moved on from that prototype codec.
We did try out the web application from the office later on. It worked ok, but you could tell that bandwidth and lag were going to be issues.
Note to anybody else reading this as well: don't take my numbers as gospel, I probably shouldn't have included them since I was not 100% certain of them. What I do know for certain is that the method was just refreshing JPEG's and you could tell the quality was low (it looked great, but at times the picture was the equivalent of a full color photoshop photo exported at a quality level of 2 or 3).
The compression definitely wasn't working across frames, just each frame.
Comments
Interesting coincidence. One of my own 'seeing the future' moments also came from OTOY. We were sitting in the Techcrunch house late on Friday afternoon and there wasn't a lot going on. We got an email in our tips mailbox from Jules, the founder and CEO of OTOY explaining that he was in San Francisco up from LA for the weekend presenting at a conference, where he had a booth setup.
He sent us some info on what OTOY was working on, something about graphics processors in a server setup that could stream games to any client. We didn't really understand what he was talking about, since 'cloud' and all that wasn't well defined at the time (this was in '08, IIRC), but there was enough there to pique our interest. We emailed back asking if we could see him that afternoon, and he said that shouldn't be a problem - he was setting the booth up that evening before the conference proper started on the weekend, and it might be the best time since it was quiet there.
We drove up, found his tiny booth - the smallest of the lot, in a dark corner, next to the much larger rooms being setup by ATI, NVidia etc. He had one screen, one PC, a couple of phones and a pocket PC (believe it was an iPaq). He plugged a playstation controller into the PC, opened up the OTOY gaming application and scrolled around a half-dozen or so games on the screen while he explained what was happening.
NVidia had built a server-side GPU cluster which they were trialling out. It was a number of servers each with a dozen or two GPU's built into them which would directly stream their output over the internet (compressed) to the client. He fired up Grand Theft Auto 3 and we watched him while he walked around the city, getting into cars, driving around and then hitting top speed over hills etc. while the framerate didn't miss a beat.
The debug stats in the top corner showed that he was using 60-80KB/sec of bandwidth while playing. He explained that each frame was being sent down over the internet from a co-lo center. The ping time was 15ms, meaning there wasn't much lag. The servers were processing the input, playing the game, and then sending the output back. The codec in the prototype was nothing more than a series of JPEG images sent at 15-60+ frames per second. They said then that they would have something much better than that at some point, and I guess this Javascript + WebGL is what he was referring to.
We still weren't super impressed, since he had a controller and a big screen, so he just as well could have been running a playstation or other console underneath the table.
But then he picked up one of the phones, opened a Java applet and connected to the same game. This time I held the phone while he controlled the player and we had this tiny phone that could not have had more processing power than an early 386 playing one of the newest computer games on the market at the time in full color, at full screen and at full speed. Totally and completely blown away. He switched in and out of other games within seconds.
The next demo was opening up a web browser, pointing it to the web address, then watching the same game render in ~30 fps as the browser pulled down JPEG images and refreshed with JPEG (again, the very simple 'codec'). Utterly amazing.
We ended up spending hours with him, while he explained their plans and how the future of gaming will all be server-side. Since then, I have seen the OTOY name come up again and again related to breakthroughs in cloud-based gaming. These guys are really smart, and really ambitious.
Techcrunch ended up writing up an intro to what we saw a couple of months later (the conference was a private conference, IIRC, and our meeting was off-the-record, if I remember correctly).
The reason why we didn't publish everything right away was because I was very skeptical about some of the claims. He was a very good salesperson, and it is always difficult to distinguish those who are smart and sales savvy but have substance, and those who are just full of shit. Jules and OTOY were definitely the former.
In terms of my questions, I was most concerned about the lag issue and how this would work with users around the world who couldn't be near servers. I went back and forward on tech issues for a while on email until I was very convinced that not only was this possible, but it was almost certainly the future of gaming (why have all these GPU's that can't be upgraded on desktops and in consoles when you can place them all in the cloud - better utilization, better upgrade path, better pricing - eg. all the cloud benefits).
Here is the later story Techcrunch wrote:
http://techcrunch.com/2008/07/09/otoy-developing-server-side...
Here are all the OTOY stories at Techcrunch from the past 6 years, you can see the things they have been working on:
http://techcrunch.com/2008/07/09/otoy-developing-server-side...
Both OnLive and Gaikai tried this, but I don't think the lag nor the economics really works out.
Consider the following: Let's say the server gets 60fps, that is a frame is rendered in 16ms. If the compression takes another 16ms (charitable!), and the ping time is another 16ms, plus 16ms to decode on the client (plus 16ms to display!), then by the time you press a the 'fire' button to shoot, until you see the output, 5-6 frames have going by, or about 80-96ms, and that's assuming some pretty optimal best case.
OnLive was measured in the range of 150ms to 250ms input lag. This is just not acceptable for twitch gaming IMHO.
Now that's some data Carmack's not going to like ;-)
> The debug stats in the top corner showed that he was using 60-80KB/sec of bandwidth while playing. He explained that each frame was being sent down over the internet from a co-lo center.
I'm wondering what kind of miracle makes a video game display streamable at that speed with any form of acceptable quality (not even talking about 1080p@60Hz), especially given the following:
> The codec in the prototype was nothing more than a series of JPEG images sent at 15-60+ frames per second.
A single acceptably lossy typical JPEG game shot at 1080p is north of 600~800kB, while even one at 720p (typical of a '08 laptop) come up being 150~300kB. Below that you start to get noticeable artifacts.
Even if the game engine/platform knows what has been updated and only send deltas of sort (again taking the series of JPEG images case), you still have to account for the worst case, that is basically the whole screen changes at once, sixty times per second.
And indeed, in the Steam video, you can see throughput data at the top: when nothing's moving the value is as expected very low, but you immediately notice the peak at 9Mbps when the L4D video ends[0], with typical values during the video at 4~6Mbps, which is much more inline with a 1080p stream. Even the fading Source logo produces a 2.5Mbps stream (which produces very small diff areas), at 85% quality. There seems to be some cap at 48Mbps.
You can also see various timings, and the adaptative compression kicking in on the 9Mbps section, where you can read:
There seems to be another timing value at the beginning, with an "encryption" label, hovering at 9ms.BTW, what's with the monotonically increasing with time stream = ${p}% value? That makes it look like a streamed video, not an interactive one. Not being suspicious but it felt weird.
[0]: https://www.youtube.com/watch?v=FRtBuP2-_pA&feature=play...
Uhm - I'd be inclined to hope that a cluster of GPUs (or just the high end ones being used for something like this) would be capable of rendering at rather higher than 60fps - whilst screen refresh rates may not match that, I would expect each frame to be rendered in rather less than 16ms.
Similarly, by using a decently parallelisable image compression codec (JPEG2000?) it should be possible to compress with the advantage of the many GPUs, too.
16ms transfer time is certainly kept generously low, but beyond that, 16ms decode is something I won't comment on due to the variability of implementation speeds (though it would be done in native code, of course) but 16ms further to display... 32ms to decode and display a JPEG? I'm pretty sure I've seen MJPEG streams at higher than 30fps before!
Back of a napkin they might be, but I think the figures in this post might require further thought.
Edit: again, to be clear, I'm not suggesting that they are using all of these advantages right now, but the idea that this can't reasonably be done for twitch gaming, even today, strikes me as bizarre, when they are trying to set up a system with whatever custom technology is required to make it work.
A GPU (or a cluster of GPUs) might be able to process, say, 10,000 frames in one second. This does not mean that the same GPUs can process one frame in (1/10,000) of a second.
Even with an infinite number of parallel GPUs, there will be an amount of latency required in copying memory to the GPU, running a job, and copying it back. After the frame is compressed, sent over the network, and picked up by the client, further delay (possibly tens of milliseconds) is added on before pixels appear on the screen.
See the discussion around John Carmack's superuser post: http://superuser.com/questions/419070/transatlantic-ping-fas...
These numbers dont add up. If the prototype codec essentially was just a series of JPEGs, i dont see where the bandwidth number of 60-80KB/s is coming from because it doesnt make sense. Each frame JPG would then have a size of 1-1.5Kilobyte considering a Framerate of 60.
OnLive and Gaikai worked quite good for what they were, but they had much higher bandwidth requirements and still didnt feature really high res.
I am probably misremembering, the comment above prompted me to try and work out what the rate could have been, it is likely that it is much higher. I tried to find the details but had no luck. They have since moved on from that prototype codec.
We did try out the web application from the office later on. It worked ok, but you could tell that bandwidth and lag were going to be issues.
Note to anybody else reading this as well: don't take my numbers as gospel, I probably shouldn't have included them since I was not 100% certain of them. What I do know for certain is that the method was just refreshing JPEG's and you could tell the quality was low (it looked great, but at times the picture was the equivalent of a full color photoshop photo exported at a quality level of 2 or 3).
The compression definitely wasn't working across frames, just each frame.
They were using 60-80KBs? I'm skeptical since just sending reasonable-sounding sound takes 128kB/s or so, video much more.