We use the Web Audio API, HTML5 Canvas (with PIXI.js) and MediaRecorder to render and play-back generative music videos on the client side. However, it turns out that MediaRecorder can only work in real time, so unless you have a reasonably beefy computer, your video render might suffer from frame drops and random hangups. In an effort to eliminate these frame drops we moved the same js rendering code that we use on the frontend into a headlessly run Firefox instance wrapped in some additional node.js logic for controlling logic flow. That required splitting the rendering process into an offline pre-processing step for FFT analysis on the audio stream, a separate rendering job for rendering the frames themselves, then using ffmpeg to combine video frames, audio data and remuxing timestamps to ensure proper playback speeds.
Comments
There is nothing there to explain how it works. Does anybody have some information?
We use the Web Audio API, HTML5 Canvas (with PIXI.js) and MediaRecorder to render and play-back generative music videos on the client side. However, it turns out that MediaRecorder can only work in real time, so unless you have a reasonably beefy computer, your video render might suffer from frame drops and random hangups. In an effort to eliminate these frame drops we moved the same js rendering code that we use on the frontend into a headlessly run Firefox instance wrapped in some additional node.js logic for controlling logic flow. That required splitting the rendering process into an offline pre-processing step for FFT analysis on the audio stream, a separate rendering job for rendering the frames themselves, then using ffmpeg to combine video frames, audio data and remuxing timestamps to ensure proper playback speeds.