ProofOfConcept

“ProofOfConcept” is my first virtual reality scene. I made it to prove out the tech stack I chose for making real-time music-driven VR experiences for web browsers.

It works in the Oculus Quest 2 browser and most desktop browsers.

Instructions

It takes a little time for the music get going so don’t turn up the volume too soon. Also note that the closer you get to the objects the music is coming from, the louder the music will get.

When the scene first loads you might see a button in the top left to unmute the sound. Click it to start the music.

If a VR device is available you’ll see a button in the bottom right for entering VR mode.

To look around on desktop and mobile devices, click and drag the mouse or drag your finger on the screen.

On desktop devices, press the WASD or arrow keys to move. Hold down the Shift or Space key to move faster, or use Caps Lock.

On VR devices, use the thumbstick to teleport.

Technical details

Javascript

I used two Javascript libraries: Csound for audio and Babylon.js for graphics.

My custom Javascript compresses to 116 kB. Source code is here.

Audio

The audio is generated in realtime using the WASM version of Csound. For the realtime graphics people out there, Csound is kind of like shaders for audio. In a browser it runs on the CPU in a separate AudioWorklet thread.

For the positional audio I reimplemented Google’s Omnitone library in Csound. It works pretty well and allows me to use a lot more sound sources than I can with other methods.

My composing workflow went like this:

1. Sequence music in Reaper DAW with plugins generated by Cabbage from my Csound scripts.
2. Bounce DAW settings and plugins to a monolithic Csound file using the steps listed here.
3. Copy/paste the monolithic Csound file and its corresponding JSON file into the Babylon.js project.

Graphics

The graphics are handled by Babylon.js. The JSON file generated by the bounced Csound script contains info needed for synchronizing the graphics to the audio. I tried sending messages directly from Csound to the graphics engine in realtime but they got bottlenecked so I embedded them into the project’s Javascript as JSON instead.