Inside the Garden of Good and Evil
Holding up a mirror to the world with Dan Tyminski
When I share a video or play someone a song in person, I like to read their faces for indications of enjoyment. Are they feeling the track? Will they laugh at a certain joke? Are they paying attention? You open your taste up for critique and feel triumphant when you trigger and share certain emotions.
Trying to simulate a similar experience on the web is difficult because of the various layers of distractions. Think about how adamant we get to make sure someone is focused on this moment in the real world. PUT YOUR PHONE DOWN! PAY ATTENTION! etc. The web impedes on our ability to grab that person’s attention because it is infinitely distracting. Are they on Messenger? How many tabs do they have open? Are you seriously playing a game right now?!
I’m a big fan of the “listening party.” Let’s round up a small group of fans and give them a first listen of the record at an interesting physical location. Everyone knows why they are there. They are excited and focused. Even Kanye West knows that the first playback of a record should be an event. We can’t forget that recorded music is art and while we will consume it personally from our pocket, it should have a grand opening as well.
Tyminski
Dan Tyminski is someone you don’t know that you know. He was the singing voice of George Clooney in the Coen brothers’ 2000 crime comedy O Brother, Where Art Thou? and provided the updated rendition of “Man of Constant Sorrow.” He’s a member of Alison Krauss and Union Station and has won 14 Grammy awards for solo and collaborative projects. He’s a bluegrass superstar but has also contributed vocals to an Avicii track. I think you can say he’s hyper-creative and curious.
That curiosity led Dan to put together his new solo masterpiece, Southern Gothic. This genre defying offering combines a modern sound and Dan’s impeccable vocals, with lyrics challenging the good and evil qualities of the south and a True Detective aesthetic. As Dan explains it, the music is like, “holding up a mirror to society without judgment or opinions.”
I was originally approached to help with this release by contributing code to a series of listening events held throughout the US. Universal Nashville and I developed screens that would filter the video image of passerbys in the aesthetic of Southern Gothic. The result was cool but limited to those events. However, I was granted the opportunity to create a digital version of this experience as well to coincide with the album release.
Seeing the impact from the listening events and latching onto Dan’s statement of “holding up a mirror to society,” I proposed a concept that would cast the filtered faces of real-time listeners into a gothic scene while allowing users the chance to stream tracks from the new album. This homage to Turntable.fm and Chat Roulette yields an experience that feels both intimate and odd. You can’t speak. Instead you are invited to focus on the music while peering through mirrors of others doing the same.
Visit the garden today from Google Chrome on your Desktop or Safari from an iPhone running iOS 11 and continue reading to learn how we built it.
Video Conference
The thought of developing an instant-on video conference experience into a website completely blows my mind but that’s precisely what Twilio set out to do when developing their Video product.
Twilio Video builds off the evolving standard of WebRTC to create a developer friendly SDK for connecting users around audio and video on their devices. Rather than provide an entire tutorial here, I would highly suggest checking out Rob Brazier’s original SIGNAL talk on the subject and then diving right into the API docs. You’ll learn about creating Rooms and adding participants to them. Did I mention it works on iPhone’s running iOS 11? 🔥
One small gotcha from my development was a need to crop all of the various video streams coming through down to a certain size which I would use as a texture later in Three.js. WebRTC doesn’t exactly make this bit clear but a bit of Stack Overflowing yielded the following gem:
video.onloadedmetadata = function(){
// this.videoHeight;
// this.videoWidth;
}That should allow you to sniff out the actual dimensions of the dynamically attached video streams.
Blending
The next step involved drawing those streams of video onto temporary HTML5 canvases and blending the video from our currently playing album video. As it turns out, it’s really easy to draw video to canvas. You simply call the drawImage method and pass in the video element as the img parameter. So if we consider these graphic layers, the user layer is drawn first and then the texture layer is blended on top. We can change the blend mode by changing the globalCompositeOperation function.
var canvas = document.createElement('canvas');
canvas.height = 128;
canvas.width = 128;
var context = canvas.getContext('2d');function render() {
context.drawImage(userVideo, 0, 0, 128, 128);
context.globalCompositeOperation = 'multiply';
context.drawImage(textureVideo, 0, 0, 128, 128); requestAnimationFrame();
}
And just like that, we’ve got a real-time video conference being drawn and textured through HTML5 canvas. However, our computer isn’t on fire yet so let’s go ahead and render those canvases in 3D. 😅
Three.js
I’m about the farthest thing from being an expert at Three.js but I did spend a bit of time in the 3D world recently due to ARKit… enough time to know how to buy a low poly tree model off Turbosquid. Find the tree you like and get it imported into your scene using the OBJLoader.
From here, you’ll want to choose where you think the “mirrors to society” will be hanging. Oh, and these mirrors are actually just plain ol’ Planes which we’ll map our canvases to. First, initialize the geometry of your plane so it fits with the sizing of your scene. Then attach your video canvas to a texture and position the it so sits in the middle of the mirror. Map that texture to a material and create your plane mesh. Finally, position the plane in your scene.
var geometry = new THREE.PlaneGeometry(60, 80, 1, 1);var texture = new THREE.Texture(canvas);
texture.repeat.x = 96 / 128;
texture.offset.x = 0.125;var material = new THREE.MeshBasicMaterial({
map: texture
});var plane = new THREE.Mesh(geometry, material);// position your plane
One more thing. You’ll want to update your texture as part of your render loop. You can do this by simply adding the line texture.needsUpdate = true after you complete the compositing of your video content.
Thanks
Thanks to the Universal Nashville team for empowering creativity for both Tyminski’s album and this experiential campaign. This album deserves it’s moment and I’m honored to be able to contribute a bit of my time to assist in that. Southern Gothic is available now.
