Building an NFT Powered Web AR Camera for the Deadmau5 Head5 Community
A Hands on Education in Web3
Two weeks ago, I published a post on Mirror which recounted the experience of minting my first NFT, a generative 3D head model from Deadmau5 under the moniker Head5. TL;DR: It was confusing and expensive but also well-produced and marketed. I sorta published the blog, dusted off my hands, and thought, “Well, that’s the end of that.” However, I am first and foremost, a web developer, and an idea began to materialize in my mind…
What if I developed a web app that allowed holders of one of these Head5 NFTs to wear them in augmented reality? Perhaps, owners could login with their crypto wallet and I would scan the blockchain for the heads they possessed and those NFTs would populate the user experience.
As it happens, I’ve been really interested in building these web ar-like camera experiences. I just gave it a try for Rezz and figured this would be the perfect opportunity to expand on the technique, while giving myself a crash course in some of the fundamentals of Web3. I also thought it would be a good opportunity to interact with other Head5 owners on the associated Discord, who I assumed would be more seasoned Web3 participants, and get some insights on this world I know little about.
The app was developed over the period of a week and lives at www.head5.camera. It was a huge hit with the Head5 community and management team. It also yielded many interesting insights about Web3 and NFTs so I was very happy to have spent time building it. Continue reading to find out how it all came together or jump down to the epilogue for a conclusion to my original Mirror post.
User Research and Scoping
Community response
I presented my rough idea to the community and was quite surprised by their first suggestion:
Don’t make it only for the NFT owners and don’t require a wallet connection.
First, some users had an issue with the inherent security risk of wallet authentication on such a simple offering. This is due to the many phishing scams that exist in the Web3 world. In addition, users in this community (and likely other NFT communities) care about the awareness and perceived value of their held NFTs. For this reason, many users wanted to make sure anyone could wear their heads and perhaps make an offer on them. Most conceded that some sort of wallet connection to verify their ownership (if they so choose) could be a good idea. Personally, I like a secure and accessible user experience so these suggestions sounded great.
Final Scope
The scope I decided on would allow anyone to provide a minted Token ID and gain access to an AR camera featuring the associated 3D head model. That user could then take a photo and download it. In addition, owners would be able to connect with their wallets to verify ownership and captured photos would include a little “verified” badge.
I was able to get to this version rather quickly and ended up expanding the scope to include: filter effects, video recording, branded frames, and simple social sharing. 😅 I had a good time making a little list of improvements every night and waking up to a strong coffee to implement them. Let’s go through some of those technical solutions.
Working with head5 Metadata and Models
Each Head5 NFT has some associated metadata and a GLTF model we can use to create our experience. At the core of the app is a Three.js powered rendering engine. Here’s how you can utilize the metadata and provided model to render one of these heads in a 3D scene.
Get Supply
First, it would be helpful to know the total amount of tokens which have been minted so far so we can make sure users are only trying to utilize NFTs which exist. In order to do this, we can use the PolygonScan API. All we need to do is provide the Head5 contract address to the appropriate method, along with our api key.
let { data: { result: supply } } = await axios.get(`https://api.polygonscan.com/api?module=stats&action=tokensupply&contractaddress=${CONTRACT_ADDRESS}&apikey=${API_KEY}`)
In addition to validation, I also used this data to add a secret feature. If you do not provide a token id on the main screen, the app will randomly provide one. It’s a great way to check out random Head5 models. There’s some really fucking weird ones.
Load Metadata from head5 API
The Head5 project also provides a little API to get the metadata for any minted NFT:
https://www.head5.io/api/metadata/0
In the response you’ll find the name
, description
, traits
, and associated model URL. I actually found that both the IPFS hosted image and model performed terribly so I ended up using the model file which is hosted on mau5trap.mypinata.cloud. To be honest, this took a bit of reverse engineering and code diving. I wish Head5 would simply offer up a more performant model URL. Anyway, with the GLTF model url handy, I was ready to load it on Three.js.
Loading GLTF model into Three.JS
Three.js is an open-source darling and the preferred way to create 3d scenes on the web. They provide a GLTF loader which you can use to load GLTF models into your scene. Check out the example they provide here.
// Initialize loader
let loader = new GLTFLoader()// Load the model
loader.load(MODEL_URL, gltf => {
// Add model to scene
scene.add(gltf.scene)
})
When I did this, I noticed that the model was quite dark and found out that I needed to adjust the renderer’s outputEncoding
setting to be THREE.RGBEFormat
. This worked well until I incorporated another visual into my scene (the camera feed) and it came through too light. I simply couldn’t figure out how to handle the conflicting image formats and I took my case to StackOverflow. Luckily, Don McCurdy had the answer. Once the main renderer was set to THREE.RGBEFormat
it was important to put any other texture in that format also. Once I adjusted my video texture’s encoding, everything looked great. All my model now needed was some lights.
Environment and Lighting
Looking closely at the internal choices of the provided Head5 model viewer in the NFT metadata, I was able to extract the exact lighting and environment texture setup. First, the lighting consists of four spotlights of varying intensity, placed around the scene. Let’s get those added.
// Initialize light params
let lights = [
{ intensity: 0.5, x: 30, y: 35, z: 50 },
{ intensity: 0.5, x: -30, y: 35, z: 50 },
{ intensity: 0.7, x: 0, y: 40, z: -40 },
{ intensity: 0.2, x: 0, y: -30, z: 0 }
]// Loop through lights
lights.forEach(light => {
// Initialize new spot light
let spotLight = new THREE.SpotLight(0xFFFFFF, light.intensity) // Position light
spotLight.position.set(light.x, light.y, light.z) // Add light to scene
scene.add(spotLight)
}
The cherry on top is the environment texture. This makes the 3d head model feel as if it is sitting in an environment. In this case, a cafe. In order to do this, we’ll load the same exact HDR file used on the Head5 model viewer and compile it into a cubeMap texture with a THREE.PMREMGenerator.
// Initialize generator
letgenerator = new THREE.PMREMGenerator(renderer)// Precompile shader
generator.compileEquirectangularShader()// Initialize loader
let loader = new RGBELoader()
// Load hdr
loader.load('/js/scene.hdr', texture => {
// Generate env map
let environment = generator.fromEquirectangular(texture).texture // Apply environment text to scene
scene.environment = environment
})
Effects
Did I say cherry on top? Sorry. There were actually a few more sprinkles to add. Three.js has a post-processing pipeline we can use to add extra effects to the app. I chose three effects that I thought would resonate with the Head5 community: glitch, pixel, and dotscreen. Three.js provides some excellent instructions for setting this up but in general, we’re replacing the normal rendering routine to that of a composer. First, import the necessary modules. Then, initialize the composer, an initial rendering pass of the scene, and a glitch pass on top.
// Initialize renderer
const composer = new EffectComposer(renderer)// Add render pass
const renderPass = new RenderPass(scene, camera)
composer.addPass(renderPass)// Add glitch pass
const glitchPass = new GlitchPass()
composer.addPass(glitchPass)
Finally, back in our render loop, we would replace the standard render with that of our composer. Dope.
composer.render()
Adding AR Functionality
Now that we know how to get our model into the scene, how do we make it move with the user’s face? While there are platforms like 8th Wall which can get you there with minimal work, I prefer to keep things open-source and roll it myself. The following is an evolution of the technique I employed on the Pure Moods Camera and my recent project for Rezz.
WebRTC Camera
First, we’re going to need access to the camera itself. We can do that by using WebRTC. Here’s the little promise method I’ve been using to get access to a user’s camera and properly resolve my app once the video is loaded.
return new Promise(async (resolve, revoke) => {
try {
// Get camera
this.stream = await navigator.mediaDevices.getUserMedia({
audio: false,
video: true
}) // Get video tag
let video = this.$refs.video // Replace video tag source
video.srcObject = this.stream // Video ready
video.onloadedmetadata = async () => {
// Resolve
resolve()
}
} catch (error) {
// Revoke
revoke(error)
}
})
Tensorflow for Face Tracking
Once we have access to the user’s video stream, we can use TensorFlow and MediaPipe to predict where the face and its landmarks are. Since I’m building this in JavaScript, I like to use the official JS package. First, we’ll load the TensorFlow model.
// Load TensorFlow model
const model = await faceLandmarksDetection.load(faceLandmarksDetection.SupportedPackages.mediapipeFacemesh, {
maxFaces: 1
})
And then, in our render loop, we can estimate where the user’s face is.
let predictions = await model.estimateFaces({
input: this.$refs.video,
flipHorizontal: true
})
When TensorFlow returns a valid prediction, it arrives in the form of 486 3D facial landmarks. Here’s an image which shows the location of each of these. This works great when dealing with 2D manipulation but we’re going to need a little help to use these landmarks easily in a 3D space.
FaceMeshFaceGeometry for Tracking Point
Luckily, Juame Sanchez has developed a Three.js helper for this TensorFlow data called FaceMeshFaceGeometry. This was discussed a bit more in my Rezz dev blog but we’re mostly interested in the ability to track specific points on our user in 3D space so we can attach our Head5 model to it. Once this helper is imported, we can pass those same predictions from above, directly to the model. Note: we’re also flipping the camera horizontally by setting the second param to true.
faceGeometry.update(predictions[0], true)
Then, back in our render loop once again, we’ll track the nose. Juame’s library will calculate a triangle defined by three provided prediction ids, and return a position
and an orthogonal basis rotation
. We can then use these values to adjust the head’s position and rotation in our scene.
// Track nose
let track = faceGeometry(5, 45, 275)// Reposition head
head.position.copy(track.position)// Rotate head
head.rotation.setFromRotationMatrix(track.position)
In addition, I followed the guidance of this pull request to incorporate scale also. Finally, we’ll use the translation functions of Three.js to tweak the position of our head as needed. The positioning I ended up with was based on my own head and it works for most. However, the community has asked for the ability to reposition and scale things themselves. Perhaps this could be added in the future.
Smooth Tracking
TensorFlow sends the head tracking data to you raw and this causes the Head model to shake around a bit. There isn’t a built in solution for this so I rolled one based on a moving average. Instead of using the latest nose tracking data, I store the last 10 results and then return the average position
, rotation
, and scale
of these. I feel like this is something that should be handled by a background process but for now I’m doing right in the client.
Capturing Photos and Videos
I’ve covered how I use HTML <canvas>
to compose new dynamic images in many dev blogs. It is such a wonderful technique and I evolved it in this app to also record video. Videos, as you know, are just a series of still image frames. First, let’s talk about rendering a single frame.
Rendering Frame
In this app, I have a renderPhoto
method which first places a screenshot of the Three.js canvas onto a new offline canvas. I found an excellent little library which abstracts the complicated code required to crop and cover the Three.js image into my frame composition. Thanks to Federico Brigante for that. Once that is drawn, I add the frame image itself on top. Then, I use the fillText
method of canvas to dynamically write in the current head’s unique head5.camera URL. Here’s an abstract version.
// Initialize photo canvas
let photoCanvas = document.createElement(‘canvas’)// Resize photo canvas
photoCanvas.height = 1080
photoCanvas.width = 1080// Get photo context
let photoContext = photoCanvas.getContext('2d')// Calculate draw parameters
let { width, height, x, y } = cover(900, 810, threeWidth, threeHeight)// Draw three canvas
photoContext.drawImage(this.$refs.three, 0, 0, threeWidth, threeHeight, 90 + x, 90 + y, width, height)// Draw frame image
photoContext.drawImage(frameImage, 0, 0)// Initialize font
photoContext.font = `24px Arial Black`
photoContext.textBaseline = 'bottom'// Head5.camera
photoContext.fillStyle = 'white'
photoContext.fillText(`head5.camera`, 90, 995)// Slash
photoContext.fillStyle = '#9B9B9B'
photoContext.fillText(`/`, 285, 995)// Token
photoContext.fillStyle = 'white'
photoContext.fillText(`${TOKEN_ID}`, 300, 995)
Before we discuss sharing this photo, let’s take a quick look at how video recording is handled.
Using RecordRTC to handle Canvas Recording
On the Pure Moods camera, I used MediaRecorder directly to create the video recording solution. To make things a bit easier on myself, I decided to use the RecordRTC library for this app. I suspect RecordRTC uses MediaRecorder also. Regardless, both solutions allow you to record video of a canvas. So, we can call our `renderPhoto` method above and simply record the `photoCanvas` to video. This actually worked surprisingly well and I was very happy to share code between photo and video capturing. First, initialize a recorder and tell it to record our canvas.
// Initialize recorder
const recorder = RecordRTC(photoCanvas, {
type: ‘canvas’
})
Then, when you’re ready, you can start recording.
recorder.startRecording()
Finally, you guessed it, we can stop recording and get the associated blob for the next step.
// Stop recording
recorder.stopRecording(() => {
// Get blob
let videoBlob = recorder.getBlob()
})
Sharing Content
Thank the web gods we have a simple way to share files on mobile devices now with the adoption of Web Share API level 2. To do this, we’ll initialize a video or photo file from a blob and then pass it to the share method. But first, we need to deal with the fact that different browsers will record different video types. I can get the type from the recorded blob but I really need to figure out the extension (webm
, mp4
, etc) also for sharing and downloading. Luckily, I found a little utility library called mime-types which does exactly this. I can pass it my type and it will return the associated extension.
// Check for native share
if (navigator.share) {
// Initialize file
let file = new File([blob], `head5-${TOKEN_ID}.${mime.getExtension(blob.type)}`, {
type: blob.type
})// Check that file can be shared
if (navigator.canShare && navigator.canShare({ files: [file] })) {
// Share file
navigator.share({
title: "Head5 Camera",
description: "Take a photo with your head5 avatar on.",
files: [file]
})
.then(() => {
console.log('Share succeeded')
})
.catch(error => {
console.error('Share failed')
})
} else {
// Download file
download()
}} else {
// Download file
download()
}
Notice how we make sure Web Share exists (and it can share this particular format of file) and fallback to simply downloading if not.
Download Content
Initiating a download is very simple. We’ll generate an object URL for the photo or video blob and then make a dynamic link tag which is clicked.
// Generate object URL for blob
let data = window.URL.createObjectURL(blob)// Initialize a tag
let link = document.createElement('a')// Update link params
link.href = data
link.download = `head5-${this.token}.${mime.getExtension(blob.type)}`// Click link
link.click()
Connecting Wallet
The last thing I want to discuss is how I integrated a wallet connection into the app. This world is completely foreign to me but I found the MetaMask documentation to be very well written and easy to follow. In the end, this integration is very simple. A user can connect with their wallet so we may know their wallet address. We can then use this address to figure out if and which Head5 NFTs they own. First, let’s get that address.
Getting Wallet Address
Correct me if I’m wrong but I believe the browser will expose an ethereum
global method from window
if an Ethereum wallet extension is installed or the user is browsing from within an Ethereum wallet app. Regardless, it’s just right there, and we can use it to request the proper permissions that lead us to the wallet address. This is wildly simple authentication.
return new Promise(async (resolve, revoke) => {
try {
// Identity Ethereum address
await ethereum.request({
method: 'eth_requestAccounts'
}) try {
// Get list of Ethereum addresses
const accounts = await ethereum.request({
method: 'eth_accounts'
}) // If first account exists
if (accounts[0]) {
// Save wallet
let wallet = accounts[0] // Load NFTs
loadNFTS() // Resolve
resolve()
} else {
// Revoke
revoke('Unable to get accounts.')
}
} catch (error) {
// Revoke
revoke('Unable to get list of addresses.')
}
} catch (error) {
// Revoke
revoke('Unable to identity your ethereum address.')
}
})
Loading a User’s NFTs from Moralis
Once we have the wallet address, we can use the developer-friendly Moralis API to query our user’s wallet for any owned Head5 NFTs. The API endpoint is exactly what you’d expect, down to the chain parameter. If any NFTs are found, I store them in an awaiting array.
return new Promise((resolve, revoke) => {
// Get head5 NFTs from user wallet
axios.get(`https://deep-index.moralis.io/api/v2/${WALLET}/nft/${CONTRACT_ADDRESS}?chain=polygon&format=decimal`, {
headers: {
'Content-Type': 'application/json',
'X-API-Key': API_KEY
}
})
.then(({ data }) => {
// If any NFTs exist
if (data.total) {
// Update NFTs array
nfts = data.result
} // Resolve
resolve()
})
.catch(error => {
// Revoke
revoke(error)
})
})
Checking Ownership
With our array of NFTs handy, we can simply check to see if the current token id is owned by the user and then adjust the experience accordingly. In our case, we augmented the renderPhoto
method to draw a little verified badge graphic on photos. Here’s that ownership check.
// Check to see if user owns NFT
return nfts.some(nft => {
return nft.token_id == TOKEN
})
Epilogue
Well, it’s been a wild two weeks since I published my original mirror post. I don’t think I’ve participated in a niche music community unofficially since the early 2000s. I wouldn’t call the Head5 discord active or rich in substance but the users there have been very helpful in expanding my knowledge of this world and helping get this app built and tested. Thanks to everyone there. I’ve shared a lot of these gained insights directly on Twitter from passively being told by someone on the Head5 team that maybe I should stop building the app (because their Snapchat integration was coming) to realizing that users who are doing this for financial reasons may not be compatible with fans doing this for other reasons. I recommend pursuing through those raw threads for more hot takes.
This project has not necessarily made me a believer in NFT ownership (I only purchased one other after this) but I can now see how my abilities might fit into this world without changing my core business much. These NFT campaigns, like an album campaign, can benefit from the experimental awareness marketing I focus on. In addition, I love building fun (useful?) utility apps around existing content and niche communities using evolving web standards. That is really interesting to me. What sort of experiences can NFTs unlock beyond bragging rights?
I also have to admit that there is something interesting about buying into an NFT community and then contributing some skill to increase the value of it. I suppose I should look into DAOs next. Can someone get me an invite to Developer DAO?
There’s so many pros and cons to this world but it all comes down to how we decide to apply the technology and offset any negative impacts. Don’t be afraid to ponder if an app is less accessible in its Web3 form vs. a Web2 form. I imagine the best solutions will exist somewhere between. If you enjoy this sort of content, consider minting the original post when gas is low.
Web3, you had my curiosity… but now you have my attention.