In-app browsers, I forgive thee not

A Behemoth case study and some complaining

This week I had the opportunity to work with polish metal band Behemoth on their newly announced album, I Loved You at Your Darkest, out October 5th. I’m no stranger to the sacrilegious music marketing arts, having done a grip of work for Ghost in the past, but as someone said to me, the lead singer of Ghost is a Scooby Doo villain compared to Behemoth’s unrelenting Nergal. I knew I had my work and religion cut out for me.

The concept we settled on required fans to recite a prayer in order to stream a clip from the new record. A simple test of a user’s allegiance to the act and their preachings. My buddy Harold sent me the following press pickup and the headline alone had me spitting coffee out all over my computer.

BEHEMOTH Offer First Preview of New Music,
But You Must Say A Prayer First (Seriously)

— Metal Injection

The experience was built on top of the Vue.js application framework Nuxt. After shopping around for a good speech recognition provider, I settled on IBM’s incredibly powerful and well documented Watson Speech to Text service. This service, powered by WebRTC, painlessly handled the recognition logic we were after. However, a real monster reared it’s ugly head during testing: the social in-app browser and it’s lack of support for new web technologies, including WebRTC.

Read on to find out how the app was built, why in-app browsers are more evil than Satan, and what you can do about it in your projects.

Prayer Recognition

“Prayer of Behemoth” prayer card

As I mentioned, speech recognition for this project is powered by IBM Watson’s Speech to Text service. In addition to providing a well documented and affordable service, the Watson developer team have also created an excellent Javascript library for adding this functionality to your project with ease. Let’s first take a look at our prayer:

Living God!
I shall not forgive!
Jesus Christ!
I forgive Thee not!

Yikes. I know. We can work with this though. One of the features of our STT (Speech to Text) service is keyword spotting so I decided to break down our prayer into keywords.

['living', 'god', 'shall', 'not', 'forgive', 'jesus', 'christ']

I figured if we interpreted our user saying 6 out of 8 of these keywords we would allow them to proceed. Given the infinite variety of voices, accents, and environments we might encounter, I configured our STT service to be as forgiving as possible. Luckily, our service has parameters for providing multiple word alternatives and tweaking the keyword threshold to let a lot of near misses through.

Here’s how I ended up configuring the Javascript library:

const stream = recognizeMic({
  format: false,
  objectMode: true,
  max_alternatives: 10,
  keywords: keywords_array,
  keywords_threshold: 0,
  word_alternatives_threshold: 0
})

Watson’s STT service will stream back results immediately so I decided to require that our users click a button when they’ve begun to pray and when they’ve finished. As they spoke, I added any matched keywords to a new array of recited keywords which I made sure to keep unique. As soon as they stopped praying, I simply checked to make sure the length of the recited array was greater than 6. Here’s a rough version of the code I ended up using.

let praying = false
let recited = []
stream.on(‘listening’, () => {
  praying = true
})
stream.on(‘data’, (data) => {
  if (data.results[0] && data.results[0].keywords_result) {
    let keywords = data.results[0].keywords_result
    for (let key in Object.keys(keywords)){
      recited.push(key)
    }
    recited = [...new Set(recited)]
  }
})
stream.on(‘stop’, () => {
  praying = false
  if (recited.length >= 6) {
    // prayer recognized
  } else {
    // try again
  }
})

Good stuff. This would also work with a non-Satanic prayer. If you’re interested in seeing what this Speech to Text library is capable of, IBM Watson has provided many examples. This marriage of WebRTC’s microphone capabilities and a powerful recognition service is what the evolving web is all about. It works well on computer and mobile browsers.

What could possibly go wrong?

In-app Browsers (aka portals to Hell)

The marketing campaigns I build are meant to be shared, loved, and despised on the Internet and especially via the big three social platforms: Facebook, Instagram, and Twitter. These are the properties artists have painstakingly built up their followings and they provide the simplest route to traction. (Behemoth has 346k followers on Twitter and 1.4 million likes on Facebook) Each of these networks also provide very popular native applications for both the iPhone and Android. I personally use a mix of mobile web and native apps for these platforms but I suspect most mobile users prefer the native applications.

When you click an external link on any of these native social apps, they are opened within the actual app — in an in-app browser. Problems arise when these in-app browsers block some of the new technologies available to the mobile web. In the case of our Behemoth campaign, these in-app browsers block the WebRTC standard’s ability to gain access to the user’s microphone. The social networks might argue that they’re protecting their user’s from possible security exploits but they can’t argue that they’re also trying to keep you within their ecosystem for marketing purposes. More time in app equals more ads seen.

In the end this makes for a terrible user experience for our app because it simply does not work as expected. As a creative developer who is fascinated by the evolution of the web, it’s also frustrating to be “held back” by these in-app browsers. Our only course of action is to encourage that user’s open the experience in their devices default browser, such as Safari on iPhone or Google Chrome on Android. The iPhone apps for Twitter and Facebook both provide a button to open the website in Safari from their in-app browser screen. However, Instagram ridiculously does not. Instagram has always been notorious about external links which is how the “link in bio” hack became so popular but not providing a link out to the device’s default browser is just plain evil. It doesn’t seem right that these apps get to run their own little Internet from within.

So, for those users that end up on your WebRTC powered experience via an in-app browser, you’ll want to provide a fallback message. Of course, WebRTC fails silently in these browsers because the current code simply stops running altogether. FML. I figured out a solution though.

A Heavenly Fallback

While the WebRTC standard provides an excellent exception event in case users don’t have the appropriate devices or simply do not allow access, this error event cannot detect an in-app browser shutting down the current code completely. One idea would be to simply detect whether or not your application is currently being served from one of these in-app browsers, but some of these browsers have decided to relay a User Agent string that looks just like a fully functional default browser. Thanks Twitter. So what’s the solution? Here’s how I do it.

I create a new variable called “prompted” and set that to null initially. Then when I’m ready to begin my getUserMedia call, I first check to see if prompted is null. If so, I set it to false. I then run my getUserMedia function. Then right beneath the getUserMedia function, I set prompted to true. That line of code below the getUserMedia function will be called immediately as long as it isn’t blocked completely. So, if prompted stays false and never gets changed to true, you should probably point your user elsewhere. In short, getUserMedia was never ran so it could not prompt your user or throw a standard error.

Here’s what that looks like in code.

if (prompted == null) {
  prompted = false
}
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
  .then((stream) => {
    // do something with stream
  })
  .catch((error) => {
    // do something with error
  })
prompted = true

In the event you find yourself in this situation, remember good copy can save a now frustrated user. I went with the following error statement: Please open this site directly in Safari to say your prayers.

Thanks

Behemoth

Thanks to IBM’s Watson team for making their excellent Speech to Text service available and for providing the node library which made integrating it a dream. Thanks to the awesome team at 5b artist management for trusting me with another one of their clients. And thanks to Behemoth for the opportunity to spread their message a bit wider. Their new album, I Loved You At Your Darkest, is out October 5th. Watch the new video for “GOD == DOG” if you dare.