Over the last few days, Apple has rolled out their “Spatial Audio supported by Dolby Atmos” feature in a new version of iOS and on their Apple Music platform. On various professional forums, it’s fair to say that some very smart audio engineers are well into the process of figuring out - well, everything. In this article, Reid Caulfield, takes a look at what we think we know so far, how this affects audio professionals and asks who needs or wants, Immersive Sound for anything?

What Kind Of Things Need Figuring Out?

Apart from “everything”? There are A/B and phase-cancellation comparisons between Apple & Dolby’s Atmos offering, vs TIDAL’s, and all of the speculation as to exactly how these various services are generating Binaural audio from their various channel-count starting points, and whether/how they are upmixing from smaller channel formats (e.g. stereo).

All of that said, there is the smell of opportunity is in the air.

What We Think We Know So Far 

• Apple can handle and stream full multi-channel music, all the way up to various Atmos speaker configurations, and stream that product to various devices, specifically, a variety of their earbud models and to their Apple TV 4k streaming device, iPhones and iPads. So if you have 12 speakers in your living room, and an AppleTV 4k, you can listen to new music, which was specifically mixed for the Atmos Music format. If you want to listen using headphones and you have the required earbuds and set the app for ‘Spatial’ - which is, I’m guessing, another word for ‘Binaural’, then you can listen to that Atmos content using well-known-and-understood downmix methodologies.

•  Apple and TIDAL seem to be in the same ballpark when it comes to their respective downmixing results. Apparently, they almost (or completely?) phase-cancel. So whatever the two systems are actually doing, they seem to be doing it the same way. That’s a very positive result.

• Apple is apparently not using Dolby’s Binaural downmix/encoding monitoring capabilities. That is, what you’d select coming out of a Dolby hardware RMU or software-based DAPS - but are instead using their own scheme, but apparently, the results are close to Dolby’s process, as they seem to be with TIDAL.

•  Why is Apple not using Dolby’s Binaural scheme? Probably licensing and/or other factors that Apple requires in order to render Binaural from multichannel in near real-time, and to a wide variety of devices and to stream out a lot of data to millions of people simultaneously. There is also speculation as we reported in our article Apple Spatial Audio With Dolby Atmos Support Released, in which we say…

  •  “Perhaps Apple has done their own thing to reduce the complexity to work on the iPhone. Initially, on the iPhone, the Dolby Atmos spatialisation was derived from the DD+JOC and then decoded ready for playback. If Apple chose to use the Dolby Atmos binaural spatialisation then the iPhone would have to sense when the headphones were being used and then switch to Dolby’s binaural render, which would be a much more complicated process as Dolby uses 3 distances to simulate the room as well as the HRTF, all of which could be too complex to do on the phone as well as the speaker rendering.”

• What, then, exactly, is Apple doing to get to Binaural and to fit it through their streaming distribution pipe? People like us have questions, and we want to see the numbers and metadata, but honestly, it looks like those really smart people I was talking about above have already basically figured out how the various algorithms are getting to a final result. No doubt this research will continue to evolve as the days and weeks go by.

•  There is some consternation in our community about there now being an additional standard that we have to meet when we mix music - Apple’s equation - since they’re not using Dolby’s standard. Earlier this week, I suggested on one of the forums that Dolby could include an “Apple Binaural” monitoring mode in their RMU and DAPS software, but that got shot down by someone really, really smart. Basically, the argument against this was, “Well, what happens when Dolby Binaural and Apple Binaural coming out of the RMU or the DAPS don’t match? We’d have to mix for two different Binaural standards.” To which I could have replied - but didn’t - well, the same thing that happens when you do a 5.1 mix, then do a standard LoRo fold down, or encode it as Dolby Stereo and discover discrepancies. Go back and figure it out, Man! For 40 years we’ve been doing this for film and (some) TV sound! Even 5.1 TV sound down to 2.0 LoRo is something that needs to be checked and, if necessary, corrected for. The exact same content can sound completely different depending on how it’s played out, even in controlled, professional environments.

• Finally, what happens when ‘standard stereo’ music - music that has not been specifically mixed and mastered for ‘Dolby Atmos for Music’ or “Apple Spatial” -  makes its way through Apple’s ‘Spatial Audio’ algorithm? Is it being upmixed to Atmos so that it can be played back immersively through AppleTV 4k? Is it being re-rendered straight to Binaural (‘Spatial’) audio, or what? Does it always stay at 2.0 LoRo, in other words, the way it was originally conceived and mixed? I should probably know the answers to these questions but right now I don’t.

 Why Do We Care So Much? How Does This Affect Sound Professionals?

This is what I really want to talk about. The technical issues surrounding Dolby Atmos for Music or Apple’s Spatial Audio technology will out soon enough. We care because, as working, professional sound engineers, editors, music mixers, TV and film mixers, and as full-on Post facilities, if we’re a bit lucky, the success of a true immersive format for TV, streaming, and music mean we extend our relevancy out another 5-15 years. New tools, more speakers, Binaural for headphones… all of this has the potential to result in more work for everybody.

Or possibly not? Dolby has been pushing Atmos Home Entertainment (HE) for TV broadcast & streaming for a while now. The truth seems to be that Atmos HE as a playout format for television content just wasn’t breaking into the home market in any meaningful way; it wasn’t captivating television audiences as had been hoped.

But this isn’t new. It was the same for 5.1 for music almost 20 years ago and 5.1 for home theatre 25 years ago; they did not set the world on fire, either, because people didn’t want to stick 6 speakers across their living rooms at home, and even if they did make the effort, they probably didn’t keep it all calibrated, or worse still didn’t even put the speakers in the right place in the room!

In my own life, we tried full 5.1 in the living room at my home years ago (twice, in two different homes, actually), but my wife is extremely bass-sensitive in her hearing profile, so I was always having to turn off the subwoofer so that she didn’t lose her mind, LOL. It wasn’t long before we were back to the two onboard TV speakers - just like the other 99% of the viewing worldwide audience.

But with Atmos HE, this history is what kicked off the sudden boom in Atmos HE ‘Soundbars’ a couple of years ago: let consumers know, early on, that they don’t have to run around installing speakers everywhere. Some of the Atmos soundbars are actually pretty good, I’m told. I guess I should buy one. Pushing the format was surely what contributed to the full development of software-based DAPS as opposed to Dolby hardware-based RMU’s, and, finally, one of the reasons that Netflix now allows Atmos mixing to take place in non-Dolby-Certified environments.

So let’s look at Dolby Atmos HE as a television-sound product vs a Music product.

Who Needs - Or Wants - Immersive Sound For Anything?

Going forward, people who record and mix music, and sound design and edit and mix for film & TV will certainly want to plan for immersive from the beginning. But, specifically on the music front, will they, and if so, for how long? Is Atmos-in-the-home destined to go the way of 3D picture in the home.

Most television content is ‘unscripted’ or ‘lightly scripted - ‘Reality’ or ‘Actuality’. Things like cooking shows, travel shows, game shows, news, etc. This makes up the vast majority of TV and streaming content, along with ‘non-premium’ documentary content. A growing number of these shows don’t even go through a proper sound edit or re-recording mixer at this point; the picture editor is mixing and throwing on some loudness plugins before output and delivering to the bare minimum delivery spec. So, the eventuality that the producers of this content will, all of a sudden, decide to pay for Atmos HE is unlikely.

Now, let’s talk about musical artists. Maybe it’s the Rolling Stones or maybe it’s a new artist who released their first album a month ago. Are they going to go back and pay for new Atmos mixes, or Binaural mixes? Do they even know what those things are, or do they care? Possibly Not. Because right now, they just want to get back to touring so they can make real money. Once the album is done, it’s done. Good luck getting them back into the studio to remix for alternative formats. Maybe Finneas and Billie Eilish will do it but, as we’ve seen, if they’re going to do it, they’re going to do it in their own facilities.

Okay, so, how about the labels? Well, they may pay for their marquee albums to be upmixed but they probably won’t, because everything will need to be re-worked entirely, from scratch, and anyway, they tried that 20 years ago with 5.1 SACD and we all know how that turned out.

If you’re the person who originally mixed a famous album from whenever maybe you’ll be called back to remix into Atmos; but probably not. And anyway, that’s what? - 500 albums, if we’re lucky? It was far less for 5.1 SACD. Neither the labels nor the artists want to deal with remixing legacy work that isn’t generating meaningful income. Do they?

But what about all of the legacy content? I mean, if Apple is doing a half-decent job of upmixing the stereo version of your favourite hit, is there room for a real, professional re-imagining of an album or song?

So if consumers don’t care and the artists, producers and distributors of the majority of content on TV or in music are not willing to pay for real 12-channel mixes, then… who cares?

Politics

There’s no way anyone is going to let you upmix - or fully remix - the entire James Bond franchise to Atmos HE. There’s no percentage in it for the producers for the money they’d have to spend to do it right. The same goes for 98% of legacy or music or television content; they’d have to call the producers back, the original mixers (if they’re not dead by now), there are “creative intent” issues that require signoff by somebody. I might well be able to make a case for remixing or, at least, upmixing absolutely everything.

And I think my case is strong: a single deliverables format to rule them all, and wouldn’t it be amazing to hear the original Star Trek series or films or every episode of Coronation Street in full Dolby Atmos? Or the first three Indiana Jones films? How about that? Or Star Wars? Maybe they’re doing it, but as far as I know, they’re not. Why?

The truth is that they just don’t want to spend the money. You & I might well be able to do a lot of that TV work in our studios at home, but they’re not going to let us. The Producers of Friends aren’t going to spend 1.5 million dollars getting to Atmos HE for 236 episodes. Viewers are just fine with Friends sounding the way it always has. Maybe the Apple Spatial Music upmix algorithm can be tweaked so that film and TV content sounds pretty good? There’s a reason that a good many of the people who pioneered film (and television) sound and DAW technology in the 1980s and 1990s are now actually working for Apple. Have they been working on Apple Spatial Audio for music all these years, or is there another show yet to drop?

So, Where Is The Opportunity?

This is what facilities and individual sound professionals all over the world are asking themselves this week: how can we make money off of Apple Spatial, Atmos Music and continue on with Dolby Atmos HE for television content? The real opportunity might be in legacy content - everything that wasn’t mixed in Atmos HE or Atmos music over the last 80 years, in other words. But who will be a part of that?

1. Are you a well-known music recording and mixing facility or engineer? Then, yes, it appears that the likes of Universal Music are working with Dolby to work through their back catalogue and to work with their current artists to produce music in Dolby Atmos so there may be opportunities in Apple Spatial And Dolby Atmos Music, but not for Atmos HE for television. Maybe you’ll get to remix some of the work you’ve already done in stereo, but who on your client roster can be convinced that the effort will pay for itself?

2. Are you a well-known post-sound-for-TV mixing facility? Then, yes, there may be opportunities in Atmos HE for television but probably not for Apple Spatial And Atmos Music. It’s just not what you do in a very segmented post-production industry.

3.  Are you already working on Dolby Atmos HE work for television content? If yes, then you’ll continue. If not, unless the soundbar market really takes off, the door is closing because it seems, consumers have already chosen not to install Atmos HE in their homes for television viewing, at least not in any meaningful way in terms of market penetration, and distributors are very cool to the idea because of the cost of Atmos deliverables. Most foreign language localization (to & from English) is basic stereo, even if the original was 5,1. I have yet to see an Atmos HE localization project.

Let’s move on.

My Adventures With Binaural Sound - Are Headphones The Future?

This is interesting and compelling, and Apple believes that headphones and earbuds are the future of sound. There seem to be very specific use-cases for Binaural sound on headphones for music, or perhaps, podcasts. And so while I have known about Ambisonics and Binaural sound, for decades, I had not experimented with it until very recently. I’ve mixed music and TV straight to mono, stereo, 5.1, 7.1, Atmos HE - but not Binaural, or Ambisonics. I’ve also upmixed to and from all of those formats, and yet, not Binaural, yet.

The use of Binaural sound does indeed fit a variety of creative applications extremely well. However, what I would really like to do is to convince our various clients that we should mix their legacy content in Atmos HE, upload that to Apple and let Apple take care of the stereo and Binaural down-conversions. This way, it will playback in Atmos HE or 5.1 or whatever, on speakers through AppleTV 4k, or binaurally in headphones, or on someone’s 5.1 system at home, or in “Straight Stereo” on speakers or headphones.

But, as you can imagine, if I mix straight to Binaural, it will cost the client ‘X’. Mixing in Atmos for Music or Atmos HE, however, will be at least “4X’. I’ll make the case, and the case is, if we go straight to Binaural, listeners won’t be able to listen to the product in anything but headphones. If they try to play it through speakers, it will fall apart. My client will probably not care about this, because they already know that their audience is actually listening to their content on headphones 98% of the time.

So, will my client pay for their legacy content to be remixed and reformatted via any methodology at all, because it sounds cooler and is more immersive? We’ll see.

The Bottom Line

Who’s going to pay for it? That’s it. That’s the whole thing.

And the answer is: Not. Very. Many... people, companies, producers, directors, artists or legacy content and IP owners.

I hope I’m wrong.

Let’s look at the very top end: Is Steven Spielberg going to authorize the tasking, energy and money it will take to remix “Schindler’s List” or “Jurassic Park” or “Jaws” up to Atmos HE? I hope he will, but that’s going to Skywalker and everybody reading this knows it. The James Bond franchise? Maybe, because I think Amazon owns it now and they don’t care about money. They might build new facilities around refurbishing legacy content, maybe poach some ‘A’ mixers away from wherever they’re currently working, and the cycle continues. But they probably won’t, because, as we’ve seen from the examples above, there’s no point f**king with “The Dark Side Of The Moon”, or “Hotel California” or “Friends” because you won’t be able to sell them to anyone who cares.

Apple can sell a streaming service that plays out these songs 10,000, or 100,000, or 1,000,000 times a day, but almost no one is profiting from this sector anymore. So will Apple pay for actual legacy upmixes? No, because they might already have a decent enough upmix algorithm that does it for free. Just like the Binaural plugin I’m testing. Phase-process a mix and (maybe) place it in a “room” and that’s enough for 98% of the listening audience.

Can we convince the producers of unscripted shows (reality, actuality, news, etc.) to do their primary creative mix in Atmos HE? No. I know this. I’ve tried this. Maybe Survivor or American Idol... maybe, if they don’t already. I mentioned earlier that reality mixing is pretty much being done by video editors and plugins at this point. A month ago I was headhunted for a position with a company that makes 100 x 90-minute movies every single year, along with 5,000 promos & trailers to accompany them. They mix in Stereo. “We’re hoping to start mixing in 5.1 in a few years time”, they told me.

So it’s no wonder that Dolby - a company I really love, by the way, for their innovation - has moved the conversation for Atmos HE for television product over to music product. Most producers of televised content don’t want to pay for Atmos HE.

Will Music Be Any Different?

From the conversations I’ve seen swirling around just over the last few days immediately following Apple’s release of Spatial Audio, I can tell you that everyone - the smartest sound engineers on the planet, actually - are running around trying to figure it out.

So while this is a very exciting time in the professional sound industry, it might be wise not to follow the Yellow Brick Road all the way to The Emerald City just yet. As a technical exercise, yes, let’s figure it all out. But as a new business sector or business model, don’t put all of your dogs into this bicycle’s basket, hoping that you’ll be able to convince the gatekeepers that they should foot the bill for it all. You can’t, and they won’t. Not again.