Over the last few years, after some high profile security breaches in the post-production sector, we have seen the rise of a ‘trust certification’ system which is required by some of the larger clients like Disney, HBO and Netflix. The process is not cheap, but perhaps a larger worry is who is approving the people doing the trust certification and is it worth the expense? These and more were all questions asked by community member Anastasios Katsaris and we asked Reid Caulfield who has written on this before to investigate.
The Question From Anastasios Katsaris
“I am an audio freelancer, 25+ years of experience, I have my own small Dolby Atmos 9.1.6 studio, which meets the Netflix requirements (thank you Netflix for keeping us freelancers into the game). I used to work with a lot of foreign clients, but because I am not certified by the Trusted Partner Network (TPN), the flow of projects has completely dried up.
So I started investigating how I could get the TPN ticket. Well, it is expensive, very expensive depending solely on the size of the facility (and its revenue). For example, I was asked to pay 2500 euros per year, plus 2000 euros for the certification paperwork. A friend who runs a much bigger studio than mine pays 2000 euros per month!
I was discussing that with him the other day and he was telling me the certification itself means nothing because, in reality, every studio has an ethernet cable hidden somewhere in the machine room and if somebody is set to leak material they will find a way.
To me, this sounds like yet another ‘tax’ but this time not from the State, but from the big industry companies who privilege the ‘chosen’. Well, to me this is ridiculous because, in reality, the material is likely to be more secure with a freelancer who works alone or with a very small and handpicked team, than with the big post houses with the legions of interns and inhouse technicians working on a salary basis.
And there is more. On what basis are the people and companies appointed to provide the certification? What are the prerequisites? Who controls them? More and more ‘assessors’ are popping up every day, most of them with absolutely no connection or understanding of the audiovisual industry, asking for a lot of money.
Even worse, there is also a ‘remote’ assessment, which is the most expensive. It is something like an interview and they ask you to fulfil the usual demands, like cameras, sandboxing the machine room etc in order to give the certification and they do this remotely without visiting the facility! If I pay 6000 euros, everything is fine. This sounds like a scam to me. What is your take on this?”
The Answer from Reid Caulfield
The answer to this question comes in two parts…
Content or Asset Trust Certification
How my small company was able to achieve most of those security requirements using a cloud-based infrastructure plan.
I wrote an article for Production Expert in October 2020 entitled Content Security - If You Work In Film And TV Post Read This Now. The article was a deep dive into the IT issues surrounding content or asset security.
This article is more of a cost assessment of asset security compliance. Some of this article will draw from a long post that I contributed to on a professional mixer forum on Facebook, as a reply to the question: How much does Content Trust Certification really matter?
Content Security - A Recap
Before we move in, here are some of the salient points from my previous Content Security article.
TPN & its predecessors were fast-tracked into place 5 years ago because of a specific post sound facility hack in 2016, whereby the hackers stole material from an unreleased Netflix show. That facility, a mainstay in Los Angeles for decades, was out of business a year and a half later.
Prior to this, of course, asset security had already been a thing for years, but the 2016 event sent everyone into a panic. This coincided with the mad rush to build post-production facilities to service Netflix’s huge content needs. SO it was a busy time for the nascent asset security compliance industry.
In 2016, of course, Netflix was on the rise as the biggest single producer of content in the world. They adopted asset security protocols that Disney had formulated years earlier: the so-called "Disney Tier One & Tier Two" levels had been in place internally for years (at Disney), when one of the people responsible for the protocol broke off and formed an independent audit company to define these practices for outside facilities - an IT consulting firm, in other words.
However, his company was quite small and couldn't keep up with the audit demands of existing post facilities who were implementing security measures, much less the absolute boom in construction of all the new facilities coming online hoping to grab Netflix post-production dollars. The ex-Disney asset security evangelist sold his company, which then became the Trust Partner Network (TPN). After selling, he then went to work at another large Hollywood studio as head of their Content IT Security.
TPN is an independent group, also too small to properly manage and audit every facility that is required to be certified, and so the costs spiralled up, creating a new barrier to entry for smaller facilities, which the big established facilities loved because it kept the newer smaller and nimbler facilities out of the game.
Then - and today - If you were building a new facility from the ground up in the U.S., then of course you would be crazy not to factor in the costs of TPN-style asset security. But throwing all of this on top of an existing, “old-school” bricks-and-mortar post facility is crazily expensive and really, really complicated.
So, what is a small/medium/garden-shed facility to do about compliance? Spoiler alert - you pay a lot of money for a compliance certificate and everything that comes after.
100% Asset Security and Facility Compliance Is (Almost) A Myth
Most facilities, no matter how big or small, cannot meet 100% of TPN's recommendations, and almost none do. Even the biggest post companies and studios are not 100% compliant because it just costs way too much. And once you actually get there, it starts falling apart very quickly. Here’s why 100% compliance is mostly a myth.
Think of the asset security document as a kind of bible. In that bible is every good and bad experience anyone has ever had regarding asset security, and the best way to deal with all of the issues that you might be presented with: it’s just a very detailed guidebook. There are chapters on equipment and best practices, and about all the gear you’ll need. There are rules on what to do if you suspect a data breach - who to call, and when, and in what order, complete with the relevant telephone numbers. But if you look closely at this bible, you’ll see that a lot of what’s in there, are actually tales and warnings - parables describing worst-case scenarios.
You’ll see the history of how we got here and where we’re headed. There’s a huge list of tools described in the bible, and guidelines - best practices - for how to set it all up. This bible is not a user manual for those tools, so while you’ll have lists of gear and technical points, this bible will not tell you how to operate those things. You’re on your own for that part of it. But you’ll also see a lot of things that just don’t apply anymore; they were built for another time, for threats that are no longer extant or viable.
It All Comes Down to IT Talent
Okay, so what do you need to start down the road to compliance? First, you’ll probably need someone to interpret this bible; an IT person familiar with this particular landscape. But you will also need politicians. People who can speak multiple languages, who will need to be able to talk back-office tech and front office management, and they need to know the culture of everyone involved, and how to talk and negotiate with those outside parties - in our case, the studios and content owners. These various skills are what go to make up the various security assessment firms. They are the gatekeepers of this bible full of rules, and they know which ones can be bent or broken, and which cannot.
This is what you’re paying for when you see the high cost of a simple security assessment and monthly consulting costs; you’re paying for technical IT expertise and for people who can speak intelligently and convincingly to outside parties, in their own languages. This is a valuable service offering as it turns out.
However, when you do sign on to the idea of ‘content security compliance’ with a firm that actually understands this technical and political bible as well as the other involved parties, it quickly becomes apparent that there’s a lot in the overall specification that is actually negotiable, and that it's possible for any given facility to kick the can down the road on many of the bible’s various rules and regulations - sometimes for years at a time.
Very often, if your compliance is lacking in any given area, your assessor will come back to you with:
“Well, as long as you promise to do it by your next assessment, twelve months from now, you should be fine.”
You’re paying those high consultancy fees in order to access that knowledge. For the average-sized post-production facility in a pre-COVID world, if you could get to 80%-90% compliance, then you were probably good for the certificate and the little plaque to put on your wall, which will probably suffice for your major network or studio client. Then the pandemic came along…
The COVID-19 Complication For Asset Security
For the first year of COVID-19, security compliance was thrown out the window any time work had to be sent to someone’s house to be done, which was a lot. Two weeks into the lockdown, however, the post-production industry in California was able to achieve exemption from the lockdown statutes. We became “necessary businesses”. Thank you, politics. Facilities started to schedule more intelligently, which is a fairly simple task, especially in multi-room post-production facilities.
Post-COVID, that compliance number came down to 70%-to-80%, a slightly easier target to hit.
But there’s another side to this: even today (October 2021), with no new lockdowns and no imposing restrictions for our industry, I am still seeing very large facilities almost completely empty. This means that a great many people in our business - engineers, mixers, editors, IT people, front-office workers - are still working from their homes; breathing their own air, using their own electricity and gear and fancy chairs.
All of those people should be adhering to asset-security compliance as well, no?
No. They’re not.
What Really Matters When It Comes to Asset Security?
With COVID-19’s arrival, from an asset security standpoint at least, every network and streamer and content owner and creator had to decide what was really important to them regarding asset security guidelines and compliance, at least if they wanted to keep the content flowing.
The IT departments of the various studios & streamers came up with a simpler compliance list, which should have been an easier target for smaller facilities and garden-shed owner/operators to hit. And yet, even those smaller targets are difficult and expensive small players to reach. So for the first year of COVID, asset security was de-facto suspended for people working from home, and for a lot of facilities as well.
Enter The Assessors: The Market at Work
When the various security-compliance assessment firms jumped on the work-from-home COVID situation early on, they were very quickly inundated with requests for security audits, requests they couldn’t handle in a timely fashion, and so their assessment and provisioning prices went up.
In Los Angeles right now, an initial remote assessment costs approximately $8,000(USD) - and that’s just the initial assessment, not the security gear - alarms and cameras - and it’s not the Cisco switches and hardware firewalls (and someone to program them), or the physical and liability insurance for everything, or the monthly maintenance or consulting fees, or the security re-audit every year... but when we delve down into these numbers, the expense of security assessment do start to make sense, besides the simple market-factoring issue of basic supply and demand.
Asset Security for Small Facilities and Garden Sheds Is a Simple Money Problem
A year ago (late 2020), my partner and I left the bricks-and-mortar post facility we had built in 2017 and moved our sound business to 80% cloud-based operations. Our clients still demanded baseline security compliance. The good news was that most of those clients had reduced their security requirements to accommodate people working from home. But, because my partner is a ‘heavy iron IT’ person, we were able to put together a cloud-based infrastructure plan that meant we could still hit our clients' pre-COVID, at-least 80% asset-security-compliance target.
We wanted to be able to declare truthfully to our clients that, asset-security-wise, it was business as usual - but cloud-based instead of on-premises. Once we had accomplished this, we looked into providing consulting services for smaller facilities and garden-shed/bedroom operators, but the bad news is that math was - is - unforgiving. I can tell you that if we had been paying for the asset-security IT, it would have been approximately $40,000(USD) for the first year of operations.
The Math Of Content Security Compliance
The major U.S. broadcasters and networks say that, in order to work with their content, you or your facility must be “fully” TPN compliant. They want 100% but no one anywhere can actually hit that target, so their real compliance target is a slightly more realistic 75%-90% during COVID. But most facilities, large and small, weren’t even hitting that pre-COVID target in the first place, not to mention that home-based workers were barely hitting any of those targets at all, even before COVID. Our company, however, was determined to hit the pre-COVID 80%-90% and to do that, we needed to consider cloud-based infrastructure. It’s still a difficult and expensive target to hit, and it doesn’t look like help is on the way.
Physics Is Not Your Friend: Broadcasters and Streamers And UHD
I’m going to digress a bit here and talk about the amount of data we deal with, on a daily basis, with me working out of my house. Once we put our cloud infrastructure into place and were awarded a passing grade from our various larger content clients, we began to notice that they started delivering UHD files - not just “4k”, or ProRes 444HQ, or ProRes Proxy, or DNxHD.
This means that a half-hour episode, of anything, is around 200GB. Hands up everyone who wants to download 10 episodes of 200GB-400GB media files and then transform them so you can put them on your Pro Tools timeline. These are essentially ‘mezzanine masters’ - the items from which all derivatives emanate.
For our sound-oriented business, we obviously don’t need those huge files to be downloaded to our local systems. The answer for us was to use cloud-based tools to manage all of that data into something that will download quickly and can be thrown onto a Pro Tools timeline.
We watermark the newly created files and add burnt-in timecode to them, all while vastly reducing the size of those huge files. This used to be something we did, using on-premises computers; now we do it all as a cloud-based function. – and it’s really fast. As a security bonus, only one copy of any given valuable media file resides on our local systems once it has been downloaded. This is another basic tenet of asset security - keep as few copies of any asset on-premises as possible. As a final commercial bonus, the original, huge media files, as well as any new files we create for local use can stay in our cloud partition, and with a few clicks, we can also sell our clients short, medium and long-term offsite backup and archival services.
The Cloud Is Still Expensive
Cloud-based operations are still expensive and can be labour-intensive. To set it up, you need a really smart, cloud-oriented IT person to do what we’ve done - moving files around up there the way we need to do while creating new ones to customer specifications - but it’s not magic. You can go learn it yourself, or hire someone who already knows how to wrangle massive, remote systems, and to deal with their limitations - because there are limitations to working with cloud-based tools and infrastructure. But those people are hard to find and they tend to be very expensive. Yes, we did it for ourselves, and we are able to profit from it, but only because we’re not paying out-of-pocket for a top-tier IT person who happens to be a partner in our small media services company.
You’re Paying the Piper Either Way
Redux: if you have/want/need on-premises IT infrastructure that complies with asset security guidelines, you’ll need an on-premises, very competent IT person who can deal with all your on-premises situation, and that person will cost at least $80,000- $120,000(USD) per year, and that’s if you’re lucky in this job market.
You can do the economics feasibility math for yourself and your situation. And at this point, the only companies that are able to offer those assessing and on-premises services to your garden shed/small/medium-sized creative businesses are large, outsource-IT firms that you already know you cannot afford or don’t want to pay. And so, you have the 8k assessment fee and the 2k monthly consulting fee. Finally, there’s the added infrastructure costs, the yearly re-audit… it goes on and on.
If instead, you want to explore cloud-based options, then there is still significant expertise required - expensive IT talent, as well as significant cloud-associated costs that measure out on a per month, per use, per-Gigabyte or per-Terabyte basis. None of this is free, obviously. Plus, you need wide Internet bandwidth, and whatever you have will never be enough.
Is The Cloud In Your Future?
For ourselves and our small enterprise, we figured out how to be mostly cloud-based. We don't have to worry about bad people stealing our client’s content from our on-premises content systems. Our backup policies, local and cloud, are incredibly robust, as per asset security general guidelines. Obviously, we follow all of the established password character-length rules (at least 13 characters), all of our computer and storage systems are encrypted; we don't do local content transforms & transcodes so there really is only one copy of a client’s content on or premises at any given time; we don't pay for insane amounts for HVAC to cool it all down.
And we can scale to super-computer proportions and back down again with a few clicks - or at least within a half hour. We can direct files where they need to go or grab them from where they need to be picked up, or have clients drop them in cloud buckets that we specifically set up for them. So, there is IT involved. Just not the same kind of IT. And any kind of IT is expensive. It is this scale that starts to make things really interesting; we do not have any processing-power limitations.
For our operation, one that is now dealing routinely with huge media files, this cloud solution works. But, built into our very partnership is, honestly, one of the world’s best IT people, so our small company can do huge things without spending huge amounts of money (out-of-pocket) on that expertise and the required hardware.
All of these asset security requirements are, indeed, a barrier to entry for smaller post players, at least in the U.S., and the only reason that we were able to figure it out, is because I know the post business, and my partner knows ‘heavy iron’ IT, both cloud and on-premises.
We’re still hard at work trying to find ways to make it more economical for small and medium-sized players. We would be happy to consult with other garden shed/small/medium/large-sized media facilities companies to help them get this done in the cloud - or even on-premises as well - but doing this for a smaller company on a smaller scale is still going to be a huge financial commitment no matter what, and the cloud infrastructure IP that we put into our situation is worth a lot as well, so the idea of just selling that part as a whole package ends up being expensive for whoever is buying it.
So, you and your clients need to decide how important asset security is and what is minimally important. From there, you have options to choose from. Everything is possible. It just comes down to time and money, and either way it’s going to be expensive to implement. Unless you got a guy...
Our Thoughts
Mike here, thank you Reid for another excellent detailed article on the issues of content security and getting a TPN. It seems that what Reid is saying is that yes, it can be worth it if you expect to do a lot of business with major media organisations in the United States who insist on compliance, but this is really only for ‘A’ & ‘B’- level film and television content.
But for smaller facilities like Anastasios, whichever way you look, it is maybe too expensive, and that is a real shame on several levels.
Without the expense of getting a TPN, smaller facilities provide cost-effective options for publishers like Netflix to get the best bang-for-buck. There is no doubt that the OTT sector is providing lots of work for the industry, but the adoption of the trust certification is another case of an unequal world.
Especially when you consider that a lot of the large facilities and the studios themselves can’t meet the levels of compliance that they impose on the smaller facilities and that the smaller facility needs to employ an expensive ‘politician’ to provide them with the necessary certification required by these large organisations.
Surely there has to be a more realistic environment for the smaller facilities to be able to meet, maybe a rewrite of the ‘bible’ Reid refers to which is aimed at the small one-person operations, who from the shed, bedroom, garden room do so much work on all types of post-production?
Surely it is in the interests of the OTT publishers like Netflix and Amazon to put in place a cost-effective and realistic compliance system that means that people like Anastasios can continue to work for production companies making content for them?
UPDATE: Netflix Respond To This Article
We reached out to Scott Kramer, who is Manager, Sound Technology | Creative Technologies & Infrastructure at Netflix and they have released a statement in response to this article…
“Netflix does not, nor has it ever required participation in TPN or any other industry security assessment program. For more information, please see our Content Security Best Practices. This document lists recommendations rather than requirements.“
We also see that there is another document on the Netflix site Home Studio Security Guidance, which provides a set of recommendations if you are working at home.
Netflix also has a dedicated Netflix Studio Information Security email address which you can use to request additional guidance or to report a ‘security incident’.