Why do we need SO MANY SERVERS??
Articles,  Blog

Why do we need SO MANY SERVERS??

October 15, 2019


“Why the heck do you guys need a whole roomful of servers just to make YouTube videos?” We get asked this… Actually a lot of other questions about our data management and our workflow all the time. So for the last year actually, ever since we got our red cameras, I’ve been meaning to do an update for you guys on how we handle such heavy footage how we share resources with such a large team and how we do all of that while maintaining our rigorous daily release schedule, so When [G Winn] reached out to us asking us to do a feature on their smooth for camera gimbal, I thought well, hey there’s an opportunity Why don’t we grab this thing and take it with us while I show you guys the way that data flows Through Linus Media Group to become a video on your screen So while I let you all appreciate the irony of a video about our overly complicated workflow shot on a cell phone with a Simple handheld gimbal, I’ll tell you guys a little bit more about the Zoom Four. So it’s got a highly integrated control panel down here with apps for both iOS and Android that allow you to control your camera, It’s got focus pull and zoom capabilities, It’s got the ability to do vertigo shots with one button through the app or manually, It’s got what they call phone go mode for instant scene transitions and it’s got both a quick standby mode and a one-click mode switch. Everything that we shot outside of the intro and the souter right now was filmed on this. It’s got a no date on it right now, But it was shot on an iPhone 10 and I’ll let you guys be the judge of the stabilized footage that comes off of it. So to demonstrate some of the challenges with working with red footage, We’re actually gonna take That intro that we just shot just now and we’re gonna ingest the whole thing to show you guys how we do it now. Red cameras are not great for recording audio. In fact even with an external preamp, It is really difficult to get anything resembling usable audio out of the camera directly. So recently we finally gave in and resorted to an external audio recorder. It really sucks to synchronize audio, especially if you’re stuck using the scratch audio that gets recorded directly to the camera, And then the SD card out of your external recorder and aligning it manually. But we recently got a little doodad called a “tentacle sync”. You get two pieces here: A master that sits on our mix pre-six, and a slave that sits on our camera you synchronize them at. The beginning of the day and what they do is they inject what’s called time code into the mag and into the SD card, So that you can easily synchronize them later on down the line. But that is far from the only problem. So the thing about the Weapon 8K, So that’s our camera recently renamed the DSM-C2… Helium or something like that, is that it records footage. they’re both the same camera at up to 8K/60 frames per second, and with the 480 or 960 gig Meg’s that’s up to hundred megabytes per second. So to put that in perspective if we were filming a simple Techlinked episode, Even though we will usually use like a twenty to one compression ratio, So that’s about four times the compression that they would probably use on something like a feature film, That means that a 10 gig clip could still end up being in the neighborhood of about – a 10 gig clip. that means that a 10 minute clip could still end up being in the neighborhood of around 50 gigs, and While it’s less of an issue for something like Linus Tech Tips or Tech Quickie when you’re recording dailies, And you’re expected to release the video on the same day that you film it like with something like a new show, Size of files ends up becoming a potential problem, so that’s why our to ingest stations which you might recognize from one of our Tech Showdown episodes are both equipped with 10 gigabit network cards. So one of them has this Asus one, which is basically the last generation version of this “Aqtion 1”, they’re both using a quantities that Allow them to be somehow under $100 for 10 gig networking. There are a number of different ways that we ingest footage depending on the type of camera we’re using, So for Red, you can see each of our video clips is actually a folder made up of smaller broken up video clips. So we’re gonna go ahead and grab that and… Copy the whole thing into our project folder… For this video, so that goes in A-Roll One. What we’re looking at right here is probably some kind of like weird buffering thing or something, But this is about what we’d expect to see in terms of our ingest speeds. It’s not the full 10 gigabit because that would be in the neighborhood of one to 1.1 gigabytes per second, But the reality of it is that you’re going to run into bottlenecks elsewhere. And in fact these Red Meg’s even though we’re using an eSATA Interface for them are just not that fast, and they only read it about 230 megs a second. Then what we got to do is Grab the audio clip off of our SD card that goes into our audio folder right here, and that’s pretty much the whole process for RED footage. But For that we wouldn’t need powerful machines, and there is a reason that each of these is running 64 gigs of ram and a 10 core Extreme Edition processor, and that’s because for our other cameras like the A7-S for example, we actually use Adobe Prelude, which is an imperfect piece of software, but Has its uses to transcode the footage when we’re bringing it in. So what we do is we take whatever project it is, and this is generally fast as possible, we still shoot that on the A7-S, we create a subfolder and we set it to transcode to Cinnaform 4k. This improves our Timeline performance when we’re scrubbing through the clips in Adobe Premiere. The other thing that we do is set a second destination for the original files that we’re not Transcoding just in case something goes wrong with the transcode, Which does happen from time to time, and we need to go back and grab the original files. Now just to give you some idea Even if we’re not using the full potential of this network connection all the time of what it’s capable of, Let’s go ahead and just grab a file off the desktop and show you just what this puppy can do. So that is saturating the read speeds of the RAID 1 SSDs that each of these machines are booted off of, so that was a 10 gigabyte file. Let’s go have a look on the other end of this Ethernet cable at how exactly that whole thing comes together now. I’ve shown you guys our server room… fair number of times, but I haven’t given you guys an update in quite a while on How exactly it’s working in here. So the main server that everyone is editing off of at the same time is this one right here. This is Whonnock server, and it’s running 24 plus 1/2 four or five six seven, so it’s running 31 Intel 750 series 1.1 terabyte NVMe SSDs, and this guy is an absolute monster. So if we fire up Performance Monitor, you can see that even though we’ve got four of our editors in office right now, And we’re seeing each of them doing a hundred and fifty, one hundred and twenty, Fifty megabytes a second of reads. This thing is barely suffering and our There we go. Our Z disk queue-death is only about 0.5. That’s the benefit of Nvme, is that nice fast responsiveness. And in fact when we were still using our SATA SSD server for every one. We were starting to run into issues Once we had the room fully staffed over there, where Adobe Premiere would crash, but not just crash. Everyone’s would crash at the same time and we narrowed that down to slow response times on our NAS. So this guy’s connected at 40 gigabits per second. This one right here has 24 SATA SSDs. And this one only gets used This is called QQ server. This one only gets used for Large projects that only one person needs to work on at a time. It’s still SSD based. So we still don’t run into any crashes or any other weird issues like that, But it doesn’t have quite the snap that an NVMe machine does. Now something that we learned a really valuable lesson about I guess it must have been about two years ago was Real-time data replication and off-site backup and that is where this guy comes in. So I want to show you guys a little trick here… I don’t know. If I’ve ever actually done this demo in a video before I’ve shown people that have come into our office. So I’m going to use my test folder here and I’m going to create a document called “Test For Video” right here. This is a text file on our main drive. Then I’m going to jump over to this hard drive based server under it. So this is running 8 8 terabyte drives for a total of 64 terabytes, And I’m just gonna pull up the Whonnock sink folder… Get that test folder open, test Iridium- Wa-pow! Just like that. So within about five to ten seconds, Whether it’s a text file or whether it’s a large video file, the synchronization software that we’re running here will Automatically dump it over to here, so if this crashes we can actually continue editing videos within about 10 to 15 minutes of Switching over everyone’s map drives. And another benefit that it gives us is that normally on a network drive when you delete a file? It’s just gone, but instead check this out. If I delete “test for video”, I’m actually going to have to do it from a separate server, but if I delete “test for video”… Boop you’re gone you’re done. And I go into here… That file is going to disappear. But… What will happen is we can go into the deletions folder and we can rescue it. This has actually saved our butts more times than I would like to admit, because Stuff does get accidentally deleted. Now all of this is only for active projects that we are working on right now. Once we’re done with something, it goes on to Petabyte Project, and we’re actually running Both phases of it right now when we originally deployed it. We were only using one of them so we could just save power on ours for all the drives that were in the other one. Not the case anymore. So Petabyte Project is now up to 777 terabytes of total space of which we’ve consumed four hundred and thirty-two. This holds all of our archived projects, so that in the event that we want to make a video that refers back to one of our other videos, We can grab the original quality files rather than downloading off YouTube like a lot of other YouTubers do. Bringing us into this room… One of the biggest challenges is not just working with red footage because the files are so heavy, one of the biggest problems that we have is that we’ve got Multiple people working off of that same shared NAS all at the same time. So this is Where the 10 gigabit connections that our editors are also using comes in. When we’re scrubbing through footage, We can actually see data rates in excess of two-and-a-half gigabit per second, so we peaked at just shy of four gigabit per second. That My friends is why we’ve got the bangin NAS with the high-speed networking. So once the edit’s done, Taron or one of our other editors will export the entire project as an MOV, but those are extremely inefficient Files, so they’re- they’re great quality, but they’re absolutely massive. So he’s gonna copy that those in the transcode flow plane. Yeah, and then you paste it here… Now what’s gonna happen there is that Water-cooled server that I built a while ago the one that’s at the very bottom of the rack, Is going to see that that file got transferred, and then it’s going to spit out the correct formats for all the different platforms that we upload to, whether it be YouTube or Flowplane or Facebook or whatever the case may be that way. The editors machine doesn’t get tied up with this. So this is cool, Once that file is done copying. It does take a second and Media Encoder is not always perfect Before it’ll pick it up, but we use VNC in order to remote into that machine So everyone can go in at the same time unlike Remote Desktop Connection, and make sure that it has actually picked up the project and that it’s transcoding to the correct format. Okay, cool. The system works, so Thanks for watching guys. If you disliked this video you can hit that button, But if you liked it, hit like, get subscribed, or maybe consider checking out where to buy the stuff We featured at the link in the video description. Also down there is our merch store which has cool shirts like this one, and our community forum, which you should totally join.

Related Posts

Integración del e-mail con el CRM de Bitrix24

Integración del e-mail con el CRM de Bitrix24

November 11, 2019
DevOps Automation tool for SQL Server database

DevOps Automation tool for SQL Server database

November 21, 2019
Official 3kliksphilip Servers

Official 3kliksphilip Servers

November 3, 2019

72 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *