“Why the heck do you guys need a whole roomful of servers just to make YouTube videos?” We get asked this… Actually a lot of other questions about our data management and our workflow all the time. So for the last year actually, ever since we got our red cameras, I’ve been meaning to do an update for you guys on how we handle such heavy footage how we share resources with such a large team and how we do all of that while maintaining our rigorous daily release schedule, so When [G Winn] reached out to us asking us to do a feature on their smooth for camera gimbal, I thought well, hey there’s an opportunity Why don’t we grab this thing and take it with us while I show you guys the way that data flows Through Linus Media Group to become a video on your screen So while I let you all appreciate the irony of a video about our overly complicated workflow shot on a cell phone with a Simple handheld gimbal, I’ll tell you guys a little bit more about the Zoom Four. So it’s got a highly integrated control panel down here with apps for both iOS and Android that allow you to control your camera, It’s got focus pull and zoom capabilities, It’s got the ability to do vertigo shots with one button through the app or manually, It’s got what they call phone go mode for instant scene transitions and it’s got both a quick standby mode and a one-click mode switch. Everything that we shot outside of the intro and the souter right now was filmed on this. It’s got a no date on it right now, But it was shot on an iPhone 10 and I’ll let you guys be the judge of the stabilized footage that comes off of it. So to demonstrate some of the challenges with working with red footage, We’re actually gonna take That intro that we just shot just now and we’re gonna ingest the whole thing to show you guys how we do it now. Red cameras are not great for recording audio. In fact even with an external preamp, It is really difficult to get anything resembling usable audio out of the camera directly. So recently we finally gave in and resorted to an external audio recorder. It really sucks to synchronize audio, especially if you’re stuck using the scratch audio that gets recorded directly to the camera, And then the SD card out of your external recorder and aligning it manually. But we recently got a little doodad called a “tentacle sync”. You get two pieces here: A master that sits on our mix pre-six, and a slave that sits on our camera you synchronize them at. The beginning of the day and what they do is they inject what’s called time code into the mag and into the SD card, So that you can easily synchronize them later on down the line. But that is far from the only problem. So the thing about the Weapon 8K, So that’s our camera recently renamed the DSM-C2… Helium or something like that, is that it records footage. they’re both the same camera at up to 8K/60 frames per second, and with the 480 or 960 gig Meg’s that’s up to hundred megabytes per second. So to put that in perspective if we were filming a simple Techlinked episode, Even though we will usually use like a twenty to one compression ratio, So that’s about four times the compression that they would probably use on something like a feature film, That means that a 10 gig clip could still end up being in the neighborhood of about – a 10 gig clip. that means that a 10 minute clip could still end up being in the neighborhood of around 50 gigs, and While it’s less of an issue for something like Linus Tech Tips or Tech Quickie when you’re recording dailies, And you’re expected to release the video on the same day that you film it like with something like a new show, Size of files ends up becoming a potential problem, so that’s why our to ingest stations which you might recognize from one of our Tech Showdown episodes are both equipped with 10 gigabit network cards. So one of them has this Asus one, which is basically the last generation version of this “Aqtion 1”, they’re both using a quantities that Allow them to be somehow under $100 for 10 gig networking. There are a number of different ways that we ingest footage depending on the type of camera we’re using, So for Red, you can see each of our video clips is actually a folder made up of smaller broken up video clips. So we’re gonna go ahead and grab that and… Copy the whole thing into our project folder… For this video, so that goes in A-Roll One. What we’re looking at right here is probably some kind of like weird buffering thing or something, But this is about what we’d expect to see in terms of our ingest speeds. It’s not the full 10 gigabit because that would be in the neighborhood of one to 1.1 gigabytes per second, But the reality of it is that you’re going to run into bottlenecks elsewhere. And in fact these Red Meg’s even though we’re using an eSATA Interface for them are just not that fast, and they only read it about 230 megs a second. Then what we got to do is Grab the audio clip off of our SD card that goes into our audio folder right here, and that’s pretty much the whole process for RED footage. But For that we wouldn’t need powerful machines, and there is a reason that each of these is running 64 gigs of ram and a 10 core Extreme Edition processor, and that’s because for our other cameras like the A7-S for example, we actually use Adobe Prelude, which is an imperfect piece of software, but Has its uses to transcode the footage when we’re bringing it in. So what we do is we take whatever project it is, and this is generally fast as possible, we still shoot that on the A7-S, we create a subfolder and we set it to transcode to Cinnaform 4k. This improves our Timeline performance when we’re scrubbing through the clips in Adobe Premiere. The other thing that we do is set a second destination for the original files that we’re not Transcoding just in case something goes wrong with the transcode, Which does happen from time to time, and we need to go back and grab the original files. Now just to give you some idea Even if we’re not using the full potential of this network connection all the time of what it’s capable of, Let’s go ahead and just grab a file off the desktop and show you just what this puppy can do. So that is saturating the read speeds of the RAID 1 SSDs that each of these machines are booted off of, so that was a 10 gigabyte file. Let’s go have a look on the other end of this Ethernet cable at how exactly that whole thing comes together now. I’ve shown you guys our server room… fair number of times, but I haven’t given you guys an update in quite a while on How exactly it’s working in here. So the main server that everyone is editing off of at the same time is this one right here. This is Whonnock server, and it’s running 24 plus 1/2 four or five six seven, so it’s running 31 Intel 750 series 1.1 terabyte NVMe SSDs, and this guy is an absolute monster. So if we fire up Performance Monitor, you can see that even though we’ve got four of our editors in office right now, And we’re seeing each of them doing a hundred and fifty, one hundred and twenty, Fifty megabytes a second of reads. This thing is barely suffering and our There we go. Our Z disk queue-death is only about 0.5. That’s the benefit of Nvme, is that nice fast responsiveness. And in fact when we were still using our SATA SSD server for every one. We were starting to run into issues Once we had the room fully staffed over there, where Adobe Premiere would crash, but not just crash. Everyone’s would crash at the same time and we narrowed that down to slow response times on our NAS. So this guy’s connected at 40 gigabits per second. This one right here has 24 SATA SSDs. And this one only gets used This is called QQ server. This one only gets used for Large projects that only one person needs to work on at a time. It’s still SSD based. So we still don’t run into any crashes or any other weird issues like that, But it doesn’t have quite the snap that an NVMe machine does. Now something that we learned a really valuable lesson about I guess it must have been about two years ago was Real-time data replication and off-site backup and that is where this guy comes in. So I want to show you guys a little trick here… I don’t know. If I’ve ever actually done this demo in a video before I’ve shown people that have come into our office. So I’m going to use my test folder here and I’m going to create a document called “Test For Video” right here. This is a text file on our main drive. Then I’m going to jump over to this hard drive based server under it. So this is running 8 8 terabyte drives for a total of 64 terabytes, And I’m just gonna pull up the Whonnock sink folder… Get that test folder open, test Iridium- Wa-pow! Just like that. So within about five to ten seconds, Whether it’s a text file or whether it’s a large video file, the synchronization software that we’re running here will Automatically dump it over to here, so if this crashes we can actually continue editing videos within about 10 to 15 minutes of Switching over everyone’s map drives. And another benefit that it gives us is that normally on a network drive when you delete a file? It’s just gone, but instead check this out. If I delete “test for video”, I’m actually going to have to do it from a separate server, but if I delete “test for video”… Boop you’re gone you’re done. And I go into here… That file is going to disappear. But… What will happen is we can go into the deletions folder and we can rescue it. This has actually saved our butts more times than I would like to admit, because Stuff does get accidentally deleted. Now all of this is only for active projects that we are working on right now. Once we’re done with something, it goes on to Petabyte Project, and we’re actually running Both phases of it right now when we originally deployed it. We were only using one of them so we could just save power on ours for all the drives that were in the other one. Not the case anymore. So Petabyte Project is now up to 777 terabytes of total space of which we’ve consumed four hundred and thirty-two. This holds all of our archived projects, so that in the event that we want to make a video that refers back to one of our other videos, We can grab the original quality files rather than downloading off YouTube like a lot of other YouTubers do. Bringing us into this room… One of the biggest challenges is not just working with red footage because the files are so heavy, one of the biggest problems that we have is that we’ve got Multiple people working off of that same shared NAS all at the same time. So this is Where the 10 gigabit connections that our editors are also using comes in. When we’re scrubbing through footage, We can actually see data rates in excess of two-and-a-half gigabit per second, so we peaked at just shy of four gigabit per second. That My friends is why we’ve got the bangin NAS with the high-speed networking. So once the edit’s done, Taron or one of our other editors will export the entire project as an MOV, but those are extremely inefficient Files, so they’re- they’re great quality, but they’re absolutely massive. So he’s gonna copy that those in the transcode flow plane. Yeah, and then you paste it here… Now what’s gonna happen there is that Water-cooled server that I built a while ago the one that’s at the very bottom of the rack, Is going to see that that file got transferred, and then it’s going to spit out the correct formats for all the different platforms that we upload to, whether it be YouTube or Flowplane or Facebook or whatever the case may be that way. The editors machine doesn’t get tied up with this. So this is cool, Once that file is done copying. It does take a second and Media Encoder is not always perfect Before it’ll pick it up, but we use VNC in order to remote into that machine So everyone can go in at the same time unlike Remote Desktop Connection, and make sure that it has actually picked up the project and that it’s transcoding to the correct format. Okay, cool. The system works, so Thanks for watching guys. If you disliked this video you can hit that button, But if you liked it, hit like, get subscribed, or maybe consider checking out where to buy the stuff We featured at the link in the video description. Also down there is our merch store which has cool shirts like this one, and our community forum, which you should totally join.
72 Comments
Ajay Koemar
9:57 You don't need to remap network drives if you use the Windows Distributed File System. Look it up in the Windows Pro IT Center.
ProW YT
solution: then, don't record in 8k, everything won't have to be that complicated and i dont need to hear ur complain on slow read speed.
Jay Doherty
Did anyone see the file anus.txt at 6:37 ?
Magnus Nilsen
I just have 15MB videos.
Aram K
Cool video but extremely inefficient workflow
Mrk 101
I wonder if linus is going to upgrade hus ingest station with new PCI 4 nvme drives
LeadUR Subconscious
gay red cameras 4o k price tag and doesnt have a decent microphone lmao you phone video shoots better who watches 4k on youtube really
Eyezzzonly
I wonder. Why on earth you still use these "Mags" or SD-Cards generally for recording. Isn´t it possible to uplink it directly to an PC or Device directly somehow? I mean come on, i cant imagine, if a film crew is using that camera, that they are sticking with 960GB Mags….how many are they using you think? – I think there must be some way. For example high-speed-Cams are using devices directly for accessing or writing the data…you have a 10G or 40G Network there. so use it to write directly….would be my suggestion
Mr Corndog
Because they video making them
Skyler Lambert
Activate Windows linis
guthexn
11:43 that dude in the background made my day
Leon GameplaysTV
0:41 i got smooth 4 too
999code
what is the synchronous software you use, you never actually said it. or did you and i missed it
Andreas Petersen
Leet
ZenThreal
12:07 ( ͡° ͜ʖ ͡°)
Steve C
So a couple of file servers with a backup sync box. C'mon more servers please.
Werewulf
At 2:38 this mans says master and a slave 😳okay buddy
Justin Orlando
No offense, but the gimble made me almost throw up
Ghryst VanGhod
may eye brows the video at 12:07?
Meinkraft
Petabyte Server gets half full after about one year, now its 2019. Is it full already?
Stefaan Codde
Why insist on 8K while the world is still looking to 1080P since the providers still give no 4K via sattelite, local Telenet also not.
encrypted
You sound a bit like todd howard
Tiger X
SO this is why im studying IT… this is what i wanna build 😀
Rawknee Salami
How many video has linus made involving this server??
Liam
ok what I don't understand is that do his editors edit the files directly in the server or do they download the files, edit the video, then upload them back to the server?
Luke E
I bet they all feel stupid for spending tens of thousands of dollars on RED cameras and even more money on upgrading their systems to handle that kind of footage when Black Magic just put out a 4K camera for $1000
Malcolm Sharpe
Hi linus, im back to computing after the big wide world, and ive found a c7000 for sale on the net. im wondering how much difference a qlogic card makes on the blade? will be far better for compiling CAD animations , for im at Uni and id start it as a business. im looking at the centre full of g6/7 blades. If you could help!! Malcolm
Ahmed Hameed
you can make it more simple , but you just rich enough 🙂 I mean comon … for simple review you dont need that red camera
C8zAr
Why film in 8k when most of the people watch your videos in 720p on their smartphone?
Avizzini888
How does you data replication work?
Nathaniel Guevin
pourquoi vous n'avez pas de filtration dans vôtre server room pour enlever la poussières.
Nathaniel Guevin
Why don't you have air purifier for your server room because you have a lot of dirt on the desk in the server room.
Karl Massie
here,s a thought stop using 8k its fucking pointless films dont even use it
Ismael Hernandez
Understood maybe half of the words he said. I am completely lost but for some reason the video is so interesting I must keep watching.
GLaDOS
what are the names of your servers, linus?
jaguar3217
13:37 length! :O
Amit Dhanokar
So basically… You dont trust YT and keep your own files
I guess that is costing you a lot
Kato Klinck
Why don’t you just clap your hands to sync the audio, pretty easy since you’d be able to see the peak in the audio read out on premier
Tey Treet
Why editors have to access real time from NAS. They can't download the related clips to their local hard drive first?
Alexblackish
1:29 isnt that vitaly's ex?
James King
Sees thumbnail
Is this a sever?
Anvar Jamal Saifi
They have $$$ to spend , so they are spending.
TheAsylum100
You have gone to anal and over the top, youtube channels far bigger dont record at this level, its about content all the way. I dont care as long as its watchable, auto mode on youtube is 720 hd, you dont really notice any difference above that for general videos. Waste of money and time your end because about 5% of pc users have monitors that can show it.
coochie nigga
Did you get an tattoo?
R__B
Why do they blur when he uses the keyboard?
By Montolla
what that issue whit white light over linus hand and screens?
By Montolla
That the only server room I saw with a 4k monitor LoL
SumriseHD
13:37 long… nice!
NTZombieSlayer 204
7:55 anyone else notice the music is synced with the nvme ssd leds ?:D
John Richardson
I couldn't pay attention to this video. All I kept thinking about was that I would have never known did this was shot with a much much cheaper camera if you didn't tell us.
mahdi hassan
You said that you recorded it with an iPhone but I saw a OnePlus 5t on that gimble
H. H.C
300 mbyte/s ? What are we getting on youtube with 1080p +?
Павел Анатольевич
13:07 using "**********************************" folders as separators in quick access area, interesting solution.
Diesel Power
least homogenous setup 2019
The Vw Fan
12:05 look at Teran 😂😂
ChillyFilly Gaming
And still using a free copy of Windows 10
Max Coplan
Dumb question, but why doesn’t LMG use dedicated Linux servers to act as the storage servers instead of Windows Server?
Tahira Javed
I love these server videoss
Big Mike
Jesus fuck, and I thought my 64gb thumb drive was big.
Gonzalo Suarez
And if shooting in 8k gives that amount of trouble, why not just record in 4k? Because 8k nowdays is useless and the amount of people who can watch someting in real 4k is also limited, shooting those massive files is kind of unnesesary, btw great videos, nice to have so many "toys"
Yannickbmgames NL
The audio is out of sync
pacman81
Literally didn't notice difference in video quality.
Gowtham Natarajan
How do you switch between servers in the monitor?
Jonathan Correa Paiva
I use the export recycle in Freenas, They said we don't needed, but im the hero now. haha
DonavonTaylor98
Solution use 1080p no one is going to youtube and going oh god i cant watch this low quality garbage :/
Edward Vazquez
You guys still use tapes as fail safes?
SHUBHAM PANCHAL
Stuff accidentally get deleted. 😖😔
Avox7Gaming
Prores vs cineform
Staton Otto
I just realized how much difference there is between what i do at my job and what you guys do on your servers. TY Linus!!!
AaronShenghao
I got Linus doing honey commercial before a video made by LTT…
Daniel Dougan
Is that a box fan behind the server cabinet?
Kuklyper
Did he actually mean raid 0 6:40 because its faster than raid 1
Next Level Code
Linus needs to dump adobe media encoder / adobe prelude for some shell scripts ffmpeg and some Linux love. Too much unreliable windows / adobe stuff going on that can be cut out of the workflow.