** UPDATE.. here is a new video with my new system.. **
If you are a photographer, or anyone else that processes very important data… this educational / workshop video on my data workflow maybe inspiring.
Join me in this complete, in-depth look at the data process starting at the point of capture, all the way to archive and everything in between.
Without doubt this is the most important thing inside of my business, and I am excited to share this with all of you. It has been comprised of over 6 years of practice, processing over 1-Million RAW images to date!
If you enjoy videos like this, please subscribe to my channel as this motivates me to continue to make more content in the future.
Thanks for watching!
To see my extended thoughts on this topic, please visit my blog post on Data Workflow here:
Special thanks to: Andrew Applepie for some of the music used in this video. You can checkout his work here:
Here are some links to the products / programs referenced in my video:
Carbon Copy Cloner:
All hard drives used:
APC Smart UPS 2200:
everything that I do inside of my business all begins and ends with some type of a workflow this is by far the most important thing in my business and honestly becomes the foundation for which everything I do and create is based on so if you're a photographer or anyone else for that matter that's in the creative space that's interested in how all this stuff works this is going to be the video for you because I'm going to show you precisely how I do so there's actually four main workflows that comprise my business first is a creative workflow and this has everything to do with photography the technical the composition the lighting setting up the shot camera settings and all that good stuff the second is a business workflow and this really speaks to the ins and outs of the business paying taxes doing the contracts communicating with clients keeping track of what's going on the operational side of running the business there's the marketing and blogging and social media components you know the post workflow the the destination of the images that are created where the pictures go and when and why and how and and then there's the data workflow and this is the in my opinion the most important part is the throughput of the data the the data integrity itself uh from the point of capture straight through into archives where that data lives how its managed where it goes that is what I'm talking about today so right out of the gate I just want to make a few points has a bit of a disclaimer number one I'm not saying that my workflow is best I don't believe in the term best I don't think there's any place for the term best to even exist in the creative spaces at all and just like the end result in which is a photograph you know everything becomes very very subjective from one person to another but what I will say is that my workflow is very effective it works incredibly well for me in my business and what my needs are as a professional point number two is this is not a workflow that was slapped together a couple weeks ago this is something that I've had in place for well over six years since I last updated my computer system it has seen and ingested and processed over a million shots in that time between myself and my assistants all the shoots and projects and videos and all the stuff that I do it's it's been amazing the third point is that all of this is very scalable so if you're just starting out or on the flip side if you have an incredibly busy studio in a storefront and so on everything can be scaled down or scaled up depending upon the volume of business that you have alright and with that said let's get to it the data workflow starts right at the point of capture I choose to use a pro-level camera I pop in a light cannon and one of the most important features of their flagship 1dx is the fact that I can write data to two different memory cards simultaneously I keep 128 gigabyte card in slot number two and this is enough to hold the entire wedding more importantly I switch out to smaller cards using memory slot 1 all day long as they become capacity so this effectively gives me what's referred to as a raid 1 data protection right out in the field if one of the memory cards becomes lost damage stolen or otherwise compromised I have all the data on a secondary card safe and secure a quick tip when using memory cards use smaller memory cards whenever possible I use the big 128 gig card as a backup and memory card slot number 2 in my 1dx and memory card slot 1 the most I'm using is either an 8 or a 16 gig card god forbid you lose a memory card you delete something you format over the card anything can happen to these memory cards by switching out to smaller cards throughout the day regardless of what you're shooting it's a way to mitigate your liability and control the risk now when I get back from the wedding or photo shoot obviously all this data needs to be ingested into my computer system here's how that's going to work first I connect both mine and my assistants master memory cards to the computer system simultaneously and start downloading all the images while I go take a shower and have a bite to eat so the big advantage of using 128 gig memory card as a backup in the camera is the fact that when I get home it makes ingesting the images very very simple because everything is on one card as opposed to having to download multiple memory cards so in a way switching out to those smaller cards those actually become the backup to this I love it all of this data is ingested into the computer system through this magic little program called photo mechanic in the entire scope of photography I would say there's probably just a few things a handful of things that you must have to be very successful photo mechanic is one of them it is the most absolute positively the most amazing program that I have ever used when it comes to moving data from point A to point B and to culling and sorting out all of my shots so the first stop on the data pipeline is what I refer to as my digital warehouse and this is AG tech G speed cue it's three terabytes in capacity it's set up as a raid 5 and it's comprised of 4 separate hard drives that are enterprise class and together make up the total capacity basically it's just a box with four hard drives in it it's a raid 5 which means that all the data that's written to it is what they call striped across all 4 drives for faster access as opposed to having just one hard drive with one drive that just spins at a constant it's nice because this hard drive can actually withstand a failure of one hard drive and the other three have enough data to rebuild itself but that's not enough so every single morning at 3 a.m. a program called Carbon Copy cloner try saying that 10 times fast and wakes up my computer it looks at the data that's on this digital warehouse Drive and it then copies it to another Drive which lives on the other side of my desk and this is called a live work backup Drive this is a GTECH raid 1 drive enclosure it's also 3 terabytes in size and it exactly mirrors everything that's on my main digital warehouse drive this is the backup drive for that hard drive so every single morning when I wake up I know that I have two sets of copies of everything that's current and live in my world and within those two different hard-driving closures they each can lose drives and still be okay now the directory structure on this drive is absolutely critical to my workflow here it is at the root level I have a folder for all the different types of work that I do weddings engagements commercial photobooth other in Austin Austin's my son he's three years old and he gets a dedicated space everywhere in my life this is my son Austin and say hi when the dialog box opens for photo mechanic I then create the following subfolder structure for each event that I do so here's a wedding example I would use the reverse date first the four-digit year the two-digit month and the two-digit day underscore lastname this just serializes all the projects so I have a way to reference them later on when things go to archive inside of the clients root folder I then make another subfolder called raw so all the images are then copied into this raw folder and from there they're sorted and called using photo mechanic and I remove all the test shots or on flattering images and I get it down to the to the core shots that are going to wind up being delivered to the client once I have that they're then copied into another subfolder okay here's how this works I drew it a lot on paper for you first you have your hard drive then you have a folder for your category in this case it's weddings then I create a folder for the each client with the reverse date underscore and their last name then I simply make a subfolder called Raw and this is where all of the images get downloaded into once I'm finished sorting out all the images they then go into this last folder which is the reverse date underscore the last name and then I add the word final to it this is then the folder that gets ingested into Lightroom and becomes a referenced catalog this now becomes the live folder and it's what will be a reference for the Lightroom for all the color correction at this point we're finished the photo mechanic both Lightroom and a master referenced catalog file live on an internal solid-state drive on my iMac by keeping the program and the catalog file on a solid-state drive the program runs very quickly and it's very easy to make edits now there's absolutely no need to store all of your raw files on a solid-state drive keep them on a fast rotational drive and as long as you're using a reference catalog it'll be just fine so within Lightroom you have what's called a catalog file and this is super important because the catalog file is what contains all the adjustments all the edits and all the data that you do to your images there are two different types of catalog structures one is managed and the other one is referenced I use both of them but for very different reasons a managed catalog file means that when you import into Lightroom it is going to copy the data the imagery from the source into this folder which is the catalog and it's going to manage all of the files inside of its own little folder structure the problem with that is that your your moving data unnecessarily through wherever it existed on a memory card or hard drive into a folder and then from there the catalog file in Lightroom is going to start to get very very large Lightroom runs a lot slower as a result for performance I use a referenced catalog file and this is by far the best way to use Lightroom and to have the fastest performance out of the program a reference system is going to go and it's going to leave the images alone on the source in my case it's that digital warehouse hard drive and it's just going to copy over the thumbnails of what the images look like into the catalog file just some quick tips here on Lightroom again keep the catalog file on the fastest drive possible ideally that's a solid-state drive you'll thank me later do not I repeat do not import all your images from your memory card directly into Lightroom that is a huge mistake by doing that you're just going to bloat up the catalog file and you're gonna be building a ton of unnecessary previews that the program has to render before we can even get to the point of sorting out all the shots huge mistake that's what flow mechanic is for there is nothing that's going to be better than that trust me don't do it inside lightly you want to keep data moving through your workflow you don't want things to just be stagnant you don't want to have like this massively large catalog file in Lightroom you don't want to have a ton of images sitting around on any one drive the best workflow out there is one that remains very transient you want to be able to have the data come in be managed and processed color corrected whatever you got to do and then off to the client and then out to some sort of an archive system so that it's off your plate and it frees up your main operating system so at this point I'm completely finished with Lightroom I've done all the processing and color correction I then simply export the project out of Lightroom online to a private gallery for the client to see pixie set is the online gallery that I use to transmit all my images to my clients they can see them they can share them they can download them it's fantastic I absolutely love it and it's active for 90 days after that time I charge a hundred dollars to pull their images from the archive drives and reactivate the gallery so the moment that the client online gallery finishes uploading I then go back to Lightroom I select the project I right-click on it and I select export as a new catalogue I select my archive drive which is another GTECH raid one enclosure I ensure that I select the dialog box that says include master files that's super important because what that is effectively going to be doing is creating a managed catalog file on the archive drive so unlike the original catalog file that I'm using to process all the data in I'm now converting over to a managed catalog for archive purposes so now what's going to happen is Lightroom is going to export out all the edits all the corrections and all it's going to bring in all of the actual raw data files that was living on my digital warehouse drive it's going to take all of that and put it into this nice little file which becomes both the catalog and the container for all those raw images and it's going to put that on the archive drive this way down the road if I need to reference a shoe or design an album or re-export if a client loses pictures whatever the case is I can just go back to the archive drive double click on that catalog file on the actual hard drive and then it's going to open Lightroom and then everything is going to be in there and I can re export it out this is amazing because then I go back to my master catalog file and I can remove that project from it because it's finished and now it lives on the archive Drive and that keeps my data workflow very transient it prevents any one of my drives from becoming capacities it keeps the catalog file the master catalog file from becoming too big and bloated and it prevents it from just slowing down if you continue to upload and import projects into Lightroom and you start to have a lot of them in there even if it's over even if it's a reference system it's going to slow down it's going to get to the point where it takes several seconds to apply adjustments to images it's just not a good working environment again the idea here is to keep the data moving right through the workflow and right through Lightroom once the project exports out I then verify the integrity on the archive drive once it's been verified I then go back to the master reference catalog file in Lightroom and I delete the project completely freeing up the space for the next event at the 90 day mark when the online client gallery expires I then go back to my digital warehouse drive and delete everything keeping in mind that everything that was delivered to the client that's in that gallery is safe and secure in a managed catalog file on the archived drive simply put this ensures that none of the main operational drives in my system ever become capacities only the archive drives and that's totally fine that's exactly how this is designed to work this just ensures that I have three months the client has three months to look at everything and make sure that nothing was missing I didn't forget to include anything and there's no discrepancies of any kind it's a little bit of a safety zone so to speak so there are a few other hard drives on my desk that are very important to the overall data work one is the data drive this contains very important backup for my QuickBooks and tax information things in my calendar my website and blog all the client contracts all the important logos and graphics for my business all that stuff is on there but most importantly this drive facilitates – automated daily backups the first is of the Lightroom master catalog file itself and the second is the database file for my client relation management software this software contains all my important client details email correspondences and all the details for the photoshoots in the wedding's so the last GTECH raid1 enclosure that I use is dedicated just for video so videos like this almond behind-the-scenes videos personal videos videos on my family and son all anything at all that the video related lives on this hard drive so the sixth and final hard drive that's connect to my computer system is just a standard GTECH one terabyte hard drive and this is dedicated to be used with apples Time Machine program this program is making hourly backups in the operating system of anything that's going on on the local machine itself things like emails and computer settings any documents that may be open and I'm actively working on that's making an hourly backup on that time machine drive it's convenient because if and when the iMac computer fails I can run out buy another computer bring it home drop it on my desk plug in this time machine drive and be able to restore everything perfectly it's important to note though I do not rely upon time machine nor this hard drive for any of the mission-critical data for that I have the hard drive enclosures and the third-party programs are making those automated backups every day you ready for this every five days one of the driving closures from all the hard drives on my desk is removed and then they're transported to my top-secret location and then each one is swapped out to last week's third rotational harddrive copy that lives off-site when I get home I just slide the drive into the enclosure I power it on and I go to sleep well I'm sleeping the drive is going to rebuild itself sector by sector ensuring that it's exactly the same on both drives when I wake up in the morning I now have a new enclosure with all the data being up-to-date in exactly the same plus another copy that now lives in a watertight military-grade Pelican case several miles away look the reality is this you can have all your data backed up in multiple locations on multiple hard drives different driving closures and all this good stuff the issue and problem still becomes your physical location anything can happen here there could be a fire that could be a foot in the middle of the night a pipe could burst and flood the whole place someone could break in I got pretty good security though I mean anything the point is anything can happen so you need to have a third copy of your important data that's got to live somewhere else you have to have a regimen in place to get the data out from your place of business and somewhere else where it's really safe and secure now I know that there is different cloud computing options out there I if that works for you then then certainly you can do that personally though I love the iCloud system for example on my iPhone if I lose my iPhone or if I break it and need another phone I have used iCloud to restore that and I think that's really fantastic however that's where it stops for me anything at all that I'm being commissioned to do where my name and reputation things that I'm contractually obligated to manage and produce those things I just need to have the peace of mind and security of knowing that I have a physical tangible copy of this data that I'm in complete control over at all times that's why I do it this way I just can't trust a third party with all this important data I mean this is my entire world this is my livelihood this is this is everything managing the data is absolutely the most important thing inside of my business so the reason why it's five days for that off-site backup is for the fact that the data still lives on the memory cards that are still camera bag from last week's shoot so before the next weekend where I might have another wedding that off-site backup is performed before I go and format the memory cards to shoot that next event or wedding or whatever it's going to be so last but certainly not least to manage all the physical power demands of the computer system the second display all the connected hard drives I have this massive power supply this is called an APC UPS 2200 and basically this is a huge battery system that is also a computer and it talks to the iMac and it's smart in the sense that if there is a power loss this device is smart enough to go into the computer and shut down the operating system safely closing out all the programs that are open it will then go and systematically power down these hard drives in the order in which it's programmed to do so this way there's no chance that any cord gets ripped out there's no chance of any data loss of any kind because of an outage this power supply will also fully operate my entire production table for up to two hours consistently so that way if I'm home and I'm in the middle of doing something important I have plenty of time to finish working on that project before I power stuff down and if I'm not home then it goes into its automatic mode and it shuts down everything all on its own anyway so either way it's great so that about wraps up the data workflow video hopefully you enjoyed the video and it was helpful to you in some way if you do have additional comments or would like me to further explain anything be sure to just comment below and I'll jump right on it if you enjoyed the video and would like to see me make more please subscribe to my channel and share the love on social media thanks so much and I'll see you in the next one you