Looking for design inspiration?   Browse our curated collections!

Return to Main Discussion Page
Discussion Quote Icon

Discussion

Main Menu | Search Discussions

Search Discussions
 
 

Harry Torque

9 Years Ago

Reminder About The Need For Multiple Backups

I've had the backup scheme of all my photos fully stress tested by reality in the past weeks.

I have all my photos on a storage server I built, a redundant unRAID array. I had a drive failure recently, but since the array is protected by a parity drive I didn't lose any files: the array can withstand single drive failure without data loss. I replaced the drive, and re-built parity to retain same level of protection.

Within hours of being back fully online, a second drive failed, this time my main image drive!

Now, if the second drive failure was while parity was being built (which takes 36 hours) or if two drives fail, I would lose all data on both drives. So I got lucky. Really lucky.

While the array was preparing new drives for the array, I woke up to the fire alarm from the restaurant downstairs. It wouldn't have been the first time there was a false alarm, but the smell of smoke made me re-assess my initial hunch and desire to drift back to sleep. I put on my slippers and bathrobe, and grabbed my last resort external backup hard drive and head out. One of the deep fryers in the restaurant had caught fire, and the smoke was billowing through doors and windows. Three firetrucks arrived within minutes, and put out the fire.

There were no casualties and no serious damage to the restaurant or my apartment, but it really made me think about my backup scheme. I've always been an advocate of offsite backups to recover from fire or water damage and theft, an often-overlooked catastrophic failure scenario. Some people justify their lack of offsite backups by saying "I will have more important things to worry about than my photos if my house burns down." Perhaps, but I've heard firemen say that the one thing people miss most are their photos. You can buy and replace pretty much everything, but you can't replace photographs. For those of us making a living from art, a fire can cripple your livelihood.

I've heeded my own advice to a certain extent: I have an offsite backup using Crashplan. I tested it out, and recovery would have been possible, but downloading the nearly terabyte of data would have taken almost a week. I did not have a second offsite backup, although I've been meaning to do one for years.

I had two local backups, one of them physically with me at the time of the fire, so I would have been able to recover critical files even if the building burned down.

But due to the fire, I will be adding another layer of backups: I have ordered two external hard drives, which I will be rotating on a monthly basis. One I will always keep at the office, and the other at home. That way I will always be able to recover from a HDD or the cloud if worst case scenario happens.

Well, the worst case scenario is zombie apocalypse. Then all the digital files will be useless. But at least I will have my trusty film cameras to document it all :)

Anyway, here is my recommendation for a bomb-proof backup scheme:

- Backup hourly or daily to a local HDD, which must be different than your working HDD or your storage array
- Backup weekly to an external HDD that you don't keep attached to the computer (this is mainly to protect against user error and virus protection). The way I have it set up is I have a script running every night at 3am, which will backup files to an external USB drive I only turn on once a week
- Backup monthly to an external HDD which you will keep at another location (family, friend or work). Same setup as above.
- Keep an online backup, but be aware that it can take weeks or months to upload your data, and even longer to recover. I use Crashplan, there are many others. But please research data and speed caps before you commit, lack of both being the reason I went for Crashplan. Also, some services will delete files after a certain time period if they don't exist on your computer anymore, which will leave you vulnerable to user error or destructive viruses.

Two additional things to note:
- Your backup is only as good as your ability to recover them. So test that you can do that easily and quickly.
- Don't just throw files on an HDD and expect it to last forever. Some backup software uses checksums; I used SyncBack Pro when I was on Windows. On Linux you can run md5 hashes to ensure that the files are intact, and that the backup HDD works. Run these tests periodically, and make sure that every file is tested, not just recently added ones!

Reply Order

Post Reply
 

Louise Reeves

9 Years Ago

Well, some of what you were saying wasn't a language I'm familiar with, but I got the gist. :-D

My 1 TB external crashed completely a few months back while I was in the process of getting the files off it. Unfortunately, I did lose almost everything, including tax reports from the last four years and it will cost me roughly $800-1000 to attempt recovery. The place I go to for computer services was not able to do so; they sold me a USB/SATD case that didn't do the trick either, although it did end up working for a previously crashed external. After that fiasco, I took all my raw files (I save my cards, I don't reuse them) and organized them on a Passport, then filed away the cards. But my finished work from 2007-2012 or work in progress, gone.

With the purchase of my HP Touch Envy laptop, I got free Cloud usage, but have yet to actually check it out. I still have to get what is on my PC to a Passport, then get the laptop work to the same Passport, then put it on the Cloud.

I will warn everyone every chance I get that, should you be looking for a file recovery program, do NOT, under any circumstances, use Stellar Photo Recovery. I paid $50 for it and it completely wiped out 300 images from my SD card. Gone. Then their "customer service", which is not in the Western Hemisphere, asked me over and over for a "screen shot" of the recovery preview which I repeatedly said does not exist because the card was wiped out. Eight different people were responding to my frustrated emails with at least half not understanding the simple problem. Seems whoever gets that day's email first answers with no clue about the previous correspondence. Very poor service and horrible program.

 

Mary Bedy

9 Years Ago

You guys have me at a loss with the technical language, but I also bought 3 years of Crashplan. I think I have 6 months left. It may take a week do download everything in case of disaster, but it's better than just completely losing it.

I do need to buy another external hard drive. My other one - a hand-me-down from my son, filled up over a year ago. In fact, since you posted this, Harry, I may just order one tonight. I only need 1 TB at this time anyway and those are all under a hundred bucks, I think. I'll just plug the computer in and let it run for the 24 hours it will take to back up.

I also back up to DVD, but I hear all the newer laptops don't even have a disc slot any more. I'll have to rethink that.

In any case, Louise THAT SUCKS. I feel bad for you. Reminds, me I have not done a backup at work for a while. Will have to plug in the external when I get in tomorrow. 99 percent of it is backed up on a server in Italy, but just in case I missed anything.....

 

Jim Hughes

9 Years Ago

Great story Harry. The failure of the RAID drives is a perfect example of the 'Law of Unintended Consequences', i.e. how something we think of as 'progress' actually leaves us worse off than before. Today we shell out for a RAID system assuming it's vastly safer than a single drive. But if the quality of the individual drives goes way down, you didn't get the security you paid for. In fact you ended up with a situation that was more complicated to deal with and arguably less secure overall.

I used to use Amazon S3 but have now switched over to Microsoft OneDrive and backup is automatic; my photos are all stored in the OneDrive folder which is automatically mirrored on the Microsoft drive. Recently I bought a new computer and simply allowed OneDrive to download all the photos to the new PC, in the background. It took a few days.

 

Dan Richards

9 Years Ago

I use a raid mirror on my main drive, and then a online back-up as well as hard copies of my work. My raided drives are on a hot-swap with fan coolers. I will say on you most used deives, never use them without hard drive fans, they get hot and burn out quicker.

 

Roy Erickson

9 Years Ago

I refuse to worry - when it's gone - it's gone. C'est la vie. When " I " am gone - it is all gone anyway - there will be no one to 'pick up the pieces'. FAA will have them - but after a year and the credit card bounces - they'll be gone from here as well. Most of them are now on two different external hard drives and about 20 flash drives. If they don't survive - like everything else in life - when it's gone it's gone. All that will survive are those that folks have bought/have printed - the rest won't even be dust to dust - it will be nothing to nothing.

 

Mary Bedy

9 Years Ago

Wow, Roy, we are waxing philosophic today ;-).

I know I have stuff I missed backing up and in fact I used to delete things from my old XP upstairs. I should also back up on my older macbook - I know it still has some room on it. But you know, there comes a point where you do what you can, and after that it's up to the universe....

 

Suzanne Powers

9 Years Ago

Harry,

Thank you for the heads up and information! I'm not sure I understand all the terminology, I am going to use DVDs, and Flickr for now.


Mary,

From what I understand internal DVD players shorten the computer battery life, most of the better laptops don't have DVD players for this reason. External DVD players don't use the computer battery only the current that runs through the cable to the computer.

I found out my cheep surge protector failed (buy only the most expensive one in the big box stores) and caused the external drive to overheat recently. I think the main internal connector melted in my Western Digital (I don't recommend this brand from Walmart, this part is prone to meltdown for this brand).

You can tell when your surge protector is no longer effective by shaking it, if there is a faint rattling sound inside that sounds like an incandescent light bulb when shaken, it is worn out and not effective.

 

Robert Frank Gabriel

9 Years Ago

Well, I use "flickr" as one of my backups for all my good photographs. It's free, it's fast, it load and downloads at full resolution in a second or two, and I can upload some 200,000 images for free.

I also make 4x6 prints from a good Epson printer of all my good images. I can always scan them or photograph them if need be.

 

Mary Bedy

9 Years Ago

Ok, I keep hearing people say they use Flikr as a backup. Can you do that without the public seeing the photos?

My son used Flikr eons ago, but since we're all on Macs and i-phones (well my husband still has Window), we can share anyway....

 

Joy McKenzie

9 Years Ago

Carbonite cloud storage runs all the time on my main computer, where I do all my work. I also have an external drive. $59 a year for Carbonite and I have peace of mind. I live in Earthquake country...San Francisco. Also FIRE country...this awful drought in California is threatening to burn the whole state away.

 

Harry Torque

9 Years Ago

@Suzanne and Mary

DVDs are not a reliable backup. I recently checked tens of DVDs from 8-12 years ago, and almost none of them worked. Bitrot is a real thing. My advice is to check your backups periodically, and don't rely on just one method.

@Jim

Yes, RAID is not backup, and I never intended it as backup. Its main attraction for me is to have huge amount of disk space on one "drive", mine is 26 terabytes.

@Robert

Flickr is not really a backup, unless you shoot JPEG only or don't mind losing RAWs. Another major problem with Flickr is how to recover 200k images (or even 1000) easily?

4x6 prints are not good for backups either, except for snapshots. Scanning quality will be very poor, from a low-resolution print.

 

Mike Savad

9 Years Ago

my set up wouldn't survive a fire, and its too much for online storage. i do have things on many drives, but i have to be more aggressive backing things up even though i do some of it daily. last week my brandnew drive died, and killed everything on the drive. i didn't lose too much, and i'm still waiting for the new one to come back... it's rather annoying.

---Mike Savad

 

Mary Bedy

9 Years Ago

CrashPlan works like Carbonite - it backs up as you're working and it sounds like it's the same - 60 bucks a year - although I bought 3 years worth as a discount (don't remember how much). I really do need to get that external drive. I know DVDs go bad.....

 

Harry Torque

9 Years Ago

@Mike It doesn't have to break the bank: CrashPlan with unlimited storage is around 4 USD per month with a four-year contract. I still don't understand how they can make money with that, but they've been around for 5+ years, which is how long I've had a contract with them. Another option is HDD; a 2TB drive is less than a 100 USD around here.

Put another way: is the downtime from recovering from your current backup and likelihood of lost files in case of fire worth more or less than 50-100 USD per year?

Your experience with a new drive is one of the reasons why I advocate running those hashes and checksums to ensure backups work. There's a script for unRAID which stress-tests the disk before you put it in the array so that it will hopefully expose the ones which are likely to fail in the first few months, but haven't looked if there are similar scripts for other OSes.

 

Samuel Sheats

9 Years Ago

Like Joy, I live in the SF bay area and use Carbonite. This is my third year with them. It literally took 6 days for the initial backup that ran all day in the background during that time. I've only had to recover files 2 or 3 times and so far the process has been very straightforward and reliable. As long as you have internet access you're good to go; even on someone else's computer, if you want.

Years ago, I learned the hard way to always disconnect your computer (and printer) from the wall/power strip/surge protector during a storm or when a neighborhood power pole transformer randomly(!) blows up. When power is finally restored after an outage, that nasty surge -- which might include a split-second voltage fluctuation -- can be unpredictable and still fry anything that's plugged in....

 

Harry Torque

9 Years Ago

@Samuel

Six days? Took me close to a year to upload several terabytes to Crashplan :P

Good point about power surges. I've lost several motherboards and routers to power outages, and bought two UPSs after one too many such incidents.

UPS separates the computer entirely from the grid. Get one which has ethernet protection as well, since that is another route where power surges can reach your computer.

I'm not entirely convinced surge protectors work, but I'm no expert. UPS also has the added benefit of battery power, so you can have your computers power down cleanly even in the case of power outage. Has come handy several times.

 

Kerri Ligatich

9 Years Ago

Years ago we lost a 3 month old computer that was on a new surge protector but did not keep the receipt. Right after that our daughters computer died, but we had the receipt and the surge protector company replaced her computer. I'm now in Hawaii and we have extreme power issues. After loosing electronics to our new washing machine, dryer, dishwasher and stove we had an electrician install a whole house surge protector to our electrical panel. We still have power flashing on and off sometimes 5 times in a row. Very hard on electronics, but the lights will change on the panel if we get an actual surge.

I just ordered from HP a 1.5 TB computer today that I plan on using just for my artwork. I have an external hard drive I was putting new work on since my computer was getting pretty full. Later I bought a backup drive to put the external work onto but it would not work with anything connected by a USB cable to my computer. So, I'm guessing some of that work will be moved to the new computer and the backup. I'll also look into a cloud service.

 

Dan Richards

9 Years Ago

Harry, in raid, that is what a mirror is for. Duplicating files on multiple drives. In that way, if one drive goes out, the other has it still, then when you replacew the bad drive, it reloads the data on that drive so again you have your back-up.
I studied the A+ manual, and started the other books for IT work, and just did not get to understanding a lot of it. The A+ was the only book I did good on.

 

Adam Jewell

9 Years Ago

If a drive crashes or something on it becomes corrupt, don't save anything else to it and run this

http://www.cgsecurity.org/wiki/PhotoRec

It's free, it works but its slow. You'll lose your file structure but can most of the time pull all the images or any other files off of it.

You'll need a second drive to save all the recovered images to.

The local computer place told me they had recovered everything that could be recovered from a drive and this pulled the remaining 700 gigs of images off if it.

 

Harry Torque

9 Years Ago

@Dan there are several different RAID methods, unRAID being just one of them. It doesn't use mirroring. Instead, it uses a parity drive. Using the data on the parity drive the array can be rebuilt if one drive fails - but not more than one.

Thanks to the parity drive you are able to physically pull a damaged drive out of the unRAID box, and the array will act like the drive is still there, with data intact. You can browse files on it, stream content, and even write to that drive which is on your desk. Doing that for the first (and second) time is a powerful demonstration of wonders of technology.

Some RAID methods do use mirroring at the expense of disk space, though.

And to re-iterate above before someone builds a gargantuan array: RAID is not backup.

 

Connie Fox

9 Years Ago

Great reminders, Harry. Thank you!

My old film cameras were stored in a drawer, and a one-day subcontractor helped himself to photography-related things around my home. I thought he had only taken my old Polaroid--but all the other film cameras are missing too. (They were in a separate location.)

Now I lock all up valuables until contractors leave, including the bonded and trustworthy maid service. But there are still some bases to cover, and it's all too easy to fail to get around to those until it's too late.

I commented to a new acquaintance on her beautiful new home, complete with all-new furniture. She said it was because a fire had destroyed their house and everything in it. The only pictures they had were copies they had sent to friends--and friends were kind enough to have a "picture party" for them as a housewarming.

 

This discussion is closed.