.
  Lightstalkers
* My Profile My Galleries My Networks

backing up your archive

after having two external hard drives get fried in one week – both were unconnected to the computer when they stopped operating. one was a maxtor one touch that would spin up but wouldn’t read, the other a western digital simply stopped turning on apropos of nothing (as far as i know) – i thought i’d ask you all if you have any better archiving system than an external hard drive b/u like i have? i have three of them, all with the same info -

luckily – any information lost was analog, so it can be recovered, but i’m concerned about the permanence of these external hard drives. burning 100 dvds also does not seem to be a good long term solution.

thanks for your help!

by [a former member] at 2005-08-04 11:27:26 UTC (ed. Mar 12 2008 ) washington, dc , United States | Bookmark | | Report spam→

I burn DVD’s of my selects and I edit pretty ruthlessly. The DVDs have no moving parts so I am somewhat more confident of them…..but with server space becoming cheaper, a secondary backup on a remote server might good idea.

Check out Digital Railroad too, they have a nice set-up and Evan is photographer friendly, right Evan?

by [former member] | 04 Aug 2005 11:08 | new orleans, United States | | Report spam→
I use hard drives in a RAID for short term storage, stuff I’m working on and my processed Jpeg archive, and burn that along with all my CF cards (I don’t edit anything out) to 2 DVD’s…one in my office, one somewhere else.

Try not to write on the DVD’s, and some people say you shouldn’t have them near your computer or monitor (electromagnetic radiation or something…).

Delkin have just launched archival DVD’s with gold coatings which they claim will last for 100 years.

http://www.delkin.com/delkin_products_archival_gold_dvd.html

If you buy a pack of 100, it compares favourably with hard drives, because for $279.99 you essentially get nearly 5 terabytes of long term storage.

It’s labour intensive to burn all those DVD’s…but so is re-scanning your analogue material after a hard drive failure. I prefer to stick with the non-moving parts option…and crossing my fingers.




by [former member] | 04 Aug 2005 13:08 | London, United Kingdom | | Report spam→
Those Gold DVDs sound good, becuase while I too am very nervous about moving parts, I also have had problems with my CDs (dont have a DVD burner yet) that sometimes punk out, so I always make a couple of them and so far have been spared any disasters. I would go with the Gold.   Imagine what all the heat and humidity down here can do to an external hard drive.  At this very moment I have a colony of ants that regularly visit my laptop.  Dont know why really.  Anyway, thanks for the recommendation.  


by Jon Anderson | 04 Aug 2005 13:08 | St Domingo, Dominican Republic | | Report spam→
Susanna— Also check out www.photoshelter.com. They do all the stuff that Digital Railroad does for quarter of the price. On top of it you can set yourself up to sell images directly online.


by Keith Bedford | 04 Aug 2005 16:08 | New York, United States | | Report spam→
Oh yeah…forgot to mention – you should try and burn your DVD’s at the slowest speed possible…I read it somewhere, dont even know if its true, and it’s a real drag…but I burn em’ reeaally sllooooowly anyway, usually at 2x or 1×.

Apparently it lessens the chances of corrupt disks.

There’s also a weblink thats been sitting in my Bookmarks for a while, but I havent had the chance to give it a good look.

It’s the Mirra Personal Server…basically a hard drive with its own IP address ( I think), so you can access the drive on the web…looks pretty interesting.

http://www.mirra.com/solutions/digital_image_backup_solution.html





by [former member] | 04 Aug 2005 17:08 | London, United Kingdom | | Report spam→
I just got a LaCie 1.6TB raid, set it up in raid 5+1, and hot-pulled two drives out to see what would happen. The thing chirped a bit, then re-built the array when i put the drives back in. Of course, no data was lost. It’s expensive, RAID will cost you 50% of the space, and you absolutely need to change the godawful fan that comes with it (the thing is incredibly noisy out of the box. 4500-rpm-low-quality-fan-at-full-speed noisy), and it "only" writes at 30 MB/sec. But it works.

I’m wondering if burning PAR sets, or something similar, wouldn’t be a good idea with DVD’s. It won’t change the number of coasters, but it might prevent some data loss with the media at some point – say 10 years from now when the dyes start to degrade. I’m not quite sure what exactly the quality problem is, and the situation has probably evolved, but i briefly discussed it with the guy in charge of the National Archives here a couple of years ago, and my understanding is that the media is inherently less stable than CD-R’s, to the point that they bypassed DVD’s even for medium-term storage. The problem with PAR’s right now is that you’d add a couple of extra steps into your workflow, where you’d have to first compress it all, then generate the PAR’s (which takes a LOOONG time) before you burn everything.

My (superficial) understanding is that burning at slower speeds might yield an advantage with low-quality media, low-quality drives or bottlenecks at the source level, or when you’re in a vibration-prone environment. The biggest problem you’ll come across is the lower-quality media, for which you can find reviews at places like dvd recordable dot org.


by [former member] | 06 Aug 2005 00:08 | lausanne, Switzerland | | Report spam→
Hi everyone-

I’m trying to work out the best way to keep my digital archive backed up and wondered what people were recommending these days. I’ve found Susanna’s post from four years ago with some suggestions, however I imagine there’s probably been some significant technology advancements since then?

Any suggestions for this technophobe would be gratefully received!

simon

by Simon Roberts | 15 Nov 2009 10:11 | Brighton, United Kingdom | | Report spam→
Any media you backup to (DVD made of gold, platinum, pewter; Hard Drives, tapes, etc) will eventually degrade and become unreadable either because the media dies or the file format will no longer be readable. So, IMHO, the only way to keep images safe is to migrate them from one media to the next as it becomes availble. Now, I have all files backed up on DVD and HD. When Blu Ray becomes affordable, I’ll migrate to Blu Ray and discard my DVDs. When solid state hard drives (or whatever the next HD technology) become affordable, I’ll migrate from HD.

In the meantime, I make sure I have plenty (five) of copies of any important file, stored in at least two locations.

by Jonathan Lipkin | 15 Nov 2009 16:11 | Brooklyn, United States | | Report spam→
I am currently using DROBO raid units. They monitor the condition of the drives, and if one is going bad, it will tell you before failure. Or, if one does fail, you can order up a new drive and replace the dead one. Info on the other drives will build the new one as the old one and all info is intact. You can also upgrade to larger and newer drives as time goes on, effectively upgrading and expanding storage without having to buy new enclosures along the way. I have 4 of them, 16 TB in all, 2 at home and a duplicate 2 at the office, and in each location 1 is for commercial work and the other for personal work. They seem pretty good so far, much better than the LaCie and Maxtor drives that would give up every year or so. What I plan on doing is swapping out the drives in each unit every 3 years no matter what, as that seems to be as long as they go before crapping out anyway. Drives are getting cheap and big, so it’s not as expensive as it was just a few years ago. DVD’s are a pain in the ass, and they are not exactly perfectly reliable either.

by Mike Peters | 15 Nov 2009 16:11 | NJ, United States | | Report spam→
I agree…migration is the key.

I too had an external drive “fry” (LaCie) or so I thought. Alas it was only the FireWire circuitry that went. Put the drive into a new external shell and voila the data was still there.

DVDs as data backup are a little more delicate than CD-Rs as the data burn is so dense it becomes easier to lose more files with a simple scratch.

I use SEAGATE external drives with great results in redundant copying.

And remember: never put your digital eggs in one giant mega-TB basket. Diversify.

by Michael Limbert | 16 Nov 2009 17:11 | Detroit , United States | | Report spam→
Thanks guys.

I’ve taken a look at the Drobo unit and it seems like it could be just the ticket.

It’s slightly more expensive than stand-alone hardrives, but once you’ve bought the casing, it can take any 3 1/2" memory drives, which are coming down in price all the time.

You can watch a demonstration on the Drobo website here- http://www.drobo.com/resources/drobodemo.php (featuring a rather excitable presenter). Useful though.

simon

by Simon Roberts | 16 Nov 2009 18:11 | Brighton, United Kingdom | | Report spam→
I use the Drobo as a secondary back up myself and have been very pleased with it. The one very important thing about Drobo to understand is; you can not pull the 4 HDD’s out and have a non-Drobo unit be able to read them. So if they were to ever go out of biz, etc. you would have to migrate your info to another system. With that, they regularly update their firmware, software and have come out with new products recently. They seem to intend on being in the game for a while.

by Bill Thomas | 16 Nov 2009 18:11 | NYC, United States | | Report spam→
Here’s another vote for the Drobo. Just remember that while it offers redundant storage, your data is still not backed up beyond the failure of 2 of the 4 drives in the array, nor against fire, theft, or a child launching it out your window. You still need to make duplicate copies of your data and store them in a separate location. I use the Drobo for my main storage, then back up regularly on 1 TB drives and keep them in a place other than my apartment.

by [former member] | 16 Nov 2009 21:11 | New York, United States | | Report spam→
I too use Drobo. It is really easy to use and offers true internal redundancy in a single place (I think better than any RAID structure).

But as the wise Preston notes, you have to also back up to another location. Fire, flood, thieves, lightning strikes, spilled whiskey, riots, insurrections, wars, terrorism, acts of God, chewing puppies, angry significant others, etc. can terminate your wonderful internally redundant system,and you are SOL.

by [former member] | 17 Nov 2009 02:11 | Washington, DC, United States | | Report spam→
Drobo all the way. I reckon we’re gonna start buying 3 1/2" drives in packs of 12 very soon, just like we’ve being doing with dvds ( gret fun ah… ). on top of drobo I am sure all of us have drives scattered around the world: mother, friends, in laws…. plus obviously the agent’s archives and Photoshelter but those are just working jpg archives. I like many here advice for Drobo cause if not you’ll never stop buying expensive drives with unnecessary casings. As we all know very well, that pile behind our desktop screens can grow pretty quickly to monstrous proportions. I just bought 500gb wd my passport drives in dubai airport like i would have bought little 4gb thumb drives just a few years ago. Until very recently i had no clue about drives. I had to learn very quickly. there are firewire 800 e sata docking units for 3 1/2" drives. It’s a pretty cheap mass storage option. Just fill up the drive and store it in a closet. very good for smaller drives that otherwise would be trashed. with video and multimedia things get easily out of control.

by [former member] | 17 Nov 2009 14:11 | Lahore, Pakistan | | Report spam→
I use CF cards for storage…in the short term they are expensive, but they are solid state, damn near indestructible and store well in a a slide sleeve. I just take the added cost in as part of my overhead, purchase them when there are deals on my preferred size and brand. I am currently working with a wholesaler to try to obtain a bulk purchase. Additionally, I burn to a DVD (safe deposit box), and to a 3.5 HD.

by Matt Wright-Steel | 17 Nov 2009 14:11 | Texas, United States | | Report spam→
I use G-Tech external hard drives with Fire Wire 800 connectivity. I make sure to purchase drives that are different in terms of serial/manufacturing numbers or production batches – hence increasing the chances to miss a bad manufacturing batch. But I use a 4 TB Drobo to back up my day-to-day work.

Long-term storage has essentially, one photo saved to three external G-Tech drives (obviously I have thousands of photos on one drive). I keep one at my home, one at my parents house and one in a safety deposit box (this option is the most expensive, but well worth it – about $100.00 per year).

DVD storage is a good short-term choice, but can be easily damaged – this option was especially not suggested from instructors at the International Center of Photography (ICP.org).

by Boots Levinson | 17 Nov 2009 18:11 | Linden, TN, United States | | Report spam→
I did not want to open this can of worms but it looks like Robert just did, so here it comes…
You can spend some money and have an ISP host your data for you. Although price has been going down, it is still another recurring bill nobody wants to deal with.
So, my advise is to setup a buddy system with a friend, mom and dad, your bro, whoever, kinda like what Robert did.

Now, when it comes to synchronizing archives, there are much much better tools for file transfers than FTP. Pretty much all of them are flavors of rsync:
http://en.wikipedia.org/wiki/Rsync
This tool was designed to do differential transfers of data (sync files and directories).
With a little reading, it is not complicated to setup and you can have your own off-site vault.
In the Windows world, I have used ViceVersa to setup the replication of a 14TB archive, but plenty of identical products are available for the mac.

by Olivier Boulot | 17 Nov 2009 20:11 | Paris, France | | Report spam→
I have a two-drive FW800 raid that I keep my recent/current work on. When I want to archive things off, I use two drives that get stored off-site. It’s not the most secure solution, but it’s a good balance between cost and security.

by Brian C Frank | 18 Nov 2009 04:11 | Des Moines, Iowa, United States | | Report spam→
Here’s what I do – I’ve been working this out over a period of a few years and think I have a pretty good system – balanced between speed and redundancy. You can also look at the DAM book which is a pretty good overview of digital asset management by Peter Krogh.

At one point, I had a RAID 1 set up – two drives in an external enclosure with a piece of hardware that made them appear as one drive. Everything that was written to the RAID got written to each drive. But I found that was too slow. What I do now is:

I have one main working drive – 500G internal SATA. I run Retrospect 8 which mirrors the drive every hour to a second drive which lives in an external enclosure. Working drives are named after letters of the hebrew alphabet. I’m up to Gimmel now. The mirrored drive is called Gimmel_mirror, which is a perfect clone of Gimmel. When I erase a file from Gimmel, it’s erased from the mirror, when I write a file, it’s copied to the mirror.

I store exernal drives in a tower with a hardware controller from fwdepot.com, which appears to be out of business. You can buy an enclosure from anywhere – I like newegg.com – which usually has a power source in it. You’ll also need some hardware to connect the enclosure to your computer. These are usually called bridgeboards, and some enclosures have them built-in. Newegg.com has some, but I can’t get to the site right now. I think that SATA drives with fw800 is fast enough. You can look into an eSATA interface which is faster. Since I’m only doing my backups over that interface, I don’t need blazing speed. If I had working files on an external drive I might look into it.

I also have retrospect running a backup server which periodically (every few hours) makes an incremental backup of my startup drive, working image drive, and laptop. Incremental backups copy only changed files, but don’t erase from the backup. This goes to a larger, 1T drive called Backup 1. Every few weeks, I move Backup 1 offsite and replace with Backup 2. I also burn RAW files to DVD. The current ‘best practice’ in backup is to have I think three or four (I forget) copies of every file. I’m compulsive, so I have five (original, mirror, backup 1, backup 2, DVD). I don’t think DVDs are a must, and it’s definitely a pain. I just do it partially out of habit – I’ve got so many already done.

As the working drives fill up, I copy files from them onto large internal drives called Image Archive 1, Image Archive 2, etc. The Aleph, Beth, etc drives move off site. I don’t backup Image Archive drives, as they don’t change and I have copies offsite incase anything goes wrong with the copies I have on my archive drives. I’ve been using Lightroom to keep track of what files are on which drives and it works quite well. iView media pro is also good, though it’s been bought by Microsoft who may have ruined it.

by Jonathan Lipkin | 18 Nov 2009 13:11 | Brooklyn, United States | | Report spam→
Whatever system you use, try to avoid using the same power supply to main disc and back up. I learnt the hard way when my main disc blew. I was so scared, my first act was to turn the back up disc on to check that that was OK. Of course, without thinking I used the same transformer, and it blew the back up disc too. Of course, it had been the power supply that had blown the first disc too, but that didn’t occur to me in time.

The main disc couldn’t be saved by data recovery people. Luckily they succeeded to rescue data from the back up.

Likewise, one lightning strike, or other spike in the current, could take out several discs at the same time. I always disconnect back up discs from the system when not in use.

With regards to keeping several back ups of files, one thing that helps is if you keep your RAW files and TIFFs/JPEGs on separate discs. Then also back up the RAW and TIFF/JPEG’s onto separate discs. That way you have 4 copies of each image on 4 different discs for starters, even if it’s only a RAW file. I make an extra back up of the TIFFs/JPEGs disc off-site, so that’s five copies. Maybe not a perfect system, but simple.

I am a little distrustful of any system that allows me to back up remotely, because I would think that would make the disc vulnerable to hacker/virus attack? And if it’s remote, then that presumably means that it’s permanently plugged in to the power supply, which must make it vulnerable to power spikes, or just wearing out through being on all the time? Unless it’s an online backup with a big corporation that itself has adequate back ups.

by Simon Crofts | 18 Nov 2009 14:11 | Edinburgh, Scotland | | Report spam→
Hey Susana!

Being totally paranoid I always try to have photos in at least two places so here’s what I do. ASAP after the shoot I ingest the stuff on to my computer. The cards do not get re-formatted until the images have been backed up to an external hard drive. Every few days I make two (2) DVD copies of all the images, one copy kept at the office and the other at home. Only then do I erase the images from my computer’s hard drive. So for every image (minus fuzzies and blinkers) I have one copy on an external hard drive for quick access plus another two copies on DVD.

My external drives of choice (for five years or so) are the Mercury Elite Firewire drives from Other World Computing (www.macsales.com). Excellent components and chip sets at decent prices.

Of course all of this will be moot as the DVDs will separate in 30 years making them unreadable and the information on everyone’s hard drives will disappear in The Great Hard Drive Crash of 2046 to be caused by a massive electro-magnetic pulse from an exploding satellite…

by James Colburn | 18 Nov 2009 16:11 | McAllen, Texas, United States | | Report spam→
I use a Drobo spread over 3 drives, which will allow any one of them to fail without an issue.

I also use 3 HDD’s in my Mac Pro tower, with a complete Library backup (using ‘Vaults’ in Aperture) on each one. Every month I make a copy of the library onto a portable HDD that I keep in a fireproof safe in the barn.

I can’t use online storage because the internet upload speeds here in New Zealand are so inadequate that it would take about 2 weeks running 24/7 to upload the library!

I have a UPS on the Mac and the Drobo.

by Marcus Adams | 18 Nov 2009 22:11 | Wellington, New Zealand | | Report spam→
Don’t confuse on site data backup with an archive. An archive needs to be off site and mirrored with your on site data. Optical disks will either chemically or physically break down and when they do there is no way to recover the data. Damaged data on magnetic hard drives can be recovered except in the most extreme conditions. Given the price of hard drives these days you are silly to bother with DVD’s as long term storage.

Something like a Drobo is great for accessing your data in the office but if your office burns down your data is lost. I use a LaCie NAS for on site data backup but my archive is on mirrored hard drives that are in flight cases. One set of drives is stored off site. When you consider how long your upload/download times are with net based storage it’s still best to just use a removable hard drive bay and pop a drive in for access to your archive.

by Jonathan Castner | 20 Nov 2009 16:11 | Denver, United States | | Report spam→
Jonathan -

I just did some looking around about DVDs, which I use in addition to multiple hard drive backups. Here is what Peter Krogh, author of The Digital Asset Management book says (summary: keep using both DVD and HD):

Andy,
I personally treat the derivatives just as I do originals – they get put in buckets, and backed up to an off-site drive as well as DVD.

As long as you understand the risks associated with omitting write-once backup, go ahead and do what suits you.

I would suggest, however, that anyone who is going to create a backup system without a write-once component should definitely implement some kind of absolute confirmation regimen. What I mean by this is that you should use something like ImageVerifier on the images (the hash-checking part) so that you can know with certainty that the images have not changed in any way, and that the hardware and filesystems are fully intact.

I’d like to see a day when DVD backup is totally superfluous. I think we’re missing some critical information at the moment. My data validation tests have shown lots of invisible corruption on hard drive. Until we get tools that can do easy and comprehensive data validation of all files, it won’t be clear if this is just my problem, or if it’s widespread. There are some new data validation tools in the new DNG spec that will help throw light on this issue.

I had a conversation with Tom Hogarty from Adobe this week where we discussed this issue. Although he does not advocate DVD as a part of a backup system, he did note that he is seeing an increasing number of corrupted images among users. (These show up with horizontal color blotches in the image area, and don’t trigger a error from existing data validation tools.)

Bottom line: Yes, I do suggest write-once media as a part of your archiving system for anything you really want to keep – at least for now.
Peter

by Jonathan Lipkin | 20 Nov 2009 17:11 | Brooklyn, United States | | Report spam→
Regarding Drobo, I hear that it’s a proprietary system, that the drives can’t be read in enclosures from other manufacturers. Is that true? I’m looking for a new archive system, any further recommendations?

by Timothy Fadek | 23 Feb 2010 11:02 | New York, United States | | Report spam→
Timothy, it’s true – my understanding is that it’s also true of all RAID systems, where if you loose 2 disks in an array at the same time, the entire array is dead.

One solution to this is building/buying an UnRaid system – www.lime-technology.com . A little bit more involved, but reasonably cheap given the amount of storage space you get, it’s essentially the same price as buying individual enclosures, with the added bonus of having a parity drive (so if one of the drives in your array fails, you don’t loose any data), and using ReiserFS on all the other drives (so in the extremely unlikely event that TWO drives fail at the same time, you only loose the data on those two drives, and can salvage all the others). Another good thing about limetech is that you’re supporting a small business – the software is coded by one guy, who is the same guy who answers emails if you have questions. One caveat is that the default build is rather noisy, so if you’re going to have it running in an office where you’re working, get the version with fan modulators. And if it’s still too noisy, consider getting more silent fans (which involves taking the box apart and replacing them). Another weak spot is that Apple file sharing isn’t built in as of yet, so you either have to go through Apple’s Samba implementation or hack around a little, a little bit more of a PITA than having straight AFP, but not a major issue at all (the feature has been requested, i’d suspect it’ll happen at some point) – at this point, it mostly means you can’t use your UnRaid box for Time Machine.

by [former member] | 23 Feb 2010 12:02 (ed. Feb 23 2010) | | Report spam→
Also : for all of you who want to really nerd off, may I suggest reading the DAM book – http://www.thedambook.com/ ? 500 pages of such exciting topics as “DNG as a Parametric Image-Editing Solution” and “Archive Directory Structure: An Overview”, as well as full chapters on choosing hardware for image storage and data migration.

by [former member] | 23 Feb 2010 12:02 | | Report spam→
That DAM book is great…it’ll put you to sleep faster than an Ambien!

Seriously, it is great, with a lot of info. But it’s all a crap shoot, where you are playing the odds of one bad thing happening against the odds of another happening.

So the best backup system is the one you actually use. If you can’t/don’t activate it easily and readily, no matter how sophisticated or redundant your hardware software system may be, you reduce your protection to zero! That’s why I start with the Drobo.

by [former member] | 23 Feb 2010 13:02 | Washington, DC, United States | | Report spam→
“Timothy, it’s true – my understanding is that it’s also true of all RAID systems, where if you loose 2 disks in an array at the same time, the entire array is dead.”

RAID-6 can lose two drives and the array can still function & be re-built. I’m not suggesting that’s necessarily the way to go, just saying…

One of the problems with most systems is bit-rot, and unfortunately this affects almost all backup systems:
http://en.wikipedia.org/wiki/Bit_rot

With larger and larger hard drives, the problem increases.

The primary risk is still “user error”, i.e. messing up your archive by accident. I rate that as a much higher risk than multiple drive failures. All of which is to say RAID alone is not enough, you need multiple backups.

The best archival system in my opinion is a ZFS-formatted array, because it prevents against bit-rot and also has a wealth of other cool functions such as automatic snapshotting etc:
http://en.wikipedia.org/wiki/Zfs

The problem is it isn’t that user-friendly, and to use it you need to run an operating system such as OpenSolaris, Nexenta, *BSD, or a few others. It was scheduled to be the next-gen filesystem of Mac OSX for a while, but Apple canned it.

That said, it’s not impossible and there’s a few guides on the net on how to build a ZFS server, which can actually be done pretty inexpensively, albeit you need to know what you are doing

PS – Here’s two articles I wrote a LONG time ago (so some info may be outdated), which may be of use:
http://www.snappertalk.com/index.php/20080604/archiving-photos/
http://www.snappertalk.com/index.php/20060309/raid-photographers/

by Ben Curtis | 23 Feb 2010 14:02 (ed. Feb 23 2010) | Cairo, Egypt | | Report spam→
Ben, my understanding was that the ZFS arrays were a real pain when you wanted to add drives (like you couldn’t directly expand your array, a parity + a data drive), but might have misunderstood (and yeah, i wasn’t thinking “exotic” raid setups – just the more usual 4/5/5+1’s that you’d usually find)

by [former member] | 23 Feb 2010 15:02 | | Report spam→
Good deal going on right now for all your back up needs:

http://www.newegg.com/Product/ComboDealDetails.aspx?ItemList=Combo.342762&nm_mc=AFC-TechBargains&nm_mc=AFC-TechBargains&cm_mmc=AFC-TechBargains-NA-_-NA-NA

by Brian C Frank | 23 Feb 2010 15:02 | Des Moines, Iowa, United States | | Report spam→
Matthias – I think you can add extra disks, just not as part of the first group/vdev, you have to create a new group, then can add that to the zpool. What the implications of that are, I’m not sure, but there’s some useful info here (this site by the way has lots of good ZFS info):

http://breden.org.uk/2008/09/01/home-fileserver-raidz-expansion/

I’ve looked at building a ZFS file server for a while, and played around with ZFS on OSX a bit, but in the end came to the conclusion that for me, the chance of data loss due to me completely messing up the ZFS system myself, vs other risks, didn’t make it worth it. If I were braver and/or smarter, I still think ZFS seems superb.

An easy way to implement it is using FreeNAS – http://freenas.org/freenas or other such OSes like Nexenta or OpenSolaris. You can even try them out virtually using VMWare. That way you can determine if you are really up to it. I wasn’t :-)

by Ben Curtis | 23 Feb 2010 15:02 | Cairo, Egypt | | Report spam→
“….and the information on everyone’s hard drives will disappear in The Great Hard Drive Crash of 2046 to be caused by a massive electro-magnetic pulse from an exploding satellite…”

Just keep loading Tri-X in to that old Leica and all will be fine.
;-)

I’m looking forward to the day when solid state memory becomes dirt cheap…

Also read an article about some new archival disks under development that are made from glass and carbon. Apparently they are being developed in cooperation with places like NASA and the Smithsonian for whom a long term storage media is of essence.

by Harry Lime | 23 Feb 2010 16:02 | London, United Kingdom | | Report spam→
how do you all deal with file data integrity?

I went through my entire library at the start of the year and found one corrupted image (the original is intact, and its a good thing I keep several backups). I would have never of noticed it unless I looked at every picture. it spooked me because I have several backups but I do not have anything in place to make sure the data stays intact.

It kind of spooked me so much that I now have an MD5 checksum file for each directory. tedious as hades, I hope there is a better way.

anyone know of a way to run a checksum on an entire directory tree on a MAC?

by Patrick O'Neill | 24 Feb 2010 05:02 (ed. Feb 24 2010) | weatherford, TX, United States | | Report spam→
I saw a couple mentions of Drobo. I just got one and love it. Also, just as a general rule, I never buy Maxtor or Western Digital. In my opinion, Maxtor is the worst there is. I’ve also had a WD drive fail though. Seagate Barracuda’s are a pretty sure bet. I probably have 10 of them and never a problem. Hitachi is pretty good as well from what I hear.

by Jon Vidar | 24 Feb 2010 05:02 | Los Angeles, United States | | Report spam→
I second avoiding Western Digital. I just returned from a month shooting and I was using two Western Digital Passport drives as on the road back up. I had constant problems with the drive ‘dropping’ the connection; not being recognised by my Mac etc.. Wasted hours of my time when it dropped a download / back up half way through. Back at my office, it took hours to copy the information across because of the same problems I had on the road, this time to a brand new Mac, so I know it wasn’t just my laptop I had with me on the road.

I can second using Seagate Barracuda drives. I have had a great run with them.

by Thomas Pickard | 24 Feb 2010 08:02 | Bangkok, Thailand | | Report spam→
guys, you’re working on anecdotal evidence here. There is no single “bad” hard drive brand, but there have been bad models in any given brand – the best bet is to mix n’ match, if you have a drobo, put a wd in one slot, a seagate in another, etc, etc, etc.

Thomas, your WD connection problems sound like a known issue with USB, and power over USB, with macs – the rear USB port on a macbook is shared with the internal stuff (keyboard ;)), and this can cause problems. Furthermore, they aren’t all powered equally – i’ve got LaCie drives that cause problems as well, or causing problems with any given cable. Using a booster (two big flat thingems in, one small out) usually solves the problem. Also (and i’m not suggesting this is your case) : when you’re using a “home” mac, connect your hard drives to the computer, not to the keyboard’s hub…

by [former member] | 24 Feb 2010 08:02 | | Report spam→
Agree with Matthias, there is good and bad all round. The key thing for longevity in my opinion is whatever drive you use for photo archiving purposes, make sure that it is in a well-cooled enclosure (particularly if you live in a hot place) i.e. with good fans. And for desktop drives, make sure you have them connected to a good UPS to prevent against power fluctuations.

by Ben Curtis | 24 Feb 2010 08:02 | Cairo, Egypt | | Report spam→
Ben, the Google study stated that “Surprisingly, we found that temperature and activity levels were much less correlated with drive failures than previously reported”… Although in really warm environments (for those of you who live in Cairo ;)), a good fan can probably only help :P
(and yay to UPS’s…)

by [former member] | 24 Feb 2010 08:02 (ed. Feb 24 2010) | | Report spam→
“And for desktop drives, make sure you have them connected to a good UPS to prevent against power fluctuations.”

But don’t rely on the UPS too much. My sister recently had a power spike that fried both her computer hard drive and the UPS too. Not sure whether that’s supposed to happen, but it did.

Simon

by Simon Crofts | 24 Feb 2010 09:02 | Edinburgh, Scotland | | Report spam→
If you do have frequent power-outs or surges, it’s good to also have a microprocessor-controlled surge protector between the UPS and power socket. And I don’t mean one of those $10 dongles. These ones are excellent and are sold all around Africa for this purpose: http://www.sollatek.com/product-detail.asp?id=948

by Ben Curtis | 24 Feb 2010 09:02 | Cairo, Egypt | | Report spam→
Matthias, re: the heat issue and Google study. Am sure their findings are correct. I just had a bunch of drives die on me that I suspected was due to overheating. But then I did have 5 drives in a desktop meant for 3, in 40+ degrees heat, with no air-con :-)
I’ve since learned my lesson, and it was one of the reasons I chose the Stardom RAID enclosure (http://www.stardom.com.tw/sohoraid_sr2_feature.html), the cooling is extremely effective. The same device is sold under a bunch of brand names such as Sans Digital in the US.

by Ben Curtis | 24 Feb 2010 09:02 | Cairo, Egypt | | Report spam→
“guys, you’re working on anecdotal evidence here. There is no single “bad” hard drive brand, but there have been bad models in any given brand – the best bet is to mix n’ match, if you have a drobo, put a wd in one slot, a seagate in another, etc, etc, etc.”

Yes and no. I have friends who are systems administrators for companies that deal with massive amounts of data and in their experience certain brands are less reliable than others. WD almost always comes out at the bottom of the pile and Seagate and Fujitsu near the top.

by Harry Lime | 24 Feb 2010 12:02 | London, United Kingdom | | Report spam→
http://www.storagereview.com/ has user-contributed reviews of drives. When you sign up, you can report on the status of all of the HDs you own – how long they are in service, when they die, etc. You have to log in and the data is hard to read, and since it’s user-contributed hardly scientific.

Looking at their data, it seems that WD had a few unreliable drives (in the 50th and 40th reliability percentile) in the mid-2000s. Earlier drives did better and there’s not enough data on recent drives.

Seagate looks a bit better, but not by a ton. Samsung did average with early drives, no data on current drives. IBM Ok.

No clear winner, from what I can tell, by manufacturer – it varies from drive to drive. You can sign up and check for yourself.

The best policy, imho, is to assume that all of your drives are inherently unreliable and will die tomorrow, and to back up regularly.

by Jonathan Lipkin | 24 Feb 2010 12:02 | Brooklyn, United States | | Report spam→
Feli, your friends’ experience is irrelevant, in part because a brand had failures in the past doesn’t mean that they currently fail, and because a brand used to be reliable doesn’t mean there is no design flaw in their current lineup, hence the importance of mixing and matching. By the time you know that a certain model is a lemon, it’s too late, because the technology has evolved anyway. This irrelevance, in the setting we’re discussing, is compounded by the drobo giving you parity, or dealing with redundant setups – if a drive fails, you don’t really care, because that’s what the system was built not to care about. What you want to prevent is improbable situation of several disks failing at the same time – the question with any piece of hardware isn’t IF it will fail, it is WHEN it will fail. But hey, suit yourself, buy identical drives from a “reliable” manufacturer…

Also, i’d be curious to have someone who actually knows about data rot and filesystems chiming in. Is Fabien Penso still around ?

by [former member] | 24 Feb 2010 12:02 | | Report spam→
Well, Matthias that’s a little like saying that all brands are exactly alike and perform identical. Do you really feel that a Yugo is as reliable as a Honda?

Obviously all drives fail and are redesigned over the years, but the issue is more complex than is obvious to you.

My ‘friends’ work for a very, very large company that shall remain nameless, who’s storage needs are only rivaled by governments. They constantly subject drives from all manufacturers to their own internal testing, because they purchase them by the trailer truckload and reliability is of the upmost importance to them.

Drives are setup under lab conditions in arrays and hammered mercilessly. Once they fail they are dissected and examined.

Turns out that the bearings for the spindle of one company in particular tend to breakdown from heat build up on a regular basis. This has been a design flaw that has existed for several years and therefore they no longer purchase equipment from this manufacturer.

During this ongoing testing the drives are driven to the point that they break down. If you analyze the data from the past years you will determine that while all drives fail eventually, some drives:

- have inherent design flaws. Sometimes a manufacturer can not use the most reliable design, because it would violate a patent. Sometimes cost containment drives a particular design.

- Some drives are manufactured to lower tolerances or from lower quality materials to keep the price point down.

I’ve been professionally involved with digital image processing for nearly 20 years. In my line of work we store massive amounts of image data and over the years you do see patterns emerge that influence your buying decisions.

by Harry Lime | 24 Feb 2010 12:02 (ed. Feb 24 2010) | London, United Kingdom | | Report spam→
I suppose that the problem with that is that the testing data of that organisation will be confidential internal information, unless they decide to publicly release it.

So it means that one can’t say ‘I know someone who has seen this confidential information and it says that the drives of manufacturer X are less reliable than the others’ because that would be libellous. If called upon to prove the truth of the statement ie. to publish the confidential testing information, one couldn’t do it.

Whether or not WD could really be bothered to sue someone on this thread for libel is another matter, but as a matter of principle and as a journalist it’s not the best/most ethical approach to take to reporting ‘facts’, IMO.

by Simon Crofts | 24 Feb 2010 12:02 (ed. Feb 24 2010) | Edinburgh, Scotland | | Report spam→
I never indicated which brand appeared to have a spindle problem. Did I now?

A lot of this is also common knowledge among many systems administrators.

I also had no idea that I need to bring along my lawyer to this discussion.

by Harry Lime | 24 Feb 2010 13:02 | London, United Kingdom | | Report spam→
Feli,

since i don’t have access to privileged information such as knowledge that “WD almost always comes out at the bottom of the pile and Seagate and Fujitsu near the top.”, or that some brands who have low reliability have a spindle problem, i’ll trust my gut feeling and spread the odds… thanks for the WD advice, though :)

by [former member] | 24 Feb 2010 14:02 | | Report spam→
There are many valid points on both sides of the argument here. For one, all drives will eventually fail. No matter what. They have moving parts and eventually they will go bad. But to say that all brands are the same is just wrong as well. I like the Yugo vs honda comparison, and I’ll take it one further and say that if I had to put four cars in my garage knowing that they will all eventually fail, I’d rather have four ferraris than four datsuns. Some things are just better bets.

Also, as a company your brand or image that is built over time says a lot about your product, and -without any insider info other than my own personal experience - I think it says something that there is recurring commentary on this thread saying that WD is not as reliable as seagate.

And yes USB has inherent flaws, especially with large amounts of data transfer IMO. If your using a mac, firewire 800 is the way to go.

jv

by Jon Vidar | 24 Feb 2010 15:02 | Los Angeles, United States | | Report spam→
Matthias,

I think everything smart to be said has been mentioned here, no need to be redundant on advices. I’ve been working professionally into software development and sysadmin for more than a decade, what I have to say to the readers reading my message :

- Make sure you understand than redundant drives (RAID, Drobo, etc) may crash at the same time, or you may permanently delete files. They DO NOT prevent you from doing backups, they only save your time when a disk crashed, as they’re still working. Best setup is probably a RAID10 setup with 4 hard drives, as you can still be going even if 2 disks crashed over 4 drives, as long as it’s not 2 consecutive drives ( a bit of luck is required ). I have a macpro with 4 drives for storage, and an SSD drive for the system (it does book 3 or 4 times faster, same goes with running application). RAID10 is also a very fast setup for managing large files.

- I use a Netgear Readynas duo for my backup http://www.readynas.com/?cat=3 , this is RAID1 setup (I strongly advice the use of at least RAID1, in case a hard drive crashes). I’ve connected the netgear to be used with TimeMachine, so I don’t have to worry about it. It has a gigabit connection so no problem.

- I also use http://www.lacie.com/us/products/product.htm?pid=11140 Lacie RAID1 external harddrive over firewire 800 when I need it. It’s almost as fast as local drive and uses raid1 in cash one drive crash. Those are also backuped on the netgear timemachine backup (yes, firewire 800 is what you should use instead of USB2.0).

- My important work is pushed on remote servers, I’m lucky enough to have fiber connections at home so uploading large files is no problem.

I would advice not to use rsync or ftp commands, backups should be fully automated else you will stop doing them, and you’ll end up loosing data. Yes you can ‘crontab’ it but if you understand those commands, then you don’t care about the previous advices and you can use your own setup.

I would advice not to use ZFS or other filesystem, they’re not ready for general use for everyone. Yes I use vservers, Xen images, and LVM for my own use, but I would advice not to do so on your own if you don’t know what you’re doing.

My advice is keep it simple, you’ll be happy the day it crashes and takes you very little time to go back to work.

by Fabien Penso | 24 Feb 2010 20:02 | san francisco, United States | | Report spam→
One more thing, if you know anyone providing the delivery of real negative film from digital files (or slides), I’m very much interested. I want a solution to keep pictures for 50 years or longer, the same way I’m scanning today films from my grand father with very high quality, as you can see at http://www.flickr.com/photos/penso/4338721940/

by Fabien Penso | 24 Feb 2010 20:02 | san francisco, United States | | Report spam→
Fablen, you could try to find a place that will record your images to something like a Rhino LVT. Extremely high resolution on film. You could do a CMY or R G B separation and so record color images on the black and white material.

I think these guys do this but I have no experience with them
http://dr5.com/

There are also many post production houses for movies workthat can shoot to a Arri laser recorder, although I think you may be limited to 4K resolution (I would need to double check the max resolution). These machines are mostly set up to record color, but obviously they can also do b/w. The estar base of the Kodak negative film is supposedly good for a minimum of 300 years. This is probably more complicated than shooting to a Rhino LVT, because you need to have your color pipeline set up for movie work. You should be able to shoot to a Rhino from Photoshop.

by Harry Lime | 24 Feb 2010 21:02 | London, United Kingdom | | Report spam→
I use Smugmug Archive (Online) now for backing up. I have researched all these sites and this is what I considered the best that will be here for a long time. Their site works great and very good customer service support. I have a Pro account that I use to back up all my family photos and the Archive for Raw,Tiff,Photoshop files. Also you can save HD video www.smugmug.com

by Dana Gonzales | 25 Feb 2010 05:02 | Los Angeles, United States | | Report spam→
I have all of my key images stored RAW on my website, hosted by Zenfolio. For the rest of the database, I keep a mess of external hard drives circulating, and am always certain to keep at least one copy of each image stored off site.
If I ever have to grab things during a fire I can go for camera gear/computer and not worry about the database…

by Jessica McGlothlin | 03 Mar 2010 21:03 | Bozeman, Montana, United States | | Report spam→

Get notified when someone replies to this thread:
Feed-icon-10x10 via RSS
Recommended
Icon_email via email
You can unsubscribe later.

More about sponsorship→

Participants

Jon Anderson, Photographer & Writer Jon Anderson
Photographer & Writer
Ocala Florida , United States
Keith Bedford, Photojournalist Keith Bedford
Photojournalist
Copenhagen , Denmark
Simon Roberts, Photographer Simon Roberts
Photographer
Brighton , United Kingdom
Jonathan Lipkin, Professor, Photographer Jonathan Lipkin
Professor, Photographer
Brooklyn , United States
Mike Peters, Photographer Mike Peters
Photographer
Nj , United States ( EWR )
Michael Limbert, Photographer, CD Michael Limbert
Photographer, CD
(Awake)
Detroit, Mi , United States ( DTW )
Bill Thomas, Photographer-Videographer Bill Thomas
Photographer-Videographer
Nyc , United States
Matt Wright-Steel, Matt Wright-Steel
Texas , United States
Boots Levinson, Photojournalist Boots Levinson
Photojournalist
New York , United States ( JFK )
Olivier Boulot, Photog Olivier Boulot
Photog
Paris , France ( CDG )
Brian C Frank, Photographer Brian C Frank
Photographer
Des Moines, Iowa , United States
Simon Crofts, Photographer Simon Crofts
Photographer
Edinburgh , Scotland
James Colburn, Photographer/Photo Editor James Colburn
Photographer/Photo Editor
Omaha, Nebraska , United States ( OMA )
Marcus Adams, Photographer & Guide Marcus Adams
Photographer & Guide
(Guide, Photographer & Fixer)
Singapore , Singapore
Jonathan Castner, Photojournalist Jonathan Castner
Photojournalist
Denver , United States
Timothy Fadek, Photographer Timothy Fadek
Photographer
Berlin , Germany
Ben Curtis, Photographer / Editor Ben Curtis
Photographer / Editor
[undisclosed location].
Harry Lime, Photographer Harry Lime
Photographer
[undisclosed location].
Patrick O'Neill, Photo Student (hahahaha) Patrick O'Neill
Photo Student (hahahaha)
Weatherford, Tx , United States
Jon Vidar, Photographer Jon Vidar
Photographer
Los Angeles, Ca , United States ( LAX )
Thomas Pickard, Photographer Thomas Pickard
Photographer
Rarotonga , Cook Islands
Fabien Penso, Software Architect Fabien Penso
Software Architect
(Nomade)
Paris , France
Dana Gonzales, Director of Photography Dana Gonzales
Director of Photography
Los Angeles , United States ( LAX )
Jessica McGlothlin, Photographer, Writer Jessica McGlothlin
Photographer, Writer
Manchester, Vermont , United States


Keywords

Top↑ | RSS/XML | Privacy Statement | Terms of Use | support@lightstalkers.org / ©2004-2014 November Eleven