This Blog Has Moved

Wednesday 30 December 2009

Storage For Photographers

Your Hard Drive Will Fail;

It’s Just A Question Of When


A yawn is the typical reaction from my colleagues when the topic of digital storage, archiving and backing up is talked about. That is, until one of them loses their work, because they weren’t paying attention to the need for a reliable back-up strategy. The yawns are then replaced by much shouting of obscenities and tears.


In this day and age of digital photography, the obsession of ever increasing megapixels, the need to shoot video and record audio, one thing is for certain; we need somewhere to store it all. The lazy and stupid will leave it all on their laptop until the disk’s full and then drag it off in a hurry, onto an external drive, usually losing stuff. Sometimes they’ll even be stuck on a job with no choice but to delete older stuff just so they can download the latest job and process it. I’ve even seen colleagues work straight of a CF card when in a hurry send a low res jpeg, forget to download the card and then format it. Disaster.


Sonnet's Fusion D800 RAID box with 8 bays


Although it’s a pain, at the end of a long day, it still pays to have a system. My personal way is to never format my CF cards until I have the same work on at least two hard drives. If it’s just on my MacBook Pro, then the CFs remain in my belt pouch until this has been backed up at home. I have a ThinkTank Photo Pixel Pocket Rocket which is always full of CF cards. I make it a point to have more than enough cards with me. I always have one small external hard drive with me too, so I might back up onto this. When on foreign assignments, I carry two external drives and back up onto these (these portable drives are never stored with my laptop. I’ll leave one in my luggage and have the other with me if leaving the laptop in a hotel or car). Once home, after a typical day in town or a trip, I’ll back up the assignment onto my Mac Pro’s Aperture Library.


I’m a big fan of Aperture and have it on all my machines. Apart from RAW processing, Aperture also acts as a fully searchable image database. At the moment I have more than 385,000 images in my Aperture library, and this number is constantly growing. RAW images are stored on an internal drive, in separate Project folders which are derived from the assignment. These are titled using a date and a name. An example would be “2009-09-28 Gordon Brown”. These Projects (with consolidated masters) are then backed up onto a Sonnet Technologies external SATA RAID box (more on which later). The same Project is also backed up onto an external hard drive which is kept off-site. It’s important to have off-site storage to secure the safety of data in the event of fire or theft. Lastly, important images, documents, video and audio are backed up to “The Cloud” (more on which later).


What’s New?


The old ways to back up were CDs and then DVDs. Blu-Ray doesn’t seem to have caught on, even though a double sided disc offers 50Gb of storage. The problem with optical storage, apart from the slow speed of writing, is that they don’t seem to have the longevity needed. I’ve had top brand CDs, kept upright in their cases and stored in cool, dark and dry conditions become unreadable after a few years. There is a 500Gb optical disc that’s being talked about, but again, how long is any media stored on that going to last? I for one certainly hope that it has proper archival stability, as it really would be useful to have.

My thoughts are that using several hard drives which are swapped out every three to four years, is the best method available. This provides speed and security. It also offers value for money, as drive prices continue to fall, with speed and capacities rising.


Seagate's FreeAgent FW800 hard drive units


If you’re wondering why I seem to have an obsession with several hard drives, and a golden rule that everything is kept on three drives at least, it’s because hard drives fail. It’s an absolute given. Anyone in IT will tell you that hard drives fail, it’s just a question of when. I’ve personally had a major brand name drive fail after 3 months of use whilst sat on a desk, but had others which have worked solidly for many years. It’s always a gamble, and you should be well prepared.


At the moment, I have 12 external drives in my office and all the associated cabling and power bricks cluttering up the floor area. After some research, I recently decided to go for a Sonnet D800 Fusion RAID ( www.sonnettech.com ). This is an external SATA RAID box with eight drive bays. This means that it will hold eight hard drives, connect to my MacPro (its multi platform) using eSATA cables (connecting to its own controller card which installs inside your computer) which means that it’s blisteringly fast, and has only three cables; two eSATA cables and one power cable. To say that it’s a neat and tidy solution would be an understatement. I’ve recently finished moving my archive over to this system and will do away with most of the external drives. This also makes the office more quite and power-efficient.

I have the drives in the D800 set up in pairs of RAID 1 (also known as Mirror RAID). This basically means that everything that is saved on the drive (and you will only see one of the drives on your computer), is automatically copied onto its partnered drive. This is transparent, automatic and at the same speed. This protects the data from a hardware failure. I then manually back up data from this unit onto an external drive via FW800. This ensures that if by accident I erase an image, or an image gets corrupted, I can get it back from this back-up. These back-up drives are then stored off-site.


In use the system is amazing. Accessing images or video on the unit is blisteringly fast as it’s working over SATA, which is much quicker than even FW800. I’ve had the unit running for weeks without powering down and it’s been absolutely stable on the Mac Pro running Leopard.


The other new term you may have heard is “The Cloud”. This is virtual storage that’s kept on servers, somewhere in the internet, sometimes even in different countries. The Cloud’s not such a new thing, but with faster broadband, it’s now becoming more usable. Apple has had “.Mac” (now called “Mobile Me” www.me.com ) for years. Part of this service has been the iDisk which has been a virtual drive, available for use by Mac and PC users. I’ve been using this system for many years, and although a little slow, it’s been solid and stable.

The other Cloud system I’ve been using for around a year is Amazon’s S3 via Jungle Disk ( www.jungledisk.com ). This mounts a virtual drive onto your desktop (it’s multi platform) and allows you to use it like any other drive. It’s a faster system than iDisk and allows you to pay for the storage you use. At the time of writing, this is $0.15 per Gb per month. Your data is then saved on Amazon’s servers either in the USA or in Europe at locations which are not disclosed.

I’ve been using the Cloud in two ways. When on assignment, if I’ve shot a particularly important image, I’ve been saving them immediately to my iDisk. This has been for back-up purposes. Also, if I’ve got documents to which I need access, as well as having copies with me, I’ve also got them on my iDisk. This also includes email and FTP account details, which means if my laptop gets stolen, I can still function by accessing this information from another machine.


Jungledisk's activity monitor shown running on a Mac Pro


As my working year continues, every few weeks, I upload the edited pictures from assignments to my Amazon S3 drive. At the end of the year, like most photographers, I look through that year’s work and select my best work for competitions. Once I have this edit sorted, and the images processed to perfection, I take the contents of this folder and also upload it to Amazon S3. This gives me an off-site back-up of the year’s de facto most important and best work.


Lastly, a word on automated back-ups. I use Apple’s Time Machine to back up everything apart from my work images. This includes emails, invoices, letters, music, family snaps and so on. It’s saved me twice so far after I accidentally deleted important information. You can get back-up software for any platform, and I urge you to also have this system in place. For me, I use a separate FW800 drive for this purpose.


One thing’s for sure; as prices tumble for memory cards and hard drives, there’s no excuse not to have a solid and dependable back-up strategy. A little time spent planning and executing this strategy will save much stress and tears; trust me, I’ve seen enough colleagues suffer.


This article was originally published in the BJP on October 07, 2009.

17 comments:

  1. Good post, always like hearing about different workflows/storage systems. I have my eye on the Drobo system. What are your thoughts on that? Mind you, I still think I need someone a bit more IT-minded than me to come in and consult for me on exactly what I need to do for my own level of data storage, which wouldn't be nearly as big as yours. I'm also a PC user though...

    ReplyDelete
  2. Thanks for the kind words.
    I'm not a fan of the Drobo system because its too complicated; their system is great theoretically as it allows you to mix and match drives and create one huge drive. However, if you read the forums, they are filled with horror stories of people loosing everything. For me, the only RAID system which makes sense (for storage) is RAID 01 and then a separate off-site backup.

    ReplyDelete
  3. Excellent article about the essential 'drudge' of backing up. At least hard drive storage is dirt cheap these days.

    ReplyDelete
  4. Many thanks. As you have so correctly said, the cost of drives is very cheap now; there really is no excuse for not having a good backup routine.

    ReplyDelete
  5. lovely post...but now I can't sleep

    ReplyDelete
  6. Thanks Edmond, since I read this on the BJP website I put more time & effort into my backups, thank goodness.

    Since reading this again last week I suffered a failure in my main image & catalogue drive.

    Thankfully I'd only lost a couple of hours of work thanks to your explanation & my efforts, and I was able to recover most of that with a great program called File Scavenger which could access the files from the failed disk.

    And now I'm wondering the best way to proceed: another, larger, portable drive or an investment I can't really afford in a RAID system!

    Funds vs. requirements ... the eternal conundrum!

    ReplyDelete
  7. Hi Paul,
    Many thanks for the kind words. I'm glad the article was useful. Sorry to hear of your problem but glad it wasn't catastrophic.
    You should at least have your work residing on two hard drives; if you let me know of the system you currently have, maybe I can then make some suggestions.

    ReplyDelete
  8. Hi Ed,
    Hope you're well and keeping warm. I am that idiot who never backed up and lost two hard drives. Very expensive! I can't stretch to RAID but my new year's resolution is to follow your lead and back up onto externals and again onto Time Machine. All the best and maybe we'll bump into each other again some time. Matt.

    ReplyDelete
  9. Hi Matt. Sorry to hear mate; that's brutal. Hard drives are so cheap now days that you should get at least two for now and stretch to RAID and a backup when funds allow. You have to think of it in terms of how those pictures you lost may make money in the future as images begin to get historical value. Also, do look into Cloud storage for your most important work. Good luck and hope to see you soon.

    ReplyDelete
  10. Hi Ed,
    thanks for your response.
    I currently use a Windows Vista laptop (desperately in need of software & hardware upgrade) and from that have portable and desktop HDDs for backups.
    I suppose what I'd like is a flexible basic system which will run with what I have now but also migrate to other systems (hopefully MBP) in the future as things grow.
    For now I'd like a RAID system with 2x500GB drives but one which I could fit 4x2TB into over time ... and finally, it needs to be idiot proof, or at least, simple for a less than totally IT-literate photographer to use. If it means fitting SATA cards for example, that might be beyond me for now, although if I have the flexibility to start off with USB and move to SATA connection over time, again so much the better!
    Thanks again,
    Paul

    ReplyDelete
  11. Hi Paul,
    You're welcome :-)
    Firstly, when you do migrate, I can't recommend the MacBook Pro highly enough. I changed over around four years ago and there hasn't been a day where I've looked back. Good news is that Mac OS X will be able to read (but not write to) your NTFS drives, so your archives will be fully accessible. Lastly, definitely have a good look at Aperture as an all in one imaging solution; raw processing, editing, captioning, adjustments and archiving / cataloguing.

    Ok, back to the present...
    Firstly, you need to use the fastest interface you have; if you have Firewire 800, that would be ideal and preferred to USB 2. Even if you laptop only has USB, it would be worthwhile getting a system with FW 800 and eSata for the future. Also, the only way to get eSata onto a laptop would be by using ExpressCards which just slot in - having said that, there are some exceptions; some eSata RAID boxes like the Sonnet Fusion D800 which I use, has a special SATA card which also does all the RAID building and monitoring, so is only suitable for a desktop computer which can take expansion cards, like a Mac Pro.
    If you'd like to get a four bay RAID box but only populate two bays to start with, it has to be a DIY job - it's very straight forward though :-)
    What you would need is to set it up as a pair of RAID 01 (Mirror) and when you add another two drive, you also set these up as a RAID 01 pair.
    Some of the enclosure that could fit the bill:
    http://www.scan.co.uk/Shop/Computer-Hardware/All/Hard-Drives-Ext/Direct-Attached-Storage-Enclosures/Enclosure-(35-525)-SATA

    I have one of these which so far I'm very pleased with:
    http://www.scan.co.uk/Products/ICY-DOCK-MB662UEAB-2S-White-Dual-Bay-SATA-to-USB-and-Firewire-400-800-External-Enclusure-Raid-PC-MA

    Simple switch on the back to choose RAID 1 and it has all the interfaces you need making it future proof.

    A word of warning when buying hard drives to fit inside these units. Firstly make sure you buy an identical pair; not only the same capacity and make, but the same model for RAID 01 to work perfectly. Secondly, check with the hardware manufacturer to see which drives ther have tested and recommend - VERY important!!

    Good luck :-)

    ReplyDelete
  12. Forgot to add; I'm a fan of WD MyBook Studio:

    http://www.scan.co.uk/Products/2TB-Western-Digital-Studio-Edition-II-USB-FW400800-ESAA

    You install the software and set it up as RAID 01 Mirror. Very simple and straight forward.

    After all this though, do remember that the ideal solution is also to have an off-site backup of your data as well.

    ReplyDelete
  13. Hi Paul..

    Like Antonio Olmos mentioned above, your article reminds us of something we feared all this time (haha).

    But it's always better to be prepared for a disaster like that.

    Nice article.

    ReplyDelete
  14. Thanks for the compliments; glad you liked teh article. However, my name is Edmond ;-)

    Also, please check out the new site for this blog:

    http://photothisandthat.co.uk/

    ReplyDelete
  15. v helpful article. I cant yet get to organize my few thousand of images.. which software to choose for sorting, organizing, captioning, exif info.. I have it all backed up properly on three separate HDs but just cant decide what software to use (!) I tried Lightroom, Bridge, all that, but didnt find The One. Help?

    ReplyDelete
  16. Aldi,
    I use Apple's Aperture for all my work. My library is over 400,000 images now and it's by far the best software I've used. Aperture 3 even supports video and audio!

    ReplyDelete