Other Sites

10 November 2006

Backing up data can be a complicated affair. That's probably why so few people do it. During the Keynote address at last summer's World Wide Developer Conference, Scott Forstall introduced a data backup product that will be part of Mac OS X 10.5 (Leopard) called Time Machine (at 31m 42s into the Keynote). Scott mentioned that in a survey of Mac users, only 26% of them back up in "any way, whatsoever." Of those 26%, the percentage who use some kind of automated backup software to produce a backup of all their files on a regular basis is just 4%.

This is a scary number. Especially these days when customers have tens or hundreds of gigabytes of music, videos, TV shows, much of which was paid for, and a lot of photos, home movies, personal and legal documents, checkbook data, shareware, and so on that is either memorabilia or has financial or legal significance. Hard disks are always getting better, but they do crash, and Murphy's Law says that your hard disk will crash at the worst possible time. One should be literate about these matters.

Let's approach this problem from a practical standpoint for the home user and see what we can come up with. We'll approach it logically and simply. If we can come up with a simple understanding, some key ideas, and some basic tools, backups won't be such a mystery, and we can breathe easier.

When we start thinking carefully about data backups, we have to break it up into three pieces.

1. The backup media
2. The backup tools
3. The policy or personal philosophy

Backup Media. Time and experience has shown over and over that the best way to back up a hard disk is with another hard disk. It's faster and more reliable. Cheaper methods such as magnetic tape are too slow writing, too slow restoring, and too prone to mechanical failure. Magnetic tape is a bad idea for the home user. Period. Trust me.

Backup Tools. This is a very difficult area, and a topic that is fraught with politics. The reason is that there are two classes of data backup software for Mac OS X. The first class has the commercial packages such as Tivoli, Legato, Veritas, EMC/Dantz, Atempo, BakBone, and so on. These packages are designed for a corporate environment with many client computers and very stressful requirements and policies. These packages, in general, are too expensive for the home user.

The second class has backup software designed for the individual or a family. For example, just to name a few, Apple Backup, ChronoSync, Deja Vu, SilverKeeper, SuperDuper, and Synchronize X. There are many, many more. Now I'm going to present an opinion here. The developers may disagree, and customers will have varying experiences, but I've been around this block, so I'm going to feel free to present my opinion. Here goes.

1. Home user backup software is modest in price and capability. Backup software written by an individual for home users has to be affordable. People tend to feel that anything more than $50 is too much to spend on a piece of software that doesn't come in a box, has no DVD, and no printed manual. Sometimes, it's basically toy-ware written to the best of the author's ability. But individuals don't make enough money on $25-$50 shareware to have a quality testing center nor do they always have the expertise to write truly iron-clad software. You're going to get what you pay for.

2. Backup software is very tricky to write. Prior to Mac OS X 10.4, many of the Unix tools used to copy files did not preserve what's called the Resource Fork of Mac-specific files. Or some of the other HFS+ extended attributes like Creator Type. Some of this meta-data about the files could be lost. In Tiger, Apple fixed all that, but a developer still has to have a very high skill level with the Mac OS X file system to write perfect backup software that accounts for huge files, funky file names, and oddball permissions. On top of that, the scheduled backup could occur while the user is active on the machine, a "live" file system (or Heaven forbid, doing an unplanned restore) so the software may have to deal with a dynamic file environment, not a fixed data set, when it starts the backup. Oops.

3. Backup software is incredibly hard to review. A reviewer can spend days or weeks with some of these low end packages and not discover the one fatal flaw that might destroy or corrupt not only the backup data but some source data. Basically, reviews focus on the user interface and whether the product seems to do what it claims to do in the simplest testing techniques. As a result, if you go to Version Tracker, do a search on "backup" you'll see dozens of software titles with hundreds of comments. Reading the comments, you'll see many, many satisfied users who've used one of these products for years. You'll find a few people who are clueless, did something dumb, and lost some data because they didn't understand their Mac or the software. And you'll almost always find someone who, with the best of practices, lost some data because the backup software had a flaw.

4. Backup software has to be perfect. Unlike shareware utilities that add some nice touch or perform a simple task, there is no substitute for perfect backups. The commercial class of products get extensive testing both in-house and by corporate customers. Catastrophic errors are rare, and methods are in place to recover from human error. But low-end backup software, written often by only one person, can have human errors that go undetected. At this point, a pissing contest starts between the author and the customer. No one is very happy. Finally, this is one instance where a public release of beta software is a very bad idea. Don't use it.

I strongly suspect that Apple itself has not been happy with the state-of-the-art, hence their announcement of Time Machine in Leopard.

Because of what I described above, the home user has to have some good strategies to account for the possibility that their selected backup package may not always work as they hoped it would.

Backup Philosophy. Because our machines and our software are not perfect, we have to think instead about our own techniques that can take those imperfections into account. Also, we have to be aware of the fact that there is a difference between a backup that is simply a copy of our working files (redundancy/synchronization) and a true backup that can go back in time. For example, if you delete a file this afternoon (and empty the trash) and your backup software synchronizes to an external disk tonight, tomorrow that file will be gone on both your home directory and the backup. (Barring exotic tools to recover "deleted" files from the hard disk.)

This is why commercial software is so exotic, capable, complex, and expensive. Data backups are formed into sets, and a named set can go back weeks in a data center that contains hundreds of terabytes. The data may be stored on special disk drives that use RAID technology. In this case, even if a single hard disk in the data center fails, data is never lost, and a user can go back to a data set saved many days or weeks ago to retrieve a file -- depending on the data center's policies.

For the home user, this is complex and expensive.

For example, let's say you have an 80 GB drive in your MacBook. Your home directory contains 40 GB. You bought an external, 150 GB FireWire drive for backup. If you do a separate, full backup of your entire home directory each night, (not over-writing), you'll be able to do this for just three days before you run out of room, and it's a terrible waste of space for all that data that didn't change.

The more practical alternative for home users is to synchronize your home directory to an external drive daily. That means that the backup on the external drive is an on-going duplicate of your home directory and only files that have changed are written out. Then, every once in awhile, you could do a complete backup on some kind of archival media, say DVDs. The problem here is two-fold: 1) the routine of that backup to DVD, requiring one to swap multiple discs, is tedious. DVDs only hold 4.7 GB each. And if you have deleted a file created since that last DVD archive, it's generally gone as soon as you synchronize. And 2) if your data starts getting corrupted, the duplicate copy will also be corrupted.

Another strategy, used by backup software, is to create a full backup, then create incremental backups each day. These preserve all data back to the time of the full backup and also reflect day-to-day changes in the incremental backups. The problem here is that the manipulation of the daily incrementals, at the time of a restore, has to be handled efficiently, transparently and reliably by the author. It's a challenge.

Finally, there is the issue of whether you're going to back up just the home directories on your Mac (yours, your spouse's, your kids) or whether you're going to make an exact, bit-for-bit copy of the hard disk inside your Mac. This is called a clone, and there is software that can do that. (SuperDuper, to name one.) This is useful in case your operating system itself gets corrupted (or mangled by a software installer). But it requires that the external drive be erased and completely dedicated to a bit-copy clone.

A Sample Course of Action

As you can see, talking about backups is enough to give one a serious headache. That's probably why only 4% of Mac users even bother. But if you're serious about protecting your data, it's possible to come up with some simple strategies that can help. What I'll describe next isn't the only way, but it's one way, and sometimes it's good to have a place to start. In time, you will improve or alter your strategy. Even the arrival of Apple's Time Machine doesn't mean you can stop invoking good practices.

1. Don't install a lot of experimental shareware. If you're fortunate enough to have an old Mac, say a CRT-based iMac or an old PowerMac G4, put it to work as a testbed to experiment with downloads. You want to keep your working Mac OS X as simple and as close to Apple's out-of-the box specs as possible to avoid conflicts or complicating your OS. That way, you won't have to clone your OS.

2. Buy an external FireWire or USB 2 drive about twice the size of your internal hard disk. Good products, which I can recommend, are the LaCie drives or the Other World Computing Mercury drives. Read reviews. You'll pay about 50-70 cents per gigabyte.

3. Read the user feedback at Version Tracker on various backup packages. Look for one that's been around for awhile, has a high version number, has been downloaded by lots of people, has good user comments about the responsiveness and customer service of the company/author, and most important, is written by one of the more well-known, larger and respected Macintosh developers. You'll need to go to their Website and size them up. Pick one and try it out even if one user or two reported a problem. Products that have been recommend to me by colleagues, submitted here without testing or endorsement, are Deja Vu and ChronoSync. Apple's Backup 3, available to Dot Mac users, is barely adequate but has the advantage of Apple behind it. Use with care and caution. A new version is rumored to be in the works.

4. Install Mac OS X on the external drive. Don't try to make it like your working OS, just keep it simple. Every once in awhile boot from that drive and use the Disk Utility to repair permissions and perform a "Repair Disk" on your main drive. This will keep your file system healthy to avoid corruption. You may want to go farther and invest in something like TechTool, just for example, from Micromat which will assist in keeping your system healthy and/or recovering from problems.

5. Once a week, in addition to whatever your purchased backup software does, do a Finder copy of your home directory (replacing the old one) to that external drive. That's why I recommended twice the size of your internal drive. That is, just drag your home directory to a folder on the external drive. Using FireWire 800, this will take about one minute per gigabyte. If your backup software fails or confuses you, you'll have a second, ahem, backup.

6. If you can afford it, buy a second external hard disk. Use whatever tool you prefer to make a backup, then put the hard disk in a safety deposit box. This seems like a bother, but if you have a burglary or a fire, all that data stored away offsite will be a life saver. Safety deposit boxes are dirt cheap. Plan to update it once a month. This is easier and far less tedious than a stack full of DVDs in your basement. Things may get better when we have 50 GB Blu-Ray discs!

Finally, lots of software looks good when it comes to backing up your data, but the real test comes when you have to restore some data. Practice with that using duplicated data and whatever software you buy. You may find that you're not happy with the restoration process and want to try a different product. And practicing a restore will pay dividends on that dreaded day when you're in a panic and really flustered.

Let's hope that Time Machine from Apple addresses most or all of these issues. The demo at WWDC appeared to show that you'll be able to go back in time (since Leopard install) to recover a single deleted file. It'll also back up your OS so that you can restore to a previous state in the event the OS gets somehow mangled. We all have great hopes for this product. Just remember, programmers are human. Computers have limits. Don't just sit back and believe that someone else, even if it's Apple, will take care of all your data. Be suspicious, be practical, and, ultimately, be informed and literate.

This is Warp Core column #83. The Warp Core archives are here:

Year 2000

Year 1999

* Your humble author also writes a column for TMO.

John Martellaro is a senior scientist and author. A former U.S. Air Force officer, he has worked for NASA, White Sands Missile Range, Lockheed Martin Astronautics, the Oak Ridge National Laboratory and Apple Computer. During his five years at Apple, he worked as a Senior Marketing Manager for science and technology, Federal Account Executive, and High Performance Computing Manager. His interests include alpine skiing, SciFi, astronomy, and Perl. John lives in Colorado. He can be contacted via his Website or the Applelinks Contact link.