Tips From the Mac Manager

It was the kind of nightmare every information systems manager dreads: the server kept crashing.

I spent most of last Friday morning wrestling our AppleShare IP server back into shape. It had crashed twice after I left Thursday afternoon - and then at least four more times Friday as I tried to fix things.

Fortunately we were prepared. Here's how you can be prepared when disaster strikes.

Back Up Everything

I can't say enough about backup. What software you use, what type of media you use, and what kind of backup rotation you use is far less important than backing up regularly. Backing up everything regularly. Backing up everything every day if possible.

We've been doing that at work since the early days of DAT tape - and on a slow LocalTalk network at that. Run backup. Every single day. Every single computer.

Over the years things got better. We got ethernet, a faster tape drive, an extra tape drive, and always a growing network. Today we have over 80 Macs on our network, everything from a IIsi through generations of Quadras and Power Macs right up to the Pismo PowerBook, Power Mac G4, iBook, and slot-loading iMac. We're backing up to AIT today, which stores about 33 GB of data per tape, and our monthly backup uses five tapes.

We're still backing up daily. Some Macs get backed up at night; others in the morning. PowerBooks are backed up as they connect to the network. Every day.

Back up everything. This is the first rule of being prepared.

Preventive Maintenance

When you're running a server - whether it's a file server, mail server, database server, or Web server - you want to run diagnostics regularly to make sure everything is in good shape. For us, that means every week.

Our tool of choice has always been Norton Utilities, going way back to the days when you could boot it off a floppy disk. Run Disk Doctor. Fix everything it finds. Then run Speed Disk.

That hasn't changed, although the software has improved over the years. Each drive on each server gets "Nortoned" every week. Anything that Norton Disk Doctor can fix is fixed. Whatever can't be fixed is trashed, along with a note to the file's author (when possible). Then we run Speed Disk not only to optimize the disk, but also because a freshly written file may have better data integrity than one that sits on the same part of the drive for months or years on end.

Norton is not the only tool; it is the one we stick with. We also have a copy of Tech Tool Pro, which I've never really used. In fact, it's so old that it may not even be practical today. (How old? Well, it came on floppy disks.) I also have my own copy of Alsoft Disk Warrior, which I keep in my briefcase. At home it's my first choice for troubleshooting, followed by Norton. But we don't own a copy at work, so I stick with Norton as much as possible.

Besides, Norton can detect things that Disk Warrior doesn't look for, such as damaged resource forks. (Likewise, Disk Warrior may find directory problems that Norton overlooks.)

To facilitate all this, I like to create a small (100-500 MB) "emergency" partition on each computer. This partition has a full copy of the Mac OS and Apple's utilities. I can boot from this even if I can't boot from Norton - as is the case with the newest iMacs, Power Macs, and PowerBooks at work.

Crash Recovery

Despite our best efforts, every now and again something goes terribly wrong. Deleted files can be recovered from backup. Messed up drives can often be recovered with Norton. But sometimes it goes beyond that.

When the drive crashes, when the server locks up, when the screen shows a bomb - time for some serious recovery work.

1. Boot from the emergency disk if possible; otherwise boot from your Mac OS, Norton, Disk Warrior, or Tech Tool CD.
2. Run Disk First Aid, which the system installer puts in the Utilities folder. This marvelous little tool comes free from Apple and can solve some very real problems. It may make running a heavy duty utility unnecessary. This is the first thing you want to run.
3. Sometimes Disk First Aid reports errors it can't fix. That's fine. That's why you've invested in powerful diagnosis and repair utilities. Even if Disk First Aid says it can fix everything, take the time to run Norton, Disk Warrior, or Tech Tool. You can generally skip the surface scan, which seems to take forever, although you should run that every now and then, too. (Note that each utility is unique. Norton excels at some types of repair, Disk Warrior at others. The more tools you have, the more likely you are to recover from your problem.) Let your software fix whatever it says it can fix.
4. If the software found problems and you told it to fix them, run diagnostics again. If it finds and fixes more problems, run it yet again. Run your utility program until it stops finding problems. I've sometimes run Norton three times to fix everything and then, for good measure, verify that it really is fixed.
5. Since you're down, this is a perfect opportunity to optimize your hard drive. Disk Warrior has a decent defragmenter, but Norton's Speed Disk remains my favorite - mostly because it does the job fast.

Repeat this process for each and every drive on your computer. Then reboot and hope for the best.

Most of the time you're up and running. Go buy yourself a Coke. You deserve it.

BOOM

And sometimes the bad stuff keeps happening, as it did last Friday. I never did figure out just where the problem was. I tried reinstalling the System, but even that didn't work.

In the end, it was backup that saved the day. After the fourth crash (my best guess - you really do get tired of counting after a while), I booted from the emergency partition, which had a copy of the Retrospect Client software used on the main partition.

Then I went to my backup server, selected the disk image from two days earlier - before we had any server problems - and used Retrospect to restore the server's boot drive.

Backup saved our server. I left Friday afternoon hoping it wouldn't crash again; it didn't. When all else failed, and that's a rare occurrence indeed, backup came through.

In over eight years, this was the first time I had to restore a server's boot drive from backup. I've had to do it two or three times on my own computer in the OS 8.5 to OS 8.6 era, but never on the server.

Considering how much work a clean system and AppleShare IP install would have been, being able to recover from backup probably saved another six to eight hours of work. Best of all, from my perspective, it meant that although I left work a bit late, it wasn't nearly as late as it could have been.

In the end, I woke up from the nightmare. Thanks to planning ahead, we were able to try several types of recovery before resorting to our backup. And when it came down to that, I'm glad we've been preparing for this for over eight years.