1998 – I’m practically ancient for this industry. I remember lusting after the TRS-80 in Radio Shack flyers back in 1977. I think it was in 1979 that I first put fingers to keyboard and used a personal computer (an Apple II+).

In all the years since, computer magazines have been my faithful companions. In the early days when I used Apple and Commodore gear, I read Byte, Creative Computing, and Compute! When I became a DOS geek, PC Magazine, Byte, and PC Tech Journal were my favorites.

Today I lean toward the Internet for the best and latest information, but I still read Macworld, InfoWorld, PC Magazine, and others regularly. (And I am singularly upset with the folks who bought Byte magazine, put it on “hiatus”, and dropped almost every good writer they had. When it comes back – and it better, I’m a paid subscriber – it won’t be the small systems cross-platform journal I’ve known and loved for almost 20 years.)

Computer magazines build their reputations on product test reports, which almost invariably include sophisticated benchmarks that look at the computer at both the component level and the application level.

MacBench, BYTEmark, and the ZD suite (CPUmark, Winstone, WinMark) are designed to fairly compare computers from different manufacturers with different CPUs and with different subsystems. Best of all, BYTEmark is was designed to do that across platforms, so it can evaluate a Pentium II vs. an Alpha vs. a PowerPC.

That may have been the reason for Byte’s demise. BYTEmark showed the upstart PowerPC 750 (a.k.a. G3) outperformed the industry darling, the Intel Pentium II. It also gave the DEC Alpha a real run for the money. Best of all, the PowerPC did this with a lower clock speed, lower power consumption, and lower chip cost.

No wonder a Win-centric publisher bought out Byte.

I am not here to condemn these reviews and benchmarks, but to take TechWeb and PC Magazine to task for unfair comparisons between the Power Mac G3 and various Pentium PCs.

John Gartner of TechWeb has been raked over the coals by the Macintosh community not only because he reported that testing by NSTL (National Software Testing Laboratory) showed PCs as faster than Macs (see his test report*), but because a second page* of the NSTL report clearly states,

“The Mac tests were run in the default graphics mode. Enabling graphics acceleration mode would make full use of high-performance adapters such as the ATI Rage II+….” (emphasis added)

Of course, a lot of readers never get to the second page to discover that this was not a level playing field. NSTL was comparing graphics accelerated Windows machines with Power Macs that did not have acceleration enabled. That’s why they carefully pointed out (near the bottom of the second page) that enabling graphics acceleration would have changed everything.

But to add injury to insult, Mr. Gartner followed up with Get Real,* a column in which he said it is fair to compare computers as NSTL did.

Sorry, John, but in all my years of reading computer magazines, I’ve never heard of anyone knowingly testing a computer with an important feature like graphics acceleration disabled, especially if it shipped with the computer.

So what if it ships with acceleration disabled? Honest testers do their best to put every computer on a level play field – even ones from Apple Computer.

But That’s Not All

Now it seems that PC Magazine may have made the same mistake. In a series of tests released on August 28 (First Looks: iMac, iMac vs. Pentium II: Test Results, iMac vs. Pentium II: What the Numbers Mean, iMac vs. Pentium II: How We Tested), it looks like ZD Labs also neglected to enable graphics acceleration, instead running their tests in “default” mode.

This is the same magazine that used to test dozens of modems or video cards side-by-side, often tweaking settings to see how much performance they could wring out. I can’t believe that today they’d rather just take equipment out of the box and test it without a single change. I thought we were dealing with geeks here.

It’s a sorry day when testers at two leading PC labs can’t be bothered to turn on the Mac’s accelerated video and judge a Macintosh on it’s true merits. Acceleration comes with the computer – all they have to do is turn it on.

Then again, that might make their favorite computers look bad.

This article has generated an unprecedented amount of email. Most of it tells me that iMacs ship with graphics acceleration enabled; I had read otherwise. Another writer say the Power Mac G3 also ships with graphics acceleration enabled, so if NSTL tested it with graphics acceleration disabled, they had to do that intentionally. (Or they may have simply assumed it was disabled since they could find no way to enable it.) Several writers have also noted that the second page I quote from has been revised since I quoted from it, so some of the words I quote are no longer present.

On the one hand, I’m glad to learn that Apple ships computers with graphics acceleration on. On the other, I wonder why the Macs didn’t benchmark better. And on the gripping hand, there’s the issue of Microsoft Office using undocumented Windows code to optimize performance, something Microsoft can’t do on the Macintosh.

And it looks like PC Magazine may be as reliable as ever.

 * These articles are no longer available online.

Further Reading

• Pentium vs. PowerPC, MacKiDo
• Motorola and PowerPC (also covers 680×0 series), PC Magazine
• PowerPC vs. Pentium II: Escargot?, Mac Musings
• Great CPUs, Past and Present, John Bayko. See especially sections on 680×0, 80×86, PA-RISC, Sparc, Alpha, PowerPC, and Merced.
• RISC, CISC, and Your Mac, MacTips

Keywords: #benchmarks #bytemark

Short link: http://goo.gl/0JvriT

searchword: unfairbenchmarks