Understanding MacBech 4.0

Measuring subsystems, not components

MacBench 3.0 measures the performance of subsystems, not the performance of individual components. For example, the hard disk is part of the disk subsystem and does not affect the results for graphics tests. So, instead of talking about individual components, it makes more sense to talk about a Mac OS system's subsystems and what composes those subsystems.

Although MacBench's tests attempt to measure the performance of individual subsystems, all these tests must run on the processor. Thus, the processor subsystem necessarily affects results for every test. In our testing, we've also noted that the disk subsystem tends to be the bottleneck for most business systems.

As we see it, a Mac OS system has the following basic subsystems: processor (including the floating point unit), disk, graphics, and CD-ROM.

The processor subsystem
We consider the components of the processor subsystem to include:

The disk subsystem
The disk subsystem includes your Mac OS system's:

Whether you have enough room on the hard disk and whether the hard disk is fragmented are also important factors in your system's disk performance.

Factors about the disk subsystem that can affect a Mac OS system's performance include:

We recommend you defragment a Mac OS system's hard disk each time you run any MacBench disk test. You can use any software defragmenting utility to defragment your system's hard disk. A defragmenting utility gathers all the separate parts of a fragmented file and sequentially orders them.

The graphics subsystem
Your Mac OS system's graphics subsystem includes:

You should remember two important facts about your Mac OS system's graphics subsystem.

Factors about the graphics subsystem that can affect MacBench results include:

The CD-ROM subsystem
Your Mac OS system's CD-ROM subsystem includes:

Factors about the CD-ROM subsystem that can affect MacBench results include:

What the Tests Do

To better understand MacBench's results for a Mac OS system, it helps to know what the test that gave you those results did when you ran it. This chapter explains what MacBench's key tests do.

Processor tests
MacBench's processor tests measure a Mac OS system's processor subsystem performance by running instruction mixes that reflect the way applications use the processor subsystem.

About the Processor test
The workload for the Processor test is a carefully weighted combination of many different modules. Each module does one or more types of real work, such as:

The combined overall workload simulates the processor behavior of leading Mac OS applications.

When you run the Processor test, MacBench runs a large instruction mix that carefully emulates the processor activity of real-world applications. The Processor test focuses solely on a Mac OS system's processor subsystem. The test performs no I/O operations or other functions that might involve a system's graphics or disk subsystems.

About the Floating Point test

The Floating Point test measures how fast your Mac OS system's floating-point unit (FPU) handles floating point calculations. MacBench's Floating Point test, based on our observations and experience, reflects the kinds of tasks the Mac OS's FPU typically performs. This test:

Hardware and software FPUs

MacBench contains both a software and hardware version of the Floating Point Test. MacBench automatically uses the software version if your Mac OS system does not have a floating point unit and the hardware version if your Mac OS system has an FPU. (You can change these defaults in the Test Settings dialog box.)

If your system has a floating point unit and you run the built-in test, your system will have a much faster score on the Floating Point Test than if you ran the software version of the test.

How we created these tests

To ensure the accuracy of the Processor test's simulated workload, we profiled a variety of leading Mac OS applications. The application profiles contained a large set of statistics about how those applications interacted with the processor. Those statistics include such important facts as the types of instructions the applications execute, the frequency of each instruction type, and how often the processor was able to find the data it needed in its RAM cache.

The Processor test reflects the results of the profiling process for Mac OS applications. During the development process, we fine-tuned the test's behavior to correlate highly with the profiled application behavior.

Processor test scores

For both tests, MacBench records how many operations the test system performs for the duration of the test. Then, MacBench compares this result to how many operations the base machine performed within the same amount of time.

For both tests, the base machine - a Power Macintosh 6100/60 - reports a score of 10.0. With these scores, bigger numbers mean better performance.

Both the Processor and Floating Point tests produce relative scores: Their results are normalized, unitless numbers meaningful only when you compare them to other MacBench 3.0 processor and floating point results. They are good indicators of the relative power of your Mac OS system's processor subsystem.

Disk tests

MacBench's disk mix tests - the Disk Mix and Publishing Disk Mix - measure the overall performance of your Mac OS system's disk subsystem. MacBench's individual disk tests run specific File Manager commands and record how many kilobytes the test system reads or writes during the test.

For both the disk and CD-ROM subsystems, File Manager commands control the majority of file I/O operations, such as copying, moving, deleting, creating, reading, and so on.

NOTE: For details on what you see onscreen during the disk tests, refer to Chapter 9, "Running MacBench's Tests."

Disk Mix

The Disk Mix provides an overall comparative measure of the disk subsystem's performance as compared to the base machine's. Here's what happens when you run the Disk Mix:

When you select the Disk Mix test, you may see a warning that the test will take some time to run. (You can turn off this warning in the Preferences dialog box.)

MacBench creates 60 Mb worth of test files. This setup process causes all disk tests to take longer on Mac OS systems with slower processors or under-powered hard drives.

NOTE: The data the Disk Mix uses is about as compressible as the data the profiled applications used, so you can get meaningful results from this test even when you're using disk compression products on your Mac OS system.

The Disk Mix is a single large test that performs the following operations on the test files:

MacBench times how long the test system runs the Disk Mix and compares that time to the base machine's. From this timing, MacBench computes the kilobytes (1,024 bytes) per second rate at which the test system read or wrote to the test files. MacBench then uses a weighted harmonic mean to calculate the final Disk Mix score, which shows how your Mac OS system performed in relation to the base machine, a Power Macintosh 6100/60.

Disk Mix scores

The Disk Mix returns a score of 10.0 for the base machine and normalizes your system's score accordingly. With this result, bigger numbers mean better performance.

Because the test mimics the kind of work applications do, you can use the results as a guide to the kind of throughput you can expect to see when working with the Mac OS system's disk subsystem.

How we created the test

We gathered information on what types of disk operations Mac OS-based applications perform by profiling 13 top-selling Mac OS-apps. The Disk Mix is an almost exact playback of these disk operations.

To ensure the accuracy of our profiling, we used commercially available and in-house tools to record application I/O operations, but without interfering with or modifying the original application. Using these tools, we recorded the File Manager commands for each of the 13 applications and combined them into one Disk Mix.

Publishing Disk Mix

The Publishing Disk Mix follows exactly the same scenario as for the Disk Mix; however, the test is based on profiling of Photoshop and QuarkXPress only and on scripts specific to the two applications. The Publishing Disk Mix is based on our profiling of power users' employing these applications on high-end publishing tasks, such as copying and moving large image files.

Its score is also a measure of how your Mac OS system performed in relation to the base machine.

Graphics tests

MacBench's graphics mix tests - the Graphics Mix and Publishing Graphics Mix - measure the overall performance of your Mac OS system's graphics subsystem. as it executes QuickDraw commands. MacBench's individual graphics tests run specific QuickDraw commands and record how many kilopixels the test system drew during the test.

QuickDraw commands control many different types of drawing tasks, such as displaying text, moving pixels from memory to screen and screen to memory, and drawing geometric objects (for example, lines, circles, arcs, and so on).

Graphics Mix

The Graphics Mix provides an overall comparative measure of the graphics subsystem's performance as compared to the base machine's. Here's what happens when you run the Graphics Mix:

When you select the Graphics Mix, you may see a warning that the test will take some time to run. (You can turn off this warning in the Preferences dialog box.)

The Graphics Mix is a single large test that reproduces the QuickDraw commands issued by our profiled applications, but instead of replaying the commands verbatim - as in the Disk Mix - the Graphics Mix represents the proportions and kinds of graphics operations our profiled applications used.

MacBench runs the Graphics Mix in six phases, with each phase lasting about 15 seconds. The phases correspond to the frequency with which each profiled application called the QuickDraw commands.

When the Graphics Mix begins a new phase, you'll see a dialog box saying, "Preparing 'Graphics Mix' test," the phase number (such as "Phase 2 of 6"), and a progress bar. What's happening behind the scenes is that MacBench is drawing every shape, every pattern, and every fill it will use in the test.

As MacBench executes each phase of the Graphics Mix, you'll see MacBench drawing many types of shapes, pictures, and text on the screen.

MacBench calculates the number of kilopixels the test system drew during each phase of the Graphics Mix. MacBench adds these numbers together and normalizes the final number to produce the Graphics Mix score.

Graphics Mix scores

The Graphics Mix returns a score of 10.0 for the base machine and normalizes your system's score accordingly. With this result, bigger numbers mean better performance.

You can use the Graphics Mix result to determine how to optimize the Mac OS system's graphics subsystem to best suit your needs. For example, the Graphics Mix results can help you determine how effectively QuickDraw accelerator cards operate in your working environment.

How we created the test

We gathered information on what types of QuickDraw operations Mac OS-based applications perform by profiling 13 top-selling Mac OS-apps. The Graphics Mix reflects the proportional use of the QuickDraw commands our profiled applications used.

To ensure the accuracy of our profiling, we used commercially available and in-house tools to record application I/O operations, but without interfering with or modifying the original application. Using these tools, we recorded the QuickDraw commands for each of the 13 applications and used the profile logs to create the Graphics Mix.

Publishing Graphics Mix

The Publishing Graphics Mix follows exactly the same scenario as for the Graphics Mix; however, the test is based on profiling of Photoshop and QuarkXPress only and on scripts specific to the two applications. The Publishing Graphics Mix is based on our profiling of power users' employing these applications on high-end publishing tasks, such as drawing complex images.

Its score is also a measure of how your Mac OS system performed in relation to the base machine.

CD-ROM tests

MacBench's CD-ROM Mix measures the overall performance of your Mac OS system's CD-ROM subsystem as it performs standard File Manager operations. MacBench's individual CD-ROM tests read data from the MacBench CD-ROM and record how many kilobytes the Mac OS system's CD-ROM subsystem read during the test.

Earlier in this chapter, we discussed the disk tests' sequential and random read tests. The sequential test reads sequential tracks on the CD-ROM. The random read test hops here and there all over the test files on the CD-ROM, reading in data wherever the drive head lands.

NOTE: For more information on sequential and random tests and the effect of block sizes on them, refer to "" on page .

CD-ROM Mix

The CD-ROM Mix provides an overall comparative measure of the CD-ROM subsystem's performance as compared to the base machine's. When you run the CD-ROM Mix, the CD-ROM that includes MacBench must be in the CD-ROM drive so MacBench can find the files it needs to run the test.

Here's what happens when you run the CD-ROM Mix:

NOTE: These files are contiguous and occupy the inner track of the MacBench CD-ROM. We put the files on the inner track of the CD-ROM to ensure compatibility with future MacBench releases. However, our studies have shown that file placement on a CD-ROM in non-inner-track configurations shows little effect on scores.

MacBench times how long the test system runs the CD-ROM Mix and compares that time to the base machine's. From this timing, MacBench computes the kilobytes (1,024 bytes) per second rate at which the test system read the test files. MacBench then uses a weighted harmonic mean to calculate the final CD-ROM Mix score, which shows how your Mac OS system performed in relation to the base machine, a Power Macintosh 6100/60.

CD-ROM Mix scores

The CD-ROM Mix returns a score of 10.0 for the base machine and normalizes your system's score accordingly. With this result, bigger numbers mean better performance.

Because the test mimics the kind of work top-selling CD-ROM-based applications do, you can use the results as a guide to the kind of throughput you can expect to see when working with the Mac OS system's CD-ROM subsystem.

How we created the test

We gathered information on what types of CD-ROM operations Mac OS-based applications perform by profiling five top-selling CD-ROM applications. The CD-ROM Mix is an almost exact playback of these operations.

To ensure the accuracy of our profiling, we used commercially available and in-house tools to record application I/O operations, but without interfering with or modifying the original application. Using these tools, we recorded the File Manager commands for each of the 5 applications and combined them into one CD-ROM Mix.

What MacBench's results mean about a system's performance

This section explains what the results of the MacBench tests can tell you about a Mac OS system's performance.

Processor results

The Processor test measures the speed and power of your Mac OS system's processor, particularly as the processor interacts with RAM. When you run the Processor test, MacBench 3.0 runs a large instruction mix that emulates the processor activity of real-world applications.

The Processor test focuses solely on the processor subsystem. It performs no I/O operations or other functions that might involve the computer's disk or graphics subsystems. You can use this test to see how effectively your system's processor and RAM work together. The faster the speed of the processor, memory, and cache, the better your system's score on this test.

Floating Point results

The Floating Point test measures how fast your Mac OS system's floating-point unit (FPU) handles floating point operations. We based the test on our experiments and observations of how real-world applications use the floating point subsystem.

Disk Mix

MacBench uses the Disk Mix to provide an overall evaluation of your Mac OS system's disk subsystem as compared to the base machine's. We gathered information on what types of disk operations Mac OS applications perform by profiling top-selling applications. This test uses File Manager calls to perform disk operations that applications routinely perform.

The Disk Mix results can tell you how fast your Mac OS system's disk subsystem executes these File Manager commands. Because the test mimics the kinds of work applications do, you can use the results as a guide to the level of throughput you can expect to see when working with the disk subsystem.

Graphics Mix

The Graphics Mix provides an overall score for your Mac OS system's graphics subsystem as compared to the base machine's. The Graphics Mix performs a variety of graphics operations that popular applications use. The test focuses on the frequency of the QuickDraw calls those applications made.

The Graphics Mix can help you determine how effectively built-in graphics or a QuickDraw accelerator card can process QuickDraw calls. You can also use these results, for example, to determine the relative speeds of different color depths on a single display card.

CD-ROM results

MacBench uses the CD-ROM Mix to provide an overall evaluation of your Mac OS system's CD-ROM drive subsystem as compared to the base machine. The CD-ROM Mix performs the same types of operations CD-ROM-based applications perform. Because the test mimics the kinds of work applications do, you can use the results as a guide to the level of throughput you can expect to see when working with the CD-ROM subsystem.

We modeled the CD-ROM tests on the individual disk tests and the Disk Mix. As a consequence, the individual CD-ROM tests perform sequential and random read (but no write) operations on blocks of data that are 512 bytes, 1 Kb, 32 Kb, 64 Kb, 128 Kb, and 1024 Kb in size. A CD-ROM test result indicates how many kilobytes per second your Mac OS system's CD-ROM subsystem read during that test.

NOTE: When you run any of MacBench's CD-ROM tests, the CD-ROM that includes MacBench must be in the CD-ROM drive so MacBench can find the files it needs to run the test.

Publishing Mixes

The Publishing Disk Mix and the Publishing Graphics Mix provide overall scores that reflect how your Mac OS system performs high-end publishing work, such as moving, reading, and displaying large graphics files. MacBench compares the score your system receives to the base machine's score on these tests.

What can the publishing mixes tell you about your Mac OS system? Let's first explain what MacBench's standard mixes tell you.

We based MacBench's Disk, Graphics, and CD-ROM mixes on the profiling of market-leading Mac OS system applications. As a result, those mixes are "market-centered," meaning they reflect the way typical users run popular applications. A market-centered approach lets MacBench reflect the center of the market and provide a more realistic view of a Mac OS system's performance.

But some Mac users are not typical - in addition to creating spreadsheets or word-processing documents they edit video files or create large images and pictures. In other words, they stress their Mac OS systems in different ways than the typical MacBench user. MacBench's standard mixes sometimes don't speak to all the needs of these users, who may have invested in disk arrays or graphics cards that are specifically tuned to speedily perform heavy-duty work. Market-centered tests can't put the kind of stress on these devices that would tell these users whether their system's throughput has actually improved.

For that reason, we developed high-end publishing disk and graphics mixes that reflect how this particular niche of users work on their Mac OS systems. In developing our publishing mixes, we profiled QuarkXPress and Photoshop and observed how power users worked on these applications. Their activities included moving, copying, and displaying large pictures and files. (By "large," we means hundreds of megabytes in size.) We took that profiling information and created a Publishing Disk Mix and a Publishing Graphics Mix.

As an indication of the differences between the standard and the high-end mixes: In the standard Disk Mix, MacBench sets a small buffer size, which correlates to what we saw in our profiled applications. But in the Publishing Disk Mix, MacBench can set a large buffer size, meaning the Mac OS system can move more data faster.

The Publishing Mixes, therefore, can tell you how quickly your Mac OS system performs high-end, demanding disk- and graphics-based work, as compared to the Power Macintosh 6100/60.

Profiling

Measuring system performance in an accurate and repeatable way is important to you. So, before we release a benchmark, we do our homework. We research current trends in the Mac OS market, analyze new software and hardware that's available, and figure out what is important to a typical user of that software and hardware.

This chapter explains why MacBench measures performance the way it does.

Using a market-centered approach to research

We use a "market-centered" approach to our research on the kinds of tests MacBench should perform. All of MacBench's tests specifically focus on the types of use best-selling applications actually receive on Mac OS systems. We picked those applications because of their high current and projected market shares by unit of the Mac OS system software market.

While MacBench can't duplicate the exact functions you perform on your Mac OS system with your applications, MacBench's tests aim for users who use applications close to the center of the sales market. This approach lets MacBench reflect the center of the software market and provides you with a more realistic view how your Mac OS system can run today's software.

Profiling is key

Profiling lets us monitor an application as it works and lets us record the subsystem-level operations it performs. We use both commercially available and in-house tools to profile many different top-selling applications as they perform typical tasks.

The results of those profiling efforts - knowing which Mac OS operations the target applications use regularly - determined the way the tests in MacBench work. Knowing that information lets us build MacBench's tests to reflect the statistical profiles of those applications.

The following list shows the applications we profiled for MacBench 3.0, their version numbers, and whether they operated in native mode on a Power Mac OS system:

How does profiling work?

Any application that wants to display a dialog box or save a file has to communicate that to the Mac operating system (OS). The Mac OS then routes the commands to the appropriate software routines: graphics and display commands to QuickDraw, file manipulation commands to the File Manager.

Our profiling of an application does not modify or interfere with the way the application interacts with the OS.

When we profile an application, we're "listening in" on its conversation with the Mac OS. We also record the conversation. When you run any of the Mix tests, you're, in a way, replaying parts of a conversation several applications have had with the Mac OS and, through the OS, the File Manager and QuickDraw.

NOTE: Many other managers in the Mac OS take part in these conversations also: the Device Manager, the Resource Manager, the Font Manager, and so on. Our Mix tests depend on these managers just as your applications do. However, our profiling focuses on the major players of the Mac OS: the File Manager and QuickDraw.

Parameter distributions

MacBench's tests don't perform the same operations over and over for the same reason you don't draw a horizontal line on the screen the same way over and over. You may draw the line at the top of the screen or the bottom of the screen, in a heavy or a light typeface, long or short.

The different ways you can draw a line represent the various parameters the line-drawing command can have. In the same way, all File Manager and QuickDraw commands have different parameters that affect how they work.

Application profiling determined the distribution of Mac OS operations and their parameters in MacBench's tests. MacBench's tests reflect the different ways applications use these operations by assigning different parameters to the same operations.

Why results differ

If you run the same MacBench test on your system repeatedly, you'll get the same score almost all the time. The margin of error in the MacBench processor and graphics scores is about 3 percent. The margin of error is slightly higher for the disk tests, but remains under 5 percent.