October 24, 2001 - Kenn Hwang and Brandon Bell
| |||
|
|
| |
|
|
Benchmarking Ethics and
the ATI Radeon October 24, 2001 - Kenn Hwang and Brandon Bell |
|
|
|
| |
Usually when a company optimizes the performance of their drivers for popular game engines or one game in particular this effort is applauded by the press and end users. However, driver optimizations have traditionally been accomplished via more efficient code. In the case of ATI's drivers, these optimizations are obtained by modifying the visual quality of textures; we'll discuss this in greater detail on subsequent pages. The question we'd like to address is the ethical implications of these drivers.
As we've discussed above, game-specific optimizations are certainly a good thing -- companies have been doing it for years. Even texture quality optimizations are nothing new. If you recall NVIDIA's 5.x driver release, texture compression was enabled by default. While this resulted in a significant performance gain, image quality was compromised. Unlike NVIDIA's experiement with texture settings in the 5.x Detonators, ATI's driver is different in the sense that its changes are undocumented and can't be toggled on or off by the end user. While this may be excusable to most gamers, it puts the development community in an awkward situation. How would a texture artist using Quake 3 check his work?
For hardware websites such as FS, test results obtained with ATI's drivers arguably shouldn't be compared to other cards. While ATI owners are forced to use the settings the driver imposes (and therefore its indicative of what an end user would experience), the settings used on the other cards aren't the same, giving ATI an unfair advantage. Quite simply it isn't an apples to apples comparison.
Whether or not this is a deceptive practice is for you to decide; right now we're going to explain what we believe ATI is doing in the drivers provided with the newer RADEON boards.
Naming Code
While we inquired into the quackifier source, we asked Andrew, our ace Gamers.com developer to create an application from scratch that does the same thing; namely, to search a single file for any string values containing "quake" or "Quake" and rename those to "quack". Within 5 minutes, we had a fully working version, shown below:
import java.io.*;
import java.util.*;
public class Test {
public static void main(String[] args) {
try {
//test string
String str = args[0];
String rep = args[1];
String inFile = args[2];
String outFile = args[3];
System.out.println( "Test String: " + str );
byte[] pattern = str.getBytes();
System.out.println( "Replace with: " + rep );
byte[] replace = rep.getBytes();
//read file into array
BufferedInputStream input = new BufferedInputStream
(new FileInputStream(inFile));
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int b = input.read();
while (b != -1) {
buffer.write(b);
b = input.read();
}
byte [] file = buffer.toByteArray();
buffer.reset();
System.out.println(inFile + " size: " + file.length + " bytes.");
// search thru file & replace
byte [] test = new byte[pattern.length];
for (int i=0; i < file.length; i++) {
if (file[i] == pattern[0] && (i + pattern.length) <= file.length ) {
System.arraycopy( file, i, test, 0, pattern.length);
if (Arrays.equals( pattern, test )) {
System.out.println("Match at: " + i);
buffer.write(replace, 0, replace.length);
i += (pattern.length - 1);
continue;
}
}
buffer.write(file[i]);
}
byte [] newFile = buffer.toByteArray();
buffer.close();
// write out new file if changed
if (! Arrays.equals(file, newFile) && ! outFile.equals("null")) {
BufferedOutputStream output = new BufferedOutputStream
(new FileOutputStream(outFile));
output.write(newFile, 0, newFile.length);
output.close();
System.out.println("Wrote new file: " + outFile);
} else {
System.out.println("File not written");
}
} catch (Exception e) {
e.printStackTrace();
}
}
public static void print(byte [] pattern) {
for (int i=0; i < pattern.length; i++) {
System.out.print( pattern[i] + " ");
}
System.out.println();
System.out.println();
}
}
Now that we have our quack, let's see what it does.
Image Quality
|
|
|
Best Quality Ever |
In the above screenshots, you can easily see that the large animated star texture is set to a low-quality texture in shot 1, even though we have explicity set Quake 3 to its highest-quality image setting. In fact, this particular texture is a pixel-perfect matchup for a low texture quality setting on an 8500 running the modified quack3, and a very close to the same setting on an Nvidia GeForce3, as the following 200% crops from the above images will show.
 |
 |
 |
| 8500 Quake3.exe | 8500 Quack3.exe | GF3 Quake3.exe |
This alone could have been enough to account for the Radeon 8500's increased benchmark scores. But wouldn't it have been a bit too obvious to anyone who does more than simply run through the benchmarks?
More Image Changes
 |
 |
 |
| 8500 Quake3.exe Best | 8500 Quack3.exe Best | 8500 Quack.exe Low |
 |
 | |
| GF3 Best Quality | GF3 Low Quality |
The first image is of course the "optimized" Quake3.exe from a stock Radeon 8500. Interestingly, what we see here is a mix between the high texture quality 2nd and 4th images, and the low texture quality of the 3rd and 5th. Notice that due to the downward angle of view in the original shot, the top of the crop is closer than the bottom, and is right at the boundary of the the bilinear filter. Apparently, the Quake-specific code in the ATI drivers will pull out more detail than the standard low texture-quality setting. The effect of this in-game is generally passable as medium-to-high texture quality, but with a 9-15% boost in framerate.
|
|
| |||
|
|
Also, a quick 400% crop of the health indicator shows even more, namely a few incongruous artifacts along smooth lines, and what appears to be 16-bit texture storage. This effect could also be a texture compression scheme, which would further reduce the bandwidth requirements at medium to high resolution.
 |
 |
 |
| 8500 Quake3.exe 32-bit textures |
8500 Quack3.exe 32-bit textures |
8500 Quack3.exe 16-bit textures |
Now that we have an idea of what ATI's "Performance Drivers" may be doing, let's run some benchmarks and see if our theories hold water!
System Setup
ABIT TH7-RAID
256MB PC800 RDRAM
ATI RADEON 7500
ATI RADEON 8500
Driver version
7.60
ASUS V8200 GeForce3 Pure
NVIDIA GeForce3 Ti
500
NVIDIA GeForce3 Ti 200
Driver version
Detonator 21.85
30GB IBM Deskstar DTLA 307030 ATA/100 Hard Drive
AFREEY
12X DVD-ROM
Windows 98 SE
Windows XP Professional
DirectX 8.0a
Desktop resolution: 800x600x16
Quake 3 Retail - 640x480 High Quality
Quake 3 Retail -
800x600 High Quality
Quake 3 Retail - 1024x768 High
Quality
Quake 3 Retail - 1280x960 High Quality
Quake 3
Retail - 1600x1200 High Quality
Quake3 Benchmarks
The GeForce2 Ti and RADEON 7500 compare very close to each other in performance. The RADEON 7500 results we've listed here are from the stock driver. When using our custom "quack" driver, the performance dropoff is typically less than 2%.
More Benchmarks
The RADEON 7500 is able to outperform the GeForce2 Ti, thanks in large part to its superior performance in the same dragothic demo. The RADEON 7500 trades positions with the GeForce2 Ti in the other tests.
Conclusion
However, several facts point towards a more conspiratorial design. For one, as big as the Quake brand is, nowadays it's not even close to being the most heavily-played FPS. However, it IS a recognized, industry-standard benchmark; as such, it was a very convenient game for ATI to dedicate resources towards optimizing. Also, the modifications to the ATI drivers completely override the graphics settings available within Quake 3. This also conveniently leaves benchmarkers in the cold because they cannot effectively change game settings back to default; any comparisons with other drivers are video cards become essentially useless.
But most damning is the fact that ATI chose not to disclose its modifications to the public, or provide a way to toggle them on and off. Do such optimizations need to remain confidential? To what end does this serve? While the static screenshots seem obvious, it takes a careful eye to catch the drivers in the act in a dynamic game environment...and if benchmarkers miss it and erroneously attribute higher scores to the Radeon's performance, could we have counted on the manufacturer to set the record straight?
|
|
| |||
|
|
|
