Introduction

We have received a lot of feedback from the DVD Benchmark and Progressive Scan Shootouts. Some of it has been positive and some of it not, but surprisingly, the majority of responses from manufacturers have been positive. The people most upset by our findings are those who own players that did not fare well in our tests. A link to the shootouts database, including our latest Shootout (January, 2003) is at the bottom of this page, immediately below "Test Results". You can search the database using criteria that are important to you. Please let us know if you like it and how we can improve it. No one else has something like this, so it is a learning experience for us too.

Our Vision

The progressive scan shootout results are not based on features, cost, build quality, or audio performance. We're primarily concerned with DVD players as video playback devices. We're not trying to do a DVD-A or SACD comparison, and we think most people use an outboard receiver or processor for Dolby Digital. Thus we care only that the DVD player sends the Dolby Digital bitstream to the receiver or processor properly, and we're confident that all of them do. We have concentrated our efforts on where it matters most: the image delivered on-screen. Our goal is to identify the DVD players most capable of reproducing an image that is most faithful to what the film-makers intended. In other words, we are looking for an ‘accurate' image.

If you are looking for reviews that detail the feature sets of players or even to make you feel good about your purchase, you might be disappointed with the shootouts. Our goal is to push each player to its limits. We want to identify its strengths and weaknesses in hopes that manufacturers will use this information to build better products that we all can enjoy.

It's worth noting (again) that we specifically designed our tests to isolate and magnify small differences between players that we think are important. If you look at the lowest-rated players and the highest-rated side-by-side playing a high quality, properly flagged movie (Superbit Fifth Element, anyone?), the differences will be subtle. This is the way most reviewers evaluate players, and we think it's a worthless technique for product reviewing.

What makes the most difference, in our opinion, is documenting when the player screws up. How often does it screw up? Under what circumstances? What kind of material does it handle poorly? How bad does it look when it screws up?

We basically think that's it's reasonable to expect your player to perform well on all discs, not just most discs.

We have worked hard to identify as many common problems as possible in terms of what you will run into with a progressive scan player. With each shootout we have tried to include new tests based on new problems that have been encountered. It is our belief that if we don't give the manufacturers something to improve on, then products will not evolve.

The discs we use for testing are not problem-specific to that disc, but are specific to the types of problems that many discs suffer from. They are the problems we test for. All too often people have associated the problems with the disc we test with. For example, we use The Big Lebowski to test for Bad Edits. People have incorrectly assumed that since they don't have Big Lebowski, they don't care about the problem. We are trying to address this in Shootout 3. We have renamed all tests to indicate what each test represents, and we simply list the disc we used to test for that problem. We have also updated Shootouts 1 and 2 to reflect this.

We chose test discs based on the fact that everyone has access to them in stores or DVD rental places. You can use the same discs that we do when evaluating a DVD player at your local dealer. Most dealers will let you try your own material.

We simply do not have enough time or resources to perform the DVD Benchmark in its entirety on every player. There has been a lot of feedback about the first two shootouts because it did not cover the basic video performance. We have decided to take some of the basic video tests from Part 1 of the Benchmark and combine them with the deinterlacing tests from Part 5. We hope this new information is what you are looking for, and we are sorry that we can't give you all of it.

Our Style

Most AV reviewers in general favor a "holistic" approach to reviewing. The idea is that the best way to evaluate a player is to play with the knobs, check out the features and output jacks, and then kick off your shoes and watch a few top-notch DVDs.

Our way is to take the thing apart, try to figure out (in obsessive detail) how all the pieces work and how to highlight the ways it differs from other similar units. It's an engineer's way of doing things.

If you tend to be fairly laid back in your viewing, and not too worried about a glitch here or there, or a disc that looks soft, things like that, then the holistic approach is probably fine.

If you get annoyed every time your player glitches, or you just are curious about how all this stuff works, then clearly our obsessive-compulsive reviewing style is right up your alley. We make no claims to have the perfect reviews - we tell people what we like, and we tell them exactly why.

Now, as to why the "other guys" will praise highly a player that we can show has serious defects, well, that's complicated. In some cases, what we consider a serious defect is not considered to be very important by some other writers.

In other cases, their testing is just not as sensitive. That's not a dig - their testing is not designed to be as sensitive. It's based on the "watch some good movies" approach, which is by its nature more slapdash. The implication is that it's important to focus on the forest (mainstream DVDs when everything is going right) and not the trees (unusual, smaller DVDs and occasional glitches). We like to dig around in the trees.

We personally don't like the holistic style of reviewing (surprise, surprise), at least for DVD players. We are gratified that there are others who feel the same way. We feel strongly that audio-visual equipment can be evaluated using an engineering approach because it has specific, measurable criteria for success. If we were reviewing restaurants, on the other hand, that wouldn't be true. Restaurants are not as amenable to that approach because there is no agreement on what constitutes a "perfect" meal. But with video, there is a holy grail: accuracy.

Would you like someone to review a computer by just sort of booting it up and playing around with the mouse, and then writing "I felt like the desktop had a stronger presence, and my computing tasks took on a greater importance, as though a veil had been lifted from my monitor"? No! You want people to measure things like speed and heat dissipation and reliability and features. And what is a DVD player if not a dedicated movie-playing computer? Some DVD players do their task of showing a glitch-free accurate image better. They do it on a wider variety of material, and they do it with fewer problems.

However, all that said, the main reason we launched into this whole shootout process is that we're huge movie fans, and we were annoyed by constant glitches distracting us from enjoying our movies. If we hadn't all been wound so tight, we probably would have just shrugged it off. Plus, we're all obsessed with finding out how things really work. However, since our first shootouts were published, manufacturers are responding, and fixing the problems. This is very gratifying.

The important thing is to enjoy the DVDs. If someone has a player that they've been enjoying, and then read our report and find out that it's not so great by our standards, does that negate their previous enjoyment? Of course not. And if they don't see any of the glitches we show, then they don't really have any reason to change players.

The Truth About the Flags

At the end of the day, we believe you should not need a fancy cadence based algorithm to deinterlace film sources. A DVD player should be able to simply use the flags on the disc to reconstruct the original frame of film. The problem is DVD authoring is far from perfect, although others may try to convince you otherwise. We are still waiting for them to provide data to backup their claims that encoding errors are nonexistent.

In the meantime, we have collected our own data. We have analyzed the content on lots of DVDs, including some of the most recent big name titles. (We have been doing this for over 2 years now.) From what we can tell with our data, DVD authoring is not improving. It seems to be staying constant, which makes sense to us because companies are trying to save money and one way to do that is to not purchase the latest authoring software or encoders every month.

Here is a small sample of DVD titles we have tested. We have included some big blockbusters as well as TV show content. We have listed the number of times the MPEG encoding has changed from 3-2 to video / 3-3 / 2-2. We have also listed if the chapter breaks on the disc have caused an unnecessary drop to video.

Match Your Socks, Not Your Electronics

Something we have always found unusual and it is not limited to DVD players or even consumer electronics, is brand loyalty. Just because you own a "Brand X" TV does not mean you need a "Brand X" DVD player. Chances are the people who designed and built the TV have no interaction with those who designed and built the DVD player. They might not even be at the same facility or even in the same city. There is also a good chance that they never tested on their own products.

When you buy a DVD player, you should ignore the brand name. First, you need to decide what is most important to you, absolute performance or features. Next figure out what the maximum amount you want to spend and start from there.

There are very few DVD player manufacturers who actually design their players from the ground up. In the main stream we know that Denon, Meridian, Panasonic, Pioneer, Sony, and Toshiba do build many of their DVD products in house. This is not an exhaustive list it is only a small sample. Most others will OEM a kit and start from there. In fact, some companies will OEM kits from multiple manufactures to build products at different price points.

As of this writing, there is only one universal player that does not suffer from the Chroma Upsampling Error (CUE). If you want a universal player for the sake of convenience, we fully understand that. Just don't expect the universal players to do everything great.

Lastly, buying one brand of products to be able to use a common remote control is a futile exercise! So called "Universal" remotes will only do the basic functions, so you still need the dedicated remote that came with the product.

A Sheep in Wolf's Clothing

A while back Jim Doolittle found a great example of why film mode deinteralcing (3-2 pulldown detection) is so important. Up until then, the best example was a test pattern, but too many people like to dismiss test patterns. We believe that the reason people dismiss video test patterns is because they are coming over from the audio side that is filled with lot a lot of voodoo and black magic. Video test patterns are extremely valuable and are used throughout the professional video world.

The example that Jim came up with is the opening of Star Trek: Insurrection. This is a long shot that pans left to right and then back again over a small village. It contains several buildings with fine detail and angles. If you do not have film mode, the angles show up as jagged instead of smooth diagonals. The problem now is that people have come to believe this is some form of torture test, which is simply not true.

We looked at the flags and they are just fine during the opening scene. There is also a strong 3-2 pulldown signature, which means any algorithm with 3-2 pulldown detection will have no problems deinterlacing this scene maintaining full vertical resolution with smooth diagonals.

This scene is only useful to demonstrate to someone why they want film mode detection. It offers nothing when comparing two different progressive scan players that have film mode detection.

Ok, we lied. There is actually a second use for that scene. Because it is a long pan, it is good to demonstrate judder caused by 3-2 pulldown. People love their HTPCs because they can remove this judder.

Another popular scene is the opening of The Fifth Element with the steps leading into the pyramid. It too is not a torture test of any kind and is just another good example of why film mode detection is so important.

Pressing On

As we have stated in Part 5, the only real way to evaluate a progressive scan player is to identify problem material. If there are no problems they will all perform the same in regards to constructing a progressive image. It is when the problems arise that the differences between players will become known.

Everyone involved with the progressive scan shootout here at Secrets is a film lover. When we sit down to watch a movie, we don't want to have to worry about what deinterlacing mode to put the player into, we simply want to be engaged by the story. We want to forget we are watching a DVD in our living room. There is nothing worse than while being immersed in the story that something happens on screen that should not have happened. When this happens you are reminded that you are at home watching video. It is like someone throwing cold water in your face.

We want a progressive scan player that is looking out for us. It is protecting us from all the unnecessary artifacts out there. With a great progressive scan player we can once again sit back, relax, and enjoy the film.

Evaluation Equipment and Process

We looked at all of the progressive DVD players using a Sony VW10HT LCD projector, which was selected because of its ability to handle a wide variety of different signal types and its very high resolution display. The projector's one major weakness is a lack of a very dark black level, but that didn't affect the tests we were making. We used the custom-made Canare LV-61S cables for most players, and a BetterCables Silver Serpent VGA to 5 RCA cable for the PC DVD players and the iScan Pro.

Players were scored on a sheet that had a pass/fail designation for each of the tests, plus areas for entering numerical scores for certain tests, and notes for anything unusual or interesting. All tests were witnessed by at least two people. For looking at the chroma separate from the luma, we used an iScan Pro, because it produces H and V sync signals even with Y'P'bP'r output, and the 10HT is able to sync to external H and V sync even in Component mode, so we were able to remove the Luma cable and still have a stable picture.

A Tektronix TDS3054 with the video option was used to measure the progressive outputs on each DVD player along with Avia and Video Essentials. We use the built-in cursors and math functions to measure the various video levels.

Notes on the Tests (How we Score)

We reworked the way we presented our tests for shootout 3, in a way that we hope will make it easier to see exactly what a particular player does or doesn't do well. Our old system graded players on whether they could handle a particular scene or disc properly, but people told us they weren't always sure what a player passing or failing a disc meant. Some people thought that if they didn't like Titanic, they didn't need to pay attention to whether a player passed the Titanic test. What was missing was a clear statement in the chart of the specific deinterlacing problem that was being tested for.

We are still using most of the same discs we used for shootouts 1 and 2, but we've broken the results down by deinterlacing issue. Some issues are covered by multiple discs, and some discs give results for multiple issues. But we think it will be easier to tell exactly why a player did or didn't do well.

In addition, we've given all the tests weights, based on how visible or distracting a particular problem is and also on how common that problem is. If a problem is not very distracting, and will only be seen on 0.1% of discs, we weighted it low. Problems that are visible on many discs, and cause significant, visible problems we weighted high. Weightings are from 1 to 10.

Most tests either pass or fail. A few have gradations of failure. If a test didn't pass, but wasn't as bad as the worst players, we in some cases, gave partial credit. Basically, each player starts with 0 points. For each test, a weighted test score is calculated, as follows:

First, the raw test score is calculated from 0 to 10. Either from pass / fail or table. (Listed with each test) Second, the weighted test score is calculated by multiplying the weighting of the test by the score and dividing by 10.

In equation form:

weighted_test_score = raw_test_score * test_weight / 10

All of the weighted test scores are added up for all the tests to form the final weighted score. Tests that have no data are counted as zero. In addition the total maximum score the player could have gotten (all the weights for all the tests that the player completed added together) is calculated.

The final score is expressed as a percentage of the maximum score for that player. In equation form:

Final_score = final_weighted_score / max_weighted_score * 100

So in other words, if the maximum score for a player, given the tests actually applied, is 150, but that player actually scored 75, the final score is 75 / 150 * 100 = 50. A different player, that had a slightly different set of tests, could have scored 200, but only scored 75. That player's score is 75 / 200 * 100 = 37.5, which is rounded to 38. All of the intermediate scores should be kept as floating point numbers, but the final score should be rounded to the nearest integer.

So any player that passes every test we gave it gets 100, and every player that fails every test we gave it gets 0.

The upside:

• All players scores can be reasonably compared, even though they had slightly different tests.

• The scores always remain in a reasonable range.

• The weightings are still very much important; high weighted tests affect the outcome much more than low-weighted tests.

The downside:

• A player doesn't get any real credit for passing a harder set of tests.

• You can get situations in which two players have essentially the same performance, but one gets a slightly higher score because it had one or two extra tests, which it passed.

We believe that the two downsides are not that bad. Looking at our data, each player was given a very comparable set of tests, and the differences in scoring are not going to amount to more than 1 or 2 points, which we've said in the notes is not a significant difference.

We have broken out the shootout into two sections Deinterlacing and Core. Deinterlacing will contain all of the progressive scan specific tests. We have grouped the deinterlacing tests based on what it is they are testing. Core will contain the chroma upsampling tests, video measurements, and some performance results.

In the sample pics below, the detailed inset pictures on the left are correct and the ones on the right are incorrect.

The tests are listed here in the order we will present them to you in the test results:

Deinterlacing

Core

Deinterlacing

Test Name:	3-2 Cadence, Film Flags
What it Measures:	Can the deinterlacer go into film mode and stay there, with standard NTSC film-sourced material.
Pass/Fail Criteria:	Deinterlacer must go into film mode and stay there for the duration of the test (about 45 seconds).
Disc(s)/Chapters used:	WHQL, Film Alternate 1
Weight:	10
Notes:	This is a fundamental deinterlacing task. Any deinterlacer that can't handle this doesn't really have a working film mode. To get to this test on WHQL: From the Main Menu, Choose Video Port Performance From the Video Port Performance menu, choose Film From the Film menu, choose Basic
Test Name:	3-2 Cadence, Alt. Flags
What it Measures:	Can the deinterlacer go into film mode and stay there, with NTSC film-source material that was authored using an encoder that turns the progressive flag off every other frame.
Pass/Fail Criteria:	Deinterlacer must go into film mode and stay there for the duration of the test (about 45 seconds).
Disc(s)/Chapters used:	WHQL, Film Alternate 2
Weight:	8
Notes:	This is just as fundamental as the previous test, but this material isn't as common. However, given the prominence of some of the titles that have this problem, we felt it was important to weight it fairly high. We removed this test from shootout #2, because in shootout 1, every player passed that passed the standard film test also passed this one. However, in doing the preliminary testing for shootout #3, we found two players that failed it. Clearly, we need to keep this around for the time being. To get to this test on WHQL: From the Main Menu, Choose Video Port Performance From the Video Port Performance menu, choose Film From the Film menu, choose Alternate
Test Name:	3-2 Cadence, Video Flags
What it Measures:	Can the deinterlacer go into film mode and stay there, with NTSC film-source material that was authored without any inverse telecine, and with no frame marked progressive (i.e., flagged as video).
Pass/Fail Criteria:	Deinterlacer must go into film mode and stay there for the duration of the test (about 45 seconds).
Disc(s)/Chapters used:	Galaxy Quest, trailer; More Tales of the City
Weight:	7
Notes:	This is not as common on major Hollywood movies. However, the results of failure are very visible, and handling film mode is fundamental to what a progressive DVD player is supposed to do.
Test Name:	3-2 Cadence, Mixed Flags
What it Measures:	Can the deinterlacer go into film mode and stay there, with NTSC film-source material which is changing back and forth from standard film-style flags to video-style flags.
Pass/Fail Criteria:	Deinterlacer must go into film mode and stay there for the duration of the test (about 45 seconds).
Disc(s)/Chapters used:	WHQL, Chapter stops 1 & 2
Weight:	6
Notes:	This particular flaw is extremely common, even on major Hollywood releases. However, it tends to happen in short bursts, especially right around the chapter stops, so it's not as bad as staying in video mode for the whole movie. To get to this test on WHQL: From the Main Menu, Choose Video Port Performance From the Video Port Performance menu, choose Variations From the Variations menu, choose Chapter Breaks From the Chapter Breaks menu, choose Alternate 1. After Alternate 1 has completed, run Alternate 2.
Test Name:	2-2 Cadence, Film Flags
What it Measures:	Can the deinterlacer go into film mode and stay there, with 30 fps progressive material with standard progressive flags. Pass/Fail criteria: Deinterlacer must go into film mode and stay there for the duration of the test (about 45 seconds).
Pass/Fail Criteria:	A player should go into film mode on this test. In the case of Natural Splendors, you should not see any visible jaggies. In the case of Avia, there should be no moiré on the zone plate. Avia is a better disc to use. We only provided Natural Splendors so you can see what happens on real material. Some players are more severe than others.
Disc(s)/Chapters used:	Natural Splendors, chapter 6; Avia, Zone plate
Weight:	5
Notes:	2-2 pulldown isn't that common in the US. In Europe, this test would be weighted 10, because almost all PAL film-sourced discs are flagged this way. We made an interesting discovery with the Avia pattern below. It is actually encoded as a progressive_sequence, which is not allowed. If you try to force it into video mode on a PC, it will still weave it together correctly with a slight loss of resolution, which suggest that PC software players actually look at this flag. None of the flag reading DVD players outside of the PC recognized this flag.
Test Name:	Film Mode, High Detail
What it Measures:	What it measures: Can the player go into film mode and stay there, with material with high spatial frequency and detail.
Pass/Fail Criteria:	Player must stay in film mode for the duration of the test (about 10 seconds)
Disc(s)/Chapters used:	Super Speedway, chapter 7, where the car goes by the stands.
Weight:	6
Notes:	This is a test that some deinterlacers ordinarily having good film mode fail, probably because the high frequency detail in the vertical direction looks like combing, and it drops into video mode to avoid artifacts. We think this is a flaw, when the material otherwise has a strong 3-2 pulldown signature. Surprisingly, this material is relatively common. If a player fails this test, it's likely that it drops to video mode much more often than necessary.
Test Name:	Bad Edit
What it Measures:	Whether the player can handle hiccups in the 3-2 cadence without combing.
Pass/Fail Criteria:	Result Criteria Points Pass 0 Combs 10 Fail 1 Comb 3 Fail+ 2+ Comb 0
Disc(s)/Chapters used:	Big Lebowski, Making-of documentary
Weight:	10
Notes:	This is a huge, huge issue, and the thing that really separates the best deinterlacers from the worst. Combing is one of the most visible artifacts, and avoiding it on material like this is difficult. We feel strongly that players that can't pass this test are not worth considering, given that there are inexpensive players that do pass it.
Test Name:	Video-to-Film Transition
What it Measures:	Can the player detect a transition from film to video, and avoid combing.
Pass/Fail Criteria:	Player must not comb during the test duration (about 30 seconds)
Disc(s)/Chapters used:	WHQL, Mixed Mode
Weight:	6
Notes:	It's relatively common for lesser deinterlacers to comb when the film cadence abruptly changes to a video cadence. This is not a common situation in most movies, but it's very common in film behind-the-scenes documentaries such as are found on special-edition DVDs. When the 60fps logo goes away, pay attention to the transition. Does it comb on this transition? You will have to watch the entire sequence, which is about 1 minute long. It may not comb on all. If it combs only just 1, it is a fail. To get to this test on WHQL: From the Main Menu, Choose Video Port Performance From the Video Port Performance menu, choose Variations From the Variations menu, choose Mixed Mode From the Mixed Mode menu, choose Basic Y
Test Name:	Recovery Time
What it Measures:	Can the player re-enter film mode in a reasonable amount of time after leaving video mode.
Pass/Fail Criteria:	Time to re-enter film mode after being in video mode:   Result Criteria Points Pass (Excellent) Within 5 frames 10 Borderline (OK) 6 to 15 frames 5 Fail (Poor) >15 frames 0
Disc(s)/Chapters used:	WHQL, Mixed Mode
Weight:	6
Notes:	The best deinterlacers return to film mode very quickly. If they stay in video mode too long, the chances of annoying deinterlacing artifacts appearing gets larger.
Test Name:	Incorrect Progressive Flag
What it Measures:	Can the player recognize that the progressive flag on the disc is not correct, and avoid combing.
Pass/Fail Criteria:	Player must not comb during the test duration (about 10 seconds for each disc)
Disc(s)/Chapters used:	Galaxy Quest, main menu intro; Apollo 13, making-of documentary
Weight:	6
Notes:	Players that always trust the flags will fail this test, because in this case, the flags are completely wrong. Material with the progressive flag set when it shouldn't be is relatively rare, but failure on this test is about as visible as it is possible to be.
Test Name:	Per-Pixel Motion-Adaptive
What it Measures:	Are the video deinterlacing algorithms per-pixel motion adaptive.
Pass/Fail Criteria:	The two parts of the test pattern had to be displayed solidly on screen without flickering, for the duration of the test.
Disc(s)/Chapters used:	Video Essentials, Zone Plate; Faroudja test disc, pendulum test.
Weight:	10
Notes:	This is the baseline of quality for video deinterlacing. Anything less is a compromise. The example below is a simulation. You will notice that the 'O' and 'K' are alternating. On the real pattern, this is happening so fast that it looks like they are just flickering but in reality, they are alternating like the animation.

Core

Test Name:	Sync Subtitle to Frames
What it Measures:	Does the MPEG decoder synchronize the subtitles with the film frames, avoiding combing in the subtitles.
Pass/Fail Criteria:	There had to be no combing in the subtitles for the duration of the test (about 30 seconds)
Disc(s)/Chapters used:	Abyss, chapter 4
Weight:	2
Notes:	This is an annoyance if you watch a lot of subtitled movies. You may or may not see a comb on the particular text below. This was created by hand to illustrate what the artifact looks like. The comb happens very fast, but we have slowed it down here so you can see it. The comb only occurs as the subtitle comes in or goes out. It is never there while the text is just sitting on screen.
Test Name:	Chroma, 3-2 Film Flags
What it Measures:	Does the player not have the Chroma Upsampling Error on standard film material.
Pass/Fail Criteria:	MPEG decoder must use a progressive chroma upsampling algorithm to pass. If it doesn't, but the player uses a deinterlacer that hides the chroma bug, we gave partial credit.   Result Criteria Points Pass No CUE 10 Borderline Masked CUE 5 Fail CUE 0
Disc(s)/Chapters used:	Toy Story, chapter 4
Weight:	10
Notes:	The chroma bug is visible on a huge variety of major Hollywood releases. Once you start seeing it, it's a huge distraction. We almost wish the weighting went to 11.
Test Name:	Chroma, 3-2 Alt. Flags
What it Measures:	Does the player not have the Chroma Upsampling Error on material that has the alternating progressive flag encoding problem.
Pass/Fail Criteria:	MPEG decoder must use a progressive chroma upsampling algorithm to pass. If it doesn't, but the player uses a deinterlacer that hides the chroma bug, we gave partial credit.   Result Criteria Points Pass No CUE 10 Borderline Masked CUE 5 Fail CUE 0
Disc(s)/Chapters used:	Monsters, Inc., Chapter 6
Weight:	8
Notes:	As with film mode, we only weight this lower because it's not as common as the "standard" flags.
Test Name:	Chroma, 2-2 Film Flags
What it Measures:	Does the player not have the Chroma Upsampling Error on material that is encoded 2-2 progressive.
Pass/Fail Criteria:	MPEG decoder must use a progressive chroma upsampling algorithm to pass. If it doesn't, but the player uses a deinterlacer that hides the chroma bug, we gave partial credit.   Result Criteria Points Pass No CUE 10 Borderline Masked CUE 5 Fail CUE 0
Disc(s)/Chapters used:	Toy Story (from the 3-disc Ultimate Edition), main menu
Weight:	8
Notes:	As with film mode, we only weight this lower because it's not as common as the "standard" flags.
Test Name:	4:2:0 Interlaced Chroma Problem
What it Measures:	Does the player have a deinterlacer or postprocessor that hides the chroma artifacts in 4:2:0 interlaced material.
Pass/Fail Criteria:	Player must mask chroma errors in interlaced material.
Disc(s)/Chapters used:	More Tales of the City, Chapter 4
Weight:	5
Notes:	This is in some ways expecting too much from current players, given that the methods for mitigating this problem have only just become known. But we thought it was a useful criterion. Next shootout, the weight will probably be higher.
Test Name:	Video Levels (Black and White)
What it Measures:	Does the player have correct voltage outputs for black and white level.
Pass/Fail Criteria:	To pass, a player has to have a black level of 0 IRE (in the darker of its black level modes, if it has two), and must output nominal white at 100 IRE +/- 2.
Disc(s)/Chapters used:	Avia, Horizontal Gray Ramp (Title 1, Chapter 103)
Weight:	8
Notes:	If a player doesn't output standard levels, consumers don't get good results out of the box, and it may be impossible to get a calibration that works properly with all the other components connected to the display. Below is a photo from one of the DVD players, as it appears on our test scope. In this particular case, it is the horizontal gray ramp found on Avia. For the video levels, we measured black and white. According to EIA770, 480p should have black at 0 IRE and white at 100 IRE. (Actually it is in mV, but we prefer to report in IRE because we believe it is easier to understand.) A consumer display must be capable of supporting multiple TV standards, which include SD, ED, and HD. For SD sources in the US, black is at 7.5 IRE. (Also called Setup or Pedestal) For the rest of the world, it is at 0 IRE. For ED and HD, black is at 0 IRE. Some DVD players allow you to turn on Setup (Set black at 7.5 IRE.) for ED sources. This makes since when your display device only has one video memory and you don't want to have to manually change the brightness setting as you switch between say cable TV and your DVD player. The problem we found is that no two DVD players call this feature by the same name. If you look at the animated image below you will see the same feature called by different names. Meridian and Pioneer call it what it is and let you select 0 IRE or 7.5 IRE. The others are a bit more cryptic. It gets even worse if you read the instruction manual and they tell you when you should adjust it. We have found some displays and video processors that will clip all information above 102 IRE. This is why we chose the +/- 2 IRE. Some DVD players allow you to adjust contrast to set the white level, but we are basing our test off of the default setting. We expect a manufacturer to do the right thing on the default memory. We found many errors on players, which suggest that manufactures are purposely changing the levels to make their player look different. Another reason for the level differences may be the tolerance of the parts used. In this case, a manufacture should be hand tweaking the players using test instrumentation as it comes off the production line. This is probably to much to ask for a $200 DVD player, but anything over $1000, especially those with a THX logo, it is expected. We had planned to use the picture controls built into each player and tell you which settings provide the most accurate image, but this became time consuming and we ran into problems where the 480i and 480p outputs were not active at the same time, which made it difficult to adjust the setting and view the output on our scope. The good news is that, in most cases, you can erase these differences by simply using a disc with test patterns to properly set contrast, brightness, and color. Once you have done this, all players will produce the black level and color. Another option, which is needed on some DVD players, is an outboard proc amp. They usually allow you to properly set all video levels before going into a display. Because of black being at 0 or 7.5 IRE, you need to be extra careful when comparing two different players or even the interlaced vs. progressive output on the same player. If your display does not offer more than one memory, you will need to adjust your player each time you switch between players. One of the players in shootout 3 has white at 106 IRE. If you compare this to another player that has white at 100 IRE you will have to either turn down contrast when you select the 106 IRE player or turn up contrast when you select the 100 IRE player to ensure both images are scaled properly. As a side note, if you plan to use the 106 IRE DVD player on a digital display, you will need a proc amp or you will not get the best possible image. Our point is that, if you see a difference in black level or color between players, it is operator error, because those differences don't exist when the voltage levels are the same.
Test Name:	Blacker-than-Black
What it Measures:	Does the player have the ability to reproduce the picture content below black.
Pass/Fail Criteria:	To pass, a player has to pass the below black on the PLUGE pattern.
Disc(s)/Chapters used:	Video Essentials, PLUGE pattern
Weight:	7
Notes:	We tested each player for the ability to pass blacker-than-black and whiter-than-white. All of the players had no problem with the whiter-than-white, so you will not see those results in the data. Below is a photo from one of the DVD players, as it appears on our test instrument. The ramp goes from 4% below black to 4% above white. The spike on the left is a marker at level 16, and the spike on the right is a marker for level 235. This pattern will be on Digital Video Essentials. It is available now on the DVD that comes with the book DVD Demystified 2nd Edition. The specification CCIR 601, which is the digital representation of the analog NTSC system, specifies the range of the luminance signal from levels 1 through 254 (Levels 0 and 255 are reserved.) with nominal 'black' at level 16 and nominal 'white' at level 235. The range of levels from 16 - 235 is where the bulk of the picture information is placed. The extra 34 levels are there for head and room toe room. Real world images contain scattered blacks (Levels 1-15) and whites (Levels 236-254). Along with these highlights and lowlights there are valid production reasons, which include the PLUGE test signal used to properly setup a display as well as keying. We believe that a digital representation of a 'natural' image requires a signal range beyond the nominal 'black' and 'white' points to look 'natural'. We like to use The Talented Mr. Ripley (Joe Kane pointed out the Talented Mr. Ripley a couple of years back) to demonstrate why BTB is needed. If below black is clipped, then the shadow detail will appear as blocky or pasty looking. This is similar to what happens when you clip white, but the pasty look is then located in the highlight areas. Above is a single frame taken from Mr. Ripley, and below is a table of levels 1-15 and how many samples on screen were set to that value. This only represents a very small number of levels in the frame, but we never said that a lot of information existed below black. Most information should reside from 16 - 235. Level Count 1 0 2 0 3 0 4 0 5 1 6 2 7 7 8 6 9 16 10 43 11 117 12 241 13 185 14 155 15 139 A big thanks goes to Ron Economos for analyzing this disc and dumping the bits used.
Test Name:	YC Delay
What it Measures:	Does the player have interchannel delay between the luma and chroma channels that meets the EIA 770 spec for consumer-grade video.
Pass/Fail Criteria:	To pass, a player has to have less than or equal to 5 nanoseconds delay between Y-Pb and Y-Pr.
Disc(s)/Chapters used:	Video Essentials; Bowtie (Title 18, Chapter 15)
Weight:	10
Notes:	When a player doesn't get this right, the video is continuously messed up. It's one of the core specifications of quality video. It is important that Y, Pb, and Pr all arrive at the same time to create an accurate image with clean, sharp, edges. In most cases the component timing is perfect up until the point it passes through the video DAC. Most errors are introduced between the DAC and the output connectors. Below is the YC pattern found on Video Essentials. If you look at the image on the far right, you will see that the left edge of the red stripe is smeared, while the right edge is dark. Compare that image with the one on the far left, where there is no smearing. The delay can be in either direction, so the smear may be on the left or right, and in some cases both sides.
	The specification EIA770 tells us what the voltage levels should be for SD (Standard definition or 480i), ED (Enhanced definition or 480p), and HD (High definition or 720p and 1080i). This specification also sets the guideline for the amount of delay allowed between channels. Below is a photo take from two different DVD players, as it appears on our test instrument. The graphic toggles between good YC timing and bad YC timing. The good timing is the one that looks like a bowtie. EIA 770 states: Less than 5ns is considered consumer grade. Less than 2ns is considered studio grade. We measure the timing between Y – Pb and Y – Pr. Most of the time when there is a delay between Y and the two color channels, it happens in the same direction. In some rare cases, the delay is in the opposite directions. This generates a delay between Pb and Pr. For example, if 'Y – Pb' is +5ns and 'Y – Pr' is -5ns, then there is a 10ns delay between Pb and Pr. We continue to use the 5ns Bowtie test pattern on Video Essentials. A single pixel is 74ns at 480i and 37ns at 480p. Smearing begins to become visible along edge transitions at 10ns with 480i and 5ns with 480p.
Test Name:	Image Cropping
What it Measures:	Does the player crop 0 lines from the top and bottom, and only a few from the left and right of the image, and is the image roughly centered.
Pass/Fail Criteria:	Result Criteria Points Pass (Excellent) < 5 L + R; < 3 Δ L - R; 0 T + B 10 Borderline (OK) < 10 L + R; < 4 Δ L - R; < 3 T + B 5 Fail (Poor) All else 0
Disc(s)/Chapters used:	Avia, Pixel Cropping pattern (Title 5, Chapter 89)
Weight:	4
Notes:	If the samples / lines are cropped, that's image information you can't see. There is no good reason to crop information off the top and bottom, and if there are samples cropped off the sides, they should be evenly taken from both sides. We will add that many consumer displays do have overscan, which will be greater than the amount of image cropping from a DVD player. This is no excuse for cropping the image. The sampling rate of luma (the black and white, brightness, portion of the signal) is 13.5 MHz, which provides 720 active samples per line. The keyword is samples, not pixels. When you see the active resolution written as 720 x 480, this implies 720 samples by 480 lines. What is the difference between samples and pixels? It is a subtle, but important, difference. Pixels have hard edges and occupy a fixed, rectangular space. The waveform generated to represent pixels has theoretically infinite bandwidth, and is as square as possible, with very quick transitions at pixel edges. Samples are instantaneous snapshots of an analog waveform. They are a way of representing an analog signal digitally. The output of a sampling system is strictly bandwidth limited, and has rounded curves with no edges at all. We will refer to the missing information from the left and right as cropped samples, and those from the top and bottom as cropped lines (a line of samples, which is equal 720 samples per line). In the NTSC system, it takes 52.65 µs (microseconds) to draw a scan line. It just so happens that it takes approximately 0.07407 µs to draw a single sample. If you multiply 0.07407 µs by 720 samples you end up with 53.3304 µs, which is 0.6804 µs more than the time it takes to draw a scan line. If you divide 0.6804 µs by 0.07407 µs, you end up with 9.19 samples less than 720 samples, or approximately 711 samples.
Test Name:	Layer Change
What it Measures:	Amount of time it takes to switch from one layer to another.
Pass/Fail Criteria:	Does the player move between disc layers quickly.   Result Criteria Points Pass (Excellent) < or = 1 Second 10 Borderline (OK) > 1 or < 2 Seconds 5 Fail (Poor) > or = 2 Seconds 0
Disc(s)/Chapters used:	WHQL, high bit rate title roll.
Weight:	4
Notes:	This is by design a worst-case layer change, involving a long seek across the disc. The layer change also occurs at the maximum bitrate. However, we find that it correlates well with the speed of all kinds of layer changes. To get to this test on WHQL: From the Main Menu, Choose Decode From the Decode menu, choose High Bit Rate From the High Bit Rate menu, choose Highest From the Highest menu, choose Title Roll The layer change will occur when you see "Warner Brothers" reach the center of the screen.
Test Name:	Responsiveness
What it Measures:	Does the player respond quickly to remote commands as well as menu navigation and chapter navigation.
Pass/Fail Criteria:	We score on a scale from 1 to 5 with 5 being the fastest.   Result Criteria Points Pass (Excellent) > or = 4.5 10 Borderline (OK) > or = 3, < 4.5 5 Fail (Poor) < 3 0
Disc(s)/Chapters used:	All discs in shootout
Weight:	6
Notes:	This is a very subjective test, but one we think is useful.
Test Name:	Video Frequency Response
What it Measures:	Amplitude of a specific frequency. (Resolution)
Pass/Fail Criteria:	Graph is green if we think response is ok. It will appear red if it is down more than -3 dB or above +.5 dB.
Disc(s)/Chapters used:	Avia, Multiburst (Title 3, Chapter 24)
Weight:	Not Weighted.
Notes:	This is the last area in the Core section. We are not going to Score the player based on the frequency response. We are simply going to show you what the frequency response of the player under test looks like. Some players allow you to alter the frequency response with a built-in sharpness control or through a preset memory. We did all of our tests with the default memory and all sharpness controls at their default, mid-point, positions. We started out by measuring Y, Pb, and Pr. We ran into a few players that had trouble on our test disc that contained the Pb and Pr sweeps. Because of this, we have stopped measuring Pb and Pr until a production disc comes out with the patterns that all players can read. If you would like to know more about this, please refer to the frequency response section in Part 1 of the DVD Benchmark. Below is a photo taken from one of the DVD players, as it appears on our test instrument, showing the multiburst pattern on Avia. You will notice that we state the frequency response out to 10 MHz. You might be wondering how we have a test pattern that goes that high. It is simple really. At 480p, all frequencies are doubled so the 5 MHz burst on Avia is really 10 MHz at 480p. We feel that this is the proper way to report the the frequency response. We often talk about ringing, but the multiburst (or sweep) patterns are not really appropriate to use when evaluating this. In the future, we will use a test pattern found on Sound & Vision Home Theater Tune-Up that contains three different edge transitions. We will break it down into Severe, Mild, and Easy tests. This is a bit more involved and is why it is not currently included. We also hope to include Phase response (Group Delay) measurements in future shootouts, and we are just waiting for some long overdue test discs, which will include the patterns necessary.

Secrets-Recommended 480i/p

We have created a “Secrets Recommended” award for all players that meet our criteria. We have broken the award out into 480i and 480p. While a player may meet our goals at 480p, it does not mean it will meet them at 480i and vice-versa.

For example, we might award our recommendation for 480p to a player that has the Chroma Upsampling Error but is masked on the progressive output. This same player would not be recommended for 480i.

We only looked at the progressive outputs in shootout number 3 so we will only hand out the award for 480p performance. It is possible that these players may pass our tests for 480i, but we have not tested them.

480i Criteria is as follows:
1. Must pass all CUE tests. (does not include ICP)
2. Must pass or "borderline" all core video basics.
3. Layer change < 2 seconds.
4. Responsiveness >= 3.

Test Results

HERE is the link to the Progressive Scan Shootout results. Try out the database, and let us know if you have any problems, and also, what improvements you would like to see. Send your suggestions to [email protected].

- Don Munsil and Stacey Spears -

Last Update: 1/27/03

"Abyss" Image Copyright 1989, Fox
"Apollo 13" Image Copyright 1995, Universal
"Avia" Image Copyright 1999, Ovation Software
"Galaxy Quest" Image Copyright 1999, Dreamworks
"Monsters, Inc." Image Copyright 2001, Pixar and Disney
"More Tales of the City" Image Copyright 1998, Showtime Networks Inc. and DVD International
"Natural Splendors" Image Coryright 2002, DVD International
"Super Speedway" Copyright 1997, IMAX
"Toy Story" Images Copyright 1995, Pixar and Disney
"WHQL DVD Test Annex 2.0" Images Copyright 1999, Microsoft Corporation

Note: We have heard of some corporations using our articles and graphics to teach courses to their staff without authorization. Secrets articles and graphics are original and copyrighted. If you wish to use them in your business, you MUST obtain permission from us first, which involves a license. Please contact the editor.

Terms and Conditions of Use