Update on “The Golden Ear Challenge”: Who Will Conquer Audio Mount Everest?

tmias_thumbLast month, we issued “The Golden Ear Challenge”: Our promise to write a glowing, feature-length article about whoever becomes the first person on record to show he or she can reliably hear an improvement offered by any super-high-resolution file format under properly controlled conditions. So far, this prize remains unclaimed.

Although I have my doubts that anyone will ace this challenge, I hold out hope that it is possible. If, in the spring of 1952, you asked the average person whether someone would ever reach the top of Mt. Everest or run a mile in under 4:00, he might have said “Nah.” But within 2 years, both of those records were broken.

It could very well be that 16/44.1 PCM or 320kbps MP3 represent the very pinnacle of audio quality as far as the human ear is concerned. (At least that’s what the overwhelming weight of science seems to suggest so far.)

But hey, it’s worth trying, right? Especially since so many marketers claim there’s a real difference. And if that difference is real, it should be acknowledged.

Claims to the Contrary

Since issuing this challenge, we’ve heard from a few people whose responses have gone a little something like this: “What are you talking about?? The difference is obvious. I hear it all the time. What are you, deaf?

Of course this is al lot like a runner in April, 1954 saying: “What are you kidding? I run a sub 4 minute mile all the time. It’s just that no one has ever been there to record it.” It’s also just like a daredevil in April, 1953 saying: “Psssshhh, Mt Everest? I’ve been to the top. And I don’t need your silly ‘photographic evidence’ to prove it.”

So what gives? Why are a handful of people so certain they can hear differences that have never yet been documented under properly controlled conditions?

The usual answer to this is either: A) placebo effect, B) expectation bias C) a poor understanding of probabilities, D) a deep unfamiliarity with the body of evidence on the subject, E) wishful thinking or F) all of the above.

(For an overview of some of these crucial pro audio concepts, I strongly recommend Ethan Winer’s classic AES panel “The Audio Myths Workshop.” If yu’ve never seen it, it might just change the way you think about sound.)

Still, I can’t hold it against anyone who is convinced he has heard real differences based on sighted tests and marketing materials, because honestly, no one is born knowing this stuff. I know I wasn’t. And the truth is you don’t need to know the science to make great-sounding recordings. (Although it can’t hurt!)

Unsuccessful Attempts so Far

My favorite thing about the Golden Ears Challenge is that it has opened the eyes and ears of a few very reasonable people.

Thanks to this challenge, I’ve had several friends and colleagues participate in properly-controlled blind tests for the very first time. By and large, they’ve been astonished by the results.

The response from these participants has usually been something along the lines of: “Well, I thought I could hear a difference – Until you stopped telling me which was which!”

I remember feeling exactly the same way when I was first exposed to blind testing. It’s a huge perspective changer. Once you’ve felt for yourself just how strong the power of our unconscious biases are, it’s hard to ever look at the world the same way again.

Through blind testing, you come to realize firsthand that there is no such thing as an “unbiased” person. There are only people who are aware of their own biases, and those who are not.

For some of my colleagues, this also turned out to be their first real experience thinking about probabilities and how they apply to listening tests.

When they think about it for a minute, listeners quickly come to realize that it’s not enough to get the right answer once in a blind test. Just like flipping a coin and correctly calling “heads” is not proof of ESP, getting the answer right in a single blind AB comparison is no proof of an audible difference.

Some of us are surprised to realize that even getting the answer right three times isn’t enough! If you were to just randomly guess, you’d get the right answer three times in a row on 1 out of every 8 tries, purely by chance. You can see how cherry picking results could be a real problem! We wouldn’t conclude that 1 in 8 people are genuine psychics for correctly guessing a coin flip three times in a row. And there’s no reason to apply a different standard to audio.

What People Can Hear

So far, all the results we’ve collected during the Golden Ear Challenge have confirmed what scientists already know about human hearing. We just haven’t found anyone – including a few very capable GRAMMY award-winning audio engineers – who has achieved a result that flies in the face of the current body of evidence.

To date, no one who has taken this challenge has gotten a statistically significant result when comparing a final playback of a 16/44.1 file or 320 kbps mp3 to any higher resolution format. This includes 24/192, 24/96, 24/88.1 and SACD. In each and every case, the results have been no better than a coin flip.

For me, the most interesting result came from one participant’s test which suggested that he might be able to tell a difference between a 24 bit and 16 bit at the low levels commonly used for tracking. If he can hear that difference, it would be no surprise. But it does give us an extra insights into why, against all evidence, some engineers might swear by higher-resolution formats for playback.

During the tracking process, engineers tend not to use the full resolution of their digital systems so that they can avoid clipping. At this stage, it may be possible to hear a difference between the noise floor of 16 bit and 24 bit audio. This has been known for a long time. But we also know that this difference disappears as soon as the file is mastered or normalized in order to use the full resolution of either format.

For instance: If you set a peak level of -18dbfs during a recording session, and compared the resulting file at 16 and 24 bits without bringing up the level, you might hear a difference. That would be like comparing an effective resolution of 13 bits to 21 bits. This is a difference we’d expect many (but not all) trained listeners to hear. But in a case like this, you’re just not using all the bits available in the file! Once you bring your peak level up to anywhere near 0dbfs for final playback, that difference in noise floor would disappear completely.

(If those last two paragraphs make no sense to you, I recommend reading our recent article on what bit depth really means for audio quality.)

Placebo effect aside, it could be that since some engineers really do hear a difference in the noise floor between 16 and 24 bits while they’re recording. (Which is to be expected.) And this might lead them to incorrectly assume that higher resolutions must be better for final playback as well. That’s a possibility that’s worth considering.

Potential Victors?

So far, out of all the responses to the Golden Ears Challenge, we’ve heard from just one confident challenger who A) was already familiar with the concepts of placebo effect and expectation bias, B) already understood the importance of blind testing and statistical significance, and C) already trains his ears with blind ABX listening tests. If anyone can win this thing, it’s bound to be somebody like this!

We made contact via social media, and although I begged him to take the challenge, he did not pursue. His claim was that he’s able to reliably differentiate between 320kbps mp3 and CD quality sound 68% of the time. If we could even get that kind of result with any statistical significance, I’d consider this challenge won. But so far, we have no confirmed evidence. Just a claim on Facebook.

We’ve also heard from another impressive candidate who’s currently studying for a PhD in ear training. He is confident that he can demonstrate a clear positive at 192kbps, and seems optimistic that he might be able to succeed at 320kbps. Reliably differentiating a 192kbps file is an impressive feat in its own right (studies suggest that the majority of musicians and engineers can’t even hear that difference) but we already know that it can be done. If we could get any solid evidence of this happening at 320kbps, it would be news worth reporting.

So there you have it: Until we find a winner, you can consider this an ongoing challenge. To date, we’ve heard a lot of talk, and seen zero evidence. So if anyone comes to you with big claims about their ability to hear any improvement offered by extreme audio resolutions, send them our way and tell them to “Take The Challenge”!

Until then, the jury says: “Come on in; The music is good and the audio’s fine.”

This entry was posted in All Stories, Featured Stories, October 2013, Technology. Bookmark the permalink. Comments are closed, but you can leave a trackback: Trackback URL.
  • Sign up for our Email Newsletter

     

  • E-news and Updates

    Subscribe to Trust Me, I'm a Scientist Find me on Facebook Find me on Twitter Email Trust Me, I'm a Scientist