Let’s Try This Again! Is a Blind Test or Null Test Best To Compare Guitar Profilers?
#tonexpedal #tonex #neuralampmodeler
Line 6 Marketplace Presets: https://bit.ly/line6marketplace-sadites
Helix Ultimate Complete Presets Vol. 1-7: https://bit.ly/3xuAD30 (NEW)
Helix Ultimate Complete Presets Vol. 1-6: https://bit.ly/3IRjVwz
Helix Ultimate Complete Presets Vol. 8: https://bit.ly/3XQCWb3
Tonex Tone Models: https://bit.ly/tonexpresets
Tonex ToneNet Tone Models: https://bit.ly/3JosdwY
Quad Cortex Presets & Captures: https://bit.ly/4dFDG92
N.A.M (Neural Amp Modeler) Tone Collections: https://bit.ly/3SOXyhx
Kemper Rigs/Profiles: https://bit.ly/3shWRTe
Tonex Pedal (Affiliate Link): https://bit.ly/3N6V7D7
Tonex One Pedal (Affiliate Link): https://bit.ly/3zzdEoz
Tonex Software (Affiliate Link): https://bit.ly/3ZyBf3s
Join this channel to get access to perks:
https://www.youtube.com/channel/UCvhdK4-QIzo76Y_RIbxZctg/join
I recently did a video that talked about whether blind listening tests are better than null tests when comparing guitar profilers. The video ruffled a lot of feathers and some people got rather upset at me about it. They also misinterpreted much of what I was saying, as I suspected might happen. In this video I re-cap what was actually said in the first video and also show three more examples that further show the points I was actually trying to make. I hope you enjoy the video and the tone models and captures!
Tonex – Sir Hombre Ultimate and DI Tone Models: https://bit.ly/4cQjX5u
N.A.M – Sir Hombre Ultimate and DI Captures: https://bit.ly/46bt1Q4
As always, thanks so much for watching…
Please visit my sponsors:
Line 6: http://www.line6.com
Vigier Guitars: http://www.vigierguitars.com
Toontrack: http://www.toontrack.com
Mission Engineering: www.missionengineering.com
Morningstar FX: http://www.morningstarfx.com
Temple Audio: http://www.templeaudio.com
0:00 – Introduction
0:25- The Only Reason I Made The First Video
0:59 – What I Was Claiming In The First Video
1:25 – Why I Used The Null Test I Showed in The First Video
2:42 – What Can A Null Test Tell Us?
3:36 – The Sound Of The Difference Is Not The Same As The Difference In The Sound
4:22 – Null Test Example 1- Extremely Subtle Differences
9:01 – Null Test Example 2 – Same Processing Not Passing Null Test
12:07 – Null test Example 3 – Real Amp Doesn’t Null With Itself
15:12 – Conclusion
15:27 – Important!
16:14 – Equally Important!
16:38 – Also Equally Important!
17:47 – Tonex – Sir Ombre Ultimate and DI Tone Models
18:50 – N.A.M – Sir Ombre Ultimate and DI Tone Models
#Lets #Blind #Test #Null #Test #Compare #Guitar #Profilers
Originally posted by UCvhdK4-QIzo76Y_RIbxZctg at https://www.youtube.com/watch?v=oNu5nrFbppw
I just want to add this comment as I think it may help some people understand even further.
The thing that some people are missing is the null test does not objectively measure ONLY differences that would be perceptible to humans. It measures ALL differences. Some of those differences may cause absolutely no difference in how a human will perceive the two sounds (I showed numerous examples of this in the two videos). Which is why I said, IN THIS PARTICULAR SCENARIO (which is all this discussion has ever been about), the null test is simply the incorrect tool for THIS job and why I specifically pointed out that everything I was talking about in the two videos is about the use of null tests vs. blind listening tests when comparing which is more accurate to the real amp when comparing N.A.M and Tonex. Hope that clarifies further.
Very cool insights, Jason! Thanks for your explanations! I think the most compelling point you're making is what tiny, imperceptible changes such as a 1 sample delay or a .01 db volume change in otherwise identical files does to the null test. Given all of these modeling devices have different latencies, that alone should dissuade people from putting so much stock into the null test. Moral of the story is: If you can't tell a difference by ear in a studio situation, there is no difference that matters in any musical situation. Buy the gear that suits your application and budget and be thankful the digital tones are so awesome now that we can even have these overblown debates. What a time to play guitar!
Great video, so which of your videos shows the proper test setup? So I get it that ears are the best method rather than a null test but would like a spectrum analyzer be a good tool to compare the original against a capture. Sometimes I get "ear blind" and can't even be sure what I listened too.
How about this fanboys. Use which ever one you think sounds better. And stop with the gas lighting. Use your ears. You know. The things that matters ? Some of you are cooked.
"Will not tell us if the difference is audible" That depends on the magnitude of the difference, and in what frequencies. When doing a null test you're probably better off analyzing the spectrum rather than just the level. It might reveal an insight into how they are processing the audio differently. And while they may not matter, so long as you get the tone you want. It might change the way you think about dialing in your tone.
Most don’t care. Let the idiots carry on in their mums basement. Who cares.
Maybe just get out and play guitar.
Stop worrying about idiots
Im so glad i got off the modeller train. Got a HX stomp…does the job sounds awesome. Thats all i need.
This whole discussion becomes ridiculous. It started with some folks using a null test so show how close some of the modelers have gotten to the original amp (which was ok). Next some people started thinking they could always use a null test to show how well a modeler performs (which was the wrong conclusion!). Now poor Jason is crucified for pointing out the mistake. Technically it comes down to the difference between sufficient and necessary. A perfect null test is a sufficient indicator for two signals sounding identical. But it is not necessary (as the 20Hz example perfectly proved.)
It goes without saying that a null test can't work, however, an ABX test is also not the right test. It won't give accurate results. The upside is, the type of blind test that will work, is much easier to do, and I'll explain why. To start with, you need a clear goal. What are you testing for, and why? In this case, "Is there an audible difference between unit A and unit B." That's it. To have a valid test, you need to test for something objective. Something like, "Does A sound better than B?", doesn't make for a valid test because sound quality is subjective. There is no right or wrong answer.
The first step when you are doing any type of testing that involves the scientific method, is you need to have some type of event or observation that can't be explained with current information. (This is why DBX won't work). So, the observation is, "I can hear a difference between A and B." You need to have that or you don't have test. Is the difference between A and B real, or is it just in my head? This is what we're going to find out. Before we start the blind test (assume I'll be the one taking the test.), you need to sit me down and do a sighted test. The reason for this, is any time you are talking about differences in sound, especially small differences. Results vary depending on the test subjects listening skills, and the equipment. For example, I may be able to hear the deference between A and B on one pair of studio monitors, but not another. In audio, this is a very common issue. So, I do a sighted test first to confirm that I can actually hear the difference in the setup that's being used to conduct the test. Once we have that, the blind test begins.
For the blind portion of the test, someone will play the same 2 short clips used in the sighted test. The key here is repetition. If there really is no difference between the 2 clips and its all in my head, doing the sighted test can't change the outcome, so long as you do enough repetitions. Why? If you play just the 2 clips one time, I have a 50% chance of guessing the correct answer. If you play the same 2 clips randomly for 25 or 50 times, I have a better chance at winning the lottery. I'm not going to guess correctly 50x in a row. And that's it. If I guess around 50% correct, its probably all in my head. If I get something like 85% or higher, then the difference is going to be real.
The reason I'm going into all this detail is anyone can put together a test like this. DBX is a pain in the ass and it won't be accurate. What I just listed, anyone can do. I also have to give full disclosure. For some reason people always get mad at me if I don't give some background info about myself. I have a masters degree in Psychology and I do sound quality for a living. It really shouldn't matter. You don't have to be an engineer to listen to music. I guess my qualifications are relevant because I have a lot of experience doing these types of tests. They're required for the type of work I do. But like I said before, anyone can do tests like this and get just as good results.
I personally don't care which one is more accurate.
My opinion is that the only valid test for guitar profilers is trying to get a sound you like out of it. If you can't do that, it doesn't matter if the whole internet tells you how true to life the amp sims are. Like, I started with the Behringer V-Amp Pro and I absolutely loved that thing. I could get really good tones out of it, despite literally everyone saying it's bad. Years later I moved to a Line6 HD500x. Everyone was saying it was great. I hated it. I have never been able to get a nice tone out of it. Did not get along with it at all. Now I have the Helix and I really like it. That's the only thing that matters.
Jason I joked in one of the groups that I'm always right. I'm glad you got the joke.
People are judging this with their eyes and not their ears. There are so many internet sonic experts out there. They're out there hanging out with the internet doctors and layers and all of those magical people who know everything about anything. I'm definitely going to subscribe to the person who starts showing how they're right about everything the rest of us are wrong about! Haha!
I watch your stuff but rarely comment because I'm absorbing the informal. I'm really new to the modeling world & am definitely all about learning as much as possible!
Keep up the awesomeness! ????????
I'm gonna need a few more Facebook blind tests and a null test or two to decide which sounds better. Bwa ha ha ha. Great info, Jason. Appreciate the diligence.
There are so many factors that could spit in the soup of a null test between modeler and some real amp, like phases not being properly aligned and difference in amplitude. Even reamping the signal twice through the same tube amp may fail the null-test, because of dirt on the mains grid or anything else interfering. Don't get me wrong: I've tested Kemper, QC, ToneX, and NAM. NAM felt and sounded best to me. But if we start doing the more scientific approach with null-tests and whatnot, then we should also look at things like aliasing. The CNNs being used are notoriously bad at avoiding that from happening.
I was curious about amp vs Tonex distortion breakup. If the breakup points are different it would cause null fail even though it sounds identical. Glad you tested this but better, amp vs same amp.
I’m soooo over guitarists wasting time in these type of arguments. IMO, if it sounds good, it is good. How does this improve your playing/musicianship? Ohhhh, it didn’t! Quit wasting time with this crap. Spend it playing.
Your patience and kindness is the thing of legends. Thank you for all of your videos Jason!
I think it's a bit weird how people obsess about matching an amp hyperperfectly vs. are you getting a sound you love in your music. Analog amps (and pedals) of the same make and model don't even match each other due to component tolerances.
The null test does tell you how comparatively similar two sources are it just 1) tells you objectively rather than subjectively when subjective difference may be more important and 2) provides evidence of an actual achieved similarity in a test rather than a maximum potentially achieved similarity.
If it sounds very low null test is achieved. It’s not 1 or zero.
It’s integrated LUFs.
The fact is that if you can ear something when it’s phase opposed, then the null test is audible.
It’s not « perfect ».
Another great null test fail example would be to take the two identical DI tracks and nudge one over just one millisecond. The sources are clearly identical but that time delay, without actually changing the content of the DI track, would cause a significant null failure. This is similar to the HX example you gave but a bit more technically visible.
Jason Sadites, cool video you deserve more subscribers
Jason Sadites, awesome video you deserve more subscribers