Twice a year, wireless industry journalists breathlessly cover the results of the biannual RootMetrics Mobile Network Performance in the United States report. 

As soon as the study is released, various news outlets jump on the numbers, and those stories tend to take a pretty surface view of the data. Sometimes it's as simple as "Verizon (VZ -4.67%) Wins Again" or maybe the headlines get nuanced enough to point out that AT&T (T -1.21%) has worked its way to a close second while T-Mobile (TMUS 0.02%) and Sprint (S) have improved.

Some media -- myself included -- have even taken a bit of a suspicious eye to the numbers, digging in a little deeper to offer context and some have even cited a competing report from OpenSignal that shows T-Mobile leading the way, trailed by Verizon, Sprint, and, lastly, AT&T. [Here's my take on the latest RootMetrics report.] But, in a broad sense, RootMetrics has been accepted as a viable standard by most, with Verizon, of course, embracing it enough to base its "A Better Network As Explained By Colorful Balls" ads on its data.

RootMetrics, however, does have one major critic. T-Mobile's John Legere has been a vocal opponent of the validity of the research and, in a recent email to the media, he said that the company "should be banned as an independent source for network benchmarking, period!"

That may sound like a bitter reaction from someone who leads a company that has consistently scored at the bottom of the report, but Legere backs up his hyperbole with some reasonable assertions. 

What is Legere saying?
Basically, Legere takes issue with the choices made by RootMetrics in how to test his company's network. Here, he charges that those decisions tilt the results in favor of its rivals:

They manipulated their testing of the T-Mobile network, choosing to turn OFF Voice over LTE, our network technology that is on every single phone we sell. VoLTE handles roughly 50% of calls made on the T-Mobile network. That is 250 million calls per day, or over 40 BILLION T-Mobile calls that RootMetrics just CHOSE to exclude in their latest tests. So the latest (and by latest, I mean up to 7 months old) RootMetrics results are worthless.

Legere also questions exactly how independent the research firm is, asking "Do we have to pay RootMetrics millions like the other carriers do to get them to stop deliberately turning off significant portions of our network and skewing results during their drive tests? Coincidence? Doubt it." RootMetrics says anyone can purchase its highly detailed data, but that is not connected to its results. 

Legere then invites people to email him (though no specific email is provided) for "more examples of their incomplete methodology."  

T-Mobile CTO Neville Ray added that the current report is based on data that is seven months old while the Verizon ad is based on year-old information. 

Verizon topped the report in overall performance again. Data source RootMetrics.

How did RootMetrics respond?
In a broad sense, RootMetrics did not really disagree with what Legere had to say. It explained its logic in an email to The Motley Fool from Vice President Julie Dey, who answered a request to respond to Legere's remarks with statements including:

Regarding why we did not test VoLTE in 2015, our goal always is to test what typical consumers experience. In the second half of 2015, VoLTE on T-Mobile was not what the majority of consumers were experiencing because it was only available on a small number of devices. T-Mobile did add more VoLTE technology over the course of [the] last six months of 2015, but due to our rigorous scientific approach we do not switch out testing methodology for any carrier once our testing begins in a half.

Basically, for better or for worse, RootMetrics said it conducts its test with devices benchmarked at the beginning of a six-month testing period then sticks with that until the next report. "Switching our methodology midstream would render a scientific apples-to-apples comparison impossible," Dey wrote.

She noted that voice-over-LTE testing will be part of the first half of 2016 testing that is currently under way.  "We are taking a wait-and-see approach to what the impact may (or may not be) for T-Mobile," Dey added.

The RootMetrics VP also responded in a second email exchange with the Fool to Legere's charges that the other carriers pay the company. She made it clear that the surveys are done independently but wireless carriers are able to purchase access to the in-depth results.

At RootMetrics, we're dedicated to remaining completely and steadfastly independent when it comes to our methodology for testing and reporting mobile performance. We self-fund our studies entirely independently and out of our own pocket. Our 2nd Half 2015 report included information from nearly 3.9 million tests with mobile devices purchased off-the-shelf, while driving nearly 232,000 miles and randomly testing outdoors and indoors at 6,607 locations with a single mission in mind: to ensure that mobile users have an independent view on network performance in order to make informed mobile decisions. It's important to understand that absolutely anyone – in the industry or not – can purchase our highly detailed data and regardless of who does, RootMetrics is committed to ensuring our scientific testing is free of bias and there is absolutely no outside influence over our testing, analysis, and results. Our results are always completely unbiased and we stand behind them 100%.

That does not mean Verizon, Sprint, and AT&T in any combination are not RootMetrics customers. They may well be and they may spend millions as Legere claimed, but Dey is responding by laying out her case that the customer base does not dictate the results of the survey.

It's a matter of perception
Legere has an issue with testing methodology and RootMetrics has responded in a way that can politely be described as "We see your point, but that's how we chose to do it," or if you take the impolite interpretation, "Stuff it." The CEO is not wrong in pointing out the age of the data and the research firm does not dispute. It's simply how it chooses to do the report.

The same could be argued on the issue of bias. If Legere considers the methodology unfair, which he clearly does, then he may draw the conclusion, even though he doesn't present evidence to back it up, that the research firm has allowed itself to be influenced by its paying customers -- which he presumes to include the other wireless carriers (or at least some of them).

This is a dispute where both sides appear right. Legere has valid points, but so does RootMetrics. The real issue is how consumers, the media, and even investors interpret the results. If you look at them through Legere's lens, then you see something very different than what the survey says and how these results get reported.