Verizon (VZ 0.11%) has made network quality the cornerstone of its advertising campaign.
The company uses the slogan "Better matters," while running ads that show how it topped the most recent RootMetrics Mobile Network Performance in the United States report. It did top the study, but that's not the entire story, given that the bottom-ranked wireless companies -- T-Mobile (TMUS +1.42%) and Sprint (S +0.00%) -- have dramatically cut into Verizon's lead.
If you use RootMetrics as the sole standard, then Verizon is still winning, albeit by a smaller margin than in the past, but if you look at other detailed studies, specifically OpenSignal's January 2016 State of Mobile Networks report then a different picture emerges.
Verizon still ranks at or near the top across most categories, but it's being pushed for the lead, and in some cases topped by T-Mobile.
A tale of two studies
RootMetrics and OpenSignal use different metrics and operate on different timetables. Both are very comprehensive tests, but they aren't merely two takes on the same operation which explains why they don't come to the same conclusions.
RootMetrics uses drive-testing. The research firm literally has testers driving all around the country taking snapshots of network performance. For its first-half-of-2015 report, the company conducted more than 6.1 million tests, driving 237,506 miles and testing 7,323 indoor locations. The research company argues on its FAQ page that it's method is superior to the real-user testing OpenSignal uses:
Test results from actual users are certainly "real," but they are simply not a good or fair way to compare performance among the national carrier. Through our scientific testing methodology, we are able to test all the networks at the same time and place using actual consumer devices that are benchmarked and quality controlled to show each of the networks consistently in the best light possible. Other anecdotal "crowd-sourced" approaches that rely on a "grab bag" of random user tests at different times or places for the carriers are neither fair nor a basis for network comparisons.
OpenSignal disagrees and argue the merits of its testing method in a email to Fool from CEO Brendan Gill:
OpenSignal tests mobile networks using millions of real world users testing networks 24/7 under normal conditions where as drive-testing is a one-off snapshot that uses test equipment to 'simulate' the network experience. It's generally limited to a much smaller number of locations and due to the difficulty in collecting a lot of data this way it often relies on out of date information (e.g. the Verizon commercial is based on data collected up to 12 months prior. The OpenSignal report is based on data collected from the last three months of 2015 so represents a much more up to date picture of network conditions).
It's very clear that both companies are taking shots at the methodology used by its rival, but each has a valid point. Because of that it's reasonable to say that both reports have merit while neither presents a definitive picture for consumers. This is very much measuring a moving target and no report or survey will be completely definitive.
To put it simply, both reports could come with a warning that says "your results may vary."
What does OpenSignal say?
OpenSignal's latest report shows that Verizon remains a top carrier, but its lead over T-Mobile is sketchy at best, which is a very recent development. In the study, you could argue that T-Mobile and Verizon sit on top where the difference depends upon geography and what you use your smartphone for. The big loser here is AT&T (T 1.25%), the clear No. 2 for RootMetrics, which scores no wins here.
Source: OpenSignal.
It's worth noting that OpenSignal offers a more complicated picture than RootMetrics, and no clear winner is declared. Verizon, for example, pulls off a slight win in Coverage 4G, which is defined as "the proportion of time LTE subscribers on each network have 4G (LTE) coverage available to them," while T-Mobile dominated 3G download speeds, and the two companies were very close in other metrics.
OpenSignal offers a very nuanced look at what consumers are experiencing. It echoes RootMetrics' findings that T-Mobile has made big strides, and it shows that the difference between the major carriers has become smaller over time. It's not clear after reading the OpenSignal results that any one carrier is "better" for every consumer.
Verizon has a coverage lead, but it's narrowing, and T-Mobile outperforms it in the markets where both are strong. This suggests that despite what Verizon says in its ads, network quality is no longer a major differentiator for consumers. Better may matter, but better is no longer unique to Verizon. T-Mobile has improved and given that it offers lower prices that should allow the upstart No. 3 carrier to continue adding subscribers.
