Paying $99 a year for an Amazon (AMZN -2.56%) Prime membership gets people into the special club the giant online retailer caters to.

It may not be a very exclusive group given that Consumer Intelligence Research Partners (CIRP) estimates that the company closed 2015 with 54 million members in the United States alone, but the perks are tremendous. In addition to video and music services, which together might be worth the price of admission, Amazon also offers Prime members free two-day shipping on millions of items.

Joining lets users of the online store buy whenever they want without having to consider whether their order hits the $35 threshold, where free shipping kicks in for everyone. It means that whether you're out of tea, which weighs a few ounces, or need a 20-pound bag of kitty litter, you can order it on and have it within two days (including Sunday, in many cases).

This has been good for Amazon given that CIRP estimated that U.S. Prime members spend an average of about $1,100 per year, compared to about $600 per year for non-members. It has also been good for customers, especially the ones who use it often enough to actually save money on shipping (about 25 times a year).

It's easy to see how Prime helps the company, and the benefits its members get, but new research shows that if you don't pay for the service, getting your order may take longer than it used to.

What does the research say?
StellaService, a company that measures the customer service performance of online retailers, found that customers who do not pay for Prime may be getting slower delivery, Fortune reported.

"For a long time, Amazon has been the leader when it comes to fulfillment," Stella Service VP Kevon Hills told the financial publication. "There are 40 companies on our list, and Amazon has always been in the top 10. This year, we've seen them fall outside of the top 10, which is the first time we've seen that happen."

Hills believes that's happening for two reasons. First, he said Amazon has put the majority of its efforts into ensuring it hits the two-day window for Prime members. Second, he believes the company's overall standing has been hurt because rivals have raised the bar.

"Three years ago, if you delivered a package in four days, that was worthy of a top ten consideration," Hills said. "Now, if you deliver a package in four days, that puts you at 20. The industry is getting faster, and we've seen Amazon fall off a little bit."

How does shipping work for non-Prime members?
Amazon offers free shipping for qualifying orders of over $35 to anyone, but the no-cost offer is not for two-day delivery, but what the retailer calls "free shipping" with a 5-8-day delivery window. On a test order I placed using an account created without Amazon Prime, buying $38.94 in yoga supplies would cost me $15.61 in shipping if I want it in two days. I could have also elected to split the difference and chosen "standard shipping," which takes 4-5 days for an extra $9.23.

Amazon shows non-Prime members what they would save on shipping if they joined the service. Image source: Amazon.

Can this hurt Amazon?
It does make sense for Amazon to make sure it hits the two-day window for Prime members, but its 5-8-day free delivery window for non-members does put it at risk to lose customers to low-cost rivals like Wal-Mart (WMT 0.46%). That physical retail giant has been investing heavily in its website and digital products. currently offers what it calls "free vaue shipping" on orders over $50. That's higher than Amazon's $35 threshold, but the brick and mortar company's free shipping only takes 4-6 days. 

That's still slow, but it's obviously sometimes better than the 5-8 day window Amazon quotes non-Prime members if they want free delivery. That, plus the fact that Wal-Mart takes online returns in its stores, might tip non-Prime members into shopping at Wal-Mart's website or app.

What should Amazon do?
Clearly, Prime members should be a priority for the online retail giant, but the company has to be very careful in how it treats non-members. Using improved delivery speeds as a carrot to get people to join makes sense, but it could also force potential regular customers to go to whatever rival offers the best combination of price and shipping speed.

The last thing Amazon wants is to create a class of customers that checks prices at Pushing and supporting Prime has helped the online retailer build a very loyal audience. Continuing that make sense, but the company should also throw some love toward loyal non-members by making sure it at least matches offers like Wal-Mart's.