Chipotle was a drag on the entire fast casual segment. Image source: Chipotle.

Blame Chipotle (CMG -1.89%), but also blame high prices and wariness over the economy for why consumers have been more cautious about eating out.

Overall, restaurant traffic was flat in the first quarter of 2016 and it stayed the same in the second, according to NPD Group. The news was especially bad in the fast-casual segment, which had been growing explosively at this time last year.

Fast-casual traffic was flat in Q1 compared to same quarter a year ago, down 3% in Q2. You can blame a lot of that on Chipotle's food-safety woes, but even taking the Mexican chain out of the equation only puts Q2 growth at 2%, down from 11% growth in 2015.

"Contributing to the stalled visit growth are consumers' uncertainties about current and future economic conditions," says Bonnie Riggs, NPD Group's restaurant industry analyst in a press release.  

These uncertainties have put a damper on overall consumer spending. Compounding the situation for the restaurant industry is the decline in food at home inflation while at the same time restaurant operators have been increasing menu prices.

Consumer concerns over pricing hit fast-casual chains like Chipotle and Panera Bread (PNRA) especially hard because they have built their business on offering higher-quality food that's more expensive than their fast-food rivals. Fast-casual chains, because they use better ingredients, also have higher costs that impacts their ability to offer deals during slower periods.

People are eating lunch out less

In general, the biggest problem for restaurants is that people are less willing to eat lunch out. The midday meal represents about 3% of all daypart visits, according to NPD.

In Q2, lunch traffic fell by 4% compared to the same quarter last year while dinner, which accounts for about 30% of daypart visits, only dropped by 1%, an improvement over the 3% decline in the first quarter of this year. Breakfast traffic grew by 1% in Q2.

The slowing lunch business can be directly linked to pricing and economic concerns. "It's political, concern about the economy. It's uncertainty, food safety, social unrest," Riggs told Nation's Restaurant News (NRN). "But probably the biggest thing is sticker shock."

NRN noted that of the total negative traffic volume at lunch, 12% came from fast casual, double its overall market share, while the average check jumped by 5%, to $8.36, the highest of any other segment.

Restaurants are not dying

For fast casual, specifically chains like Chipotle and Panera -- which market around having better quality and more natural foods (which cost more) -- economic uncertainty will always be a problem. If people are worried about their jobs or the economy, they may not be willing to pay more for pork from a pig raised down the street or "clean" bacon.

Worries about money will push people away from pricier choices, seemingly especially at lunch, but Riggs was clear that NPD's research had some positives as well.

"The good news in all of this is that consumers made 61.3 billion restaurant visits this past year," she said. "They are not giving up dining out at restaurants and other foodservice outlets. It's true that in this flat market it's a battle for visit share but there are restaurants that are winning. The winning operators focus on their customers' needs and deliver on them."