The new energy boom in the U.S. has captured the attention of the entire world, with massive new discoveries and production increases having had a sizable impact both on long-term prospects for oil stocks as well as on the entire supply and demand relationship for oil and gas across the globe. Yet for investors, oil stocks haven't performed nearly as well as that boom might suggest.

Looking back over the past year, a simple S&P 500 index fund has outperformed many key oil stocks, both domestic and foreign. In particular, ExxonMobil (NYSE:XOM) has delivered less than half the gains of the S&P, while British giant BP (NYSE:BP) has been stuck right at the halfway point. Most of the biggest oil stocks in the U.S. and abroad have delivered only single-digit-percentage gains so far this year, compared to the S&P's double-digit rise. Let's take a look at what's behind the underperformance in oil stocks and whether the industry is poised to recover in the near future.

It's not easy being big
When most people think about the energy boom, they tend to focus on the small exploration and production companies that have gone from being unknown players in the industry to well-known names focusing on key geographical areas with huge production potential. For instance, the Bakken play in North Dakota has turned the state into the highest-growth area in the nation, and Kodiak Oil & Gas (NYSE:KOG) and several other tiny exploration and production companies have ridden the wave of success in boosting their production levels and reaping the Bakken's benefits. Kodiak in particular has leveraged its future success to the Bakken, with plans to spend all of its capital budget in the region.

For the largest oil companies, however, the story is much different. For them, even new discoveries that have opened up opportunities for oil fields that were thought to be entirely played out are relatively insignificant compared to the massive amounts of oil and gas that they currently produce. As a result, big oil stocks have to rely on innovation and asset purchases just to stay even with natural declines in existing-well output. During the first quarter, Exxon reported a 3.5% drop in production compared to the year-ago period, while France's Total (NYSE:TOT) saw 2% production declines.

Moreover, the success of oil companies in replacing reserves has been mixed. Exxon and Chevron (NYSE:CVX) have both seen reserves rise more than enough to offset their production, but Total and Royal Dutch Shell have fallen short, with reserve replacement ratios in the 85%-95% range. Competition among major oil companies to replace depleted fields with new opportunities has driven up asset prices, narrowing margins and making oil stocks look less attractive.

The coming margin crunch
In particular, the quest for more production has pushed energy companies outside their traditional comfort zone, with exploration now centering on high-cost, high-reward unconventional areas like offshore ultra-deepwater fields, Canada's oil sands, and the lucrative shale plays that have popped up around the world. All of these methods allow oil companies to claim gains on their reserves, but actually producing oil and gas from them involves much greater amounts of investment and requires energy prices to remain higher in order to be profitable.

Again, small companies have an easier job in building up assets, as they can move the needle with pure plays on lucrative areas. The long-term challenge that big oil companies face in achieving the same success poses a greater risk, and that's a big part of why oil stocks have suffered lately.

Will oil stocks become leaders again?
With production being such a limiting factor for oil companies, you can expect energy prices to be the primary factor in determining profits. A sustained move for oil above $100 and a continuing recovery in natural gas prices could push oil stocks higher, but with higher supplies of energy products and continuing new discoveries, such a move looks unlikely in the short run.

Fool contributor Dan Caplinger has no position in any stocks mentioned. You can follow him on Twitter @DanCaplinger. The Motley Fool recommends Chevron and Total. Try any of our Foolish newsletter services free for 30 days. We Fools may not all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors. The Motley Fool has a disclosure policy.