The only big surprise in the fourth-quarter results of big-and-tall retailer Casual Male (NASDAQ:CMRG) was the plus-sized tax benefit it realized.

The company reported that sales in the quarter, which included an additional week, were $146 million compared to last year's $130 million. Profits, meanwhile, came in at $38.7 million, or $0.83 per share, while net income last year was just $13.5 million, or $0.33 per share.

While this sounds impressive, both periods enjoyed a reversal of deferred tax assets of $30 million and $3 million, respectively. After normalizing tax rates, it would have generated per-share profits of $0.20 and $0.17 per share, respectively, which was in line with what analysts had predicted as they exclude such one-time events. Perhaps it would be better to look at Casual Male's operating profits, which take interest and tax benefits out of the equation, and showed a near 13% increase over the year before.

As the business has been growing, management said it has been taking market share from the competition. Like Men's Warehouse (NYSE:MW) and Jos. A. Bank (NASDAQ:JOSB), Casual Male experienced a weak December primarily because of unseasonably warm weather. However, it was able to turn that around in January and report much stronger sales. Overall in the quarter, same-store sales (sales at stores open for at least a year) increased by 7.5%, which was from a particularly tough comparison of 7.9% comp growth the year before.

Sales in the core business of Casual Male XL and Casual Male Rochester have been expanding steadily, with comps up 9% for the year. The retailer now wants to take in a much larger portion of the market by targeting segments beyond its primary retail customer.

In 2006, the company acquired online plus-sized "lifestyle" retailer SuperSizeWorld.com, which offers products and accessories aimed at the big-and-tall customer. Casual Male originally planned to call it ThinkBig.com, but as the company has been reorganizing itself away from the "big-and-tall" name, it decided to create a more cohesive image and call it LivingXL.com. That's in line with the rebranding of its stores as Casual Male XL.

Another acquisition Casual Male expects to pay off down the road as it tries to dominate the plus-size category will be its Jared M. deal. The specialty retailer typically caters to athletes and entertainment personalities, and the average customer spends around $40,000 per year in the store; some drop as much as $200,000 annually. Where the proprietor of that business was stressed to his limits servicing his customers alone, Casual Male believes it will be able to expand the presence with its backing.

With operating margins growing, an expanding presence beyond traditional retail, and the opportunity to tap into a specialized clientele niche, Casual Male's growth prospects seem to be a supersized opportunity.

Looking for other smaller companies with outsized potential? Check out Motley Fool Hidden Gems, which selects the best small-cap prospects overlooked by the rest of Wall Street. A 30-day trial subscription gives you full access to all the market-beating recommendations.

Fool contributor Rich Duprey owns shares of Casual Male, but does not own any of the other stocks mentioned in this article. You can see his holdings here. The Motley Fool has a disclosure policy.