FBR Group (NYSE:FBR) has had a confusing structure. It's a real estate investment trust (REIT) that also has an investment bank. So late last week, FBR had a spin-off of the FBR Capital Markets (NASDAQ:FBCM) division. The stock traded up 5.88% on its first day of trading, but investors may be concerned that things are getting crowded.

About 72% of FBR Capital's business comes from capital raising and advisory work. Looking back at 2006, the firm raised about $10.9 billion in equity for clients and participated in 28 M&A deals. A nice boost came from FBR Capital's acquisition of investment bank Legacy Partners Group. The deal added more than two dozen bankers and expanded FBR Capital's presence in the consumer and health-care sectors.

FBR Capital also wants a piece of the bonanza in hedge funds and private equity deals. So last year, private equity firm Crestview purchased a 12% stake in FBR Markets. The firm has about $1.5 billion under management and a management team with extensive experience at Goldman Sachs.

FBR Capital's prospectus says it has an advantage because the big players like Morgan Stanley (NYSE:MS), Goldman, and Lehman Brothers focus mainly on larger market-cap deals. While this is true, FBR still faces lots of competition from boutiques such as KBW (NYSE:KBW), Evercore (NYSE:EVR), Cowen (NASDAQ:COWN), Thomas Weisel (NASDAQ:TWPG), and Lazard.

So long as the financial markets continue to grow, the competition should not be a problem. There are more than enough deals to go around, and FBR has a strong platform. Yet whenever there are a lot of IPOs in a sector, Foolish investors should be cautious, and the same should go for boutique investment banks.

Further Foolishness:

Fool contributor Tom Taulli, author of The Complete M&A Handbook, does not own shares mentioned in this article. He is currently ranked 2,149 out of 29,896 rated investors in Motley Fool CAPS.