I'm always canvassing the market for investment ideas, and looking at leaders in a respective sector is a good place to start. In the market for auctioning high-end collectibles and fine art, Sotheby's (NYSE:BID) and Christie's collectively dominate the landscape. Christie's can be quickly eliminated, as it is private, leaving minor players, including Escala (NASDAQ:ESCL), Collectors Universe (NASDAQ:CLCT), or Ritchie Bros. Auctioneers (NYSE:RBA). Unfortunately, none has the name recognition or compelling business models of the major auctioneers. That leaves Sotheby's -- but after some initial due diligence, I'm finding the company's investment merits a bit abstract.

Painting Sotheby's
Sotheby's generates the majority of its revenue from the auctioning of property, fine and decorative art, jewelry, and collectibles. My initial expectation was that this business would be highly profitable, akin to eBay's (NYSE:EBAY) ability to reap a cut of the many transactions on its website. Sotheby's has higher fixed costs than its cyberspace counterpart, but its operating margin was still decent, at slightly over 30% for the last 12 months, well above the 20.4% average for the market. Thanks in part to significant interest expense, net margins of 16% ran just slightly ahead of the market average of 13.7%. Not bad overall, and Sotheby's margins ran very close to eBay's over the past year, although eBay is growing faster and has a better longer-term track record.

In terms of track record, Sotheby's doesn't paint a very steady picture, as sales have only grown 5.2% on average over the past five years. Meanwhile, net income and cash flow trends look more like a Picasso painting than a modern piece, with consistent, visible, straight-line trends. That's somewhat understandable; Sotheby's and Christie's had a rough couple years before 2003, as they were accused of colluding to fix auction commission fees back in 2000. Plus, the art and collectible market has a reputation for being highly cyclical and subject to the whims of economic fluctuations and the tastes of fickle high-end collectors and art dealers.

A frothy market?
Fortunately, the art market is in the midst of a three-year boom, with record-setting auction activity and prices being paid for rare works of art. Just last week, a Christie's auction generated nearly $500 million, while Sotheby's held its own Impressionist and Modern Art event in New York, with an estimated $297 million in activity, highlighted by the sale of Paul Cezanne's Nature Morte aux Fruits for $37 million.

This boom is doing wonders for Sotheby's stock chart, as it is trading near a five-year high of $31.75. The stock did fall over 6% Friday after third-quarter results came in below expectations, but the first and third quarters are seasonal lows in the art market. Annual fluctuations aside, the million-dollar question is how long frothy art market conditions might continue.

The smart money weighs in
Fellow fool Stephen Ellis recently highlighted the news that a well-respected value shop sold its entire stake in Sotheby's, while another shareholder significantly pared its holdings. This leads me to believe that now may not be the best time to enter the bidding for some shares. And while debt to capital levels have come down significantly as management repurchased a big chunk of its stock last year, the ratio is still high, at just under 50%.

Fool's final word
Overall, I'm still intrigued by Sotheby's auctioneering prowess and believe its brand name and industry leadership allow for a decent economic moat and the ability to continue generating solid profitability. But right now, I'd recommend waiting for a more compelling entry point into the stock. After all, the 2000 scandal is still in the back of investors' minds.

Art-market proponents may argue that we have reached a new paradigm, as wealth levels skyrocket for high-end Russian, Chinese, and emerging market collectors. But the more likely outcome is that the industry will sputter after becoming overheated, as has happened time and time again: Just look at the dot-com, housing, and commodity investment realms for proof.

For related Foolishness:

eBay is a Motley Fool Stock Advisor pick. Take the newsletter dedicated to the very best of David and Tom Gardner's selections for a 30-day free spin .

Fool contributor Ryan Fuhrmann has no financial interest in any company mentioned. Feel free to email him with feedback or to discuss any companies mentioned further. The Fool has an ironclad disclosure policy.