For two decades, Novell (NASDAQ:NOVL) has been trying find new growth markets in enterprise software. Unfortunately, its latest attempt has failed. Fed up with the lack of growth, Novell's board staged a palace coup -- throwing out Jack Messman, the CEO, and Joseph Tibbetts, the CFO. Novell's president, Ronald Hovsepian, will replace Messman.

Simply put, Novell's performance has been lackluster, especially the first-quarter results. Revenues declined from $290 million to $274 million. As for net income, it was only $2 million or break-even per share. The company also issued lower guidance.

The second quarter was also weak. Revenues fell from $297 million to $278 million. Novell produced net income of $3 million or $0.01 per share. Since May, the stock had lost a quarter of its value before the news of the coup came out.

Yes, it's hard to believe that Novell was once a stellar growth company. Back in the 1980s, its flagship product, NetWare, was the standard operating system for computer networks. This was in contrast to Microsoft's (NASDAQ:MSFT) DOS system, which was focused on stand-alone computers.

Of course, Microsoft eventually encroached on Novell's business. As a result, Novell made attempts to diversify into other market segments, such as buying WordPerfect and the Quattro Pro spreadsheet. A few years later, Novell sold the products -- at a massive loss -- to Corel. In 1997, Google's (NASDAQ:GOOG) current CEO, Eric Schmidt, became the CEO of Novell, and he launched Internet-based products to help reinvigorate growth. Yet again, it was to no avail.

Then, a few years ago, Novell made another attempt to remake itself. This time it became an open-source software developer, a feat accomplished with a variety of acquisitions, such as Ximian and SUSE. Interestingly enough, IBM (NYSE:IBM) helped finance the acquisition of SUSE.

Open source essentially is a New-Age software model. That is, the software code is made available to anyone for free (downloadable from the Internet). Programmers around the globe can develop the software. For example, open source products like Linux and MySQL are becoming serious alternatives to software from biggies like Microsoft, Oracle (NYSE:ORCL), and others.

The problem for Novell is the weak execution of its open source strategy. I talked to Ilan Seyahek, the co-founder and CTO of Jitterbit, an open source company. He commented that "First, SuSE added no real value beyond the popular Linux incumbents, so we could not develop and test on it as the distribution of choice. Also, Novell does not project itself as a company with the DNA to support communities of developers. Novell is great for Network Support Engineers, who are very different from developers."

With Novell's NetWare business ebbing -- and its open source initiatives failing to get traction -- it's hard to see where the company will find growth. The new CEO, who has 17 years of experience at IBM, can take the short-run moves of cutting costs and perhaps ditching the NetWare business. But as for becoming a viable player in open source, that is likely to come much later -- if ever.

The company could make some short-term moves to improve its profitability, like cutting excess headcount, and the stock upswing seems to reflect these short-term moves. But again, the top line has been weak for the company -- not something that can be solved overnight. Given the intense competition (in fact, Oracle has hinted it will start its own Linux business) and upheaval at Novell, the company has a lot of challenges ahead.

Microsoft is a Motley Fool Inside Value recommendation. To find more top-shelf stocks at bargain-bin prices, sign up today for a free 30-day guest pass .

Fool contributor Tom Taulli does not own shares mentioned in this article.