The majority of stocks fall in a gray area. They're not necessarily good or bad, just right for some investors and wrong for others. Annaly Capital Management (NLY 0.55%) and American Capital Agency (AGNC 0.55%) are great examples of this.

For instance, Annaly's 10.4% yield and American Capital Agency's 11.4% yield are huge selling points. However, for investors who bought -- or are thinking about buying -- in hopes of earning a stable and reliable dividend, it might be time to sell. 

3. Unstable dividend

The two mortgage real estate investment trusts' lack of dividend stability has everything to do with their business model. Both companies borrow based on short-term interest rates and buy long-term residential housing debt. They earn the difference between the two.

NLY Dividend Chart

As you can see in the chart above, when the gap in yield between the two-year Treasury and the 10-year Treasury widens, Annaly's earnings and dividend increase. The opposite is true when spreads narrow. The catch is that interest rates are highly unpredictable. and therefore so is the dividend from Annaly and American Capital Agency.

2. They're getting riskier
While neither company's dividend has proven reliable, that does not mean their business is unsustainable. In fact, their chief asset class -- packaged residential mortgage debt -- is guaranteed against default by Fannie Mae and Freddie Mac

American Capital Agency did not go public until May 2008, but holding safer assets is one major reason Annaly survived the housing bubble in 2006 and 2007 while similar companies holding riskier assets got hammered.  

But times are changing. With the acquisition of CreXus in January 2013, Annaly has been buying commercial real estate and commercial loans. The portfolio is small, only 12% of equity, but it exposes Annaly to the risk of losing principal, or credit risk. 

Moreover, Fannie Mae and Freddie Mac have designed investments to dump some of their credit risk from the roughly $5 trillion worth of mortgage loans they guarantee onto private capital. In the companies' third-quarter conference calls, Annaly suggested these deals are on its radar, while American Capital Agency President Gary Kain said the high quality of these investments isn't getting enough attention. 

Lastly, during American Capital Agency's investor day in October, management suggested it could invest in nonagency securities. Though very similar to what the company currently holds, nonagencies are not protected against default. 

With interest rates near all-time lows, it's not surprising to see Annaly and American Capital Agency thinking outside the box; in fact, I would be disappointed if they weren't. But looking forward, all signs point toward both companies taking on riskier assets.

1. Better options
If an unreliable dividend and a potentially riskier portfolio doesn't have you running for the hills, then the best reason to sell is because there are better opportunities.

NLY 3 Year Total Returns (Daily) Chart

Like Annaly and American Capital Agency, Two Harbors (TWO 1.48%) is a mortgage REIT, it has a 9.4% yield, and it invests in similar assets. Its total return, however, has smoked those of Annaly and American Capital Agency over the last three years. 

This is due to Two Harbors' greater diversity of assets. Following the financial crisis ,the company loaded its portfolio with subprime mortgages -- risky loans to borrowers with iffy credit -- it bought at dirt cheap prices, and it made a killing as the housing market rebounded. 

During American Capital Agency's investor day, the company mentioned its interest in excess mortgage servicing rights, loan origination, and nonagency securities -- all three of which Two Harbors is already involved in. 

For me, if Annaly and American Capital Agency are interested in expanding beyond their bread and butter, why not invest in a company such as Two Harbors that has already proven its ability to successfully invest in more credit-sensitive assets?

Ultimately, there is an ocean of interesting and potentially lucrative high-yielding stocks out there, and Annaly and American Capital Agency are only the tip of the iceberg.