Dividends are a key feature for mortgage REITs. Image source: Getty Images.

Real estate investment trusts have been a go-to staple for income investors for years because of the lucrative dividend distributions they pay. Within the REIT sector, mortgage REITs are especially generous in returning capital to shareholders, and Annaly Capital Management (NLY 0.55%) and American Capital Agency (AGNC 0.55%) are the giants in the industry. With their double-digit yields and a favorable interest rate environment, both mortgage REITs have produced extremely strong returns, and investors want to know which one looks like a better pick for an income portfolio.

Let's look more closely at Annaly and American Capital Agency using a range of common metrics to evaluate their merits.

Stock performance and valuation

Mortgage REITs have been seen as vulnerable to rising interest rates, but the Federal Reserve has thus far failed to pull the trigger on rate hikes, and that has helped support share prices in the industry over the past year. Annaly is up a whopping 27% since June 2015, while American Capital Agency has gained a still-impressive 17% over the same timeframe.

Price-to-earnings ratios aren't generally the best measure of valuation for real estate investment trusts, and many investors instead focus on the relative value of the shares compared to the REIT's book value. Based on current share prices and the most recently available book value figures for the two REITs, American Capital Agency looks slightly cheaper. American Capital Agency shares trade at about 88% of book value, while Annaly Capital has a slightly smaller discount to book, trading at 94% of book value. The fact that both REITs have a discount to their book values shows the concern among investors about the interest rate environment, but American Capital carries a slightly more favorable discounted value.

NLY Total Return Price Chart

NLY Total Return Price data by YCharts.

Dividends

People buy mortgage REITs because of their yields, and both Annaly and American Capital Agency meet the test on the dividend front. Based on its most recent payout, Annaly's dividend yield is around 11%; American Capital comes with an even higher 12.3% yield.

However, it's important to put each REIT's dividends in the context of where its payouts were in the past. Between late 2011 and the end of 2013, American Capital Agency cuts its dividend by more than half. Since then, though, it has done a good job of keeping its payout relatively stable, with just a single cut. Similarly, Annaly reduced its quarterly payout from $0.60 per share to $0.30 over the same timeframe, but it has kept its dividend stable since then.

NLY Dividend Chart

NLY Dividend data by YCharts.

Some investors might prefer American Capital Agency for another reason: It made the shift to pay dividends on a monthly basis. With its higher payout and fairly similar dividend history to Annaly Capital, American Capital Agency gets the slight nod on the payout front.

Growth prospects and risk

Mortgage REITs grow by acquiring assets, and they carry risk from their leveraged portfolios. Annaly Capital has given investors a textbook example of this growth strategy in recent months, announcing it would purchase Hatteras Financial in a cash and stock deal that prices Hatteras at about $1.5 billion. The deal gives Hatteras investors a choice to elect between cash, stock, or a combination of both, but overall, Annaly will pay for the deal 65% in stock and 35% in cash. As Annaly sees it, the move will expand and diversify its portfolio to include a wider array of mortgage-backed securities, residential whole loans, and mortgage servicing rights. It will also cement Annaly's leadership position in the mortgage REIT industry, and it bolsters Annaly's liquidity position in the industry. Annaly's current debt to common equity ratio of about 6 is fairly consistent with the company's long-term trends.

For American Capital Agency, a slightly higher debt to common equity ratio of almost 7.5 indicates slightly heightened risk levels. Interestingly, though, it's going through a slightly different transaction, seeing its manager American Capital purchased by rival Ares Capital. To help facilitate the deal, American Capital Agency is buying the portion of American Capital's business that represents the mortgage REIT's own external manager, called American Capital Mortgage Management, for $562 million. As a result of the deal, American Capital Agency hopes to keep its management team available to it, and CEO Gary Kain argues the transaction will result in "substantial cost savings that will be realized over time from the elimination of the management fee." Such moves could give American Capital Agency a competitive edge over REITs that have external managers.

Both American Capital Agency and Annaly Capital face the same industry conditions, and they look fairly similar even as both undergo substantial changes. Currently, American Capital Agency looks like a slightly better buy, but both are likely to rise and fall with conditions in the interest rate market for the rest of this year and beyond.