After being shunned for most of 2022, the mortgage real estate investment trust (REIT) sector regained some popularity as investors sensed the Federal Reserve's tightening cycle is wrapping up, which will remove a major headwind for the sector.

The two biggest mortgage REITs, AGNC Investment (AGNC 0.99%) and Annaly Capital Management (NLY 1.33%), have mounted quite the rally over the past several months. Are these two REITs still buys, and if so, which one is the better buy? 

A pile of coins, a miniature house, and a calculator.

Image source: Getty Images.

Mortgage REITs are different from traditional REITs

Most traditional REITs develop properties and then lease them out to tenants. The idea is that rental income will cover the interest costs on the debt the REIT used to finance the property. The property developments could be anything from shopping malls, to cellphone towers, to data centers.

Mortgage REITs don't invest in real property. They invest in real estate debt (i.e., mortgages). Mortgage REITs will use borrowed money to finance the purchase of mortgage-backed securities. This is similar to using margin to buy stocks. Agency mortgage REITs invest in mortgage-backed securities, which are guaranteed by the U.S. government. Non-agency REITs invest in mortgages that are not guaranteed. For a mortgage REIT, the difference between its cost of borrowing and the amount it earns on its mortgage portfolio drives its returns. In that way, it is very similar to a bank. 

Mortgage REITs performed abysmally for most of 2022, but in the final quarter of the year, they had a furious rally. You can see the chart of recent performance below:

AGNC Chart

AGNC data by YCharts

The driver for the year-end rally was the perception that the Fed is ready to pivot from a hawkish to a neutral posture. Another major factor is that mortgage-backed securities began outperforming Treasuries after a long stretch of underperformance. Even with this recent performance upsurge, these stocks are still trading down 21% to 25% on a year-over-year basis. Both companies still have dividend yields in the low- to mid-teen percentages. 

AGNC Investment has extremely limited credit risk

AGNC Investment is the purest agency mortgage REIT out there. It invests almost exclusively in government-guaranteed mortgage-backed securities. This means AGNC takes almost no credit risk. If the borrower on one of the underlying mortgages stops paying, the government will ensure that AGNC gets all of its scheduled principal and interest payments. The trade-off for no credit risk is that AGNC needs to use a lot of leverage, which means it has heavy interest rate risk and is vulnerable if mortgage-backed securities underperform its hedges. 

For AGNC Investment, the story is all about MBS's outperformance and underperformance. MBSs underperformed Treasuries last year such that the difference in yield between MBSs and Treasuries (called MBS spreads) reached levels last seen during the 2008 financial crisis. There were a lot of technical factors driving this underperformance, but fears that the Fed would sell its vast holdings of mortgage-backed securities into the market were a big one. That said, selling $2.7 trillion worth of MBSs into the market is easier said than done, and the Fed probably isn't seriously contemplating this. 

Annaly has higher risk ... and a higher yield

Annaly Capital invests in agency mortgage-backed securities, mortgage servicing rights and other non-government-guaranteed mortgage assets. Agency mortgage-backed securities represent 91% of Annaly's assets so its portfolio is pretty similar to AGNC Investment. The mortgage servicing rights are an unusual asset in that they increase in value as interest rates rise. The mortgage servicer handles the administrative tasks of the mortgage on behalf of the investor, collecting payments, forwarding interest and principal, ensuring property taxes are paid, and handling defaults. The servicer gets 0.25% of the mortgage as a fee every year. The right to perform this service is worth something, and it is capitalized as an asset. The non-guaranteed mortgages are largely loans that don't fit into the government's approved credit box and are therefore ineligible for a government guarantee. These loans are nothing like the subprime loans of yesteryear -- they are conservative loans made primarily to professional real estate investors. 

A recession is a bigger risk for Annaly

Given that Annaly and AGNC's portfolios are similar, the big driver as to which is the better buy will be whether the economy enters a recession and whether mortgage-backed security spreads continue to normalize or start increasing again.

If MBS spreads begin to increase, then the "best buy" is neither stock. Both will give back their recent outperformance and will be vulnerable to dividend cuts. Assuming MBS spreads continue to normalize, the current environment will be favorable for both stocks.

If the economy enters a recession, Annaly will see increased costs on its servicing portfolio and could see mortgage servicing rights valuations decrease as rates fall and costs increase. A recession will also increase the likelihood of credit losses on its residential credit portfolio. Annaly has a dividend yield of 15.5% versus AGNC, which has a yield of 12.5%. Both are trading at massive premiums to third-quarter book value per share. Ordinarily, mortgage REITs trade at book or below, so the market is already pricing in good numbers for the fourth quarter.

The bottom line is Annaly is riskier, but you are getting an extra 3 percentage points in dividend yield for taking that risk. That said, housing is in for a rough 2023 and AGNC Investment simply doesn't have that exposure. I would invest in AGNC Investment if it came to a choice between the two.