The first thing that most investors will probably notice about Redwood Trust (RWT 0.88%) is its huge 8.8% dividend yield. For investors searching out dividend stocks, that high of a yield is as irresistible as a flame to a moth. In the end, however, if you take the time to look at this real estate investment trust's (REIT's) history, you'll probably err on the side of caution and look elsewhere. Here's why.

Simple REITs and complicated ones

REIT stalwart Realty Income (O -0.17%) has a 5.5% dividend yield, which is fairly attractive. The average REIT's yield is around 4.9%, using the Vanguard Real Estate ETF as a proxy, and the S&P 500 index's yield is roughly 1.5%. Realty Income's yield also happens to be near its highest levels over the past decade, hinting that the stock is on sale right now.

O Chart

O data by YCharts

But the real reason to like Realty Income stock is how boring it is. The REIT basically buys physical properties and leases them out, which is exactly what you would do if you owned a rental property. This is a simple business to understand and the model has supported 29 consecutive years worth of annual dividend increases. That's a story that dividend investors could easily get behind.

But what about the much higher dividend yield that Redwood Trust is offering? On an absolute basis, it is roughly 3.3 percentage points larger, which equates to a 60% bump in the income you would collect. That's hard to ignore.

The problem is that Redwood Trust isn't as easy to understand as Realty Income. Redwood Trust is a mortgage REIT. It originates mortgages for single homes and apartments, owns a portfolio of mortgages, and bundles mortgages up into bond-like investments that it sells to others. This is not something that most people could do on their own. It would require a fair amount of effort to fully understand the company's three main business segments and even more to keep tabs on them over time.

The proof is in the dividend

For a big yield, though, some income investors might be willing to put in the extra legwork. Here's the problem: Redwood is more of a total return investment than a dividend stock. If you go back to the turn of the century, the stock is down roughly 40%. But the total return, which assumes the reinvestment of dividends, is up 430%. The high yield is a key part of the equation here.

RWT Chart

RWT data by YCharts

Basically, a big dividend allows for more shares to be purchased while the stock price is low. For investors thinking about total return and asset allocation Redwood Trust might be a good choice for adding mortgage exposure to an otherwise diversified portfolio. But this is mostly going to be big investors like insurance companies and pension funds. If you are looking to eventually live off of the passive income your portfolio generates, history suggests that Redwood Trust will be a bad choice.

RWT Chart

RWT data by YCharts

Look at the dividend history above (the orange line): It is volatile at best with multiple dividend cuts. The stock price (the purple line) has basically followed the dividend lower over time. That's quite normal for a mortgage REIT, but it is a terrible outcome for a dividend investor who is trying to supplement Social Security in retirement. You have less income and less capital!

Lower-yielding, boring REITs are better

For most investors, finding a reliable dividend-paying REIT like Realty Income, despite its more modest yield, will be a better choice than chasing yield with a more complicated REIT like Redwood Trust. That's not to suggest the mortgage REIT is a bad company, it just isn't designed to provide a reliable income stream over time. If you are tempted by Redwood Trust's fat yield, you should probably step back and consider a safer, more reliable option.