When it comes to your money, you need to be able to trust the people you work with. But unless your broker comes to work every day with a "Bernie Madoff Is My Hero" button on his jacket, it's not always easy to separate the shysters from the many reputable professionals out there. How can you tell whether your advisor has your best interests at heart -- or is taking advantage of you?

Later in this article, I'll point to three warning signs to watch out for in your relationship with your broker or financial advisor. But first, let's take a quick look at why the financial advice industry is so fraught with peril.

A broken business model?
 Perhaps the biggest difference between financial advisors and most other professionals you work with is evident in how they get paid. If your house needs a new roof, your car's not running right, or you need your lawyer to draft your family's wills, the transaction couldn't be simpler: The professionals you hire do their work, and you write them a check.

But with financial advice, someone got the bright idea that if customers never had to write a check to hire an advisor, they'd be more likely to use one. So with the cooperation of financial services providers like insurance companies and mutual fund managers, financial advisors figured out many ways to get compensated for their work on the back end -- in some cases without investors ever having to see a penny's worth of fees actually come out of their accounts.

What to watch out for
 So, if advisors don't explicitly tell you how much you end up paying them, then it's up to you to figure it out for yourself. Here are some things for you to watch out for:

1. Load mutual funds 
 One common way that advisors get paid is by selling mutual funds that charge sales loads. Loads come in two main varieties: upfront and deferred. With an upfront load, a portion of the money you invest gets funneled off to pay your advisor and related expenses, leaving you with a reduced balance from the start. A deferred load, on the other hand, lets you invest the full amount you commit to the fund. But if you sell the fund within a set period of time -- typically five to seven years -- then you'll have to pay a sales charge, which will get deducted from the proceeds.

Load funds are big business, with American Funds, Franklin Resources' (NYSE: BEN) Franklin Templeton funds, and AllianceBernstein (NYSE: AB) all attracting attention from advisors by making sure they get compensated for selling the funds. Some of these funds have reasonably good track records, but if the only funds your advisor ever recommends are load funds, it could indicate a conflict of interest. Also, companies like Ameriprise (NYSE: AMP), whose in-house advisors have their own proprietary Columbia Management funds to sell, can give advisors incentives to target certain funds over others that may be better for you.

2. Unnecessary annuities
 Annuities are controversial in the financial world because they carry high commissions for advisors who sell them, and most come with fees that are higher even than mutual funds. In addition, surrender fees can cost you a big chunk of your investment if you need your money back sooner than you expected -- with those surrender fees often going to reimburse annuity sellers Hartford Financial (NYSE: HIG), Genworth Financial (NYSE: GNW), and many of their peers for the commissions they paid to the selling agent.

Annuities have legitimate purposes, though, including tax deferral, certain guaranteed benefits, and the right to convert the annuity into a stream of lifetime income payments. But if your advisor has you buy annuities even though you don't get much benefit from those features, then it's time to ask some tough questions.

3. Pressure to trade
 If your advisor gets paid by commission, there's obvious pressure to increase the number of transactions. So even for investors who can largely build a strategy and stick with it for years, advisors have incentives to make more frequent moves.

Of course, sometimes, changes are warranted. But if you're a long-term investor and your advisor has you acting like a day-trader, you've made a bad match -- and it's time to look elsewhere.

Be careful out there
 It's important to remember that many financial advisors do a good job and deserve to get paid for the work they do. But often, you'd be much better off writing a check to a fee-only planner to set up a good long-term investing strategy than by accepting the "free" advice of someone who gets paid on commission.

When it comes to smart investing, putting together your own portfolio can be the safest way to play. To discover some stock ideas that could be a good fit for you, check out the Motley Fool's latest special report on retirement. Inside, you'll find three promising stock picks for long-term investors. It won't cost you a thing, but don't wait: Get your free report today while it's still available.

Tune in every Monday and Wednesday for Dan's columns on retirement, investing, and personal finance. You can follow him on Twitter here.