In an ideal world, your broker would help you get the best deal possible whenever you made a transaction. But unfortunately, you have to watch your broker closely to protect your own best interest. From high-commission products to nickel-and-dime fees, brokers are in business to make money -- and if you're not careful, you'll be left footing more than your fair share of the bill.

Many customers have never heard of one source of potential revenue for brokers, but it's an increasingly important part of the business. As competition among exchanges for stocks and other securities has heated up, exchanges have decided that a great way to entice your broker to place your trades with them is to offer cold hard cash for the privilege.

Making money that you'll never see
It wasn't that long ago that investors didn't have many choices in where to buy and sell stocks. If a stock was listed on a given exchange, that's where you'd typically go to trade shares.

The boom in new exchanges, however, has given investors -- and, more importantly, the brokers who typically serve them -- alternatives on where to place trades. In theory, that should be a good thing for investors. With multiple exchanges, brokers can route trades through whichever one will give their customers the best execution price.

To attract traffic, upstart exchanges started offering brokers a payment to place orders there. In some cases, exchanges then charged an extra fee to anyone who filled one of those offers. So in other words, brokers were rewarded for adding liquidity to the exchange but had to pay a premium if they reduced liquidity by taking out one of the orders.

Over time, the practice became commonplace enough that even NYSE Euronext (NYSE: NYX) and Nasdaq OMX (Nasdaq: NDAQ) started doing it. According to a recent article in The New York Times, during the first three months of 2012, the Nasdaq paid more than $300 million in rebates, while the New York Stock Exchange paid a somewhat smaller proportion of its overall revenue. A recent Woodbine Associates study suggests that rebates may cost investors as much as $5 billion annually.

The practice isn't limited to stocks, either. For instance, CBOE Holdings (Nasdaq: CBOE) offers rebates for certain types of options trades on its C2 platform.

What it means for you
In an ideal world, none of this would make any difference to you. In deciding where to place your trades, your broker would aim to get you the best deal possible after factoring in any rebates. If the broker earned a rebate, it would pass that money along to you in the form of lower commissions. Interactive Brokers (Nasdaq: IBKR), for instance, gives customers who use its cost-plus pricing structure the benefit of any rebates it gets from exchanges.

But some brokers keep any rebates to offset general expenses. For instance, TD AMERITRADE (Nasdaq: AMTD) says in its order-execution FAQ: "The majority of exchanges and market makers provide incentives for brokers to route orders to them. Typically, this involves a rebate or payment to the broker for routing orders to that exchange or market maker. … This payment is used to offset the costs of doing business and ultimately helps to reduce the overall cost to our clients."

If you're not a frequent trader, then these small boosts in execution price -- which often amount to less than a penny per share -- probably won't make a huge difference to your investing results. But even investors who never trade stocks at all can be affected. The Woodbine study pointed to mutual fund companies facing the same situation when they make trades for their accounts, and with much larger trades involved, those pennies add up.

Watch out
Trade rebates are just another way that brokers can end up having a conflict of interest with their clients. Making sure that you learn as much as you can about how your broker gets compensated will put you in a better position to understand your broker relationship. You may not be able to avoid paying slightly more, but it will at least give you a more accurate picture of what your true costs are.

Finding the best broker you can is important to your long-term financial success. So is making the best investments you can. The Motley Fool's special report on long-term investing can help you figure out where to find the best stock prospects for the long haul, with three names submitted for your consideration. Let me invite you to get your free copy right now.

Do you need more help finding a broker that will do a better job of meeting your needs? Go to the Fool's Broker Center and learn more about the brokers that cater to investors like you.