LONDON -- In my 30 years as an investor, the only type of business that I've always refused to invest in is the tobacco companies. Yesterday I added banks to that list, and I'm now having a good think about whether insurance companies should be on there as well.

The straw that broke this particular camel's back was the revelation that the FTSE 100 member Barclays (LSE: BARC.L) was fined 290 million pounds for manipulating the London Interbank Offer Rate (LIBOR), and that many other banks are being investigated for behaving similarly.

Even though tit may be let off the hook following reports that it was encouraged to do this by the Bank of England and "Whitehall officials," I'm steering clear as I'm no longer prepared to invest in a sector where it seems to be standard practice to rip off your customers and companies' accounts are becoming increasingly hard to interpret.

Ridiculously complex
The balance sheets and profit and loss accounts of banks (and insurers) are such complex beasts that even the experts can be fooled. The big problem is that it is easy to turn a loss into an accounting profit in this sort of business by making overoptimistic assumptions about the future such as the level of bad debts, claims, and the returns that you'll earn upon your investments.

Many towns and cities in America are now facing bankruptcy because they did just this with their pension schemes over many years and ran up massive deficits as a result. You can only deny reality for so long before the truth eventually catches up with you.

History tells us that banks love to gear up their exposure by using derivatives, financial instruments that Warren Buffett called "weapons of financial mass destruction" back in 2003. This lead to the crash of 2008-09, when several years of passing off bad debt as good debt nearly led to the collapse of the American financial system.

The three that I do own
That said, I've only ever owned shares in one bank, and I still own shares in three financial companies, which I'm not planning to sell. The largest holding is in Buffett's conglomerate Berkshire Hathaway (NYSE: BRK-A)(NYSE: BRK-B), which is really a group of insurance companies combined with a much larger collection of non-insurance businesses like the Burlington Northern Santa Fe railroad.

Then I have a much smaller stake in Prem Watsa's Fairfax Financial Holdings (NYSE: FFH), a diversified insurer that's known as Canada's equivalent of Berkshire Hathaway because Watsa's philosophy is similar to Buffett's.

Finally, there's a very small holding in Lloyds Banking Group (LSE: LLOY.L), which is a pure punt on the economic recovery. The old Lloyds TSB used to be Britain's most conservatively run big bank, but its directors threw more than 200 years of history out of the window during the financial crisis of 2008, when they decided to ruin the bank by buying HBOS. I'm hoping that enough of the old culture remains.

Buffett and Watsa have demonstrated over the years that they take their responsibilities very seriously. They both own a substantial proportion of their respective companies, so they have plenty of skin in the game, and this is a great incentive to not bet the farm on a reckless acquisition, as The Royal Bank of Scotland did when it bought ABN-AMRO.

Do they understand themselves?
Another thing that puts me off financial companies is that I doubt whether anyone -- and I really do mean anyone, including the top management -- has a complete picture of their business. The multinational banks with investment banking arms seem to be particularly prone to this because they have become so complex.

No doubt there will be a few bargains thrown up in the wake of the fallout from the LIBOR scandal. But I feel much more confident that Mark Cavendish will win today's stage in the Tour de France than of my being able to make money by buying the shares of most banks.

However, as Nathan Rothschild said, you should "buy at the sound of cannons." Given the hammering that banks and other financials, such as the composite insurer Aviva, have taken over the past few years, investors who are bolder than I could do nicely by taking a close look at the sector. I'll sit this one out, as there are plenty more fish in the investment sea.

Buffett buy signal! The billionaire investor has found an attractive large cap right here in Britain! Discover what he bought and the price he paid in this special report, "The One U.K. Share Warren Buffett Loves" -- it's free.

Further investment opportunities: