Here's a thought. Socially responsible investing is neither socially responsible nor investing.

Yes, I know that sounds funny coming from a guy who refused to eat at Yum! Brands' (NYSE:YUM) Taco Bell because of its refusal to do the right thing for the laborers who picked its tomatoes. But the more I look at the movement that tries to do good through investment, the more I see potholes.

Does it really matter?
First off, there's the "It's none of my business" factor. Do I think people are smart to use tobacco? No. I also don't think they're wise to eat at McDonald's until they can barely fit behind the steering wheel, or throw their money away in casinos, or give their teenagers giant SUVs that flip over when they change lanes too quickly, or jimmy themselves into $300 a pair premium denim jeans just so they can feel better about their butts.

But you know what? It's none of my business. Moreover, human nature being what it is, people are going to engage in these activities anyway. And many will derive a great deal of satisfaction from such activities. And so long as the companies providing the products are sound businesses producing profits for shareholders, I'll reserve the right to make them part of my business and suggest that other investors do the same.

Unlike you, I DO care
If you're more pronounced in your politics than I am, hey, good for you. Knock yourself out. Feel free to choose companies that fit your needs. (Your choice is also none of my business.) But this just brings us to my second problem with socially conscious investing: the bogus factor of funds that want to get your business by assuaging your guilt.

SRI undoubtedly has a strong appeal. Consider the psyche of a generation of young investors who have come of age with reports of bad corporate behavior and protests by bearded anti-globalization mobs. A crowd that listens to guys like Coldplay's Chris Martin saying things like "I think shareholders are the great evil of this modern world." (Never mind that without the capital provided by shareholders to form recording companies and bankroll the warblings of skinny, weepy crooners, Martin might be singing in the shower instead of living in the lap of luxury and sleeping with an American movie star.)

The natural response of the fund industry, which, like any other business, has to compete for fad-commerce, is to offer socially responsible funds, with a promise to offer returns without the guilt. But what, exactly, is socially responsible or socially conscious? Good question.

It is what they say it is.
To me, the biggest problem with so-called socially conscious funds is that what is "responsible" will largely depend on what your definition of "responsible" is. That means the usual suspects -- guns, tobacco, gambling -- are usually right out. From there on out, responsibility is in the eye of the beholder. The trouble for a critical beholder like me is that the easy answers peddled by these funds may not really stand up to scrutiny.

For instance, many SRI funds say they exclude polluters, yet they often include a lot of tech companies. The computer biz may seem cleaner than oil refining, but only if you don't know what goes into making a circuit board. How about what happens to all this techno-junk when it goes obsolete -- according to business plans, of course -- in about two weeks' time? (Hint: Thousands of tons of it end up exported for "recycling" to poor African nations, where it sits around in seeping piles.) Where, then, do you come down on a company like Dell (NASDAQ:DELL) or Hewlett Packard (NYSE:HPQ)?

Or take energy. Many funds say they exclude nuclear energy. Easy to see why. People fear the big, bad nuke. But this only increases our reliance on other technologies. Are coal- and gas-fired alternatives better? Global warming, anyone? How about coal-mining accidents? Other ill-health effects from airborne particulate pollution? These are estimated to kill between 50,000 to 100,000 Americans every year.

Interested in morality? Then your fund likely avoids casinos. But how do you know how much money Verizon (NYSE:VZ) generates by providing the conduits that carry phone sex, or Internet gambling and porn? I'm betting it's more than none.

The answers aren't easy, and that is precisely the problem. The fund managers' judgment of what is socially responsible enough -- ExxonMobil (NYSE:XOM) has women on the board! -- might not actually fit with your politics. But by promising to do the thinking for you, they only relieve you of the guilt, not the responsibility you may think you have.

Returns, anyone?
I noticed something funny when I started looking into SRI funds. Too many of them discuss their returns in a vacuum, or against other SRI funds. This is a bogus, but understandable, situation, and it's probably a response to the facts: Most of the well-known SRI funds I've seen recommended by various financial sources underperform the market.

How about taking a look at one of the old standards of this field, the Pax World Balance Fund (FUND:PAXWX)? World? Balance? Sure sounds nice. Until you look at the tape. How's a 50% lag on the S&P 500 sound to you?

It was the same story with every SRI fund I could find. Even the Neuberger Berman Socially Responsible (FUND:NBSRX) fund chosen by our fund guru, Shannon Zimmerman, has a pretty unimpressive track record in my book, although it has done better over the past few years.

The Foolish bottom line
If you want to use your money to do good, more power to you. I wholeheartedly encourage you to do so. But do it with open eyes, and don't be fooled by the financial world's latest marketing gimmick. SRI funds may promise you guilt-free returns, but they're only likely to deliver what the vast majority of funds give you: crummy returns.

If you really want to make a difference, dig into companies you like, invest where you think you'll make some money, and make a difference. And along the way, share the wealth you accumulate directly with the causes that concern you. (Foolanthropy would be a good place to start.)

Whatever you do, don't leave the thinking to someone else. There's nothing socially responsible about that.

Wait! You're not done. This is just a quarter of the Duel! Don't miss the Bull opening argument and the Bull and Bear rebuttals. Even when you're done, you're still not done. You can vote and let us know who you think won this Duel.

Seth Jayson holds funds only when he's forced to by retirement plans. At the time of publication, he had positions in no company or fund mentioned here. View his stock holdings and Fool profile here. Dell is a Motley Fool Stock Advisor recommendation. The NB Socially Responsible fund is a Champion Funds recommendation. Fool rules are here.