In college, I worked as a valet at a fancy hotel in Los Angeles. When an expensive car drove in, I used to always think, "Wow, he's rich!" But as I got to know these "rich" people, I saw a different side. A few opened up about their finances (people love talking to valets), and I couldn't believe their stories. Some of them weren't that successful. Certainly not like I imagined. Instead, they made modest incomes and spent most of it on a car. It's amazing how fast you can go from admiring someone to feeling bad for them.

I learned something from that. When you meet someone who owns a $100,000 car, you only know one thing about their wealth: That they have $100,000 less in the bank, or $100,000 more in debt, than they did before they bought the car. That's the only information you have.

We rarely think of it that way. So much of our perception of wealth is driven by what we see other people buying. Since we can't see their bank accounts, that's all we have to go on. But it gives us a distorted view of wealth. Some people we think are wealthy really aren't; they just spend most of their income. Others we think of as less well-off are actually the rich ones. They're rich not despite driving an old car, but because of it.

Financial wealth isn't what you see. It's what you don't see.

My colleague Bill Mann wrote a great article on debt, suggesting: "There is no faster way to feel rich than to spend lots of money on really nice things. But the way to be rich is to spend money you have, and to not spend money you don't have. It's really that simple."

That's excellent advice, but I'd take it a step further. The only way to be rich is to not spend money you have. That's the only way to accumulate wealth.

The reason most of us don't ever get rich is because we have conflicting desires: We want to be rich and spend a lot of money, without realizing that the latter makes the former difficult to achieve. When most people say they want to be a millionaire, what they really mean is "I want to spend a million dollars," which is literally the opposite of being a millionaire.

At some point, you have to pick whether you want to be rich or buy stuff. There's no way around it. The arithmetic will win in the end. 

Check back every Tuesday and Friday for Morgan Housel's columns on finance and economics. 

Follow Morgan Housel on Twitter @TMFHousel. The Motley Fool has a disclosure policy.