By now, you're probably tired of the market doldrums -- and the rallies that then finish in more declines. But before you decide to throw in the investment towel, turn to someone who has been through many down markets and crashes. This person has continued to invest during the tough times, and over time, he's been successful.

I'm talking about Warren Buffett. At the helm of Berkshire Hathaway, he's delivered a 19.8% compounded annual increase over the past 57 years. That's compared to a 9.9% gain for the S&P 500 Index.

It's definitely worth listening to Buffett's wisdom, especially during difficult times. One particular comment may pull you out of the bear market blues and inspire you to keep building a portfolio that could soar over time.

Difficult market times

Buffett's comment has to do with difficult market environments, such as the one we're going through today. Here's what the billionaire investor had to say: 

The most common cause of low prices is pessimism -- sometimes pervasive, sometimes specific to a company or industry. We want to do business in such an environment, not because we like pessimism but because we like the prices it produces. It's optimism that is the enemy of the rational buyer.

All of this means that many investors feel negative about the market at certain times. Right now is a perfect example -- and that's the time to invest. That's because this general pessimism leads many to sell -- and that results in great prices on stocks that still have amazing long-term potential.

Buffett calls optimism the "enemy" because when the market is soaring, everyone wants to get in -- and sometimes, investors will try to get in at any price. Down the road, they may regret paying too much for those once-skyrocketing stocks.

How does Buffett handle these difficult market environments? He doesn't panic. Instead, he continues investing.

Buffett doesn't even mind when his favorite stocks -- ones he already owns -- drop during market crashes. In the Berkshire Hathaway shareholder letter back in 2008 -- during a financial crisis -- Buffett shared more inspiring thoughts.

"We enjoy such price declines if we have funds available to increase our positions," Buffett wrote. "Whether we're talking about socks or stocks, I like buying quality merchandise when it is marked down."

Follow Buffett's lead

If you're inspired by these Buffett comments, what's the best thing you can do right now? Follow Buffett's lead and invest in quality companies that are suffering now but have what it takes to excel over the long term.

A good example is Costco Wholesale (COST -0.69%). The warehouse giant has declined 13% over the past year. At the same time, revenue and net income have continued to grow. That's because Costco offers bargain-basement prices on essentials -- something folks are looking for right now.

Costco relies on membership fees for profit. And the good news here is the company's membership renewal rates remain high, at more than 90%.

All of this makes Costco a company that can grow during good times and bad. Today, it's trading at 34 times forward earnings estimates, down from more than 45 last year.

An example in healthcare is Pfizer (PFE 0.02%). Yes, the company faces declines in coronavirus vaccine sales, compared to earlier in the pandemic. But Pfizer still predicts blockbuster sales from its vaccine and coronavirus treatment. The company also has a solid portfolio of blockbusters -- and 19 candidates in the pipeline are expected to reach the market within 18 months.

Today, trading at 12 times forward earnings estimates, Pfizer looks dirt cheap.

There are other examples across industries. So instead of forgetting about investing during down markets, remember that they can be key times in your investment story. Like Buffett, you can pick up great deals -- and possibly watch the value of your portfolio take off over time.