The pace of U.S home construction slipped in May with many Americans still struggling to afford new houses.

Builders started work at a seasonally adjusted annual rate on 1.01 million homes last month, the Commerce Department said Tuesday. That was down 6.5% from 1.07 million in April.

Construction firms began work on fewer single-family houses, condominiums and apartments last month.

Home construction has struggled to gain much traction this year, limiting its ability to contribute as much to broader economic growth as it has in the past. Many would-be buyers face higher mortgage rates than at this time last year, while builders are selling fewer new homes but charging more for them. That has reduced the number of possible buyers and the number of construction jobs. Builders employ 1.49 million fewer workers than they did at the start of the Great Recession in December 2007, a loss of roughly 20%.

"We expect housing to contribute positively to 2014 economic growth, but the magnitude of its contribution likely will be much smaller than that reported for the past two years," said Dana Saporta, director of U.S. economics at the bank Credit Suisse.

In May, construction tailed off in the Northeast, Midwest and West. Only the South experienced greater building activity in May.

Housing starts have risen 9.4% over the past 12 months. But apartments account for most of the gains, suggesting that more Americans will be renting instead of owning homes.

The growth in apartment buildings points to an economy in which more Americans are renting, rather than buying homes. Following the housing bust and recession, Americans have had to deal with relatively flat wages and job insecurity, both obstacles to saving for a down payment. The home ownership rate was 64.8% at the start of the year, down from a peak of 69.2% during 2004.

Applications for building permits, a gauge of future activity, fell 6.4%  in May to an annual rate of 991,000.

Now that the spring and summer buying season is in full swing, homebuilding appears to be struggling.

Builder confidence has improved slightly but remains pessimistic.

The National Association of Home Builders/Wells Fargo builder sentiment index rose to 49 in June, up from 45 in May. A reading below 50 indicates that builders consider the conditions for new construction to be poor. The index had been above 50 from June through January.

Sales of new homes rose 6.4% in April to a seasonally adjusted annual rate of 433,000, according to a separate Commerce Department report. That comes after sales declined 6.9% in March to an annual pace of 407,000, while buying slipped 4.4% in February.

Median prices on new homes are $275,800, a 10% increase since the beginning of 2013.

All of this comes amid a broader real estate slowdown after significant sales and price gains in the first half of 2013. Both prices and borrowing costs are higher.

Average rates on a 30-year, fixed mortgage are 4.2%, up from a low of 3.93% over the past year, according to Freddie Mac.

Existing-homes were purchased at a yearly clip of 4.65 million in April. That's below the 5.1 million homes sold last year. New home usually represent about 20% of all homes sold, but that figure has fallen recently to 10%, according to Zillow, a real estate data firm.