Warren Buffett says that the housing market remains the key to the economic recovery, and the key to getting the U.S.’s unemployment rate to fall significantly. And he says he thinks housing may turn around sooner than many think.

“Unemployment will fall significantly, in my view, when we get back up to a million housing starts,” Buffett tells Charlie Rose. “We’ve recovered on corporate profits, we’ve recovered in terms of getting the banks back in shape — banks are in good shape now — we’ve recovered in all kinds of areas. Corporate America is doing fine in most areas. It’s not doing fine in things tied to residential construction. That won’t come back until we work off the excess inventory. But the amount of excess inventory is not as high as a lot of people think, in my judgment.”

Buffett says he thinks the housing market could work off its excess inventory before mid-2013. The broad impact of the housing market, which impacts not only homebuilders but other industries (carpet companies are one example), is what’s needed to make a big dent in the unemployment rate, he says. When the U.S. gets back up to a million housing starts a month, Buffett says the unemployment rate will fall below 7%.

Buffett also discusses entitlement reform; the need for politicians to get things done rather than use issues “as weapons”; and how the U.S. is becoming a nation of “super-haves” and have-nots. He reiterates his belief that the country won’t go into a double-dip recession, unless one of two things occurs: Confidence in the country’s leaders is eroded to such a point that people talk themselves into a “huge funk”, or the European debt troubles spill over to the U.S., both of which he says are unlikely.