With the federal government running huge deficits, Warren Buffett says it’s time to raise taxes on the wealthy — who, he says, often end up paying much lower tax rates than working-class Americans. And, he says, doing so won’t drive wealthy investors away from stocks, as some argue.

“I have worked with investors for 60 years and I have yet to see anyone — not even when capital gains rates were 39.9 percent in 1976-77 — shy away from a sensible investment because of the tax rate on the potential gain,” Buffett writes in a New York Times op-ed piece. “People invest to make money, and potential taxes have never scared them off.”

Buffett also says that higher tax rates don’t hurt job creation. To those who argue otherwise, he says, “I would note that a net of nearly 40 million jobs were added between 1980 and 2000. You know what’s happened since then: lower tax rates and far lower job creation.”

Buffett says that the wealthiest Americans often pay a far lower percentage of their income as taxes than do middle-class Americans. A big reason: Much of the wealthiest people’s incomes come from investments; much of the middle class’s incomes come from wages, meanwhile, which are usually taxed at a higher rate than capital gains. In fact, noting that the IRS keeps data on the 400 Americans with the highest incomes, Buffett says that 88 of the 400 reported no wages at all in 2008, though every one of them had capital gains. “Some of my brethren may shun work but they all like to invest. (I can relate to that),” Buffett writes.

Congress needs to change things as it looks to resolve the budget crisis, Buffett says. He advocates keeping tax rates the same for 99.7% of Americans, but upping them for the wealthiest 0.3%. “For those making more than $1 million — there were 236,883 such households in 2009 — I would raise rates immediately on taxable income in excess of $1 million, including, of course, dividends and capital gains,” Buffett says. “And for those who make $10 million or more — there were 8,274 in 2009 — I would suggest an additional increase in rate.”

Buffett says the wealthy — including himself — have been “coddled long enough by a billionaire-friendly Congress. It’s time for our government to get serious about shared sacrifice.” And, he says, most of the “mega-rich” people that he knows wouldn’t mind paying more taxes.