Tar baby?

I was reading that many black leaders were outraged by Mitt Romney using the term "tar baby" in a speech about the troubled Big Dig project.

BOSTON (AP) -- Gov. Mitt Romney has apologized for referring to the troubled Big Dig construction project as a "tar baby" during a fundraiser with Iowa Republicans, saying he didn't know anyone would be offended by the term some consider a racial epithet.

This is what he said:

"The best thing politically would be to stay as far away from that tar baby as I can," he told a crowd of about 100 supporters in Ames, Iowa."

I remember when Tony Snow said it recently as well. I have, of course, heard of the story of the "doll made of tar that traps Br'er Rabbit. It has come to be known as a way of describing a sticky mess." But I have never heard of it being a derogatory term for black people.

I grew up in Mississippi during the 60's and I heard just about every derogatory remark one can possibly make about a black person and that was never one of them.

So where did the "derogatory" part come from? I'd like to know.