Growing up in East Texas, and the purportedly "Oldest Town in Texas", I've seen the Confederate flag so often that it's pretty much stopped registering. In and of itself, I don't see it as being especially racist, though I sometimes question the sense and taste of those who display on every surface in their homes.
Saying the Civil War was about NOTHING but slavery is a bit like saying the Crusades were about religion and only religion. Slavery was a big motivator, both economically and socially, but there were other economic, sociopolitical and even philosophical aspects behind it as well. Many (not all, of course, but many) of the Confederate fighters - including the African-American ones, and Robert E. Lee himself - were simply supporting their homeland out of a sense of patriotism and their state's rights to autonomy (back then, the common mindset was that you were a Texan or a Virginian before you were an American.) It can and has also been argued that the Union had no right, under the Constitution, to prevent the secession. Is it a good thing they did? Probably.
Now, I'm not saying it's totally invalid to see the Confederate flag as a symbol of racism, because that is mixed up in its history. But in many cases, that's not the intent of the one who displays it: more often, I think it's meant to be a symbol of Southern identity and history. "This is where I'm from, for good or bad." While that notion of loyalty to the land is rather alien to me personally, it is a major factor in this part of American history.
People will always see what they want to see in these things, though.