Oh, I'm pretty sure a label can be applied, accurately or not, whether or not you accept the label. Some people are labelled crazy, but aren't and some crazy people never get called crazy for one reason or another. No, it isn't fair, either. You could even be given a label that is perfectly correct, insofar as it goes, that you hadn't even though of. I believe I mentioned somewhere already how that happened to me. Someone called me a "four-wheeler," which was a perfectly accurate word to describe what I was doing in my spare time then but I'd never thought of myself that way.
They're just descriptive words, not accusations.