The nature of communication on the internet makes for some very weird signifiers - I remember getting into fountain pens and finding out that a pretty undesirable group was into them as well and it was on the cusp of being a signifier for politics I don't agree with. Luckily it never got to that threshold, but basically something can become a signifier for something else just by virtue of volume - Pepe is probably the best, clearest cut example, with different groups literally mass posting pepes as much as possible in a battle to "own" that signifier.

I was originally going to say "Hey, just do you" but then I totally get that feeling of "I'm into X just because I like it but for some stupid reason X signifies Y which I really don't care for" and it sucks.