I think this may get into the intent-vs.-context thing, which is always sort of complicated and depressing and unsatisfying when I find myself running across examples in my own life, because there's not really a happy answer. So you may not have intended to be racist when you flinched from a large black guy because the concern for you was the "large guy" rather than the "black" aspect, but since we live in a racist society where black men have been repeatedly villainized and cast as dangerous to white womanhood in a number of social narratives, the flinch still plays into those narratives even if you didn't intend it to. So while it doesn't mean "Kaie as a person = horrible, evil racist" or anything like that, it does still contribute to racist structures in a way that it wouldn't if it was a large white guy you flinched away from.
Which doesn't mean that you need to become totally fine with men invading your personal space. And of course, that tendency for men to crowd women also supports a social structure of sexism - even if in this case his intent wasn't to go "oh, silly women don't need personal space like manly men do!" but rather "whoops, didn't see her there because I was so busy worrying about what I need to do tonight" or whatever may have been his reason, the action in the context of a sexist society was still another example of women's space /comfort/ need to not be sat on being treated as less important than men's.
Sorry, that got all rambly.
I'm a wicked young lady but I been trying hard lately
Oh fuck it, I'm a monster, I admit it!
It makes me so mad my blood really starts a-going
La la la la, la la la lie
Sooner or later, we all gotta die
Curse of Millhaven- Nick Cave & the Bad Seeds