A Peculiar Prejudice

People are mad if I don't want to get close to them in some way but not others.  I feel like I'm a murder victim and they are in prison.