Usually people will say that when they don't really know the person who passed but they care about you enough to sympathize with your situation and how you feel.

I guess it's just one of those irrational things. Like, there is no real, rational reason for it to be upsetting and you get what people mean by it. When it's all you hear people say it can be almost draining, if that makes sense