Being read as angry

Both Novice and Saint seem to read me as much more angry / aggressive than I intend. It feels like any mild irritation I express, as well as annoyance, exasperation, even sometimes fear, sorrow or passionate engagement, is often read as anger. 

This is completely baffling to me. I can often feel annoyed, or scared, or worried, or tired, but very, very rarely do I feel angry. Slightly more often after starting testosterone, but still it's extremely rare. My reactions typically lean towards fear, self-hatred or sorrow in 99% of possible situations, and less than 1% towards anger or aggression. To be accused of having a feeling I really don't recognize at all... It feels really confusing, and scary. I'm basically being told that my reality isn't real, that what I think I communicate isn't the message they receive. 

When it was just Saint, I thought it was a consequence of his traumatic childhood, coupled with ADHD and depression. He often perceives things as much more negative than they really are, sees things in black&white. Not just with regards to me, but in general. So if I say I don't like something he's done, he'll very quickly jump to the conclusion that I don't like him. This makes communications really difficult. 

We've tried having me name my emotions more often, so that he isn't left guessing. Since his guesses often lean more serious/angry/dramatic than what I really feel. So I'll tell him: "I'm feeling mildly annoyed at this", or "I'm fed up by this, it bothers me" or "I'm not feeling angry, I'm feeling sad and scared". I think that's helped a bit, but I'm not sure. Hard to go against a life of conditioning...

Anyway, Novice told me last night that they ALSO perceive me as angry/aggressive. That's two people close to me, independently telling me the same thing. I can't brush that off, or blame it on their mental health issues. There's got to be something there, I'm just not seeing it. 

And even if there wasn't... Even if they're both objectively wrong... Does it matter? The effect is still there: They perceive me as angry, aggressive. The two adults I love the most in this world, are telling me that the way I behave is hurtful, damaging, scary. Objective truth doesn't matter, because subjectively to them this feels true. I'm causing them harm. 

I can't have that. I have to fix it. Fix me.

I have no idea how, as I don't even recognize this expressed anger in myself. Saint and I talk fairly loudly with each other, regardless of mood or feelings involved. But if I unwittingly raise my voice around Novice, they're quick to correct me. So I don't think it's a volume issue. 

I don't scream, shout, throw things, slam doors or hit stuff. I'm not like X, who shouted at me, tore down a shelf at my feet, and made a hole in the wall with his fist right next to where I was sitting. I was damaged by those months with him, traumatized. I don't do any of these things, but that doesn't mean I'm not causing them harm. 

I'm left really, really puzzled. 

Could it be a masking issue? I know autistic people often are perceived as angry/upset when they aren't masking. Perhaps I don't mask as well when I'm stressed / sad / tired / annoyed? That could affect my intonation, make my sentences flatter or more clipped, I'm probably smiling less etc. Could that be why I'm perceived as angry? 

If that's the case: Should I be expected to always remain high-masking in front of my loved ones? Am I really causing harm by not masking? This can't possibly be the whole explanation, there's got to be other things there as well. Verbal- or non-verbal things they're perceiving as anger, that I'm not aware of. 

How the FUCK (frustration at myself, not anger at anyone else) do I work on something, when I have no idea what the problem is?!? No tools to recognize it, no way to do damage control?! This feels like another part of social interactions that I wasn't aware even existed. I can't analyze and master something, if I don't know it exists. 

Well, now I know this exists. And I know it's causing harm. I'm causing harm. So I'll just have to figure it out. I don't want to hurt them. Where do I even begin??

Update: Tried googling, but coming up short. All the anger management stuff is focused on explosive rage. I don't do that, and I don't feel that. On the contrary, I don't feel I loose control at all. I even tried a Norwegian page, but encounter the same problem. How can I work on something I can't perceive?
I'm sure there's something out there, but I apparently don't even know what to search for. I feel lost.