What you describe is hyperacusis, and while it can be direct pain from noise, in other cases it can be more of a discomfort, and even on an emotional level.
I have read other users that have difficulty with clinking glasses, silverware, and dishes, and need hearing protection to do the dishes. I have not experienced this, but have experienced discomfort with more lower frequency sounds, such as loud vehicles, and lawn equipment.
One day it was quite striking as I was outside, with no hearing protection, and enjoying the sun and the sounds. I was fine for almost an hour, and then my neighbor across the street turned on a vacuum to clean his car. I was pretty far away, but that sound instantly annoyed me, and I could not wait to go inside as it was so uncomfortable, although not objectively loud as I was plenty far away. Once I got away, the discomfort stopped instantly, so I know exactly what you are talking about.