I don't care.
As this indicates complete detachment and disinterest, it is the most emotionally-charged of the four. You can do what you like, I don't care, it is nothing to me.

I don't mind.
This is a gentle statement indicating general assent and support.

It doesn't matter.
This is often rather like I don't mind. Alternatively, it is very like never mind.

Never mind.
This carries a strong sense of forgiveness or reassurance. It's OK, I know you have done something silly, but we can make the world OK, between us.