You have the right to consent to a conversation.
I learned this with my abuser, who didn't think I had this right. I also didn't think I had this right, part of my childhood conditioning, but once I realized I did and started to assert boundaries, I quickly learned that he didn't think I had this right. He accused me of all kinds of things (like abuse, manipulation, avoidance, punishment) when I said things like, "I don't want to talk about this right now. We can discuss it tomorrow when I'm feeling better." I was drinking his kool-aid, so he did manage to pull me back in many times, until I fully absorbed that I DO in fact have the right to withdraw my consent to talk.
You can consent to touch. You can consent to sex. You can consent to being in the presence of someone. You can consent to talk to someone. Anything else is coercion.
If someone does not like you withdrawing consent (for anything), and this is a dealbreaker for them, then the responsible thing for them is to make decisions about their own damn self. Maybe they need more sex, or more talk, or whatever than you can consistently give them. The mature thing is for them to accept the situation and decide their own boundaries. Maybe even, "I need someone who wants the same things I do, so I will find a different relationship." No fuss, no drama.
It's also ok to discuss this with them, not as a threat or as a way to change their behavior, but in order to negotiate the needs of both in the relationship.
That isn't manipulation, so long as you're not trying to *control* someone's behavior... that's just you accepting someone as they are and withdrawing your own consent or setting more distant boundaries of your own.
Ultimately it comes down to: Are you trying to make someone do something or be a way they don't want to be? Or are you accepting themself as they are and making your own choices for your own life?