Can my ex force me to change custody arrangements?

I left my kids’ father after a police report for domestic violence. The case was dismissed, but that doesn’t mean nothing happened. After we split, he quickly entered a new relationship and texted our kids, saying he had moved on and didn’t want anything to do with them. He even tried to give up all parental and financial responsibilities, but that didn’t go through.

Later, he blamed me for “ruining his life” and eventually became single again. After some time, he started seeing our youngest child every other weekend, from Friday to Monday. It’s not a written agreement, but I have proof of this arrangement.

Now, he wants to change the custody schedule—not for work reasons but to fit better with his new relationship. I’ve already arranged the current schedule to work with my job, and changing it would disrupt not only my routine but also my workplace. He’s pressuring me, saying I need to adjust my job to fit his plans.

Can he legally force me to change the arrangement? And does the fact that he’s unemployed make a difference in this situation? I feel stuck and need advice.