The answer is: it depends. You can probably refuse, yes. However, your ex-husband's parents can file for grandparent visitation rights from the court. This is a complicated area of Family Law and you should probably seek an attorney if and when they file.
Nothing in this communication should be considered legal advice. Nor should this communication lead you to believe that an attorney-client relationship has been created. Only after thorough analysis of your case with you in my office can I provide effective legal advice on your specific issue.