If your company wants to limit the times or days when referees can book a phone call, you can set it up on company level.
As an administrator, press your profile picture, select Company Settings and go to the Communications tab. There is a setting where you can select the time slots for when referees are allowed to propose times for reference calls. You can press the weekday to block the entire day if you, for example, want to block the weekend completely. There must be at least 30 hours a week for the referee to choose from.
When the referee selects the "I would rather be called" option, the calendar is now blocked off the times/days you selected.
All time slots are still available for you as a recruiter to select if you yourself want to send and propose times to a referee.