Michael Grade (BBC boss) hated Doctor Who full stop and wanted it axed, but I think during Colin's era the budgets were so low that the whole production suffered severely, which gave Grade even more ammunition even though he created the problem. But yes, he felt that Colin had been miscast in the role and when Sylvester was cast John Nathan Turner (DW Producer) was forced to have screen tests made to ensure Sylvester was the best option.....JNT then chose other actors to screen test purposely to make him come across as the best.The Season 23 story was to be called "The Nightmare Fair", and would indeed have returned the Toymaker, portrayed so excellently by Michael Gough during the Hartnell era. I always thought Colin Baker was underrated as the Doctor, and from what I read, he was fired because a BBC boss had a personal grudge against him.
As for HPH being cast as The Toymaker....very much looking forward to seeing this. The 60th special(s) are really sounding great fun.