A former Premier League referee has sparked a backlash after revealing he used to entertain himself by "joking about" and playing "games" while officiating matches.