I've seen this in more than one book. A horse refuses to open its mouth for the bit, so the handler holds its nostrils shut.
This is both cruel and ineffective.
Unlike humans, horses are not able to breathe through their mouth. So, they won't open their mouth if they can't breathe through their nose, they'll just...well...likely hurt you pulling away.
The correct technique for forcing a horse's mouth open is to insert your thumb into the side of the mouth, where the bit rests, and press down firmly on the horse's gum.