In some styles of riding, it's considered desirable for the horse to hold his head in a vertical position. In others - not so much.
So, why is this? What is with the different head positions? It's often said that it's "to look pretty" or "to balance better."
Neither is the likely original truth of this. Okay, here are some quick visual aids:
This is a show jumper making a turn. His head is actually fairly low for a jumper, but it's still up, the nose pointed slightly forward (image source Paul Keleher via Wikimedia Commons).
And this is a dressage horse, with the head held vertically.
Both horses are pretty well balanced. So, why the head thing? The actual likely reason has to do with the horse's visual range. Horses have a cone of binocular vision that extends outward from their forehead. They also have a rectangular pupil, giving them a horizontal area of visual acuity (the "reverse" of cats, with their vertical slit pupils).
The jumping horse's cone of best vision is directed forward towards the next jump. The dressage horse's is actually directed at the ground in front of his feet. Which might not seem to be where you want your horse looking - except that dressage originated on the battle field. If you're on a messy battle field with downed men, downed horses, dropped weapons and all sorts of other nasties on the ground around you - then "where he's putting his hooves" is probably exactly where you want your horse to be looking!
Western horses also often keep their head vertical - because when crossing rough terrain or difficult trails then again, you probably want your horse looking at the ground.
So - it all has to do with vision. (Note: This is not a popular thing but a personal theory, but I'm fairly solid on it.