A quick look through horse ads and you'll see that most of the male horses advertised are geldings.
The vast majority of male horses are neutered. This practice may date back to very close to the beginnings of domestication. The Lapps capture, neuter, and release many of the male reindeer in the herds they follow.
Gelding is not done to limit the number of foals produced per se. You can only reduce population growth in a harem breeder species by removing females - despite the fact that some people promote geld and release as a way of controlling Mustang populations (it won't work).
We geld most male horses for two reasons:
1. To remove inferior males from the breeding pool.
2. To reduce aggression and make the animals easier to handle.
Male horses produce a lot of testosterone. Stallions have a high libido and are very aggressive. They are harder to train than mares and can be dangerous to handle at times, especially if not properly trained and socialized.
Castration just makes them much more mellow, even, and easy to deal with - to the point where many horsemen prefer geldings to mares (not all - I actually have a fondness for a good mare myself). Castrating before puberty can also cause a slight, but noticeable increase in adult height.
Most domestic horses are gelded at a year. Some people believe in gelding at a younger age, but this is controversial. Ideally, though, you want your riding or work horse to be gelded prior to puberty, as horses gelded afterwards can retain some stallion-like behavior.
The exception is the racing industry, where colts are often kept intact until they can be tried on the racetrack. As performance is the only thing these horses are judged by (sometimes to the expense of soundness and temperament) it makes sense to keep them as entires until they've raced a few times. This does mean that some male Thoroughbreds can be a little studdy...I've had one go into display behavior on me because a strange gelding went off on a trail ride with a mare he fancied. He's a gelding, but he clearly wasn't "done" until after puberty.
Miniature horses are often kept entire because people think they're easier to handle because of their size.
A Paint/Draft cross gelding.