If you look at English riders, you'll see that most of the time we wear beige or tan breeches. Sometimes you'll see white ones at a show, and sometimes a rainbow of colors, but beige or tan appear to be traditional. Why is this?
The answer is, as far as I know just that, tradition. White breeches, however, are generally only worn when working in show arenas. Beige and earthy colors are for following hounds or the trail.
So I honestly suspect - although I don't know - that the origin of wearing those colors was simple: They don't show the dirt and dust inevitably picked up on a long ride and still look smart at the end of the day.