This phrase may or may not have anything to do with horses. To "get one's goat" is to make one
angry or mad, and it more likely came from a naval tradition of keeping ship's goats.
However, there's another popular etymology which, while it has no evidence, makes some sense.
In order to reduce stress, racehorses that travel a lot often have companions that go with them. A variety of animals are used - miniature horses, donkeys, chickens, etc. Goats, however, are one of the most common. Goats and horses get on well together - in some parts of Europe co-grazing goats with horses is done for pasture management.
Stealing a racehorse's companion would make the horse antsy, stressed, and even angry - and thus unlikely to perform well on race day.
There's no actual proof this has anything to do with the saying, but it's a cute idea.