It's a trope that a horse can "smell your fear" and may act up as a result.
The truth is that horses absolutely can tell when a handler is afraid. Some horses will act up. Others will actually try to calm the fearful person down (the best lesson horses).
However, there's no evidence they do this by "smelling" your fear. It's vaguely possible that domestic horses can pick up on and identify human fear pheromones...but this is not normal across species.
So, how do they tell?
Horses can easily read the body language of another horse. Domestic horses readily learn to read human body language and tactile cues - that's how we train them, after all.
Horses can't smell your fear - but they can feel the tension in your muscles if you're riding them, see your body language if you aren't and also hear your heart rate. It might seem that they're smelling your fear - actually, you're telling them you're afraid in signals that might as well be shouting in "horse."
(If you're handling or riding a horse and think it's picking up on your fear, try slowing your breathing - which will help both of you calm down).