Some photography is gated behind the physical requirements of keeping up with the subjects, and unsurprisingly, action adventure photography is one of those genres.
The first thing to note about this video is that it's a lightly shrouded advertisement for Nikon's Z 6 II, by Nikon Europe. Generally, I'm not interested in sharing that sort of content, but this has value outside of that and is worth your time.
Under the campaign #CreateYourLight, Nikon follows outdoor, adventure, and urban photographer, Ray Demski, as he shoots runners up in the mountains. The location and light is beautiful, but what struck me most was how difficult the shooting must be on the photographer, physically. You are quite clearly have to keep up with your subjects, and presumably hike difficult terrain to get to the locations in the first place. What makes this more difficult — as all us photographers know too well — is camera gear is seldom light. So, in essence, you're doing what the subject athletes have to do, with the addition of a pack weighing you down!
Nevertheless, the results are fantastic and it's a genre I'd love to try. Sometimes the physical nature of getting a shot adds to the sense of accomplishment.
I've never done a formal study or anything, but over the years I've read about many such outdoor/adventure photographers. And it seems to me, many if not most either used to do the activity they photograph, or currently are part of the whole "scene." Ray Demski's bio says he's an accomplished athlete, Wikipedia lists Jimmy Chin as a professional climber AND photographer (and film director), and while I can't name any names,I know I've read the bios of top surfing photographers who were once accomplished surfers themselves. It all helps with both getting the trust of the athletes and clients, and with being able to keep up on a shoot. Because like you say, it can't be easy!