What problems in society are both mathematically challenging, verifiable, and competitively rewarding?

It might also be helpful to note that many people, myself included, believe that the correlation between "technically hard" and "interesting" decreases a lot after you go away from college classes into the real world. 

See eg https://www.benkuhn.net/hard/ for an exposition of this belief.

As a high-level comment, it seems bad to structure the world so that the smartest people compete against each other in zero-sum games.  It's definitely the case that zero-sum games are the best way to ensure technical hardness, as the games will by construction be right at the threshold of playability.  But if we do this we're throwing most of the value away in comparison to working on positive-sum games.

More of a skill set than a problem, but data science / machine learning would be my nomination. It’s one of the hottest fields for hiring right now, with computer science more generally being a top earning college major vs. lower earnings for fields like economics, mathematics, statistics, and physics. (See figures here: https://www.wsj.com/amp/articles/which-college-graduates-make-the-most-11574267424.) It’s very mathematically challenging, especially at the highest levels of ML. It doesn’t necessarily have the same gamesmanship aspect as trading stocks of profits depending on winning or losing against another human being, but you are optimizing models and being rewarded for predictive accuracy. (You could also try Kaggle if you’re really looking for competition.)

Most importantly from an EA perspective, it’s good training for contributing to AI Safety, but also offers great impact opportunities for the right person even if they never work on AI Safety. This post and comment describe some of opportunities for having impact with AI beyond working on AI safety, including biomedical research and public health research (Post: https://forum.effectivealtruism.org/posts/LHZBcqyCkYqmZLzij/?commentId=iea66e3TxsnFWbHoS#comments ).

Personally I studied economics and statistics before getting some work experience and realizing that CS and ML would be more useful across a broad range of roles. Maybe that’s my bias, but if you have math/STEM inclinations, I’d say you could do worse than learning some Python or majoring in CS.

As a person from a similar background (IMO medalist, etc.) who has considered similar criteria, one field I have been thinking a lot about is formal verification. It's mathematically challenging in that it can involve a lot of complicated, unintuitive mathematical logic, and it's verifiable in that you can know when the proof checker succeeds that you got things "right", although I don't think there are rewards like in trading or ML competitions, nor is it competitive in the way you describe. (Personally, I now think this last point is a good thing—I loved the math but hated the zero-sum nature of math competitions. But, if this counts for anything, I think there are people trying to build an AI to "win an IMO gold medal" with formal proofs.)

I am not very confident about the impact of such a career, but I think there's at least a decent chance of high impact. 80000 Hours has a short review with some reasons to believe in its potential and some cases for being bearish, although the latter cases focus on AI safety applications and I think applications to cybersecurity are more direct and also have solid potential for impact. Also, I suspect that a formal verification career that's optimized for impact will look completely different from one that's optimized to be mathematically challenging. But I would guess that, at the very least, the path to high impact is more plausible than derivatives trading or finance.