Given our current understanding of mathematics and physics, it is much more plausible that a mathematician (or small team) could achieve a breakthrough advance in prime factorization than that an inventor (or small team) could achieve time travel or an FTL drive. Plausibility is not essential in speculative fiction, but it improves the sense of being grounded, and tends to make ideas age better.