The most-mentioned books on r/math.
Looking for more? Similar subreddits are algorithms, AskScienceDiscussion, genetics, medicalschool, and rstats.