I recently coerced my friend into reading the first two chapters of a fantasy book of my choosing. The problem is, I can't decide which, because I really want to hook him into reading fantasy. The only fantasy books he's read thus far are a few Harry Potter and Lord of the Rings. He was not impressed. He normally reads physics textbooks, math, etc.
This is an old thread, but mention of Physics and math made me think of Time Shifters by Shanna Lauffey. It's a time travel series and a lot of science gets slipped in so smoothly that you don't realize you've been reading Einstein's theories if you're not looking for it. Book four really goes into current temporal theories, quoting known experts in the field like Hawkings, Thorne and Davies.
It's also a good adventurous read with a strong female protagonist. Win-win.
Normally reads physics textbooks... I might ease him into fantasy by way of sci-fi. Larry Niven has some cool short novels that turn on real science points -- like Neutron Star. And the first Heechee book, Gateway is quite good. The Martian.
The Warlock series by Christopher Stasheff is fairly sf-in-a-fantasy-trapping -- the protagonist is a space explorer showing up on a medieval-trapping world where the "witches" have telepathy and telekinesis. (It's sort of a fallen galactic empire, spreading back out to the distant colonies that were left isolated in the collapse.)
Or some really hard SF like James Halperin's two books -- The Truth Machine and the First Immortal. The first one is our world with the invention of a lie detector that actually works, and is cheaply mass-produced. The implications for everything -- crime, politics, science, business, personal relationships... he explores it all. The second is from the PoV of someone who chooses to freeze himself at death and then his great-something-grandson invents the nanotechnology to reassemble frozen brains and brings him back to life in a strange distant future.
The single-question What-if fiction can appeal to a physics-textbook reader where swords-and-sorcery adventure might not.
Sci-fi: Fantasy's gateway drug for nerds.
If he really only reads textbooks, it's probably a lost cause, though. Reading for knowledge and reading for the emotional ride is a different pursuit.