Doctors can tell if you've had sex recently if they are testing for that but why the hell is a doctor going to look inside your vagina for a physical anyway? They can only tell if you've had sex recently if they do specific tests for it. As mentioned previously by other contributors, your hymen
can break for all sorts of reasons so it's no indication that you're not a virgin. If he says anything then tell him not to tell but it shouldn't even be an issue anyway because of doctor-patient confidentiality.