A lot of people who are athletic have a lower pulse rate. 40 is getting kind of low though. As long as his blood pressure is ok they may just keep an eye on him. If he complains of dizziness
or seems a little sluggish or confused I would think it was too low for him and take him back to ANY doctor or emergency room. Don't wait for a cardiologist even if they might send you to one later. If that doctor told you he should not be exhausted or cannot drive then they think there is a problem. That is not normal. If that is all they told you then I would call them back and say you want more information. You have the right to have them explain why. If they will not help you get a second opinion
as soon as you can!!