welcome to HealthcareMagic
You should worry about having baby after 1 year of staying with husband and having unprotected relations.Relax do not think there is any problem immediately.The criteria to get pregnancy
-patent fallopian tubes
-normal semen analysis
When all these things are fine there are high chances of pregnancy.Also to increase your chances you should have sex on alternate days from 10th -21st day of your every cycle .Keep lying down for at least an hour or more after sex do not get up as soon as it is finished.Even as your doctor suggested Letroz will help but what i suggested are the precautions and your doctor has given medicine;together it may help.
All the best