Management of fibroid is done if it is symptomatic with pain , excess bleeding, or is associated with infertility
As you said your fibroid is huge (you havent mentioned the size) it can cause problems in your pregnancy
especially if its within the uterine cavity
or distorting the uterine cavity as it would prevent implantation
or cause abortion
of an implanted pregnancy.
So its good if you get it operated and then go ahead with your pregnancy.
But you have to keep in mind that many times the infertility can be due to several other reasons despite having a fibroid. So before removal of fibroid for infertility per se , you should rule out other causes of infertility . Its important before surgery especially if you are taking it for conceiving.