has the potential to make a women infertile however a small fibroid has less chances. We need to rule out the other possible reasons and correct them first before we resort to get this removed.
Poly cystic ovaries are second reason to be scared, though completely manageable it takes a lot of patience and participation from your side.
Follow up well with your doctor. It also makes sense to take a second opinion
from an expert Infertility
Specialist in your area. Technology can increases the outcome.