You can opt for surgical removal...
So it turned out to be schwannoma
). Both these are very benign and chances of malignant cancerous changes is extremely rare. Studies reveal cancer risks are lower than 1% with schwannoma.
Though schwannoma grow slowly, unlike fibroma it has more tendency to cause compressive symptoms. Therefore your doctors suggestion to get that removed. You can plan this surgery as per your convenience.
Let me know if you are looking for more information about this tumour.