Viral infections such as chicken pox
are most contagious at the early stages when it is difficult to be certain one is infected.
As you never had infected, you remain vulnerable and especially more if you have not had the vaccines.
Suggest you get serology done to see if you have protective antibodies
in which case you are naturally protected or get the vaccine course.
If within the next week, you develop any fever, chills
better not visit the new born and the family.