With your description of the swellint it is looking lymphnode only.
Any infection in concerned drainage
area of one lymphnode lead to enlargement of that particular lypmnode as it tries to limit the infection.
Previpous ear infection
might be the cause of lymphnode enlargement and the site you described contains the lyphnodes(postauricular and posterior cervical lypmnodes).
Nothing to be scared as you did blood tests and they were normal.
Consult your ENT surgeon and discuss your concerns with him.