There should not be anything serious.
Normally, the left HANGS lower than the right for most men.
There is a muscle called DARTOS in the scrotum
that contracts and relaxes based on the temperature changes of the body and that is how and why you feel it.
Since your doctor has examined you and felt nothing abnormal, there should be no worry.