Brief Answer:
loose skin is common but not the pain

Detailed Answer:
Hi.
Thanks for your query, albeit short and specific.
To recapitulate: Age not mentioned - no mention about visit to a Doctor or medication (mentioning these will help me) - loose skin which holds testicles - some times it pains on lateral side in night - wants to know whether to worry about or is common for all.

This is definitely not common.
Loose skin is normal and common for all. It is the function of scrotal skin to be loose to lose the temperature to keep the testicles cooler than the normal body temperature for it to function normally.
The causes of the pain on the lateral side can be:
Mild fullness or inflammation of the epididymis and/or testis.
There can be a hernia stretching the nerves in the inguinal region.
And other reasons if any needs a clinical evaluation and investigations for proper diagnosis and treatment if needed.

I hope this answer helps you, please feel free to ask for further relevant queries if you feel that there is a gap of communication. Please give details about age and any other information that you would like to share so that we can discuss further.

---