I agree with you, as there is no chest pain
or short of breathing or tightness in the chest then there is no reason to worry about that.
as you said you did some heavy physical work which seems the cause for your arm pain
, most likely muscular but can be also from trapped nerve in the cervical spine
try simple analgesia such as paracetamol
if no improvement in the next few days then see your family doctor /GP
hope you get better soon