if indeed you are having a reaction to the soap, them the first step is to avoid using the soap, which it sounds like you are doing already. you can usually use a hydrocortisone cream
to help with this type if irritation, and this is available in most pharmacies over the counter. if things get worse or persist despite avoiding the soap, you should see your doctor as it may be something else and require treatment.
I hope this helps, please let me know if you have any more questions.