Yes you have to be worried because diarrhea
can lead to dehydration
which can lead to fatal complication.
You need to see doctor, mean while give the baby as much fluids as possible, if you have ORS( oral rehydration salts with zinc) the better. Continue breast feeding and if on formula milk continue but make sure the hygiene is well maintained(hand washing, feeding utensils are clean etc)