Sounds like my little boy at the moment also a premmy.. when he was like this I would always take him to the drs or hospital as sometimes they need a little help with the breathing. mines has an inhaler now which I find helps - maybe something you should ask about..
NO!!!! take him into the hospital! You should never not think the worst! Jus to b sure you go in and see whats really going on! especially is he has suffered those in his past as anew baby!