Brief Answer:
Don't need to worry, it is self limited...

Detailed Answer:
Hello,
Coming directly to your query, i would like to ask you few questions like whether colour of semen is red brown or only brown?? whether it is the first time you have noticed? whether you have any signs of urinary tract infection? Whether you are taking any medications for any chronic illness?

I would like to tell you that blood in semen (hematospermia) can be present due to various causes like urinary tract infection, prostatitis, vesiculitis or sometimes due to idiopathic.

Also want to add that if you have found it brown semen only once or twice, then you don't have any problem and in most cases it is self limited, so you need not to worry much about it.

So at this point, it seems it is of no emergency and don't require any investigation or treatment.

---