Good question. For some reason (vB, say no more) Latest Activity is only displaying new threads as you say, but that is not the expected behaviour. I've logged a ticket to get them to explain themselves, but in the mean time, this is what I do.
If you look at this image, the speech bubble that is blue indicates that there are unread replies, whereas the grey one indicates that there are not.
If for some reason you want to clear one without reading it, you can click the 'Mark Channels Read' link at the bottom of the homepage. Does that all make sense?