I love this topic. I've debated it at length several times as a result of choosing this Discourse platform (with its infinite scroll) as my community platform of choice.
There are a few issues that are frequently raised.
In some cases infinite scroll messes with screen readers, but not the most popular ones, so that is possibly an issue for a very small percentage of users (but most likely none).
Focus is lost when navigating back
Discourse actually handles this so it's not an issue. (Scroll down through this page a while, click on a link, click the back button and you'll find that you're exactly where you were when you clicked the link.)
It's not expected behaviour
Fair enough in the old days, but now... Facebook
Unable to access footer content
We made sure that all necessary home page footer content was accessible through the main navigation, and in the footer of posts.
Inability to get to the end
This argument has always confounded me somewhat – there is nothing of value at the end so why do you need to get to it?
I don't believe that infinite scroll is always good or always bad, but rather something that should be decided on a case by case basis by weighing up the pros and cons.
NNGs stance on infinite scrolling is that it's not recommended for goal-oriented finding tasks, but that it is "advantageous for content that streams constantly and has a relatively flat structure, where each unit of content belongs at the same level of hierarchy and has similar chances of being interesting to users".