I think this is a good idea, and a better option than doing testing on individual users and reiterating after each test.
However, the first test you describe is a perfectly valid form of UX testing, known as a focus group, but you do have to be aware of the strengths and weaknesses of the process. Primarily, focus groups will give you a good insight into what your target audience might be while giving you good ideas on concepts and patterns that are the basis of your UX. Focus groups are also great for reaching a larger number of people at once.
They cannot, however, give you very good detailed feedback, nor solid feedback on usability in general. What’s more, focus groups will inherently give more biased feedback, as humans will bias there responses based upon the group surrounding them. Removing additional participants allows for more in-depth answers, a better insight to general usability standards, and less (though not a complete absence of) bias in responses.
Depending on your goals, one option might be better than the other.