Great to hear. I love collaborative design sessions!
Several sessions of 4 hours seems like a lot? Great that you have the time for it though. Watch out for low blood-sugar levels, which tend to make the 'group discussion' parts a bit more animated than ideal.
The brainstorming side of things is where I believe the magic happens—the benefits of collaborative design sessions come from generating the ideas together, being included in the design process with hands on activities, and busting through "designers block" with the benefit of different roles/perspectives. The consensus emerges from the group reaching the same conclusions. I wouldn't necessarily separate the idea selection from the brainstorming. Are you avoiding the brainstorming mode because of strict project design constraints?
Here are two of the most common approaches to shaping the activities and process that combine the group brainstorming with the selection and refinement of ideas:
A 'Group whiteboard sketching' approach is better suited to ideating within certain constraints. You can sketch up the general/existing layout on the whiteboard, and take turns sketching/explaining an idea by removing or adding to the previous design (or passing if they don't have anything to add), then discussing each idea as a group before moving to the next sketcher. This tends to generate less innovation, but facilitates working within the constraints. Lots of good discussion comes out of it! And time can slip away while you're having fun.
Alternately, a 'Design Studio method' where you gather all the resources you need (people, paper, drawing materials, reference material) in one room, and give the team a problem to solve. You then work through steps of a) sketching for a few minutes on your own, b) explaining your sketches for a minute or two each, and then c) critiquing each idea as a group for a few mins. Rinse and repeat a couple of times. After a couple of rounds, you may start seeing some convergence, and the facilitator might increase the design fidelity in the following round (more fine detail). This method doesn't do so well when the project has strict constraints where divergent ideas aren't helpful, but you can focus in on specific parts of a design quite nicely.