I’m older and have a kid so my priorities are probably very different to yours ut as a general rule I would think about the following:
Location, is it a 2 hour daily commute? That 2 hours adds up to a big chunk of your time that could be used for other pursuits or further development.
People and environment - You are going to be interacting with these people more than anyone else in your life.and spending more waking hours there than anywhere else perhaps even than home.
What sort of clients do they have? Getting some good names under your belt, even at the sacrifice of doing exactly what you want can make it easier to get jobs in the future.
I’ve been contracting for the last few years though recently started a new perm job and have found every job to be very different to the next. I have done UI and branding for an engineering software firm, personas for a big data institute, User analysis for a Software reporting company and now wireframing, UX and FE dev for a CRM/CMS integration agency. I have found that every interviewer has caught on to different aspects of my skills and experience so having a wide range of skills but with a clear objective of what you want and do want can help your interviewers too.