That’s a good question and recently I was wondering the same. When I used to work in agencies a general rule of thumb for browser support was around >3% and to support more we charged extra. As for resolution, it was all 960px back then (I’m showing my age) and these were for sites that had about 100,000 visitors on a good day.
More recently, I was discussing the same thing with the developers where I work and asked if 3% would be a reasonable cut off, as we were running into users that had nasty little low res windows machines that were 1024x600. I work in education, and schools buy the cheapest computers they can.
The product has over a million users. 3% would mean 300,000 users. 300,000 users running into trouble would put a significant load on support staff.
Basically, you have to way up the impact of not supporting a certain size, and a simple metric of % of users affected is insufficient. You need to weigh up how severely they will be affected on top of the number.
I hope those anecdotes help.