Yeah, you're totally not alone. =)
Just like most visual and graphic designers, many UXers have a good eye for detail—we're working in a mostly visual medium, and it's one way we can flex our muscles and prove our skillz, plus the emotional impact of a design can be pretty nuanced. But as you say, it can really get in the way sometimes, and waste precious time that is much better invested elsewhere.
I find it helpful to split my wireframes and mockups into three types, with different levels of fidelity to the final design:
For thinking - by hand, very sketchy and blocky, most of the time using shorthand placeholders for content. I'm usually the only one who usually sees and works with these as they're completed in a flurry as I work towards getting my thoughts in order and articulating a possible solution to a problem, but this is also great for collaborative sessions or knocking through some ideas with others.
For communicating - Once I've settled on a proposed direction, I set up something with a slightly longer lifespan with the other designers around me. These are still very sketchy, but include real content and more attention to detail (including some annotation and references), and are still mostly done by hand unless it's a larger project with a lot of iterations for which I'll often invest in digitising them in Balsamiq or Axure. If it's a small adjustment of an existing design I might jump into Photoshop or Illustrator or HTML and move things around a bit.
For testing - these screens need to work without explanation because they're being tested in usability sessions, so will need real content and plenty of detail. I've tested with hand sketched designs on paper or clickable prototypes, all the way through to pretty much finished designs. Depends on where we're up to in the process, and what we're actually testing.
It's amusing to compare it with the Uncanny Valley concept in robotics and 3D animation; if you're working with something that looks relatively finished, people seeing it have a higher expectation of it being pixel perfect or else it just gets weird and distracting.
Balsamiq made such a breakthrough for digital mockups when they embraced the sketchy, imperfect style. =)
Sounds like you should stick to your guns and steer away from perfecting every pixel!