Hobbies and the tenure-track

I'm thinking about taking on a big hobby/project that is tenuously related to my research. Of course, I know that I don't have much time for non-research stuff until I have tenure (and probably not even then). So the question is, how close does it have to be to be synergistic with my research? If it's interesting enough to me, should I do it even if it slightly hurts my (other) research? Does the fact that I am losing sleep thinking about it mean that I should take it on, or that I shouldn't?