Jerry, yes, that's the flipside I worry about. Or rather, I'm not so much worried about losing my own Faith (although it is a real concern for the obvious reasons), but I'm way more concerned about raising kids in a mixed household. It would be my responsibility as the mother to make sure that my kids are strong in their Faith, and grow up to be responsible, strong Catholics. If there wasn't the correct emphasis on religion from my spouse, that job could be much more difficult, especially in the teenage years: "Oh, well, Dad doesn't care that much about religion, so why should I?" type of response. For this reason, I think it would be difficult for me to build a long-term relationship with a non-Catholic because there are just too many variables, but I know that every time I think I have a plan, God will surprise me with a twist, so I'm definitely not closing my mind or heart to the possibility of a mixed marriage. No matter how much I'd really like to avoid that decision, God's Will be done in all things.
Yes, raising the children in the faith is very much a valid concern, as well. Even when the non-Catholic parent agrees to allow the children to be raised in and practice the faith, often it does not turn out that way.
Regarding your comment concerning "emphasis on religion from my spouse" -- Keep in mind that a spouse whose emphasis is on the wrong religion may present far more hurdles than one who ignores religion.