If it is something I think is stupid, then we have a problem. Ex. Bowling. I think it's stupid. So if someone were in a league and played 4 nights a week, and spent all weekend attending to waxing his bowling balls, then yes, we have a problem.
I think grownups should have balance in their life and any man who is so into a sport that it interferes with his grown up intimate relationship, then yes, there is going to be a problem.
Most women are not usually that into a sport that it skews their whole life.
If you mean, I like ice skating and he likes golf, and we both regularly do our own sport, and neither is good at the other's sport, that is fine with me.
Identifying the main hazard of an intense enthusiasm in playing sports as its interference "with his grown up intimate relationships" was a good way to say it.