I agree. "Baby" can sound a little minimizing to the woman. But if both like it, no biggie. I wonder if a man would like it if his wife/girlfriend called him "her little boy."
Why not? At 41 and 3rd of 7 I'm Mom's baby broy. 6'4" I'm my sister's baby brother!