Yes, you can give baby goats water in a bottle.
Many goat owners use bottles to provide water to kids (baby goats), especially in their early months. It's often done to ensure they are getting enough fluids, particularly if they aren't readily drinking from other sources. Just be sure the bottle and nipple are clean. Introducing the bottle gradually helps the kid adjust to the taste and drink willingly.