The Ice Age movie in which Ellie is pregnant and has a baby is Ice Age: Dawn of the Dinosaurs.
Ellie's Pregnancy in Ice Age: Dawn of the Dinosaurs
In Ice Age: Dawn of the Dinosaurs, Ellie and Manny are a couple, and Ellie is expecting their first child. This is a central plot point of the film. This makes it clear which Ice Age film features the arrival of their baby.
