The definition of high altitude illness begins at 8,000 ft so there is not a concern until over that, assuming your infant is otherwise well.
Steve C is correct, there are suggestions that smaller children are more susceptible to high altitude illness. May have something to do with how the arteries in their lungs respond to the atmospheric changes.
No offense to asking your pediatrician, but they may not be knowledgeable about this unles they have a specific personal interest, or live in/near a climbing/altitude type community.
Here is some expert advice. This link is from the International Society for Mountain Medicine, Consensus Statement on Children at High Altitude. Much of it is medical jargon, but there are some practical tips scattered in there as well.
Harvey
http://www.ismmed.org/ISMM_Children_at_Altitude.htm