The oxygen level in the air is lower at high altitudes. Short term stay (less than 2 weeks) at around 12,000 feet (3500 meters) of altitude will usually not affect your baby.
Long term stay in high altitudes may affect the size of your baby. Also, nausea can often worsen if you travel to higher altitudes.
Airplane travel is not a cause to worry, though, since all commercial planes have pressurized cabins and the oxygen levels remain adequate, even if the plane is at very high altitudes.