What flight controller are you using? I had this issue with a full Naze board, and once I covered it with foam, the problem was solved. Turns out that at low altitude, prop wash can affect the barometer and throw things out of sync. Of course this wouldn't apply if you were flying in rate mode with the baro disabled.