Maybe the replaced potentiometer could be bad from the start. My recommendation is to see if you can find a good quality potentiometer and see the drifting goes away again
open