If the object does not move at constant velocity, then it will have different velocities at the initial and final locations. According to work-kinetic energy theorem, the external force will impart some extra kinetic energy. But we associate potential energy to the forces like gravitational force, spring force and coulomb force. So the external agency should not impart any kinetic energy when the object is taken from initial to final location.