The speeding up of this track is also evident on the edited mono version that was released as a single a month or two before the album's release, and on the French CBS EP 5660. Significantly, the accompanying edited version of "Pledging My Time" is at correct musical pitch on both of these releases.
For those interested, the editing on these 45 rpm releases removes two verses from each of these songs. "Rainy Day Women" has verse 3 edited out, and the track then fades out in the instrumental section between verses 4 & 5. "Pledging My Time" has verse 2 edited out, and then fades out at the end of verse 4 - the fade starts while Dylan is actually still singing.
For more information about Dylan's 1965-66 singles , see this page.