Numerical study was carried out on a transonic compressor rotor, NASA Rotor 37, with different configurations of circumferentially contoured endwalls to understand the fundamental flow mechanisms of casing treatment effect. The result showed the wall contouring above the blade leading edge had significant stall margin improvement and insignificant efficiency and pressure ratio drops. By applying the second invariant Q, two major vortex structures were observed near the stall conditions. One was tip leakage vortex and the other was rotating instability vortex. The rotating instability vortex seemed to be induced by the interaction between the inflow and reverse flow near the endwall. In the case with wall contouring, enlarged clearance gap encouraged the tip clearance flow and relieved the interaction that caused the rotating instability vortex. Alleviation of the vortex led to the improvement of stall margin as a consequence.