The relationship between the microstructure and the yield strength after T6 tempering was investigated using 6061 aluminum alloy manufactured at various levels of temperature and strain rate during hot forging. Non-recrystallized structures (continuous recrystallization structure) were formed by hot forging at low Zener–Hollomon parameter (Z parameter) conditions, which consisted of fine grains surrounded by high angle grain boundaries and contained low angle grain boundaries inside. Increasing the Z parameter formed fine-grained structures, resulting in increased yield strength. Increasing further the Z parameter formed recrystallization, having coarse recrystallized structures (discontinuous recrystallization structure) of hundreds of micrometers in diameter with high angle grain boundaries, resulting in significantly reduced yield strength. The yield strength of the material with recrystallized grain structures was less dependent on the grain size. On the other hand, the yield strength of the material with non-recrystallized structures was severely dependent on the grain size, roughly in accordance with the previous data. Subgrain strengthening appeared to be more effective than recrystallized grain strengthening. In consideration of the effect of the texture on the yield strength using Schmidt factor, τ′crss (the value equivalent to critical resolved shear stress in the slip direction. τ′_CRSS − τ′_CRSS0 = k′d′^−m′, τ′_CRSS = s·σ_0.2, s: the averaged Schmidt factor in the tensile direction. d′: grain sizes in the slip direction) was less dependent on d′. In consideration of the texture, the yield strength of 6061-T6 is essentially less dependent on the grain size as reported previously.