As the medical demand is projected to increase along with the population aging in Japan, the geographical distribution of physicians is a significant concern for society and policymakers. To implement effective measures on geographical physician distribution, this study aimed to describe and compare the distribution of physicians by specialty in 2000, 2010 and 2016 in Japan, and examine whether practice setting was associated with distribution. To quantify the geographical physician distribution by specialty, we calculated the Gini coefficients of physicians working at clinics or hospitals in 2000, 2010, and 2016. We used the basic geographic unit for medical care planning in Japan, a secondary medical area, as the study unit. To show the association between the geographical distribution of physicians in each specialty and their practice setting, we categorized specialties into two groups by the proportion of physicians in that specialty working in hospitals, and showed aggregated Lorenz curves for each category. The overall geographical distribution of physicians appeared to improve during the study period, but varied by specialty. Those in specialties, where at least 90% of physicians work in hospitals such as anesthesiologists and radiologists, were more clustered, as shown by the Lorenz curves and the Gini coefficients. Similar distributional differences were also found even when we excluded physicians working in clinics, meaning that the distributional variation could be explained by other factors than the distribution of hospitals. These results suggest that the nature of practice in each specialty strongly affects the geographical distribution of specialists.