Since the coronavirus disease 2019 (COVID-19) emerged in Wuhan, China, in December 2019, it has rapidly spread worldwide, and the number of cases is also increasing in Japan. The number of COVID-19 cases in Japan in the early stages was not uniform, and cases were largely concentrated in several prefectures. There was a strong, positive correlation between the distribution of COVID-19 cases and the number of foreign travelers as well as Chinese travelers, at prefectural level, with coefficients of 0.68 (P < 0.0001) and 0.60 (P < 0.0001), respectively. Moreover, phylogenetic tree analysis revealed that all the registered SARS-CoV-2 detected from January 23 to February 29, 2020, belonged to Chinese lineage, while those detected in March 2020 belonged to American and European lineages. Only 14 (20.3%) were infected outside Japan; however, the majority of the cases (79.7%) were infected domestically. In conclusion, a higher number of COVID-19 cases were identified in prefectures with more Chinese travelers, supporting the importance of enforcing policies that restrict the entry of overseas travelers to control COVID-19 spread. These findings highlight the risk of secondary transmission in the community caused by apparent or silently imported cases.