Imām Ibn al-Jawzī (d. 597/1201) summarized the scholars' opinions in his book "Zād al-Masīr" as follows:

1. They are the People of the Book. [Ibn ʿAbbās, Abū al-ʿĀliyah, Qatādah, Muqātil] Those who hold this view explain that the People of the Book argued with the Prophet ﷺ, saying, "Why did you abandon the direction of prayer towards Bayt al-Maqdis? If it was misguided, then you have defied God by adhering to it, and if it was guided, then why have you turned away from it?" Qatādah also relates that they said, "The man has longed for his ancestral house and the religion of his people."
2. They are the Arab polytheists. [Al-Suddī from his teachers] Those who hold this view explain that the polytheists argued, saying, "He has returned to your direction of prayer, and soon he may return to your religion."

Who are the "people" who would have no argument?