What precedent? Congress did not assume sovereignty over the District until it was ready to move there in 1800. Until that happened, that territory still belonged to Virgina and Maryland. I agree that retrocession would be a desirable course of action. However that gets complicated by the 23rd Amendment. If you remove all the residents of D.C., who gets to decide who those 3 electors D.C. has under the 23rd are? Does Congress get to pick them?
Under the 23rd Amendment, Congress determines the method of appointment of electors. In the legislation that proposed combining DC with Maryland for congressional voting, Congress would determine that no electors be chosen from DC.
Or perhaps they could apportion them on the basis of the national popular vote.