Protesters gathered in front of the governor's house in Kandahar after around 3,000 families were asked to leave the colony, according to the former government official who witnessed the crowds.
Footage from local media showed crowds of people blocking a road in the city.
The affected area is predominantly occupied by the families of retired army generals and other members of the Afghan security forces.
The families, some of whom had lived in the district for almost 30 years, had been given three days to vacate, the official, who had spoken to some of those affected, said.
Taliban spokesmen did not immediately respond to a request for comment on the evictions.