Paul compares not having Christ with slavery and having Christ with being free. For the Jews, they were like children who were to inherit an estate. As long as they were children, the inheritance is held in trust. In Old Testament times the Jews had the promise that their Messiah was coming, but could not enjoy the full benefits until he actually arrived. For Gentiles, they worshipped false gods and observed feasts and practices that were useless. Neither group should have any reason for returning to slavery.
We return to slavery anytime we go back to our old way of life without Christ. It might be a religious ritual or behavior or it could be a sinful behavior. It was the deliverance from sin and slavery that attracted us to Jesus; why go back?