'I was friends with them': Lawyer has mixed feelings trying to boot Fani Willis from case