What I am seeing is there is no way to batch delete custom permission sets when they have people assigned to them. It can be difficult to do it manually for a nation with a lot of control panel users.
Thank you for the suggestion. Right now it is a manual process to delete the permission set from a person which makes removing the permission set difficult. We will consider looking into this but don't have a timeline at the moment.