Xbox One Friends & Followers Lists Explained!
Unfortunately, there is no set way to remove followers from your list. The “Block unblock” method has been reported to work, but its nothing official. There is no mention of removing followers. If you haven’t already, you should post that this feature be added in a future update.