You can remove users from an Access group at any time. You can do this in 2 ways either through the Access Rights tab or through the individual user's setting.
Remove a user from a role through Access Rights
Click on the Admin Gearwheel, select the Access Rights, then All Roles. Scroll down to the Group that you wish to remove the user form. Click on the number of users to the right of the role name, then click on the bin to remove them from the group.

This can also be done in bulk by using the checkboxes to select the users you want to remove then under tools selecting Remove Users:

Remove a user from a role through the User settings
Click on the Admin gearwheel and then on Users. Select the user that you want to edit and then click on the Rights tab. Click on the bin next to the role that you wish to remove from the users.
Alternatively, you can remove them from all roles by pressing Remove All Roles.