There is the option to remove a contact from a location without deleting that contact from the database, or the option to delete the contact entirely. Once viewing the location details in the pop up window click on the Contacts tab.
Select the contact you would like to remove and their details will appear in a second window. Click the small green pencil icon to enable editing. Scroll to the bottom of the details page and select the small red trash can icon. You will be given the option to either Delete From Location or Delete Contact. Both actions will ask you to confirm.