Check if $GET['action'] is 'add' or 'delete', and then perform the appropriate action with $GET['id'].
Note that it might be easier (assuming that ID's are unique) if you store them such as:
$array[$_GET['id']] = true;
That way it's simple to delete them later:
unset($array[$_GET['id']]);
Also note that instead of storing boolean TRUE, you could store any other data associated with each ID that you wanted to.