When you're clicking the link you are posting back to the server with the GET data "action" and "id". On the Memory.php page you load up the array for the user from stored data. It may be like (0 => 1, 1 => 3, 2 => 5), so don't mix up your key-value pairs.
To add to the array just tack on a blank index:
$arr[] = $_GET['id'];
sort($arr);
To delete it, you must determine the numeric index of the id:
$x = array_search($_GET['id'],$arr);
if ($x !== false) { unset($arr[$x]); }
sort($arr);
use the sort() function to re-index your keys after you make changes to the array.