You've already got the "what", I'm just offering a couple of additional viewpoints as to the "why". Doesn't hurt to look at it from another couple of angles.
If you like you can think about how a compiler for a language like Java or C++ or C# operates. (This is for statically-typed languages, not dynamically-typed languages like PHP, but the behaviour from the language-user's perspective is the same). The compiler needs to allocate memory for each entity created (of course), so it needs to work out how big the object will be.
For primitive properties it's easy enough: bytes are one byte, integers are four bytes, and so on. Add those up and you get the total size of the object, right? But what if one of those properties is itself an object? Recurse and add the calculated size of the property? But what if the object can contain an instance of its own class? Infinite recursion?
Instead, for non-primitive properties, what is actually stored in the object is the memory address of where the property's value is to be found. And a memory address is a primitive type (four bytes on a 32-bit processor, eight bytes on a 64-bit processor), so it can be stored directly and the total size of the object can be calculated and allocated.
Different languages hide this to different degrees. C doesn't hide it at all (and gets much of its power from the fact), C++ would like to hide it but it's C underpinnings won't let it. Java hides it thoroughly, while C# lets you mess with it so long as you explicitly state that what you're doing is "unsafe" (with the .NET framework providing classes that encapsulate most of the reasons you'd want to).
The difference between those languages and PHP is that in those languages variables have types, while in PHP the type belongs to the value stored in the variable. But reference is still the key word.
Consider a doubly-linked list structure, where every element is an object that has a $next field (which is null for the last element in the list) and a $previous field (which is null for the first element in the list). The fields work like you'd expect: $foo->next is the next element after $foo and $bar->previous is the previous element before $bar. But this can only be achieved if objects are stored and passed around as references - because if $foo->next really did contain the next object (instead of just a reference) $bar->previous really did contain the previous object, then $foo->next->previous would blow up in your face.