Writing PHP extension, mind boggled by pointer madness

[sneakyimp]

I'm reading Sara Golemon's book, Extending and Embedding PHP so that I can try and write a PHP extension (long story...).

I've been looking through a lot of PHP code examples (JSON, serialize functions, etc.) and am baffled by the pointer usage. I coded in C at university years and years ago but never really grokked pointers. I'm hoping someone can help me wade through this section of the book and better understand what is going on. I'll italicize the parts I find confusing:

====
DATA RETRIEVAL
In order to retrieve a variable from userspace, you'll need to look in whatever symbol table it's stored in. The following code segment shows using the zend_hash_find() function for this purpose:

{
  zval **fooval;

  if (zend_hash_find(EG(active_symbol_table),
        ("foo", sizeof("foo"),
        (void**)&fooval) == SUCCESS) {
    php_printf("Got the value of $foo!");
  } else {
    php_printf("$foo is not defined.");
  }
}

A few parts of this example should look a little funny. Why is fooval defined to two levels of indirection? Why is sizeof() used for determining the length of "foo" ? Why is &fooval, which would evaluate to a zval***, cast to a void**? If you asked yourself all three of these questions, pat yourself on the back.

First, it's worth knowing that HashTables aren't only used for userspace variables. The HashTable structure is so versatile that it's used all over the engine and in some cases it makes perfect sense to want to store a non-pointer value. A HashTable bucket is a fixed size, however, so in order to store data of any size, a HashTable will allocate a block of memory to wrap the data being stored. In the case of variables, it's a zval being stored, so the HashTable storage mechanism allocates a block of memory big enough to hold a pointer. The HashTable's bucket uses that new pointer to carry around the zval and you efffectively wind up with a zval** inside the HashTable. The reason for storing a zval* when HashTables are clearly capable of storing a full zval will be covered in the next chapter.

When trying to retrieve that data, the HashTable only knows that is has a pointer to something. In order to populate the pointer into a calling function's local storage, the calling function will naturally dereference the local pointer, resulting in a variable of indeterminate type with two levels of indirection (such as void**). Knowing that your "indeterminate type" in this case is zval*, you can see where the type being passed into zend_hash_find() will look different to the compiler, having three levels of direction rather than two. This is done on purpose here so a simple typecase is added to the function call to silence compiler warnings.

...

If zend_hash_find() locates the item you're looking for, it populates the dereferenced pointer provided with the address of the bucket pointer it allocated when the requested data was first added to the HashTable and returns an integer value matching the SUCCESS constant...

Any help coming to grips with this bit of info would be much appreciated. I find the double- and triple-indirection and casting of triple indirection to double indirection (void**) especially confusing.

[laserlight]

sneakyimp wrote:

I find the double- and triple-indirection and casting of triple indirection to double indirection (void) especially confusing.

From what I see, the idea is that if you want to find an object of type T, you need to create a T to point to it. You then pass the address of the T to zend_hash_find as zend_hash_find needs to modify the T to point to the object found and yet have this change reflected in the caller (i.e., simulating pass by reference where what you're conceptually passing is a T). Thus, you have a T that is converted to a void. The thing is, you want to find a zval, i.e., T is zval, so you create a zval and then have a zval* that is converted to a void.

Now, conversion of a pointer to object type (whether it be zval, zval or zval) to void* happens implicitly, but this does not hold true when converting to void. Hence, the type cast is required since the type of the corresponding parameter in zend_hash_find is void.

[NogDog]

I.

Hate.

Pointers.

(I know, zero help to you, but one reason I haven't touched C in at least 15 years now. 🙂 )

[sneakyimp]

laserlight;11044067 wrote:

From what I see, the idea is that if you want to find an object of type T, you need to create a T* to point to it.

Is this really the case? If I want an int, Can I not define an int and then pass-by-reference to the search function which will modify the address contents, no?

int search_result;
if (search_function(&search_result) == SUCCESS) {
  cout << search_result << endl; // this should output an int, no?
}

I can certainly see an alternative where I explicitly create a pointer, but I'm still baffled why we are declaring two levels of indirection.

laserlight;11044067 wrote:

You then pass the address of the T to zend_hash_find as zend_hash_find needs to modify the T to point to the object found and yet have this change reflected in the caller (i.e., simulating pass by reference where what you're conceptually passing is a T).

Ok this is the sort of english-that-does't-really-elucidate-the-concept that I'm having trouble with. Let me know if this is what you are suggestion:

// define pointer to a object of type T
T *myPtr;

// define your search fn
void my_search_fn(T *t_ptr) {
  T search_result; // NOT a pointer, but an actual T object
  // do some kind of search to find search_result, reassign t_ptr to the memory address 
  t_ptr = &search_result;
}

// perform the search
my_search_fn(myPtr);

laserlight;11044067 wrote:

Thus, you have a T that is converted to a void

Lolwut? I'm not following this leap to a second level of indirection?

laserlight;11044067 wrote:

The thing is, you want to find a zval, i.e., T is zval, so you create a zval and then have a zval that is converted to a void.

This is also confusing. It's not clear to me why I want to find the address of a zval object rather than just the zval object itself. In the original example, I don't understand why fooval is a zval rather than just a zval. I'm guessing it has something to do with the function zend_hash_find expecting a void for its fourth parameter. I also don't understand why it expects a void and not just a void.

laserlight;11044067 wrote:

Now, conversion of a pointer to object type (whether it be zval, zval or zval) to void happens implicitly, but this does not hold true when converting to void. Hence, the type cast is required since the type of the corresponding parameter in zend_hash_find is void.

I truly appreciate that clarification, but don't really understand why the parameter is pData is declared as void** and not just void. I'm guessing it has something to do with the compiler being finicky about what actions we migh want to perform on pData? Or perhaps it's two levels of indirection because we want to return a reference to some object, possibly an array or complex object on the heap so we need a void* and then we are also passing by reference so we have the second level of indirection? i.e., one level of indirection to accommodate a complex heap object, the other level to allow passing by reference to the calling function?

[sneakyimp]

NogDog;11044069 wrote:

I.

Hate.

Pointers.

(I know, zero help to you, but one reason I haven't touched C in at least 15 years now. 🙂 )

Not especially helpful for coding, but definitely helpful for my anxiety. Pointers are so dumb. I'm looking for some kind of overview about how to cope with them, but they all seem to devolve into a million specific examples rather than offering any overview. In particular, I fail to see the need for multiple levels of indirection.

[laserlight]

sneakyimp wrote:

Is this really the case? If I want an int, Can I not define an int and then pass-by-reference to the search function which will modify the address contents, no?

You could, if the hash table interface were designed for that, but it apparently isn't. There's a plausible reason why not though: since the hash table can store objects of arbitrary size, it is possible that an object that is relatively expensive to copy might be stored in it. In that case, for efficiency you might prefer to deal with pointers to that object rather than copy the entire object.

sneakyimp wrote:

Let me know if this is what you are suggestion:

// define pointer to a object of type T
T *myPtr;

// define your search fn
void my_search_fn(T *t_ptr) {
  T search_result; // NOT a pointer, but an actual T object
  // do some kind of search to find search_result, reassign t_ptr to the memory address 
  t_ptr = &search_result;
}

// perform the search
my_search_fn(myPtr);

No, that won't work: t_ptr is local to my_search_fn. Assigning to it does not change myPtr. Your my_search_fn function has no net effect.

sneakyimp wrote:

Lolwut? I'm not following this leap to a second level of indirection?

Perhaps you need a simpler example then. Here's a C program that calls a function to set the value of an int:

#include <stdio.h>

void foo(int *x)
{
    *x = 123;
}

int main(void)
{
    int n = 0;
    foo(&n);
    printf("%d\n", n);
    return 0;
}

Here's a C program that calls a function to set the value of a pointer:

#include <stdio.h>
#include <stdlib.h>

void bar(int **x)
{
    int *y = malloc(sizeof(*y));
    if (y)
    {
        *y = 123;
        *x = y;
    }
}

int main(void)
{
    int *p = 0;
    bar(&p);
    if (p)
    {
        printf("%d\n", *p);
        free(p);
    }
    return 0;
}

In the first program, it is obvious why you need a pointer to an int: you want to modify the int in foo. If you only passed an int, then x = 123 would still work, but in main n would remain with the value of 0. In the second program, you need the second level of indirection for the same reason: you want to modify the pointer to int in bar. If you only passed a pointer to int, then the malloc call and assignment would still work, but in main p would remain a null pointer.

sneakyimp wrote:

It's not clear to me why I want to find the address of a zval object rather than just the zval object itself.

It is not clear to me either, but according to the author:

• In the case of variables, it's a zval* being stored

• The reason for storing a zval* when HashTables are clearly capable of storing a full zval will be covered in the next chapter.

sneakyimp wrote:

I truly appreciate that clarification, but don't really understand why the parameter is pData is declared as void** and not just void. I'm guessing it has something to do with the compiler being finicky about what actions we migh want to perform on pData? Or perhaps it's two levels of indirection because we want to return a reference to some object, possibly an array or complex object on the heap so we need a void and then we are also passing by reference so we have the second level of indirection? i.e., one level of indirection to accommodate a complex heap object, the other level to allow passing by reference to the calling function?

It could have been a void*. My guess is that it is a design decision to remind the programmer that a pointer to a pointer is expected, and the reason for that would be related to your guess about "passing by reference so we have the second level of indirection".

[Weedpacket]

• Every type in C has a fixed size in bytes, so if you want to store something that comes in various sizes you have to allocate space for it (and remember how much you allocated), and use a pointer type to refer to it.

• Everything is passed by value, so if you want to modify something you have to pass a reference as the value - i.e., a pointer.

• For nearly 20 years, (long enough to result in a lot of compiler backward-compatibility problems) you could not directly pass a struct to a function; you had to pass a pointer (which, in light of the previous entry, was probably what you wanted anyway).

• Likewise when returning a struct.

• Speaking of returning, you can't just whack together an arbitrary tuple of values when you want to return more than one (such as an success/error code and the actual result); instead of declaring a struct for each occasion when you wanted to (which would then need to be examined by the calling function to find out what happened), it was easier to return one of the values and pass references to variables that would be modified to contain the others (but everything is passed by value, so....)

Even without the architectural reason for pointers (C is pretty low-level; a pointer is just a memory location and its type just says how many bytes to starting from that location belong to it), the pointer ends up covering a lot of ground. On the plus side, it simplifies the language; on the minus side, it leaves exposed the program's complexity. On the plus side, it solves a lot of implementation needs. On the minus side, it means pointers show up everywhere.

[sneakyimp]

Thanks so much for the clarifications and examples, laserlight. And Weedpacket thank you for the bullet pointed list of important concepts. Laserlight's examples are helpful, especially the second one.

I see that PHP's zval declaration has pointers:

typedef union _zvalue_value {
    long lval;
    double dval;
    struct {
        char *val;
        int len;
    } str;
    HashTable *ht;
    zend_object_value obj;
} zvalue_value;

With regard to Weedpacket's first point, I was previously laboring under the impression that this structure alone would be enough to pass data of an arbitrary size, but I understand from Laserlight's second example that simply passing a zval to a function does not permit that function to modify it, possibly returning a search result or initializing it, etc. This certainly helps to clarify why one would want to pass a zval's address to a function:

zval myZVal;
myZVal.lval = 456;

void search_fn(zval *return_val) {
  // do some search, change data etc.
  *return_val.lval = 123;
}

search_fn(&myZVal);

Weedpacket wrote:

Even without the architectural reason for pointers (C is pretty low-level; a pointer is just a memory location and its type just says how many bytes to starting from that location belong to it), the pointer ends up covering a lot of ground. On the plus side, it simplifies the language; on the minus side, it leaves exposed the program's complexity. On the plus side, it solves a lot of implementation needs. On the minus side, it means pointers show up everywhere.

This kind of explanation why is very helpful for my morale. It's sort of like telling a soldier why he's fighting a terrible war and living in a muddy hole.

In particular, "everything is passed by value" is a very helpful concept. I vaguely recall that the stack is the mechanism used for this.

Must one pass the address of an object for a function to modify it?. More specifically, if you want a function to modify one of the parameters you supply, is it necessary to use the ampersand in front of a parameter, or can a function effect modifications on a data structure if you simply pass a pointer (or pointer to pointer, etc.)? It would be awesome if we could say something like "a function could never modify a var you pass to it unless you supply the ampersand." I suspect this is not the case.

[laserlight]

sneakyimp wrote:

Must one pass the address of an object for a function to modify it?

No, e.g., a file scope variable could be modified from within a function without its address being passed to the function. Of course, you're probably thinking of a variable that is not already in the scope of the given function, in which case the answer is yes: if you want to modify an object whose identifier is not in scope from within that function, then you need to pass a pointer to that object as an argument.

sneakyimp wrote:

More specifically, if you want a function to modify one of the parameters you supply, is it necessary to use the ampersand in front of a parameter, or can a function effect modifications on a data structure if you simply pass a pointer (or pointer to pointer, etc.)?

No, it isn't. For example, if your intent is to modify the contents of a dynamically allocated array, you might pass a pointer to the first element of the dynamically allocated array. This pointer could have been obtained by calling malloc, not by the use of the address-of (ampersand) operator.

sneakyimp wrote:

It would be awesome if we could say something like "a function could never modify a var you pass to it unless you supply the ampersand." I suspect this is not the case.

If you "supply the ampersand", i.e., if you take the address of the object, then the actual argument that you're passing is a pointer, not the object that you're conceptually passing. Furthermore, from within the function, the formal parameter could be modified unless it were declared const, e.g.,

void baz(int n)
{
    n = 123; /* valid, though it has no net effect */
}

void bazz(const int n)
{
    n = 123; /* invalid, but to the caller the const makes no difference */
}

[sneakyimp]

Thanks again for clarifications. I've managed, despite these baffling pointers everywhere, in making some progress on my project. I am feeling better, but expect that a true understanding will only come with time spent coding with them.

[laserlight]

sneakyimp wrote:

Thanks again for clarifications.

You're welcome 🙂

sneakyimp wrote:

I've managed, despite these baffling pointers everywhere, in making some progress on my project. I am feeling better, but expect that a true understanding will only come with time spent coding with them.

Well, at the time when I was being introduced to C++ at pre-university level, an older friend of mine who apparently did some C or C++ programming courses in his younger days preemptively warned me about all the trouble I was going to face with pointers and pointer arithmetic. Personally, I never understood why he had so much problems with that: with a single level of indirection, it seems pretty straightforward. Things get a little tricky when dealing with two levels of indirection, but it should be manageable, though I still try to avoid it where feasible. The problem comes when I am faced with the work of a three star programmer, which is why I was disturbed when I glanced over the code snippet from post #1, but after I realised that the third level of indirection was only in passing (literally), it was not so bad.