Question about Endianness and byte length

sneakyimp

I'm working on a PHP Extension which is supposed to serialize PHP data into AMF3 format. The spec on page 2 describes a few data types that it deals with:

U16 - An unsigned 16-bit integer in big endian (network) byte order
U32 - An unsigned 32-bit integer in big endian (network) byte order
DOUBLE - 8 byte IEEE-754 double precision floating point value in network byte order (sign bit in low memory).

I'm wondering about ints in PHP and how they might vary on different CPUs. Do we have any guarantee about the byte length of a PHP int? Is it always 4 bytes or could it be 8 bytes? Or two bytes? Is it really an int or a long?

What about Endianness? I've read quite a bit of AMF3 conversion code and they appear to try and account for endianness when dealing with floats/doubles, but not with ints. Are ints affected by the system's endianness? What about longs?

Lastly, can anyone recommend an affordable way to gain access to the various systems where I might test 32-bit, 64-bit systems of both big and little endianness?

Any help would be much appreciated.

johanafm

All of the protocol layers in TCP/IP are defined to be big endian, which means that any LE system needs to swap byte ordering for internet communication. So, if you are using existing functionality for network communication, this should allready be handled. Otherwise you can swap byte order through four macros: host to network short, host to network long, network to host short and network to host long

#if defined(BIG_ENDIAN) && !defined(LITTLE_ENDIAN)

  #define htons(A) (A)
  #define htonl(A) (A)
  #define ntohs(A) (A)
  #define ntohl(A) (A)

#elif defined(LITTLE_ENDIAN) && !defined(BIG_ENDIAN)

  #define htons(A) ((((uint16_t)(A) & 0xff00) >> 8) | \
                    (((uint16_t)(A) & 0x00ff) << 8))
  #define htonl(A) ((((uint32_t)(A) & 0xff000000) >> 24) | \
                    (((uint32_t)(A) & 0x00ff0000) >> 8)  | \
                    (((uint32_t)(A) & 0x0000ff00) << 8)  | \
                    (((uint32_t)(A) & 0x000000ff) << 24))

  #define ntohs  htons
  #define ntohl  htonl

#else

  #error "Must define one of BIG_ENDIAN or LITTLE_ENDIAN"

#endif

There seems to be few guarantees about integer size in PHP. The only thing you know is that it is at least 32 bits, and that it might be 64. But PHP does define PHP_INT_SIZE as well as PHP_INT_MAX

a maximum value of about two billion is the usual value (that's 32 bits signed). 64-bit platforms usually have a maximum value of about 9E18

You should probably note that there is no signed integer in PHP, and that overflow causes automatic conversion to float.

Yes, ints are affected by endianness. 0x1122 might be stored either as (BE) 11 22 or as (LE) 22 11.

No idea what you call affordable, but one possible solution would be to use an "endian switchable" system such as PowerPC. Supposedly you can change endianness through a machine state register flag. You should be able to get one of Apple's old G4 models for 50-200$. But do double check that you really can switch endianness first of course...

And if you want to be thorough, what about mixed endian systems? :p

sneakyimp

Thanks Johana.

I am writing an AMF3 serializer PECL extension. I've played around with amfext and it doesn't really work correctly. I therefore need to concern myself with byte ordering.

I've seen htonl/ntohl mentioned as possibilities but it's my understanding that these are a) not always defined or available and b) limited to 32-bit unsigned integers. As I mention in just a sec, I think I have 32-bit integers sorted.

I have not yet found the definition within PHP that will let me know the value of PHP_INT_SIZE. I'll keep my eyes out for that. Thanks for that tip.

It's also helpful to know that integers are LE/BE and therefore must be reversed if they are to be written from a LE machine. AMFEXT just reverses them without checking the system endianness which should work fine for linux, osx, and windows, but which will probably break on big-endian systems.

I've been in touch with a couple of folks on the PECL-DEV mailing list and there are some good suggestions about how to determine the machine's endianness. This run-time check is probably my favorite:

littleEndian = ((int*)"BAA") | 1;

It was also recommended that I just check for a compile-time define WORDS_BIGENDIAN but I'm wary of this approach because it depends an autoconfigure script being used and properly adding the define, etc. It also doesn't handle architectures that might switch endianness. There is some discussion on the list now about whether there should maybe be macros to deal with endianness.

As it turns out, AMF3 serialization uses a variable-length integer encoding scheme which only permits 29 bit precision. Any int too large or too small gets serialized as a IEEE 754 double. Given that PHP integers are at least 32 bits, this seems to have solved my problems for integers.

I'm worried now about doubles. That's the reason for my other post. I'm guessing that to be thorough, I need to find out how many bytes are involved and to be safe I need to avoid writing any adjacent bytes should I need to cast a double as a char pointer.

You should probably note that there is no signed integer in PHP, and that overflow causes automatic conversion to float.

I'm not sure what this means? Could you clarify? Do you mean "no unsigned integer in PHP?"

And if you want to be thorough, what about mixed endian systems?

Criminy! It never ends.

johanafm

sneakyimp;10960718 wrote:
I've seen htonl/ntohl mentioned as possibilities but it's my understanding that these are a) not always defined or available

But you can easily check this with #if !defined and then proceed to define them yourself.

sneakyimp;10960718 wrote:
b) limited to 32-bit unsigned integers.
(...)
I'm worried now about doubles. That's the reason for my other post. I'm guessing that to be thorough, I need to find out how many bytes are involved

Ok, I've never had to deal with this myself, but since all data is stored as bytes, it seems to me that you'd be able to use the exact same approach to revers endianness for anything, wether it's int, float or UTF-8 characters.
A double is an 8 byte float (according to IEEE 754-2008, also see wikipedia entry for double). As far as endianness goes, wether it's 8 bytes or something else should not matter (except for exactly how the reversal algorithm needs to be constructed). But assume 8 bytes for the sake of an example, and just think about how a decimal number is stored, i.e.

1 bit sign
11 bit exponent
52 bit fraction

1/3 would be represented as 0x3fd5 5555 5555 5555 which means that memory inspection on BE systems would show exactly that. I never did much C, and it's a long long time ago, but on an LE system, the following

double d = 1.0/3;
void *i = &d;
for (int j = 0; j < (sizeof(double) / sizeof(char)); ++j) {
	printf("%x ", *( (char *) i) + (j * sizeof(char)) ));
}

ought to show

55 55 55 55 55 55 d5 3f

What the code does (or is supposed to do anyway), is to store the value of floating point division 1/3, and then step through it one byte at the time to inspect the byte ordering.
The conclusion is that even though you are no longer dealing with integers, all you need to know is the size of what you store, and then perform your "reverse endianness" accordingly.
"Size of what you store" would of course refer to the underlying data type. I.e. for a string it's not the size of the string, but rather the size of the characters: either single bytes (iso 8859-1) or multiple bytes (utf-8).

sneakyimp;10960718 wrote:
and to be safe I need to avoid writing any adjacent bytes should I need to cast a double as a char pointer.

I don't doubt you, but I don't see the problem myself, and I wouldn't mind an explanation.

sneakyimp;10960718 wrote:
I'm not sure what this means? Could you clarify? Do you mean "no unsigned integer in PHP?"

Ah yes, unsigned. And thinking about it, I suppose that shouldn't pose any problems for you.

sneakyimp;10960718 wrote:
Criminy! It never ends.

Hehe. But personally I wouldn't worry about this (but perhaps that's why I don't write extensions).
At least not initially. If you create functions to handle BE/LE and pass your doubles, ints etc through these, all you have to do is add the mixed endian handling there later down the road if you decide it's needed.
And at the very least I'd start by finding out what system(s) use this. If there are any, and if they are still in use, also find out if PHP even compiles on them...

sneakyimp

Thanks so much johanafm. Your input has been very helpful and encouraging.

I've posted my double reverser (sounds like a football play) in the other thread. My machine is of course little endian so the reversal is required and it does seem to be working. I agree it would be awesome to make a generic byte reverser -- let's discuss that in the other thread.

It's great that you mentioned UTF8. Having gotten the double serialization working in my project, my next chore is string serialization and strings are encoded as -- wait for it -- UTF8!! I find myself wondering if I must reverse the bytes within a single UTF8 char inside my string.

I see from this table that Ñ is a two-byte char in UTF8 with an char encoding of C391. I created this little flash program:

var data:* = "Ñ";
// your byte array you want to save
var bytes: ByteArray = new ByteArray();
// this results in data being AMF3 serialized
bytes.writeObject(data);
// set up correct url request using post in binary mode
var request:URLRequest = new URLRequest ("http://192.168.1.2/flash_testing.php");
var loader: URLLoader = new URLLoader();
request.contentType = "application/octet-stream";
request.method = URLRequestMethod.POST;
request.data = bytes;
loader.load( request );

It posts the AMF3-serialized data to this PHP file:

<?php
$var = $GLOBALS[ 'HTTP_RAW_POST_DATA' ];
file_put_contents('flash_testing.txt', $var);
?>

The contents of flash_testing.txt show the following:

My-Mac:fmog sneakyimp$ xxd /usr/local/apache2/htdocs/flash_testing.txt
0000000: 0605 c391                                ....

The first two bytes of which are markers to indicate type and length and some flags. The second two are the C391 we expect for Ñ.

Then I created a PHP file in notepad and saved it as UTF8 encoded:

<?php
$str = "Ñ";
file_put_contents('utf8.txt', $str);
?>

The output:

My-Mac:htdocs sneakyimp$ xxd utf8.txt
0000000: c391                                     ..

I'm guessing this means I won't need to reverse the bytes that correspond to a UTF8 char. That said, I don't really see any need to reverse the bytes of a string at this time.

EDIT: forgot my ut8.php contents