I'm thinking about writing a socket server/API that would accept product data in JSON format.
The issue I'm having (and maybe I'm not alone ...?) is how to determine the boundaries of an object; that is, when does one object end and the next object begin?
Is there a conventional solution for this?
Cogent thoughts, Questions, Pointers, slaps in the general direction of my head and links to RTFM are all appreciated. :-)
I'm probably missing an important point here, but since JSON is just a string, does it really matter where you split it into chunks? Or do you want to somehow be able to process some of the JSON data before the entire string is transmitted?
That's really a good question. I was thinking of the difficulty of transmitting 1000's of products in one transaction. I had assumed that we'd want to allow the system(s) to give us one "product" (JSON object) at a time, over, and over until they were done.
I suppose we might just read all the data into some specific location, keyed by the end-user's API-key, and they put it back together and parse it later ... is that what you're thinking? I wonder how long it would take to transfer, say, JSON string(s) for 10,000 products?
dalecosp;11036733 wrote:socket server/API that would accept product data in JSON format.
(…how to determine the boundaries of an object; that is, when does one object end and the next object begin?
If you are using existing technology to handle all of this, such as HTTP via a web server, you won't have to deal with the problem because it has been handled by others. If you are doing it yourself by reading from sockets (and possibly writing to them as well), you will have to decide for yourself. For example
1. Fixed length messages
2. Include message size in the message (possibly using some sort of message header)
3. Assign the meaning of "EOM" (end of message) to some character, such as \0x0
You should probably read about how to handle the possible lack of terminating character, message size or messages exceeding fixed length limits.
If you look at http, you are not supposed to send bytes below... "some number" , in the actual message 
Let's say 0x32 (but do look it up yourself. It could be 0x20 for all I know). (some) Bytes below that are used as control characters, such as 0x0. If you want to see this in action try running serialize() on a php object with protected / private variables and then sending the serialized string in a http response without first sanitizing it.
dalecosp;11036739 wrote:That's really a good question. I was thinking of the difficulty of transmitting 1000's of products in one transaction. I had assumed that we'd want to allow the system(s) to give us one "product" (JSON object) at a time, over, and over until they were done.
That is definitely one possibility. But if you require the same format no matter the number of products being transmitted, there will be no difference in processing server side.
For example, you might require the following format: An array of products, which each element being one product.
I.e. in php that would look like
$products = [
[
'id' => 'p001',
'name' => 'Shoe'
],
[
'id' => 'p145',
'name' => 'Penguin'
],
[
'id' => 'p001',
'name' => 'Lambtron'
],
];
And in json
[{"id":"p001","name":"Shoe"},{"id":"p145","name":"Penguin"},{"id":"p001","name":"Lambtron"}]
If someone are to send you one procuct, you'd still require them to send you an array of products. You'd still access it the same way as several products, using iteration
if (!is_array($products)) {
# error
exit;
}
if (count($products) === 0) {
# error
exit;
}
foreach ($products as $prod) {
# do stuff
# doesn't matter if it is one or several products
}
dalecosp;11036739 wrote:I wonder how long it would take to transfer, say, JSON string(s) for 10,000 products?
The limit on number of products that can be transfered at once is memory. But it will always be faster to send 10k products in one request than in 10k requests. Setting up each request equals overhead. The time required will also be proportional to the size of the strings involved. For example, sending "product_id : product_serial_number_00001"
will obviously use up more bandwidth than
"id: 1"
Perhaps a stupid example, but you get the idea of reducing string sizes should it be relevant / significant.
dalecosp;11036739 wrote:I suppose we might just read all the data into some specific location, keyed by the end-user's API-key, and they put it back together and parse it later ... is that what you're thinking?
If you only handle one incoming connection at the time, storing the string elsewhere for processing by some other program might be a good idea. But this should only need doing for optimization purposes. I doubt you'd need to resort to such things if you use an appropriate approach. For example, how about nginx which is awesome at handling loads of concurrent connections.
If you need end user API keys etc, you could either include that information in the message, changing the above to
$request = [
'auth' => [
'user' => 'billy-bob',
'key' => 'api-key',
],
'length' => 3,
'products' => [
[
'id' => 'p001',
'name' => 'Shoe'
],
[
'id' => 'p145',
'name' => 'Penguin'
],
[
'id' => 'p001',
'name' => 'Lambtron'
],
],
];
dalecosp;11036739 wrote:Is there a conventional solution for this?
Cogent thoughts, Questions, Pointers, slaps in the general direction of my head and links to RTFM are all appreciated. :-)
I'd say http and a web server sounds rather conventional, most likely by creating a REST API. Variations on my example exist. For example, you may require pre-authentication (login) before allowing a user access to your REST API. However, it should be noted that some purists believe that this breaks the spirit of REST - If you maintain auth info, you are storing state. I'd say, if it works - whatever… Either way, you will have to propagate auth info somehow.
Writing the server from the grounds up sounds exhausting and unnecessary. I personally wouldn't do it unless I had very specific demands I needed to meet, or as a learning experience. I do not know where you'd find articles covering everything you need, but you may find the basics of message boundaries here: http://www.codeproject.com/Articles/11922/Solution-for-TCP-IP-client-socket-message-boundary