Handling gzipped xml

mikawhat

I'm working on a live betting feed xml file which is much more complicated than i thought... From the url of the feed's provider, i'm getting a gzip file, which has no extension. Using 7zip from windows i'm able to extract the data, in a file with no extension. Renaming the file to .xml, i'm able to handle it with simplexml_load_file().

Howevere, this file contains huge amounts of data, which will crash my browser (both ie and ff).

My questions regarding this task, from the beginning are:
1. how can i get the initial file with php, having the url. (remote server uses windows authentication).
2. how to handle the file, which as i mentioned is without extension but is a gzip.
3. extracted xml file is about 120MB, which may cause the server ruuning slow(?). Any suggestions would be really useful!

dagon

file,curl,file_get_contents

2.gzfile or one of the other gz* functions depending on how you do 1

3.don't process the file while the user is waiting, do it daily or hourly (whatever you need) to produce the result you need.

mikawhat

Is there a problem with windows authentication when using those methods?
Is gzip like all other compressed files format (zip, rar), a file containing another file? Using gunzip -c file it will produce a file? I've tried this and is still working for 30 minutes now...

dagon

never done any php development on windows, so i don't know
you simply replace file fopen fread with gzopen gzread ..

mikawhat

i mean http authentication, not windows...sorry

dagon

mikawhat;10912657 wrote:
i mean http authentication, not windows...sorry

i don't see were that fits in to the process

mikawhat

well, if you try the xml feed url from your browser it will ask you for username and password. I guess running a script should also somehow use a username & password...

dagon

$fp = gzopen("http://name:password@server/directory/page", "r");

Weedpacket

For a huge XML file, [man]xml_parse[/man] and its related functions would seem more appropriate: it's a stream-based parser, so you'd process the data while you're reading the file, instead of reading and parsing the entire file before doing any processing.

mikawhat

This will get the gzip

$fp = gzopen("http://name:password@server/directory/page", "r");

This will handle the xml

For a huge XML file, xml_parse and its related functions would seem more appropriate: it's a stream-based parser, so you'd process the data while you're reading the file, instead of reading and parsing the entire file before doing any processing.

From the downloaded gzip, to the xml to be used, i guess there are some processes required for converting somehow the gzip to xml... is that right? Sorry for asking much but i'm not familiar with this stuff...

Weedpacket

mikawhat wrote:
From the downloaded gzip, to the xml to be used, i guess there are some processes required for converting somehow the gzip to xml... is that right?

Assuming that the gzip file is a compressed XML file ... see dagon's posts above.

mikawhat

OK, i'm now able to get the xml from the remote server, and save it into an array. This array has many sub-arrays, so if i start (already tried it and seems quite lame) doing the following, it will take ages until i find out the actuall structure of the array(xml)

foreach ($array as $key=>value) {
 ...check if there's a new array and then do another foreach etc etc
}

I was wondering if there's an easier way to get what i want from that array. I've never worked with xmls before so i have no idea what could be used for getting the required data..