Web Fetching

[deleted]

Hi,

I've been trying to get some information of a site so that i can be used on another site.

Here's the url:
http://www.synnex.com.au/Default.aspx?tabi...ct&export=False

I've been successful in extracting the required information, however there is more info on more pages. The problem is that these links use javascript.

If you look at the bottom where it's got 1-10, they are links with javascript, while the next 10, back 10, etc. are form fields.

I am looking to some how automatically navigate through these links, e.g. the Next button, using php so that each page can have the required info extracted.

Thanks.

dgrinberg

Hi,

I think that the link you gave is incorrect. It is only going to the front page of the Synnex site for me.

I assume you are going to a list of products, and wanting to cycle through the other pages.

What version of PHP are you using? If you are using PHP5 maybe the DOM extention can help you?

Have a look here for an example:
http://www.sitepoint.com/article/coming-soon-webserver-near/15

It enables you to DOM parse a HTML document.

Other than that I think more info is required. (or at least access to the page you are working with)

[deleted]

Apologies dgrinberg for that link not working. I clicked it and did the same for me. If you goto the link however and then click products, and press the search button (without changing any of the worm info), it will then display the list of products in which i am able to extract.

I am wanting to move to the next page and then extract it's contents, however because they are using javascript for these links it makes it a bit more difficult. They are not trying to stop people accessing it however, as the info i am extracting is for one of their clients.

The php5 option looks interesting, however don't have the time to research it more thoroughly. Do you know of any other possibilities.

Thanks.

dgrinberg

Yuck, DOTNET!!!!

I am not sure that you are going to be able to do what you want to do.

The only way to do it is by HTTP posting your required page number. But there will be a whole lot of session variables that you wont be able to replicate!

Just investigating this would be a 'truck' load of work.

Really the only thing I can point you to is PHP's CURL extension. It enables you to do HTTP posting. Maybe that will help?

Sorry I can be any more help.

Good Luck!!

dgrinberg

Do you have any contact with the builders of the other site?

Maybe you could get them to build a page which just lists them all in one page? Or better still in XML?

weekender

firstly, do you have permission to use the data on the site?

from past experience on phpbuilder, people take a very dim view on content-leeching. if you do have permission, maybe the site supplies some kind of rss feed or something.

[deleted]

Originally posted by micksworld
They are not trying to stop people accessing it however, as the info i am extracting is for one of their clients.

This is legitimate, the client i am doing the work for has a login and is able to download a csv of the product database, however this file is missing information that is required.

dgrinberg

I am sure that if you are requesting legitimate stuff that they should be able to add what you require in the CSV!

[deleted]

I wish they had have prodvided the info that is needed in the csv. There are other companies that are 'apparently' spidering the products with no problems, according to one of the tech support guys that works for synnex. Please feel free to email their webmaster and ask if it's legitimate, i emailed him 3 days ago and still haven't heard back about how i should get this info.