This is extremely difficult, you will run into many problems. You may think it's possible on win32 using the Internet explorer method described above - but I think you'll find that that doesn't give you enough control.
Some things to consider:
- Security considerations - will the browser you use execute Javascript or other things
- Popups, dialogue boxes - will dialogue boxes and popups cause your application to hang waiting for the browser to complete? WHat happens if the user enters a URL requiring HTTP authentication, with a redirect (or redirect loop), HTTPS, goes to a FTP site, redirects to a file: URL, has a dodgy SSL certificate, requires NTLM authentication, contains frames, contains an object which the browser would normally use a helper app to display etc?
- If you are capturing the physical (or virtual) screen of a real session, how are you going to stop two requests coming in at once? Are you going to queue the requests and use a daemon?
- How are you detecting / handling errors which happen?
- What is the resource usage of these embedded browsers? Will it hose your server?
- Do you support plugins (Flash, Java, active X etc)
- How do you detect when the page is fully loaded? What is "fully loaded" anyway? Do you have to wait for backgrounds to load, etc? (NB: In my implementation I found this to be a problem)
I wrote an implementation which used a XUL application running in Xulrunner (Mozilla) to create a socket server application which listened for requests from PHP. It sent the results back asynchronously into PHP (Via HTTP, base64 encoded PNGs I think), and PHP further processed them.
This was complicated but worked well.
There are a LOT of issues. You need to be really careful.
Whatever you do, you have to be very careful that the browser doesn't crash / hang / spit out a dialogue box.
I found that I had to override various XPCOM objects in Mozilla to prevent dialogue boxes which would otherwise impede progress. I don't know if you get that level of control in MSIE.
Mark