How do I set the User-Agent when writting a spider?

[johnh]

As the subject states, I'm starting to write a spider. I want to make it a good spider and not trample everything it sees and so it reads the robots.txt file, etc, but how do I set the "User-Agent" field and the "Referer" field? That they will see in their logs?

Thanks
JOhn

[johnh]

Never mind, I found it. But in case anyone else does a search (which has been a long one for me) here is a big clue.

GET $path HTTP/1.1$crlf"
."Host: $host $crlf$crlf"
."User-Agent: $user_agent $crlf$crlf";