A Webget Client

Here's a simple client that takes a remote host to fetch a document from, and then a list of documents to get from that host. This is a more interesting client than the previous one because it first sends something to the server before fetching the server's response. 

    #!/usr/bin/perl -w
    use IO::Socket;
    unless (@ARGV > 1) { die "usage: $0 host document ..." }
    $host = shift(@ARGV);
    foreach $document ( @ARGV ) {
        $remote = IO::Socket::INET->new( Proto     => "tcp",
                                         PeerAddr  => $host,
                                         PeerPort  => "http(80)",
                                        );
        unless ($remote) { die "cannot connect to http daemon on $host" }
        $remote->autoflush(1);
        print $remote "GET $document HTTP/1.0\n\n";
        while ( <$remote> ) { print }
        close $remote;
    }

The web server handing the ``http'' service, which is assumed to be at its standard port, number 80. If your the web server you're trying to connect to is at a different port (like 1080 or 8080), you should specify as the named-parameter pair, PeerPort => 8080. The autoflush method is used on the socket because otherwise the system would buffer up the output we sent it. (If you're on a Mac, you'll also need to change every "\n" in your code that sends data over the network to be a "\015\012" instead.)

Connecting to the server is only the first part of the process: once you have the connection, you have to use the server's language. Each server on the network has its own little command language that it expects as input. The string that we send to the server starting with ``GET'' is in HTTP syntax. In this case, we simply request each specified document. Yes, we really are making a new connection for each document, even though it's the same host. That's the way you always used to have to speak HTTP. Recent versions of web browsers may request that the remote server leave the connection open a little while, but the server doesn't have to honor such a request.

Here's an example of running that program, which we'll call webget: 

    shell_prompt$ webget www.perl.com /guanaco.html
    HTTP/1.1 404 File Not Found
    Date: Thu, 08 May 1997 18:02:32 GMT
    Server: Apache/1.2b6
    Connection: close
    Content-type: text/html


    <HEAD><TITLE>404 File Not Found</TITLE></HEAD>
    <BODY><H1>File Not Found</H1>
    The requested URL /guanaco.html was not found on this server.<P>
    </BODY>

Ok, so that's not very interesting, because it didn't find that particular document. But a long response wouldn't have fit on this page.

For a more fully-featured version of this program, you should look to the lwp-request program included with the LWP modules from CPAN.