Retrieving a web page

Start of a web browser...

We build upon the capabilities of previous examples by showing how to grab the contents of a web page from the Internet. This example shows how to dump the HTML data of the Rowley Associates home page at www.rowley.co.uk/index.htm.

Select and build the project

In the examples for your board, you'll find a Networking Projects solution, and within that a Retrieve Web Page project. Double-click that project to make it active and press F7 to build.

Sockets

Double-click the file example_retrieve_web_page.c in the Source Files folder and it will open in the code editor. This example is longer than before, but then it does much more than previous examples.

Once the domain name is resolved, the example needs to communicate with the web server in order to download the web page. It does this by creating a socket and connecting the socket to the server:

// Open a socket to the host on port 80.
s = ctl_tcp_socket();
if (s < CTL_NO_ERROR)
  example_terminate("No sockets available\n");
example_check_status(ctl_tcp_connect(s, addr, HTONS(80), 1000));

ctl_tcp_socket creates a new socket and ctl_tcp_connect connects that socket to the server. The parameters to ctl_tcp_connect are:

The socket, created by ctl_tcp_socket.
The IP address of the server to connect to. The IP address in this example is resolved using DNS using ctl_dns_get_host_by_name.
The TCP/IP port to connect to. HTTP connections use port 80, and HTONS converts the port number from host byte order to network byte order as required by the TCP/IP Library.
The maximum time to wait for the connection to be made. In this example we are prepared to wait one second for the connection to be established.

Sending the request

Once the socket is established, you start to communicate with the server using a GET request. For reference, the HTTP protocol is fully described in RFC2616.

The GET request consists of the command, the headers, and a blank line to terminate the headers:

ctl_tcp_printf(s, "GET http://%s/%s HTTP/1.0\r\n", host, name);
ctl_tcp_printf(s, "Accept: text/plain\r\n");
ctl_tcp_printf(s, "Host: %s\r\n", host);
ctl_tcp_printf(s, "\r\n");
ctl_tcp_push(s);

The application sends:

The GET command specifying the URL and the protocol to use. In this case, the URL is composed of the host name and the page we are interested in. Following that is the protocol to use, HTTP/1.0, which simplifies the following code somewhat.
The headers. This tells the server the MIME type of the response and the host we are addressing.
A blank line which indicates that the headers are complete.

An important difference

One thing to notice is the call to ctl_tcp_push: this ensures that the data sent to the socket gets pushed to the network and sent out on the wire. The TCP/IP Library buffers data on a socket until a TCP segment is full, when it is pushed to the network—to flush a partially-filled segment, call ctl_tcp_flush. This makes the TCP/IP Library different from classic TCP stacks which will typically flush a partially-filled segment to the network after a short timeout.

Reading the response

Once the headers are sent, the example reads the response from the server using repeated calls to tcp_read_line. We specified HTTP/1.0 which requests the server to close the connection after sending all its data, and take advantage of the fact that when a socket is closed, we'll receive an error if we try to read more from it, and we exit the loop:

// Process response.
for (;;)
  {
    // Try to read a whole line from the web server.
    stat = ctl_tcp_read_line(s, line_buffer, sizeof(line_buffer)-1);

    // Ensure the buffer is terminated.
    line_buffer[sizeof(line_buffer)-1] = 0;

    // Process return status.
    if (stat == CTL_NET_ERR_TIMEDOUT)
      {
        // Didn't get anything, loiter...
      }
    else if (stat < CTL_NO_ERROR)
      {
        // Error reading the socket or the socket closed?
        break;
      }
    else
      {
        // Dump response.
        printf("%s\n", line_buffer);
      }
  }

// Make sure socket is closed.
ctl_tcp_shutdown(s);

Before exiting, we close the socket. If the socket is already closed because the server closed it, closing it a second time makes no difference.

See if it works

Power up the board and run the code. In the CrossWorks Debug Terminal you will see something similar to this:

DHCP: awaiting IP address assignment
DHCP: awaiting IP address assignment
DHCP: awaiting IP address assignment
DHCP: IP address is 10.0.0.44 and subnet mask is 255.255.255.0
Using DNS server 10.0.0.8
www.rowley.co.uk resolved to 178.236.4.60
Connecting to www.rowley.co.uk (178.236.4.60)...
Requesting ...
HTTP/1.1 200 OK
Date: Mon, 09 Sep 2013 13:17:33 GMT
Last-Modified: Thu, 29 Aug 2013 08:37:15 GMT
Content-Type: text/html
Content-Length: 13841
Connection: keep-alive
Server: AmazonS3

<!DOCTYPE HTML>
<html>
…

Job done!

You now have a way to communicate with an HTTP server. You'll find that many servers will communicate in much the same way: a command, some headers, a blank line, and read the response, so you have a starting point at least.

The code

// Retrieve a web page.

#include "libnet/ctl_net_api.h"
#include "libplatform/platform.h"
#include "libplatform/platform_network.h"
#include "example_support.h"

// Static data.
static char line_buffer[512];

static void
example_retrieve_web_page(const char *host, const char *name)
{
  CTL_NET_IPv4_ADDR_t addr;
  CTL_SOCKET_t s;
  CTL_STATUS_t stat;
  char str[16];

  // Try to resolve host.
  stat = ctl_dns_get_host_by_name(host, &addr, 2000);

  // Did this resolve?
  if (stat < CTL_NO_ERROR)
    {
      // No.
      example_terminate("Could not resolve www.rowley.co.uk!\n");
    }
  else
    {
      // Yes, print the resolved IP address.
      printf("%s resolved to %s\n",
             host,
             ctl_ip_sprint_addr(str, addr));
    }

  // User needs to know...
  printf("Connecting to %s (%s)...\n",
         host,
         ctl_ip_sprint_addr(str, addr));

  // Open a socket to the host on port 80.
  s = ctl_tcp_socket();
  if (s < CTL_NO_ERROR)
    example_terminate("No sockets available\n");
  example_check_status(ctl_tcp_connect(s, addr, HTONS(80), 1000));

  // Send the request
  printf("Requesting %s...\n", name);
  ctl_tcp_printf(s, "GET http://%s/%s HTTP/1.0\r\n", host, name);
  ctl_tcp_printf(s, "Accept: text/plain\r\n");
  ctl_tcp_printf(s, "Host: %s\r\n", host);
  ctl_tcp_printf(s, "\r\n");
  ctl_tcp_push(s);

  // Process response.
  for (;;)
    {
      // Try to read a whole line from the web server.
      stat = ctl_tcp_read_line(s,
                               line_buffer, sizeof(line_buffer)-1,
                               CTL_TIMEOUT_DELAY, 4000);

      // Ensure the buffer is terminated.
      line_buffer[sizeof(line_buffer)-1] = 0;

      // Process return status.
      if (stat == CTL_NET_ERR_TIMEDOUT)
        {
          // Didn't get anything, loiter...
        }
      else if (stat < CTL_NO_ERROR)
        {
          // Error reading the socket or the socket closed?
          break;
        }
      else
        {
          // Dump response.
          printf("%s\n", line_buffer);
        }
    }

  // Make sure socket is closed.
  ctl_tcp_shutdown(s);
}

int
main(void)
{
  // Initialize platform.
  platform_initialize();

  // Start networking, wait for an IP address.
  example_check_status(example_bring_up_full_networking());
  example_check_status(example_await_assigned_ip_address());

  // Send headers, read web page.
  example_retrieve_web_page("www.rowley.co.uk", "");

  // Done.
  return example_finish();
}