Tracking a FedEx Package with an HTML Scraper

Saturday Sep 28th 2002 by Jonathan Eisenzopf
Share:

One way to develop powerful VoiceXML applications is to leverage existing Web applications by simulating a Web browser and scraping the results via a programmable HTTP Web agent. This example allows users to track a Federal Express package, using the fedex.com Web site.

One way to develop powerful VoiceXML applications is to leverage existing Web applications by simulating a Web browser and scraping the results via a programmable HTTP Web agent. This example allows users to track a Federal Express package, using the fedex.com Web site.

Overview

One of the selling points of VoiceXML is that it can work with your existing Web infrastructure. Applications can be developed using the same tools, run on the same server over the same Internet connection. So it makes sense that VoiceXML applications should be able to leverage existing code.

Hopefully many of your Web application have been componentized into re-usable libraries, so that other interfaces can be developed using the same code. In reality, most Web applications are flat scripts. Others are not even available to you at all. One way of leveraging Web applications in the latter two examples is to write a layer of code between the voice interface and an existing Web or wireless interface. This exercise involves emulating the intended client, submitting form input, and parsing results into a form that can be played back by a VoiceXML interface.

The example that we're going to look at is a Perl script that uses the Federal Express Web tracking application available at http://www.fedex.com/us/tracking/. I will be referring to line numbers in the source code throughout this article, so you might want to open the example in another browser or print it out for reference. You can also download the code without line numbers.

The application is totally self-contained. There aren't any other external files or programs needed to run the script other than a Perl interpreter and the LWP and CGI modules. The script will run on Windows and Unix servers and on any Web server that supports CGI.

How it Works

An example of a typical dialog is below:


Computer: Please enter your federal express tracking number 
now.
Customer: 1232343
Computer: Package 1232343 is as follows. Left FedEx Sort 
Facility NEWARK NEW JERSEY. The last update for this package 
occurred on August six at twelve thirty nine.

The first time the script is run, the user is prompted for their Fedex tracking number. If the user doesn't enter a number, they are given additional instructions and prompted for the number again. When the user enters a number on their telephone keypad, it's sent back to the script as the track_num form field. This number is appended to the Fedex tracking URL and sent using an LWP agent. LWP is a Perl library for simulating a Web browser agent. The page that comes back from the Fedex site is parsed for the tracking information. We also check for an error message. If an error occurred, the user is prompted again for a tracking number, otherwise, the tracking information is played and the call is ended.

fedex_tracker.pl

This Perl script utilizes the LWP libraries, which allow us to send queries to a remote Web server. This library, along with the CGI library, are both loaded on lines 1 and 2. On line 10, we check for the existence of the track_num parameter. If it exists, we go ahead and send it to the Fedex Web site by executing the &parse_html function on line 11. If an error occurs, meaning that the tracking number was invalid, we output some VoiceXML content that plays an error message and re-runs the script. If the request was successful, the code on lines 26 to 54 is executed, which plays the tracking information that was retrieved.

If the track_num form field was not passed to the script, the VoiceXML content on lines 55 to 79 is sent back to the VoiceXML gateway. This is the dialog that asks the user for the tracking number. So if a number is not sent, this is the dialog that gets executed first.

Basically, the script will send back the proper dialog for the following states:

  • The user is running the script for the first time; the user is prompted for a number
  • The user has just entered a number; the VoiceXML form submits the number back to the script, which submits the number to the Fedex site and plays the results.
  • The user entered an incorrect number; the user hears an error message and plays the error message.

    The guts of the script are in the parse_html subroutine starting on line 81. This is where we send the tracking number to the Fedex Web site and parse the results. This is also where we need to do a little regular expression fancy footwork. On line 83, we build the full contents of the URL that we will be submitting the information to (including the tracking number). Fortunately, the Fedex script allows us to submit data via the GET method (instead of POST), so we can simply call the get function of the LWP library, passing the URL we just built. The function passes the resulting HTML back, which is stored in the $page variable.

    So now we have a real big text string that contains an HTML page. How do we get that information out of all that text? Well, by finding a pattern to the results and using that as a mark for where the data starts. To figure this out, I used the Fedex application from a Web browser and viewed the HTML source. I found that a unique block of HTML occurs just before the tracking data begins. The result are the regular expression on line 91 which says, find this text CLASS="resultstableheader" followed by zero or more characters .*? followed by a </tr> tag, followed by a whole bunch of text (.*?) up to a </table> tag. This last bit of text up to the </table> tag is where our data is located. We want to grab that and save it on line 92.

    Now that we have the glob of text containing the data, we have to parse it with more regular expressions. To accomplish this in one fatal swoop, I created a compound regular expression on lines 94 and 95 that finds a row of data and splits it into pieces. Lines 97 to 112 split the rows into records and saves the information in an array of hashes. Each array item contains the message, date, time, and notes that are related to the record. Fedex lists not only the current status, but the whole history of a package. While we're only playing back the first record, which is the latest tracking information, you can reuse this data structure and set of regular expressions to play the entire history.

    Once the data structure has been built, it returns the array back to the caller. You can see where we reference the first record of this data structure on lines 27 to 31, where we're creating local copies, which are included in the VoiceXML output.

    Conclusion

    Well, we just created a pretty useful VoiceXML application in not much more than 100 lines of code. That's impressive I think. The opportunity to voice enable any number of Web applications is at hand, and here's some code you can use to get started.

    About Jonathan Eisenzopf

    Jonathan is a member of the Ferrum Group, LLC based in Reston, Virginia that specializes in Voice Web consulting and training. He has also written articles for other online and print publications including WebReference.com and WDVL.com. Feel free to send an email to eisen@ferrumgroup.com regarding questions or comments about the VoiceXML Developer series, or for more information about training and consulting services.

  • Share:
    Home
    Mobile Site | Full Site
    Copyright 2017 © QuinStreet Inc. All Rights Reserved