The Impact of Legal Challenges on the Evolution of Web-based Intelligent Agents

• Author: Yun Wan; Yue Liu
• Position: Department of Computer Science, University of Houston - Victoria; Norwegian Research Center for Computers and Law, Faculty of Law, University of Oslo
• Pages: 112-119

Key words: Intelligent Agent, Internet, Data Extraction, Shopbots, Metabots, Database right, Trespassing in cyberspace

Page 112

1. Introduction

The Web provides an ideal opportunity for more efficient information-brokerage models (Maes, 1994). As a result, we have seen numerous Web-based agents emerge since 1994.

For example, A Web-based auction agent like eBay could reach a potentially unlimited number of participants compared to its brick-and-mortar counterpart. A Web-based travelling agent like expedia.com could seamlessly integrate airfare, hotel, and car-rental service. It can then cross-sell travelling products when travellers plan their trip online. Former premium ticketing agent, TicketMaster, also has its online presence, ticketmaster.com, which essentially has transformed itself into a Web-based ticketing agent.

The Web also provides powerful technology to allow shoppers, from one site, to compare price information for the same or similar products and services from multiple sites. We call such technology comparison-shopping agents or shopbots because they serve as the buyer's agent to retrieve price information from other websites (Wan, Menon, & Ramaprasad, 2003). These shopbots dis-intermediated the traditional role of agents because they were competing for the same customer. However, shopbots are able to re-package the information they have collected in a more efficient way.

There is a natural limit on the number of websites a shopbot can handle, so one shopbot will not be able to cover all sites a user wants to search. As a result, shopbots emerged and were distributed unevenly, with both overlapping coverage and their own unique coverage. Thus, another type of shopbots or so-called MetaBots emerged. Instead of searching websites directly, MetaBots search those shopbots and retrieve and aggregate the information from the latter (Etzioni, 1997).

As we will illustrate in this paper, the legal challenges for these three types of emerging Web-based technology and business models are Web-unique and, in some circumstances, rather complex. The possible consequence of lawsuits against them greatly influenced the evolution path of these three categories, especially for the two Web-only categories, Shopbots and MetaBots.

Essentially, shopbots end up with a more favourable legal environment than MetaBots in terms of their information retrieval behaviour. Consequently, the former developed into a fully mature business model while the latter largely stalled in its development.

2. The Case of Shopbots and Comparison-Shopping Services

Since the first launch of BargainFinder in 1995, comparison-shopping as an effective online shopping mode has attracted millions of consumers as well as large number of small vendors. It has also inspired the creativity of at least two generations of techno-entrepreneurs to develop ingénue shopbots that could retrieve more and more complex product information(Wan, 2005).

Page 113

However, this category of B2C ecommerce has also experienced the most complicated legal challenges. As a result, the evolution of this information industry category was probably more influenced by lawsuits rather than by technology advances.

In order to be familiar with the details of the legal challenges in comparison-shopping services, we need to introduce some typical scenarios of online shopping, as well as the roles of comparison-shopping services. Assuming an online shopper visits an online store like bestbuy.com to look for the price of a digital camera and then clicks through to another online store like Amazon.com for the price of the same camera. Both are certainly legitimate actions. Now if we go a step further and suppose that this online shopper employs a shopping assistant, who is representing the shopper, visits these two stores and retrieves the price for the online shopper. Is this legitimate?

This is exactly what the comparison-shopping agent, BargainFinder, did in 1995 and such a query was blocked by some online music vendors (Krulwich, 1996). Thus, the first legal challenge involved the rights of online vendors to block the query of comparison-shopping agents.

Now we go a step further, is it legitimate for a comparison-shopping agent to visit these two online stores, and cache the price of the same digital camera, and then allow all incoming online shoppers to retrieve the price without going back to these two stores repeatedly?

This is the general practice of most comparison-shopping agents (Choi, 2001; Jeanette, 2004; Rebecca, 1999). By caching the price information from online vendors, they improve their own efficiency as well as that of online vendors. However, if data is retained on agents' databases, the legal risk of agents increases unless the online vendors could benefit from such actions. So far, the potential legal risks of comparison-shopping services have just been between online vendors and agents. Yet, the true challenge for comparison-shopping services may lie in the data extraction among agents themselves.

Again, suppose there is a comparison-shopping agent that does not collect price information from online vendors. Instead, it collects such information from a few other comparison-shopping agents (such agents are usually called "Meta-Search" agents or Metabots). Do the latter Shopbots have the right to prohibit the Metabots from querying its own search engine? Should the Metabots compensate the Shopbots or the vendor?

Many comparison-shopping services provide rating information on products, vendors and consumers/buyers (on auction sites). Will such information be transferable with the producer or must it be retained by the site where the information was originally obtained?

With these preliminary questions, we review the development of comparison-shopping services and examine how legal challenges influenced their development track and techno-business model in the next section. We will illustrate the development of shopbots by dividing the history of its evolution from 1995 to 2005 into three stages. Major conflicts of stakeholders, along with the resulting legal challenges and their influence, are discussed within these stages.

3. The Nascent Stage (1995~96): The Dilemma of Vendors

The early stage of comparison-shopping services mainly revolved around the online vendors and shopbots. The central legal challenge is whether shopbots have the right to collect information from vendor sites; Or should an agent be allowed to deep-link to another website without permission?

Back in 1995, when BargainFinder was first launched, there were two different vendor attitudes: cooperation and blockade. Typically, small online vendors preferred to be searched by shopbots, while more popular online vendors were hesitant (Krulwich, 1996).

For small online vendors, the major business hurdle was visibility(Wan, 2005). They could usually offer competitive pricing, but their websites could only reach a limited number of consumers due to budget constraints on advertising. The comparison-shopping service works like an indirect promotion and leveled ground of competition with more established competitors, which usually asked higher prices for the same product. Thus, they generally preferred the crawling of shopbots.

Since established popular online vendors usually charged premium price, they were afraid of losing their business when the index of comparison is based on price. Also, many popular online vendors generated their revenue from banner ads from their website; the link used by shopbots brought the shopper to the product page directly, thus potentially reduced their ad revenue though it could have been that this additional revenue might not have happened without the referral of shopbots.

However, there were also potential gains from being listed in a comparison-shopping service for vendors in both categories because it turned out that online shoppers were not mere bargain finders, but were also concerned about service qualities, such as delivery and return policy, as revealed by later studies (Brynjolfsson & Smith, 2000). Thus, popular vendors probably also benefited...