http://blogs.wsj.com/digits/2012/12/23/how-the-journal-tested-prices-and-deals-online/

December 23, 2012

How the Journal Tested Prices and Deals Online

In an article published in Monday's paper, [1] the Wall Street Journal reports on Internet retailers that show different prices and products to different consumers. To get these results, the Journal conducted a series of tests on a variety of commercial websites. Here's an overview of the process.

To conduct preliminary tests, the Journal built custom software that allowed it to look at Web pages while simulating visits from different computers. The system specifically allowed the following:

1. tests from different "IP addresses"--numbers that can provide information on location--by running the computer traffic through "proxy" computers in different places that the Journal had purchased the use of

2. masking of something called the Web browsing "user agent" so that the simulated visit appeared to be coming from different Web browsers, including the default browsers on the iPhone and other common smartphones; and

3. use of "profiles" with different browsing histories and cookies--small files that websites can store on a person's computer. The Journal created such profiles and gathered cookies itself by browsing different sites on the Web.

In building this software, the Journal used an open-source system called Selenium that allows automated tests of Web browsing.

In selecting types of sites to test, the Journal relied primarily on earlier published reports or speculation that a site was altering prices or deals, including comments on online forums. Of particular importance was research first published in October [2] by Jakub Mikians of Universitat Politecnica de Catalunya and Laszlo Gyarmati, Vijay Erramilli and Nikolaos Laoutaris of Telefonica Research. The Journal also tested sites known to use software that allows for display of different products to different users.

When the Journal observed significant differences at an online store in preliminary examinations, it followed up with additional analysis to determine whether the differences followed any observable pattern. In many cases, they did not; many price variations appeared to be the result of randomized testing, a common practice of online retailers. In other cases, however, the variations were strongly, or entirely, correlated with the simulated visitor's location, browser, or other factors. The most salient of those findings are discussed in Monday's article.

The Journal's follow-up testing typically required additional, custom-written software programs to tease out the specific factors involved in a given site's offerings. Programming languages and tools used in this phase of testing and analysis included: Python, Ruby, Selenium, mitmproxy, csvkit, and R.

The Journal's process can be seen in its testing of Staples.com, the main example in Monday's article. Initial testing with Selenium indicated that Staples.com sometimes offered different prices to certain visitors, apparently depending on their IP address. But the results weren't clear-cut. Using mitmproxy and Chrome Developer Tools, the Journal found that Staples.com stored what appeared to be the current visitor's estimated location in a small text file known as a cookie, under a variable named "zipcode."

The Journal observed that it could preemptively set this variable in order to make Staples.com seem to react as though the visitor were coming from the set location. Further tests revealed that the test computer would then consistently receive the range of prices in line with that ZIP Code. These findings were corroborated by a series of standard browsing tests and crowdsourced tests through a company called Houdini Inc. that manages crowdsourced projects.

The Journal developed a series of small programs, written in Ruby and R, to test and analyze the prices given to ZIP Codes across the country. Between Dec. 14 and Dec. 16., the Journal simulated visits to Staples.com from all of the more than 42,000 U.S. ZIP Codes, testing the price of a Swingline stapler 20 times in each. (The Journal used the list provided by federalgovernmentzipcodes.us, which combines the U.S. Postal Service's data with various geographic and demographic data.) The results of this test are described in Monday's article and can be explored in this interactive graphic. [3]

One note: IP addresses map imperfectly to ZIP Codes, so real-world visitors testing the Staples site might not receive the exact predicted results.

-- Jeremy Singer-Vine, Ashkan Soltani and Jennifer Valentino-DeVries

[1] http://online.wsj.com/article/SB10001424127887323777204578189391813881534.html

[2] http://conferences.sigcomm.org/hotnets/2012/papers/hotnets12-final94.pdf

[3] http://online.wsj.com/article/SB10001424127887323476304578198373572451206.html