Question
Empirix.com did a study on the performance of six online auction sites. EBay did not do well:
http://www.empirix.com/Empirix/Corporate/News+and+Info/Press+Releases/Web+Application/Online+Auction+Benchmarks.html?page=home&link=left _pr_one
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Empirix Study of Online Auction Sites Finds as Many as One in Eight Transactions Fail
Amazon Auctions had overall top performance, besting eBay on response times and transaction success rates.
WALTHAM, Mass. - Oct. 7, 2003 - Empirix Inc., the leading provider of integrated test and monitoring solutions for Web, voice and network applications, today reported the results of a two-week study of the performance of six online auction sites. The study went beyond home-page availability to measure customer experience - whether customers can complete a multi-step transaction successfully, and how quickly.
Key findings from the report, titled "Benchmark Study of Online Auction Site Performance
August-September 2003", include:
Amazon Auctions achieved the best overall performance, with the second-shortest transaction length of 5.68 seconds, the best consistency and the lowest transaction failure rate of 0.52 percent.
BidVille had the shortest average transaction length at 3.90 seconds.
eBay exhibited the longest transaction lengths, with an average of 13.20 seconds and the second-highest transaction failure rate at 3.97 percent.
uBid had the highest transaction failure rates at 11.76 percent - meaning approximately one in eight transactions failed.
Online auction sites monitored as part of the study were: Amazon Auctions (http://auctions.amazon.com/); BidVille (http://www.bidville.com/); eBay (http://www.ebay.com/); ePier (http://www.epier.com/); uBid (http://www.ubid.com/); and Yahoo! Auctions (http://auctions.yahoo.com/).
Empirix measured three customer experience metrics during the course of the study period: Efficiency (total transaction length in seconds), Consistency (variability of transaction length) and Reliability (was the transaction completed successfully). Results were as follows:
http://www.users.eesc.com/bhearsch/p...ions_table.gif
All measurements were performed by Empirix's virtual user agents, which navigate Web applications just like actual customers and measure the quality of the customer's experience.
Transaction failure rates can translate into millions of dollars in lost business
"We were quite surprised at the transaction failures rates found at some of the sites," noted Joe Alwan, vice president of Marketing for Empirix. "uBid and eBay's failure rates in particular were very high, at one in eight and one in 20 transactions respectively. That doesn't bode well for a good customer experience at either site." Alwan noted that in many industries, quality is measured in terms of "five nines" or 99.999 percent success rates. "None of these sites approached five nines of quality - even Amazon Auctions, with a failure rate of just 0.52 percent, has work to do," said Alwan.
According to the latest figures from the META Group*, unplanned application downtime has a major bottom-line impact on online retailers. The annual cost of unplanned downtime for an online retailer with 99.999 percent availability is just $1,950. But the annual cost of downtime for an online retailer with 95 percent availability is $9.75 million. "A transaction success rate of 95 percent may sound good, relatively speaking, but the cost impact of that five percent of downtime can be tremendous," explained Alwan. "By reducing their errors rates by just one percent, companies can save millions of dollars, while providing a better quality of service to their customers."
Transaction length can drive customers to other alternatives
Alwan also noted wide discrepancies in transaction lengths among the auction sites, ranging from a speedy 3.90 seconds for upstart BidVille to a sluggish 13.20 seconds for eBay. Certainly some of that may be attributable to eBay's larger database of listings; however, Alwan pointed out that companies should be able to optimize the performance of their Web applications to handle the expected load. "The bottom line is this study indicates that it takes a customer three times longer to complete a transaction at eBay than it does at BidVille," he said. "And everyone is pressed for time. Long waits could drive buyers to try competitors or alternatives such as search engines and comparison-shopping services."
Poor consistency can ruin an otherwise well performing application
The consistency of transaction lengths also varied widely. Amazon Auctions had the lowest standard deviation of transaction length, indicating that its customers enjoy the most consistent experience. uBid, on the other hand, had the highest standard deviation, meaning its customers received the least consistent experience, with transaction times as long as 185 seconds - far longer than the average Web user would wait. "Consistency is an important metric to measure for critical business applications," explained Alwan. "Inconsistent performance indicates a low level of internal control over an application, so customers are just as likely to receive a poor experience as a good one. Considering that a customer relationship is only as strong as your last interaction, inconsistency can have a large impact on a company's ability to keep its customers coming back."
Alwan noted that good application testing and monitoring practices can help companies improve the quality of experience customers enjoy at their sites. "The key is to navigate applications and business processes exactly as users do and measure everything about their experience - including wait times, content, context-sensitive application features, and user-specific data," said Alwan. "Pre-deployment testing and production monitoring - both from the user's perspective - can have a tremendous impact on quality. The goal is to catch potential problems before they impact the customer experience, rather than relying on customers to point problems out to you."
The results of this study were highlighted in a story published in today's Wall Street Journal Online (http://www.wsj.com/). Interested parties can access the full Empirix report, "Benchmark Study of Online Auction Site Performance, August-September 2003," at no charge at http://www.empirix.com/.
Study Methodology
Empirix examined the sites using its FarSightSM service, which monitors Web application performance from a network of virtual user agents located throughout the global Internet. Virtual user transactions navigated to the home page, searched for "digital camera," clicked through to the first listing and verified that the listing page loaded correctly. The virtual user transactions were repeated at 15-minute intervals, 24 hours a day, at each site from August 31 through September 15, thus yielding approximately 1,200 data points for each company's Web application.<HR></BLOCKQUOTE>
Blanche
Answer
Blanche
I am not sure that 13 secs to complete a transaction is that big a deal. I mean, if you do 5 or 10 buy transactions a day, the extra secs do not up to a whole lot.
Also, the fast response time on the smaller site may be due to little traffic. Could be that is the only transaction going on.
Seems to me the study is a lot of hoopla over nothing.
Answer
Hi commentary. This is the part that caught my eye:
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>and the second-highest transaction failure rate at 3.97 percent.<HR></BLOCKQUOTE>
I think this means that 1 out of every 20 transactions failed on eBay. I'm assuming this is due to site glitches, server errors, page not found, etc.
Blanche
Answer
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by commentary:
I am not sure that 13 secs to complete a transaction is that big a deal. <HR></BLOCKQUOTE>
MMMMMMMmmmmmmm. It is my consistent experience that it often takes 13 seconds just to get the search page to load. And then it can take a minute or more for the results to come up, if at all.
For some of us, the snail's pace of eBay on a modem is infuriating. Not to mention all the error messages received during the course of searching.
I believe the study, using fairly professional web folks, and probably up to date equipment, understated the problem. At least for the majority of Americans who are not on DSL or do not have the latest bells and whistles. And that is where the buyers are.
Larry
Answer
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>I am not sure that 13 secs to complete a transaction is that big a deal. I mean, if you do 5 or 10 buy transactions a day, the extra secs do not up to a whole lot.<HR></BLOCKQUOTE>
But I wonder how the results would have looked if they used a dial up connection to do the tests?
I "sniped" the other night (my definition not RD's) and I was worried I was going to miss the bid when I went in with One Minute and twenty seconds - I actually got in with 12 seconds to spare, but that type of result must mean that bids are being lost to buyers and sellers all over the place.
A friend with ISDN has learned to snipe at 16 seconds - and missed two of his snipes on the same night.
Of course, the seconds do not add up a whole lot, they only affect livelihoods of those sellers who do not know that they lost the bids......
Kevin
Answer
Kevin
I think I have posted here that one should never try to snipe as close to auction end as possible. It makes no sense to do so.
I agree with you that it takes a long time to load certain screens on eBay. If one sees an item with minutes to go, one almost have to start with getting a bid screen up. Otherwise, it takes too long to process the bids.
I personally use sniping software almost 100% of the time so the screen load is not an issue. I think I missed about 3 bids out of hundreds so I am not sure if there is a problem on eBay. But, then my snipe time buffer is never less than 20 secs so I am not sure if that is what helps.
Answer
Blanche
I saw that statement also but do not buy it. If I enter a transaction and it fails to register, I will simply go back and resubmit. So unless I am sniping, the first failure to submit should not have prevented me from doing a second submit.
If a buyer really wants something, I think this is a non-issue.
Answer
Sorry Commentary,
I don't see site reliability on an economic platform / market place the size of Ebay, as being a non-issue. The problems compounded seriously at the time that they added:
(a) Powered By IBM logo's in the upper right of every page.
(b) sending data to doubleclick when you enter a search (Blanche gave me the settings to get around this, but users should not have to recalibrate to work on a live economy site of the nature of Ebay)
(c) changes to the banners at the top of search pages (Blanche gave me the settings to get around this, but users should not have to recalibrate to work on a live economy site of the nature of Ebay).
(d) when adjustments were made to the SYI listing form that created a page hang of up to 2 minutes on every page as you work through a multi page listing form that undermines productivity.
I know several people who have quit using the site entirely, and more people who now only search for their core interests because of the time "wasted" by trying to do wider searches. All of these changes happened since May and have affected the site seriously.
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>I think I have posted here that one should never try to snipe as close to auction end as possible. It makes no sense to do so.<HR></BLOCKQUOTE>
Whether or not it makes sense, Ebay fostered this by previously having a more efficient site. You may blame the buyers for having a petty competitive attitude to competitive buying, but Ebay has gotten less and less efficient and it is losing bids from some of those who are manually sniping (all reports indicate that sniping programs still work efficiently).
The fact is that Ebay is a major and allegedly very profitable business. It **SHOULD** be running efficiently, because the nature of the market that it embraces requires it. Ebay has made some very good improvements to its search engine capabilities this year, but they have undermined it by waiting for banner ad's and tracking software before loading the results. Google runs an even larger search engine, but I can load 6 search results - including typing in the search term - in the time it takes to load one Ebay search if I have the settings of the average user.
The importance of the "live market" nature of the Ebay economy means that these are definitely not non-issues - BUT Ebay can get away with this because they hold a virtual monopoly of the industry.
Bids are being lost through these performance issues. Bidders are reducing what they look at because of these performance issues, and some have left. In a soft economy, this does nothing to help sellers. Ebay *should* easily be able to conduct a much more efficient and effective site. This will help their bottom line as well as that of their paying customers.
Just how I see it, Kevin
Answer
Kevin
I am not sure we are talking about the same issue here. I am discussing the 13 secs reference in the above article. I am not talking about all the changes that eBay has made to hamper or slow down the site for both listing and bidding.
But, if one reflects upon the old days, eBay was more unstable as a site. Yet, prices were still better in the past than today. Site stability is the not the sole cause of the decline on eBay. I personally think the modern junk has more to do with it. Also, some rare items are truly not rare. They are just buried in someone's closet.
Answer
I believe that we are both talking about the 5.68 to 13.2 ratio of performance between Amazon and Ebay above. Whilst the 8 second difference can be viewed as a non-issue, the problem is compounded many fold by using dial up connection, and by more remote location of users. My guess is that at least 60% of users are on dial up (it is a guess so I may be way wrong here) and for most of those people the 13.2 seconds holds no relevance. I would be very happy if Ebay's pages loaded at 13.2 seconds, but that is an unreasonable expectation.
However, in spite of whatever the reliability issues were 4 years ago, people who I have spoken to in the last four months have changed their useage of Ebay due to how the site now performs. It may be unreasonable for them to expect Ebay's technology to improve at a level equivalent of the rest of the net, but that expectation was fostered by Ebay running quickly until the recent changes that have made it a chore. *This* has nothing to do with what is being listed, this has to do with getting around the site efficiently and being able to find whatever they are looking for. A standard run of searches fell from taking me about 15 to 20 minutes, to taking me 45 minutes until Blanche gave me the fix. The 25 minutes could have been used to look for other items to bid on, BUT several people I have spoken to in person have stopped doing wider searches because of the time consumed in trying to load pages. This stupid innefficiency is reflected in the results above, and it is losing bids for sellers on Ebay. If the difference really was only 8 seconds, I would agree that it is a non-issue, but that 8 seconds is only a reflection of a more serious flaw in Ebay's operation, and dial up users see the worst of that flaw.
Kind Regards, Kevin