
If 125 people go to eBay and enter the same search term, will they see the same results?
Over the years, we've heard claims that eBay search brings back inconsistent results - a concern for sellers who expect their listings to be visible to shoppers. We set out to test those claims, launching an experiment to try and better understand marketplace search engines.
Over a 24-hour period, 125 readers volunteered to go to eBay.com and enter five specified search terms and report back their results in a blind survey. They searched using Best Match with no filtering. The survey was active for 24 hours to minimize the natural fluctuation of new and ending eBay listings. And we instructed users to "Make sure that you have "ALL CATEGORIES" enabled in the eBay search bar and "BEST MATCH" chosen in the Sort Order drop-down menu."
Here are the results across all browsers and time zones:
Diaper Bag:
84%: 37,000 listings (+/-200 listings)
12%: 50,000 listings (+/-200 listings)
4%: International participants
Tackle Box:
72%: 7,300 listings (+/-200 listings)
24%: 16,600 listings (+/-200 listings)
4%: International participants
Hires Root Beer Metal Sign:*
44%: 49 listings
24%: 38 listings
12%: 41 listings
8%: 44 listings
4%: 50 listings
4%: 48 listings
4%: International participants
* Since it is a somewhat rare collectible, there may have been auctions ending (or starting) throughout the 24-hour testing period.
Black Cocktail Dress:
63%: 453,000 listings (+/-2000 listings)
21%: 363,000 listings (+/-1000 listings)
8%: 376,000 listings (+/-1000 listings)
4%: 472,000 listings (+/-1000 listings)
4%: International participants
Star Wars Figure:
64%: 76,000 listings (+/-1000 listings)
20%: 117,500 listings (+/- 500 listings)
12%: 65,000 listings (+/-1000 listings)
4%: International participants
Note that we had a small number of international readers participate in the experiment, as indicated in the numbers above. All of them reported fewer search results than US readers because not all items show up internationally.
One thing the test indicated was that some participants did not realize that eBay's Best Match filters search by category in some (or many) cases. Shoppers may assume after conducting a broad search that it is as broad as it can get, not realizing Best Match may limit results to a certain category.
For example, those who took it on faith that searching "tackle box" without specifying a category would bring back all listings for tackle boxes may have only seen 7,300 listings.
Note that we asked participants to indicate which browser they were using - those using Firefox were the ones who tended to report the higher results. Here's a breakdown of which browsers people used when searching eBay for this experiment:
Browser Used:
Chrome: 40%
Firefox: 40%
Internet Explorer: 12%
Other: 8%
We also noted that some search terms were more consistent than others - why would a commodity item like a black cocktail dress bring back a different number of results for different people searching within a 24-hour period?
The experiment indicates that sellers who worry that their listings may be excluded from eBay search results have good reason for concern, and we'll continue to run tests such as these.
Do you have any theories as to why eBay Best Match may bring back substantially different results for different people?