AuctionBytes Blog
Covering auctions, collectibles and marketplace selling.

AuctionBytes Blog The AuctionBytes Blog has been giving a voice to online merchants since its launch in 2005. Named one of the world's top 30 blogs in 2008 by "Blogging Heroes." Weigh in with your thoughts on the joys and pitfalls of selling online.
Sat Dec 17 2022 16:34:37

Etsy Raises Concerns over Testing It Runs

By: Ina Steiner

Sponsored Link

Etsy sellers aren't the only ones who have concerns about testing the company runs on its platform. Two techies at the company recently raised concerns about it on Etsy's Code as Craft blog.

Etsy data scientists Stephane Shao and Clare Burke were surprisingly candid about the potential for unintended negative consequence of running so many tests on its platform - which Etsy calls "experiments."

Etsy's widespread use of experimentation empowers its teams, they said, but it also means that "many different teams end up experimenting concurrently on shared pages or closely related products."

And that can render isolated results from individual experiments misleading, they said - and worse, it can lead to "negative interactions" where two or more tests, each with the potential to have a positive impact on their own, have a lessor or even negative impact when tested together.

The authors of the blog post used a hypothetical example involving the navigation bar on Etsy that features a black shopping cart icon on a white background. If one team ran a test to turn the white background to orange, and another team independently ran a test where it turned the shopping cart from black to orange, the shopping cart would no longer be visible to shoppers (the orange shopping cart wouldn't be seen against the orange background):


Over the years, sellers on eBay have also expressed concerns about testing the company does on its marketplace. The tests can be disconcerting, since marketplaces general don't disclose when they are running tests. 

Testing can also waste sellers' (and buyers') time as they try to figure out why they're suddenly seeing something different. (The old, "is it a glitch, a test, or a permanent change?" routine.)

Some tests simply seem like change for the sake of change. Remember when eBay suddenly turned eBay Store icons from red to blue in 2013 without telling anyone?

Three years ago, Etsy completed its platform migration to the Google Cloud and heralded the ability to run even more tests as a result ("Etsy Ramps Up Testing after Completing Google Migration.") At the time, Chief Technology Officer Mike Fisher explained the benefits:

"Our migration to Google Cloud has enabled us to shift more than 15% of our engineering headcount from daily infrastructure management to improving the customer experience. The combination of increased brain power and significant compute power has dramatically sped up our innovation pipeline. In fact, Etsy's experiment velocity was up 115% in 2019; this process leads to meaningful insights that help us meet ever-evolving buyer and seller needs, ultimately delivering a more personalized, seamless experience."

Now, Shao and Burke are looking at how to look at Etsy experiments more holistically, which you can read about on the Etsy Code as Craft blog.

While some sellers deplore encountering constant testing, some appreciate that marketplaces at least test changes for unintended consequences before they roll them out sitewide.

Have you noticed any differences in the level and nature of testing Etsy and eBay have been doing on their marketplaces? What are the best and worst tests you recall on either platform? What would you like Etsy techies to know about the impact of testing on your business?



Comments (12) | Permalink

Readers Comments

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: I_Fix_Dolls This user has validated their user name.

Mon Dec 19 01:03:18 2022

It hasn't affected me, but there are quite a few sellers who have been negatively affected by the STUPID idea of making the sellers NEGATIVE FEEDBACK pinned at the TOP of the feedback. It has been like that for MONTHS. How unfair is THAT?
Other testing has affected sellers trying to list or edit their items, The adjusting of the thumbnail pictures still doesn't "stick" when you save, which wastes the seller's time. Etsy may think testing is harmless, but it is not. So tired of having "Behind the scenes" screwing with the pages. Why don't they work on making "Search" work the way it should, or being able to just list without wading through a bunch of "Glitches".. Have they never heard of "If it ain't broke, DON'T FIX IT"?

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: Stone Cutter This user has validated their user name.

Mon Dec 19 01:32:11 2022

***The ignorance of a non-techie CEO***

I worked in tech for many years....family and friends are also in tech. In my opinion, the best run tech companies come from CEO’s who are techies, otherwise the company is nothing more than a sales driven organization run by an ignorant executive team who believes engineering is never that complicated to matter.

If Etsy engineers are publicly expressing concerns about how their codebase is generated, this is a REALLY bad sign; they’ve pretty much had it!  I wouldn’t be surprised if engineers begin to bail to find other jobs.

Chad Dickerson - Etsy’s old CEO, Chad, was a techie. Whenever Etsy made design changes, the changes were much more methodical and streamlined, while Etsy would kindly let Sellers know of these changes, on the Etsy forum.

Josh Silverman - Silverman is not a techie, but a business salesman. And how does any non-techie CEO know who to really qualified to lead its engineering department. I’ve worked for crappy CTO’s who worked for business CEO’s who knew nothing about tech.  The best run tech companies I worked for were run by techies. Since Silverman took over Etsy, the site is filled with obvious bugs that never get fixed. Not only does Etsy generate too many so called “tests”, but they fail at proper Quality Assurance testing before implementing new changes.  And Etsy rudely never tells Sellers of any up-and-coming code changes, which I find incredibly insulting.

For many years, I have always felt that Etsy’s Board of Directors will regret not hiring a technical leader. I’ve been watching Etsy reviews on Glassdoor. Older Etsy engineers say that Etsy use to hire quality engineers, but anymore, they just hire anyone to fill a position.

Engineering teams that are not well run, will eventually fail. There are so many technical problems on Etsy that I know are costing me sales. And my customers tell me of problems when attempting to purchase products that have been on Etsy for years, yet Etsy does nothing about the technical bugs I report.

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: Stone Cutter This user has validated their user name.

Mon Dec 19 09:52:27 2022

***Engineering vs. Business Design***

“It hasn't affected me, but there are quite a few sellers who have been negatively affected by the STUPID idea of making the sellers NEGATIVE FEEDBACK pinned at the TOP of the feedback. It has been like that for MONTHS. How unfair is THAT?”

The problem, stated above, may not be an engineering flaw but rather a design flaw promoted by Etsy’s business & marketing departments, which is a whole different topic of discussion. The role of engineering is to develop what the “business” requires. It’s when functionality flat out does not work which is then the responsibility of the engineering teams.

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: ebayout This user has validated their user name.

Mon Dec 19 18:08:52 2022

GIGO never grows old.

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: ebayout This user has validated their user name.

Mon Dec 19 18:22:58 2022

Does Etsy have too many programmers with nothing to do...like eBay?

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: AntiqueGirly This user has validated their user name.

Mon Dec 19 19:59:14 2022

The negative reviews, in most cases are many years old, maybe when people were  just getting started and made some mistakes but have since learned. It’s not fair that potential buyers see that above all the Gr8 reviews.

Search is still a horrible horrible mess, showing all sorts of things that are irrelevant to tour search.

Also that if you choose highest price as an option most times you barely get any results! Why on earth would Etsy block the higher prices items!

All this with little to customer support!    

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: Sa11yanne This user has validated their user name.

Tue Dec 20 03:26:51 2022

I have been in constant tests for the last 2 years or so, it been so ridiculous that i thought that my shop had actually been hacked, i was logging on everyday and seeing things had been moved around or added to my store which had no benefit to me at all and all were a waste of time like all the tests they keep doing on the whole Etsy site, and most of the tests they do run can be really damaging to shop sales and not just mine, they now have a new search box that keeps coming in and out, with is just above the old search section, its a complete waste of time as its all the same items that are showing in the main search anyway and i am sure its very confusing for buyers, nothing ever works correctly on Etsy, the site is a complete shambles and the App is forever messing up, my last problem on the App was where my items completely disappeared from my shop when someone clicked on one of my items from the search and was happening for months and only found out when i had a customer contact me, i would of been none the wiser as dont use the App and have a feeling its happening all over again, just makes me laugh as they are trying to push the App on everyone and it does not even work correctly! with all these constant tests is just damaging the site and the shopping experience for the buyers, they to do not know if they are coming or going with the constant changes, i was on Ebay for years and never saw any tests or changes in all that time, and just recently gone back on there because of the decline of views and sales on the Etsy site, and still looks the same to me except the shop fronts have changed, but thats it, and thats 15 years or more from when i started using Ebay, there is no benefit to all these tests that keep happing on Etsy and they need to stop, i have been in a new test since May  where if you get a negative review its gets pinned to everything, i have had 3 out of over 1000 reviews and they still insist its the best ones to show the buyers as they click on one of my items, this has had a major impact to my sales and now take a quarter of what i used to take since this started happening, gone from 4k to around 1k if im lucky, buyers see a negative review at the top and just think wow they must have loads of negative reviews if thats the first one they see and then just go else where, the whole things a complete joke, but is not effecting all shops, and the shops with hundreds of really awful reviews do not seem to be having this at all! so its appears only certain shops are being targeted, this is actually not a test its been stated and there to stay but i think this is effecting the shops more so that have been Etsy test shops like my own, just wish they would stop messing all the time and actually sort major problems out with the site, and the more they keep doing tests the worse the whole site becomes.

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: Queen of the Jungle This user has validated their user name.

Tue Dec 20 09:16:33 2022

It's not just the constant testing- it's the entire culture of Etsy which seems centered around their hatred of the sellers.

I hate to be dramatic but it seems there is no other explanation for their sadistic manipulations of the site on a daily basis.

Why would anyone in management think it's a good idea to sabotage sellers by prominently displaying irrelevant, inaccurate and OLD reviews on their listings?

What the hell is wrong with these morons who are supposed to be highly educated tech people?   I guess they all got top marks in gender studies and DEI which supposedly qualifies them to run multi-billion dollar tech companies.

Total disgrace and unfair to both sellers and buyers.

If they are so smart, let them first fix SEARCH.  Then screw around with the other things.

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

This user has validated their user name. by: The End

Tue Dec 20 21:04:38 2022

So, Etsy is concerned about itself ?.....
Need a  better venue or 2......

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: eadovintage This user has validated their user name.

Wed Dec 21 05:49:05 2022

I am also in the experiment / test with negative review placement. Luckily (lol??) my reviews they are pinning to my listings are not horrible like some other sellers. I am still making some sales. Unfortunately some sellers have absolutely terrible sales killing reviews being prominently displayed under all of their listings. It is the absolute stupidity of this test from a sellers stand point that keeps me actively posting and bumping the 2 threads in the forums. Any new thread opened by sellers experiencing this is quickly closed. They monitor the threads pretty close and delete anything they don't want us to know. Or mute the sellers who are most outraged. What is really egregious is selecting a group of sellers for this experiment instead of it being applied to everyone. We haven't figured out the parameters of inclusion but we are working on it. We need some tech minded people to help us. Anybody out there to help?

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: Jacqui This user has validated their user name.

Wed Dec 21 06:08:36 2022

I too am a victim of Etsy's constant testing and am sick of it. They need to put a lot more effort into fixing problems that actually exist, like:
1) The flawed Star Seller Program
2) Search function that doesn't work properly
3) Boot out the thousands of resellers
4) Non-existent seller support
5) A fair review system where sellers can't be targeted by competitors

Instead, they keep increasing seller fees and use the money to run even more idiotic tests with meaningless outcomes. I mean, how stupid do you have to be to think that pinning irrelevant negative reviews to every listing in a seller's shop will have a positive outcome for anyone? Unless they're deliberately targeting vintage, handmade and OOAK sellers in order to drive business away from them.

It appears that Josh Silverman is trying to recreate another eBay, chock full of resellers selling crap. He obviously didn't learn his lesson when he messed up the first time round.

Perminate Link for Etsy Raises Concerns over Testing It Runs   Etsy Raises Concerns over Testing It Runs

by: Sierra This user has validated their user name.

Sat Dec 24 12:44:25 2022

I agree with @I_Fix_Dolls ... and other comments here. I've sold on Etsy since 2006 and have seen many annoying things. Several years ago, Etsy did announce some of the testing they were doing. They made it clear that wasn't a comprehensive list of ALL the tests they were doing, but at least knowing some of them helped a bit! Then they took that away ... probably so the competition wouldn't know what was being tested!

Problems with cropping the thumbnail while listing come and go, but my current biggest annoyance is when I click the link in the sale notification email to print a shipping label. The screen I end up on will NOT generate a label after I've filled in everything. There aren't any error messages so that, along with many years of Etsy selling experience, make me very confident it's not something I'm suddenly doing wrong. I have to cancel out of that screen and go back to my Orders list, check the box to the left of the order for which I want to print a label, then click the "print a label" button and then I end up back at the same screen which didn't work previously ... except NOW it works!

After YEARS of not being able to save a listing as a draft, I recently learned that was because I usually copy a previous listing. Apparently only brand new listings from scratch can be saved as drafts? At least this *does* work. Filling in all the piddly stuff (i.e., why I prefer to start with an existing listing!) takes time, but at least I can save as a draft.

I *soooooo* wish Etsy would kick out the resellers! I've reported dozens (probably hundreds) of shops over the years. I don't always have proof that the item wasn't handmade by the seller or that the shop is a reseller, but I give Etsy my best argument ... such as there is simply NO WAY that any authentic artisan could weave rugs, throw pottery, make jewelry, and blow glass in quantities to make as many sales as they've made. I've recently noticed more "fair trade" sellers popping up; these are just resellers portraying their shop as being beneficial to artisans in less-developed countries.

I'm definitely annoyed with Etsy's current practice of showing negative reviews first. I believe this is actually illegal per the FTC; my understanding is that negative reviews aren't supposed to be treated any differently than positive reviews. I've seen less-than-5-star reviews at the top of my reviews a couple of times, but not very often, thank goodness! However, since each Etsy user sees things differently, shoppers may be seeing things differently than me.  

@StoneCutter said they wouldn't be surprised if other engineering types started leaving Etsy. Mike Fisher IS leaving, or maybe has already left by now. He was Etsy's CTO for many years. So tech things on Etsy may get worse before they get better ... depending on the new CTO's knowledge and experience.  

As for Etsy search, it seems to be taking after Amazon, i.e., showing shoppers things they THINK they might be interested in (RE: Amazon, I got this information directly from an Amazon employee - not Seller Support staff - a couple of years ago; it was the final straw and why I decided to stop selling on Amazon). I've seen the same thing happen on other sites too.



Login is required to post comments.
To sign in to leave a comment, fill in the form below. If you have not yet signed up for AB Verify, or if you'd like more information, go to the Registration Page
.

Login for AB Verify
Be sure and use your email address and password to log in.

 
Email:
Password:
 
 Forgot Your Password?
 Even though you are signed in with the AuctionBytes Blog, you will have to sign in to the EcommerceBytes blog. But you can sign in with your existing AB Verify info.