Ina Steiner EcommerceBytes Blog
News and insight focusing on ecommerce.
by Ina Steiner, Editor of EcommerceBytes.com
Tue July 26 2022 22:28:40

Questions Marketplaces Should Ask before Running Tests

By: Ina Steiner

Sponsored Link

It can feel like marketplaces are incessantly running tests to make tweaks or test new features, which can prove disruptive to sellers' routines. And because marketplaces rarely disclose the existence of a test, never mind the nature of a given test, it has gotten to the point sellers can't differentiate between tests and glitches.

So when Etsy engineer Theresa Eades wrote about tests in a post on the Etsy Code as Craft blog, we thought sellers would be interested in what she shared.

First, she disclosed some examples of issues her team encountered in two prior tests (which Etsy calls "experiments"):

"Two years ago, when my team worked on adding video to the listing page, we had to restart the experiment because we forgot to include specific event logging for our feature. 

"Last year we implemented the key functionality for an experiment with the "add to cart" button, but neglected to get accessibility and text strings translated early, costing us a delay till the translations could complete."

She then explained how she created a standard checklist to help reduce such pitfalls when conducting product experiments.

Here are some of the more interesting questions on her checklist:

- Is there anything in my experiment that could degrade performance of the site?

- What populations should be included or excluded from the experiment? 

- When should users see this feature? (Which pages, signed in/signed out, mobile, desktop, etc.)

- Where/when should bucketing occur?

- Will the experiment conflict with any other experiments? Do the experiments need to run exclusively?

- What countries should the experiment run in (can impact translations)?

Take a look at the others and let us know what you think of tests on Etsy and elsewhere. What problems do you encounter, and what changes would you like Etsy and other marketplaces to implement to reduce disruption to your daily tasks?

Comments (9) | Leave Comment | Permalink
Readers Comments

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: Snapped This user has validated their user name.

Wed Jul 27 06:54:15 2022

“Two years ago….we forgot…”

“Last year…neglected to…”

“ She then explained how she created a standard checklist..”

That’s the problem, right there.  Experiment first, THEN ask questions?  Why wasn’t such a ‘checklist’ developed and mandated before the first hack was unleashed?  It has to come to being as a ‘personal’ and ‘evolving’ tool years after the fact?  Then blogged about as if it were some new ‘discovery’?

It’s quite evident this - and too many other - foundational elements in complex interfaced system’s SW T&E are sorely lacking for all these platforms.  One doesn’t need a 25+ year experience base in DoD systems integration and SW T&E (since before ‘computer science’ existed as a discipline) to see that.

SW testing has three levels that must be examined and proved operational BEFORE the new version code is dumped on the unsuspecting user, and even then as a ‘test’ only, prior to official release and reliance.

Start with specification testing.  Independent of any ‘utility’ assessment, does the code DO what it was SPECIFIED to do when the button is pushed?  (Example - does the number 6 change to a number 9 as designed?). That should be exhausted in a lab setting, disconnected from any operational interface.  And it should be done for each ‘box’ n the system chain impacted independently, be it by code changed, or due to a changed interface message as result of code change.  

Next comes functional testing.  With the code processing as specified, is the JOB (result) what was specified?  Is the man/machine interface supported and viable?  Are the data within tolerances?  Any smoke or seizure issues?  Is that new number 9 displayed in the correct place for the correct time in the correct user application?

Then comes integration testing.  Again, per specification, if I push the button on box A, does box C respond correctly without melting down something unintended in Box B, arrayed between them?  When I change the 6 to a 9, does it scramble the other digits in the database somehow?

And ALL of this should be done on specially designed TEST platforms, identical as possible to real world conditions, but not the actual real world systems who’s ‘results’ can wreak havoc on lives and livelihoods.  

Software testing is a science.  It’s not a ‘craft’ to be conducted as an afterthought to coding.  Not hunt and peck and see what happens so one can get better at it.  Results should be predicted and predictable, and if not, stop THERE and understand it, then fix it before proceeding.

And THEN, only THEN, can the new version be tried ‘in the field’.  But even so, all users should be specifically alerted, set population limited to ‘test conditions’, and data still collected, so any further ‘glitches’ can be corrected BEFORE an official release.  That’s the ‘beta’ testing most are familiar with.

Moreover, strict version control must be maintained.  That’s called configuration management, (what works with what, and what doesn’t), and it seems to be a foreign concept to these guys.  Especially critical if there are user ‘classes’ or ‘types’ or ‘systems’ that are to be ‘excluded’ from access to the changed versions by operational convenience or necessity.  It’s also critical to maintain in case the grand experiment goes south, so the previous ‘working’ version(s) can be quickly re-established.

The OP author has it right to establish and rely on a checklist.  And, though not mentioned, specific and established (certified) test PROCEDURES.  But that should exist BEFORE the first test ever occurs, not be developed from them, after the fact.  And just as an aside, it can’t be both ‘personal’ and ‘standard’.  

Clearly, adequate SW T&E costs money.  Much cheaper to toss it out there and THEN (try to) fix (what SEEMS to be) the problems - but as it has been clearly demonstrated, not only does that make finding and fixing the actual problem more difficult and takes more time, there’s no guarantee the fix is adequately tested either.  Sound familiar?

And what we are left with is Swiss cheesed code, constantly glitching, constantly being “worked hard” on by cluelessly disconnected “teams”.  

One last thought for the toolbox.  Don’t just test what ‘should’ work.  Test what shouldn’t work too, to be sure not only it doesn’t, but doesn’t cause some other issue.  

And this is by no means limited to an ‘Etsy’ problem either.  

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: shut1968 This user has validated their user name.

Wed Jul 27 14:37:29 2022

A long long time ago Ebay use to do beta testing before making live changes to the site, there would be almost like a second site with the changes and sellers would get invited to test it out before hand for any bugs.. then when it went live for everyone if there was an issue you got credits etc etc for any down time... somewhere along the way they just said screw sellers, make the changes and fix it as you go.. I personally believe it should be illegal for any Marketplace selling site to do changes without beta testing first since people's business and livelihood are not affected.

I can remember one site about 15 years ago. can't think of the name of it now, it was a decent selling site back then, "offer" or something like that did an update that crashed the site and wiped most of it's database in mid November, right before the holiday rush and didn't get it back up and running right for 6 months at least.. screwed all their sellers and was never the same from that point forward.

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: LDWCallsOut This user has validated their user name.

Wed Jul 27 20:00:30 2022

Are you thinking of iOffer?  They used to dominate the counterfeit luxury brand market online, and then Etsy took that over.

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

This user has validated their user name. by: The End

Wed Jul 27 21:17:19 2022

Ya, Ya, Ya,...Blah, Blah Blah.....
I was there when Etsy first started.....around 2008.
It was sleek and functional.
I brought the action.
And I could make a decent living.
The new "humanoids" ruined it.
It's useless as good steady income now.
The new owners crippled Etsy beyond repair.
And I firmly believe they meant to do it.

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: Bubbles This user has validated their user name.

Thu Jul 28 00:31:38 2022

Like most web platforms - they throw new stuff at the wall to see if sticks - then they TRY to fix the problems it has created when it doesn't - usually creating even more problems that never get fixed.

They waste so much time, money & resources making changes this way instead of having a duplicate site to experiment with and get it working perfectly BEFORE applying it to the main website.

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: spooky This user has validated their user name.

Thu Jul 28 09:16:57 2022

last month the eBay app for iphone became unusable to list an item. One glitch after another. I contacted eBay Twitter and told them about it. After a week I contacted same again and they said there is a ticket on it and they are aware and working on it. I knew what it was and told them - they had made "upgrades" attempting to have the seller buy high visibility add ons at extra cost but those links had made the app unstable. It took 5 weeks to correct. Multi billion dollar company can not get the most simple basic things to work!  

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: Queen of the Jungle This user has validated their user name.

Thu Jul 28 09:22:07 2022

Until the geniuses at Etsy can fix the search function, they shouldn't bother with anything else.

All of their engineers and tech people are politically correct hires who "check all the boxes" when it comes to race, sex, political affiliation.  People who work for Etsy are not there because of merit or skill.

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: LDWCallsOut This user has validated their user name.

Thu Jul 28 10:45:28 2022

WJW, Queen of the Jungle!  Your post nailed it.  

About Etsy's politically selected hires, that does seem to be the way employees are selected.  It's not only the tech people.  I remember more than a decade ago an Etsy staffer (anyone remember Hey Michelle?) posting in the Forums that she felt bad when processing reports of reselling that would get items removed or shops closed because that could interfere with a reseller's ability to earn a living!  Like, who do you think you're working for, sweetness?  

Last week I was in an Etsy ''experiment'' with search results pages.  At the top of the first page of results were two new lines of smaller thumbnails.  The top one was headed ''most popular'' for (my search term), and the second on was ''priced from $50 to $100'' for (my search term).  I am using a larger screen desktop, and the experiment pushed the actual search results down enough on the page so that no real results were completely visible above the bottom of the screen! without scrolling down.  Who dreams up this kind of feature?

Even worse, the items shown in the 2 experimental lines didn't match my search terms at all!  When you are searching for handbags, what do pet items (for dogs only), items for babies and pre-school age children, wedding items, and crafting supplies fit in?  And when I tried searching for other kinds of items, the experiment was always off in a similar way.

After Etsy started including description text keywords in their algorithm for search results, search has gotten much worse -- more than I would have thought possible.  With the holiday shopping season just weeks away now, it might be a good idea for Etsy to clean up the mess they made with reading descriptions for keywords rather than try to force shoppers to look at more dog, baby, toddler, wedding, and DIY crafting supply listings.  

Perminate Link for Questions Marketplaces Should Ask before Running Tests   Questions Marketplaces Should Ask before Running Tests

by: Sierra This user has validated their user name.

Sat Jul 30 12:18:50 2022

@Snapped - I agree with what I could understand in your comment. YES, testing should be done on a mirrored server, as it used to be, but I'm told that's become incredibly expensive ... so that's apparently why many/most sites now test on the live site. I **totally** agree with your comment about ''cluelessly disconnected teams.'' For YEARS, I've had a feeling that Etsy's internal teams don't communicate well, if at all.

@TheEnd - Etsy started in June 2005. I opened my Etsy shop in August 2006 and have been there ever since. As a public company, aren't the shareholders the ''owners''? Did you mean the new management (Josh, in particular, who isn't so new anymore)? Etsy started downhill in October 2013 when ''production partners'' (then called ''outside manufacturers'') was officially announced. I hope someone can get Etsy cleaned up and make it a better version of what it was originally intended to be, but I'm not holding my breath on that.

@LDW - I remember ''HeyMichelle!'' Several years ago, Josh spoke about his ''cushion to couch'' strategy; this is about selling higher-priced goods and is apparently a strategy to make lower-priced imports sink farther down in search. As for seeing irrelevant listings in Etsy search, they're just copying Amazon. Again.  



Login is required to post comments.
To sign in to leave a comment, fill in the form below. If you have not yet signed up for AB Verify, or if you'd like more information, go to the Registration Page
.

Login for AB Verify
Be sure and use your email address and password to log in.

 
Email:
Password:
 
 Forgot Your Password?
 Even though you are signed in with the AuctionBytes Blog, you will have to sign in to the EcommerceBytes blog. But you can sign in with your existing AB Verify info.