Living and Dying by the Algorithms

So, how does the algorithm work? This is the great question with e-commerce and online marketing across various social media, search, and commerce platforms. Since the handmade e-commerce site in question went public in 2015, the priorities of the company change after each quarterly board meeting. There is a cycle and culture of pushing for ways to endlessly improve and be tinkered with — AI and bots crawl the site and can remove and shut down shops with little to no warning. Trends can be decided upon and implemented based on keyword searches of a couple hundred individuals, the lists of celebrity influencers, a feature in an advertisement or a selection by an employee of the company.

We are here to focus on the reviews, however, as no marketplace that exists is perfect for consumers and business owners. In essence, the higher a given shop’s reviews, the more frequently their products appear in search results, thus resulting in more sales (Collinger and Malthouse, 2015). This is good, right? Because as a consumer, you would want only the best products shown to you when you search for them rather than the worst or least popular — but what if that popularity was artificially downgraded for smaller sellers and falsely upgraded for drop-shippers or sellers that weren’t hand-making their products? The site attempted to remedy any customer dissatisfaction with a mathematical formula, the details of which were hidden from both sellers and buyers, called the the Order Dissatisfaction Rating (ODR) (Glassenberg, 2020). This ODR included the amount of customer complaint cases brought against a seller and shop ratings, among other metrics. If your ODR rose to an amount higher than was defined as acceptable (at the time less than 1% of reviews could be 1 or 2 stars, over the course of 90 days), your shop would be warned, closed and/or you would be suspended from using the site. Unfortunately, only about 10% of purchasers review their orders, and this metric only took into account the reviews that were made, not the overall percentage of orders fulfilled with satisfied customers.

The ODR approach was ended in 2020 because so many shop dissatisfaction ratings increased after customers were unhappy with the disruptions to shipping and the overall logistics of the planet’s supply chain due to the COVID-19 pandemic. Those customers that felt out of control took it out on sellers in the form of low reviews. In 2021, a program was created called the “Star Seller Program,” which was a way to display the ODR to consumers and sellers. Previously, this ODR metric was only accessed with a non-navigable link which was made available on a Reddit post or discussion boards hosted by other organizations under a now-deleted section of the site called “customer service performance.” The link is now broken. This new program began in 2021 and required that 95% of a shop’s reviews over 90 days be five star reviews, meaning that for every 4 star review a shop needed 19 five star reviews to maintain their standing as a star seller. If a shop only receives reviews on 10% of their sales, a single 4-star review would require 190 additional sales with exclusively five star reviews to recover their star seller standing. In 2022, after considerable seller feedback, this policy was remedied so that a shop had to maintain a 4.8 average star rating over 90 days, significantly increasing the ability for shops to be a part of the program that sets them apart from other sellers.