Penny Predictor

Now that we have the punchline in the heading, let's earn it :)

Welcome!

If you aren't familiar with Penny auctions, you might be surprised to hear about a brand new Xbox and 4k TV selling for $171.90.

Penny auction sites have been providing these deals using a unique bidding structure. Every auction participant pays for the ability to place a bid. When a bid is placed, the price increases by $0.01 and the time remaining resets to 10 seconds.

For the Xbox, this means 17190 bids were placed. If each bid costs $0.15 to place, the penny auction site makes a total revenue of $2750.40

Here's a list of 5 randomly selected auctions along with the revenue the site generated.

$50 Home Depot® Digital Gift Card + 100 Bids

Final Price: $36.68

Site Revenue: $586.88

Luxury Linens™ Premium Ultra Soft 6 Piece Bed Sheet Set - Queen - Sage

Final Price: $8.02

Site Revenue: $128.32

125 Bid Pack!

Final Price: $6.55

Site Revenue: $104.80

Longvadon Genuine Italian Alcantara iPhone 11 Pro Max Case - Midnight Black

Final Price: $9.40

Site Revenue: $150.40

Kamikoto - Kanpeki Knife Set + Yanagiba Knife & Sharpening Whetstone

Final Price: $28.42

Site Revenue: $454.72

Revenue calculator

End price $171.90

Neat, but can we make money with this?

If we were really good at predicting when an auction ends, we could minimize the bids we place (our cost) while maximizing our chances of winning.

People pick up on patterns, but it's easy to become too fixated on a few aspects. Humans are good at simplifying problems, but they lose some nuance with that simplification.

We're going to use machine learning to try to avoid bias. Don't worry, this is homegrown organic stuff. Since our input isn't text, it doesn't make sense to use an LLM like ChatGPT for this. Let's use the right tool for the job ;)

Machine learning tries to find the simplest explanation for a problem, so it'll come to a simple conclusion. After all, If you only ever watched 3 auctions and they all ended at $1.10, you may be inclined to believe that all auctions end at $1.10.

In order to find patterns, we need data so that our “simplest explanation” fits what actually happens in the world. Maybe auctions that happen on the weekends typically end with higher prices. If we only train on 5 days of data, we'll miss that insight.

Collecting Data

When predicting the end of an auction, we likely want to know a few things:

1. When users place bids
1. What time of day/day of week
2. How much time was left on the clock
2. Who places those bids
3. Metadata about the auction
1. How much is this item worth?
2. Any special deals?

I collected roughly 2 weeks of data with second-precision. I recorded just shy of 10 million bids in over 10,000 auctions. The raw data (and the code) is available here.

Clean the data

Cleaning the data basically means making the data easier to understand. If the model needs to make sense of the messy data and draw insights, it's not gonna be as effective as just drawing insights. The more focused the model's task is and the less unrelated work it has to do, the better.

The code for data cleaning is available here.

Training the model

Now that we have the data and know what we're predicting (if this bid is the end of the auction) let's go over our strategy for training the model.

We want the model to have a bit of context for what's happening with the auction. As such, we're going to give it the 9 most recent bids and some metadata about the auction.

Our model architecture looks like this:

Don't worry if this doesn't make sense/seems unapproachable, it took me about a month to end up on this architecture. I'm kinda winging it, but as far as I can tell that's just machine learning :)

The LSTM layer allows us to pass in a 9x8 matrix of data. The LSTM is useful since it can see how each element is changing from bid-to-bid. For example, it can see that a bid placed at 10 seconds is followed by a bid placed at 3 seconds and learn what that means. The LSTM basically lets us tell the model “hey, these are 9 sequential instances of the same information about bids, so they're probably related/similar”.

Then we also pass in a vector (list) of 12 metadata items. These two get normalized (basically big numbers get divided down and small numbers get multiplied up till they're in roughly the same ballpark as eachother) and are concatenated together.

Finally, we have a few dense layers to allow the ML model to work its magic and (hopefully) understand the data.

Let's try training it!

After feeding it roughly 60% of the bids, we'll validate on 20% and ask it for predictions on the remaining 20%.

Here's 10 random predictions:

1. .000
2. .000
3. .003
4. .002
5. .000
6. .000
7. .000
8. .000
9. .000
10. .000

Well this doesn't seem very helpful - they're all nearly 0%! What happened here?

We can find out by looking at the number of positive (auction ended) and negative (auction did not end) datapoints we trained with.

Well this makes sense. If 99.999% of the time an auction doesn't end, and you ask someone to predict if an auction is going to end, it's a safe bet to just say it doesn't end. Especially since the penalty for a false positive (saying it will end when it doesn't) is the same as the penalty for a false negative (saying it won't end when it does). It's safer for the model to just predict that the auction never ends.

So how can we overcome this?

There's three main strategies for this.

1. We can oversample our positive class, meaning we duplicate a bunch of examples of an auction ending.
1. This can cause the model to memorize what those data points look like rather than actually learning a strategy.
2. We can undersample our negative class, meaning we delete a bunch of examples of an auction not ending.
1. This throws out 99.93% of our dataset, which doesn't seem ideal for learning broad generalities.
3. Class Weights
1. This adjusts the reward/penalty for getting an answer wrong so that it's 99x worse to misclassify an auction ending bid as it is to misclassify an auction continuing.
2. Basically we're telling the model it's worth it to predict the end as 50% likely, even if it's really 0.5% likely.

Here's 10 random predictions:

1. .000
2. .010
3. .569
4. .284
5. .374
6. .000
7. .767
8. .000
9. .593
10. .587

Much better!

Since we used class weights, 50% no longer means the model thinks it's 50% likely to end on that bid. We need to correct for the change by using a threshold.

A threshold is a cutoff point where items above that point are classified as a positive prediction (auction will end), and items below that threshold are classified as a negative prediction (auction will not end).

Here's two graphs showing the tradeoff in false negatives/false positives as you adjust the threshold.

True positive vs False positive

False positives are bad since they tell you that an auction is going to end when it actually won't.

True negative vs False negative

False negatives are bad since they tell you that the auction won't end when it actually will.

When using this, you likely want to minimize the number of false positives and false negatives, but decreasing one increases the other.

Once you have a threshold you're ok with, let's continue on. No changing it once you see the real-world data - that ruins the point of analyzing at these charts!

So how good is this model, anyways?

It depends what you're asking it to do.

Let's make a really bad benchmark that ignores a lot of the realities of the situation ;)

Imagine the model was able to place bids. If it guesses the ending bid correctly, it wins the item. We'll use the website-provided "buy-it-now" price as the item's value.

Placing a bid costs the model $0.15. We'll optimize the threshold to maximize profit.

The best split (0.02) spends $142,920 in bids/buying the items it wins and produces a revenue of $537,148 every 7 days (A profit of $394,227.63 per week).

Nearly $400k profit a week?

Thats... a lot of money.

And the model is here for free - it's actually running locally in your browser right now. But beware, you aren't gonna make that much using the model.

Flaws

Now that's not to say you can't make a fraction of that $400k, just that there are a bunch of reasons why the model isn't optimal for earning you that amount.

Firstly, the model predicts whether a certain user is going to win. Since you can't assume the identity of other bidders, you placing a bid will shift the predicted odds. You could theoretically feed your future moves into the model to determine the best time to place a bid, but that requires you to stay in the auction (and spend money bidding).

I really made a model ideal for the website owners to use for prolonging auctions with fake bids. When my model detects an auction will likely end, just place a few fake bids till the site turns (even more of) a profit.

Secondly, the real value of the items aren't even close to the buy-it-now price. Who's gonna buy a secondhand chandelier for ~$1,750? Liquidating that is gonna be a pain.

The site does have a way to convert wins back into bids, which is how they get people to bid on things like chandeliers.

Well this has been fun

When I started this project I was in middle school. Obviously I haven't been working on it this whole time (I'm now at my first job post-college), but it's been something I've come back to and looked at from different angles. I think I'll always come back and take a stab at related problems, but this seems like a good stopping point.

Sources

Further reading

While I didn't discover this until late in the process, I was able to replicate their results in Table 1. Since so many bids fall into the 8 & 9 second bucket, it's easy to classify those so their accuracy numbers are buoyed a bit by that. The final bid is more challenging to classify.

I did like their analysis of types of bidders. I could imagine a system that has long-term memory tracking the "type" of bidder (or metrics about them) and feeding that into the model for improved results.

Tech used

Rusty version of python's pandas. It seems to only have APIs that it can make fast. As a result it takes a bit to port your pandas flow, but once you do it'll be quick.