ghost · GitHub

@philbooth

Between 14-Mar-2018 and 19-Mar-2018, the content server instances ran
out of disk IOPS because of increased logging caused by heavy traffic
from Fx 59. In order to backfill the lost events, this change adds a
new script that recreates Amplitude events from flow events, which we
keep around in an S3 bucket.
The transformation process is not perfect. For instance, we don't have
the device_id property in the flow data, and it's used to calculate the
insert_id when present. Since the insert_id is what Amplitude uses to
deduplicate events, this is a pretty important caveat.
So, in order for this script to be useful, we need one of the following
things to be true:
* We know that the events that made it to Amplitude did not contain a
  device_id. My hunch is that this is true for most content server
  events, but I can't be certain.
* We can ask Amplitude to delete all content server events for the days
  that we are re-importing. Seems unlikely, although we haven't asked
  the question.
* We are able to deduplicate the events ourselves, by comparing the uid,
  timestamp and event type to the events that made it to Amplitde. This
  one should definitely be true, but that is for a different script.

Read the original on github.com ↗