Deduplicating My VictoriaMetrics Data

Although most of my time-series data still lives in InfluxDB, I also have VictoriaMetrics running and playing host to daily todo list stats along with a few other things.

However, I recently realised that I screwed up when deploying it: I'd assumed that de-duplication of data was a given, but actually dedupe is disabled by default.

In order to fill gaps, my cronjobs tend to write data for multiple days at a time, which has resulted in duplication of the data, causing inflation when reporting total values.

This short post talks about enabling the deduplication feature in VictoriaMetrics as well as tidying up existing data.

---

Impact of Duplicated Data

Quite a few of the stats that I collect in VictoriaMetrics are ultimately visualised by summing or counting values:

sum_over_time(todo_list_completed{db="workload_stats"}[1d])

This results in a graph like the following

Now, I work bloody hard and do get through a lot in a day, but 1600 items is a ridiculous amount: assuming an 11 hour work day (12 with lunch), that's 2.42 items per minute.

Admittedly, this is an extreme example (I was probably debugging the script and running multiple times), but it makes the issue quite clear.

We can use the VictoriaMetrics export endpoint to dump data to see the extent of the duplication (note: this metric has a single series, so I've not bothered filtering by label):

$ curl -s "https://$MY_VM/api/v1/export?match[]=todo_list_completed" \
  | jq  '.timestamps' \
  | sort \
  | uniq -c \
  | sort -nr \
  | head -n 5
     40   1780696800000,
     40   1780610400000,
     40   1780524000000,
     31   1781215200000,
     30   1781128800000,

That first timestamp is 5 Jun, and the values exist some 40 times. The graph reports 1520 items for that day, so if we divide that by 40, we can predict that the de-duplicated value would probably be 38.

---

My Misunderstanding

Although I know that the storage model is very different, I'd assumed that VictoriaMetrics' handling of out of order writes worked something like InfluxDB.

If the same duplicates were written into InfluxDB, they'd essentially never have surfaced:

• During query, the values would have been merged, only returning the latest
• TSM compactions also perform a merge, physically removing the older values

However, although VictoriaMetrics supports deduplication it's not enabled by default, allowing duplicate values to amass and surface during aggregation.

---

Enabling Deduplication

Enabling de-dupe support is just a case of passing a command line flag:

-dedup.minScrapeInterval=[interval]

The value passed defines the acceptable interval of the data itself: setting it to 15m means that records with timestamps within 15 minutes of each other will be collapsed into a single record (using whichever has the highest value).

My VM instance hosts stats that are written (roughly) once a day, so I updated my docker-compose to set the interval to 12h:

    victoria_metrics:
        restart: always
        image: victoriametrics/victoria-metrics:v1.122.0
        container_name: victoria_metrics
        command:
         - "-selfScrapeInterval=5s"
         - "-storageDataPath=victoria-metrics-data"
         - "-retentionPeriod=100y"
         - "-disablePerDayIndex"
         - "-dedup.minScrapeInterval=12h"
        ports:
         - 8428:8428
        volumes:
         - /home/ben/docker_files/files/victoria/data:/victoria-metrics-data

Realistically, it would have had the same effect if I'd set 1s or 15m - the main thing is that, to preserve data, the interval should be less than or equal to the granularity of the data itself.

---

Handling Older Data

There was, however, a catch.

Deduplication is performed during background merges (compactions in InfluxDB parlance) as well as at query time.

Merges are only performed if a partition has been touched, which means that old/existing data could potentially not be de-duplicated for quite some time (if ever).

Arguably, this may not matter (do we really need to care about duplication in data that we're not actively querying?), but leaving it as-is would also leave scope for confusion in future.

Happily, though, it's an issue that's quite easy to address: we can tell VictoriaMetrics to perform merges on specific partitions, triggering deduplication in the process:

curl 'https://$MY_VM/internal/force_merge?partition_prefix=2026_06'

I first set VictoriaMetrics up around June last year, so I wrote a short loop to iterate through the months since then

d="2025-06"
while [ "$d" != "2026-07" ]
do
    echo "Merging $d"
    # The partition prefix uses underscores, so replace hyphen with underscore
    # in the querystring
    curl "https://$MY_VM/internal/force_merge?partition_prefix=${d/-/_}"
    # Give the merge time to run before kicking off the next
    # we're relatively data light so a minute should be enough
    sleep 60
    d=$(date -d "$d-01 + 1 month" +'%Y-%m')
done

Note: on systems with lots of data, merges can be quite I/O intensive, so you'll probably want to allow more than 60s between calls (this is also why I didn't simply do two calls with prefixes of 2025 and 2026).

VictoriaMetrics logs when de-dupes start, as well as when they finish:

2026-06-14T11:44:39.052Z    info    VictoriaMetrics/lib/storage/table.go:495    start removing duplicate samples for partition (/victoria-metrics-data/data/big/2025_12, /victoria-metrics-data/data/small/2025_12)
2026-06-14T11:44:39.293Z    info    VictoriaMetrics/lib/storage/table.go:499    finished removing duplicate samples for partition (/victoria-metrics-data/data/big/2025_12, /victoria-metrics-data/data/small/2025_12) in 0.242 seconds

Once all of the merges had completed, rechecking the export for duplicates confirmed that they had all been deduped away:

$ curl -s "https://$MY_VM/api/v1/export?match[]=todo_list_completed" | jq  '.timestamps' | sort | uniq -c | sort -nr | head -n 5
      1   1781301600000
      1   1781215200000,
      1   1781128800000,
      1   1781042400000,
      1   1780956000000,

My graphs also reported much more reasonable values for the original time-range:

We can see, too, that the original prediction of 38 items was correct.

---

Comparing Values

It's important to note that, even with deduplication enabled, VictoriaMetrics behaves differently to InfluxDB.

InfluxDB's merges prefer the latest value, whereas VictoriaMetrics selects the maximum value.

So, if we consider a duplicated datapoint:

some_measurement,tag=foo field1=10 1781435963000000000
some_measurement,tag=foo field1=5 1781435963000000000

The two databases will return different values when querying that timestamp:

Database	Value
InfluxDB	5
VictoriaMetrics	10

This doesn't matter too much for the data that I'm storing in VM (subsequent writes will always be >= the original), but it is something that's worth keeping in mind.

---

Conclusion

Switching between software always brings challenges but, in this case, I was quite heavily misled by my InfluxData heritage and relied on functionality that I've long been able to take for granted.

Although this post highlights quite an extreme example, the impact that it had on my stats was generally more subtle - doubling or tripling values - reported values were out, but often still within a semi-plausible range.

Thankfully, resolving that has proven to be quite simple.

If I'd been willing to wait, most of the historic data would probably have deduplicated without me forcing merges, but doing so was simpler and helps ensure that I have certainty about the state of data going forwards.