Generating and Embedding Vector Data with TimescaleDB

I did an experimentation with storing and searching audio embeddings that are generated via the VGGish model and stored in TimescaleDB's vector database. My goal was for generating an embeddings to ultimately gain more experience in working with vectors in TimescaleDB. With TimescaleDB you can analyze audio content, find similar audio segments, and even build an audio-based retrieval system. This script extends the original VGGish inference demo, and can be found on my Github page.

It features:

• Generate 128-dimensional audio embeddings from WAV files using Google's VGGish model

• Store embeddings in TimescaleDB with vector search capabilities

• Process individual files or entire directories of audio

• Perform similarity searches to find matching audio content

• Includes time-series capabilities for temporal queries

• Built on industry-standard tools: TensorFlow, PostgreSQL, and TimescaleDB

I use Google's VGGish neural network to convert audio into a consistent "embedding" representation - a 128-dimensional vector that captures the audio's characteristics. These embeddings are then stored in TimescaleDB with the vector extension, which enables fast similarity searches.

AudioVec implements a vector-based audio retrieval system by utilizing Google's pre-trained VGGish CNN to generate 128-dimensional embeddings from audio spectrograms. These audio files you can generate simply by recording ambient audio in your room if you wish.

The system processes input WAV files by segmenting them into approximately one-second chunks, converting each segment to mel-frequency spectrograms, and feeding these through the VGGish model to produce fixed-size embedding vectors that capture audio features like timbre, pitch, and temporal patterns. These embeddings are then stored in TimescaleDB with the pgvectorscale extension, which creates specialized DISKANN indexes optimized for high-dimensional vector similarity searches using cosine distance metrics.

Query operations perform approximate nearest neighbor searches across the embedding space, with TimescaleDB's time-series capabilities enabling temporal filtering and the vector extension's SIMD-optimized distance calculations providing sub-millisecond similarity matching across millions of audio segments, effectively creating a content-based audio search engine that operates on learned acoustic representations rather than metadata.

The total process is:

1. Audio files are loaded and segmented into ~1 second chunks

2. Each segment is processed through the VGGish model

3. The resulting 128-dimensional embeddings are stored with metadata

4. TimescaleDB organizes the data for efficient retrieval

git clone https://github.com/jamesfreire/audiovec.git
cd audiovec
pip install tensorflow tensorflow-hub numpy scipy psycopg2-binary

VGGish depends on the following Python packages:

• numpy

• resampy

• tensorflow

• tf_slim

• six

• soundfile

These are all easily installable via, e.g., pip install numpy (as in the sample installation session below). Any reasonably recent version of these packages should work.

Download the TensorFlow Model Garden

• https://github.com/tensorflow/models/tree/master

• Place vggish_to_timescale.py file within models/research/audioset/vggish

VGGish also requires downloading two data files:

• Place these within models/research/audioset/vggish

• VGGish model checkpoint, in TensorFlow checkpoint format.

• Embedding PCA parameters, in NumPy compressed archive format.

Instructions on how to install TimescaleDB is available on their site

The code will automatically install the vector extension and setup the audio embeddings table, here called audio_embeddings, or whatever you define at the command line. If you wish to set it up manually you can execute:

psql -U postgres -c "CREATE DATABASE audio_vectors;"
psql -U postgres -d audio_vectors -c "CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;"
psql -U postgres -d audio_vectors -c "CREATE EXTENSION IF NOT EXISTS vector;"

• Run within models/research/audioset/vggish

# Process a directory of audio files
python vggish_to_timescaledb.py \
                 --wav_dir /path/to/audio \
                 --checkpoint /path/to/custom/vggish_model.ckpt \
                 --pca_params /path/to/custom/vggish_pca_params.npz \
                 --db_name audio_embeddings\
                 --db_user postgres \
                 --db_password your_password

You can also process an individual wav file using:

--wav-file /path/to/audio.wav

The code will also generate synthetic audio if no input is provided.

vggish_to_timescaledb.py creates a audio_embeddings table with the following structure:

CREATE TABLE audio_embeddings (
    timestamp TIMESTAMPTZ NOT NULL,
    audio_file TEXT NOT NULL,
    segment_id INTEGER NOT NULL,
    embedding vector(128),
    metadata JSONB,
    PRIMARY KEY (timestamp, audio_file, segment_id)
);

The metadata field contains JSON with additional information:

• segment_duration_s: Duration of the audio segment

• segment_start_time_s: Start time within the original file

• embedding_dimension: Size of the embedding vector (128)

• sample_rate: Sample rate of the original audio

-- Find audio similar to a specific embedding, place the actual embedding as the array below
SELECT
    audio_file,
    segment_id,
    timestamp,
    1 - (embedding <=> '[0.1, 0.2, ..., 0.3]'::vector) AS similarity_score
FROM
    audio_embeddings
ORDER BY
    embedding <=> '[0.1, 0.2, ..., 0.3]'::vector
LIMIT 10;

-- Find segments similar to an existing segment
WITH target_embedding AS (
    SELECT embedding
    FROM audio_embeddings
    WHERE audio_file = 'your_audio_file.wav' AND segment_id = 5
)
SELECT
    ae.audio_file,
    ae.segment_id,
    ae.timestamp,
    1 - (ae.embedding <=> te.embedding) AS similarity_score
FROM
    audio_embeddings ae,
    target_embedding te
WHERE
    ae.audio_file != 'your_audio_file.wav' OR ae.segment_id != 5
ORDER BY
    ae.embedding <=> te.embedding
LIMIT 10;

-- Find similar segments within a time range, place the actual embedding as the array below
SELECT
    audio_file,
    segment_id,
    timestamp,
    1 - (embedding <=> '[0.1, 0.2, ..., 0.3]'::vector) AS similarity_score
FROM
    audio_embeddings
WHERE
    timestamp BETWEEN '2025-05-01' AND '2025-05-15'
ORDER BY
    embedding <=> '[0.1, 0.2, ..., 0.3]'::vector
LIMIT 10;

1. Index Tuning: Configure the DISKANN index parameters based on your dataset size and search requirements:

CREATE INDEX audio_embeddings_embedding_idx
ON audio_embeddings USING diskann (embedding vector_cosine_ops)
WITH (ef_construction = 128, m = 16, ef_search = 64);

1. Parameters to tune:

   • ef_construction: Higher values increase build time but enables more accurate search results (64-512)

   • m: Maximum number of connections per node (8-64)

   • ef_search: Controls the accuracy vs. speed tradeoff for queries (higher = more accurate but slower). This parameter specifies the size of the dynamic candidate list used during search. Defaults to 40. Higher values improve query accuracy while making the query slower

2. Chunking: Process large collections in smaller batches to manage memory usage.

3. Hypertable Tuning: Customize the chunk size based on your query patterns:

SELECT set_chunk_time_interval('audio_embeddings', INTERVAL '1 day');

For non-WAV formats, use ffmpeg to convert first:

# Convert MP3 to WAV
ffmpeg -i input.mp3 -acodec pcm_s16le -ar 44100 -ac 1 output.wav

I found this a great way to get started with experimenting with generating embeddings along with trying out an incredibly fast database that is free to use. Enjoy!

• TimescaleDB for the vector extension capabilities

• Google's AudioSet team for the VGGish model

• TensorFlow team for TensorFlow Hub