Getting a higher confidence in spaCy similarity

Posted on Jun 24, 2023

Note: This article was written a while ago and may contain outdated information. Please verify the details before relying on it. If I express opinions or recommendations, they might not reflect my current views. For this reason, I recommend checking for more recent articles on the same topic.

That’s a fast and small addendum to my recent post about similarity calculation of my blog posts. You can find part 1 here.

I used spaCy to get the similarities between my posts, was unhappy with that.

Well, the solution is far easier than I thought.

After some research I found out that using another model leads to far better results!

I used en_core_web_lg in conjunction with universal-sentence-encoder:

nlp = spacy.load("en_core_web_lg")
nlp.add_pipe('universal_sentence_encoder')

And this leads to really good results.

The post that I showed as an example in my last post has now fairly good similar posts:

{
    "post": "https://tty4.dev/development/istio-as-authorization-part-1/",
    "similar": [
            "https://tty4.dev/development/istio-as-authorization-part-2/",
            "https://tty4.dev/development/ext-authz-java/",
            "https://tty4.dev/development/k8s-intro/kubernetes-workloads-resources-lifecycle-hooks/",
            "https://tty4.dev/development/k8s-intro/kubernetes-services/",
            "https://tty4.dev/development/helm-hands-on-overriding-values/"
    ]
}

I think it’s good enough to embed it into my Jenkins pipeline and do it automatically when publishing new posts.

Getting a higher confidence in spaCy similarity

Want to know more?