Vertex AI Router: An open source API back-end

A few weeks ago I was integrating a little AI helper into the Vendaly app that I’m building at the moment to support filling out forms and making it easier. I realised that this code might actually be useful for others and have decided to open source it under the Apache 2.0 license 🎉

It’s a extremely lightweight and very fast API in typescript that serves a http endpoint for other services to connect to. The idea here is to serve it internally to your cloud services rather than to the public internet, but you can of course do that if you want to.

It comes with a single endpoint as a starting point for you to call which summarises text for you. The idea is that you create your own endpoints with your own system instructions.

Here’s the code: https://github.com/jgunnink/vertex-ai-router

To start making changes, fork the repository. Once you’ve completed your code changes and are ready to deploy, connect your repository of the fork to cloud build and then make sure the cloud build service account has the right permissions to read, write to artifact registry, and permissions to deploy to cloud run, and the cloudbuild job which is included in the repo will build, push and deploy the code to cloud run.

Once it’s deployed, you’ll need to give another service the permission to call it, assuming you’re deploying it without public access, and then you’ll also need to make sure the service account the service is running as has the right permissions to call Vertex.

To call Vertex, you can use one of the predefined roles, but personally I couldn’t find one which only had the permissions to make an inference call. So I made a custom role for it: Vertex AI Predictor

It’s only got the one IAM permission: aiplatform.endpoints.predict

To wrap up, vertex-ai-router is a lightweight, TypeScript-based bridge designed to help you deploy internal AI endpoints quickly. By leveraging Cloud Run and Cloud Build, you can maintain a secure, private interface for your services to interact with Vertex AI without exposing them to the public internet.

The project is open-source under the Apache 2.0 license. If you want to streamline how your cloud services handle text summarisation or custom system instructions, feel free to fork the repo and get started. Link to repo on Github: https://github.com/jgunnink/vertex-ai-router