Adding a Real-Time Subtitles to OBS Stream
TL;DR
Using OBS for camera capture and AWS Transcribe WebSocket API I was able to add a real time subtitles on top of my camera feed, which can then be used, via OBS Virtual Cam plugin, as an input for Zoom/Skype/Meet calls.
The Intro
Since corona isolation started, we have been flooded with screenshots of people hanging in calls, working, and trying to get on with their lives. After a while, you get bored with your same old meetings, so people, being creative as they are, are now starting to play around with green screen options in Zoom. And you can see some really cool examples like those:
I was wondering, can you go one step further, and add real time subtitles to your stream.
After some research, I found Amazon Transcribe and, better yet, it turns out that they have support for web sockets and real time translation.
Ok, Jump To The Juicy Part
List of prerequisites
- Amazon IAM role with Access ID and secret key
- OBS Studio
- Camera
- Microphone
Unless you are using an admin amazon user (which is never a good idea), first you need to give your user a rights to use this AWS service. We do this by attaching this policy to your AWS user.
After you are done with this, go and checkout this awesome example repository by AWS team.
https://github.com/aws-samples/amazon-transcribe-websocket-static
From their README file:
1. Clone the repo
2. run npm install
3. run npm run-script build to generate dist/main.js
Once you've bundled the JavaScript, all you need is a webserver.
For example, from your project directory:
npm install --global local-web-server
wsLeave this running in a terminal, and open your browser on http://localhost:8000
You should now see a UI like this:
As you can see, the app is working, I've just said "Hello", and got a response from AWS Transcribe. Thing is, this UI is not the best candidate for subtitles, so we are gonna tweak css styles a bit, and make it more friendly for OBS.
Lets change styles for body and #transcript text area
body {
margin: 20px;
background-color: rgba(0, 0, 0, 0);
}
#transcript {
margin: 30px 0;
white-space: pre-wrap;
background: #00ff00;
color: white;
font-weight: bold;
font-family: Arial, Helvetica, sans-serif;
font-size: 26px;
text-shadow: 1px 1px 0px #000000;
}
Awesome, nice green background, perfect for chroma key (aka green screen) effect.
Now, start your OBS studio, and add a new scene (I named mine CAM + SUBS)
Next, select that scene, and add two sources, first add a camera (Video Capture Device) and then on top of that, add window capture (I named mine CAM and Chrome Capture.)
In window capture screen select chrome window (if it's not showing in the dropdown, you might need to select show windows with empty names)
You should now get something like this:
Good news is - things work. We just need to fix it so that only green part remains on the screen.
Right-click on chrome source, and select Filters
First, add a crop filter, and play with top/bottom/left/right numbers, until only green text area remains
One final step is to add Chroma Key filter (it's using green by default) and your green will disappear, leaving only text that comes from AWS over your camera video.
Final result:
One final tweak is to move/scale subtitles to a position of your liking.
If you are on Windows, good news for you is that there is OBS Virtual cam plugin, where you can make this output to a virtual camera device, and then use this as a input for Zoom/Skype/Meet videos. For Mac, there is bounty open to create native virtual cam support, there are even some users who forked OBS and made it work in their repos, but I still didn't test those.
Important Disclaimer!
I made this just as a quick demo, and spent ~20 minutes playing around with this. Even during that short period, I saw some nasty words come out as a result of:
- me being too far from a microphone
- me not being a native English speaker
So, before you decide to show off in front of your colleagues or your boss, consider implementing a blacklist, so you don't get into trouble.
Pro-tip:
Since you are likely to refresh this Chrome window many times, it might be easier to write your Access ID and secret key directly into the index.html file, so you don't have to paste them every time you refresh the page.
Follow me on twitter @kodisha
Discussion on Hacker News