I'm astonished by how quickly three months have passed, and by the lack of progress I've made. However, I managed to find my stride in the final four weeks, as you'll see here:
Here's a breakdown of the last three months:
In the first month, I spent considerable time setting up my server. I contemplated purchasing an energy-efficient computer like a Nuc, but since I already owned a gaming PC, I decided to use it. The major drawback is its high energy consumption, roughly ~100W/hr.
I occasionally play performance-demanding games, so I wanted to maximize my server's performance while running it 24/7. Initially, Proxmox seemed the most viable option, but its setup and maintenance were overly complicated. Consequently, I opted for Hyper V. I also planned to transfer a few services to offset the cost of running my server, but maintaining some of the service is such a pain in the ass. I’m still using it for a few things, but not sure if I will keep.
Takeaways: I wouldn't recommend an average person to run a home server. It's quite time-consuming with minimal added benefits. Yes, you “own” your data and have more control, but in terms of user experience it is always worst than any of the paid services out there. Despite this, it served as a good learning experience, and made me appreciate certain paid services and their cost efficiency :D
The last four weeks where very productive from an engineering perspective *haha*.
Choosing My Initial Tech Stack
Initially, I chose Next.js + Python as my primary tech stack, but found it frustrating. I struggled with error handling in Python, which hindered my workflow. I selected Python because I wanted to learn more about it, and it's the preferred language for anything related to ML/AI. However, I realized that if Python was necessary, I could always integrate it into my stack later.
Instead, I settled on Next.js and Node.js. Another crucial decision was whether to use the new app router or the page directory approach. I opted for the latter, given its extensive documentation, reduced "magic", and easy-to-understand filename-route relationship. I also believed it would facilitate onboarding other team members, since it's entirely JS-based.
For hosting, I chose Vercel after considering Render, Railway, and Fly.io. Despite contemplating Amazon Lambda, my experience with setting up my server made me realize I didn't want to deal with production engineering. I needed to keep things simple. Vercel, being the native Next.js platform, requires zero configuration.
Vercel is the Shopify for developers like me. It lets me focus on the product without worrying about configuration and scaling. It just works.
I explored Prisma and TRPC, but ultimately decided against them. While they make sense in terms of developer experience, learning a new system seemed unnecessary at this point.
Takeaways: Keeping things simple during project setup is crucial. Select a tech stack that is designed to work together. Additional frameworks or infrastructure can be integrated when necessary.
Choosing 3rd Party APIs for Text to Speech Conversion
In my previous post, I decided to use Play.ht and explored Elevenlabs. However, when I understood the average word count of an article, the associated costs, the API limitations for long-form audio, and that all these services are wrappers around Google's API, I decided to reconsider Google's text to speech.
Cost of Text to Speech for 1 Million Characters:
Elevenlabs: $228 ($99/month + $120 extra usage)
Play.ht: ~$100/month (200k words)
Google: $16 or $160 (Studio voice)
Amazon Polly: $16
Initially, I struggled with Google's API. Unlike other services that only require an API key for authentication, Google uses a credential file that doesn't work in a serverless environment like Vercel. Eventually, I discovered that I needed to convert the credentials to a base64-encoded string and decode it in the app. This seemed needlessly complex.
Despite the hurdles, I got it to work and managed to simplify my code significantly. At one point, merging multiple WAV files was causing memory exhaustion, making it unscalable.
Google permits 1 Million Bytes, roughly equivalent to 1 Million characters. That translates to 150k-200k words or approximately 20 hours of audio. Recently, I discovered Amazon Polly, which is on par with, if not superior to, Google in terms of pricing and performance. For now, I'm content with Google.
Takeaways: If a 3rd Party API is merely a wrapper for another API, it's better to use the original source. It's cheaper and grants more control, unless the 3rd party adds value through additional functionality. Most services primarily build interfaces around their API, which is typically secondary.
Defining My Priorities
In retrospect, focusing solely on building the past month was acceptable as I needed to hone my skills. However, from a project management perspective, I wasted time without doing any validation. Since I'm a one-man team, I need to be ruthless with time management. At the moment, I'm the bottleneck for everything. To enhance my planning, I've started using Linear.
Takeaways: Having someone to keep you accountable and focused on the most important tasks would be invaluable. I need a virtual PM!
Launching My Project by the End of July
My first side project is nearly complete. Users will receive summaries of bookmarked articles as podcast episodes, simply by using the browser extension. The summaries are impressively good - a 10-20 minute read is condensed into a 2-minute audio summary.
Exploring the AI Tech Stack
A friend recently approached me with an idea that required an AI tech stack. This piqued my interest in the makeup and associated costs of such a stack. I'm still in the early stages, but here are some points to note:
Out-of-the-box LLMs are general-purpose tools: They're good enough for most products, retrieving general information, generating text and images, and processing information. However, if they require specialized knowledge, they fall short. The main challenge is figuring out how to supplement an LLM with your own information. LLMs don't have memory - ChatGPT simulates memory by sending previous requests each time, making it more costly and increasing latency. There are two main ways to create specialized LLMs and add a form of memory:
Converting Documents to Embeddings: This method is simpler, straightforward, and cost-effective. You convert a set of documents (information) into embeddings, store them in a vector database, and add the most relevant information as part of the prompt. You don't alter the LLM but provide it with relevant context.
The most common choice for creating embeddings is OpenAI’s embeddings API endpoint. Send your text to their API, and it converts the text to embeddings. You can store these in a Vector Database like Supabase. It costs about $0.0001/1K tokens, which seems reasonable and not overly expensive.Fine-tuning an LLM for Your Use Case: Fine-tuning isn't necessarily superior to embeddings, but rather complementary. Training the model to adapt to the nuances of your specific use cases leads to more accurate results. This requires less context, shorter prompts, and improves result quality. Fine-tuning might be expensive, but I'm not yet sure.
Startup stack: Open AI/Azure Open AI + a Vector database such as Supabase/Neon should give you a pretty good out of the box experience for getting started. The other options seems like using the open source library Sentence Transformers for embeddings and host a model like the new Llama2 model on replicate.
I look forward to the next 3 months and will hopefully get more stuff done than in the last three months *haha*.
Open questions around the AI tech stack:
Whats cost of running embeddings?
At what does it make sense to fine-tune a model?
How much data do you need to make it actual useful?
How does OpenAI function change the game?
Whats the cost difference between hosting a model on Replicate vs. OpenAI?
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.