The feed of updates to Manish R Jain
Docovia is the land for your docs, a place your docs can call home. It hosts, processes, archives, extracts text, enriches the doc via LLM , and makes it all searchable via Typesense. You can see a live read-only demo of Docovia here . The source code is available on GitHub . I’ve been self-hosting paperless-ngx for a while now, and I owe it my real gratitude. It got me away from the clutches of…
Should I buy or rent? Should I sell now or keep it? These questions keep people up at night, and for good reasons. We’re often talking about decisions involving hundreds of thousands of dollars and decades of financial impact. The problem? Most advice is either oversimplified (“renting is throwing money away”) or buried in spreadsheets that require an MBA to understand. So people guess. They go…
To allow Postgres connections from a remote server, or from within a docker container, you’d want to allow it to listen to external traffic. This requires doing two settings, one in # /etc/postgresql/15/main/postgresql.conf listen_addresses = '*' # This would listen to all addresses. # Could also be set to 0.0.0.0 for the same effect. and another in # /etc/postgresql/15/main/pg_hba.conf # TYPE…
Today, after visiting VFS office to apply for my child’s Overseas Citizen of India document, I was turned back because I didn’t provide both the parents’ documents, just mine. Now, when uploading the docs to Indian government site, they only asked for one parent. But VFS , the “ private” entity in charge of collecting documentation, was adamant that it must be both parents. Meanwhile, their…
A few months ago, I was talking about an exciting new product with a friend. As I described the efficiencies this product could offer, my friend remarked: “ Sure, it saves some time. But is it addressing a ‘ hair on fire’ problem?” The best explanation I could find for a “ hair on fire” problem is by Michael Seibel : If your friend was standing next to you and their hair was on fire, that fire…
As companies become bigger, the “ shared non-responsibility” expands vastly — where no one in particular is fully responsible, because someone else is responsible for a portion. CEO isn’t responsible, because they hired another C-level or VP . VP isn’t responsible, because they rely on a lead. Lead isn’t responsible, because they need something from a cross-functional lead. The cross-functional…
If you’re using Cloudflare to manage traffic to your origin server, you might want to restrict incoming traffic to only come from Cloudflare. This is crucial for protecting your origin server from potential unauthorized access. A handy tool that can aid in achieving this on an Ubuntu server is the UncomplicatedFirewall . For those unfamiliar, UFW is a user-friendly front-end for managing iptables…
We’re running Postgres for the production deployment of Struct, a ( not-yet-launched ) chat platform we’re building to deal with the disorganized chaos of live communication. I’m using PostgreSQL to store the data — which I have started to enjoy because of its decades long improvements around useability, documentation and in particular, the command-line tool psql (though, some things around schema…
I recently witnessed a goroutine leak in my Go server. It looked like this. goroutine profile: total 41053 20524 @ 0x43a656 0x44a7fe 0x65c512 0x46bec1 # 0x65c511 net/http.(*persistConn).writeLoop+0xf1 /usr/local/go/src/net/http/transport.go:2410 20523 @ 0x43a656 0x44a7fe 0x65b425 0x46bec1 # 0x65b424 net/http.(*persistConn).readLoop+0xd84 /usr/local/go/src/net/http/transport.go:2227 Searching…
I’m using Postgres for my new startup. The startup is still in stealth mode, so I can’t talk about it much. But, given the focus of the startup is on communication platforms, Postgres made a lot of sense. This is the first time I’ve ever used Postgres, and honestly, I was quite impressed by just the sheer amount of online documentation and help available to get started. My first impressions of…
Original: Sep 14, 2022 In a previous post , I showed how to put together a cheap load balancer server for $500. I later had an idea to use that server for storage too. Storage servers don’t need much CPU , just a lot of disk space. And this one had very little disk, and a pretty decent CPU . The combination of both the workloads — a load balancer and a NAS is a good one. Typically, you’d pick up a…
Spent the last hour wrecking my head against an npm error, trying search after search with Google, leading nowhere. Finally copied over the issue to ChatGPT, and it accurately identified the issue, giving me a copy-paste environment variable to set. And it worked, instantly! 🚀
Originally Posted: Aug 24, 2022 Dealing with huge web3 data, I see myself doing this a lot: moving data directories across machines. In this case, we have a 250 GB directory with 10K files. The network connection between machines is ~5 MBps. But using these techniques, we can achieve a 6x faster transfer (~30 MBps). There’re two ways to do this: Use compressed tar (Newer and Older) Use rsync…
Originally written: Sep 7, 2022 In /etc/security/limits.conf , add * soft nofile 1000001 * hard nofile 1000001 Log out, log back in again, and you should have: $ ulimit -Sn 1000001 $ ulimit -Hn 1000001 Original Source Update (Dec 6, 2022) If you want to automate this via Salt , this is what you can add to your config. open_file_limits: file.blockreplace: - name: /etc/security/limits.conf -…
In a previous post , I wrote about the best way to transfer a directory across the network. In this post, I compare Brotli v Gzip v Zstd v LZ4 on blockchain dataset to determine which compression can give me the fastest transmission rates to move data around. I took a Badger DB directory that I needed to transmit over the wire, and ran various compression algorithms on it to see how fast can they…
In my setup, where I have an HTTPS load balancer, sending traffic over HTTP to the server, aka reverse proxy, Caddy outperforms NGINX by 4x. That’s 400% better performance via Caddy over an HTTPS connection than NGINX . I’m using RPC Bench for this test. Disclaimer: YMMV . If Nginx is working out great for you, continue using it. I have switched from Nginx to Caddy after I experienced this…
As you might have picked up, I run baremetal servers. They provide the best performance-to-cost ratio. A problem I’ve been trying to solve for some time now has been to securely expose a performant service, which is load-balanced across a bunch of servers. I first tried using Cloudflare Tunnels, a tunnel per server. These tunnels then get load balanced using Cloudflare Loadbalancer. Turns out,…
Ethereum logs always stump newcomers. A typical Ethereum log looks like this: $ curl https://stream.0xfast.com/beta/eth-mainnet/15000000 -s | jq '.transactions[].logs' ... { "address" : "0x9e0905249ceefffb9605e034b534544684a58be6" , "topics" : [ "0xc42079f94a6350d7e6235f29174924f928cc2ac818eb64fed8004e115fbcca67" , "0x00000000000000000000000068b3465833fb72a70ecdf485e0e4c7bd8665fc45" ,…
ETH : https://eth-uswest.0xfast.com/stream/free?range=1337 Status Page: https://status.0xfast.com Motivation As I’ve ventured into the web3 space, I’ve been astounded by how unintuitive getting your hands on web3 data is. The first thing I wanted to do was to download the blockchain data and play with it. Seems simple enough, right? Not quite. While there are several web3 API services available,…
Setting up Nginx with Let’s Encrypt is surprisingly easy, with certbot. I won’t go into details here. This tutorial does a good job: https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/
This blog post was on the front page of Hacker News for a day — Link It’s been almost nine months since I left Dgraph Labs, a company that I spent 6 years of my life building — after a layoff followed by a failed Series B round (unsurprisingly, and in that order). During this time, I’ve had a chance to reflect upon what happened, what went wrong, what went right, where did we succeed, and where…
I put together a build for a 2.5 Gbps server, which can run Nginx and act as a load balancer for my other servers. Here’s all the parts in pcpartpicker: https://pcpartpicker.com/user/mrjn/saved/dQPWcf Fractal Design Core 1000: $50 at Central Computers This has become the case of my choice. It’s cheap and just really a delight to work with. I have put together 2 of these already, and just enjoyed…
I’m working on serving Polygon data via Outserv. As part of the process, I downloaded the first 15 million Polygon blocks and stored them as JSON . To avoid any duplicates while loading this data into Outserv, each object in JSON has a unique ID . That’s true for Blocks and Transactions that have hash and Accounts that have address . But, is not true for Logs, which don’t have any globally unique…
I was recently looking for a working solution to parsing Smart Contracts in the EVM chains. I couldn’t find a simple to follow, working solution based on Go. This post aims to fill that gap. In this post, we’d parse Polygon bridge transactions. Let’s look at this particular transaction . This txn belonging to block 31740481 in Polygon L2 chain is interacting with a smart contract here . Get a…
Originally Published: July 13, 2022 Update (Sep 14, 2022): I’ve switched to using ZFS because it has checksums for every block — I can’t imagine trusting a storage without checksums. This blog post covers how to setup ZFS and switch away from mdadm. # We want to create a RAID 0 on /dev/sda and /dev/sdb drives. # Create Partitions on drives first. $ sudo fdisk /dev/sda Welcome to fdisk (util-linux…
sudo apt update sudo apt install usb-creator-gtk sudo -E usb-creator-gtk # -E would maintain env variable This would search for ISO files in Downloads, and show you the USB stick that’s plugged in. You can then follow the interface to create a bootable disk.
Break the cycle. Run Outserv
I posted in the OpenSea discord channel that I need help with getting OpenSea API key so I can write an importer for Outserv . And I got pinged by 0xEider. Yes, 0xEider is part of OpenSea community team. Just that, this one was… not the same 0xEider. Watch and enjoy! 1080p and 2x playback speed recommended.
Decentralization — the dispersion or distribution of functions and powers. — Merriam Webster Today, I’m excited to announce the launch of Outserv — a GraphQL-powered Blockchain Search Engine. Outserv enables you to run production-grade GraphQL Search APIs over any blockchain data 10x faster than existing mechanisms. Outserv is a unique mix of GraphQL, Search, and Cache systems — all put together…
Sustainable — of, relating to, or being a method of harvesting or using a resource so that the resource is not depleted or permanently damaged. - Merriam-Webster Dictionary Open source is built on the principles of transparency, participation and collaboration. And open source software is thriving. It’s everywhere. Every decently complex software is using some open source software underneath. But…