I've been extremely reticent to add additional data stores, because of the complication they add, in trying to sync all of them up. Redis, melisearch, elasticsearch, all of these would add a ton of complication.

Spot on. People sometimes think they can just throw new technologies at a problem to solve it. There's almost nothing that would make lemmy less maintainable than adding additional data stores / sources of truth.

It would add complexity, that is something I am not going to disagree, but I don't think they add unjustifiable complexity. Most websites running at country scale will use a Lucene, Solr, or more recently Elasticsearch for covering search. Taking load off the database is critical, because all extra concurrent load makes all transactions slower until it gets to the point it buckles. Even if it doesn't buckle, the delayed reaction times impact on users and their experience and eventually those doubts start to build up about whether this is good or reliable.

I suggested the following because from what I have seen, at scale these technologies are favoured. I don't know any large website that allows database searching without any form of caching. Care to give an example? I seriously fear you're letting your own view cloud what happens in practice and you are starting to believe that what you are doing is more correct that all alternatives that have made a different conclusion. Maybe you're right and industry standards are wrong, but that's a mighty bold assertion and a big leap of faith.

I've heard similar things a few times now and I feel like there must be some common misconception. Every single site admin so far runs lemmy on a single server, including the database. PostgreSQL scales perfectly well to fill a single server of any size. We're far from its limits. Rust also scales perfectly well to fill a server of any size. Moving some stuff to a different application doesn't improve the situation any more than improving the performance with the tools we already have. Adding more random tools / different applications will take up the same memory and the same CPU time lemmy and PG is using. It would just add more overhead, more complexity, and more multiple-sources-of-truth issues.

I'm getting some titanic "this ship can't sink" vibes here. Most of my experience is that most shiny tech that promises a lot, generally has a moment where it doesn't scale as advertised. Eventually projects are spun off late to try to deal with this. I raised the point in the hope of starting the discussion early, but it seems like the sites have to buckle and there to be no option until there is going to be a consideration of something like this.

It isn't some common misconception though, most top websites when accessing immutable content, will try to cache first to cut load. If a query is run more than 96 times a day, a 15 min cache is going to provide a reduction in the amount of work, assuming they are evenly divided. They are returning a result rather than doing the same computations again. Yes, the data can maybe be stale, but who needs the data to be that real time for search. Even an hour cache is hardly an issue from a search perspective.

In tech, it's important to use the best tool for the job, it isn't always advisable to stick with simple stacks when the demands are greater. The last few weekends, there has been bad actors really testing the limits of Lemmy, and they seem quite motivated. By allowing search straight onto the DB, you're putting the DB in the hands of bad actors which is a very risky move. So far, it's not going smoothly. They're going to keep probing and poking where their is weaknesses.

I didn't even want thumbs up of this idea, I just wanted it to be considered. I'm quite disappointed (and a little insulted) that it was fobbed off with some weird insinuation that I was naively assuming throwing technology at something would magically fix it. I don't know Rust, or Diesel, so unfortunately couldn't contribute from a code perspective. I was hoping I could share some of my experience, but I'm getting the impression that isn't welcomed. Good luck.