Un sitio de @falconius
Context The coding ability of LLMs is growing continuously, and less and less code is being written by human SWEs (Software Engineers). LLMs can generate large amounts of code in a short time, and we SWEs cannot perform a super exhaustive code review; we simply do not have the necessary time. It may also happen that the SWE [ ]
Models constantly change and improve but evals persist Look at the data No amount of evals will replace the need to look at the data, once you have a evals good coverage you ll be able to decrease the time but it ll be always a must to just look at the agent traces to identify possible [ ]
Why? The biggest hurdle when someone wants to evaluate a machine learning classification task is the burdensome process of manual labeling: time consumption, boredom and human errors are some of the problems related to this task. These issues are even more critical if we want to know how is the model performing in production periodically, [ ]
I ve been eager for a long time to work on a project where LLMs negotiate with each other, to see if prompt changes can improve the negotiators. I hadn t been able to find much time to work on the project, but thanks to Cursor and Claude Sonnet, I was able to make progress. Basically, the [ ]
A New Software Development Paradigm The increasing adoption of large language models (LLMs) in software development has brought significant changes, especially for teams integrating third-party LLMs via APIs. Many teams that once built traditional, deterministic software products are now shifting toward probabilistic systems, a change that requires a new approach to ensure product quality and [ ]
Evals, or evaluations, are a set of tests used to measure prediction models performance on a specific task. Why? Any product using LLMs (Large Language Models) in production environments should have evals. And not only that, evaluations should be built even before writing the code. For those coming from a software development background, it would [ ]
I ve been thinking and doing some coding to make different AI s negotiate with each other. What is the best medium of exchange of information between AI s? In a simple bargaigning, a seller tries to sell her product at a given price. Then a potential buyer makes an offer, lower than the asking price and the [ ]
If you have or are building a software product with the intention of making it last over time, it would be wise to consider the constant evolution that AI is going to make in the coming months and years. One of the areas where AI (especially LLMs) is making the most progress is in code [ ]
Evaluations before coding. BM25 before vector search. RAG before finetuning. Deterministic before stochastic outputs.
There s been an explosion after the public launch of ChatGPT, followed by the leakage of the LLaMA weights and the open access to a OpenAI API. LLM s enabled an incredible amount of applications that couldn t be done before or at least not with the current level of success. Human language is going to become the [ ]