Software Research, Development, Testing, and Education
My standard from-scratch implementation of kernel ridge regression (KRR) trains uses the explicit Cholesky inverse of a RBF kernel matrix. It s possible to train by computing a Cholesky decomposition and then using a solve method. The explicit inverse approach looks Continue reading The post Kernel Ridge Regression via Cholesky Decomposition with a Solve Function Using From Scratch C# appeared…
I wrote an article titled Computing Dataset Variance Inflation Factor (VIF) Using C# in the August 2026 edition of Microsoft Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2026/08/03/computing-dataset-variance-inflation-factor-vif-using-csharp.aspx. The variance inflation factor (VIF) of a dataset column is a value that measures the Continue reading The post Computing…
Zoltar is my NFL football prediction computer program. It uses a neural network and a type of reinforcement learning. The first game of the 2026 NFL season will be played on Wednesday, September 9, with the New England Patriots at Continue reading The post NFL 2026 Football Early Predictions – Zoltar Likes the Broncos and the Seahawks appeared first on James D. McCaffrey .
A simple decision tree for regression will almost always overfit the training data where the model accuracy on the training data is near 100% but accuracy on new, previously unseen data is very poor. The simple idea of bagging tree Continue reading The post I Use AI to Improve My Bagging Tree Regression System Using C# appeared first on James D. McCaffrey .
I recently refactored my basic kernel support vector regression system, implemented from scratch, using Python. I tested my refactored code using synthetic data, but I figured I should run the system on some real world data. One of the standard Continue reading The post The Boston Area House Price Problem With From-Scratch Support Vector Regression Using Python appeared first on James D. McCaffrey…
I recently (yesterday) made major revisions to my personal C# matrix QR decomposition using the Householder algorithm. In my work environment, I use matrix QR decomposition for matrix pseudo-inverse of a matrix of training data. So, my next step after Continue reading The post New Version of Matrix Pseudo-Inverse With QR Decomposition (Householder Algorithm) Using C# appeared first on James D.…
One morning before work, I realized that I hadn t written any JavaScript code for several weeks. For mental exercise, I decided to implement a regression system (to predict a single numeric value), using a neural network with exactly two hidden Continue reading The post Deep Neural Network Regression From Scratch Using JavaScript appeared first on James D. McCaffrey .
The goal of a machine learning regression problem is to predict a single numeric value. For example, you might want to predict the price of a house in a particular area based on square footage, year built, number of bedrooms, Continue reading The post Yet Another Metric for Regression Model Evaluation: Mean Absolute Scaled Error (MASE) Implemented Using C# appeared first on James D. McCaffrey .
There are three ways to train a basic linear regression model: 1.) using stochastic gradient descent, 2.) using left pseudo-inverse (normal equations) via Cholesky inverse, 3.) using relaxed Moore-Penrose pseudo-inverse via one of many possible inverses. Each of the three Continue reading The post New Version of Linear Regression Trained Using MP Pseudo-Inverse via QR-Householder Using C# appeared…
In machine learning, if training data is multicollinear, the interpretability of the resulting model will likely be poor (the details are tricky and out of the scope of this post). Multicollinear data has two or more rows that are highly Continue reading The post Checking Machine Learning Training Data for Multicollinearity Using VIF (Variance Inflation Factor) With From-Scratch JavaScript…
In a nutshell, engineers and researchers: a.) Have a tendency to continue investing time, money, and effort into a failing venture simply because they have already invested resources in it. b.) Have a tendency to believe that a complex solution Continue reading The post Machine Learning and Psychology: Sunk Cost Fallacy, Complexity Fallacy, and Obnoxious Researchers appeared first on James D.…
In the back of my mind, I wasn t entirely happy with my current version of matrix QR decomposition using the Householder algorithm. The current version worked fine, but the underlying code just didn t feel quite right in some way that Continue reading The post New Version of Matrix QR Decomposition With the Householder Algorithm Using C# appeared first on James D. McCaffrey .
The goal of a machine learning regression problem is to predict a single numeric value. For example, a bank might want to predict the maximum safe loan amount for a customer, based on age, account balance, annual income, and so Continue reading The post Support Vector Regression With SGD Training Using JavaScript appeared first on James D. McCaffrey .
I fed my decision tree regression system, implemented using C#, to several AI systems and asked the AI to analyze it for correctness. I was quite impressed that the AI found a few rare edge cases where my code could Continue reading The post I Use AI to Improve My Decision Tree Regression Using C# appeared first on James D. McCaffrey .
I ran into an interesting problem recently. I had a set of training data and I needed to condense it to a smaller size (number of items). This is a rare scenario because in most cases you want as much Continue reading The post Pruning Machine Learning Training Data Using an RBF Kernel with C# appeared first on James D. McCaffrey .
One Sunday morning, I figured I d put together a demo four different different regularization techniques for linear regression: L1, L2, weight decay, and input noise. I have implemented all these techniques before, but never in the same program. I decided Continue reading The post Linear Regression with L1, L2, Weight Decay, and Input Noise Regularization from Scratch Using SGD Training with…
One morning before work, I figured I d implement kernel support vector regression (SVR), from scratch, using Python. It took me a bit longer than expected but I got a demo up and running. My implementation uses hard-wired RBF (radial basis Continue reading The post Support Vector Regression with SMO Training from Scratch Using Python appeared first on James D. McCaffrey .
I ran into an interesting problem recently. I had a set of training data and I needed to condense it to a smaller size (number of items). This is a rare scenario: in most cases you want as much training Continue reading The post Pruning Machine Learning Training Data Using an RBF Kernel with Python appeared first on James D. McCaffrey .
An Extra Trees Regressor ( Extremely Randomized Trees ) is an ensemble machine learning algorithm that builds multiple randomized decision trees to predict continuous numerical values. It differs from Random Forest regressions by introducing greater randomization during node splitting and utilizing the Continue reading The post Extra Trees Regression From Scratch Using Python appeared first on…
In machine learning, if training data is multicollinear, the interpretability of the resulting model will likely be poor (the details are tricky and out of the scope of this post). Multicollinear data has two or more rows that are highly Continue reading The post Checking Machine Learning Training Data for Multicollinearity Using VIF (Variance Inflation Factor) from Scratch C# appeared first on…