Data Science and Statistical Consulting
This post just serves as a basic notes write up of getting started with YOLO for object detection. YOLO (You Only Look Once) 1 is a real-time computer vision object detection system that uses a single neural network to predict bounding boxes and class probabilities directly from full images in one evaluation. There are many versions of YOLO from v1 up to (at time of writing) v11. I have chosen to…
A palace in the middle of a fictional city is protected by a large wall. During the night, in protest, a citizen randomly opens fire at the palace walls with a machine gun from a fixed but hidden vantage point, riddling the long, straight palace walls with bullet holes. You are called in the morning to the Emperors office to analyse the bullet holes in the wall and determine where the shooter was…
Can we trust the outputs that we are receiving from GenAI products? Well trust is built in part, on transparency. The reason more traditional data science and analytics work found its way into mainstream decision making is because it was transparent and trusted. Most serious data scientists I speak to spend a lot of time worrying about reproducibility . From a scientific and computational…
Have you ever looked for a hotel, Airbnb, restaurant and checked out the reviews? Always go for the highest rating right? Sure..although that 5 star apartment only has 3 reviews…Ooo this one has more, oh but its only 4.72 stars…hmmm. Shit. What do I do? We know higher reviews are better than lower reviews. But we also know more reviews are more reliable than fewer reviews. Questions… Can we…
tl;dr The use of classification algorithms to model binary response data are so ubiquitous there is a risk that in some settings this type of model is inappropriate. In cases when the response data occur over time and subjects may drop out (due to death, loss to follow up etc), traditional methods like logistic regression will not result in accurate estimations. Time to event models like survival…
Do you ever wake up at night, slightly disoriented, breathless, wondering how artificial neural nets are able to optimise weights without encountering discontinutities? Do you ever stare at your boss’s mouth while it’s moving but you can’t hear a word they are saying because you realise the words we use are just an artibrary token system that doesn’t make much sense until we use our little monkey…
I had an interesting discussion with a client recently about the value proposition for collecting large amounts of time series data. I thought it might be worthwhile recording my thoughts here in case of interest to others. I want to highlight 3 key applications where time series analysis can add value. I want to answer three common questions I get on the topic. What is time series data? Time…
There seems to be a subconscious deterrent from using specialised tools like R as part of a production or core tech stack, as if it’s somehow not compatible with other forms of technology. Which is clealy BS. Over the past year a suprising number of my projects have involved ways to package and integrate complex R code into an existing application. There are many ways to do this, but few are as…
No plan survives first contact with IT In most organisations the single biggest challenge you will face as a data scientist is dealing with IT (Information Technology / Technical Support) to support and enable your data science workflows in Production. Why? It’s not really anyone’s fault, it’s just a lack of alignment. IT teams don’t care heaps about enabling 1% of users to run wild with Open…
The first ever OceaniaR meeting took place this week in Melbourne! The event took place the day before the Monash WOMBAT 2024 conference and was designed as a hackathon/unconference style event with a focus on promoting projects and participation from R users in the Oceania region including Australia, New Zealand and the Pacific Islands. It was such as nice day with R developers, researchers,…
Earlier this year I was fortunate enough to work with EcoHealth Alliance on a major project focused on two WHO priority zoonoses, Rift Valley fever virus and Crimean-Congo hemorrhagic fever virus. Over a 10 year period, research teams sampled over 800 people and livestock, sampled over 300 rodents, and conducted ecological site characterizations at 150 sites in each Tanzania and South Africa. To…
In previous posts I have discussed frameworks for thinking about data science projects. When it comes to actually writing R code, there are a number of coding workflows you can adopt to get the work done. For example: Monolithic Notebook (RMarkdown, Quarto) Directory structure with run.R control scripts (e.g. ProjectTemplate) Opinionated pipeline tools ( {targets} , {drake} etc) R Package So which…
In a previous post I discussed the charming nostalgia of CRISP-DM as a data analysis workflow choice. A modern limitation of this model is how it handles the Deployment step. In the olden days this was usually some form of static report to a client. It’s a loaded topic now with deployment usually having something to do with giving money to AWS/MSFT or a battle to the death with your IT department.…
The cross-industry standard process for data mining (CRISP-DM) is a process model and framework for carrying out data mining projects. ‘Data mining’ is quite a funny term and will instantly carbon-date you if you use it. It very much of a late 90’s era. If you were around then your WFH setup probably looked like this. So good. Still room for a CD burner Nostalgia aside, it remains a popular choice…
A recent paper published in ‘Ethics and Information Technology’ titled ChatGPT is bullshit 1 is fast becoming one of my favorite papers. Not least because it uses the word bullshit 165 times. The hype surrounding the use of Large Language Models (LLM’s) such as ChatGPT, Bard, Llama, Claude etc has been inescapable lately. I have become increasingly concerned with the general willingness to accept…
Earlier in my career, in the afternath of the GFC I worked as trader for an online share trading platform. Everyday I would get blasted with a firehose of emotion from investors. People day-trading on their accounts, needing help placing exotic trades or dealing during volatile markets. It’s easy to get swept up in the excitement. Sweating on every tick of the market. Listening to the live market…
A defiant Aussie family has refused to sell their farm-land property despite the entire neighborhood being converted into a new housing estate. Is this real? Where is it? Could I geolocate it using just OSINT 1 techniques?… Yeah of course. I have a loose theory that no matter who or where you are, there is probably sufficient data for a sufficiently motivated person to find you. The Challenge I…
Are you about to scale up your data analytics team to do more AI/ML work? Here are 5 things you need to know up front to make your life easier. These are tips focused on enterprise-level organisations wanting to implement Microsoft’s Azure Machine Learning studio . But I’m sure the content translates well to other settings. 1) People - Process - Platform (pick any three) Effective analytics and…
If you have hired a data scientist or run a team of data people, you may not be an expert yourself (that’s why you hired an expert, right?). So how do you know you are receiving a quality predictive model and not some BS that was thrown carelessly at a black-box machine learning model. The consequences of poorly built statistical models are not trivial. In 2015, Amazon realized its ‘AI recruiting…
The most undervalued skill in delivering value with data science teams is picking projects that are likely to succeed. There is no shortcut - it takes years of hard earned experience. A number that seems to be floating around is 80% of data science projects will FAIL. Ouch. Many of these types of numbers are ‘predictions’ from consultancies who stand to benefit from making big claims. “Through…