Short, random tutorials and posts
The ONNX Runtime package makes it easy to run TensorFlow models in PHP. This short tutorial will show you how. It’s based on this tutorial from tf2onnx. We’ll use SSD Mobilenet, which can detect multiple objects in an image. First, download the pretrained model from the official TensorFlow Models project and this awesome shot of polar bears. Photo from the U.S. Fish and Wildlife Service Install…
DVC is a version control system for machine learning datasets and models. It allows you to store large files outside of Git while keeping them versioned. In a few steps, you can have it working on Heroku. This tutorial assumes you’re already using DVC and are ready to deploy your app to Heroku. Getting Started Use the Apt buildpack from Heroku to install DVC. heroku buildpacks:add --index 1…
Git LFS allows you to version large files while storing them outside of your Git repository. Heroku doesn’t have built-in support for it, so a few additional steps are needed to make it work. Keep in mind that Heroku’s maximum slug size is 500 MB compressed. This tutorial assumes you’re already using Git LFS and are ready to deploy your app to Heroku. Getting Started We’ll use two buildpacks: one…
I’m happy to announce another round of machine learning gems for Ruby. Like in the last round , many use FFI or Rice to interface with high performance C and C++ code. Let’s dive in. Data Frames Rover is a data frame library, which is a popular data structure for data analysis and machine learning. It’s powered by Numo and uses columnar storage for fast operations on columns. Forecasting…
Curious to try machine learning in Ruby? Here’s a short cheatsheet for Python coders. Data structure basics Numo: NumPy for Ruby Daru: Pandas for Ruby Libraries Category Python Ruby Multi-dimensional arrays NumPy Numo Data frames Pandas Daru , Rover Visualization Altair Vega Predictive modeling Scikit-learn Rumale Gradient boosting XGBoost , LightGBM XGBoost , LightGBM Deep learning PyTorch ,…
In August, I set out to improve the machine learning ecosystem for Ruby and wasn’t sure where it would go. Over the next 5 months, I ended up releasing 16 libraries and learned a lot along the way. I wanted to share some of that knowledge and introduce some of the libraries you can now use in Ruby. The Theme There are many great machine libraries for Python, so a natural place to start was to see…
Photo by Bruce Hong 2023 Update: Check out Polars Ruby as well. NumPy and Pandas are two extremely popular libraries for machine learning in Python. Last post, we looked at Numo , a Ruby library similar to NumPy. As luck would have it, there’s a library similar to Pandas as well. It’s called Daru, and it’s the focus of this post. Overview Daru is a data analysis library. Its core data structure is…
Photo by Jonas Svidras NumPy is an extremely popular library for machine learning in Python. It provides an efficient way to work with large, multi-dimensional arrays. What you may not know is Ruby has a library with similar functionality. It’s called Numo, and in this post, we’ll look at what you can do with it. Basic Operations Numo’s core data structure is the multi-dimensional array, which has…
Welcome to another installment of deep learning in Ruby. Today, we’ll look at FER+ , a deep convolutional neural network for emotion recognition developed at Microsoft. The project is open source, and there’s a pretrained model in the ONNX Model Zoo that we can get running quickly in Ruby. First, download the model and this photo of a park ranger. Photo from Yellowstone National Park We’ll use…
The ONNX Model Zoo has a number of interesting pretrained deep learning models. Thanks to the ONNX Runtime, we can run them in Ruby. Today, we’ll look at artistic style transfer. Here’s the model we’ll use. First, download the pretrained model and this awesome shot of a lynx. Photo from the U.S. Fish and Wildlife Service Install the ONNX Runtime, MiniMagick, and Numo::NArray gems. MiniMagick…
The ONNX Runtime gem makes it easy to run TensorFlow models in Ruby. This short tutorial will show you how. It’s based on this tutorial from tf2onnx. We’ll use SSD Mobilenet, which can detect multiple objects in an image. First, download the pretrained model from the official TensorFlow Models project and this awesome shot of polar bears. Photo from the U.S. Fish and Wildlife Service Install…
Ruby isn’t a common choice for machine learning, but companies running Ruby can get tremendous value from it. I’m happy to announce it’s now possible to build advanced models in TensorFlow, Scikit-learn, PyTorch, and a number of other tools, and score them in Ruby with minimal friction. To do this previously, you’d need to either: Shell out Create a microservice Use a bridge like PyCall or RSRuby…
I’m happy to announce that XGBoost - and its cousin LightGBM from Microsoft - are now available for Ruby! XGBoost and LightGBM are powerful machine learning libraries that use a technique called gradient boosting. Gradient boosting performs well on a large range of datasets and is common among winning solutions in ML competitions. XGBoost and LightGBM are already available for popular ML languages…
I’m happy to announce that Lockbox now supports Mongoid. This makes it easy to add application-level encryption to your MongoDB documents. Blind Index also now supports Mongoid for cases where you need to query for exact matches. Get the latest versions of Lockbox and Blind Index today!
I’ve created a few Ruby gems over the years, and there are a number of patterns I’ve found myself repeating that I wanted to share. I didn’t invent them, but have long forgotten where I first saw them. They are: Rails Migrations Rails Dependencies Testing Against Multiple Dependency Versions Testing Against Rails Coding Your Gemspec Let’s dig into each of them. In the examples, the gem is called…
Note: Searchkick Pro is no longer available. Searchkick makes it easy to add intelligent search to Rails applications. It was launched in 2013 back when Elasticsearch 0.90 was all the rage. To date, there have been almost 3 million downloads. Today, I’m happy to announce the launch of Searchkick Pro, an extension for Searchkick with a number of great features. The most notable are: Reliable…
A new version of Lockbox was just released with support for types, making it easier to encrypt non-string fields. class User < ApplicationRecord encrypts :born_on, type: :date encrypts :salary, type: :integer end Previously, you’d need to perform typecasting yourself, making it harder to work with encrypted fields. All of these types are supported: date datetime boolean integer float binary json…
Some Ruby features like scrypt and hkdf require OpenSSL 1.1. Here’s how to make it work on Mac: Install rbenv and OpenSSL 1.1 brew install rbenv ruby-build openssl@1.1 Install Ruby RUBY_CONFIGURE_OPTS="--with-openssl-dir=/usr/local/opt/openssl@1.1" \ rbenv install 2.6.3 Open an interactive shell to confirm it worked rbenv shell 2.6.3 irb And run require "openssl" OpenSSL::OPENSSL_VERSION…
This is an update to Securing User Emails in Rails with a number of improvements: Works with Devise’s email changed notifications Works with Devise’s reconfirmable option Stores encrypted data in a single field You only need to manage a single key Email addresses are a common form of personal data, and they’re often stored unencrypted. If an attacker gains access to the database or backups, emails…
Blind indexing is an approach to securely search encrypted data with minimal information leakage. I’m happy to announce that Blind Index 1.0 was just released! Here are the key improvements. Stronger Algorithm This release adds support for Argon2id and makes it the default algorithm. Argon2 is a memory-hard function. You specify the amount of memory required to compute a hash, and if an attacker…
Encrypting sensitive data at the application-level is crucial for data security. Since writing Securing Sensitive Data in Rails , I haven’t been able to shake the feeling that encryption in Rails could be easier and cleaner. To address this, I created a library called Lockbox . Here are some of the principles behind it. Easy to Use, Hard to Misuse Many cryptography mistakes happen during…
Suppose a worst-case scenario happens: an attacker finds a remote code execution vulnerability and creates a reverse shell on one of your web servers. They then find the database credentials, connect to your database, and steal the data. For unencrypted data and data encrypted at the storage level, it’s game over. The attacker has it all. If data is encrypted at the application level with…
bcrypt has been a great choice for safely storing passwords. However, as time has passed, a better alternative has emerged: Argon2 . OWASP now recommends Argon2 for new applications. With a little bit of code, you can use Argon2 with Devise. Devise supports custom encryptors . However, it requires a separate column to store a salt, which isn’t needed as Argon2 stores the salt in the password hash…
Hybrid cryptography allows certain servers to encrypt data without the ability to decrypt it. This can greatly limit damage in the event of a breach. Suppose we have a service that sends text messages to customers. Customers enter their phone number through the website or mobile app. With hybrid cryptography, we can set up web servers to only encrypt phone numbers. Text messages can be sent…
It's important to understand where personal data is stored in your applications. Personal data that’s not encrypted at the application level is especially vulnerable in the event of a breach. pdscan is a command line tool to help you identify this data. It uses data sampling and column naming to find data and produces minimal database load. It scans for: Last names Email addresses IP addresses…
It feels like data breaches are showing up every week in the news. If you haven’t taken a second look at how you’re storing sensitive data, now is probably a good time. Users trust you with the privacy and security of their information. This guide will walk through what data is sensitive, best practices for storing it, and pitfalls to avoid. What’s Sensitive? The National Institute of Standards…
When you connect to a database, Postgres uses the sslmode parameter to determine the security of the connection. There are many options, so here’s an analogy to web security: disable is HTTP verify-full is HTTPS All the other options fall somewhere in between, and by design, make less guarantees of security than HTTPS in your browser does. This includes the default prefer . The Postgres docs have…
Use client-side encryption to encrypt your data before sending it to S3. You can provide an encryption key to use directly or a KMS key for envelope encryption. With envelope encryption, a data encryption key is retrieved from KMS and used to encrypt the file. An encrypted version of the key is stored in the object metadata. When downloading the file, the encrypted key is sent to KMS to be…
Many companies start out with a single web application. As the team and codebase grow, things feel less organized and common tasks like booting the app and running the test suite take longer and longer. It can be tempting to turn to microservices to alleviate some of this pain. However, distributed systems add a significant amount of complexity and mental overhead. Before you decide to split apart…
Organizations today have more data than ever. Predictive modeling is a powerful way to use this data to solve problems and create better experiences for customers. For instance, do a better job keeping items in stock by predicting demand or lower costs by predicting fraud. If you use Ruby on Rails, it can be tough to know how to incorporate this into your app. We’ll go over four patterns you can…
Many companies today run infrastructure where machines or containers can be replaced at any time, so you can’t depend on them for permanent storage. One place this is especially painful is the Rails console. Console history can save a lot of typing. This is where Archer comes in. Add it your project, and it’ll begin to use the database to store history. Archer supports multiple users so everyone…
Encryption is a common way to protect sensitive data. Generating a secure key is an important part of the process. attr_encrypted , the popular encryption library for Rails, uses AES-256-GCM by default, which takes a 256-bit key. So how can we generate a secure one? If you’re in a hurry, feel free to skip to the answer . Take 1 One way to generate a key is: SecureRandom.base64(32).first(32) This…
Slack signs its requests so you can verify they’re authentic. Here’s a method you can use in your Rails controllers for it. def request_verified? timestamp = request.headers["X-Slack-Request-Timestamp"] signature = request.headers["X-Slack-Signature"] signing_secret = ENV.fetch("SLACK_SIGNING_SECRET") if Time.at(timestamp.to_i) < 5.minutes.ago return false # expired end basestring =…
Here’s how to use Vault for public key infrastructure. Update: Vault now has a great article on this Install the latest version of Vault and jq sudo apt-get install unzip jq wget https://releases.hashicorp.com/vault/0.9.0/vault_0.9.0_linux_amd64.zip unzip vault_0.9.0_linux_amd64.zip sudo mv vault /usr/local/bin Start Vault (we use development mode for this tutorial) vault server -dev Create a PKI…
QR decomposition is a stable way to solve linear regression . require "matrix" x = Matrix.columns([[1, 1, 1, 1, 1], [1, 2, 3, 4, 5], [4, 2, 5, 6, 1]]) y = Matrix.column_vector([145, 225, 355, 465, 515]) You can use the extendmatrix gem to do decomposition in pure Ruby. Givens rotations are faster , but the implementation appears to have a bug. require "extendmatrix" r = x.houseR q = x.houseQ Next,…
Jupyter notebooks are a great alternative to the Rails console for doing exploratory data analysis and building predictive models. Here’s how to get setup: First, install Jupyter . With Homebrew, use: brew install jupyterlab Add to your Gemfile group :development do gem 'iruby', require: false gem 'ffi-rzmq', require: false end Run bundle install bundle exec iruby register --force Start Jupyter…
The upsert gem is great for individual upserts, but for performant bulk upserts, use the activerecord-import gem. Add a unique index on the columns to upsert on (if it’s not your primary key) class AddUpsertIndexOnForecasts < ActiveRecord::Migration[5.2] def change add_index :forecasts, [:date], unique: true end end Prep your records records = [ {date: "2018-01-01", value: 10}, {date:…
There is an updated version of this post. The GDPR goes into effect next Friday. Whether or not you serve European residents, it’s a great reminder that we have the responsibility to build systems in a way that protects user privacy. Email addresses are a common form of personal data, and they’re often stored unencrypted. If an attacker gains access to the database or backups, emails will be…
With the GDPR just around the corner, here are two useful ways to protect your users’ IP addresses. Both support IPv4 and IPv6, and are included in the ip_anonymizer gem. Masking This is the approach Google Analytics uses for IP anonymization : For IPv4, the last octet is set to 0 For IPv6, the last 80 bits are set to zeros require "ipaddr" def mask_ip(ip) addr = IPAddr.new(ip) if addr.ipv4? # set…
TPC-H is a database benchmark. git clone https://github.com/gregrahn/tpch-kit.git cd tpch-kit/dbgen make -f Makefile.osx Create the database and load the schema createdb tpch psql tpch -f dss.ddl Generate data ./dbgen -vf -s 1 Load the data for i in `ls *.tbl`; do table=${i/.tbl/} echo "Loading $table..." sed 's/|$//' $i > /tmp/$i psql tpch -q -c "TRUNCATE $table" psql tpch -c "\\copy $table FROM…
TPC-DS is a database benchmark. git clone https://github.com/gregrahn/tpcds-kit.git cd tpcds-kit/tools make OS=MACOS Create the database and load the schema createdb tpcds psql tpcds -f tpcds.sql Generate data ./dsdgen -FORCE -VERBOSE Load the data for i in `ls *.dat`; do table=${i/.dat/} echo "Loading $table..." sed 's/|$//' $i > /tmp/$i psql tpcds -q -c "TRUNCATE $table" psql tpcds -c "\\copy…
Rollup is a great tool for building libraries. “Webpack for apps, and Rollup for libraries” Run: yarn add rollup rollup-plugin-buble rollup-plugin-commonjs rollup-plugin-node-resolve rollup-plugin-uglify --dev Add to package.json : { "main": "dist/my-project.js", "module": "dist/my-project.esm.js", "scripts": { "build": "rollup -c" } } Add dist/ to your .gitignore . Create rollup.config.js with:…
Securing database traffic inside your network can be a great step for defense in depth. It’s also a necessity for Zero Trust Networks . Both Amazon RDS and PgBouncer have built-in support for TLS, but it’s a little bit of work to get it set up. This tutorial will show you how. Direct Connections The first step is to make sure all direct connections are secure. Luckily, Amazon RDS has a parameter…
Simple rules to follow when creating metrics Over time: You must see how metrics change over time. Ideally you can view them by day, week, and month. No pie charts! In groups: Metrics require balance. Think of the extremes. If you over-optimize for one metric, what problem will it create? Weighted appropriately: If different times of the week or geographic areas are more important, your metrics…
Install Vault , as well as JQ for JSON parsing brew install vault jq Start the dev server vault server -dev Then open another window. For this demo, we’ll create a new Postgres database. createdb myapp Create a Postgres user for Vault to manage other users psql -c "CREATE USER vault WITH CREATEROLE ENCRYPTED PASSWORD 'secret';" myapp And create a role to grant to temporary users. This is where you…
AWS makes it easy to enable server-side encryption on many of its services, but it also provides ways to do client-side encryption well. Here are a few ways in Ruby. S3 Gem: aws-sdk-s3 client = Aws::S3::Encryption::Client.new( kms_key_id: "alias/my-key" ) client.put_object( body: File.read("test.txt"), bucket: "my-bucket", key: "test.txt" ) resp = client.get_object( bucket: "my-bucket", key:…
It’s been over 2 years since PgHero 1.0 was released as a performance dashboard for Postgres. Since then, a number of new features have been added. checks for serious issues like transaction ID wraparound and integer overflow the ability to capture and view query stats over time suggested indexes to give you a better idea of how to optimize queries (check out Dexter for automatic indexing) PgHero…
Your database knows which queries are running. It also has a pretty good idea of which indexes are best for a given query. And since indexes don’t change the results of a query, they’re really just a performance optimization. So why do we always need a human to choose them? Introducing Dexter . Dexter indexes your database for you. You can still do it yourself, but Dexter will do a pretty good…
Setting up database users for an app can be challenging if you don’t do it often. Good permissions add a layer of security and can minimize the chances of developer mistakes. The three types of users we’ll cover are: Type Description Read Write Modify migrations Schema changes ✓ ✓ ✓ app Reading and writing data ✓ ✓ analytics Data analysis and reporting ✓ Before we jump into it, there’s something…
How I personally start new apps Create Project Get the latest version of Rails gem install rails Create a new app rails new <name> -d postgresql --skip-turbolinks Don’t fret too much over the name - you can easily update it later Version Control Add Git git add . git commit -m "Hello app" App Config Make a few updates to config/application.rb Disable unwanted generators config.generators do |g|…