Justin Pinkney's home on the web
A dataset of 187 thousand high resolution face images from movie trailers! Download it from huggingface . # Before the advent of giant web scale image datasets FFHQ used to be considered a big dataset full of images of faces on which many, many GANs were trained. One of the many issues with FFHQ (side note: don’t get me wrong there are lots of others, but I was focused on this one at the time) is…
Back in the GAN days most of my time was spent playing with famous models like StyleGAN and pix2pix . But another of my favourite models, which was much less well known, was Swapping Autoencoder , a lovely model from Adobe research for which all the code and checkpoints were released [1] 1. It was also the method (if not the exact model) used for the landscape mixer feature in Photoshop . This is…
Just want to play with the model? Try it on Lambda labs or Huggingface I'm doing more catching up with write ups of past models I trained, this time it's the turn of "Image Mixer" at the time I did at least share the checkpoint and a demo , but here's a more detailed write up of the process of training the model itself. Even before I worked there I was very impressed with Midjourney's blend…
I'm still trying to catch up on writing up some of the "old" diffusion experiments I did and models I made during 2022. This is another quick write up of a model designed to generate new images of a person's face given a single image of that face with no further fine tuning. There are now better models available for this purpose, in particular take a look at IPAdapter face and PhotoMaker . Mixing…
Outputs of the Pokemon fine-tuned Würstchen v2 model for prompts: "Hello Kitty", "Old man looking at the moon", "Donald Trump", adn "Yoda". Top row EMA weights bottom row normal. The second half of 2023 saw a ton of image generation models come out (including our own Midjourney v6 😊), some open source some not and most interesting in their own way. One particularly neat little model was Würstchen…
Sidenotes # I just added side notes to my blog, thanks to my move to 11ty (from the bloated mess that was Gatsby) and the wonderful sites and articles people have shared on the small web (as well as the power of view source ) it's been easy and a pleasure! I always want to add margin/side/foot notes to my posts, it's probably a sign of a bad excessively detail oriented writing style. Because…
I've long had an obsession with trying to create a purely generated train journey with machine learning tools. Cruising through the latent space, inside the internal world of a model is super appealing, and this post documents my years long efforts to do using various generative models along the way. In the end the rise of diffusion models made what I was trying to do possible, and if you want to…
This post is time-travelling a little. For some reason I never blogged about the image variations stable diffusion model I trained, so this is a bit of a recap on the how and why of the image variation model and to collect some links and experiments using it. The model itself is a little old now and there are other models which are similar might be better like Karlo v1.0 and Kandinsky (probably…
The other web # The recently accelerated enshitification of Twitter made me realise that too much of what I do and make is trapped in that platform. I feel this especially when I start trying to explain projects or experiments I've worked on to people by sharing tweets, why isn't it on my blog? Its probably not an understatement to say that Twitter has changed the course of my career, it's how…
https://youtu.be/rsxUcmHNH9s I recently signed up to Derrick Schultz 's online algorithmic film making class . I'm excited to be taking one of Derrick's classes, I've been a big fan for a long time of both his work and teaching (not just because his YouTube channel has long been my default response when people would ask me "How do I StyleGAN?"). I'm also keen to have something I can focus on as an…
Classic paintings of flowers cut and re-arranged. A long time ago I made these digitally cutout and collaged images of flowers. They were all created using public domain flower paintings, U2net for segmentation and image magick for collaging. --> -->
I've been doing a bunch of quick experiments for my improved CLIP Image conditioned version of Stable Diffusion v1. We can actually do lots of fun stuff with this model including transforming this Ghibli house into a lighthouse! First off you can find the actual model on huggingface hub . Throughout the post we'll be using this crop from a Ghibli frame as our test image. This is what the standard…
Playing with the variations model a bit more. You can take the CLIP image embedding of an image then do latent editing with a direction in CLIP text space. original variation "colour oil painting" - "bw photo" "modern dslr photo" - "old bw photo" original "old woman" -"young woman" "modern dslr photo" - "oil painting" original "detailed painting" - "child's drawing" "archviz render" - "child's…
If you want to just try out the model look here: All of this work was done as part of my role at Lambda Labs and all the real details of how the model was made, and how you can make one yourself are in this post on the Lambda Examples repo . This blog post just has a few extra details and notes on my experience Girl with a pearl earring, Cute Obama creature, Donald Trump, Boris Johnson, Totoro,…
Repository: https://github.com/justinpinkney/pixel2style2pixel/ A little while back I ported a couple of recent machine learning models to Runway: " Encoding in Style " (aka Pixel2Style2Pixel or PSP)[^psp] and " Style Space Analysis "[^style-space]. They work well together, PSP for encoding an existing image into the latent space of a Generative model, and Style Space Analysis for editing that…
Download the dataset: V2 As part of my paper Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains [^rdgi] I use a dataset of Ukiyo-e face images for training a StyleGAN model, this post contains a link to, and details of that dataset. Updates # V2 - Removed 28 bad quality images (poor alignment or not face). V1 - Initial release used in the paper Resolution…
If you'd like to keep Toonify Yourself free for everyone to play with, please consider donating to cover running costs at Ko-fi: So Doron Adler and I recently released our toonification translation model at our Toonify Yourself website. It turned out to be pretty popular with tens of thousands people visiting in the 22 hours it was running for, submitting almost a quarter of a million images for…
TLDR: If you want a Colab Notebook to toonify yourself click here: If you're interested in how the website Toonify Yourself works, see this followup post In a previous post I introduced the idea of Layer Swapping (or more generally network blending) for StyleGAN models. I briefly pointed to a fantastic model created by Doron Adler that generates almost photo-realistc people who seem to have come…
Making Ukiyo-e portraits real # In my previous post about attempting to create an ukiyo-e portrait generator I introduced a concept I called "layer swapping" in order to mix two StyleGAN models[^version]. The aim was to blend a base model and another created from that using transfer learning, the fine-tuned model . The method was different to simply interpolating the weights of the two…
Input space and Feature space are different sides of the looking glass # Neural networks (for image tasks) take in images and put them through a series of, typically convolutional, transformations. The world of those input images consisting of red, green, and blue pixel values we can call input space and the intermediate values that get computed in the network, the transformed versions of the…
This is old, since then I gave up the pain of Gatsby and updated the site to run on 11ty I've had many different versions of a home page over the years. I've left traces on lots of different social media sites (but only really use Twitter now), kept a food blog[^recipes], posted about generative art on a wordpress blog, even written my own simple static site generator (for no apparent reason).…
These cakes are a lie, they're all generated by a neural network. Random fake cakes are nice, but eventually my grand idea is to make a bespoke cake designer (I'm in this for the latent space you know!). Before I can do that I need to train a model, let's call it CakeGAN. So here is a delightful recipe for how to bake your very own non-existant cakes. How to bake a CakeGAN # Ingredients # Data -…
This is all old. Since then I've updated my blog to run on 11ty and life is much simpler. Below is a big image produced by my neural network feature visalisation library Sumie please zoom in. This page is mostly just a test of the integration of Leaflet for displaying big images on my Gatsby powered homepage. Below are some brief scratchings on how this works so I don't forget. What is MDX…
I maintain two collections of links to StyleGAN models pre-trained on a variety of datasets: Awesome Pretrained StyleGAN Awesome Pretrained StyleGAN 2 Most of these have been shared via the very active StyleGAN creative community on twitter, and if you're aware of any others then please send them my way. Either create an issue or fill out one of the following forms: Pretrained models are useful…
Everyone who's ever seen output from GANs has probably seen faces generated by StyleGAN. Now you can do the same in MATLAB! StyleGAN (and it's successor) have had a big impact on the use and application of generative models, particularly among artists. Much of this has been a combination of accessible and (fairly) straightforward to run code, great stability in training, a particularly well formed…
I've spent some time training a StyleGAN2 model on ukiyo-e faces. Here are some results from training and some experimentation with model interpolation. --> Dataset # I've spent some time collecting face images from museum images, suprisingly AWS Rekognition does a reasonable job of detecting faces and landmarks. As most of the images provided by museums are not very high resolution I've also used…
Of all the GAN architectures pix2pix is a personal favourite. It popularised the use of GANs for image to image translation, it's nice and simple, trains relatively quickly, and invariably produces some surprisingly pleasant results. If you don't know what pix2pix is, see the original project page which has some nice examples and a demo. Or just search Twitter for #pix2pix for some fun examples.…
Takes a picture and rearranges the pixels to make another. A little generative art Java application I made a long time ago which implements an algorithm to take all the pixels from an image and use them to make a new image, based on a (sort of) sorting algorithm. https://youtu.be/FCp3UCBGCmc This was all inspired by József Fejes's all RGB sorting work. I implemented his basic concept for pixel…
Midjourney # I work as a machine learning researcher at Midjourney, working on state of the art text-to-image models. Before # Lambda Labs - I worked as a Senior Machine Learning Researcher at Lambda Labs mostly with a focus on computer vision and generative models. MathWorks - I used to work as a senior consultant at MathWorks helping people to: develop computer vision and deep learning…
Get a fast and accurate face and facial feature detector for MATLAB here Intro # Everyone pretty much takes good quality face detection for granted these days, and it's essentially a solved problem in computer vision. Everyone's photo app can detect all the face you care about and computer vision competition results are increasingly focussing on performance for difficult, small, or occluded faces.…
My biophysics phd was focused on single molecule fluorescence methods for studying DNA-protein interactions. A bit more specifically I used single molecule methods such as FRET (Forster resonance energy transfer), PIFE (protein induced fluorescence enhancement), and a method I developed, TFM (tethered fluorophore motion), to study a range of DNA processing proteins (DNA polymerase, recombinases,…
Processing is a great at many things, it's so easy to do so much, including displaying things in 3D. Still it's easy to make things grind to a halt when trying to do to much. But there are a lot of things you can do to speed things up like PShape recording , using Vertex Arrays to display lots of points, or writing GLSL Shaders . Loading a .obj model in Processing is as easy as using loadShape(),…
A prototype of a stolen idea from a friend of DNA sequence displaying LED lamp. http://www.youtube.com/tCwi4XRZ3RU My first with controlling addressable LEDs with Processing and the fantastic Fadecandy board from Micah Scott . Of course it would be way more fitting to have two LED strips and arrange them in a double helix...
Having worked on a few computer vision projects at work recently I've been interested in trying to understand what the computer is seeing. A lot of image processing algorithms involve putting the image through filters, or transforms, or extracting local descriptions of portions of the image. Often these are modelled around what the human vision system is understood to be doing, but of course our…
I recently saw some great work by Lee Riggs, using Maya to create wonderful abstract landscapes out of millions of small coloured block or pillars. They're fantastic images, and I wanted to try and emulate the effect in Blender as part of my ongoing efforts to try and get to grips with the program. It turns out it was a lot easier than I expected, just using a couple of modifiers and some…
testing After seeing many amazing water drop photos all around the internet I tried out some quick attempts at droplet collision photos. Armed with a cheap speedlite, a tray of water, and a syringe it was actually fairly easy to get some decent splashes. It still takes a lot of lucky timing, and many failed attempts, so building an arduino controlled water dropper sounds like a pretty fun project…