I have a habit of building cool things and then immediately misplacing them. Once I wrote a whole Python script for bulk-resizing 10,000 images, used it exactly once, and spent the next two years rewriting it from scratch every time I needed it again. This story is that pattern, but worse. I built a diffusion model that could clone a person's face from a single photo. And then I lost the checkpoint. Not deleted. Not corrupted. Just gone. Somewhere on one of the twelve hard drives I have accumulated like a digital hoarder. I still look for it sometimes, late at night, when I should be sleeping.
The Dream: Dreambooth But Lazy
When Stable Diffusion dropped, Dreambooth followed soon after. It was magic. You give it a handful of photos of your face, it fine-tunes the whole model, and suddenly you can generate yourself as an astronaut or a renaissance painting or whatever. People loved it. Apps sprouted up overnight, died, and sprouted again. Humans are, as it turns out, extremely into their own faces.
But Dreambooth is a whole process. You need multiple images. You need to train. It takes time. And I am fundamentally lazy. I wanted a model where you hand it one photo of a person and it just works. No fine-tuning, no waiting, just face in, face out, like a vending machine for fake people.
I figured someone smarter than me was going to build this anyway, but they were taking too long. So I started coding.
Attempt One: Face Recognition Embeddings, AKA How To Overfit In One Easy Step
My first idea was simple. I took Stable Diffusion and ripped out the text conditioning entirely. No prompts, no "a photo of a person," just a face crop fed through a pretrained face recognition network, and the resulting embedding would guide generation. The model would learn to turn that identity vector into a new image of the same person, with different hair, background, expression, whatever.
I trained it on FFHQ, a dataset of 70,000 aligned face images. That sounds like a lot. It is not a lot. By modern image generation standards, 70,000 images is basically a napkin sketch.
The results on training data looked great. Genuinely promising. The model was producing new images that clearly had the same nose, the same face shape, the same vibe.
Then I ran it on validation images. Faces the model had never seen.
It failed. Hard.
The model had overfit on the 70,000 training identities. It memorized their face embeddings instead of learning a general concept of "a face." I tried data augmentation on the face crops before feeding them to the recognition network. Clearly not enough. The face recognition network produced embeddings that were too similar between different photos of the same person. And apparently, too easy for the diffusion model to just memorize.
I stared at the outputs for a while. Some features did transfer. There was definitely something about nose shapes that stuck. But these images did not look like the original people. They looked like cousins. Distant cousins. Cousins you met once at a wedding and could not pick out of a lineup.
Attempt Two: CLIP To The Rescue
The face recognition network was too specific. It was designed to tell faces apart, not to understand what makes a face look like a face in general. So I swapped it for the obvious choice: CLIP image embeddings.
CLIP is the workhorse of a thousand diffusion projects. It is not specialized for faces, which turns out to be exactly what you want. It produces a broad, fuzzy representation of whatever is in the image, which is much harder for a diffusion model to memorize.
I followed the same pipeline. Crop face, extract CLIP embedding, feed it into the cross-attention layers instead of text. Train on FFHQ with CLIP as the encoder.
This was better. The model generalized to unseen faces. It was not perfect. The likeness was not Dreambooth-level. But it was a single image, no fine-tuning, and the output was recognizably the same person. I was getting somewhere.
The Pivot: Trailer Faces and the Retrieval Trick
Around this time, I built a dataset I am still unreasonably proud of: Trailer Faces HQ. I scraped movie trailers frame by frame and extracted every face I could find. The result was larger than FFHQ, and crucially, it had multiple shots of the same actors, both from different scenes in the same trailer and from different movies entirely.
This opened up a new training strategy. Instead of taking a single face crop and trying to augment it with random transforms, I could use retrieval. For each training step, I would take a face crop from one frame, then search the dataset for a different image of the same actor, and use that as the target. The model would learn to generate a new image of the person it saw in the conditioning crop, but with the pose, lighting, and expression from the retrieved image.
This was the right idea. Training with retrieval forced the model to learn actual identity transfer, because the conditioning image and the target image were always different photos.
The Fun Part: Mixing Faces and Backgrounds
The model worked. But I had removed text conditioning entirely, so the face embedding was the only control. You could generate new images of a person, but you could not tell it what kind of image to generate. So I tried something else.
I trained a version with two embeddings: one for the face region, and one for the rest of the image (hair, background, lighting, accessories). The face region was extracted as a crop. The background was the original image with the face masked out in gray.
When you use the same image for both embeddings, you get more photos of that person in a similar setting. Boring but expected.
When you mix them. That is where it gets interesting.
Grab the face embedding from person A and the background embedding from person B. The model generates person A with the hair, lighting, hat, background of person B. It is basically Face Mixer from the old StyleGAN days, but powered by diffusion.
And it kind of worked. Many generated images were recognizably the original person. They picked up features from the background image. Some things changed unexpectedly. Gender sometimes flipped. Expression tended to stick, which makes sense given how crudely I split the image.
I was delighted. This was exactly the kind of vibe I wanted. Face mixing but with diffusion.
The Punchline: Where The Hell Is That Checkpoint?
All of this happened in late 2022. I am writing this in early 2024. The gap has nothing to do with being busy. I lost the checkpoint.
I was going to release the model. I had the config files, the training scripts, the dataset. But the actual trained weights. Gone. I have twelve hard drives. I have folders called "checkpoints_old," "checkpoints_final," "checkpoints_FINAL_v2," "checkpoints_DO_NOT_DELETE." None of them contain the right file. I have searched. I have used grep on filenames. I have checked backup drives. I have stared at directory trees hoping the right combination of letters would materialize.
It did not.
So this blog post is not a triumphant release announcement. It is a eulogy. Here is what I built, here is how it worked, and here is the hard drive it is probably gathering cosmic rays on in a drawer somewhere.
If you want to recreate it, the configs are out there, the datasets are downloadable, and the approach is described above. Maybe you will have better luck keeping track of your files than I did.
Reflections on the Face Mixer Approach
Looking back, there are a few things I wish I had tried. Embedding interpolation between faces. SDEdit-style image-to-image starting from the generated output. Multiple classifier-free guidance scales applied independently to the face and background embeddings. Would have been fun experiments. I did not do them because I lost the weights.
I also want to give a nod to the GAN-era work that did this exact thing years ago. StyleGAN face mixing was a whole aesthetic. People made art with it. It is funny to see the same idea resurface with diffusion models, with all the same strengths and weaknesses, just with more compute and fancier math.
Dataset Notes
The three datasets used in this work are all publicly available: FFHQ (70k aligned face images), CelebA-HQ (30k celebrity faces at high resolution), and Trailer Faces HQ (a larger dataset built from movie trailer frames, with multiple shots of the same actors across different scenes and films). The retrieval-based training approach is the most interesting direction here, and I suspect someone with better organizational habits than me could push it much further.