Animate 2D faces using Thin Plate Spline Motion Model (TPSM)

Описание

Let's animate our 2D art or photography using TPSM or Thin Plate Spline Motion Model, this is a local install tutorial, but I also left a link for an online source that will let you do this immediately.

This is fairly simple install, but I ran into a lot of issues unrelated to the installation instructions. All the tutorials I could find were for Linux, so I had to do a few minor tweaks to make it run on windows 11.

Here is an alternate way w/ cleaner results but limitations https://www.youtube.com/watch?v=nMBjtEbNfoc&ab_channel=ArtificiallyIntelligent

Need to upscale your video? https://pollinations.ai/p/QmWF8xtJ47uk2QARcPNutp7JBPJLVV11vKD7eAV4d2v9tp/create
or https://github.com/k4yt3x/video2x

Downloads
Here is the github you will need for the local install - https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model

Here is the google drive where you get your checkpoints - https://drive.google.com/drive/folders/1pNDo1ODQIb5HVObRtCmubqJikmR7VVLT

Here is where to get the code for pytorch - https://pytorch.org/

Here is the site to get Anaconda - https://www.anaconda.com/products/distribution

Here is the site to download Git if do not want to grab it via Anaconda - https://git-scm.com/downloads

Optional CUDA driver download - https://developer.nvidia.com/cuda-downloads

Need a tutorial to install Anaconda and Pytorch? Check out Nerdy Rodents tutorial on it - https://www.youtube.com/watch?v=OjOn0Q_U8cY&t=106s&ab_channel=NerdyRodent

========= installation============================

Step 1. Creating your Virtual environment (

########### Creating A Virtual Environment
conda create --name thin-plate-spline python=3.9

########## Run this once per session
conda activate thin-plate-spline
CD Thin-Plate-Spline-Motion-Model folder location (i.e. CD C:\Users\%\Desktop\Thin-Plate-Spline-Motion-Model)
Set CUDA_VISIBLE_DEVICES=0

########## Running TPSM testing Code (USE THIS ONE IF YOU HAVE CUDA DRIVERS)

#Assets folder only
python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image assets/source.png --driving_video assets/driving.mp4 --result_video Test_Results.mp4

=========================================================

#Assets folder only with CPU only (USE THIS IF YOU DON'T HAVE CUDA DRIVERS)

python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image assets/source.png --driving_video assets/driving.mp4 --result_video Test_Results.mp4 --CPU

#Custom Folders for images(input1), video(input2), output(output)
python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image images/source.png --driving_video videos/driving.mp4 --result_video output/Test_Results_001.mp4 --find_best_frame

#Code broken down

python demo.py --config config/vox-256.yaml \
--checkpoint checkpoints/vox.pth.tar --source_image assets/source.png \
--driving_video assets/driving.mp4 --find_best_frame \
--result_video Test_Results_001.mp4 \
--result_video output/Test_Results_001.mp4 \
--find_best_frame \
--CPU

#########Don't forget you will have to activate and change to the correct directory to use this tool.
Activate code: conda activate thin-plate-spline
Change to Correct Directory - CD Thin-Plate-Spline-Motion-Model folder location


===================More info if your curious==============
############## What the code does
conda create --name thin-plate-spline python=3.9 # Created 3.9 python environment, we will only do this once
conda activate thin-plate-spline # Activates the environment, we will run this every time we start up anaconda and wish to use TPSM

############### Running the program
Set CUDA_VISIBLE_DEVICES=0 # This will activate GPU0 for CUDA Optimized drivers this will not turn off CPU processing
python demo.py --config config/vox-256.yaml # This executes config
--checkpoint checkpoints/vox.pth.tar --source_image assets/source.png #This selects your pre-trained models in this example it is selecting vox
--driving_video assets/driving.mp4 # This will select a driving video from the "assets" folder and the file name as "driving.mpg"
--result_video Test_Results_001.mp4 # This is a command for your output video and it is going to the root folder which should be the thin-plate-spline-motion-model that you unzipped from the github repository
--result_video output/Test_Results_001.mp4 # This command will send your output to the folder named output in the root directory and the file will be named Test_Results_001
--find_best_frame # This will try and optimize to create a better result
--CPU #this will be used if you are running AMD or you didn't install CUDA drivers

Like Art? check out my Deviant Art site: https://www.deviantart.com/justthetipwithdaniel

Pics at end are from MJ community

#stablediffusion #aiartcommunity #aiart #texttovideo #TPSM