Adoro: A deepfake app that lets you perform Italian opera classics using only a selfie

A new starter Workflow from Gradient helps you build a web application that can turn a selfie into a deepfaked video of singing an Italian opera song. This tutorial introduces the new starter Workflow.

2 months ago   •   6 min read

By David
Table of contents
Video walkthrough of this tutorial

We recently launched a new Gradient site that features a new campaign called "Make anything with Gradient."

In one of the vignettes, Stewart asks if he can use Gradient to make a very specific machine learning application – one that will let him create a deepfake of him singing an Italian opera classic.

Check out the video if you haven't seen it yet:

A new Gradient campaign launched recently called "Make anything with Gradient."

Today we're going to show you a new starter Workflow in Gradient called Adoro which will help you create your own version of this deepfake app – complete with a trained model to transfer motion from a driving video onto a static selfie.

The app that we will deploy will look like this:

The Adoro web app that we will create with Gradient Workflows will have a simple UI for taking a selfie and submitting that selfie for motion transfer from the driving video.

And the output we'll be able to create with the app will look something this:

The Adoro app will allow us to generate lip sync videos like this one!

Let's get started!

The Adoro starter project

Adoro takes advantage of a project called First Order Motion Model for Image Animation. The model was originally released by a team of researchers at NeurIPS 2019 and the original source code is available on Github here.

The Workflow will deploy a rudimentary web app that contains a user interface for interacting with the first order motion model to create new deepfaked videos.

The Adoro template comes pre-loaded with a driver video that we created. The driver video is a 256px x 256px video of an actor lip-syncing the timeless classic "La donna e mobile."

The driver video provides the reference for the final output video. The first order model will apply the head and facial movements of the driver video onto the static image that we provide.

What we will be doing next is creating a Workflow to transfer the motion and the music from the driver video onto a static selfie image of our designation.

Create a new project and connect Github

If this is your first Gradient Project, you'll be prompted to create a project and give it a name.

The first thing we'll need to do is create a new project to contain our Adoro Workflow. From the Gradient console, create a new project and give it a name.

Next, we'll create a new Workflow within our new project.

At this point if we haven't already granted Gradient access to Github we'll want to do so now. This will allow Gradient to write the Adoro Workflow to a new repository in Github.

We will want to grant Gradient access to at least one repo on Github if we haven't already done so.

Next we'll select the Adoro template. We'll also select the Github account or organization we want to write to, and we'll give our repo a name.

Create a new Workflow from the Gradient console and select the Adoro template.

We can now create our new Workflow! Once we select Create Workflow we should be able to see our Workflow processes running.

Once we start the Adoro Workflow we'll be taken to the Workflow view to see the processes running.

The Adoro Workflow contains two jobs: ValidateModel and CreateDeployment.

The YAML for the Workflow looks like this:

'on':
  github:
    branches:
      only: main
jobs:
  ValidateModel:
    uses: script@v1
    with:
      image: paperspace/first-order-model
      script: |-
        python3 demo.py --relative --adapt_scale \
          --config config/vox-256.yaml \
          --checkpoint /inputs/model/vox.pt \
          --driving_video /app/driving.mp4 \
          --source_image /app/source.png \
          --result_video /app/result.mp4
    inputs:
      model:
        type: dataset
        with:
          ref: gradient/voxpt
    outputs:
      results:
        type: dataset
        with:
          ref: demo-dataset
    resources:
      instance-type: P4000
  CreateDeployment:
    uses: script@v1
    with:
      image: paperspace/gradient-sdk
      script: >-
        cat > ./deployment.yaml <<EOF

        image: paperspace/adoro-server:1.0.0

        port: 8000

        resources:
          replicas: 1
          instanceType: P4000
        EOF

        apt update > /dev/null

        apt-get install -y jq

        gradient deployments create --name adoro-${RANDOM} --projectId
        ${PROJECT_ID} --spec ./deployment.yaml |awk '{print $3}'>
        ./deployment.id


        echo

        echo "Adoro can be accessed at URL:"

        gradient deployments get --id $(cat ./deployment.id)|jq
        '.deploymentSpecs[0].endpointUrl' -r
    needs:
      - ValidateModel
    resources:
      instance-type: C3
context:
  event:
    github:
      ref: b3355a1c37af89f014709f300dc8b3f335673229
      url: https://github.com/dbanys/adoro-project

Let's break down what we're doing.

First, we'll tell the Workflow to use the main branch of the new repo that we are creating:

'on':
  github:
    branches:
      only: main

Next, we'll define a pair of jobs, ValidateModel and CreateDeployment. In ValidateModel we'll tell Gradient to use a P4000 GPU instance type to pull an image from DockerHub called paperspace/first-order-model.

jobs:
  ValidateModel:
    uses: script@v1
    with:
      image: paperspace/first-order-model
      script: |-
        python3 demo.py --relative --adapt_scale \
          --config config/vox-256.yaml \
          --checkpoint /inputs/model/vox.pt \
          --driving_video /app/driving.mp4 \
          --source_image /app/source.png \
          --result_video /app/result.mp4
    inputs:
      model:
        type: dataset
        with:
          ref: gradient/voxpt
    outputs:
      results:
        type: dataset
        with:
          ref: demo-dataset
    resources:
      instance-type: P4000

Within the ValidateModel step we are also defining a number of variables for the demo.py file to execute, including the config and checkpoint files, as well as the driving video (the video above with the actor), the source image (our selfie), and the output video.

Next, we define the CreateDeployment step which will use a C3 CPU instance type to host a very simple web server. The web server is defined by the DockerHub image paperspace/adoro-server.

jobs:
  ValidateModel:
    ...
  CreateDeployment:
    uses: script@v1
    with:
      image: paperspace/gradient-sdk
      script: >-
        cat > ./deployment.yaml <<EOF

        image: paperspace/adoro-server:1.0.0

        port: 8000

        resources:
          replicas: 1
          instanceType: P4000
        EOF

        apt update > /dev/null

        apt-get install -y jq

        gradient deployments create --name adoro-${RANDOM} --projectId
        ${PROJECT_ID} --spec ./deployment.yaml |awk '{print $3}'>
        ./deployment.id


        echo

        echo "Adoro can be accessed at URL:"

        gradient deployments get --id $(cat ./deployment.id)|jq
        '.deploymentSpecs[0].endpointUrl' -r
    needs:
      - ValidateModel
    resources:
      instance-type: C3

Finally our Workflow defines a context parameter. This refers to the particular git commit that triggered this particular Workflow run.

context:
  event:
    github:
      ref: b3355a1c37af89f014709f300dc8b3f335673229
      url: https://github.com/dbanys/adoro-project

Once our Workflow finishes running, let's take a look at the logs.

When the Workflow finishes running we should see a web address specified in the logs.

Everything looks good!

Reviewing the logs, we can see that the CreateDeployment step of the Workflow has generated a public URL to view our new we app. The logs should look like this (note that you will have a different URL for your own application):

Adoro can be accessed at URL:
da9503c3f4d6c4d90ace21c3769fa81af.clg07azjl.paperspacegradient.com

This is what we should see when we visit the URL:

The Adoro Workflow deploys the Adoro web app to a public web address

From here all we need to do is snap a picture and let the application work its magic!

We'd love to see your Adoro! Let us know @hellopaperspace what you're able to create!

Bring this project to life

Spread the word

Keep reading