Digital Verse - Metaverse of the Digital Avatars

Materials

https://drive.google.com/drive/folders/1YMkA0egela66tr3zzgOw7-jKlx4zDF_d?usp=sharing

Demo access

Landing -> https://digitalverse.ai

Link to web demo -> https://app.digitalverse.ai/

Demo accounts -> login: tomcruise@digitalverse.ai password: tomcruise@digitalverse.ai

login: margotrobbie@digitalverse.ai password: margotrobbie@digitalverse.ai

Support: @panichm (telegram account)

Recommends for good results: upload small videos (5-20 seconds long), keep your face away from the camera for better quality, use videos with standard lighting.

Problem/Annotation

Synthetic media is a realistic transformation of audio and video using artificial intelligence. Currently, there are several applications based on this technology, but it’s developing rapidly, attracting more and more public attention.

• On one hand, this technology is a holy grail for advertisers and filmmakers which can give them endless opportunities to use any faces of any celebrities in their projects. With the help of our platform, the celebrity will be able to pronounce the text of the commercial in all the world's known languages. The advertiser will have a chance to create a separate commercial for ten thousand products, having only one digitized version of the celebrity's face. Film producers won't have to pay multimillion-dollar royalties to celebrities, it will be enough to buy their face.

• On the other hand, without proper regulation this technology is a sophisticated threat for businesses and individuals. The illegal use of faces is gaining momentum. Debates about the originality of the synthetic videos and lawsuits attract lots of attention, thereby encouraging the creation of content with celebrities without their consent. Technology is evolving fast and it’s only a matter of time before synthetic videos will be no longer distinguishable from the original.

We believe that with the help of blockchain this problem can be solved, NFT have a potential to create metaverse of the digital avatars of the users and regulate the relationship between video producers and celebrities, while copyright and data protection laws still cannot.

Solution

Our solution is a blockchain-based NFT marketplace of digital faces with a platform for synthetic videos generation.

Platform architecture -> https://drive.google.com/file/d/1R9-Pf1u-HHQ4s0dC99Ys2QAIVhVW7u0k/view?usp=sharing

The user (for example celebrity) will be able to digitize his face by uploading a video of himself to the platform. Digital avatar of the user will be presented on the platform for the potential customers. After receiving the offer to buy the face, the user will be able to choose whether to sell it for this particular video/commercial. In case of consent, an NFT token gets created.

After the token is minted, potential video creators can purchase this NFT, thereby acquiring the right to use the celebrity's face to create one synthetic video.

With our solution, video creators will be able to purchase a digitized face and produce a video with this face on the same platform. And all this without the need for real filming and with digital confirmation on the blockchain. The NFT will eliminate the need to obtain IPR rights, which will be assigned to the video content producers with the purchase of the token.

Concept & Feasibility

We have identified 2 main areas where our solution may be in demand.

Video marketing

Video production today is a very expensive process that includes the rent of cameras, studios and payments for the work of actors.

There are several stages of the video creation process:

• Concept development (plot, plans, etc.);
• Pre-production (preparation of scripts, equipment);
• Shooting;
• Post-production (editing of the footage);

Once a commercial is filmed, it is very difficult to make any edits, they are expensive and time-consuming. But instead of shooting a new video for marketing localization, language replacement and other corrections, it's enough to simply edit the existing one on the post production stage using our technology. For example, we are able to remove all parasite words from the video, replace phrases and translate a video into different languages ​​with a natural voice and facial expressions of the actor.

Our target audience is video creation agencies, film studios, and individual authors.

Market Size -> https://drive.google.com/file/d/1hGHJvlBlQcjCMBtt5lkaf5ShKJxGW2xy/view?usp=sharing

PAM - $234B. The global film and video market is expected to grow from $234.91 billion in 2020 to $251.92 billion in 2021 with an annual growth rate of 7.2%. And projected to reach $318.23 billion in 2025.

TAM - $163B. ~70% from the total video market is formed by customers who may potentially need our product. That leaves us with $163B.

SAM - $32B. Due to the novelty of the technology, we assume that of all customers who could potentially need our solution, 20% will want to try it at this stage.

SOM - $20B. We don't have competitors in the AI field at the moment, which is also explained by the novelty of technology. Our main competitors are traditional video editing softwares (like Adobe Premiere Pro). Those applications partially perform the functionality of our solution, but they are more expensive and time-consuming.

As a monetization, we propose using either a fixed price for processing one video or a subscription for using our application.

Information Security

15.000 of Deep Fake videos were discovered on the Internet in 2019. Which is an 84% increase from just 8.000 in 2018. By now, this number has grown dramatically.

Large companies like Facebook are already sounding the alarm about the potential impact of Deep Fake videos. However, this problem concerns almost every media service - youtube, social networks, etc.

To solve this problem we are aiming to provide to Deep Fake creators the regulated field for their creativity and developments. We are on the way to create the first licensing and copyright management platform using NFTs. And enable creators to fairly buy rights for the “faces”.

With the help of NFT the owner of the token will be able to confirm the IPR for the video.

Our target audience - video content creators, influencers, celebrities, bloggers.

The planned monetization model is a one-time payment for checking/minting one video.

Project Design

We have developed a scalable architecture that consists of several modules:

Module for replacing faces on video

It consists of machine learning algorithms written in Python. There are 3 main parts:

• Data preparation

Each uploaded video is divided into frames, on each frame we detect the faces, process them in a certain way (align, improve the resolution) and use in next modules.

• Model training

For each celebrity/user we train his own machine learning model (aka Digital Avatar) on the GPU using previously created faceset, which later allows you to replace faces/synchronized lips almost instantly without any additional training.

• Video merging

Using previously created Digital Avatar, we replace faces on frames from uploaded video and generate a new video using various additional functions for color correction, quality improvement and etc.

Our main difference from other deepfake projects is that we have automated the process of creating videos as much as possible (you don't need a lot of manual work, just upload a video and thats it) and due to separate Digital Avatar for each user, we can almost instantly generate videos with high quality. Also, it is possible to additionally train the model on specific video and get a better quality/resolution (for b2b requests).

Smart contracts

Supports Ethereum/Avalanche blockchains and allows to mint and transfer NFT tokens.

NFT Token smart contract functions:

• MintTokenToAddress(address owner, string memory metadataURI) Allows you to mint a NFT token.
• TransferFrom(address from, address to, uint256 tokenId) Allows you to transfer any NFT token to another address.
• Implementation of access control

NFT Market smart contract. Allows you to list and sell created NFT tokens, functions:

• CreateMarketItem(address nftContract,uint256 tokenId,uint256 price)
• CreateMarketSale(address nftContract,uint256 itemId)
• FetchMarketItems() public view returns (MarketItem[] memory)
• FetchMyNFTs() public view returns (MarketItem[] memory)

Backend

Written in golang for interacting with machine learning algorithms, implementing business logic and interacting with the blockchain.

Front-end

Implemented on React.js.

Current implemented workflow in the demo

1. The user uploads a video with his face recorded in different angels and get his personal Digital Avatar;
2. User select a desired video for faceswap from pre-don video gallery or upload his own custom video;
3. With the help of the face replacement module, the video is divided into frames, a faces is detected on each frame. Using a machine learning model, this faces are replaced by another and then a new synthetic video is generated;
4. Next step is to upload the synthetic video to IPFS storage;
5. Then the link to the video in IPFS storage and additional information (name, description) is sent to the smart contract to create an NTF token. The token can be transferred, sold.

Roadmap

We plan to further develop the platform. In the current demo version we already implemented the interface for bloggers/influencers, so they could generate different synthetic videos. Soon we will release a full version of NFT marketplace of the Digital Avatars. Our next steps:

Update a personal account for celebrities / users, where they can:

1. Be able to accept offers from companies/brands to use a Digital Avatar;
2. Add support for voice digitalization and lips synchronization based on audio tracks.

Create a personal account for business with the ability to:

1. Upload the video and select the person to replace the face / voice;
2. Or just send your script and we will record the video ourselves;
3. Buy NFT token of generated video.

• Business - Create Step 1 -> https://drive.google.com/file/d/1PmXg5tjHULTTJY0qk6loft4mkf9foMjA/view?usp=sharing
• Business - Create Step 2 -> https://drive.google.com/file/d/13a6qc-fKbzX3Xzgu7ZpY2jXo_7pQsRTU/view?usp=sharing

Improve the structure of smart contracts:

1. So that user can transfer rights to use his face;
2. Smart contract for NFT shop.

Update machine learning algorythms:

1. Train models with higher resolution. Current resolution is 320px, it is possible to improve it up to 448-640px, i.e. quality boost in 1.5-2x times.
2. Fix problems with artifacts during fast head movement and different lighting/color gradation, improve face detection algorithms.