Generative Modeling as Sampling

What is Generative Modeling#

Objects as Vectors#

The objects being generated are identified as vectors \(z\in \mathbb{R}^d\). (for flow and diffusion models, we focus on applications to continuous data.)

Examples:

• Image. An image can be represented by an element \(z\in \mathbb{R}^{H\times W\times 3}\) where \(H\) is the pixels of the image in height, and \(W\) the width; the factor of \(3\) accounts for three color channels (RGB). For every pixel and every color channel we are given an intensity value in \(\mathbb{R}\).
• Video. A video is simply a series of images in time. We have \(T\) frams, then a video is \(z\in \R^{T\times H\times W\times 3}\).
• Molecular structure. For example, represent the structure of amocule by a matrix \(z=(z^1,...,z^N)\in \R^{3\times N}\) where \(N\) is the number of atoms in the molecule and each \(z^i\in \R^3\) describes the location of that atom.

Generation as Sampling#

Generating an object \(z\) is modeled as sampling from the data distribution \(z\sim p_\text{data}\).

A generative model is a machine learning model that allows us to generate samples from \(p_\text{data}\).

Since there are many possible objects that can be generated, and there is no one single “best” object, there is a spectrum of objects that fit  better or worse. This idversity of possible objects are thought of as a probability distribution and called the data distribution \(p_{\text{data}}\). How “good” an object fits is replaced by how “likely” it is under the data distribution \(p_\text{data}\).

Hence mathematically, the task of generation can be described as sampling from the (unknown) distribution \(p_{\text{data}}\). We usually assume access to a finite number of examples sampled independently from \(p_\text{data}\) which together serve as a proxy for the true distribution.

A dataset consists of a finite number of samples \(z_1,...,z_N\sim p_\text{data}\).

In many cases we want to generate an object conditioned on some data \(y\), e.g., then rephrase this as sampling from a conditional distribution:

Conditional generation involves sampling from \(z\sim p_\text{data}(\cdot|y)\) with \(y\) being a conditioning variable. The choice of \(y\) is typically arbitrary rather than fixed.

Here \(p_\text{data}(\cdot|y)\) is called the conditional data distribution. Since \(y\) is typically not fixed, we seek a single model which may be conditioned on any such choice of \(y\). Techniques for unconditional generation are readily generalized to the conditional case, therefore first we may focus exclusively on the unconditional case.

How to do Generative modeling#

Assume that we have access to some initial distrbution \(p_\text{init}\) that we can sample from, the goal of a generative modeling is then to transform samples from \(x\sim p_\text{init}\) into samples from \(p_\text{data}\). In the majority of applications we take \(p_\text{init}\) to be a simple Gaussian. Thus the process is basically from noise to data.