loss=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt=−∬∫q(x)q(zt−1,zt∣x)dxlogp(zt−1∣zt)dzt−1dzt=∬∫q(x)q(zt−1,zt∣x)logq(zt−1∣zt,x)dxdzt−1dztThis Term Is Constant And Is Represented ByC1−∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)dxdzt−1dzt−C1=∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)q(zt−1∣zt,x)dxdzt−1dzt−C1=∬q(x)q(zt∣x)∫q(zt−1∣zt,x)logp(zt−1∣zt)q(zt−1∣zt,x)dzt−1dzxdzt−C1=∬q(x)q(zt∣x)KL[q(zt−1∣zt,x)∣p(zt−1∣zt)]dxdzt−C1∝∬q(x)q(zt∣x)KL[q(zt−1∣zt,x)∣p(zt−1∣zt)]dxdzt(6.8)(6.9)(6.10)(6.11)(6.12)(6.13)(6.14)(6.15)
The Diffusion Probability Model[1][2] is currently the main method used in image and video generation, but due to its abstruse theory, many engineers are unable to understand it well. This article will provide a very easy-to-understand method to help readers grasp the principles of the Diffusion Model. Specifically, it will illustrate the Diffusion Model using examples of one-dimensional random variables in an interactive way, explaining several interesting properties of the Diffusion Model in an intuitive manner.
\n
The diffusion model is a probabilistic model. Probabilistic models mainly offer two functions: calculating the probability of a given sample appearing; and generating new samples. The diffusion model focuses on the latter aspect, facilitating the production of new samples, thus realizing the task of generation.
\n
The diffusion model differs from general probability models (such as GMM), which directly models the probability distribution of random variables. The diffusion model adopts an indirect approach, which utilizes random variable transform(shown in Figure 1a) to gradually convert the data distribution (the probability distribution to be modeled) into the standard normal distribution, and meanwhile models the posterior probability distribution corresponding to each transformation (Figure 1b-c). Upon obtaining the final standard normal distribution and the posterior probability distributions, one can generate samples of each random variable ZT…Z2,Z1,X in reverse order through Ancestral Sampling. Simultaneously, initial data distribution q(x) can be determined by employing Bayes theorem and the total probability theorem.
\n
One might wonder: indirect methods require modeling and learning T posterior probability distributions, while direct methods only need to model one probability distribution, Why would we choose the indirect approach? Here's the reasoning: the initial data distribution might be quite complex and hard to represent directly with a probability model. In contrast, the complexity of each posterior probability distribution in indirect methods is significantly simpler, allowing it to be approximated by simple probability models. As we will see later, given certain conditions, posterior probability distributions can closely resemble Gaussian distributions, thus a simple conditional Gaussian model can be used for modeling.
\n
\n
Figure 1: Diffusion probability model schematic
","transform_en":"
To transform the initial data distribution into a simple standard normal distribution, the diffusion model uses the following transformation method: \nZ=αX+1−αϵwhereα<1,ϵ∼N(0,I)(1.1)\nwhere X∼q(x)is any random variable，Z∼q(Z) is the transformed random variable。
\n
This transformation can be divided into two sub-transformations。
\n
The first sub-transformation performs a linear transformation (αX) on the random variable X. According to the conclusion of the literature[3], the linear transformation makes the probability distribution of Xnarrower and taller, and the extent of narrowing and heightening is directly proportional to the value of α.
\n
This can be specifically seen in Demo 1, where the first figure depicts a randomly generated one-dimensional probability distribution, and the second figure represents the probability distribution after the linear transformation. It can be observed that the curve of the third figure has become narrower and taller compared to the first image. Readers can experiment with different α to gain a more intuitive understanding.
\n
The second sub-transformation is adding independent random noise(1−αϵ). According to the conclusion of the literature[4], adding independent random variables is equivalent to performing convolution on the two probability distributions. Since the probability distribution of random noise is Gaussian, it is equivalent to performing a Gaussian Blur operation. After blurring, the original probability distribution will become smoother and more similar to the standard normal distribution. The degree of blurring is directly proportional to the noise level (1−α).
\n
For specifics, one can see Demo 1, where the first figure is a randomly generated one-dimensional probability distribution, and the third figure is the result after the transformation. It can be seen that the transformed probability distribution curve is smoother and there are fewer corners. The readers can test different α values to feel how the noise level affect the shape of the probability distribution. The last figure is the result after applying all two sub-transformations.
\n
","likelihood_en":"
From the transformation method (equation 1.1), it can be seen that the probability distribution of the forward conditional probability q(z∣x) is a Gaussian distribution, which is only related to the value of α, regardless of the probability distribution of q(x). \nq(z∣x)=N(αx,1−α)(2.1)\nIt can be understood by concrete examples in Demo 2. The third figure depict the shape of q(z∣x). From the figure, a uniform slanting line can be observed. This implies that the mean of q(z∣x) is linearly related to x, and the variance is fixed. The magnitude of α will determine the width and incline of the slanting line.
\n
","posterior_en":"
The posterior probability distribution does not have a closed form, but its shape can be inferred approximately through some technique.
\n
According to Bayes formula, we have\nq(x∣z)=q(z)q(z∣x)q(x)(3.1)
\n
When z takes a fixed value, q(z) is a constant, so the shape of q(x∣z) is only related to q(z∣x)q(x).\nq(x∣z)∝q(z∣x)q(x)wherezisfixed(3.2)
\n
From Equation 2.1, we can see that q(z∣x) is a Gaussian distribution, so we have\nq(x∣z)∝2π(1−α)1exp2(1−α)−(z−αx)2q(x)=α1GaussFun2πσ1exp2σ2−(x−μ)2q(x)wherezisfixedwhereμ=αzσ=α1−α(3.3)(3.4)
\n
It can be observed that the GaussFun part is a Gaussian function of x, with a mean of αz and a variance of α1−α, so the shape of q(x∣z) is determined by the product of GaussFun and q(x).
\n
According to the characteristics of multiplication, the characteristics of the shape of the q(x∣z) function can be summarized.
\n
\n
When the variance of the Gaussian function is small (small noise), or when q(x) changes slowly, the shape of q(x∣z) will approximate to the Gaussian function, and have a simpler function form, which is convenient for modeling and learning.
\n \n
When the variance of the Gaussian function is large (large noise), or when q(x) changes drastically, the shape of q(x∣z) will be more complex, and greatly differ from a Gaussian function, which makes it difficult to model and learn.
\n
\n\n
The specifics can be seen in Demo 2. The fourth figure present the shape of the posterior q(x∣z), which shows an irregular shape and resembles a curved and uneven line. As α increases (noise decreases), the curve tends to be uniform and straight. Readers can adjust different α values and observe the relationship between the shape of posterior and the level of noise. In the last figure, the blue dash line represents q(x), the green dash line represents GaussFun in the equation 3.4, and the orange curve represents the result of multiplying the two function and normalizing it, which is the posterior probability q(x∣z=fixed) under a fixed z condition. Readers can adjust different values of z to observe how the fluctuation of q(x) affect the shape of the posterior probability q(x∣z).
\n
The posterior q(x∣z) under two special states are worth considering.
\n
\n
As α→0, the variance of GaussFun tends to ∞, and q(x∣z) for different z almost become identical, and almost the same as q(x). Readers can set α to 0.001 in Demo 2 to observe the specific results.
\n \n
As α→1, the variance of GaussFun tends to 0, The q(x∣z) for different z values contract into a series of Dirac delta functions with different offsets equalling to z. However, there are some exceptions. When there are regions where q(x) is zero, the corresponding q(x∣z) will no longer be a Dirac delta function, but a zero function. Readers can set α to 0.999 in Demo 2 to observe the specific results.
\n
","forward_process_en":"
For any arbitrary data distribution q(x), the transform(equation 2.1) in section 2 can be continuously applied(equation 4.1~4.4). As the number of transforms increases, the output probability distribution will become increasingly closer to the standard normal distribution. For more complex data distributions, more iterations or larger noise are needed.
\n
Specific details can be observed in Demo 3.1. The first figure illustrates a randomly generated one-dimensional probability distribution. After seven transforms, this distribution looks very similar to the standard normal distribution. The degree of similarity increases with the number of iterations and the level of the noise. Given the same degree of similarity, fewer transforms are needed if the noise added at each step is larger (smaller α value). Readers can try different α values and numbers of transforms to see how similar the final probability distribution is.
\n
The complexity of the initial probability distribution tends to be high, but as the number of transforms increases, the complexity of the probability distribution q(zt) will decrease. As concluded in section 4, a more complex probability distribution corresponds to a more complex posterior probability distribution. Therefore, in order to ensure that the posterior probability distribution is more similar to the Conditional Gaussian function (easier to learn), a larger value of α (smaller noise) should be used in the initial phase, and a smaller value of α (larger noise) can be appropriately used in the later phase to accelerate the transition to the standard normal distribution.
\n
In the example of Demo 3.1, it can be seen that as the number of transforms increases, the corners of q(zt) become fewer and fewer. Meanwhile, the slanting lines in the plot of the posterior probability distribution q(zt−1∣zt) become increasingly straight and uniform, resembling more and more the conditional Gaussian distribution.
By substituting Equation 4.1 into Equation 4.2, and utilizing the properties of Gaussian distribution, we can derive the form of q(z2∣x) \nz2=α2(α1x+1−α1ϵ1)+1−α2ϵ2=α2α1x+α2−α2α1ϵ1+1−α2ϵ2=N(α1α2x,1−α1α2)(4.5)(4.6)(4.7)
\n
In the same way, it can be deduced recursively that\nq(zt∣x)=N(α1α2⋯αtx,1−α1α2⋯αt)=N(αtˉx,1−αtˉ)whereαtˉ≜j=1∏tαj(4.8)
\n
Comparing the forms of Equation 4.8 and Equation 2.1, it can be found that their forms are completely consistent. If only focusing on the final transformed distribution q(zt), then the t consective small transformations can be replaced by one large transformation. The α of the large transformation is the accumulation of the α from each small transformation.
\n
In the DDPM[2] paper, the authors used 1000 steps (T=1000) to transform the data distribution q(x) to q(zT). The probability distribution of q(zT∣x) is as follows:\nq(zT∣x)=N(0.00635x,0.99998)(4.9)
\n
If considering only marginal distribution q(zT), a single transformation can also be used, which is as follows:\nZT=0.0000403X+1−0.0000403ϵ=0.00635X+0.99998ϵ(4.10)\nIt can be seen that, after applying two transforms, the transformed distributions q(zT∣x) are the same. Thus, q(zT) is also the same.
\n
","backward_process_en":"
If the final probability distribution q(zT) and the posterior probabilities of each transform q(x∣z),q(zt−1∣zt) are known, the data distribution q(x) can be recovered through the Bayes Theorem and the Law of Total Probability, as shown in equations 5.1~5.4. When the final probability distribution q(zT) is very similar to the standard normal distribution, the standard normal distribution can be used as a substitute.
\n
Specifics can be seen in Demo 3.2. In the example, q(zT) substitutes N(0,1), and the error magnitude is given through JS Divergence. The restored probability distribution q(zt) and q(x) are identified by the green curve, and the original probability distribution is identified by the blue curve. It can be observed that the data distribution q(x) can be well restored, and the error (JS Divergence) will be smaller than the error caused by the standard normal distribution replacing q(zT).\nq(zT−1)q(zt−1)q(z1)q(x)=∫q(zT−1,zT)dzT=∫q(zT−1∣zT)q(zT)dzT…=∫q(zt−1,zt)dzt=∫q(zt−1∣zt)q(zt)dzt…=∫q(z1,z2)dz1=∫q(z1∣z2)q(z2)dz2=∫q(x,z1)dz1=∫q(x∣z1)q(z1)dz1(5.1)(5.2)(5.3)(5.4)\nIn this article, the aforementioned transform is referred to as the Posterior Transform. For example, in equation 5.4, the input of the transform is the probability distribution function q(z1), and the output is the probability distribution function q(x).The entire transform is determined by the posterior q(x∣z1). This transform can also be considered as the linear weighted sum of a set of basis functions, where the basis functions are q(x∣z1) under different z1, and the weights of each basis function are q(z1). Some interesting properties of this transform will be introduced in Section 7.
\n
In Section 3, we have considered two special posterior probability distributions. Next, we analyze their corresponding posterior transforms.
\n
\n
When α→0, the q(x∣z) for different z are almost the same as q(x). In other words, the basis functions of linear weighted sum are almost the same. In this state, no matter how the input changes, the output of the transformation is always q(x).
\n
When α→1, the q(x∣z) for different z values becomes a series of Dirac delta functions and zero functions. In this state, as long as the support set of the input distribution is included in the support set of q(x), the output of the transformation will remain the same with the input.
\n
\n\n
In Section 4, it is mentioned that the 1000 transformations used in the DDPM[2] can be represented using a single transformation\nZT=0.0000403X+1−0.0000403ϵ=0.00635X+0.99998ϵ(5.5)
\n
Since α=0.0000403 is very small, the corresponding standard deviation of GaussFun (Equation 3.4) reaches 157.52. However, the range of X is limited within [−1,1], which is far smaller than the standard deviation of GaussFun. Within the range of x∈[−1,1], GaussFun should be close to a constant, showing little variation. Therefore, the q(x∣zT) corresponding to different zT are almost the same as q(x). In this state, the posterior transform corresponding to q(x∣zT) does not depend on the input distribution, the output distribution will always be q(x).
\n
Therefore, theoretically, in the DDPM model, it is not necessary to use the standard normal distribution to replace q(zT). Any other arbitrary distributions can also be used as a substitute.
\n
Readers can conduct a similar experiment themselves. In Demo 3.1, set start_alpha to 0.25, end_alpha to 0.25, and step to 7. At this point, q(z7)=0.000061X+1−0.000061ϵ, which is roughly equivalent to DDPM's q(zT). Click on apply to perform the forward transform (plotted using blue curves), which prepares for the subsequent restoring process. In Demo 3.2, set the noise_ratio to 1, introducing 100% noise into the tail distributionq(z7). Changing the value of nose_random_seed will change the distribution of noise. Deselect backward_pdf to reduce screen clutter. Click on apply to restore q(x) through posterior transform. You will see that, no matter what the shape of input q(z7) may be, the restored q(x) is always exactly the same as the original q(x). The JS Divergence is zero. The restoration process is plotted using a red curve.
\n
","fit_posterior_en":"
From the front part of Section 3, it is known that the posterior probability distributions are unknown and related to q(x). Therefore, in order to recover the data distribution or sample from it, it is necessary to learn and estimate each posterior probability distribution.
\n
From the latter part of Section 3, it can be understood that when certain conditions are met, each posterior probability distribution q(x∣z),q(zt−1∣zt) approximates the Gaussian probability distribution. Therefore, by constructing a set of conditional Gaussian probability models p(x∣z),p(zt−1∣zt), we can learn to fit the corresponding q(x∣z),q(zt−1∣zt).
\n
Due to the limitations of the model's representative and learning capabilities, there will be certain errors in the fitting process, which will further impact the accuracy of restored q(x). The size of the fitting error is related to the complexity of the posterior probability distribution. As can be seen from Section 3, when q(x) is more complex or the added noise is large, the posterior probability distribution will be more complex, and it will differ greatly from the Gaussian distribution, thus leading to fitting errors and further affecting the restoration of q(x).
\n
Refer to Demo 3.3 for the specifics. The reader can test different q(x) and α, observe the fitting degree of the posterior probability distribution q(zt−1∣zt) and the accuracy of restored q(x). The restored probability distribution is ploted with orange, and the error is also measured by JS divergence.
\n
Regarding the objective function for fitting, similar to other probability models, the cross-entropy loss can be optimized to make p(zt−1∣zt) approaching q(zt−1∣zt). Since (zt−1∣zt) is a conditional probability, it is necessary to fully consider all conditions. This can be achieved by averaging the cross-entropy corresponding to each condition weighted by the probability of each condition happening. The final form of the loss function is as follows.\nloss=−∫q(zt)∫q(zt−1∣zt)logp(zt−1∣zt)dzt−1Cross Entropydzt=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt(6.1)(6.2)
\n
KL divergence can also be optimized as the objective function. KL divergence and cross-entropy are equivalent[10]\nloss=∫q(zt)KL(q(zt−1∣zt)∣p(zt−1∣zt))dzt=∫q(zt)∫q(zt−1∣zt)p(zt−1∣zt)q(zt−1∣zt)dzt−1dzt=−∫q(zt)Cross Entropy∫q(zt−1∣zt)logp(zt−1∣zt)dzt−1dzt+Is Constant∫q(zt)∫q(zt−1∣zt)logq(zt−1∣zt)dz(6.3)(6.4)(6.5)
\n
The integral in equation 6.2 does not have a closed form and cannot be directly optimized. The Monte Carlo integration can be used for approximate calculation. The new objective function is as follows:\nloss=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt≈−i=0∑Nlogp(Zt−1i∣Zti)where(Zt−1i,Zti)∼q(zt−1,zt)(6.6)(6.7)
\n
The aforementioned samples (Zt−1i,Zti) follow a joint probability distribution q(zt−1,zt), which can be sampled via an Ancestral Sampling. The specific method is as follows: sample X,Z1,Z2…Zt−1,Zt step by step through forward transforms (Formulas 4.1~4.4), and then reserve (Zt−1,Zt) as a sample. This sampling process is relatively slow. To speed up the sampling, we can take advantage of the known features of the probability distribution q(zt∣x) (Formula 4.8). First, sample X from q(x), then sample Zt−1 from q(zt−1∣x), and finally sample Zt from q(zt∣zt−1). Thus, a sample (Zt−1,Zt) is obtained.
\n
Some people may question that the objective function in Equation 6.3 seems different from those in the DPM[1] and DDPM[2] papers. In fact, these two objective functions are equivalent, and the proof is given below.
\n
For Consistent Terms, the proof is as follows:
\n
loss=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt=−∬∫q(x)q(zt−1,zt∣x)dxlogp(zt−1∣zt)dzt−1dzt=∬∫q(x)q(zt−1,zt∣x)logq(zt−1∣zt,x)dxdzt−1dztThis Term Is Constant And Is Represented ByC1−∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)dxdzt−1dzt−C1=∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)q(zt−1∣zt,x)dxdzt−1dzt−C1=∬q(x)q(zt∣x)∫q(zt−1∣zt,x)logp(zt−1∣zt)q(zt−1∣zt,x)dzt−1dzxdzt−C1=∬q(x)q(zt∣x)KL[q(zt−1∣zt,x)∣p(zt−1∣zt)]dxdzt−C1∝∬q(x)q(zt∣x)KL[q(zt−1∣zt,x)∣p(zt−1∣zt)]dxdzt(6.8)(6.9)(6.10)(6.11)(6.12)(6.13)(6.14)(6.15)
\n
In the above formula, the term C1 is a fixed value, which does not contain parameters to be optimized. Here, q(x) is a fixed probability distribution, and q(zt−1∣zt) is also a fixed probability distribution, whose specific form is determined by q(x) and the coefficient α.
\n
For the Reconstruction Term, it can be proven in a similar way.
Therefore, the objective function in equation 6.1 is equivalent with the DPM original objective function.
\n
Based on the conclusion of the Consistent Terms proof and the relationship between cross entropy and KL divergence, an interesting conclusion can be drawn:\nminp∫q(zt)KL(q(zt−1∣zt)∣p(zt−1∣zt))dzt⟺minp∬q(x)q(zt∣x)KL[q(zt−1∣zt,x)∣p(zt−1���zt)]dxdzt(6.19)\nBy comparing the expressions on the left and right, it can be observed that the objective function on the right side includes an additional variable X compared to the left side. At the same time, there is an additional integral with respect to X, with the occurrence probability of X, denoted as q(x), serving as the weighting coefficient for the integral.
\n
Following a similar proof method, a more general relationship can be derived:\nminpKL(q(z)∣p(z))⟺minp∫q(x)KL(q(z∣x)∣p(z))dx(6.20)
\n
","posterior_transform_en":"
Contraction Mapping and Converging Point
\nq(x)=∫q(x,z)dz=∫q(x∣z)q(z)dz(7.1)\n\n
Through extensive experiments with one-dimensional random variables, it was found that the Posterior Transform exhibits the characteristics of Contraction Mapping. This means that, for any two probability distributions qi1(z) and qi2(z), after posterior transform, we get qo1(x) and qo2(x). The distance between qo1(x) and qo2(x) is always less than the distance between qi1(x) and qi2(x). Here, the distance can be measured using JS divergence or Total Variance. Furthermore, the contractive ratio of this contraction mapping is positively related to the size of the added noise.\ndist(qo1(z),qo2(z))<dist(qi1(x),qi2(x))(7.2)
\n
Readers can refer to Demo 4.1, where the first three figure present a transform process. The first figure is an arbitrary data distribution q(x), the third figure is the transformed probability distribution, and second figure is the posterior probability distribution q(x∣z). You can change the random seed to generate a new data distributionq(x), and adjust the value of α to introduce different degrees of noise.
\n
The last two figures show contraction of the transform. The fourth figure displays two randomly generated input distributions and their distance, divin. The fifth figure displays the two output distributions after transform, with the distance denoted as divout.
\n
Readers can change the input random seed to toggle different inputs. It can be observed from the figures that divin is always smaller than divout for any input. Additionally, if you change the value of α, you will see that the smaller the α(larger noise), the smaller the ratio of divout/divin，indicating a larger rate of contraction.
\n
According to the Banach fixed-point theorem[5], a contraction mapping has a unique fixed point (converged point). That is to say, for any input distribution, the Posterior Transform can be applied continuously through iterations, and as long as the number of iterations is sufficient, the final output would be the same distribution. After a large number of one-dimensional random variable experiments, it was found that the fixed point (converged point) is located near q(x). Also, the location is related to the value of α; the smaller α (larger noise), the closer it is.
\n
Readers can refer to Demo 4.2, which illustrates an example of applying posterior transform iteratively. Choose an appropriate number of iterations, and click on the button of Apply, and the iteration process will be draw step by step. Each subplot shows the transformed output distribution(green curve) from each transform, with the reference distribution q(x) expressed as a blue curve, as well as the distance div between the output distribution and q(x). It can be seen that as the number of iterations increases, the output distribution becomes more and more similar to q(x), and will eventually stabilize near q(x). For more complicated distributions, more iterations or greater noise may be required. The maximum number of iterations can be set to tens of thousands, but it'll take longer.
\n
For the one-dimensional discrete case, q(x∣z) is discretized into a matrix (denoted as Qx∣z), q(z) is discretized into a vector (denoted as qi). The integration operation ∫q(x∣z)q(z)dz is discretized into a matrix-vector multiplication operation, thus the posterior transform can be written as\nqoqoqo=Qx∣zqi=Qx∣zQx∣zqi…=(Qx∣z)nqi1 iteration2 iterationn iteration(7.3)(7.4)(7.5)\nIn order to better understand the property of the transform, the matrix (Qx∣z)n is also plotted in Demo 4.2. From the demo we can see that, as the iterations converge, the row vectors of the matrix (Qx∣z)n will become a constant vector, that is, all components of the vector will be the same, which will appear as a horizontal line in the denisty plot.
\n
\n
Figure 2: Only one component in support
\n\n
The following will prove that with some conditions, the posterior transform is a contraction mapping, and there exists a unique point, which is also the converged point. The proof assumes that the random variable is discrete, so the posterior transform can be regarded as a single step transition of a discrete Markov Chain. The posterior q(x∣z) corresponds to the transfer matrix. Continuous variables can be considered as discrete variables with infinite states.
\n\n
When q(x) is greater than 0, the posterior transform matrix q(x∣z) will be greater than 0 too. Therefore, this matrix is the transition matrix of an irreducibleaperiodic Markov Chain. According to the conclusion of the literature [13], this transformation is a contraction mapping with respect to Total Variance metric. Therefore, according to the Banach fixed-point theorem, this transformation has a unique fixed point(converged point).
\n \n
When q(x) is partially greater than 0, and the support of q(x) (the region where q(x) is greater than 0) consists only one connected component (Figure 2), several conclusions can be drawn from equation (3.4)：\n\n\n
When z and x are within the support set, since both q(x) and GaussFun are greater than 0, the diagonal elements of the transfer matrix {q(x∣z)∣z=x} are greater than 0. This means that the state within the support set is aperiodic.
\n\n
When z and x are within the support set, since GaussFun's support set has a certain range, elements above and below the diagonal {q(x∣z)∣x=z+ϵ}is also greater than 0. This means that states within the support set are accessible to each other, forming a Communication Class[14], see in Figure 2b.
\n \n
When z is within the support set and x is outside the support set, q(x∣z) is entirely 0. This means that the state within the support set is inaccessible to the state outside the support set (Inaccessible Region in Figure 2b)
\n \n
When z is outside the support set and x is inside the support set, due to the existence of a certain range of the support set of GaussFun, there are some extension areas (Extension Region in Figure 2b), where the corresponding {q(x∣z)∣x∈support} is not all zero. This means that the state of this part of the extension area can unidirectionally access the state inside the support set (Unidirectional Region in Figure 2b).
\n \n
When z is outside the support set and x is outside the support set, the corresponding q(x∣z) is entirely zero. This implies that, states outside the support set will not transit to states outside the support set. In other words, states outside the support set only originate from states within the support set.
\n\n\n
\nFrom (c), we know that states within the support set will not transition to states outside of the support set. From (a) and (b), we know that the states within the support set are non-periodic and form a Communicate Class. Therefore, the states within the support set independently form an irreducible and non-periodic Markov Chain. According to the conclusion of Theorem 11.4.1 in reference [7], as n→∞, q(x∣z)n will converge to a constant matrix, with each column vector in the matrix being identical. This implies that for different values of z, q(x∣z)n are the same (as seen in Figure 2c). In Addition, according to (d) and (e), there exist some states z, which are outside of the support set, that can transition into the support set and will carry information from within the support set back to the outside. Thus, the corresponding q(x∣z)n for these z states (the q(x∣zex) region in Figure 2c) will equal the corresponding q(x∣z)n in the support set (the q(x∣zsup) region in Figure 2c).\n
\n\n
\nTherefore, it can be concluded that when the state is confined within the support set and two extension regions, limn→∞q(x∣z)n will converge to a fixed matrix, and each column vector is identical. Hence, for any input distribution, if posterior transforms are continuously applied, it will eventually converge to a fixed distribution, which is equal to the column vector of the converged matrix. Based on the conclusion from the literature [9], when a iterative transform converges to a unique fixed point, this transform is a Contraction Mapping with respect to a certain metric. \n
\n
\n\n
When q(x) is partially greater than 0, and multiple connected component exist in the support set of q(x), and the maximum distance of each connected component can be covered by the support set of corresponding GaussFun, the states within each connected domain constitute only one Communicate Class. As shown in Figure 3, q(x) has two connected component. On the edge of the first component, the support set of GaussFun corresponding to q(x∣z=−0.3) can span the gap to reach the second component, so the states of the first component can access the states of the second component. On the edge of the second component, the support set of GaussFun corresponding to q(x∣z=0) can also span the gap to reach the first. Thus, the states of the second component can access the states of the first component, so these two component form a Communicate Class. Therefore, similar to the case with a single component, when states are confined to each component, gaps, and extension areas, the posterior transform has a unique iterative convergence point, which is a contraction mapping with respect to a certain metric.
\n\n
When q(x) is partially greater than 0, and multiple connected component exist in the support set of q(x), and the maximum distance of each connected component cannot be covered by the support set of corresponding GaussFun, the states within each component constitute multiple Communicate Classes, as shown in Figure 4. Under such circumstances, as n→∞, q(x∣z)n will also converge to a fixed matrix, but not all the column vectors are identical. Therefore, the posterior transforma is not a strict contraction mapping. However, when the state of the input distribution is confined to a single Communicate Class and its corresponding extension, the posterior transform is also a contraction mapping with a unique convergence point.
\n\n\n
\n
Figure 3: Two components which can communicate with each other
\n\n
\n
Figure 4: Two components which cannot communicate with each other
\n\n
Additionally, there exists a more generalized relation about the posterior transform that is independent of q(x∣z): the Total Variance distance between two output distributions will always be less than or equal to the Total Variance distance between their corresponding input distributions, that is\ndist(qo1(x),qo2(x))<=dist(qi1(z),qi2(z))\nThe proof is given below in discrete form:\n∥qo1−qo2∥TV=∥Qx∣zqi1−Qx∣zqi2∥TV=m∑∣n∑Qx∣z(m,n)qi1(n)−n∑Qx∣z(m,n)qi2(n)∣=m∑∣n∑Qx∣z(m,n)(qi1(n)−qi2(n))∣≤m∑n∑Qx∣z(m,n)∣(qi1(n)−qi2(n))∣Absolute value inequality=n∑∣(qi1(n)−qi2(n))∣m∑Qx∣z(m,n)m∑Qx∣z(m,n)=1=n∑∣(qi1(n)−qi2(n))∣(7.6)(7.7)(7.8)(7.9)(7.10)(7.11)\nIn this context, Qx∣z(m,n) represents the element at the m-th row and n-th column of the matrix Qx∣z, and qi1(n) represents the n-th element of the vector qi1.
\n
The relationship between the converged distribution and the input distribution q(x) cannot be rigorously proven at present.
\n
Anti-noise Capacity In Restoring Data Distribution
\nFrom the above analysis, we know that when certain conditions are satisfied, the posterior transform is a contraction mapping. Therefore, the following relationship exists: \ndist(q(x),qo(x))<dist(q(z),qi(z))(7.12)\nWherein, q(z) is the ideal input distribution, q(x) is the ideal output distribution, qi(x) is any arbitrary input distribution, and qo(x) is the output distribution obtained after transforming qi(z). \n\n
The above equation indicates that the distance between the output distribution qo(x) and the ideal output distribution q(x) will always be less than the distance between the input distribution qi(z) and the ideal input distribution q(x). Hence, the posterior transform has certain resistance to noise. This means that during the process of restoring q(x)(Section 5), even if the tail distributionq(zT) contains some error, the error of the outputed distribution q(x) will be smaller than the error of input after undergoing a series of transform.
\n
Refer specifically to Demo 3.2, where by increasing the value of the noise ratio, noise can be added to the tail distributionq(zT). Clicking the \"apply\" button will gradually draw out the restoring process, with the restored distribution represented by a red curve, and the error size will be computed by the JS divergence. You will see that the error of restored q(x) is always less than the error of q(zT).
\n
From the above discussion, we know that the smaller the α (the larger the noise used in the transform process), the greater the contractive ratio of the contraction mapping, and thus, the stronger the ability to resist noise.
\n
","deconvolution_en":"
As mentioned in the Section 1, the transform of Equation 2.1 can be divided into two sub-transforms, the first one being a linear transform and the second being adding independent Gaussian noise. The linear transform is equivalent to a scaling transform of the probability distribution, so it has an inverse transformation. Adding independent Gaussian noise is equivalent to the execution of a convolution operation on the probability distribution, which can be restored through deconvolution. Therefore, theoretically, the data distribution q(x) can be recovered from the final probability distribution q(zT) through inverse linear transform and deconvolution.
\n
However, in actuality, some problems do exist. Due to the extreme sensitivity of deconvolution to errors, having high input sensitivity, even a small amount of input noise can lead to significant changes in output[11][12]. Meanwhile, in the diffusion model, the standard normal distribution is used as an approximation to replace q(zT), thus, noise is introduced at the initial stage of recovery. Although the noise is relatively small, because of the sensitivity of deconvolution, the noise will gradually amplify, affecting the recovery.
\n
In addition, the infeasibility of deconvolution restoring can be understood from another perspective. Since the process of forward transform (equations 4.1 to 4.4) is fixed, the convolution kernel is fixed. Therefore, the corresponding deconvolution transform is also fixed. Since the initial data distribution q(x) is arbitrary, any probability distribution can be transformed into an approximation of N(0,I) through a series of fixed linear transforms and convolutions. If deconvolution restoring is feasible, it means that a fixed deconvolution can be used to restore any data distribution q(x) from the N(0,I) , this is clearly paradoxical. The same input, the same transform, cannot have multiple different outputs.
APP: This Web APP is developed using Gradio and deployed on HuggingFace. Due to limited resources (2 cores, 16G memory), the response may be slow. For a better experience, it is recommended to clone the source code from github and run it locally. This program only relies on Gradio, SciPy, and Matplotlib.
\n
Author: Zhenxin Zheng, Senior computer vision engineer with ten years of algorithm development experience, Formerly employed by Tencent and JD.com, currently focusing on image and video generation.
loss=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt=−∬∫q(x)q(zt−1,zt∣x)dxlogp(zt−1∣zt)dzt−1dzt=∬∫q(x)q(zt−1,zt∣x)logq(zt−1∣zt,x)dxdzt−1dztThis Term Is Constant And Is Represented ByC1−∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)dxdzt−1dzt−C1=∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)q(zt−1∣zt,x)dxdzt−1dzt−C1=∬q(x)q(zt∣x)∫q(zt−1∣zt,x)logp(zt−1∣zt)q(zt−1∣zt,x)dzt−1dzxdzt−C1=∬q(x)q(zt∣x)KL(q(zt−1∣zt,x)∥p(zt−1∣zt))dxdzt−C1∝∬q(x)q(zt∣x)KL(q(zt−1∣zt,x)∥p(zt−1∣zt))dxdzt(6.8)(6.9)(6.10)(6.11)(6.12)(6.13)(6.14)(6.15)
The Diffusion Probability Model[1][2] is currently the main method used in image and video generation, but due to its abstruse theory, many engineers are unable to understand it well. This article will provide a very easy-to-understand method to help readers grasp the principles of the Diffusion Model. Specifically, it will illustrate the Diffusion Model using examples of one-dimensional random variables in an interactive way, explaining several interesting properties of the Diffusion Model in an intuitive manner.
\n
The diffusion model is a probabilistic model. Probabilistic models mainly offer two functions: calculating the probability of a given sample appearing; and generating new samples. The diffusion model focuses on the latter aspect, facilitating the production of new samples, thus realizing the task of generation.
\n
The diffusion model differs from general probability models (such as GMM), which directly models the probability distribution of random variables. The diffusion model adopts an indirect approach, which utilizes random variable transform(shown in Figure 1a) to gradually convert the data distribution (the probability distribution to be modeled) into the standard normal distribution, and meanwhile models the posterior probability distribution corresponding to each transformation (Figure 1b-c). Upon obtaining the final standard normal distribution and the posterior probability distributions, one can generate samples of each random variable ZT…Z2,Z1,X in reverse order through Ancestral Sampling. Simultaneously, initial data distribution q(x) can be determined by employing Bayes theorem and the total probability theorem.
\n
One might wonder: indirect methods require modeling and learning T posterior probability distributions, while direct methods only need to model one probability distribution, Why would we choose the indirect approach? Here's the reasoning: the initial data distribution might be quite complex and hard to represent directly with a probability model. In contrast, the complexity of each posterior probability distribution in indirect methods is significantly simpler, allowing it to be approximated by simple probability models. As we will see later, given certain conditions, posterior probability distributions can closely resemble Gaussian distributions, thus a simple conditional Gaussian model can be used for modeling.
\n
\n
Figure 1: Diffusion probability model schematic
","transform_en":"
To transform the initial data distribution into a simple standard normal distribution, the diffusion model uses the following transformation method: \nZ=αX+1−αϵwhereα<1,ϵ∼N(0,I)(1.1)\nwhere X∼q(x)is any random variable，Z∼q(Z) is the transformed random variable。
\n
This transformation can be divided into two sub-transformations。
\n
The first sub-transformation performs a linear transformation (αX) on the random variable X. According to the conclusion of the literature[3], the linear transformation makes the probability distribution of Xnarrower and taller, and the extent of narrowing and heightening is directly proportional to the value of α.
\n
This can be specifically seen in Demo 1, where the first figure depicts a randomly generated one-dimensional probability distribution, and the second figure represents the probability distribution after the linear transformation. It can be observed that the curve of the third figure has become narrower and taller compared to the first image. Readers can experiment with different α to gain a more intuitive understanding.
\n
The second sub-transformation is adding independent random noise(1−αϵ). According to the conclusion of the literature[4], adding independent random variables is equivalent to performing convolution on the two probability distributions. Since the probability distribution of random noise is Gaussian, it is equivalent to performing a Gaussian Blur operation. After blurring, the original probability distribution will become smoother and more similar to the standard normal distribution. The degree of blurring is directly proportional to the noise level (1−α).
\n
For specifics, one can see Demo 1, where the first figure is a randomly generated one-dimensional probability distribution, and the third figure is the result after the transformation. It can be seen that the transformed probability distribution curve is smoother and there are fewer corners. The readers can test different α values to feel how the noise level affect the shape of the probability distribution. The last figure is the result after applying all two sub-transformations.
\n
","likelihood_en":"
From the transformation method (equation 1.1), it can be seen that the probability distribution of the forward conditional probability q(z∣x) is a Gaussian distribution, which is only related to the value of α, regardless of the probability distribution of q(x). \nq(z∣x)=N(αx,1−α)(2.1)\nIt can be understood by concrete examples in Demo 2. The third figure depict the shape of q(z∣x). From the figure, a uniform slanting line can be observed. This implies that the mean of q(z∣x) is linearly related to x, and the variance is fixed. The magnitude of α will determine the width and incline of the slanting line.
\n
","posterior_en":"
The posterior probability distribution does not have a closed form, but its shape can be inferred approximately through some technique.
\n
According to Bayes formula, we have\nq(x∣z)=q(z)q(z∣x)q(x)(3.1)
\n
When z takes a fixed value, q(z) is a constant, so the shape of q(x∣z) is only related to q(z∣x)q(x).\nq(x∣z)∝q(z∣x)q(x)wherezisfixed(3.2)
\n
From Equation 2.1, we can see that q(z∣x) is a Gaussian distribution, so we have\nq(x∣z)∝2π(1−α)1exp2(1−α)−(z−αx)2q(x)=α1GaussFun2πσ1exp2σ2−(x−μ)2q(x)wherezisfixedwhereμ=αzσ=α1−α(3.3)(3.4)
\n
It can be observed that the GaussFun part is a Gaussian function of x, with a mean of αz and a variance of α1−α, so the shape of q(x∣z) is determined by the product of GaussFun and q(x).
\n
According to the characteristics of multiplication, the characteristics of the shape of the q(x∣z) function can be summarized.
\n
\n
When the variance of the Gaussian function is small (small noise), or when q(x) changes slowly, the shape of q(x∣z) will approximate to the Gaussian function, and have a simpler function form, which is convenient for modeling and learning.
\n \n
When the variance of the Gaussian function is large (large noise), or when q(x) changes drastically, the shape of q(x∣z) will be more complex, and greatly differ from a Gaussian function, which makes it difficult to model and learn.
\n
\n\n
The specifics can be seen in Demo 2. The fourth figure present the shape of the posterior q(x∣z), which shows an irregular shape and resembles a curved and uneven line. As α increases (noise decreases), the curve tends to be uniform and straight. Readers can adjust different α values and observe the relationship between the shape of posterior and the level of noise. In the last figure, the blue dash line represents q(x), the green dash line represents GaussFun in the equation 3.4, and the orange curve represents the result of multiplying the two function and normalizing it, which is the posterior probability q(x∣z=fixed) under a fixed z condition. Readers can adjust different values of z to observe how the fluctuation of q(x) affect the shape of the posterior probability q(x∣z).
\n
The posterior q(x∣z) under two special states are worth considering.
\n
\n
As α→0, the variance of GaussFun tends to ∞, and q(x∣z) for different z almost become identical, and almost the same as q(x). Readers can set α to 0.001 in Demo 2 to observe the specific results.
\n \n
As α→1, the variance of GaussFun tends to 0, The q(x∣z) for different z values contract into a series of Dirac delta functions with different offsets equalling to z. However, there are some exceptions. When there are regions where q(x) is zero, the corresponding q(x∣z) will no longer be a Dirac delta function, but a zero function. Readers can set α to 0.999 in Demo 2 to observe the specific results.
\n
","forward_process_en":"
For any arbitrary data distribution q(x), the transform(equation 2.1) in section 2 can be continuously applied(equation 4.1~4.4). As the number of transforms increases, the output probability distribution will become increasingly closer to the standard normal distribution. For more complex data distributions, more iterations or larger noise are needed.
\n
Specific details can be observed in Demo 3.1. The first figure illustrates a randomly generated one-dimensional probability distribution. After seven transforms, this distribution looks very similar to the standard normal distribution. The degree of similarity increases with the number of iterations and the level of the noise. Given the same degree of similarity, fewer transforms are needed if the noise added at each step is larger (smaller α value). Readers can try different α values and numbers of transforms to see how similar the final probability distribution is.
\n
The complexity of the initial probability distribution tends to be high, but as the number of transforms increases, the complexity of the probability distribution q(zt) will decrease. As concluded in section 4, a more complex probability distribution corresponds to a more complex posterior probability distribution. Therefore, in order to ensure that the posterior probability distribution is more similar to the Conditional Gaussian function (easier to learn), a larger value of α (smaller noise) should be used in the initial phase, and a smaller value of α (larger noise) can be appropriately used in the later phase to accelerate the transition to the standard normal distribution.
\n
In the example of Demo 3.1, it can be seen that as the number of transforms increases, the corners of q(zt) become fewer and fewer. Meanwhile, the slanting lines in the plot of the posterior probability distribution q(zt−1∣zt) become increasingly straight and uniform, resembling more and more the conditional Gaussian distribution.
By substituting Equation 4.1 into Equation 4.2, and utilizing the properties of Gaussian distribution, we can derive the form of q(z2∣x) \nz2=α2(α1x+1−α1ϵ1)+1−α2ϵ2=α2α1x+α2−α2α1ϵ1+1−α2ϵ2=N(α1α2x,1−α1α2)(4.5)(4.6)(4.7)
\n
In the same way, it can be deduced recursively that\nq(zt∣x)=N(α1α2⋯αtx,1−α1α2⋯αt)=N(αtˉx,1−αtˉ)whereαtˉ≜j=1∏tαj(4.8)
\n
Comparing the forms of Equation 4.8 and Equation 2.1, it can be found that their forms are completely consistent. If only focusing on the final transformed distribution q(zt), then the t consective small transformations can be replaced by one large transformation. The α of the large transformation is the accumulation of the α from each small transformation.
\n
In the DDPM[2] paper, the authors used 1000 steps (T=1000) to transform the data distribution q(x) to q(zT). The probability distribution of q(zT∣x) is as follows:\nq(zT∣x)=N(0.00635x,0.99998)(4.9)
\n
If considering only marginal distribution q(zT), a single transformation can also be used, which is as follows:\nZT=0.0000403X+1−0.0000403ϵ=0.00635X+0.99998ϵ(4.10)\nIt can be seen that, after applying two transforms, the transformed distributions q(zT∣x) are the same. Thus, q(zT) is also the same.
\n
","backward_process_en":"
If the final probability distribution q(zT) and the posterior probabilities of each transform q(x∣z),q(zt−1∣zt) are known, the data distribution q(x) can be recovered through the Bayes Theorem and the Law of Total Probability, as shown in equations 5.1~5.4. When the final probability distribution q(zT) is very similar to the standard normal distribution, the standard normal distribution can be used as a substitute.
\n
Specifics can be seen in Demo 3.2. In the example, q(zT) substitutes N(0,1), and the error magnitude is given through JS Divergence. The restored probability distribution q(zt) and q(x) are identified by the green curve, and the original probability distribution is identified by the blue curve. It can be observed that the data distribution q(x) can be well restored, and the error (JS Divergence) will be smaller than the error caused by the standard normal distribution replacing q(zT).\nq(zT−1)q(zt−1)q(z1)q(x)=∫q(zT−1,zT)dzT=∫q(zT−1∣zT)q(zT)dzT…=∫q(zt−1,zt)dzt=∫q(zt−1∣zt)q(zt)dzt…=∫q(z1,z2)dz1=∫q(z1∣z2)q(z2)dz2=∫q(x,z1)dz1=∫q(x∣z1)q(z1)dz1(5.1)(5.2)(5.3)(5.4)\nIn this article, the aforementioned transform is referred to as the Posterior Transform. For example, in equation 5.4, the input of the transform is the probability distribution function q(z1), and the output is the probability distribution function q(x).The entire transform is determined by the posterior q(x∣z1). This transform can also be considered as the linear weighted sum of a set of basis functions, where the basis functions are q(x∣z1) under different z1, and the weights of each basis function are q(z1). Some interesting properties of this transform will be introduced in Section 7.
\n
In Section 3, we have considered two special posterior probability distributions. Next, we analyze their corresponding posterior transforms.
\n
\n
When α→0, the q(x∣z) for different z are almost the same as q(x). In other words, the basis functions of linear weighted sum are almost the same. In this state, no matter how the input changes, the output of the transformation is always q(x).
\n
When α→1, the q(x∣z) for different z values becomes a series of Dirac delta functions and zero functions. In this state, as long as the support set of the input distribution is included in the support set of q(x), the output of the transformation will remain the same with the input.
\n
\n\n
In Section 4, it is mentioned that the 1000 transformations used in the DDPM[2] can be represented using a single transformation\nZT=0.0000403X+1−0.0000403ϵ=0.00635X+0.99998ϵ(5.5)
\n
Since α=0.0000403 is very small, the corresponding standard deviation of GaussFun (Equation 3.4) reaches 157.52. However, the range of X is limited within [−1,1], which is far smaller than the standard deviation of GaussFun. Within the range of x∈[−1,1], GaussFun should be close to a constant, showing little variation. Therefore, the q(x∣zT) corresponding to different zT are almost the same as q(x). In this state, the posterior transform corresponding to q(x∣zT) does not depend on the input distribution, the output distribution will always be q(x).
\n
Therefore, theoretically, in the DDPM model, it is not necessary to use the standard normal distribution to replace q(zT). Any other arbitrary distributions can also be used as a substitute.
\n
Readers can conduct a similar experiment themselves. In Demo 3.1, set start_alpha to 0.25, end_alpha to 0.25, and step to 7. At this point, q(z7)=0.000061X+1−0.000061ϵ, which is roughly equivalent to DDPM's q(zT). Click on apply to perform the forward transform (plotted using blue curves), which prepares for the subsequent restoring process. In Demo 3.2, set the noise_ratio to 1, introducing 100% noise into the tail distributionq(z7). Changing the value of nose_random_seed will change the distribution of noise. Deselect backward_pdf to reduce screen clutter. Click on apply to restore q(x) through posterior transform. You will see that, no matter what the shape of input q(z7) may be, the restored q(x) is always exactly the same as the original q(x). The JS Divergence is zero. The restoration process is plotted using a red curve.
\n
","fit_posterior_en":"
From the front part of Section 3, it is known that the posterior probability distributions are unknown and related to q(x). Therefore, in order to recover the data distribution or sample from it, it is necessary to learn and estimate each posterior probability distribution.
\n
From the latter part of Section 3, it can be understood that when certain conditions are met, each posterior probability distribution q(x∣z),q(zt−1∣zt) approximates the Gaussian probability distribution. Therefore, by constructing a set of conditional Gaussian probability models p(x∣z),p(zt−1∣zt), we can learn to fit the corresponding q(x∣z),q(zt−1∣zt).
\n
Due to the limitations of the model's representative and learning capabilities, there will be certain errors in the fitting process, which will further impact the accuracy of restored q(x). The size of the fitting error is related to the complexity of the posterior probability distribution. As can be seen from Section 3, when q(x) is more complex or the added noise is large, the posterior probability distribution will be more complex, and it will differ greatly from the Gaussian distribution, thus leading to fitting errors and further affecting the restoration of q(x).
\n
Refer to Demo 3.3 for the specifics. The reader can test different q(x) and α, observe the fitting degree of the posterior probability distribution q(zt−1∣zt) and the accuracy of restored q(x). The restored probability distribution is ploted with orange, and the error is also measured by JS divergence.
\n
Regarding the objective function for fitting, similar to other probability models, the cross-entropy loss can be optimized to make p(zt−1∣zt) approaching q(zt−1∣zt). Since (zt−1∣zt) is a conditional probability, it is necessary to fully consider all conditions. This can be achieved by averaging the cross-entropy corresponding to each condition weighted by the probability of each condition happening. The final form of the loss function is as follows.\nloss=−∫q(zt)∫q(zt−1∣zt)logp(zt−1∣zt)dzt−1Cross Entropydzt=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt(6.1)(6.2)
\n
KL divergence can also be optimized as the objective function. KL divergence and cross-entropy are equivalent[10]\nloss=∫q(zt)KL(q(zt−1∣zt)∥p(zt−1∣zt))dzt=∫q(zt)∫q(zt−1∣zt)p(zt−1∣zt)q(zt−1∣zt)dzt−1dzt=−∫q(zt)Cross Entropy∫q(zt−1∣zt)logp(zt−1∣zt)dzt−1dzt+Is Constant∫q(zt)∫q(zt−1∣zt)logq(zt−1∣zt)dz(6.3)(6.4)(6.5)
\n
The integral in equation 6.2 does not have a closed form and cannot be directly optimized. The Monte Carlo integration can be used for approximate calculation. The new objective function is as follows:\nloss=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt≈−i=0∑Nlogp(Zt−1i∣Zti)where(Zt−1i,Zti)∼q(zt−1,zt)(6.6)(6.7)
\n
The aforementioned samples (Zt−1i,Zti) follow a joint probability distribution q(zt−1,zt), which can be sampled via an Ancestral Sampling. The specific method is as follows: sample X,Z1,Z2…Zt−1,Zt step by step through forward transforms (Formulas 4.1~4.4), and then reserve (Zt−1,Zt) as a sample. This sampling process is relatively slow. To speed up the sampling, we can take advantage of the known features of the probability distribution q(zt∣x) (Formula 4.8). First, sample X from q(x), then sample Zt−1 from q(zt−1∣x), and finally sample Zt from q(zt∣zt−1). Thus, a sample (Zt−1,Zt) is obtained.
\n
Some people may question that the objective function in Equation 6.3 seems different from those in the DPM[1] and DDPM[2] papers. In fact, these two objective functions are equivalent, and the proof is given below.
\n
For Consistent Terms, the proof is as follows:
\n
loss=−∬q(zt−1,zt)logp(zt−1∣zt)dzt−1dzt=−∬∫q(x)q(zt−1,zt∣x)dxlogp(zt−1∣zt)dzt−1dzt=∬∫q(x)q(zt−1,zt∣x)logq(zt−1∣zt,x)dxdzt−1dztThis Term Is Constant And Is Represented ByC1−∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)dxdzt−1dzt−C1=∬∫q(x)q(zt−1,zt∣x)logp(zt−1∣zt)q(zt−1∣zt,x)dxdzt−1dzt−C1=∬q(x)q(zt∣x)∫q(zt−1∣zt,x)logp(zt−1∣zt)q(zt−1∣zt,x)dzt−1dzxdzt−C1=∬q(x)q(zt∣x)KL[q(zt−1∣zt,x)∥p(zt−1∣zt)]dxdzt−C1∝∬q(x)q(zt∣x)KL(q(zt−1∣zt,x)∥p(zt−1∣zt))dxdzt(6.8)(6.9)(6.10)(6.11)(6.12)(6.13)(6.14)(6.15)
\n
In the above formula, the term C1 is a fixed value, which does not contain parameters to be optimized. Here, q(x) is a fixed probability distribution, and q(zt−1∣zt) is also a fixed probability distribution, whose specific form is determined by q(x) and the coefficient α.
\n
For the Reconstruction Term, it can be proven in a similar way.
Therefore, the objective function in equation 6.1 is equivalent with the DPM original objective function.
\n
Based on the conclusion of the Consistent Terms proof and the relationship between cross entropy and KL divergence, an interesting conclusion can be drawn:\nminp∫q(zt)KL(q(zt−1∣zt)∥p(zt−1∣zt))dzt⟺minp∬q(x)q(zt∣x)KL(q(zt−1∣zt,x)∥p(zt−1∣zt))dxdzt(6.19)\nBy comparing the expressions on the left and right, it can be observed that the objective function on the right side includes an additional variable X compared to the left side. At the same time, there is an additional integral with respect to X, with the occurrence probability of X, denoted as q(x), serving as the weighting coefficient for the integral.
\n
Following a similar proof method, a more general relationship can be derived:\nminpKL(q(z)∥p(z))⟺minp∫q(x)KL(q(z∣x)∥p(z))dx(6.20)\nA detailed derivation of this conclusion can be found in Appendix A.
\n
","posterior_transform_en":"
Contraction Mapping and Converging Point
\nq(x)=∫q(x,z)dz=∫q(x∣z)q(z)dz(7.1)\n\n
Through extensive experiments with one-dimensional random variables, it was found that the Posterior Transform exhibits the characteristics of Contraction Mapping. This means that, for any two probability distributions qi1(z) and qi2(z), after posterior transform, we get qo1(x) and qo2(x). The distance between qo1(x) and qo2(x) is always less than the distance between qi1(x) and qi2(x). Here, the distance can be measured using JS divergence or Total Variance. Furthermore, the contractive ratio of this contraction mapping is positively related to the size of the added noise.\ndist(qo1(z),qo2(z))<dist(qi1(x),qi2(x))(7.2)
\n
Readers can refer to Demo 4.1, where the first three figure present a transform process. The first figure is an arbitrary data distribution q(x), the third figure is the transformed probability distribution, and second figure is the posterior probability distribution q(x∣z). You can change the random seed to generate a new data distributionq(x), and adjust the value of α to introduce different degrees of noise.
\n
The last two figures show contraction of the transform. The fourth figure displays two randomly generated input distributions and their distance, divin. The fifth figure displays the two output distributions after transform, with the distance denoted as divout.
\n
Readers can change the input random seed to toggle different inputs. It can be observed from the figures that divin is always smaller than divout for any input. Additionally, if you change the value of α, you will see that the smaller the α(larger noise), the smaller the ratio of divout/divin，indicating a larger rate of contraction.
\n
According to the Banach fixed-point theorem[5], a contraction mapping has a unique fixed point (converged point). That is to say, for any input distribution, the Posterior Transform can be applied continuously through iterations, and as long as the number of iterations is sufficient, the final output would be the same distribution. After a large number of one-dimensional random variable experiments, it was found that the fixed point (converged point) is located near q(x). Also, the location is related to the value of α; the smaller α (larger noise), the closer it is.
\n
Readers can refer to Demo 4.2, which illustrates an example of applying posterior transform iteratively. Choose an appropriate number of iterations, and click on the button of Apply, and the iteration process will be draw step by step. Each subplot shows the transformed output distribution(green curve) from each transform, with the reference distribution q(x) expressed as a blue curve, as well as the distance div between the output distribution and q(x). It can be seen that as the number of iterations increases, the output distribution becomes more and more similar to q(x), and will eventually stabilize near q(x). For more complicated distributions, more iterations or greater noise may be required. The maximum number of iterations can be set to tens of thousands, but it'll take longer.
\n
For the one-dimensional discrete case, q(x∣z) is discretized into a matrix (denoted as Qx∣z), q(z) is discretized into a vector (denoted as qi). The integration operation ∫q(x∣z)q(z)dz is discretized into a matrix-vector multiplication operation, thus the posterior transform can be written as\nqoqoqo=Qx∣zqi=Qx∣zQx∣zqi…=(Qx∣z)nqi1 iteration2 iterationn iteration(7.3)(7.4)(7.5)\nIn order to better understand the property of the transform, the matrix (Qx∣z)n is also plotted in Demo 4.2. From the demo we can see that, as the iterations converge, the row vectors of the matrix (Qx∣z)n will become a constant vector, that is, all components of the vector will be the same, which will appear as a horizontal line in the denisty plot.
\n
In the Appendix B, a proof will be provided that, when q(x) and α satisfy some conditions, the posterior transform is a strict Contraction Mapping.
\n
The relationship between the converged distribution and the input distribution q(x) cannot be rigorously proven at present.
\n
Anti-noise Capacity In Restoring Data Distribution
\nFrom the above analysis, we know that when certain conditions are satisfied, the posterior transform is a contraction mapping. Therefore, the following relationship exists: \ndist(q(x),qo(x))<dist(q(z),qi(z))(7.12)\nWherein, q(z) is the ideal input distribution, q(x) is the ideal output distribution, qi(x) is any arbitrary input distribution, and qo(x) is the output distribution obtained after transforming qi(z). \n\n
The above equation indicates that the distance between the output distribution qo(x) and the ideal output distribution q(x) will always be less than the distance between the input distribution qi(z) and the ideal input distribution q(x). Hence, the posterior transform has certain resistance to noise. This means that during the process of restoring q(x)(Section 5), even if the tail distributionq(zT) contains some error, the error of the outputed distribution q(x) will be smaller than the error of input after undergoing a series of transform.
\n
Refer specifically to Demo 3.2, where by increasing the value of the noise ratio, noise can be added to the tail distributionq(zT). Clicking the \"apply\" button will gradually draw out the restoring process, with the restored distribution represented by a red curve, and the error size will be computed by the JS divergence. You will see that the error of restored q(x) is always less than the error of q(zT).
\n
From the above discussion, we know that the smaller the α (the larger the noise used in the transform process), the greater the contractive ratio of the contraction mapping, and thus, the stronger the ability to resist noise.
\n
","deconvolution_en":"
As mentioned in the Section 1, the transform of Equation 2.1 can be divided into two sub-transforms, the first one being a linear transform and the second being adding independent Gaussian noise. The linear transform is equivalent to a scaling transform of the probability distribution, so it has an inverse transformation. Adding independent Gaussian noise is equivalent to the execution of a convolution operation on the probability distribution, which can be restored through deconvolution. Therefore, theoretically, the data distribution q(x) can be recovered from the final probability distribution q(zT) through inverse linear transform and deconvolution.
\n
However, in actuality, some problems do exist. Due to the extreme sensitivity of deconvolution to errors, having high input sensitivity, even a small amount of input noise can lead to significant changes in output[11][12]. Meanwhile, in the diffusion model, the standard normal distribution is used as an approximation to replace q(zT), thus, noise is introduced at the initial stage of recovery. Although the noise is relatively small, because of the sensitivity of deconvolution, the noise will gradually amplify, affecting the recovery.
\n
In addition, the infeasibility of deconvolution restoring can be understood from another perspective. Since the process of forward transform (equations 4.1 to 4.4) is fixed, the convolution kernel is fixed. Therefore, the corresponding deconvolution transform is also fixed. Since the initial data distribution q(x) is arbitrary, any probability distribution can be transformed into an approximation of N(0,I) through a series of fixed linear transforms and convolutions. If deconvolution restoring is feasible, it means that a fixed deconvolution can be used to restore any data distribution q(x) from the N(0,I) , this is clearly paradoxical. The same input, the same transform, cannot have multiple different outputs.
APP: This Web APP is developed using Gradio and deployed on HuggingFace. Due to limited resources (2 cores, 16G memory), the response may be slow. For a better experience, it is recommended to clone the source code from github and run it locally. This program only relies on Gradio, SciPy, and Matplotlib.
\n
Author: Zhenxin Zheng, Senior computer vision engineer with ten years of algorithm development experience, Formerly employed by Tencent and JD.com, currently focusing on image and video generation.
Figure 3: Two component which can communicate with each other
\n\n
\n
Figure 4: Two component which cannot communicate with each other
\n\n
另外，后验概率变换存在一个更通用的关系，与q(x∣z)的具体值无关: 两个输出分布的之间的Total Variance距离总是会小于等于对应输入分布之间的Total Variance距离，即\ndist(qo1(x),qo2(x))≤dist(qi1(z),qi2(z))(B.1)\n下面通过离散的形式给出证明：\n∥qo1−qo2∥TV=∥Qx∣zqi1−Qx∣zqi2∥TV=m∑∣n∑Qx∣z(m,n)qi1(n)−n∑Qx∣z(m,n)qi2(n)∣=m∑∣n∑Qx∣z(m,n)(qi1(n)−qi2(n))∣≤m∑n∑Qx∣z(m,n)∣(qi1(n)−qi2(n))∣Absolute value inequality=n∑∣(qi1(n)−qi2(n))∣m∑Qx∣z(m,n)m∑Qx∣z(m,n)=1=n∑∣(qi1(n)−qi2(n))∣(B.2)(B.3)(B.4)(B.5)(B.6)(B.7)\n其中，Qx∣z(m,n)表示矩阵Qx∣z的第m行第n列的元素，qi1(n)表示向量qi1的第n个元素。
\n
","cond_kl_en":"
This section mainly introduces the relationship between KL divergence and conditional KL divergence. Before the formal introduction, we will briefly introduce the definitions of entropy and conditional entropy, as well as the inequality relationship between them, in preparation for the subsequent proof
\n
Entropy and Conditional Entropy
\nFor any two random variables Z,X, the Entropy is defined as follows[16]：\nH(Z)=∫−p(z)logp(z)dz(A.1)\nThe Conditional Entropy is defined as followed [17]：\nH(Z∣X)=∫p(x)∫−p(z∣x)logp(z∣x)dzEntropydx(A.2)\nThe following inequality relationship exists between the two：\nH(Z∣X)≤H(Z)(A.3)\nIt is to say that the Conditional Entropy is always less than or equal to the Entropy, and they are equal only when X and Z are independent. The proof of this relationship can be found in the literature [17].\n\n
KL Divergence and Conditional KL Divergence
\nIn the same manner as the definition of Conditional Entropy, we introduce a new definition, Conditional KL Divergence, denoted as KLmathcalC. Since KL Divergence is non-symmetric, there exist two forms as follows. \nKLC(q(z∣x)∥p(z))=∫q(x)KL(q(z∣x)∥p(z))dxKLC(q(z)∥p(z∣x))=∫p(x)KL(q(z)∥p(z∣x))dx(A.4)(A.5)\n\n
Similar to Conditional Entropy, there also exists a similar inequality relationship for Conditional KL Divergence:\nKLC(q(z∣x)∥p(z))≥KL(q(z)∥p(z))KLC(q(z)∥p(z∣x))≥KL(q(z)∥p(z))(A.6)(A.7)\nIt is to say that the Conditional KL Divergence is always less than or equal to the KL Divergence, and they are equal only when X and Z are independent.
\n
The following provides proofs for the conclusions on Equation A.5 and Equation A.6 respectively.
\n
For equation A.6, the proof is as follows:\nKLC(q(z∣x)∥p(z))=∫q(x)KL(q(z∣x)∥p(z))dx=∬q(x)q(z∣x)logp(z)q(z∣x)dzdx=−∬−q(x)q(z∣x)logq(z∣x)dzdxCondtional Entropy Hq(Z∣X)−∬q(x)q(z∣x)logp(z)dzdx=−Hq(Z∣X)−∫{∫q(x)q(z∣x)dx}logp(z)dz=−Hq(Z∣X)+∫−q(z)logp(z)dzCross Entropy=−Hq(Z∣X)+∫q(z){logp(z)q(z)−logq(z)}dz=−Hq(Z∣X)+∫q(z)logp(z)q(z)dz+∫−q(z)logq(z)dzEntropy Hq(Z)=KL(q(z)∥p(z))+Hq(Z)−Hq(Z∣X)≥0≤KL(q(z)∥p(z))(A.8)(A.9)(A.10)(A.11)(A.12)(A.13)(A.14)(A.15)(A.16)\nIn this context, equation A.15 applies the conclusion that Conditional Entropy is always less than or equal to Entropy. Thus, the relationship in equation A.6 is derived.
\n
For equation A.6, the proof is as follows:\nKL(q(z)∥p(z))=∫q(z)logp(z)q(z)dx=∫q(z)log∫p(z∣x)p(z)dzq(z)dz=∫p(x)dx∫q(z)logq(z)dz−∫q(z)log∫p(z∣x)p(x)dxdz∫p(x)dx=1≤∬p(x)q(z)logq(z)dzdx−∫q(z)∫p(x)logp(z∣x)dxdzjensen inequality=∬p(x)q(z)logq(z)dzdx−∬p(z)q(z)logp(z∣x)dzdx=∬p(x)q(z)(logq(z)−logp(z∣x))dzdx=∬p(x)q(z)logp(z∣x)q(z)dzdx=∫p(x){∫q(z)logp(z∣x)q(z)dz}dx=∫p(x)KL(q(z)∥p(z∣x))dx=KLC(q(z)∥p(z∣x))(A.15)(A.16)(A.17)(A.18)(A.19)(A.20)(A.21)(A.22)(A.23)(A.24)\nThus, the relationship in equation A.7 is obtained.
\n
Another important conclusion can be drawn from equation A.15.
\n
The KL Divergence is often used to fit the distribution of data. In this scenario, the distribution of the data is denoted by q(z) and the parameterized model distribution is denoted by pθ(z). During the optimization process, since both q(z∣x) and q(x) remain constant, the term H(Z)−H(Z∣X) in Equation A.15 is a constant. Thus, the following relationship is obtained:\nminpθKL(q(z)∥pθ(z))⟺minpθ∫q(x)KL(q(z∣x)∥pθ(z))dx(A.25)
\n
Comparing the above relationship with Denoised Score Matching[18](equation A.26), some similarities can be observed. Both introduce a new variable X, and substitute the targeted fitting distribution q(z) with q(z|x). After the substitution, since q(z|x) is a conditional probability distribution, both consider all conditions and perform a weighted sum using the probability of the conditions occurring, q(x), as the weight coefficient.\nminψθ21∫q(z)∥∥ψθ(z)−∂z∂q(z)∥∥2dz⟺minψθ∫q(x)21∫q(z∣x)∥∥ψθ(z)−∂z∂q(z∣x)∥∥2dzScore Matching of q(z∣x)dx(A.26)
\n
The operation of the above weighted sum is somewhat similar to Elimination by Total Probability Formula .\nq(z)=∫q(z,x)dx=∫q(x)q(z∣x)dx(A.27)
\n
","proof_ctr_en":"
\n
Figure 2: Only one component in support
\n\n
The following will prove that with some conditions, the posterior transform is a contraction mapping, and there exists a unique point, which is also the converged point.
\n
The proof will be divided into several cases, and assumes that the random variable is discrete, so the posterior transform can be regarded as a single step transition of a discrete Markov Chain. The posterior q(x∣z) corresponds to the transfer matrix. Continuous variables can be considered as discrete variables with infinite states.
\n\n
When q(x) is greater than 0, the posterior transform matrix q(x∣z) will be greater than 0 too. Therefore, this matrix is the transition matrix of an irreducibleaperiodic Markov Chain. According to the conclusion of the literature [13], this transformation is a contraction mapping with respect to Total Variance metric. Therefore, according to the Banach fixed-point theorem, this transformation has a unique fixed point(converged point).
\n \n
When q(x) is partially greater than 0, and the support of q(x) (the region where q(x) is greater than 0) consists only one connected component (Figure 2), several conclusions can be drawn from equation (3.4)：\n\n\n
When z and x are within the support set, since both q(x) and GaussFun are greater than 0, the diagonal elements of the transfer matrix {q(x∣z)∣z=x} are greater than 0. This means that the state within the support set is aperiodic.
\n\n
When z and x are within the support set, since GaussFun's support set has a certain range, elements above and below the diagonal {q(x∣z)∣x=z+ϵ}is also greater than 0. This means that states within the support set are accessible to each other, forming a Communication Class[14], see in Figure 2b.
\n \n
When z is within the support set and x is outside the support set, q(x∣z) is entirely 0. This means that the state within the support set is inaccessible to the state outside the support set (Inaccessible Region in Figure 2b)
\n \n
When z is outside the support set and x is inside the support set, due to the existence of a certain range of the support set of GaussFun, there are some extension areas (Extension Region in Figure 2b), where the corresponding {q(x∣z)∣x∈support} is not all zero. This means that the state of this part of the extension area can unidirectionally access the state inside the support set (Unidirectional Region in Figure 2b).
\n \n
When z is outside the support set and x is outside the support set, the corresponding q(x∣z) is entirely zero. This implies that, states outside the support set will not transit to states outside the support set. In other words, states outside the support set only originate from states within the support set.
\n\n\n
\nFrom (c), we know that states within the support set will not transition to states outside of the support set. From (a) and (b), we know that the states within the support set are non-periodic and form a Communicate Class. Therefore, the states within the support set independently form an irreducible and non-periodic Markov Chain. According to the conclusion of Theorem 11.4.1 in reference [7], as n→∞, q(x∣z)n will converge to a constant matrix, with each column vector in the matrix being identical. This implies that for different values of z, q(x∣z)n are the same (as seen in Figure 2c). In Addition, according to (d) and (e), there exist some states z, which are outside of the support set, that can transition into the support set and will carry information from within the support set back to the outside. Thus, the corresponding q(x∣z)n for these z states (the q(x∣zex) region in Figure 2c) will equal the corresponding q(x∣z)n in the support set (the q(x∣zsup) region in Figure 2c).\n
\n\n
\nTherefore, it can be concluded that when the state is confined within the support set and two extension regions, limn→∞q(x∣z)n will converge to a fixed matrix, and each column vector is identical. Hence, for any input distribution, if posterior transforms are continuously applied, it will eventually converge to a fixed distribution, which is equal to the column vector of the converged matrix. Based on the conclusion from the literature [9], when a iterative transform converges to a unique fixed point, this transform is a Contraction Mapping with respect to a certain metric. \n
\n
\n\n
When q(x) is partially greater than 0, and multiple connected component exist in the support set of q(x), and the maximum distance of each connected component can be covered by the support set of corresponding GaussFun, the states within each connected domain constitute only one Communicate Class. As shown in Figure 3, q(x) has two connected component. On the edge of the first component, the support set of GaussFun corresponding to q(x∣z=−0.3) can span the gap to reach the second component, so the states of the first component can access the states of the second component. On the edge of the second component, the support set of GaussFun corresponding to q(x∣z=0) can also span the gap to reach the first. Thus, the states of the second component can access the states of the first component, so these two component form a Communicate Class. Therefore, similar to the case with a single component, when states are confined to each component, gaps, and extension areas, the posterior transform has a unique iterative convergence point, which is a contraction mapping with respect to a certain metric.
\n\n
When q(x) is partially greater than 0, and multiple connected component exist in the support set of q(x), and the maximum distance of each connected component cannot be covered by the support set of corresponding GaussFun, the states within each component constitute multiple Communicate Classes, as shown in Figure 4. Under such circumstances, as n→∞, q(x∣z)n will also converge to a fixed matrix, but not all the column vectors are identical. Therefore, the posterior transforma is not a strict contraction mapping. However, when the state of the input distribution is confined to a single Communicate Class and its corresponding extension, the posterior transform is also a contraction mapping with a unique convergence point.
\n\n\n
\n
Figure 3: Two components which can communicate with each other
\n\n
\n
Figure 4: Two components which cannot communicate with each other
\n\n
Additionally, there exists a more generalized relation about the posterior transform that is independent of q(x∣z): the Total Variance distance between two output distributions will always be less than or equal to the Total Variance distance between their corresponding input distributions, that is\ndist(qo1(x),qo2(x))<=dist(qi1(z),qi2(z))(B.1)\nThe proof is given below in discrete form:\n∥qo1−qo2∥TV=∥Qx∣zqi1−Qx∣zqi2∥TV=m∑∣n∑Qx∣z(m,n)qi1(n)−n∑Qx∣z(m,n)qi2(n)∣=m∑∣n∑Qx∣z(m,n)(qi1(n)−qi2(n))∣≤m∑n∑Qx∣z(m,n)∣(qi1(n)−qi2(n))∣Absolute value inequality=n∑∣(qi1(n)−qi2(n))∣m∑Qx∣z(m,n)m∑Qx∣z(m,n)=1=n∑∣(qi1(n)−qi2(n))∣(B.2)(B.3)(B.4)(B.5)(B.6)(B.7)\nIn this context, Qx∣z(m,n) represents the element at the m-th row and n-th column of the matrix Qx∣z, and qi1(n) represents the n-th element of the vector qi1.\n
![](\"file/fig1.png\")
![](\"file/fig2.png\")
![](\"file/fig3.png\")
![](\"file/fig4.png\")