Note More Theoretical Reports: https://arxiv.org/pdf/2303.08797 Noise Scheduler Research: https://arxiv.org/pdf/2301.10972

Note 1. Introduce v_pred. As for DDPM noise scheduler 1.1 definition: v = \sqrt{\bar{\alpha_t}} \epsilon - \sqrt{1-\bar{\alpha_t}} x_0 1.2 The conversion btw epsilon pred and velocity pred: \epsilon_{pred} = \sqrt{\bar{\alpha_t}} v_{pred} + \sqrt{1-\bar{\alpha_t}} x_t

Note 1. use (v-prediction, epsilon loss) the loss. v_pred = uvit ( z_t , logsnr_t ) eps_pred = sigma_t * z_t + alpha_t * v_t

Note 1. Following the U-Net initialization strategy, zero-initializing the final convolutional layer in each block before any residual connections, DiT regresses γ, β, and dimension-wise scaling parameters α that are applied immediately before any residual connections within the DiT block.

Note 1. Generation Process: (i) Stochastic interpolant framework decouples the formulation of xt from the forward SDE. 2. Model prediction: (i) Learn the velocity field v(x, t) and use it to express the score s(x, t) when using an SDE for sampling. 3. Optimal choice of wt will always be model prediction and interpolant dependent. 4. from a DiT model (discrete, score prediction, VP interpolant) to a SiT model (continuous, velocity prediction, Linear interpolant)

Note 1. Extend the 2D image-based VAE into a 3D VideoVAE with CausalConv3D. 2. Encode a long video with a divide-and-merge strategy. 3. Caption Model: 3.1 The temporal encoder is implemented with [Token Turing Machines](https://github.com/google-research/scenic/tree/main/scenic/projects/token_turing).

Note 1. Follow-up work: APG(https://arxiv.org/pdf/2410.02416) 1.1 Leaning more on the orthogonal component significantly attenuates this saturation side effect in generations while maintaining the quality-boosting benefits of CFG. 1.2 APG performs best when applied to the denoised predictions rather than the noise prediction.

Note 1. Training Receipt - Initialize the T2I model with a low-cost class-condition model; - Pretrain on text-image pair data rich in information density; - Fine-tuning with superior aesthetic quality data; 2. adaLN-single - one global set of shifts and scales is computed only at the first block which is shared across all the blocks, denoted as shared_adaln_cond; - a layer-specific trainable embedding, denoted as adaln_cond; adaptively adjusts the scale and shift parameters in different blocks

Note 1. Gap btw training & inference: the initial noises corrupted from real videos remain temporally correlated at the low-frequency band. 2. Free-Init Procedure 2.1 Initialize an independent Gaussian noise; 2.2 DDIM denoising to generate a clean video latent; 2.3 Obtain noisy version video latent through forward diffusion; 2.4 Combine the low-frequency components of this video latent with the high-frequency components from random Gaussian noise; 2.5 Repeat;

Note Known as SD-3 1. Change the distribution over t from the uniform distribution to the one giving more weight to intermediate timesteps by sampling them more frequently. 2. Use a ratio of 50 % original and 50 % synthetic captions. 3. MM-DiT

Note 1. When increasing the image size, the optimal noise scheduling shifts towards a noisier one (due to increased redundancy in pixels). This is more important in video generation.

Note 1. Argue that treating spatial and temporal modeling in a separable way causes motion artifacts, temporal inconsistencies, or generation of dynamic images rather than videos with vivid motion. 2. Follow-Up; Mind the Time: https://mint-video.github.io/src/MinT-paper.pdf 2.1 use interval guidance in CFG to mitigate the oversaturation issue

Note 1. guidance is harmful toward the beginning of the chain (high noise levels), largely unnecessary toward the end (low noise levels), and only beneficial in the middle.

Note 1. For Spatio-Temporal Attention, 2D RoPE for spatial & temporal. Insert a learnable [NEXT] token to differentiate image patches across different rows is enough for Spatial. No need for 3D RoPE. 2. Do not include dynamic resolution training in our main training stages. Instead, after convergence, fine-tuning the model for a few steps (10K-20K) with dynamic resolutions enables it.

Note 1. validation loss is a proxy for generation quality.

Note 1. process condition image tokens uniformly with text and noisy image tokens, integrating them into a unified sequence. Not using the direct addition of hidden states b/c constrains token interactions.

Note 1. Retain Full 3D Attention in the first and last two layers. 2. first train a Full 3D Attention model on 256 × 256 images; then inherit the model weights and replace Full 3D Attention with Skiparse Attention 3. adding slight Gaussian noise to the conditional images to enhance generalization during fine-tuning

Note 1. remove the positional embedding in DiT and find no quality loss. 2. AE-F32C32; skip the 256px; gradually fine-tuning the model to 1024px, 2K and 4K 3. Replace T5 with LLM as Text Encoder. Using T5 text embedding as key, value, and image tokens (as the query) for x-attention training results in extreme instability, with training loss frequently becoming NaN.

Note 1. Token down-sampling at middle layers: reduced the sequence length of the image keys and values by four times in middle layers making the whole network resemble a traditional convolution U-Net with only one level of down sampling. 2. improved these captioning conditions by generating multi-level captions to reduce dataset bias and prevent model overfitting. 3. we looped through the gradients of all model parameters and counted how many gradients exceeded a specific gradient-value threshold.

Note 1. As we scale up the spatial resolution, we observe the model producing slow or nearly static motion. 2. Using causal temporal attention also results in a significant drop in both quality and total scores. 3. Using interpolation of the RoPE embeddings yields improved VBench scores compared to extrapolation. 4. Observe staleness happens when we scale our model to 8B with >= 512 resolutions, probably due to the model being more easily overfitting to follow the first frame.

Note pixel-wise noise + patch-wise noise