TY - GEN
T1 - Polarity Sampling
T2 - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
AU - Humayun, Ahmed Imtiaz
AU - Balestriero, Randall
AU - Baraniuk, Richard
N1 - Funding Information:
Humayun and Baraniuk were supported by NSF grants CCF-1911094, IIS-1838177, and IIS-1730574; ONR grants N00014-18-12571, N00014-20-1-2534, and MURI N00014-20-1-2787; AFOSR grant FA9550-22-1-0060; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.
Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - We present Polarity Sampling, a theoretically justified plug-and-play method for controlling the generation quality and diversity of any pre-trained deep generative network (DGN). Leveraging the fact that DGNs are, or can be ap-proximated by, continuous piecewise affine splines, we derive the analytical DGN output space distribution as a function of the product of the DGN's Jacobian singular values raised to a power p. We dub p the polarity param-eter and prove that p focuses the DGN sampling on the modes (p < 0) or anti-modes (p > 0) of the DGN output-space probability distribution. We demonstrate that nonzero polarity values achieve a better precision-recall (quality-diversity) Pareto frontier than standard methods, such as truncation, for a number of state-of-the-art DGNs. We also present quantitative and qualitative results on the improve-ment of overall generation quality (e.g., in terms of the Fréchet Inception Distance) for a number of state-of-the-art DGNs, including StyleGAN3, BigGAN-deep, NVAE, for different conditional and unconditional image generation tasks. In particular, Polarity Sampling redefines the state-of-the-art for StyleGAN2 on the FFHQ Dataset to FID 2.57, StyleGAN2 on the LSUN Car Dataset to FID 2.27 and Style-GAN3 on the AFHQv2 Dataset to FID 3.95. Colab Demo.
AB - We present Polarity Sampling, a theoretically justified plug-and-play method for controlling the generation quality and diversity of any pre-trained deep generative network (DGN). Leveraging the fact that DGNs are, or can be ap-proximated by, continuous piecewise affine splines, we derive the analytical DGN output space distribution as a function of the product of the DGN's Jacobian singular values raised to a power p. We dub p the polarity param-eter and prove that p focuses the DGN sampling on the modes (p < 0) or anti-modes (p > 0) of the DGN output-space probability distribution. We demonstrate that nonzero polarity values achieve a better precision-recall (quality-diversity) Pareto frontier than standard methods, such as truncation, for a number of state-of-the-art DGNs. We also present quantitative and qualitative results on the improve-ment of overall generation quality (e.g., in terms of the Fréchet Inception Distance) for a number of state-of-the-art DGNs, including StyleGAN3, BigGAN-deep, NVAE, for different conditional and unconditional image generation tasks. In particular, Polarity Sampling redefines the state-of-the-art for StyleGAN2 on the FFHQ Dataset to FID 2.57, StyleGAN2 on the LSUN Car Dataset to FID 2.27 and Style-GAN3 on the AFHQv2 Dataset to FID 3.95. Colab Demo.
KW - Deep learning architectures and techniques
KW - Image and video synthesis and generation
UR - http://www.scopus.com/inward/record.url?scp=85136750650&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85136750650&partnerID=8YFLogxK
U2 - 10.1109/CVPR52688.2022.01038
DO - 10.1109/CVPR52688.2022.01038
M3 - Conference contribution
AN - SCOPUS:85136750650
T3 - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
SP - 10631
EP - 10640
BT - Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 19 June 2022 through 24 June 2022
ER -