SELF-SIMILARITY IN DEEP NEURAL NETWORK MODULES FOR IMAGES AND VIDEOS

Self-similarity within and across video frames is used in today's best methods for image/video restoration and generation in the form of attention and warping modules within deep neural networks (DNNs). Self-similarity is a microcosm of the greater mission to understand images by first understanding components within the image. While this design philosophy mirrors how generic DNNs build features, DNNs are at the mercy of many practical limitations such as available training data, specified loss function, network size, and training time. Even when properly scaled up, DNNs are still limited by observed correlations. Robust generalization necessitates that DNNs learn underlying scientific principles, but today's DNNs have not demonstrated this understanding. In short, DNNs should be learning scientific relationships, but they do not because science is not in the data. This limitation is the inspiration for a larger research goal to shift DNNs from data-only learning to data-driven learning by designing architectures that explicitly incorporate assumptions about our data via inductive bias. This thesis contributes new ideas to principally improve DNNs for images and videos by incorporating our knowledge of self-similarity into modules.