Defining Information Content with Second Order Representation Shifts

 Personnel: Ryan Benkert, Mohit Prabhushankar


Goal/Motivation: The goal of active learning is to identify the most informative unlabeled data points for annotation. Ideal algorithms select samples that most prominently aid in model performance improvement.  

Challenges: In active learning, algorithms rank data points by information content and select the most informative samples for annotation. Therefore, the success of an active learning algorithms relies on two factors: 1) The definition of highly “informative” samples; and 2) How well the model approximates information content when constrained by limited training data. In other words, a successful active learning algorithm depends on what it considers “informative”, and how well it produces importance scores. Incorporating both 1) and 2) within an effective algorithm represents a major challenge in the field of active learning. As a result, the performance of several approaches varies significantly across different datasets or parameter choices and renders practical deployment uncertain.

High Level Description of the Work: Several popular strategies define information content on the model representation of data points directly. However, model representations are infamously unpredictable, especially when few training points are available [1, 2, 3](a common case in active learning). 

At OLIVES, we pursue a fundamentally different approach. Instead of utilizing the model representation directly, we define information content on representation shifts. Our definition is not only more accurate in active learning settings, but is also simpler to approximate when few data points are available. In Figure~1, we show examples of highly informative images based on different information content definitions. We see that informative samples as per our definition are significantly harder to distinguish or are ambiguous which implies a higher information content. For instance, our “informative" samples are ambiguous images with complex shapes or ambiguous colors (e.g. the frog image in the top row; third image from the right side). In contrast, the other definitions consider clear images informative that are similar among each other. In GauSS [4], we combine the definition with efficient sampling techniques to improve the active learning performance on natural images. With FALSE [5], we gear the definition towards unlabeled data points directly to improve the active learning performance on noisy data.

References:

  1. J. Lee and G. AlRegib, "Gradients as a Measure of Uncertainty in Neural Networks," in IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, Oct. 2020.

  2. G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, "Backpropagated Gradient Representations for Anomaly Detection," in Proceedings of the European Conference on Computer Vision (ECCV), SEC, Glasgow, Aug. 23-28 2020. [PDF] [Code]

  3. D. Temel, G. Kwon*, M. Prabhushankar*, and G. AlRegib, "CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition," in Advances in Neural Information Processing Systems (NIPS) Workshop on Machine Learning for Intelligent Transportation Systems, Long Beach, CA, Dec. 2017

  4. R. Benkert, M. Prabhushankar, G. AlRegib, A. Parchami, and E. Corona, "Gaussian Switch Sampling: A Second Order Approach to Active Learning," IEEE Transactions on Artificial Intelligence (TAI), submitted on Jun. 4 2022.

  5. R. Benkert, M. Prabhushankar, and G. AlRegib, "Forgetful Active Learning With Switch Events: Efficient Sampling for Out-of-Distribution Data," in IEEE International Conference on Image Processing (ICIP), Bordeaux, France, Oct. 16-19 2022.