Compositional Zero-Shot Learning using Multi-Branch Graph Convolution and Cross-layer Knowledge Sharing

Article Type

Research Article

Publication Title

Pattern Recognition

Abstract

The purpose of the Compositional Zero-Shot Learning (CZSL) is to recognize new state-object compositions of known objects and known states. For example, the CZSL model should recognize young cat when the model has seen images of a few state-object compositions like young tiger, old tiger and old cat. The visual features of a state may have significant variation across different compositions of the state with different objects. For example, in the compositions peeled apple and peeled orange, the state peeled has different visual features. This context dependency of state features is difficult to learn from the annotated images of different compositions. We propose a Graph Convolutional Network (GCN) with two distinct branches for object and state recognition. GCN utilizes its ability to aggregate features from the non-Euclidean neighbourhood. This aggregation ability of GCN can help our model to capture the intricate dependencies between visual features of state and object. We also propose a novel cross-layer knowledge sharing strategy for the purpose of reducing ambiguity in learning state features due to context dependency. The proposed cross-layer knowledge sharing helps in identifying a set of objects having feasible compositions with a particular state and thereby reducing the ambiguity in the state features. Finally, we propose a feasibility based penalization to better regularize the joint prediction from the two branches of the network. The proposed algorithm is evaluated on the challenging benchmarks and competitive results in comparison to state-of-the-art algorithms have been achieved.

DOI

10.1016/j.patcog.2023.109916

Publication Date

1-1-2024

Link to Full Text

Share

COinS