Deep continual hashing with gradient-aware memory for cross-modal retrieval

•Decoupling the code generation and hash learning is one way to continually hashing.•Label- hash code pairs are maintained to keep the encoding ability for seen concepts.•The worth-storing data is picked out by recording loss trajectory of each sample.•Gradient projection improves the probability of codes unchanged after model updated.