Identifying and characterizing highly similar notes in big clinical note datasets.

•We aimed to use a scalable algorithm to de-duplicate notes in big datasets.•We use an algorithm to minimize pairwise comparisons consisting of three phases.•Duplicate notes were very prevalent in our institutional electronic medical record.