Towards Better Character-based Word Vectors

● Learn word vectors in larger wikipedia dataset (22 G). ● Run more evals for Byte Pair Search method. BPS Example: Input: low 5, lower 2, newest 6, widest 3 Both (s, t) and (e, s) pair has subword pair frequency 9, but in the every iteration of BPE, BPE will only merge one pair. Merge order might influence subwords vocabulary and final word segmentation results. To address this issue, BPS will go through all the possibilities and returning more relevant subwords.