44f80b5872b1bb9679d15b8230c1731fd26ac527,torchnlp/text_encoders/static_tokenizer_encoder.py,StaticTokenizerEncoder,__init__,#StaticTokenizerEncoder#Any#Any#Any#Any#Any#,15

Before Change


        self.lower = lower
        self.tokenize = tokenize
        self.append_eos = append_eos
        self.tokens = Counter()

        for text in sample:
            self.tokens.update(self._preprocess(text))

After Change


        for text in sample:
            self.tokens.update(self.tokenize(text))

        self.stoi = RESERVED_STOI.copy()
        self.itos = RESERVED_ITOS[:]
        for token, count in self.tokens.items():
            if count >= min_occurrences:
Italian Trulli
In pattern: SUPERPATTERN

Frequency: 3

Non-data size: 3

Instances


Project Name: PetrochukM/PyTorch-NLP
Commit Name: 44f80b5872b1bb9679d15b8230c1731fd26ac527
Time: 2018-03-10
Author: petrochukm@gmail.com
File Name: torchnlp/text_encoders/static_tokenizer_encoder.py
Class Name: StaticTokenizerEncoder
Method Name: __init__


Project Name: NTMC-Community/MatchZoo
Commit Name: 4bc0cb5d2924a63cf06f641b7cf36f799885f33f
Time: 2018-12-26
Author: 948280670@qq.com
File Name: matchzoo/processor_units/processor_units.py
Class Name: WordHashingUnit
Method Name: transform


Project Name: anttttti/Wordbatch
Commit Name: 7170cdf9c6ed8beacd93738b0ec1c97cfbc23b6e
Time: 2018-04-12
Author: antti.puurula@yahoo.com
File Name: wordbatch/wordbatch.py
Class Name: WordBatch
Method Name: process