22bce0332c94639df80820a439e4c1893a5552f8,mindsdb/libs/data_sources/file_ds.py,FileDS,_getDataIo,#FileDS#Any#,58
Before Change
// lets try to figure out if its a csv
try:
data.seek(0)
full = len(data.read())
data.seek(0)
bytes_to_read = int(full*0.3)
dialect = csv.Sniffer().sniff(data.read(bytes_to_read))
data.seek(0)
After Change
try:
data.seek(0)
first_few_lines = []
i = 0
for line in data:
i += 1
first_few_lines.append(line)
if i > 500:
break
dialect = csv.Sniffer().sniff("".join(first_few_lines))
data.seek(0)
// if csv dialect identified then return csv
if dialect:
In pattern: SUPERPATTERN
Frequency: 4
Non-data size: 8
Instances Project Name: mindsdb/mindsdb
Commit Name: 22bce0332c94639df80820a439e4c1893a5552f8
Time: 2019-02-01
Author: george@cerebralab.com
File Name: mindsdb/libs/data_sources/file_ds.py
Class Name: FileDS
Method Name: _getDataIo
Project Name: mindsdb/mindsdb
Commit Name: bafbfaa5718e1de72b805237d2c350aec11de9fe
Time: 2019-02-01
Author: george@cerebralab.com
File Name: mindsdb/libs/data_sources/file_ds.py
Class Name: FileDS
Method Name: _getDataIo
Project Name: HazyResearch/fonduer
Commit Name: 66c21553343edf0e76c3c728ac3fa10b1fb6720b
Time: 2018-12-20
Author: SenWu@users.noreply.github.com
File Name: src/fonduer/parser/preprocessors/csv_doc_preprocessor.py
Class Name: CSVDocPreprocessor
Method Name: __len__
Project Name: HazyResearch/fonduer
Commit Name: 66c21553343edf0e76c3c728ac3fa10b1fb6720b
Time: 2018-12-20
Author: SenWu@users.noreply.github.com
File Name: src/fonduer/parser/preprocessors/tsv_doc_preprocessor.py
Class Name: TSVDocPreprocessor
Method Name: __len__