We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
数据量少于BULK_COUNT未执行插数,修改了一下:
def init_ES(): pie = ProcessIntoES() # 创建ES的index pie.create_mapping() start_time = time.time() index = 0 count = 0 action_list = [] BULK_COUNT = 1000 # 每BULK_COUNT个句子一起插入到ES中 for line in open(pie.music_file, 'r', encoding='utf8'): if not line: continue item = json.loads(line) index += 1 action = { "_index": pie._index, "_type": pie.doc_type, "_source": { "question": item['question'], "answers": '\n'.join(item['answers']), } } action_list.append(action) if index >= BULK_COUNT: pie.insert_data_bulk(action_list=action_list) index = 0 count += 1 print(count) action_list = [] if index < BULK_COUNT: pie.insert_data_bulk(action_list=action_list) end_time = time.time() print("Time Cost:{0}".format(end_time - start_time))
The text was updated successfully, but these errors were encountered:
No branches or pull requests
数据量少于BULK_COUNT未执行插数,修改了一下:
The text was updated successfully, but these errors were encountered: