feat: add generative recommendation tokenizer. #317
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
magicheng0816
requested review from
Clement-Wang26,
RobbieLeung,
walsonyang and
yq33victor
yq33victor
reviewed
Nov 5, 2025
| @@ -0,0 +1,96 @@ | |||
| #pragma once | |||
There was a problem hiding this comment.
nit: add header like this:
/* Copyright 2025 The xLLM Authors. All Rights Reserved.
Copyright 2024 The ScaleLLM Authors. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://github.com/jd-opensource/xllm/blob/main/LICENSE
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================*/
| std::lock_guard<std::mutex> lock(instance_map_mutex_); | ||
|
|
||
| auto it = instance_map_.find(version); | ||
| if (it == instance_map_.end()) { |
There was a problem hiding this comment.
why we need to find twice from instance_map_? Line25 above has already do this.
|
|
||
| static void DestroyInstance(const std::string& version) { | ||
| std::lock_guard<std::mutex> lock(instance_map_mutex_); | ||
| instance_map_.erase(version); |
There was a problem hiding this comment.
should erase version from instance_version_list_ ?
| path.append(tokenizer_args_.vocab_file()).string(); | ||
|
|
||
| LOG(INFO) << "model_version:" << model_version; | ||
| LOG(INFO) << "vocab_full_path:" << vocab_full_path; |
There was a problem hiding this comment.
nit: :)
LOG(INFO) << "model_version:" << model_version
<< ", vocab_full_path:" << vocab_full_path;
| ->initialize(vocab_full_path)) | ||
| << "Failed to initialize vocab dict from " << vocab_full_path; | ||
| } else { | ||
| LOG(INFO) << "vocab file is not set"; |
There was a problem hiding this comment.
nit: maybe log(error) or log(fatal) or return false ?
| @@ -0,0 +1,138 @@ | |||
| #include "rec_vocab_dict.h" | |||
| } | ||
|
|
||
| bool RecVocabDict::get_items_by_tokens(const RecTokenTriple& rec_token_triple, | ||
| std::vector<int64_t>* item_ids) const { |
There was a problem hiding this comment.
nit: maybe use & here is better.
std::vector<int64_t>* item_ids -> const std::vector<int64_t>& item_ids
CHECK(!item_ids.empty());
| @@ -0,0 +1,94 @@ | |||
| #pragma once | |||
|
|
||
| /** | ||
| * @brief initialize instance, parse vocab file | ||
| * @param vocab_file vocab file, need full path |
There was a problem hiding this comment.
nit: maybe we can align the comment format.
// @brief initialize instance, parse vocab file
// @param vocab_file vocab file, need full path
// ...
| @@ -0,0 +1,52 @@ | |||
| #include "rec_tokenizer.h" | |||
liutongxuan
reviewed
Nov 6, 2025
| std::sort(model_weights_files_.begin(), model_weights_files_.end()); | ||
|
|
||
| //@todo: 'false' will be replaced with generative recommendation judgment | ||
| if (false) { |
There was a problem hiding this comment.
If it doesn't work yet, shouldn't add this logic for now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.