Add a groupby operator #1123
Merged
Add a groupby operator #1123
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bohou-aryn
force-pushed
the
clustering
branch
from
955cd3c to
db0d219
Compare
Closed
HenryL27
approved these changes
Jan 30, 2025
| def init_embedding(row): | ||
| doc = Document.from_row(row) | ||
| return {"vector": doc.embedding, "cluster": -1} | ||
|
|
There was a problem hiding this comment.
can you add a "assert self.context.exec_mode == ExecMode.RAY" in here?
| return context.read.document(doc_list) | ||
|
|
||
| def test_groupby_count(self, fruits_docset): | ||
| aggregated = fruits_docset.groupby("text_representation").count() |
There was a problem hiding this comment.
What do the Documents in aggregated look like at this point?
Comment on lines
+39
to
+41
| def init(embeddings, K, init_mode): | ||
| if init_mode == "random": |
There was a problem hiding this comment.
supernit: init_mode could be an Enum but str is fine too. I guess would be nice to have the list of known init_modes in the exception?
HenryL27
reviewed
Jan 31, 2025
Comment on lines
1
to
2
| from ray.data._internal.aggregate import Count | ||
| from ray.data.aggregate import AggregateFn |
There was a problem hiding this comment.
from typing import TYPE_CHECKING
if TYPE_CHECKING:
<ray imports>
HenryL27
reviewed
Jan 31, 2025
|
|
||
| return DocSet(self._docset.context, DatasetScan(serialized)) | ||
|
|
||
| def count(self) -> DocSet: |
HenryL27
reviewed
Jan 31, 2025
|
|
||
| @staticmethod | ||
| def update(embeddings, centroids, iterations, epsilon): | ||
| i = 0 |
bohou-aryn
force-pushed
the
clustering
branch
2 times, most recently
from
8a6aadd to
3653e2f
Compare
This includes generally three steps: 1. materialize a document's embedding 2. initialize centroids randomly 2. iterate the kmeans process until converge, this is based on ray dataset map group and aggregate operators. The result centroids could be used for downstream work.
bohou-aryn
force-pushed
the
clustering
branch
2 times, most recently
from
907625d to
fdfc705
Compare
bohou-aryn
force-pushed
the
clustering
branch
from
fdfc705 to
59c08ad
Compare
Implement groupby based on ray dataset groupby and show how a general entity clustering could be used together with kmeans clustering.
bohou-aryn
force-pushed
the
clustering
branch
from
59c08ad to
b25ee78
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement groupby based on ray dataset groupby and show how a general
entity clustering could be used together with kmeans clustering.