Skip to content

Add support for DSS URLs#209

Draft
philipp-fischer wants to merge 7 commits intodevelopfrom
feature/dss
Draft

Add support for DSS URLs#209
philipp-fischer wants to merge 7 commits intodevelopfrom
feature/dss

Conversation

@philipp-fischer
Copy link
Collaborator

Make EPath and metadatasets support paths like this:

__module__: megatron.energon
__class__: MetadatasetV2
splits:
  train:
    blend:
      - path: dss://charts123_zh@v0
        weight: 1
        aux:
          media_source: dss://charts2500@v0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants