Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Edge-casing for multi-GPU HF-to-NeoX conversion #1065

Merged
merged 5 commits into from
Nov 1, 2023
Merged

Edge-casing for multi-GPU HF-to-NeoX conversion #1065

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fc17ada
edge-casing for multiGPU hf to sequential case
haileyschoelkopf Oct 30, 2023
4579a33
cleanup whitespace
haileyschoelkopf Oct 30, 2023
71ca6cf
Update NeoXArgs docs automatically
invalid-email-address Oct 30, 2023
7c50e77
Merge branch 'main' into fix-conversion-from-hf
Quentin-Anthony Oct 31, 2023
6cb4077
Update NeoXArgs docs automatically
invalid-email-address Oct 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions configs/neox_arguments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Logging Arguments

- **git_hash**: str

Default = a97bd1f
Default = 4579a33

current git hash of repository

Expand Down Expand Up @@ -605,7 +605,7 @@ Optimizer Arguments

Default = adam

Type of optimizer to use. Choose from ['adam', 'onebitadam', 'cpu_adam', 'cpu_torch_adam', 'sm3', 'madgrad_wd', 'sgd', 'lion']
Type of optimizer to use. Choose from ['adam', 'onebitadam', 'cpu_adam', 'cpu_torch_adam', 'sm3', 'madgrad_wd', 'sgd']
NOTE: sgd will use MuSGD from Mup. Mup must be enabled for this optimizer.


Expand Down
6 changes: 3 additions & 3 deletions tools/ckpts/convert_hf_to_sequential.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,15 +519,15 @@ def get_non_existing_dir(tmp_dir):
model, optimizer, _, lr_scheduler = deepspeed.initialize(
model=model,
optimizer=optimizer,
args=neox_args,
# args=neox_args,
Quentin-Anthony marked this conversation as resolved.
Show resolved Hide resolved
lr_scheduler=lr_scheduler,
dist_init_required=False,
model_parameters=None,
config_params=neox_args.deepspeed_config,
mpu=mpu if not neox_args.is_pipe_parallel else None,
)

if os.environ["OMPI_COMM_WORLD_RANK"] == "0":
if os.environ.get("OMPI_COMM_WORLD_RANK", "1") == "0":
os.makedirs(f"{tmp_cache_dir}", exist_ok=True)

torch.distributed.barrier()
Expand Down Expand Up @@ -566,7 +566,7 @@ def get_non_existing_dir(tmp_dir):
print("==========================================")
convert(hf_model, ckpt_dir=ckpt_dir, output_dir=args.output_dir)

if os.environ["OMPI_COMM_WORLD_RANK"] == "0":
if os.environ.get("OMPI_COMM_WORLD_RANK", "1") == "0":
# cleanup temp dir
os.system(f"rm -r {tmp_cache_dir}")

Expand Down