Skip to content

Reformat README and update development instructions #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 19, 2024
Merged

Reformat README and update development instructions #115

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 45 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,53 @@
# Distributed

The `Distributed` package provides functionality for creating and controlling multiple Julia processes remotely, and for performing distributed and parallel computing. It uses network sockets or other supported interfaces to communicate between Julia processes, and relies on Julia's `Serialization` stdlib package to transform Julia objects into a format that can be transferred between processes efficiently. It provides a full set of utilities to create and destroy new Julia processes and add them to a "cluster" (a collection of Julia processes connected together), as well as functions to perform Remote Procedure Calls (RPC) between the processes within a cluster. See the `API` section for details.
The `Distributed` package provides functionality for creating and controlling
multiple Julia processes remotely, and for performing distributed and parallel
computing. It uses network sockets or other supported interfaces to communicate
between Julia processes, and relies on Julia's `Serialization` stdlib package to
transform Julia objects into a format that can be transferred between processes
efficiently. It provides a full set of utilities to create and destroy new Julia
processes and add them to a "cluster" (a collection of Julia processes connected
together), as well as functions to perform Remote Procedure Calls (RPC) between
the processes within a cluster. See the `API` section for details.

This package ships as part of the Julia stdlib.

## Using development versions of this package

To use a newer version of this package, you need to build Julia from scratch. The build process is the same as any other build except that you need to change the commit used in `stdlib/Distributed.version`.

It's also possible to load a development version of the package using [the trick used in the Section named "Using the development version of Pkg.jl" in the `Pkg.jl` repo](https://github.com/JuliaLang/Pkg.jl#using-the-development-version-of-pkgjl), but the capabilities are limited as all other packages will depend on the stdlib version of the package and will not work with the modified package.
### On Julia 1.11+
In Julia 1.11 Distributed was excised from the default system image and became
more of an independent package. As such, to use a different version it's enough
to just `dev` it explicitly:
```julia-repl
pkg> dev https://github.com/JuliaLang/Distributed.jl.git
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it just dev Distributed.jl or is the issue that it is not registered in the registry?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it's not registered:

(@v1.11) pkg> dev Distributed.jl
    Updating registry at `~/.julia/registries/General.toml`
ERROR: The following package names could not be resolved:
 * Distributed (8ba89e20-285c-5b6f-9357-94700520ee1b in manifest but not in project)

Can we register it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, would need to be something similar to JuliaRegistries/General#89713

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going off this checklist I've opened two PRs that should complete excision and make Distributed upgradeable:

```

### On older Julia versions
To use a newer version of this package on older Julia versions, you need to build
Julia from scratch. The build process is the same as any other build except that
you need to change the commit used in `stdlib/Distributed.version`.

It's also possible to load a development version of the package using [the trick
used in the Section named "Using the development version of Pkg.jl" in the
`Pkg.jl`
repo](https://github.com/JuliaLang/Pkg.jl#using-the-development-version-of-pkgjl),
but the capabilities are limited as all other packages will depend on the stdlib
version of the package and will not work with the modified package.

## API

The public API of `Distributed` consists of a variety of functions for various tasks; for creating and destroying processes within a cluster:
The public API of `Distributed` consists of a variety of functions for various
tasks; for creating and destroying processes within a cluster:

- `addprocs` - create one or more Julia processes and connect them to the cluster
- `rmprocs` - shutdown and remove one or more Julia processes from the cluster

For controlling other processes via RPC:

- `remotecall` - call a function on another process and return a `Future` referencing the result of that call
- `Future` - an object that references the result of a `remotecall` that hasn't yet completed - use `fetch` to return the call's result, or `wait` to just wait for the remote call to finish
- `Future` - an object that references the result of a `remotecall` that hasn't
yet completed - use `fetch` to return the call's result, or `wait` to just
wait for the remote call to finish.
- `remotecall_fetch` - the same as `fetch(remotecall(...))`
- `remotecall_wait` - the same as `wait(remotecall(...))`
- `remote_do` - like `remotecall`, but does not provide a way to access the result of the call
Expand Down Expand Up @@ -49,6 +76,15 @@ For controlling multiple processes at once:

### Process Identifiers

Julia processes connected with `Distributed` are all assigned a cluster-unique `Int` identifier, starting from `1`. The first Julia process within a cluster is given ID `1`, while other processes added via `addprocs` get incrementing IDs (`2`, `3`, etc.). Functions and macros which communicate from one process to another usually take one or more identifiers to determine which process they target - for example, `remotecall_fetch(myid, 2)` calls `myid()` on process 2.

**Note:** Only process 1 (often called the "head", "primary", or "master") may add or remove processes, and manages the rest of the cluster. Other processes (called "workers" or "worker processes") may still call functions on each other and send and receive data, but `addprocs`/`rmprocs` on worker processes will fail with an error.
Julia processes connected with `Distributed` are all assigned a cluster-unique
`Int` identifier, starting from `1`. The first Julia process within a cluster is
given ID `1`, while other processes added via `addprocs` get incrementing IDs
(`2`, `3`, etc.). Functions and macros which communicate from one process to
another usually take one or more identifiers to determine which process they
target - for example, `remotecall_fetch(myid, 2)` calls `myid()` on process 2.

**Note:** Only process 1 (often called the "head", "primary", or "master") may
add or remove processes, and manages the rest of the cluster. Other processes
(called "workers" or "worker processes") may still call functions on each other
and send and receive data, but `addprocs`/`rmprocs` on worker processes will
fail with an error.
Loading