-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathindex.html
88 lines (66 loc) · 18.3 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
<!DOCTYPE html>
<html lang="en"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><title>Home · ParallelProcessingTools</title><meta name="title" content="Home · ParallelProcessingTools"/><meta property="og:title" content="Home · ParallelProcessingTools"/><meta property="twitter:title" content="Home · ParallelProcessingTools"/><meta name="description" content="Documentation for ParallelProcessingTools."/><meta property="og:description" content="Documentation for ParallelProcessingTools."/><meta property="twitter:description" content="Documentation for ParallelProcessingTools."/><meta property="og:url" content="https://oschulz.github.io/ParallelProcessingTools.jl/stable/"/><meta property="twitter:url" content="https://oschulz.github.io/ParallelProcessingTools.jl/stable/"/><link rel="canonical" href="https://oschulz.github.io/ParallelProcessingTools.jl/stable/"/><script data-outdated-warner src="assets/warner.js"></script><link href="https://cdnjs.cloudflare.com/ajax/libs/lato-font/3.0.0/css/lato-font.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/juliamono/0.050/juliamono.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/fontawesome.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/solid.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/brands.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.8/katex.min.css" rel="stylesheet" type="text/css"/><script>documenterBaseURL="."</script><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js" data-main="assets/documenter.js"></script><script src="search_index.js"></script><script src="siteinfo.js"></script><script src="../versions.js"></script><link class="docs-theme-link" rel="stylesheet" type="text/css" href="assets/themes/catppuccin-mocha.css" data-theme-name="catppuccin-mocha"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="assets/themes/catppuccin-macchiato.css" data-theme-name="catppuccin-macchiato"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="assets/themes/catppuccin-frappe.css" data-theme-name="catppuccin-frappe"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="assets/themes/catppuccin-latte.css" data-theme-name="catppuccin-latte"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="assets/themes/documenter-dark.css" data-theme-name="documenter-dark" data-theme-primary-dark/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="assets/themes/documenter-light.css" data-theme-name="documenter-light" data-theme-primary/><script src="assets/themeswap.js"></script></head><body><div id="documenter"><nav class="docs-sidebar"><div class="docs-package-name"><span class="docs-autofit"><a href>ParallelProcessingTools</a></span></div><button class="docs-search-query input is-rounded is-small is-clickable my-2 mx-auto py-1 px-2" id="documenter-search-query">Search docs (Ctrl + /)</button><ul class="docs-menu"><li class="is-active"><a class="tocitem" href>Home</a><ul class="internal"><li><a class="tocitem" href="#Distributed-computing"><span>Distributed computing</span></a></li><li><a class="tocitem" href="#Multithreading"><span>Multithreading</span></a></li><li class="toplevel"><a class="tocitem" href="#Waiting-and-sleeping"><span>Waiting and sleeping</span></a></li><li class="toplevel"><a class="tocitem" href="#Exception-handling"><span>Exception handling</span></a></li><li class="toplevel"><a class="tocitem" href="#File-I/O"><span>File I/O</span></a></li></ul></li><li><a class="tocitem" href="api/">API</a></li><li><a class="tocitem" href="LICENSE/">LICENSE</a></li></ul><div class="docs-version-selector field has-addons"><div class="control"><span class="docs-label button is-static is-size-7">Version</span></div><div class="docs-selector control is-expanded"><div class="select is-fullwidth is-size-7"><select id="documenter-version-selector"></select></div></div></div></nav><div class="docs-main"><header class="docs-navbar"><a class="docs-sidebar-button docs-navbar-link fa-solid fa-bars is-hidden-desktop" id="documenter-sidebar-button" href="#"></a><nav class="breadcrumb"><ul class="is-hidden-mobile"><li class="is-active"><a href>Home</a></li></ul><ul class="is-hidden-tablet"><li class="is-active"><a href>Home</a></li></ul></nav><div class="docs-right"><a class="docs-navbar-link" href="https://github.com/oschulz/ParallelProcessingTools.jl" title="View the repository on GitHub"><span class="docs-icon fa-brands"></span><span class="docs-label is-hidden-touch">GitHub</span></a><a class="docs-navbar-link" href="https://github.com/oschulz/ParallelProcessingTools.jl/blob/main/docs/src/index.md" title="Edit source on GitHub"><span class="docs-icon fa-solid"></span></a><a class="docs-settings-button docs-navbar-link fa-solid fa-gear" id="documenter-settings-button" href="#" title="Settings"></a><a class="docs-article-toggle-button fa-solid fa-chevron-up" id="documenter-article-toggle-button" href="javascript:;" title="Collapse all docstrings"></a></div></header><article class="content" id="documenter-page"><h1 id="ParallelProcessingTools.jl"><a class="docs-heading-anchor" href="#ParallelProcessingTools.jl">ParallelProcessingTools.jl</a><a id="ParallelProcessingTools.jl-1"></a><a class="docs-heading-anchor-permalink" href="#ParallelProcessingTools.jl" title="Permalink"></a></h1><p>This Julia package provides some tools to ease multithreaded and distributed programming.</p><h2 id="Distributed-computing"><a class="docs-heading-anchor" href="#Distributed-computing">Distributed computing</a><a id="Distributed-computing-1"></a><a class="docs-heading-anchor-permalink" href="#Distributed-computing" title="Permalink"></a></h2><p>Julia provides native support for distributed computing on multiple Julia processes that run in parallel on the same or on different machines. ParallelProcessingTools add some machinery to make some aspects of this even easier.</p><p>An internal elastic cluster manager (<a href="api/#ParallelProcessingTools.ppt_cluster_manager"><code>ppt_cluster_manager</code></a>, a modified version of <code>ParallelProcessingTools.ElasticManager</code>), started on demand, allows for starting (<a href="api/#ParallelProcessingTools.runworkers"><code>runworkers</code></a>) an stopping (<a href="api/#ParallelProcessingTools.stopworkers"><code>stopworkers</code></a>) worker processes in a dynamic fashion. The worker processes can also be started outside of the Julia session (<a href="api/#ParallelProcessingTools.worker_start_command"><code>worker_start_command</code></a> and <a href="api/#ParallelProcessingTools.write_worker_start_script"><code>write_worker_start_script</code></a>), this can be useful to add worker to a running Julia session via manually controlled batch jobs, for example. Workers can be started locally (<a href="api/#ParallelProcessingTools.OnLocalhost"><code>OnLocalhost</code></a>), via SLURM (<a href="api/#ParallelProcessingTools.SlurmRun"><code>SlurmRun</code></a>), or via HTCondor (<a href="api/#ParallelProcessingTools.HTCondorRun"><code>HTCondorRun</code></a>). Other methods to start workers (e.g. via SSH) may be added in the future (contributions are very welcome).</p><p>The elastic cluster manager automatically adds new workers to an automatically created dynamic worker pool (<a href="api/#ParallelProcessingTools.ppt_worker_pool-Tuple{}"><code>ppt_worker_pool</code></a>) of type <a href="api/#ParallelProcessingTools.FlexWorkerPool"><code>FlexWorkerPool</code></a> that optionally supports oversubscription. Users can <code>take!</code> workers from the pool and <code>put!</code> them back, or use <a href="api/#ParallelProcessingTools.onworker"><code>onworker</code></a> to send work to workers in the pool without exceeding their maximum occupancy.</p><p>Since workers can appear and disappear dynamically, initializing them (loading packages, etc.) via the standard <code>Distributed.@everywhere</code> macro is problematic, as workers added afterwards won't be initialized. Parallel processing tools provides the macro <a href="api/#ParallelProcessingTools.@always_everywhere-Tuple{Any}"><code>@always_everywhere</code></a> to run code globally on all current processes, but also store the code so it can be run again on future new worker processes. Workers that are part of a <a href="api/#ParallelProcessingTools.FlexWorkerPool"><code>FlexWorkerPool</code></a> will be updated automatically on <code>take!</code> and <code>onworker</code>. You can also use <a href="api/#ParallelProcessingTools.ensure_procinit"><code>ensure_procinit</code></a> to manually update all workers to all <code>@always_everywhere</code> used so far.</p><p><a href="api/#ParallelProcessingTools.AutoThreadPinning"><code>AutoThreadPinning</code></a>, in conjunction with the package <a href="https://github.com/carstenbauer/ThreadPinning.jl/"><code>ThreadPinning</code></a>, provides a convenient way to perform automatic thread pinning (e.g. inside of <code>@always_everywhere</code>, to apply thead pinning to all processes). Note that <code>ThreadPinning.pinthreads(AutoThreadPinning())</code> works on a best-effort basis and that advanced applications may require customized thread pinning for best performance.</p><p>Some batch system configurations can result in whole Julia processes, or even a whole batch job, being terminated if a process exceeds its memory limit. In such cases, you can try to gain a softer failure mode by setting a custom (slightly smaller) memory limit using <a href="api/#ParallelProcessingTools.memory_limit!"><code>memory_limit!</code></a>.</p><p>For example:</p><pre><code class="language-julia hljs">ENV["JULIA_DEBUG"] = "ParallelProcessingTools"
ENV["JULIA_WORKER_TIMEOUT"] = "120"
using ParallelProcessingTools, Distributed
@always_everywhere begin
using ParallelProcessingTools
using Statistics
import ThreadPinning
pinthreads_auto()
# Optional: Set a custom memory limit for worker processes:
# myid() != 1 && memory_limit!(8 * 1000^3) # 8 GB
end
runmode = OnLocalhost(n = 4)
# runmode = lkSlurmRun(slurm_flags = `--ntasks=4 --cpus-per-task=8 --mem-per-cpu=8G`)
display(worker_start_command(runmode))
# Add some workers and initialize with all `@always_everywhere` code:
old_nprocs = nprocs()
_, n = runworkers(runmode)
@wait_while nprocs() < old_nprocs + n
ensure_procinit()
# Show worker resources:
pool = ppt_worker_pool()
display(pool)
display(worker_resources())
# Confirm that Distributions is loaded on a worker:
worker = last(workers())
@fetchfrom worker mean(rand(100))
# Some more init code
@always_everywhere begin
X = rand(100)
end
# Add some more workers, we won't run `ensure_procinit()` manually this time:
old_nprocs = nprocs()
_, n = runworkers(runmode)
@wait_while nprocs() < old_nprocs + n
# Worker hasn't run @always_everywhere code yet, so it doesn't have `mean`:
worker = last(workers())
display(@return_exceptions @userfriendly_exceptions begin
@fetchfrom worker mean(X)
end)
# Using `take!` on a `FlexWorkerPool` automatically runs init code as necessary:
pid = take!(pool)
try
remotecall_fetch(() -> mean(X), pid)
finally
put!(pool, pid)
end
# `onworker` (using the default `FlexWorkerPool` here) does the same:
onworker(mean, X)
# If we don't need workers processes for a while, let's stop them:
stopworkers()</code></pre><p>We can also use SLURM batch scripts, like this (e.g. "batchtest.jl"):</p><pre><code class="language-julia hljs">#!/usr/bin/env julia
#SBATCH --ntasks=4 --cpus-per-task=8 --mem-per-cpu=8G --time=00:15:00
using Pkg; pkg"activate @SOME_JULIA_ENVIRONMENT"
ENV["JULIA_DEBUG"] = "ParallelProcessingTools"
ENV["JULIA_WORKER_TIMEOUT"] = "120"
using ParallelProcessingTools, Distributed
@always_everywhere begin
using ParallelProcessingTools
import ThreadPinning
pinthreads_auto()
end
_, n = runworkers(SlurmRun(slurm_flags = `--cpu-bind=cores --mem-bind=local`))
@wait_while maxtime=240 nprocs() < n + 1
resources = worker_resources()
display(resources)
stopworkers()</code></pre><p>This should run with a simple</p><pre><code class="language-shell hljs">sbatch -o out.txt batchtest.jl</code></pre><p>and "out.txt" should then contain debugging output and a list of the worker resources.</p><h2 id="Multithreading"><a class="docs-heading-anchor" href="#Multithreading">Multithreading</a><a id="Multithreading-1"></a><a class="docs-heading-anchor-permalink" href="#Multithreading" title="Permalink"></a></h2><p>To test multithreading performance and help debug and optimize multithreaded code, ParallelProcessingTools provides the utility macros <a href="api/#ParallelProcessingTools.@onthreads-Tuple{Any, Any}"><code>@onthreads</code></a> to run code explicitly on the selected Julia threads (all threads can be listed using <a href="api/#ParallelProcessingTools.allthreads-Tuple{}"><code>allthreads</code></a>).</p><p>You can use the macro <a href="api/#ParallelProcessingTools.@critical-Tuple{Any}"><code>@critical</code></a> to prevent code that may suffer from race conditions in parallel to other code fenced by <code>@critical</code>.</p><p>The macro <a href="api/#ParallelProcessingTools.@mt_out_of_order-Tuple{Any}"><code>@mt_out_of_order</code></a> is useful to run different code on in parallel on Julia threads.</p><h1 id="Waiting-and-sleeping"><a class="docs-heading-anchor" href="#Waiting-and-sleeping">Waiting and sleeping</a><a id="Waiting-and-sleeping-1"></a><a class="docs-heading-anchor-permalink" href="#Waiting-and-sleeping" title="Permalink"></a></h1><p>In a parallel computing scenario, on threads, distributed processes or both, or when dealing with I/O operations, code often needs to wait. In addition a timeout mechanism is often necessary. Julia's standard <code>wait</code> function can only waits a single object without a timeout. (<code>waitany</code>, requires Julia >= v1.12, can be used to wait for multiple tasks).</p><p>ParallelProcessingTools provides a very flexible macro <a href="api/#ParallelProcessingTools.@wait_while-Tuple"><code>@wait_while</code></a> to wait for custom conditions with an optional timeout, as well as the functions <a href="api/#ParallelProcessingTools.wait_for_all"><code>wait_for_all</code></a> and <a href="api/#ParallelProcessingTools.wait_for_any"><code>wait_for_any</code></a> that can wait for different kinds of objects, also with an optional timeout.</p><p>The functions <a href="api/#ParallelProcessingTools.sleep_ns-Tuple{Integer}"><code>sleep_ns</code></a> and <a href="api/#ParallelProcessingTools.idle_sleep-Tuple{Integer, Real, Real}"><code>idle_sleep</code></a> can be used to implement custom scenarios that require precise sleeping for both very short and long intervals.</p><h1 id="Exception-handling"><a class="docs-heading-anchor" href="#Exception-handling">Exception handling</a><a id="Exception-handling-1"></a><a class="docs-heading-anchor-permalink" href="#Exception-handling" title="Permalink"></a></h1><p>Exceptions throws during remote code execution can be complex, nested and sometimes hard to understand. You can use the functions <a href="api/#ParallelProcessingTools.inner_exception"><code>inner_exception</code></a>, <a href="api/#ParallelProcessingTools.onlyfirst_exception"><code>onlyfirst_exception</code></a> and <a href="api/#ParallelProcessingTools.original_exception"><code>original_exception</code></a> to get to the underlying reason of a failure more easily. The macro <a href="api/#ParallelProcessingTools.@userfriendly_exceptions-Tuple{Any}"><code>@userfriendly_exceptions</code></a> automatizes this to some extent for a given piece of code.</p><p>To get an exception "in hand" for further analysis, you can use the macro <a href="api/#ParallelProcessingTools.@return_exceptions-Tuple{Any}"><code>@return_exceptions</code></a> to make (possibly failing) code return the exceptions instead of throwing it.</p><h1 id="File-I/O"><a class="docs-heading-anchor" href="#File-I/O">File I/O</a><a id="File-I/O-1"></a><a class="docs-heading-anchor-permalink" href="#File-I/O" title="Permalink"></a></h1><p>File handling can become more challenging when working in a parallel and possibly distributed fashion. Code or whole workers can crash, resulting in corrupt files, or workers may become disconnected, but still write files and clash with restarted code (resulting in race conditions and may also result in corrupt files).</p><p>ParallelProcessingTools provides the functions <a href="api/#ParallelProcessingTools.write_files"><code>write_files</code></a> and <a href="api/#ParallelProcessingTools.read_files"><code>read_files</code></a> to implement atomic file operations, on a best-effort basis (depending on the operating system and underlying file systems).</p></article><nav class="docs-footer"><a class="docs-footer-nextpage" href="api/">API »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.8.1 on <span class="colophon-date" title="Thursday 20 February 2025 11:50">Thursday 20 February 2025</span>. Using Julia version 1.11.3.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>