Skip to content

Commit baca254

Browse files
authored
Update file transfer guides (#824)
* Deprecating squid * Promoting osdf:/// and file:/// transfer plugins
1 parent 29f2363 commit baca254

10 files changed

+198
-888
lines changed

_layouts/file_avail.html

+5-18
Original file line numberDiff line numberDiff line change
@@ -9,40 +9,27 @@ <h2>Which Option is the Best for Your Files?</h2>
99
<tr>
1010
<th>Input Sizes</th>
1111
<th>Output Sizes</th>
12-
<th>Link to Guide</th>
1312
<th>File Location</th>
14-
<th>How to Transfer</th>
13+
<th>Syntax for <code>transfer_input_files</code></th>
1514
<th>Availability, Security</th>
1615
<!--<th>Special Considerations</th>-->
1716
</tr>
1817

1918
<tr>
2019
<td>0 - 100 MB per file, up to 500 MB per job</td>
2120
<td>0 - 5 GB per job</td>
22-
<td><a href="file-availability.html">Small Input/Output File Transfer via <b>HTCondor</b></a></td>
2321
<td><code>/home</code></td>
24-
<td><b>submit file</b>; filename in <small><code>transfer_input_files</code></small></td>
25-
<td>CHTC, UW Grid, and OSG; works for <i>your</i> jobs</td>
22+
<td>No special syntax</td>
23+
<td>CHTC and external pools</td>
2624
<!--<td><font color="red">DO NOT USE HTCondor transfer for files in <code>/staging</code> OR <code>/squid</code></font></td>-->
2725
</tr>
2826

29-
<tr>
30-
<td>100 MB - 1 GB per repeatedly-used file</td>
31-
<td>Not available for output</td>
32-
<td><a href="file-avail-squid.html">Large Input File Availability Via <b>Squid</b></a></td>
33-
<td><code>/squid</code></td>
34-
<td><b>submit file</b>; http link in <small><code>transfer_input_files</code></small></td>
35-
<td>CHTC, UW Grid, and OSG; files are made *publicly-readable* via an HTTP address</i></td>
36-
<!--<td>large files unique to individual jobs are better for large data staging</td>-->
37-
</tr>
38-
3927
<tr>
4028
<td>100 MB - TBs per job-specific file; repeatedly-used files > 1GB</td>
4129
<td>4 GB - TBs per job</td>
42-
<td><a href="file-avail-largedata.html">Large Input and Output File Availability Via <b>Staging</b></a></td>
4330
<td><code>/staging</code></td>
44-
<td><b>job executable</b>; copy or move within the job</td>
45-
<td>a portion of CHTC; accessible only to <i>your</i> jobs</td>
31+
<td><code>osdf:///</code> or <code>file:///</code></td>
32+
<td>all of CHTC/external pools or a subset of CHTC</td>
4633
<!--<td>special submit "Requirements"</td>-->
4734
</tr>
4835

_redirects/file-avail-squid.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
layout: redirect
3+
redirect_url: /uw-research-computing/htc-job-file-transfer
4+
permalink: /uw-research-computing/file-avail-squid
5+
---

_redirects/file-availability.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
layout: redirect
3+
redirect_url: /uw-research-computing/htc-job-file-transfer
4+
permalink: /uw-research-computing/file-availability
5+
---

_uw-research-computing/file-avail-squid.md _uw-research-computing/archived/file-avail-squid.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22
highlighter: none
33
layout: file_avail
44
title: Transfer Large Input Files Via Squid
5+
published: false
56
guide:
6-
category: Manage data
77
tag:
88
- htc
99
---

_uw-research-computing/file-availability.md _uw-research-computing/archived/file-availability.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@ highlighter: none
33
layout: file_avail
44
title: Small Input and Output File Availability Via HTCondor
55
alt_title: Transfer Small Input and Output
6+
published: false
67
guide:
7-
category: Manage data
88
tag:
99
- htc
1010
---

_uw-research-computing/check-quota.md

+40-23
Original file line numberDiff line numberDiff line change
@@ -9,29 +9,25 @@ guide:
99
---
1010

1111
The following commands will allow you to monitor the amount of disk
12-
space you are using in your home directory on our (or another) submit node and to determine the
13-
amount of disk space you have been allotted (your quota).
14-
15-
If you also have a `/staging` directory on the HTC system, see our
16-
[staging guide](file-avail-largedata.html#5-checking-your-quota-data-use-and-file-counts) for
17-
details on how to check your quota and usage.
18-
\
19-
The default quota allotment on CHTC submit nodes is 20 GB with a hard
20-
limit of 30 GB (at which point you cannot write more files).\
21-
\
22-
**Note: The CHTC submit nodes are not backed up, so you will want to
12+
space you are using in your home directory on the access point and to determine the
13+
amount of disk space you have been allotted (your quota).
14+
15+
The default quota allotment in your `/home` directory is 20 GB with a hard
16+
limit of 30 GB (at which point you cannot write more files).
17+
18+
**Note: The CHTC access points are not backed up, so you should
2319
copy completed jobs to a secure location as soon as a batch completes,
2420
and then delete them on the submit node in order to make room for future
25-
jobs.** If you need more disk space to run a single batch or concurrent
26-
batches of jobs, please contact us ([Get Help!](get-help.html)). We have multiple ways of dealing with large disk space
27-
requirements to make things easier for you.
21+
jobs.** Disk space provided is intended for *active* calculations only, not permanent storage.
22+
If you need more disk space to run a single batch or concurrent
23+
batches of jobs, please contact us ([Get Help!](get-help.html)). We have multiple ways of dealing with large disk space requirements to make things easier for you.
2824

2925
If you wish to change your quotas, please see [Request a Quota Change](quota-request).
3026

31-
**1. Checking Your User Quota and Usage**
27+
**1. Checking Your `/home` Quota and Usage**
3228
-------------------------------------
3329

34-
From any directory location within your home directory, type
30+
From any directory location within your `/home` directory, use the command
3531
`quota -vs`. See the example below:
3632

3733
```
@@ -42,18 +38,39 @@ Disk quotas for user alice (uid 20384):
4238
```
4339
{:.term}
4440

45-
The output will list your total data usage under `blocks`, your soft
41+
The output will list your total data usage under `space`, your soft
4642
`quota`, and your hard `limit` at which point your jobs will no longer
47-
be allowed to save data. Each of the values given are in 1-kilobyte
43+
be allowed to save data. Each value is given in 1-kilobyte
4844
blocks, so you can divide each number by 1024 to get megabytes (MB), and
49-
again for gigabytes (GB). (It also lists information for ` files`, but
50-
we don\'t typically allocate disk space by file count.)
45+
again for gigabytes (GB). (It also lists information for number of `files`, but
46+
we don't typically allocate disk space in `/home` by file count.)
47+
48+
**2. Checking Your `/staging` Quota and Usage**
49+
------------------------------------------------
50+
Users may have a `/staging` directory, meant for staging large files and data intended for
51+
job submission. See our [Managing Large Data in HTC Jobs](file-avail-largedata) guide for
52+
more information.
53+
54+
To check your `/staging` quota, use the command `get_quotas /staging/username`.
55+
56+
```
57+
[alice@submit]$ get_quotas /staging/alice
58+
Path Quota(GB) Items Disk_Usage(GB) Items_Usage
59+
/staging/alice 20 5 3.18969 5
60+
```
61+
{:.term}
62+
63+
Your `/staging` directory has a disk and item quota. In the example above, the disk quota is
64+
20 GB, and the items quota is 5 items. The current usage is printed in the following columns;
65+
in the example, the user has used 3.19 GB and 5 items.
66+
67+
To request a quota increase, [fill out our quota request form](quota-request).
5168

52-
**2. Checking the Size of Directories and Contents**
69+
**3. Checking the Size of Directories and Contents**
5370
------------------------------------------------
5471

55-
Move to the directory you\'d like to check and type `du` . After several
56-
moments (longer if you\'re directory contents are large), the command
72+
Move to the directory you'd like to check and type `du` . After several
73+
moments (longer if the contents of your directory are large), the command
5774
will add up the sizes of directory contents and output the total size of
5875
each contained directory in units of kilobytes with the total size of
5976
that directory listed last. See the example below:

0 commit comments

Comments
 (0)