Skip to content

Commit 20b722a

Browse files
tennleetennlee
committed
Simplify and standardise the README for each sub-package
1 parent b65f84b commit 20b722a

5 files changed

Lines changed: 17 additions & 118 deletions

File tree

‎packages/data/README.md‎

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,6 @@
1-
# Environmental Data Intelligence Toolkit (pyearthtools) Data Indexing
1+
# PyEarthTools Data Indexing
22

3-
Data tools to support and reduce the complexity of accessing common Earth System Datasets.
3+
This is the data sub-package which forms a part of the [PyEarthTools package](https://github.com/ACCESS-Community-Hub/PyEarthTools).
44

5-
**Continue to the [documentation site](https://git.nci.org.au/bom/dset/pyearthtools-package/documentation).**
5+
Documentation for this package is maintained at the [documentation site](https://pyearthtools.readthedocs.io/en/latest/).
66

7-
## Adding archives
8-
9-
`pyearthtools` itself contains no archives, and rely's on extra addons to include them, currently only an NCI addon is available, at [NCI](https://git.nci.org.au/bom/dset/pyearthtools-package/archives/nci.git)
10-
11-
For examples, and documentation on the use of these tools, please see `/Examples/`.

‎packages/pipeline/README.md‎

Lines changed: 3 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,5 @@
1-
# Pipeline Version 2
1+
# PyEarthTools Pipeline
22

3-
A rework of the pipeline setup to improve parallelisation and overall usability.
3+
This is the pipeline sub-package which forms a part of the [PyEarthTools package](https://github.com/ACCESS-Community-Hub/PyEarthTools).
44

5-
Integrated fully with `pyearthtools.data` to build pipelines of operations applied to data.
6-
7-
## Examples
8-
9-
A pipeline should be a sequence of steps, with optional branches.
10-
11-
Here is an example pipeline, for use with PanguWeather.
12-
13-
```python
14-
15-
import pyearthtools.data
16-
import pyearthtools.pipeline
17-
18-
19-
data_preperation = pyearthtools.pipeline.Pipeline(
20-
(
21-
pyearthtools.data.archive.ERA5(['msl', '10u', '10v', '2t']),
22-
pyearthtools.data.archive.ERA5(['z', 'q', 't', 'u', 'v'], level_value = [50, 100, 150, 200, 250, 300, 400, 500, 600, 700, 850, 925, 1000])
23-
),
24-
pyearthtools.pipeline.operations.xarray.Merge(),
25-
pyearthtools.pipeline.operations.xarray.Sort(['msl', '10u', '10v', '2t', 'z', 'q', 't', 'u', 'v']),
26-
pyearthtools.pipeline.operations.Transforms(
27-
apply = pyearthtools.data.transforms.coordinates.standard_longitude(type = '0-360') + pyearthtools.data.transforms.coordinates.ReIndex(level = 'reversed')
28-
),
29-
pyearthtools.pipeline.operations.xarray.reshape.CoordinateFlatten('level'),
30-
pyearthtools.pipeline.operations.xarray.conversion.ToNumpy(),
31-
pyearthtools.pipeline.operations.numpy.reshape.Squish(1),
32-
)
33-
```
34-
35-
This pipeline can then be viewed as a graph
36-
37-
```python
38-
data_preperation.graph()
39-
```
40-
41-
![Pipeline Graph](./assets/pipeline_example.svg)
42-
43-
## Installation
44-
45-
### Pypi
46-
47-
```shell
48-
pip install pyearthtools-pipeline
49-
```
50-
51-
### On Gadi
52-
53-
On gadi prebuilt modules exist
54-
55-
```shell
56-
module use /scratch/ra02/modules
57-
module load pyearthtools
58-
```
5+
Documentation for this package is maintained at the [documentation site](https://pyearthtools.readthedocs.io/en/latest/).

‎packages/training/README.md‎

Lines changed: 3 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,5 @@
1-
# pyearthtools Training
1+
# PyEarthTools Training
22

3-
Using `pyearthtools.data` this package aims to significantly speed up ML model investigations and experimentations.
3+
This is the training sub-package which forms a part of the [PyEarthTools package](https://github.com/ACCESS-Community-Hub/PyEarthTools).
44

5-
With `pyearthtools.pipeline` a data stram can be configured to prepare the data for training or prediction. Once the model is wrapped in a `Wrapper`, it can be trained upon, or connected with a `Predictor` for inference.
6-
7-
## Available Frameworks
8-
9-
As wrappers are needed to connect the model to the rest of the system only the following frameworks are available, but it would be easy to add more,
10-
11-
| Name | Description |
12-
| ---- | ----------- |
13-
| Lightning | Pytorch Lightning - Managed pytorch code |
14-
| ONNX | Complied models for prediction |
15-
| XGBoost | Gradient boosted Trees |
16-
17-
18-
## Usage
19-
20-
First create the datamodule
21-
22-
```python
23-
import pyearthtools.training
24-
import pyearthtools.pipeline
25-
26-
datamodule = pyearthtools.training.data.default.PipelineDefaultDataModule(
27-
pyearthtools.pipeline.Pipeline.sample(),
28-
train_split = pyearthtools.pipeline.iterators.DateRange('2000-01-01T00', '2000-02-01T00', '1 day')
29-
)
30-
# Training Mode
31-
datamodule.train()
32-
```
33-
34-
Then create the model, and the model wrapper
35-
36-
```python
37-
38-
model = MODEL_GOES_HERE
39-
model_wrapper = pyearthtools.training.wrapper.FRAMEWORK.Wrapper(model, datamodule)
40-
```
41-
42-
If the framework supports it, run training
43-
44-
```python
45-
model_wrapper.fit()
46-
```
47-
48-
Or if only predictions are available, connect with a Prediction Wrapper
49-
```python
50-
predictor = pyearthtools.training.wrapper.predict.PREDICTION_TYPE(model_wrapper)
51-
predictor.predict('2000-01-01T00')
52-
```
5+
Documentation for this package is maintained at the [documentation site](https://pyearthtools.readthedocs.io/en/latest/).

‎packages/tutorial/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1-
# PyEarthTools tutorials
1+
# PyEarthTools Tutorials
22

3-
Supporting code for `PyEarthTools` [tutorial notebooks](../../notebooks/tutorial/).
3+
This is the tutorial sub-package which forms a part of the [PyEarthTools package](https://github.com/ACCESS-Community-Hub/PyEarthTools).
4+
5+
Documentation for this package is maintained at the [documentation site](https://pyearthtools.readthedocs.io/en/latest/).

‎packages/utils/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1-
# pyearthtools Utilities
1+
# PyEarthTools Utils
22

3-
A common source for commonly used utility functions.
3+
This is the utils sub-package which forms a part of the [PyEarthTools package](https://github.com/ACCESS-Community-Hub/PyEarthTools).
4+
5+
Documentation for this package is maintained at the [documentation site](https://pyearthtools.readthedocs.io/en/latest/).

0 commit comments

Comments
 (0)