Commit d099bd6d authored by lukas leufen's avatar lukas leufen

Merge branch 'release_v0.11.0' into 'master'

Release v0.11.0

Closes #136, #138, #139, #140, #141, #142, #143, #144, #151, #152

See merge request !135
parents 7f0b5ac2 a15bf606
Pipeline #46089 passed with stages
in 7 minutes and 17 seconds
......@@ -60,7 +60,7 @@ Thumbs.db
htmlcov/
.pytest_cache
/test/data/
/test/test_modules/data/
/test/test_run_modules/data/
report.html
/TestExperiment/
/testrun_network*/
......
# Changelog
All notable changes to this project will be documented in this file.
## v0.11.0 - 2020-08-24 - Advanced Data Handling for MLAir
### general
- Introduce advanced data handling with much more flexibility (independent of TOAR DB, custom data handling is
pluggable), #144
- default data handler is still using TOAR DB
### new features
- default data handler using TOAR DB refactored according to advanced data handling, #140, #141, #152
- data sets are handled as collections, #142, and are itable in a standard way (StandardIterator) and optimised for
keras (KerasIterator), #143
- automatically moving station map plot, #136
### technical
- model modules available from package, #139
- renaming of parameter time dimension, #151
- refactoring of README.md, #138
## v0.10.0 - 2020-07-15 - MLAir is official name, Workflows, easy Model plug-in
### general
......
#!/usr/bin/env bash
# run coverage twice, 1) for html deploy 2) for success evaluation
python3.6 -m pytest --cov=src --cov-report term --cov-report html test/ | tee coverage_results.out
python3.6 -m pytest --cov=mlair --cov-report term --cov-report html test/ | tee coverage_results.out
IS_FAILED=$?
......
MIT License
Copyright (c) 2020 Lukas Leufen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
\ No newline at end of file
......@@ -5,45 +5,213 @@ learning (ML) models for the analysis and forecasting of meteorological and air
# Installation
* Install __proj__ on your machine using the console. E.g. for opensuse / leap `zypper install proj`
* A c++ compiler is required for the installation of the program __cartopy__
* Install all requirements from `requirements.txt` preferably in a virtual environment
* Installation of MLAir:
* Either clone MLAir from its repository in gitlab (link??) and use it without installation
MLAir is based on several python frameworks. To work properly, you have to install all packages from the
`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
packages built for your operating system. Name names of these package may differ for different systems, we refer
here to the opensuse / leap OS. The geo plot can be removed from the `plot_list`, in this case there is no need to
install the geo packages.
* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap `zypper install proj`
* (geo) A c++ compiler is required for the installation of the program **cartopy**
* Install all requirements from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/machinelearningtools/-/blob/master/requirements.txt)
preferably in a virtual environment
* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
device.
* Installation of **MLAir**:
* Either clone MLAir from the [gitlab repository](https://gitlab.version.fz-juelich.de/toar/machinelearningtools.git)
and use it without installation (beside the requirements)
* or download the distribution file (?? .whl) and install it via `pip install <??>`. In this case, you can simply
import MLAir in any python script inside your virtual environment using `import mlair`.
## Special instructions for installation on Jülich HPC systems
import MLAir in any python script inside your virtual environment using `import mlair`.
_Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC
setup files as a skeleton and customise it to your needs._
# How to start with MLAir
The following instruction guide you through the installation on JUWELS and HDFML.
* Clone the repo to HPC system (we recommend to place it in `/p/projects/<project name>`).
* Setup venv by executing `source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for
all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes. <br>
You have to enter the HPC project's budget name (--account flag).
* The default external data path on JUWELS and HDFML is set to `/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`.
<br>To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`:
`data_path=<your>/<custom>/<path>`.
* Execute `python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
* Execute either `sbatch run_juwels_develgpus.bash` or `sbatch run_hdfml_batch.bash` to verify that the setup went well.
* Currently cartopy is not working on our HPC system, therefore PlotStations does not create any output.
In this section, we show three examples how to work with MLAir.
Note: The method `PartitionCheck` currently only checks if the hostname starts with `ju` or `hdfmll`.
Therefore, it might be necessary to adopt the `if` statement in `PartitionCheck._run`.
## Example 1
We start MLAir in a dry run without any modification. Just import mlair and run it.
```python
import mlair
# Security
# just give it a dry run without any modification
mlair.run()
```
The logging output will show you many informations. Additional information (including debug messages) are collected
inside the experiment path in the logging folder.
```log
INFO: mlair started
INFO: ExperimentSetup started
INFO: Experiment path is: /home/<usr>/mlair/testrun_network
...
INFO: load data for DEBW001 from JOIN
...
INFO: Training started
...
INFO: mlair finished after 00:00:12 (hh:mm:ss)
```
* To use hourly data from ToarDB via JOIN interface, a private token is required. Request your personal access token and
add it to `src/join_settings.py` in the hourly data section. Replace the `TOAR_SERVICE_URL` and the `Authorization`
value. To make sure, that this **sensitive** data is not uploaded to the remote server, use the following command to
prevent git from tracking this file: `git update-index --assume-unchanged src/join_settings.py`
## Example 2
Now we update the stations and customise the window history size parameter.
```python
import mlair
# our new stations to use
stations = ['DEBW030', 'DEBW037', 'DEBW031', 'DEBW015', 'DEBW107']
# expanded temporal context to 14 (days, because of default sampling="daily")
window_history_size = 14
# restart the experiment with little customisation
mlair.run(stations=stations,
window_history_size=window_history_size)
```
The output looks similar, but we can see, that the new stations are loaded.
```log
INFO: mlair started
INFO: ExperimentSetup started
...
INFO: load data for DEBW030 from JOIN
INFO: load data for DEBW037 from JOIN
...
INFO: Training started
...
INFO: mlair finished after 00:00:24 (hh:mm:ss)
```
## Example 3
Let's just apply our trained model to new data. Therefore we keep the window history size parameter but change the stations.
In the run method, we need to disable the trainable and create new model parameters. MLAir will use the model we have
trained before. Note, this only works if the experiment path has not changed or a suitable trained model is placed
inside the experiment path.
```python
import mlair
# our new stations to use
stations = ['DEBY002', 'DEBY079']
# same setting for window_history_size
window_history_size = 14
# run experiment without training
mlair.run(stations=stations,
window_history_size=window_history_size,
create_new_model=False,
trainable=False)
```
We can see from the terminal that no training was performed. Analysis is now made on the new stations.
```log
INFO: mlair started
...
INFO: No training has started, because trainable parameter was false.
...
INFO: mlair finished after 00:00:06 (hh:mm:ss)
```
# Customised workflows and models
# Custom Workflow
MLAir provides a default workflow. If additional steps are to be performed, you have to append custom run modules to
the workflow.
```python
import mlair
import logging
# Customise your experiment
class CustomStage(mlair.RunEnvironment):
"""A custom MLAir stage for demonstration."""
def __init__(self, test_string):
super().__init__() # always call super init method
self._run(test_string) # call a class method
def _run(self, test_string):
logging.info("Just running a custom stage.")
logging.info("test_string = " + test_string)
epochs = self.data_store.get("epochs")
logging.info("epochs = " + str(epochs))
# create your custom MLAir workflow
CustomWorkflow = mlair.Workflow()
# provide stages without initialisation
CustomWorkflow.add(mlair.ExperimentSetup, epochs=128)
# add also keyword arguments for a specific stage
CustomWorkflow.add(CustomStage, test_string="Hello World")
# finally execute custom workflow in order of adding
CustomWorkflow.run()
```
```log
INFO: mlair started
...
INFO: ExperimentSetup finished after 00:00:12 (hh:mm:ss)
INFO: CustomStage started
INFO: Just running a custom stage.
INFO: test_string = Hello World
INFO: epochs = 128
INFO: CustomStage finished after 00:00:01 (hh:mm:ss)
INFO: mlair finished after 00:00:13 (hh:mm:ss)
```
## Custom Model
Each model has to inherit from the abstract model class to ensure a smooth training and evaluation behaviour. It is
required to implement the set model and set compile options methods. The later has to set the loss at least.
```python
import keras
from keras.losses import mean_squared_error as mse
from keras.optimizers import SGD
from mlair.model_modules import AbstractModelClass
class MyLittleModel(AbstractModelClass):
"""
A customised model with a 1x1 Conv, and 3 Dense layers (32, 16
window_lead_time). Dropout is used after Conv layer.
"""
def __init__(self, window_history_size, window_lead_time, channels):
super().__init__()
# settings
self.window_history_size = window_history_size
self.window_lead_time = window_lead_time
self.channels = channels
self.dropout_rate = 0.1
self.activation = keras.layers.PReLU
self.lr = 1e-2
# apply to model
self.set_model()
self.set_compile_options()
self.set_custom_objects(loss=self.compile_options['loss'])
def set_model(self):
# add 1 to window_size to include current time step t0
shape = (self.window_history_size + 1, 1, self.channels)
x_input = keras.layers.Input(shape=shape)
x_in = keras.layers.Conv2D(32, (1, 1), padding='same')(x_input)
x_in = self.activation()(x_in)
x_in = keras.layers.Flatten()(x_in)
x_in = keras.layers.Dropout(self.dropout_rate)(x_in)
x_in = keras.layers.Dense(32)(x_in)
x_in = self.activation()(x_in)
x_in = keras.layers.Dense(16)(x_in)
x_in = self.activation()(x_in)
x_in = keras.layers.Dense(self.window_lead_time)(x_in)
out = self.activation()(x_in)
self.model = keras.Model(inputs=x_input, outputs=[out])
def set_compile_options(self):
self.compile_options = {"optimizer": SGD(lr=self.lr),
"loss": mse,
"metrics": ["mse"]}
```
This section summarises which parameters can be customised for a training.
## Transformation
......@@ -98,8 +266,34 @@ scaling values instead of the calculation method. For method *centre*, std can s
class: `xr.DataArray` with `dims=["variables"]` and one value for each variable.
## Inception Model
See a description [here](https://towardsdatascience.com/a-simple-guide-to-the-versions-of-the-inception-network-7fc52b863202)
or take a look on the papers [Going Deeper with Convolutions (Szegedy et al., 2014)](https://arxiv.org/abs/1409.4842)
and [Network In Network (Lin et al., 2014)](https://arxiv.org/abs/1312.4400).
# Special Remarks
## Special instructions for installation on Jülich HPC systems
_Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC
setup files as a skeleton and customise it to your needs._
The following instruction guide you through the installation on JUWELS and HDFML.
* Clone the repo to HPC system (we recommend to place it in `/p/projects/<project name>`).
* Setup venv by executing `source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for
all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes. <br>
You have to enter the HPC project's budget name (--account flag).
* The default external data path on JUWELS and HDFML is set to `/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`.
<br>To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`:
`data_path=<your>/<custom>/<path>`.
* Execute `python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
* Execute either `sbatch run_juwels_develgpus.bash` or `sbatch run_hdfml_batch.bash` to verify that the setup went well.
* Currently cartopy is not working on our HPC system, therefore PlotStations does not create any output.
Note: The method `PartitionCheck` currently only checks if the hostname starts with `ju` or `hdfmll`.
Therefore, it might be necessary to adopt the `if` statement in `PartitionCheck._run`.
## Security using JOIN
* To use hourly data from ToarDB via JOIN interface, a private token is required. Request your personal access token and
add it to `src/join_settings.py` in the hourly data section. Replace the `TOAR_SERVICE_URL` and the `Authorization`
value. To make sure, that this **sensitive** data is not uploaded to the remote server, use the following command to
prevent git from tracking this file: `git update-index --assume-unchanged src/join_settings.py`
......@@ -17,7 +17,7 @@ sys.path.insert(0, os.path.abspath('../..'))
# -- Project information -----------------------------------------------------
project = 'machinelearningtools'
project = 'MLAir'
copyright = '2020, Lukas H Leufen, Felix Kleinert'
author = 'Lukas H Leufen, Felix Kleinert'
......@@ -118,7 +118,7 @@ latex_elements = {
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'machinelearningtools.tex', 'MachineLearningTools Documentation',
(master_doc, 'mlair.tex', 'MLAir Documentation',
author, 'manual'),
]
......
Get started with MachineLearningTools
=====================================
Get started with MLAir
======================
<what is machinelearningtools?>
Install MLAir
-------------
MLT Module and Funtion Documentation
------------------------------------
MLAir is based on several python frameworks. To work properly, you have to install all packages from the
`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
packages built for your operating system. Name names of these package may differ for different systems, we refer
here to the opensuse / leap OS. The geo plot can be removed from the `plot_list`, in this case there is no need to
install the geo packages.
Install MachineLearningTools
----------------------------
* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap `zypper install proj`
* (geo) A c++ compiler is required for the installation of the program **cartopy**
* Install all requirements from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/machinelearningtools/-/blob/master/requirements.txt)
preferably in a virtual environment
* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
device.
* Installation of **MLAir**:
* Either clone MLAir from the [gitlab repository](https://gitlab.version.fz-juelich.de/toar/machinelearningtools.git)
and use it without installation (beside the requirements)
* or download the distribution file (?? .whl) and install it via `pip install <??>`. In this case, you can simply
import MLAir in any python script inside your virtual environment using `import mlair`.
Dependencies
How to start with MLAir
-----------------------
In this section, we show three examples how to work with MLAir.
Example 1
~~~~~~~~~
We start MLAir in a dry run without any modification. Just import mlair and run it.
.. code-block:: python
import mlair
# just give it a dry run without any modification
mlair.run()
The logging output will show you many informations. Additional information (including debug messages) are collected
inside the experiment path in the logging folder.
.. code-block::
INFO: mlair started
INFO: ExperimentSetup started
INFO: Experiment path is: /home/<usr>/mlair/testrun_network
...
INFO: load data for DEBW001 from JOIN
...
INFO: Training started
...
INFO: mlair finished after 00:00:12 (hh:mm:ss)
Example 2
~~~~~~~~~
Now we update the stations and customise the window history size parameter.
.. code-block:: python
import mlair
# our new stations to use
stations = ['DEBW030', 'DEBW037', 'DEBW031', 'DEBW015', 'DEBW107']
# expanded temporal context to 14 (days, because of default sampling="daily")
window_history_size = 14
# restart the experiment with little customisation
mlair.run(stations=stations,
window_history_size=window_history_size)
The output looks similar, but we can see, that the new stations are loaded.
.. code-block::
INFO: mlair started
INFO: ExperimentSetup started
...
INFO: load data for DEBW030 from JOIN
INFO: load data for DEBW037 from JOIN
...
INFO: Training started
...
INFO: mlair finished after 00:00:24 (hh:mm:ss)
Example 3
~~~~~~~~~
Let's just apply our trained model to new data. Therefore we keep the window history size parameter but change the stations.
In the run method, we need to disable the trainable and create new model parameters. MLAir will use the model we have
trained before. Note, this only works if the experiment path has not changed or a suitable trained model is placed
inside the experiment path.
.. code-block:: python
import mlair
# our new stations to use
stations = ['DEBY002', 'DEBY079']
# same setting for window_history_size
window_history_size = 14
# run experiment without training
mlair.run(stations=stations,
window_history_size=window_history_size,
create_new_model=False,
trainable=False)
We can see from the terminal that no training was performed. Analysis is now made on the new stations.
.. code-block::
INFO: mlair started
...
INFO: No training has started, because trainable parameter was false.
...
INFO: mlair finished after 00:00:06 (hh:mm:ss)
Customised workflows and models
-------------------------------
Custom Workflow
~~~~~~~~~~~~~~~
MLAir provides a default workflow. If additional steps are to be performed, you have to append custom run modules to
the workflow.
.. code-block:: python
import mlair
import logging
class CustomStage(mlair.RunEnvironment):
"""A custom MLAir stage for demonstration."""
def __init__(self, test_string):
super().__init__() # always call super init method
self._run(test_string) # call a class method
def _run(self, test_string):
logging.info("Just running a custom stage.")
logging.info("test_string = " + test_string)
epochs = self.data_store.get("epochs")
logging.info("epochs = " + str(epochs))
# create your custom MLAir workflow
CustomWorkflow = mlair.Workflow()
# provide stages without initialisation
CustomWorkflow.add(mlair.ExperimentSetup, epochs=128)
# add also keyword arguments for a specific stage
CustomWorkflow.add(CustomStage, test_string="Hello World")
# finally execute custom workflow in order of adding
CustomWorkflow.run()
.. code-block::
INFO: mlair started
...
INFO: ExperimentSetup finished after 00:00:12 (hh:mm:ss)
INFO: CustomStage started
INFO: Just running a custom stage.
INFO: test_string = Hello World
INFO: epochs = 128
INFO: CustomStage finished after 00:00:01 (hh:mm:ss)
INFO: mlair finished after 00:00:13 (hh:mm:ss)
Custom Model
~~~~~~~~~~~~
Data
~~~~
Each model has to inherit from the abstract model class to ensure a smooth training and evaluation behaviour. It is
required to implement the set model and set compile options methods. The later has to set the loss at least.
.. code-block:: python
import keras
from keras.losses import mean_squared_error as mse
from keras.optimizers import SGD
from mlair.model_modules import AbstractModelClass
class MyLittleModel(AbstractModelClass):
"""
A customised model with a 1x1 Conv, and 3 Dense layers (32, 16
window_lead_time). Dropout is used after Conv layer.
"""
def __init__(self, window_history_size, window_lead_time, channels):
super().__init__()
# settings
self.window_history_size = window_history_size
self.window_lead_time = window_lead_time
self.channels = channels
self.dropout_rate = 0.1
self.activation = keras.layers.PReLU
self.lr = 1e-2
# apply to model
self.set_model()
self.set_compile_options()
self.set_custom_objects(loss=self.compile_options['loss'])
def set_model(self):
# add 1 to window_size to include current time step t0
shape = (self.window_history_size + 1, 1, self.channels)
x_input = keras.layers.Input(shape=shape)
x_in = keras.layers.Conv2D(32, (1, 1), padding='same')(x_input)
x_in = self.activation()(x_in)
x_in = keras.layers.Flatten()(x_in)
x_in = keras.layers.Dropout(self.dropout_rate)(x_in)
x_in = keras.layers.Dense(32)(x_in)
x_in = self.activation()(x_in)
x_in = keras.layers.Dense(16)(x_in)
x_in = self.activation()(x_in)
x_in = keras.layers.Dense(self.window_lead_time)(x_in)
out = self.activation()(x_in)
self.model = keras.Model(inputs=x_input, outputs=[out])
def set_compile_options(self):
self.compile_options = {"optimizer": SGD(lr=self.lr),
"loss": mse,
"metrics": ["mse"]}
__version_info__ = {
'major': 0,
'minor': 10,
'minor': 11,
'micro': 0,
}
from mlair.run_modules import *
from mlair.run_modules import RunEnvironment, ExperimentSetup, PreProcessing, ModelSetup, Training, PostProcessing
from mlair.workflows import DefaultWorkflow, Workflow
from mlair.run_script import run
from mlair.model_modules import AbstractModelClass
def get_version():
......
......@@ -13,7 +13,8 @@ DEFAULT_START = "1997-01-01"
DEFAULT_END = "2017-12-31"
DEFAULT_WINDOW_HISTORY_SIZE = 13
DEFAULT_OVERWRITE_LOCAL_DATA = False
DEFAULT_TRANSFORMATION = {"scope": "data", "method": "standardise", "mean": "estimate"}
# DEFAULT_TRANSFORMATION = {"scope": "data", "method": "standardise", "mean": "estimate"}
DEFAULT_TRANSFORMATION = {"scope": "data", "method": "standardise"}
DEFAULT_HPC_LOGIN_LIST = ["ju", "hdfmll"] # ju[wels} #hdfmll(ogin)
DEFAULT_HPC_HOST_LIST = ["jw", "hdfmlc"] # first part of node names for Juwels (jw[comp], hdfmlc(ompute).