Commit 80665641 authored by lukas leufen's avatar lukas leufen

dev is now included

parents a0d2a6cc 0d9ea05d
Pipeline #41956 passed with stages
in 5 minutes and 39 seconds
......@@ -73,7 +73,7 @@ report.html
# secret variables #
####################
/src/configuration/join_settings.py
/mlair/configuration/join_settings.py
# ignore locally build documentation #
######################################
......
# Changelog
All notable changes to this project will be documented in this file.
## v0.10.0 - 2020-07-15 - MLAir is official name, Workflows, easy Model plug-in
### general
- Official project name is released: MLAir (Machine Learning on Air data)
- a model class can now easily be plugged in into MLAir. #121
- introduced new concept of workflows, #134
### new features
- workflows are used to execute a sequence of run modules, #134
- default workflows for standard and the Juelich HPC systems are available, custom workflows can be defined, #134
- seasonal decomposition is available for conditional quantile plot, #112
- map plot is created with coordinates, #108
- `flatten_tails` are now more general and easier to customise, #114
- model classes have custom compile options (replaces `set_loss`), #110
- model can be set in ExperimentSetup from outside, #121
- default experiment settings can be queried using `get_defaults()`, #123
- training and model settings are reported as MarkDown and Tex tables, #145
### technical
- Juelich HPC systems are supported and installation scripts are available, #106
- data store is tracked, I/O is saved and illustrated in a plot, #116
- batch size, epoch parameter have to be defined in ExperimentSetup, #127, #122
- automatic documentation with sphinx, #109
- default experiment settings are updated, #123
- refactoring of experiment path and its default naming, #124
- refactoring of some parameter names, #146
- preparation for package distribution with pip, #119
- all run scripts are updated to run with workflows, #134
- the experiment folder is restructured, #130
## v0.9.0 - 2020-04-15 - faster bootstraps, extreme value upsamling
### general
- improved and faster bootstrap workflow
......
MIT License
Copyright (c) 2020 Lukas Leufen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
\ No newline at end of file
This diff is collapsed.
......@@ -17,7 +17,7 @@ sys.path.insert(0, os.path.abspath('../..'))
# -- Project information -----------------------------------------------------
project = 'machinelearningtools'
project = 'MLAir'
copyright = '2020, Lukas H Leufen, Felix Kleinert'
author = 'Lukas H Leufen, Felix Kleinert'
......@@ -55,7 +55,7 @@ extensions = [
autosummary_generate = True
autoapi_type = 'python'
autoapi_dirs = ['../../src/.']
autoapi_dirs = ['../../mlair/.']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
......@@ -118,7 +118,7 @@ latex_elements = {
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'machinelearningtools.tex', 'MachineLearningTools Documentation',
(master_doc, 'mlair.tex', 'MLAir Documentation',
author, 'manual'),
]
......
Get started with MachineLearningTools
=====================================
Get started with MLAir
======================
<what is machinelearningtools?>
Install MLAir
-------------
MLT Module and Funtion Documentation
------------------------------------
MLAir is based on several python frameworks. To work properly, you have to install all packages from the
`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
packages built for your operating system. Name names of these package may differ for different systems, we refer
here to the opensuse / leap OS. The geo plot can be removed from the `plot_list`, in this case there is no need to
install the geo packages.
Install MachineLearningTools
----------------------------
* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap `zypper install proj`
* (geo) A c++ compiler is required for the installation of the program **cartopy**
* Install all requirements from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/machinelearningtools/-/blob/master/requirements.txt)
preferably in a virtual environment
* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
device.
* Installation of **MLAir**:
* Either clone MLAir from the [gitlab repository](https://gitlab.version.fz-juelich.de/toar/machinelearningtools.git)
and use it without installation (beside the requirements)
* or download the distribution file (?? .whl) and install it via `pip install <??>`. In this case, you can simply
import MLAir in any python script inside your virtual environment using `import mlair`.
Dependencies
How to start with MLAir
-----------------------
In this section, we show three examples how to work with MLAir.
Example 1
~~~~~~~~~
We start MLAir in a dry run without any modification. Just import mlair and run it.
.. code-block:: python
import mlair
# just give it a dry run without any modification
mlair.run()
The logging output will show you many informations. Additional information (including debug messages) are collected
inside the experiment path in the logging folder.
.. code-block::
INFO: mlair started
INFO: ExperimentSetup started
INFO: Experiment path is: /home/<usr>/mlair/testrun_network
...
INFO: load data for DEBW001 from JOIN
...
INFO: Training started
...
INFO: mlair finished after 00:00:12 (hh:mm:ss)
Example 2
~~~~~~~~~
Now we update the stations and customise the window history size parameter.
.. code-block:: python
import mlair
# our new stations to use
stations = ['DEBW030', 'DEBW037', 'DEBW031', 'DEBW015', 'DEBW107']
# expanded temporal context to 14 (days, because of default sampling="daily")
window_history_size = 14
# restart the experiment with little customisation
mlair.run(stations=stations,
window_history_size=window_history_size)
The output looks similar, but we can see, that the new stations are loaded.
.. code-block::
INFO: mlair started
INFO: ExperimentSetup started
...
INFO: load data for DEBW030 from JOIN
INFO: load data for DEBW037 from JOIN
...
INFO: Training started
...
INFO: mlair finished after 00:00:24 (hh:mm:ss)
Example 3
~~~~~~~~~
Let's just apply our trained model to new data. Therefore we keep the window history size parameter but change the stations.
In the run method, we need to disable the trainable and create new model parameters. MLAir will use the model we have
trained before. Note, this only works if the experiment path has not changed or a suitable trained model is placed
inside the experiment path.
.. code-block:: python
import mlair
# our new stations to use
stations = ['DEBY002', 'DEBY079']
# same setting for window_history_size
window_history_size = 14
# run experiment without training
mlair.run(stations=stations,
window_history_size=window_history_size,
create_new_model=False,
trainable=False)
We can see from the terminal that no training was performed. Analysis is now made on the new stations.
.. code-block::
INFO: mlair started
...
INFO: No training has started, because trainable parameter was false.
...
INFO: mlair finished after 00:00:06 (hh:mm:ss)
Customised workflows and models
-------------------------------
Custom Workflow
~~~~~~~~~~~~~~~
MLAir provides a default workflow. If additional steps are to be performed, you have to append custom run modules to
the workflow.
.. code-block:: python
import mlair
import logging
class CustomStage(mlair.RunEnvironment):
"""A custom MLAir stage for demonstration."""
def __init__(self, test_string):
super().__init__() # always call super init method
self._run(test_string) # call a class method
def _run(self, test_string):
logging.info("Just running a custom stage.")
logging.info("test_string = " + test_string)
epochs = self.data_store.get("epochs")
logging.info("epochs = " + str(epochs))
# create your custom MLAir workflow
CustomWorkflow = mlair.Workflow()
# provide stages without initialisation
CustomWorkflow.add(mlair.ExperimentSetup, epochs=128)
# add also keyword arguments for a specific stage
CustomWorkflow.add(CustomStage, test_string="Hello World")
# finally execute custom workflow in order of adding
CustomWorkflow.run()
.. code-block::
INFO: mlair started
...
INFO: ExperimentSetup finished after 00:00:12 (hh:mm:ss)
INFO: CustomStage started
INFO: Just running a custom stage.
INFO: test_string = Hello World
INFO: epochs = 128
INFO: CustomStage finished after 00:00:01 (hh:mm:ss)
INFO: mlair finished after 00:00:13 (hh:mm:ss)
Custom Model
~~~~~~~~~~~~
Data
~~~~
Each model has to inherit from the abstract model class to ensure a smooth training and evaluation behaviour. It is
required to implement the set model and set compile options methods. The later has to set the loss at least.
.. code-block:: python
import keras
from keras.losses import mean_squared_error as mse
from keras.optimizers import SGD
from mlair.model_modules import AbstractModelClass
class MyLittleModel(AbstractModelClass):
"""
A customised model with a 1x1 Conv, and 3 Dense layers (32, 16
window_lead_time). Dropout is used after Conv layer.
"""
def __init__(self, window_history_size, window_lead_time, channels):
super().__init__()
# settings
self.window_history_size = window_history_size
self.window_lead_time = window_lead_time
self.channels = channels
self.dropout_rate = 0.1
self.activation = keras.layers.PReLU
self.lr = 1e-2
# apply to model
self.set_model()
self.set_compile_options()
self.set_custom_objects(loss=self.compile_options['loss'])
def set_model(self):
# add 1 to window_size to include current time step t0
shape = (self.window_history_size + 1, 1, self.channels)
x_input = keras.layers.Input(shape=shape)
x_in = keras.layers.Conv2D(32, (1, 1), padding='same')(x_input)
x_in = self.activation()(x_in)
x_in = keras.layers.Flatten()(x_in)
x_in = keras.layers.Dropout(self.dropout_rate)(x_in)
x_in = keras.layers.Dense(32)(x_in)
x_in = self.activation()(x_in)
x_in = keras.layers.Dense(16)(x_in)
x_in = self.activation()(x_in)
x_in = keras.layers.Dense(self.window_lead_time)(x_in)
out = self.activation()(x_in)
self.model = keras.Model(inputs=x_input, outputs=[out])
def set_compile_options(self):
self.compile_options = {"optimizer": SGD(lr=self.lr),
"loss": mse,
"metrics": ["mse"]}
__version_info__ = {
'major': 0,
'minor': 9,
'minor': 10,
'micro': 0,
}
from src.run_modules import *
from src.workflows import DefaultWorkflow, Workflow
from mlair.run_modules import RunEnvironment, ExperimentSetup, PreProcessing, ModelSetup, Training, PostProcessing
from mlair.workflows import DefaultWorkflow, Workflow
from mlair.run_script import run
from mlair.model_modules import AbstractModelClass
def get_version():
......
......@@ -29,8 +29,8 @@ DEFAULT_TARGET_VAR = "o3"
DEFAULT_TARGET_DIM = "variables"
DEFAULT_WINDOW_LEAD_TIME = 3
DEFAULT_DIMENSIONS = {"new_index": ["datetime", "Stations"]}
DEFAULT_INTERPOLATE_DIM = "datetime"
DEFAULT_INTERPOLATE_METHOD = "linear"
DEFAULT_INTERPOLATION_DIM = "datetime"
DEFAULT_INTERPOLATION_METHOD = "linear"
DEFAULT_LIMIT_NAN_FILL = 1
DEFAULT_TRAIN_START = "1997-01-01"
DEFAULT_TRAIN_END = "2007-12-31"
......
......@@ -6,7 +6,8 @@ import re
import socket
from typing import Tuple
ROOT_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
# ROOT_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
ROOT_PATH = os.getcwd()
def prepare_host(create_new=True, data_path=None, sampling="daily") -> str:
......@@ -23,35 +24,38 @@ def prepare_host(create_new=True, data_path=None, sampling="daily") -> str:
:return: full path to data
"""
hostname = get_host()
user = getpass.getuser()
runner_regex = re.compile(r"runner-.*-project-2411-concurrent-\d+")
if hostname == "ZAM144":
path = f"/home/{user}/Data/toar_{sampling}/"
elif hostname == "zam347":
path = f"/home/{user}/Data/toar_{sampling}/"
elif hostname == "linux-aa9b":
path = f"/home/{user}/mlair/data/toar_{sampling}/"
elif (len(hostname) > 2) and (hostname[:2] == "jr"):
path = f"/p/project/cjjsc42/{user}/DATA/toar_{sampling}/"
elif (len(hostname) > 2) and (hostname[:2] in ['jw', 'ju'] or hostname[:5] in ['hdfml']):
path = f"/p/project/deepacf/intelliaq/{user}/DATA/toar_{sampling}/"
elif runner_regex.match(hostname) is not None:
path = f"/home/{user}/mlair/data/toar_{sampling}/"
else:
raise OSError(f"unknown host '{hostname}'")
if not os.path.exists(path):
if data_path is None:
hostname = get_host()
user = getpass.getuser()
runner_regex = re.compile(r"runner-.*-project-2411-concurrent-\d+")
if hostname == "ZAM144":
data_path = f"/home/{user}/Data/toar_{sampling}/"
elif hostname == "zam347":
data_path = f"/home/{user}/Data/toar_{sampling}/"
elif hostname == "linux-aa9b":
data_path = f"/home/{user}/mlair/data/toar_{sampling}/"
elif (len(hostname) > 2) and (hostname[:2] == "jr"):
data_path = f"/p/project/cjjsc42/{user}/DATA/toar_{sampling}/"
elif (len(hostname) > 2) and (hostname[:2] in ['jw', 'ju'] or hostname[:5] in ['hdfml']):
data_path = f"/p/project/deepacf/intelliaq/{user}/DATA/toar_{sampling}/"
elif runner_regex.match(hostname) is not None:
data_path = f"/home/{user}/mlair/data/toar_{sampling}/"
else:
data_path = os.path.join(os.getcwd(), "data", sampling)
# raise OSError(f"unknown host '{hostname}'")
if not os.path.exists(data_path):
try:
if create_new:
check_path_and_create(path)
return path
check_path_and_create(data_path)
return data_path
else:
raise PermissionError
except PermissionError:
raise NotADirectoryError(f"path '{path}' does not exist for host '{hostname}'.")
raise NotADirectoryError(f"path '{data_path}' does not exist for host '{hostname}'.")
else:
logging.debug(f"set path to: {path}")
return path
logging.debug(f"set path to: {data_path}")
return data_path
def set_experiment_path(name: str, path: str = None) -> str:
......
......@@ -3,7 +3,7 @@ __author__ = 'Lukas Leufen'
__date__ = '2020-07-08'
from src.helpers import to_list, remove_items
from mlair.helpers import to_list, remove_items
import numpy as np
import xarray as xr
import pickle
......@@ -17,8 +17,8 @@ import copy
from typing import Union, List, Tuple, Dict
import logging
from functools import reduce
from src.data_handler.station_preparation import StationPrep
from src.helpers.join import EmptyQueryResult
from mlair.data_handler.station_preparation import StationPrep
from mlair.helpers.join import EmptyQueryResult
number = Union[float, int]
......
......@@ -19,7 +19,7 @@ from itertools import chain
import numpy as np
import xarray as xr
from src.data_handler.advanced_data_handler import AbstractDataPreparation
from mlair.data_handler.advanced_data_handler import AbstractDataPreparation
class BootstrapIterator(Iterator):
......
......@@ -3,9 +3,9 @@ __author__ = 'Lukas Leufen'
__date__ = '2020-07-17'
from src.helpers import to_list
from src.data_handler.station_preparation import StationPrep
from src.data_handler.advanced_data_handler import DefaultDataPreparation
from mlair.helpers import to_list
from mlair.data_handler.station_preparation import StationPrep
from mlair.data_handler.advanced_data_handler import DefaultDataPreparation
import os
from typing import Union, List
......@@ -53,7 +53,7 @@ if __name__ == "__main__":
"sampling": 'daily',
"target_dim": 'variables',
"target_var": 'o3',
"interpolate_dim": 'datetime',
"interpolation_dim": 'datetime',
"window_history_size": 7,
"window_lead_time": 3,
"neighbors": ["DEBW034"],
......
......@@ -65,7 +65,6 @@ class DataCollection(Iterable):
return list(self._mapping.keys())
class KerasIterator(keras.utils.Sequence):
def __init__(self, collection: DataCollection, batch_size: int, batch_path: str, shuffle_batches: bool = False,
......
......@@ -13,9 +13,9 @@ import numpy as np
import pandas as pd
import xarray as xr
from src.configuration import check_path_and_create
from src import helpers
from src.helpers import join, statistics
from mlair.configuration import check_path_and_create
from mlair import helpers
from mlair.helpers import join, statistics
# define a more general date type for type hinting
date = Union[dt.date, dt.datetime]
......@@ -39,7 +39,7 @@ class AbstractStationPrep(object):
class StationPrep(AbstractStationPrep):
def __init__(self, station, data_path, statistics_per_var, station_type, network, sampling,
target_dim, target_var, interpolate_dim, window_history_size, window_lead_time,
target_dim, target_var, interpolation_dim, window_history_size, window_lead_time,
overwrite_local_data: bool = False, transformation=None, store_data_locally: bool = True,
min_length: int = 0, start=None, end=None, **kwargs):
super().__init__() # path, station, statistics_per_var, transformation, **kwargs)
......@@ -53,7 +53,7 @@ class StationPrep(AbstractStationPrep):
self.sampling = sampling
self.target_dim = target_dim
self.target_var = target_var
self.interpolate_dim = interpolate_dim
self.interpolation_dim = interpolation_dim
self.window_history_size = window_history_size
self.window_lead_time = window_lead_time
self.overwrite_local_data = overwrite_local_data
......@@ -99,7 +99,7 @@ class StationPrep(AbstractStationPrep):
f"statistics_per_var={self.statistics_per_var}, " \
f"station_type='{self.station_type}', network='{self.network}', " \
f"sampling='{self.sampling}', target_dim='{self.target_dim}', target_var='{self.target_var}', " \
f"interpolate_dim='{self.interpolate_dim}', window_history_size={self.window_history_size}, " \
f"interpolate_dim='{self.interpolation_dim}', window_history_size={self.window_history_size}, " \
f"window_lead_time={self.window_lead_time}, overwrite_local_data={self.overwrite_local_data}, " \
f"transformation={self._print_transformation_as_string}, **{self.kwargs})"
......@@ -144,7 +144,7 @@ class StationPrep(AbstractStationPrep):
return coords.rename(index={"station_lon": "lon", "station_lat": "lat"}).to_dict()[str(self)]
def call_transform(self, inverse=False):
self.transform(dim=self.interpolate_dim, method=self.transformation["method"],
self.transform(dim=self.interpolation_dim, method=self.transformation["method"],
mean=self.transformation['mean'], std=self.transformation["std"],
min_val=self.transformation["min"], max_val=self.transformation["max"],
inverse=inverse
......@@ -164,10 +164,10 @@ class StationPrep(AbstractStationPrep):
self.make_samples()
def make_samples(self):
self.make_history_window(self.target_dim, self.window_history_size, self.interpolate_dim)
self.make_labels(self.target_dim, self.target_var, self.interpolate_dim, self.window_lead_time)
self.make_observation(self.target_dim, self.target_var, self.interpolate_dim)
self.remove_nan(self.interpolate_dim)
self.make_history_window(self.target_dim, self.window_history_size, self.interpolation_dim)
self.make_labels(self.target_dim, self.target_var, self.interpolation_dim, self.window_lead_time)
self.make_observation(self.target_dim, self.target_var, self.interpolation_dim)
self.remove_nan(self.interpolation_dim)
def read_data_from_disk(self, source_name=""):
"""
......@@ -658,13 +658,13 @@ if __name__ == "__main__":
sp = StationPrep(data_path='/home/felix/PycharmProjects/mlt_new/data/', station='DEBY122',
statistics_per_var=statistics_per_var, station_type='background',
network='UBA', sampling='daily', target_dim='variables', target_var='o3',
interpolate_dim='datetime', window_history_size=7, window_lead_time=3,
interpolation_dim='datetime', window_history_size=7, window_lead_time=3,
) # transformation={'method': 'standardise'})
# sp.set_transformation({'method': 'standardise', 'mean': sp.mean+2, 'std': sp.std+1})
sp2 = StationPrep(data_path='/home/felix/PycharmProjects/mlt_new/data/', station='DEBY122',
statistics_per_var=statistics_per_var, station_type='background',
network='UBA', sampling='daily', target_dim='variables', target_var='o3',
interpolate_dim='datetime', window_history_size=7, window_lead_time=3,
interpolation_dim='datetime', window_history_size=7, window_lead_time=3,
transformation={'method': 'standardise'})
sp2.transform(inverse=True)
sp.get_X()
......
......@@ -9,8 +9,8 @@ from typing import Iterator, Union, List, Dict
import pandas as pd
import requests
from src import helpers
from src.configuration.join_settings import join_settings
from mlair import helpers
from mlair.configuration.join_settings import join_settings
# join_url_base = 'https://join.fz-juelich.de/services/rest/surfacedata/'
str_or_none = Union[str, None]
......