Commit 7f0b5ac2 authored by lukas leufen's avatar lukas leufen

Merge branch 'release_v0.10.0' into 'master'

Release v0.10.0

Closes #147, #146, #145, #135, #134, #130, #127, #124, #123, #122, #121, #119, #116, #114, #113, #112, #110, #109, #108, and #106

See merge request toar/machinelearningtools!125
parents e945b365 ab723a1c
Pipeline #43674 failed with stages
in 9 minutes and 25 seconds
......@@ -27,6 +27,9 @@
*.sql
*.sqlite
*.sqlite3
*-err.[0-9]*
*-out.[0-9]*
*.ipynb_checkpoints
# OS generated files #
######################
......@@ -49,6 +52,7 @@ Thumbs.db
##########################
/tmp/
/logging/
/HPC_logging/
# test related data #
#####################
......@@ -61,6 +65,16 @@ report.html
/TestExperiment/
/testrun_network*/
# experiment path #
# #################
????-??-??_????-??_network
# secret variables #
####################
/src/join_settings.py
\ No newline at end of file
/mlair/configuration/join_settings.py
# ignore locally build documentation #
######################################
/docs/_build
\ No newline at end of file
stages:
- init
- test
- docs
- pages
### Static Badges ###
......@@ -131,6 +132,31 @@ coverage:
- badges/
- coverage/
#### Documentation ####
sphinx docs:
stage: docs
tags:
- machinelearningtools
- zam347
before_script:
- chmod +x ./CI/update_badge.sh
- ./CI/update_badge.sh > /dev/null
script:
- pip install -r requirements.txt
- pip install -r docs/requirements_docs.txt
- chmod +x ./CI/create_documentation.sh
- ./CI/create_documentation.sh
after_script:
- ./CI/update_badge.sh > /dev/null
when: always
artifacts:
name: pages
when: always
paths:
- badges/
- webpage/
#### Pages ####
pages:
stage: pages
......@@ -138,15 +164,23 @@ pages:
- zam347
- base
script:
# badges
- mkdir -p public/badges/
- cp -af badges/badge_*.svg public/badges/
- ls public/badges/
# coverage
- mkdir -p public/coverage
- cp -af coverage/. public/coverage
- ls public/coverage
# test
- mkdir -p public/test
- cp -af test_results/. public/test
- ls public/test
# docs
- mkdir -p public/docs
- cp -af webpage/. public/docs
- ls public/docs
# summary
- ls public
when: always
artifacts:
......@@ -157,9 +191,11 @@ pages:
- badges/
- coverage/
- test_results/
- webpage/
cache:
key: old-pages
paths:
- public/badges/
- public/coverage/
- public/test/
- public/docs/
# Changelog
All notable changes to this project will be documented in this file.
## v0.10.0 - 2020-07-15 - MLAir is official name, Workflows, easy Model plug-in
### general
- Official project name is released: MLAir (Machine Learning on Air data)
- a model class can now easily be plugged in into MLAir. #121
- introduced new concept of workflows, #134
### new features
- workflows are used to execute a sequence of run modules, #134
- default workflows for standard and the Juelich HPC systems are available, custom workflows can be defined, #134
- seasonal decomposition is available for conditional quantile plot, #112
- map plot is created with coordinates, #108
- `flatten_tails` are now more general and easier to customise, #114
- model classes have custom compile options (replaces `set_loss`), #110
- model can be set in ExperimentSetup from outside, #121
- default experiment settings can be queried using `get_defaults()`, #123
- training and model settings are reported as MarkDown and Tex tables, #145
### technical
- Juelich HPC systems are supported and installation scripts are available, #106
- data store is tracked, I/O is saved and illustrated in a plot, #116
- batch size, epoch parameter have to be defined in ExperimentSetup, #127, #122
- automatic documentation with sphinx, #109
- default experiment settings are updated, #123
- refactoring of experiment path and its default naming, #124
- refactoring of some parameter names, #146
- preparation for package distribution with pip, #119
- all run scripts are updated to run with workflows, #134
- the experiment folder is restructured, #130
## v0.9.0 - 2020-04-15 - faster bootstraps, extreme value upsamling
### general
- improved and faster bootstrap workflow
- new plot PlotAvailability
- extreme values upsampling
- improved runtime environment
### new features
- entire bootstrap workflow has been refactored and much faster now, can be skipped with `evaluate_bootstraps=False`, #60
- upsampling of extreme values, set with parameter `extreme_values=[your_values_standardised]` (e.g. `[1, 2]`) and
`extremes_on_right_tail_only=<True/False>` if only right tail of distribution is affected or both, #58, #87
- minimal data length property (in total and for all subsets), #76
- custom objects in model class to load customised model objects like padding class, loss, #72
- new plot for data availability: `PlotAvailability`, #103
- introduced (default) `plot_list` to specify which plots to draw
- latex and markdown information on sample sizes for each station, #90
### technical
- implemented tests on gpu and from scratch for develop, release and master branches, #95
- usage of tensorflow 1.13.1 (gpu / cpu), separated in 2 different requirements, #81
- new abstract plot class to have uniform plot class design
- New time tracking wrapper to use for functions or classes
- improved logger (info on display, debug into file), #73, #85, #88
- improved run environment, especially for error handling, #86
- prefix `general` in data store scope is now optional and can be skipped. If given scope is not `general`, it is
treated as subscope, #82
- all 2D Padding classes are now selected by `Padding2D(padding_name=<padding_type>)` e.g.
`Padding2D(padding_name="SymPad2D")`, #78
- custom learning rate (or lr_decay) is optional now, #71
#!/bin/bash
# reset status
echo "failure" > status.txt
# create webpage folder
BRANCH_NAME=$( echo -e "${CI_COMMIT_REF_NAME////_}")
mkdir -p webpage/
mkdir -p webpage/recent
#for w in master develop
#do
# if [[ "${CI_COMMIT_REF_NAME}" == "$w" ]]; then
# mkdir -p "webpage/${BRANCH_NAME}"
# fi
#done
mkdir -p "webpage/${BRANCH_NAME}"
cd docs || {
echo "no docs to build available";
echo "incomplete" > status.txt;
echo "no docs to build avail" > incomplete.txt;
exit 0; }
echo "${CI_COMMIT_TAG}"
make clean
make html
IS_FAILED=$?
# copy results
cp -r ./_build/html/* "../webpage/${BRANCH_NAME}/."
cp -r ./_build/html/* ../webpage/recent/.
if [[ "${CI_COMMIT_REF_NAME}" = "master" ]]; then
cp -r ./_build/html/* ../webpage/.
fi
cd ..
# report if job was successful
if [[ ${IS_FAILED} == 0 ]]; then
echo "success"
echo "success" > status.txt
echo "build" > success.txt
exit 0
else
echo "failed"
exit 1
fi
......@@ -2,7 +2,7 @@
# 'running', 'success' or 'failure' is in this file
if [[ -e status.txt ]]; then
EXIT_STATUS=`cat status.txt`
EXIT_STATUS=$(cat status.txt)
else
EXIT_STATUS="running"
fi
......@@ -26,14 +26,14 @@ elif [[ ${EXIT_STATUS} = "success" ]]; then
BADGE_SUBJECT="passed"
BADGE_COLOR="brightgreen"
if [[ -e success.txt ]]; then
SUCCESS_MESSAGE=`cat success.txt`
SUCCESS_MESSAGE=$(cat success.txt)
BADGE_SUBJECT="${SUCCESS_MESSAGE}"
fi
elif [[ ${EXIT_STATUS} = "incomplete" ]]; then
EXIT_STATUS_MESSAGE=`cat incomplete.txt`
EXIT_STATUS_MESSAGE=$(cat incomplete.txt)
BADGE_SUBJECT="${EXIT_STATUS_MESSAGE}"
EXIT_STATUS_RATIO="$(echo ${EXIT_STATUS_MESSAGE} | (grep -oP '\d*') | head -1)"
printf "%s\n" ${EXIT_STATUS_RATIO}
EXIT_STATUS_RATIO="$(echo "${EXIT_STATUS_MESSAGE}" | (grep -oP '\d*') | head -1)"
printf "%s\n" "${EXIT_STATUS_RATIO}"
if [[ "${EXIT_STATUS_RATIO}" -lt "${FAILURE_THRESHOLD}" ]]; then
BADGE_COLOR="red"
else
......@@ -71,10 +71,10 @@ printf "%s\n" "${SHIELDS_IO_NAME//\#/%23}"
SHIELDS_IO_NAME="$( echo -e "${SHIELDS_IO_NAME//\_/__}" )"
SHIELDS_IO_NAME="$( echo -e "${SHIELDS_IO_NAME//\#/%23}")"
curl "https://img.shields.io/badge/${SHIELDS_IO_NAME}" > ${BADGE_FILENAME}
curl "https://img.shields.io/badge/${SHIELDS_IO_NAME}" > "${BADGE_FILENAME}"
echo "https://img.shields.io/badge/${SHIELDS_IO_NAME}"
SHIELDS_IO_NAME_RECENT="RECENT:${SHIELDS_IO_NAME}"
curl "https://img.shields.io/badge/${SHIELDS_IO_NAME_RECENT}" > ${RECENT_BADGE_FILENAME}
curl "https://img.shields.io/badge/${SHIELDS_IO_NAME_RECENT}" > "${RECENT_BADGE_FILENAME}"
echo "${SHIELDS_IO_NAME_RECENT}" > testRecentName.txt
#
......@@ -82,10 +82,10 @@ if [[ ! -d ./badges ]]; then
# Control will enter here if $DIRECTORY doesn't exist.
mkdir badges/
fi
mv ${BADGE_FILENAME} ./badges/.
mv "${BADGE_FILENAME}" ./badges/.
# replace outdated recent badge by new badge
mv ${RECENT_BADGE_FILENAME} ./badges/${RECENT_BADGE_FILENAME}
mv "${RECENT_BADGE_FILENAME}" "./badges/${RECENT_BADGE_FILENAME}"
# set status to failed, this will be overwritten if job ended with exitcode 0
echo "failed" > status.txt
......
["DENW094", "DEBW029", "DENI052", "DENI063", "DEBY109", "DEUB022", "DESN001", "DEUB013", "DETH016", "DEBY002", "DEBY005", "DEBY099", "DEUB038", "DEBE051", "DEBE056", "DEBE062", "DEBE032", "DEBE034", "DEBE010", "DEHE046", "DEST031", "DEBY122", "DERP022", "DEBY079", "DEBW102", "DEBW076", "DEBW045", "DESH016", "DESN004", "DEHE032", "DEBB050", "DEBW042", "DEBW046", "DENW067", "DESL019", "DEST014", "DENW062", "DEHE033", "DENW081", "DESH008", "DEBB055", "DENI011", "DEHB001", "DEHB004", "DEHB002", "DEHB003", "DEHB005", "DEST039", "DEUB003", "DEBW072", "DEST002", "DEBB001", "DEHE039", "DEBW035", "DESN005", "DEBW047", "DENW004", "DESN011", "DESN076", "DEBB064", "DEBB006", "DEHE001", "DESN012", "DEST030", "DESL003", "DEST104", "DENW050", "DENW008", "DETH026", "DESN085", "DESN014", "DESN092", "DENW071", "DEBW004", "DENI028", "DETH013", "DENI059", "DEBB007", "DEBW049", "DENI043", "DETH020", "DEBY017", "DEBY113", "DENW247", "DENW028", "DEBW025", "DEUB039", "DEBB009", "DEHE027", "DEBB042", "DEHE008", "DESN017", "DEBW084", "DEBW037", "DEHE058", "DEHE028", "DEBW112", "DEBY081", "DEBY082", "DEST032", "DETH009", "DEHE010", "DESN019", "DEHE023", "DETH036", "DETH040", "DEMV017", "DEBW028", "DENI042", "DEMV004", "DEMV019", "DEST044", "DEST050", "DEST072", "DEST022", "DEHH049", "DEHH047", "DEHH033", "DEHH050", "DEHH008", "DEHH021", "DENI054", "DEST070", "DEBB053", "DENW029", "DEBW050", "DEUB034", "DENW018", "DEST052", "DEBY020", "DENW063", "DESN050", "DETH061", "DERP014", "DETH024", "DEBW094", "DENI031", "DETH041", "DERP019", "DEBW081", "DEHE013", "DEBW021", "DEHE060", "DEBY031", "DESH021", "DESH033", "DEHE052", "DEBY004", "DESN024", "DEBW052", "DENW042", "DEBY032", "DENW053", "DENW059", "DEBB082", "DEBB031", "DEHE025", "DEBW053", "DEHE048", "DENW051", "DEBY034", "DEUB035", "DEUB032", "DESN028", "DESN059", "DEMV024", "DENW079", "DEHE044", "DEHE042", "DEBB043", "DEBB036", "DEBW024", "DERP001", "DEMV012", "DESH005", "DESH023", "DEUB031", "DENI062", "DENW006", "DEBB065", "DEST077", "DEST005", "DERP007", "DEBW006", "DEBW007", "DEHE030", "DENW015", "DEBY013", "DETH025", "DEUB033", "DEST025", "DEHE045", "DESN057", "DENW036", "DEBW044", "DEUB036", "DENW096", "DETH095", "DENW038", "DEBY089", "DEBY039", "DENW095", "DEBY047", "DEBB067", "DEBB040", "DEST078", "DENW065", "DENW066", "DEBY052", "DEUB030", "DETH027", "DEBB048", "DENW047", "DEBY049", "DERP021", "DEHE034", "DESN079", "DESL008", "DETH018", "DEBW103", "DEHE017", "DEBW111", "DENI016", "DENI038", "DENI058", "DENI029", "DEBY118", "DEBW032", "DEBW110", "DERP017", "DESN036", "DEBW026", "DETH042", "DEBB075", "DEBB052", "DEBB021", "DEBB038", "DESN051", "DEUB041", "DEBW020", "DEBW113", "DENW078", "DEHE018", "DEBW065", "DEBY062", "DEBW027", "DEBW041", "DEHE043", "DEMV007", "DEMV021", "DEBW054", "DETH005", "DESL012", "DESL011", "DEST069", "DEST071", "DEUB004", "DESH006", "DEUB029", "DEUB040", "DESN074", "DEBW031", "DENW013", "DENW179", "DEBW056", "DEBW087", "DEST061", "DEMV001", "DEBB024", "DEBW057", "DENW064", "DENW068", "DENW080", "DENI019", "DENI077", "DEHE026", "DEBB066", "DEBB083", "DEST063", "DEBW013", "DETH086", "DESL018", "DETH096", "DEBW059", "DEBY072", "DEBY088", "DEBW060", "DEBW107", "DEBW036", "DEUB026", "DEBW019", "DENW010", "DEST098", "DEHE019", "DEBW039", "DESL017", "DEBW034", "DEUB005", "DEBB051", "DEHE051", "DEBW023", "DEBY092", "DEBW008", "DEBW030", "DENI060", "DEST011", "DENW030", "DENI041", "DERP015", "DEUB001", "DERP016", "DERP028", "DERP013", "DEHE022", "DEUB021", "DEBW010", "DEST066", "DEBB063", "DEBB028", "DEHE024", "DENI020", "DENI051", "DERP025", "DEBY077", "DEMV018", "DEST089", "DEST028", "DETH060", "DEHE050", "DEUB028", "DESN045", "DEUB042"]
[
"DENW094",
"DEBW029",
"DENI052",
"DENI063",
"DEBY109",
"DEUB022",
"DESN001",
"DEUB013",
"DETH016",
"DEBY002",
"DEBY005",
"DEBY099",
"DEUB038",
"DEBE051",
"DEBE056",
"DEBE062",
"DEBE032",
"DEBE034",
"DEBE010",
"DEHE046",
"DEST031",
"DEBY122",
"DERP022",
"DEBY079",
"DEBW102",
"DEBW076",
"DEBW045",
"DESH016",
"DESN004",
"DEHE032",
"DEBB050",
"DEBW042",
"DEBW046",
"DENW067",
"DESL019",
"DEST014",
"DENW062",
"DEHE033",
"DENW081",
"DESH008",
"DEBB055",
"DENI011",
"DEHB001",
"DEHB004",
"DEHB002",
"DEHB003",
"DEHB005",
"DEST039",
"DEUB003",
"DEBW072",
"DEST002",
"DEBB001",
"DEHE039",
"DEBW035",
"DESN005",
"DEBW047",
"DENW004",
"DESN011",
"DESN076",
"DEBB064",
"DEBB006",
"DEHE001",
"DESN012",
"DEST030",
"DESL003",
"DEST104",
"DENW050",
"DENW008",
"DETH026",
"DESN085",
"DESN014",
"DESN092",
"DENW071",
"DEBW004",
"DENI028",
"DETH013",
"DENI059",
"DEBB007",
"DEBW049",
"DENI043",
"DETH020",
"DEBY017",
"DEBY113",
"DENW247",
"DENW028",
"DEBW025",
"DEUB039",
"DEBB009",
"DEHE027",
"DEBB042",
"DEHE008",
"DESN017",
"DEBW084",
"DEBW037",
"DEHE058",
"DEHE028",
"DEBW112",
"DEBY081",
"DEBY082",
"DEST032",
"DETH009",
"DEHE010",
"DESN019",
"DEHE023",
"DETH036",
"DETH040",
"DEMV017",
"DEBW028",
"DENI042",
"DEMV004",
"DEMV019",
"DEST044",
"DEST050",
"DEST072",
"DEST022",
"DEHH049",
"DEHH047",
"DEHH033",
"DEHH050",
"DEHH008",
"DEHH021",
"DENI054",
"DEST070",
"DEBB053",
"DENW029",
"DEBW050",
"DEUB034",
"DENW018",
"DEST052",
"DEBY020",
"DENW063",
"DESN050",
"DETH061",
"DERP014",
"DETH024",
"DEBW094",
"DENI031",
"DETH041",
"DERP019",
"DEBW081",
"DEHE013",
"DEBW021",
"DEHE060",
"DEBY031",
"DESH021",
"DESH033",
"DEHE052",
"DEBY004",
"DESN024",
"DEBW052",
"DENW042",
"DEBY032",
"DENW053",
"DENW059",
"DEBB082",
"DEBB031",
"DEHE025",
"DEBW053",
"DEHE048",
"DENW051",
"DEBY034",
"DEUB035",
"DEUB032",
"DESN028",
"DESN059",
"DEMV024",
"DENW079",
"DEHE044",
"DEHE042",
"DEBB043",
"DEBB036",
"DEBW024",
"DERP001",
"DEMV012",
"DESH005",
"DESH023",
"DEUB031",
"DENI062",
"DENW006",
"DEBB065",
"DEST077",
"DEST005",
"DERP007",
"DEBW006",
"DEBW007",
"DEHE030",
"DENW015",
"DEBY013",
"DETH025",
"DEUB033",
"DEST025",
"DEHE045",
"DESN057",
"DENW036",
"DEBW044",
"DEUB036",
"DENW096",
"DETH095",
"DENW038",
"DEBY089",
"DEBY039",
"DENW095",
"DEBY047",
"DEBB067",
"DEBB040",
"DEST078",
"DENW065",
"DENW066",
"DEBY052",
"DEUB030",
"DETH027",
"DEBB048",
"DENW047",
"DEBY049",
"DERP021",
"DEHE034",
"DESN079",
"DESL008",
"DETH018",
"DEBW103",
"DEHE017",
"DEBW111",
"DENI016",
"DENI038",
"DENI058",
"DENI029",
"DEBY118",
"DEBW032",
"DEBW110",
"DERP017",
"DESN036",
"DEBW026",
"DETH042",
"DEBB075",
"DEBB052",
"DEBB021",
"DEBB038",
"DESN051",
"DEUB041",
"DEBW020",
"DEBW113",
"DENW078",
"DEHE018",
"DEBW065",
"DEBY062",
"DEBW027",
"DEBW041",
"DEHE043",
"DEMV007",
"DEMV021",
"DEBW054",
"DETH005",
"DESL012",
"DESL011",
"DEST069",
"DEST071",
"DEUB004",
"DESH006",
"DEUB029",
"DEUB040",
"DESN074",
"DEBW031",
"DENW013",
"DENW179",
"DEBW056",
"DEBW087",
"DEST061",
"DEMV001",
"DEBB024",
"DEBW057",
"DENW064",
"DENW068",
"DENW080",
"DENI019",
"DENI077",
"DEHE026",
"DEBB066",
"DEBB083",
"DEST063",
"DEBW013",
"DETH086",
"DESL018",
"DETH096",
"DEBW059",
"DEBY072",
"DEBY088",
"DEBW060",
"DEBW107",
"DEBW036",
"DEUB026",
"DEBW019",
"DENW010",
"DEST098",
"DEHE019",
"DEBW039",
"DESL017",
"DEBW034",
"DEUB005",
"DEBB051",
"DEHE051",
"DEBW023",
"DEBY092",
"DEBW008",
"DEBW030",
"DENI060",
"DEST011",
"DENW030",
"DENI041",
"DERP015",
"DEUB001",