Hands-on Data Preparation
Description:
From a process point of view, CRoss-Industry Standard Process for Data Mining (CRISP-DM) describes six major steps for any data analysis project. Having gained business understanding, required data need to be identified and semantically understood, which requires domain knowledge, data engineering and data analysis knowledge. In this phase, it is essential to assess the quality of the data (shit in, shit out). Data understanding is the starting point for data integration and preparation. For the integration, uncountable technologies are available, however, none of them is the silver-bullet. The analysis’ goal and planned analysis approach influence the technology stack as well as an existing infrastructure. The analysis also poses concrete requirements on the data. The task of data preparation step is to extract the required data from their sources through transformation, cleaning, filtering, missing value treatment, etc. The explorative character of data analyses as well as the strong influence of the data preparation on the results of the analysis require that data preparation is performed repeatedly while parameters are changed based on evaluation results. In addition, due to the exploratory nature of an analysis it is often seen only in the course of the project, which data is actually important, how data needs to be prepared, and which data lead to better results. Consequently, data scientist spend a lot of time solely with data preparation (up to 70% of the project effort). Within this hands-on Seminar, you will learn, based on concrete examples, how you kick-start your data analysis projects (i.e., data integration, data preparation). We will show, based on our experience from many projects, where the caveats lie and how you safely ship around them. During the seminar, we will use different examples from process improvement (e.g., predictive maintenance) and will work together to implement data preparation steps in Jupyter Notebooks. Participants need to bring their Laptops and shall be allowed to install “Anaconda”.
Presenters:
Andreas Jedlitschka and Julien Siebert are with the Institute of Experimental Software Engineering at Fraunhofer, Germany.