Lecture: kartičky | data-mining

organization

practicals (40 %)
- homework (traditional or trial alternative): 10 points
- quizzes: 5×5 = 25 points
  - in Moodle throughout the semester
- project (traditional or trial alternative): 15 points
  - usually, students propose their own topics
  - with a presentation
- we can even get some bonus point for cooperation during practicals
exam (60 %)
- online in-class test (during lab practical) 15 %
- final exam (written and oral part) 45 %

traditional version: no AI, individual

traditional project – can be in teams

trial alternative (for project+homework)

can be solved individually or in small teams involving two or (three) students
AI tools allowed
- if used, test at least two and compare their benefits/drawbacks
involves two presentation
- traditional one
- software-oriented one – instead of the homework!
agreement with the publication of the project is necessary

Introduction

data mining from databases

non-trivial process of gaining implicit, previously not known, but potentially useful information from the data
originated in the 90s (there was not enough data before)
knowledge discovery in databases (KDD)
data mining (DM) – business intelligence (BI) and big data

Introduction

foundations

Introduction

data mining is an interactive and iterative process

data preparation
- we build one table containing all the relevant data
- selection
- preprocessing
- transformation
the actual “data mining” – we find patterns in the data
interpretation – found knowledge shall be evaluated from the point of view of the end user (manager, customer, etc.)

Introduction

PoV of a manager

there's a topical issue
goal of the data mining process is to obtain as much information as possible that is relevant to solving the problem
example
- find groups of customers of a department store to offer special services to
- the found groups can be interpreted as segments in the given market area
steps
- form a team: data analyst, domain expert, expert on databases, …
- specify the problem
- obtain all data available
  - we should also obtain the external data describing the environment of the analyzed processes (time period of the year, advertising, political issues, weather, …)
- select the methods
  - clustering, classification, exploratory data analysis, association rules, decision trees, genetic algorithms, Bayesian networks, neural networks
  - visualization methods – helpful for presentation
- preprocess the data
- mine the data
- interpret the results
  - we may need to create an analytical report
  - make the results easy to understand
  - the output can also mean to carry out a reasonable action

Introduction

tasks

classification and prediction
- goal: predict a continuous or discrete value based on some attributes
- interpretation may be challenging
- prediction: weather forecast, stock prices, …
- we should be able to cover the entire domain (all the data may be useful for a reasonable prediction)
description
- goal: find a dominant structure or relationships
- we may ignore some of the information; the extracted knowledge does not need to be that precise (but it should be easily understandable)
looking for “nuggets”
- goal: find some interesting knowledge (does not have to fully cover the given concept)

Introduction

real tasks

segmentation and classification of bank clients
causes of failures in telecommunication networks
causes of change of service provider
prediction of power consumption
analysis of the patient database in a hospital
- Florence Nightingale
- Ignaz Semmelweis
market basket analysis

Methodologies

SEMMA

sample – select data for modeling
- may include sampling, imputation (adding other useful information, e.g. adding seasons of the year to the data about the sales), partitioning (train-test-validation split)
explore – visual exploration and dimensionality reduction
modify – prepare the objects, values, and variables for data modeling; transform the data
model – apply data mining techniques (decision trees, regression models, NNs, …)
- create models providing relevant outcome
asses – evaluate the results of modeling (assess their reliability and usefulness)

Methodologies

CRISP-DM – cross-industry standard process for data mining; a robust general-purpose model

business understanding
- determine our business objective
- assess our present situation, what data we have
- risk assessment
- setting KPIs
data understanding
- collect and describe initial data; explore and visualize it
- verify the quality of the data
data preparation
- cleaning, integration (merging), aggregation, …
modeling
evaluation