[Introduction to Data Scientists] Descriptive Statistics and Simple Regression Analysis ♬

Last night, I summarized [Introduction to Data Scientists] Basics of scientific calculation, data processing, and how to use graph drawing library. From tonight, I will finally use them to get into the main subject. Tonight I will summarize descriptive statistics and simple regression analysis. I will supplement the explanations in this book. 【Caution】 ["Data Scientist Training Course at the University of Tokyo"](https://www.amazon.co.jp/%E6%9D%B1%E4%BA%AC%E5%A4%A7%E5%AD%A6%E3 % 81% AE% E3% 83% 87% E3% 83% BC% E3% 82% BF% E3% 82% B5% E3% 82% A4% E3% 82% A8% E3% 83% B3% E3% 83 % 86% E3% 82% A3% E3% 82% B9% E3% 83% 88% E8% 82% B2% E6% 88% 90% E8% AC% 9B% E5% BA% A7-Python% E3% 81 % A7% E6% 89% 8B% E3% 82% 92% E5% 8B% 95% E3% 81% 8B% E3% 81% 97% E3% 81% A6% E5% AD% A6% E3% 81% B6 % E3% 83% 87% E2% 80% 95% E3% 82% BF% E5% 88% 86% E6% 9E% 90-% E5% A1% 9A% E6% 9C% AC% E9% 82% A6% I will read E5% B0% 8A / dp / 4839965250 / ref = tmm_pap_swatch_0? _ Encoding = UTF8 & qid = & sr =) and summarize the parts that I have some doubts or find useful. Therefore, I think the synopsis will be straightforward, but please read it, thinking that the content has nothing to do with this book.

Chapter 3 Descriptive Statistics and Simple Regression Analysis

Chapter 3-1 Types of statistical analysis

3-1-1 Descriptive and inference statistics

Statistical analysis methods are divided into descriptive statistics and inference statistics.

3-1-1-1 Descriptive statistics

"A method to grasp the characteristics of the collected data, organize it in an easy-to-understand manner, and make it easy to see. Calculate the characteristics of the data by calculating the mean, standard deviation, etc., classify the data, and express it using figures and graphs. Descriptive statistics are what you do. "

3-1-1-2 Inference statistics

"The idea of inference statistics is to perform a precise analysis using a model based on a probability distribution from only partial data, and infer the whole to obtain statistics." "It is also used to predict the future from historical data. This chapter describes simple regression analysis, which is the basis of inference statistics. More complex inference statistics will be dealt with in the next four chapters."

3-1-2 Importing the library
import numpy as np
import scipy as sp
import pandas as pd
from pandas import Series, DataFrame

import matplotlib as mpl
import seaborn as sns
import matplotlib.pyplot as plt
sns.set()

from sklearn import linear_model

Install sklearn below.


$ sudo pip3 install scikit-learn

As shown below, it seems that it can also be used with Rasipi4.

 $ python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sklearn import linear_model 
>>> 

For the time being, python3-sklearn-doc was not found, but it seems that it could be installed under Debian / Ubuntu.

$ sudo apt-get install python3-sklearn python3-sklearn-lib

Chapter 3-2 Data reading and interaction

...abridgement 3-2-1-5 From the following site, get the data student.zip with the following program. https://archive.ics.uci.edu/ml/machine-learning-databases/00356/student.zip

import requests, zipfile
from io import StringIO
import io

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00356/student.zip'
r = requests.get(url, stream = True)
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall()

The following four files have been expanded. student.txt student-mat.csv student-merge.R student-pcr.csv

3-2-2 Data reading and confirmation

Connect to the above import and execute the following

student_data_math = pd.read_csv('./chap3/student-mat.csv')
print(student_data_math.head())

Data; You can check the delimiter.

school;sex;age;address;famsize;Pstatus;Medu;Fedu;Mjob;Fjob;reason;guardian;traveltime;studytime;failures;schoolsup;famsup;paid;activities;nursery;higher;internet;romantic;famrel;freetime;goout;Dalc;Walc;health;absences;G1;G2;G3
0  GP;"F";18;"U";"GT3";"A";4;4;"at_home";"teacher...                                                                                                                                                                              
1  GP;"F";17;"U";"GT3";"T";1;1;"at_home";"other";...                                                                                                                                                                              
2  GP;"F";15;"U";"LE3";"T";1;1;"at_home";"other";...                                                                                                                                                                              
3  GP;"F";15;"U";"GT3";"T";4;2;"health";"services...                                                                                                                                                                              
4  GP;"F";16;"U";"GT3";"T";3;3;"other";"other";"h... 

Change the read to; and reload.

student_data_math = pd.read_csv('./chap3/student-mat.csv', sep =';')
print(student_data_math.head())

It looked beautiful.

  school sex  age address famsize Pstatus  Medu  Fedu  ... goout Dalc Walc health  absences  G1  G2  G3
0     GP   F   18       U     GT3       A     4     4  ...     4    1    1      3         6   5   6   6
1     GP   F   17       U     GT3       T     1     1  ...     3    1    1      3         4   5   5   6
2     GP   F   15       U     LE3       T     1     1  ...     2    2    3      3        10   7   8  10
3     GP   F   15       U     GT3       T     4     2  ...     2    1    1      5         2  15  14  15
4     GP   F   16       U     GT3       T     3     3  ...     2    1    2      5         4   6  10  10

[5 rows x 33 columns]
3-2-3 Check the nature of the data

print(student_data_math.info())

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 395 entries, 0 to 394
Data columns (total 33 columns):
 #   Column      Non-Null Count  Dtype
---  ------      --------------  -----
 0   school      395 non-null    object
 1   sex         395 non-null    object
 2   age         395 non-null    int64
 3   address     395 non-null    object
 4   famsize     395 non-null    object
 5   Pstatus     395 non-null    object
 6   Medu        395 non-null    int64
 7   Fedu        395 non-null    int64
 8   Mjob        395 non-null    object
 9   Fjob        395 non-null    object
 10  reason      395 non-null    object
 11  guardian    395 non-null    object
 12  traveltime  395 non-null    int64
 13  studytime   395 non-null    int64
 14  failures    395 non-null    int64
 15  schoolsup   395 non-null    object
 16  famsup      395 non-null    object
 17  paid        395 non-null    object
 18  activities  395 non-null    object
 19  nursery     395 non-null    object
 20  higher      395 non-null    object
 21  internet    395 non-null    object
 22  romantic    395 non-null    object
 23  famrel      395 non-null    int64
 24  freetime    395 non-null    int64
 25  goout       395 non-null    int64
 26  Dalc        395 non-null    int64
 27  Walc        395 non-null    int64
 28  health      395 non-null    int64
 29  absences    395 non-null    int64
 30  G1          395 non-null    int64
 31  G2          395 non-null    int64
 32  G3          395 non-null    int64
dtypes: int64(16), object(17)
memory usage: 102.0+ KB

Looking at the contents of cat student.txt, it seems that this data has the following contents.

$ cat student.txt
# Attributes for both student-mat.csv (Math course) and student-por.csv (Portuguese language course) datasets:
1 school - student's school (binary: "GP" - Gabriel Pereira or "MS" - Mousinho da Silveira)
2 sex - student's sex (binary: "F" - female or "M" - male)
3 age - student's age (numeric: from 15 to 22)
4 address - student's home address type (binary: "U" - urban or "R" - rural)
5 famsize - family size (binary: "LE3" - less or equal to 3 or "GT3" - greater than 3)
6 Pstatus - parent's cohabitation status (binary: "T" - living together or "A" - apart)
7 Medu - mother's education (numeric: 0 - none,  1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education)
8 Fedu - father's education (numeric: 0 - none,  1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education)
9 Mjob - mother's job (nominal: "teacher", "health" care related, civil "services" (e.g. administrative or police), "at_home" or "other")
10 Fjob - father's job (nominal: "teacher", "health" care related, civil "services" (e.g. administrative or police), "at_home" or "other")
11 reason - reason to choose this school (nominal: close to "home", school "reputation", "course" preference or "other")
12 guardian - student's guardian (nominal: "mother", "father" or "other")
13 traveltime - home to school travel time (numeric: 1 - <15 min., 2 - 15 to 30 min., 3 - 30 min. to 1 hour, or 4 - >1 hour)
14 studytime - weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3 - 5 to 10 hours, or 4 - >10 hours)
15 failures - number of past class failures (numeric: n if 1<=n<3, else 4)
16 schoolsup - extra educational support (binary: yes or no)
17 famsup - family educational support (binary: yes or no)
18 paid - extra paid classes within the course subject (Math or Portuguese) (binary: yes or no)
19 activities - extra-curricular activities (binary: yes or no)
20 nursery - attended nursery school (binary: yes or no)
21 higher - wants to take higher education (binary: yes or no)
22 internet - Internet access at home (binary: yes or no)
23 romantic - with a romantic relationship (binary: yes or no)
24 famrel - quality of family relationships (numeric: from 1 - very bad to 5 - excellent)
25 freetime - free time after school (numeric: from 1 - very low to 5 - very high)
26 goout - going out with friends (numeric: from 1 - very low to 5 - very high)
27 Dalc - workday alcohol consumption (numeric: from 1 - very low to 5 - very high)
28 Walc - weekend alcohol consumption (numeric: from 1 - very low to 5 - very high)
29 health - current health status (numeric: from 1 - very bad to 5 - very good)
30 absences - number of school absences (numeric: from 0 to 93)

# these grades are related with the course subject, Math or Portuguese:
31 G1 - first period grade (numeric: from 0 to 20)
31 G2 - second period grade (numeric: from 0 to 20)
32 G3 - final grade (numeric: from 0 to 20, output target)

Additional note: there are several (382) students that belong to both datasets . 
These students can be identified by searching for identical attributes
that characterize each student, as shown in the annexed R file.
3-2-4 Quantitative and qualitative data

・ Quantitative data The data is represented by continuous values to which the four arithmetic operations can be applied, and the ratio is meaningful. Example: number of people, amount of money, etc. ・ Qualitative data It is discontinuous data to which the four arithmetic operations cannot be applied, and is used to express the state. Example; ranking, category, etc.

Gender is qualitative data

print(student_data_math['sex'].head())
0    F
1    F
2    F
3    F
4    F
Name: sex, dtype: object

The number of absentees is quantitative data

print(student_data_math['absences'].head())
0     6
1     4
2    10
3     2
4     4
Name: absences, dtype: int64
3-2-4-2 Calculate the average value for each axis
print(student_data_math.groupby('sex')['age'].mean())
sex
F    16.730769
M    16.657754
Name: age, dtype: float64

Women study.

print(student_data_math.groupby('sex')['studytime'].mean())
sex
F    2.278846
M    1.764706
Name: studytime, dtype: float64

Descriptive statistics

3-3-1 Histogram
fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
y1 = student_data_math['absences']
ax1.hist(y1, bins = 10, range =(0.0,max(y1)))
ax1.set_ylabel('count')
ax1.set_xlabel('absences')
plt.grid(True)
plt.show()

Figure_13absences.png

3-3-2 Mean, median, mode
print('Average value{}'.format(student_data_math['absences'].mean()))
print('Median{}'.format(student_data_math['absences'].median()))
print('Mode{}'.format(student_data_math['absences'].mode()))

Mean 5.708860759493671
Median 4.0
Mode 0 0
dtype: int64

Enlarge the figure above to verify it in the figure.

fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
y1 = student_data_math['absences']
ax1.hist(y1, bins = 30, range =(0.0,30))  #,max(y1)
x0 = student_data_math['absences'].mean()
ax1.plot(x0+0.5, 70,  'red', marker = 'o',markersize=10,label ='mean')
x0 = student_data_math['absences'].median()
ax1.plot(x0+0.5, 70, 'blue', marker = 'o',markersize=10,label ='median')
x0 = student_data_math['absences'].mode()
ax1.plot(x0+0.5, 70, 'black', marker = 'o',markersize=10,label ='mode')
ax1.legend()
ax1.set_ylabel('count')
ax1.set_xlabel('absences')
plt.grid(True)
plt.show()

Figure_13absences30m-.png

3-3-3 Variance and standard deviation

Definition formula Variance $ σ ^ 2 $

σ^2 = \frac{1}{n}\Sigma_{i=1}^{n}(x_i-\bar{x})^2

Standard deviation $ σ $ std(standered deviation)

σ = \sqrt{\frac{1}{n}\Sigma_{i=1}^{n}(x_i-\bar{x})^2}
print('Distributed{}'.format(student_data_math['absences'].var(ddof=0)))
print('standard deviation{}'.format(student_data_math['absences'].std(ddof = 0)))
print('standard deviation{}'.format(np.sqrt(student_data_math['absences'].var())))
Variance 63.887389841371565
Standard deviation 7.99295876640006
Standard deviation 8.00309568710818

Plot the mean ± standard deviation.

fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
y1 = student_data_math['absences']
ax1.hist(y1, bins = 30, range =(0.0,30))  #,max(y1)
x0 = student_data_math['absences'].mean()
ax1.plot(x0+0.5, 70,  'red', marker = 'o',markersize=10,label ='mean')
x1 = student_data_math['absences'].std(ddof=0)
ax1.plot(x0+x1+0.5, 70, 'blue', marker = 'o',markersize=10,label ='mean+std')
ax1.plot(x0-x1+0.5, 70, 'black', marker = 'o',markersize=10,label ='mean-std')
ax1.legend()
ax1.set_ylabel('count')
ax1.set_xlabel('absences')
plt.grid(True)
plt.show()

Figure_13absences30m-std.png

3-3-4 Summary statistics and percentile values

Percentile values are ranked when the total number is 100 25th percentile, 25th percentile, 1st quartile The 75th is the 75th percentile, the third quartile 50th percentile, median

print('Summary statistics', student_data_math['absences'].describe())
Summary statistic count 395.000000
mean       5.708861
std        8.003096
min        0.000000
25%        0.000000
50%        4.000000
75%        8.000000
max       75.000000
Name: absences, dtype: float64
Find the interquartile range

25th percentile; describe (4) 75th percentile: describe (6) Difference; describe (6)-describe (4) print ('75-25 Percentile', student_data_math ['absences'] .describe () [6]-student_data_math ['absences'] .describe () [4]) 75-25 Percentile 8.0

fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
y1 = student_data_math['absences']
ax1.hist(y1, bins = 30, range =(0.0,30))  #,max(y1)
x0 = student_data_math['absences'].median()
ax1.plot(x0+0.5, 70,  'red', marker = 'o',markersize=10,label ='median')
x1 = student_data_math['absences'].describe()[4]
ax1.plot(x1+0.5, 70, 'blue', marker = 'o',markersize=10,label ='25percentile')
x1 = student_data_math['absences'].describe()[6]
ax1.plot(x1+0.5, 70, 'black', marker = 'o',markersize=10,label ='75percentile')
ax1.legend()
ax1.set_ylabel('count')
ax1.set_xlabel('absences')
plt.grid(True)
plt.show()

Figure_13absences30mpp.png

3-3-4-2 describe () for all rows
print('Full column summary statistics', student_data_math.describe())
Full column summary statistics
              age        Medu        Fedu  traveltime  ...    absences          G1          G2          G3
count  395.000000  395.000000  395.000000  395.000000  ...  395.000000  395.000000  395.000000  395.000000
mean    16.696203    2.749367    2.521519    1.448101  ...    5.708861   10.908861   10.713924   10.415190
std      1.276043    1.094735    1.088201    0.697505  ...    8.003096    3.319195    3.761505    4.581443
min     15.000000    0.000000    0.000000    1.000000  ...    0.000000    3.000000    0.000000    0.000000
25%     16.000000    2.000000    2.000000    1.000000  ...    0.000000    8.000000    9.000000    8.000000
50%     17.000000    3.000000    2.000000    1.000000  ...    4.000000   11.000000   11.000000   11.000000
75%     18.000000    4.000000    3.000000    2.000000  ...    8.000000   13.000000   13.000000   14.000000
max     22.000000    4.000000    4.000000    4.000000  ...   75.000000   19.000000   19.000000   20.000000

[8 rows x 16 columns]

3-3-5 Box plot

Box plot is (minimum value, number 1). 1 quartile, median, 3rd quartile, maximum) is expressed by a box and a whiskers as follows.

fig, (ax1,ax2) = plt.subplots(2, 1, figsize=(8,2*4))
y1 = student_data_math['G1']
ax1.hist(y1, bins = 30, range =(0.0,max(y1)))  #,max(y1)
x0 = student_data_math['G1'].median()
ax1.plot(x0+0.5, 60,  'red', marker = 'o',markersize=10,label ='median')
x1 = student_data_math['G1'].describe()[4]
ax1.plot(x1+0.5, 60, 'blue', marker = 'o',markersize=10,label ='25percentile')
x1 = student_data_math['G1'].describe()[6]
ax1.plot(x1+0.5, 60, 'black', marker = 'o',markersize=10,label ='75percentile')
ax2.boxplot(y1)
ax2.set_xlabel('G1')
ax2.set_ylabel('count')
ax1.legend()
ax1.set_ylabel('count')
ax1.set_xlabel('G1')
plt.grid(True)
plt.show()

Figure_14-G1pp.png

fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
y1 = [student_data_math['G1'],student_data_math['G2'],student_data_math['G3'],student_data_math['absences']]
ax1.boxplot(y1,labels=['G1', 'G2', 'G3', 'absences'])
ax1.set_xlabel('category')
ax1.set_ylabel('count')
ax1.legend()
ax1.set_ylabel('count')
ax1.set_xlabel('category')
plt.grid(True)
plt.show()

Figure_14-G1pp4.png

3-3-6 Coefficient of variation

The coefficient of variation CV is the standard deviation σ / mean $ \ bar {x} $ The coefficient of variation does not depend on the scale, and the degree of dispersion can be seen.

print(student_data_math.std()/student_data_math.mean())
age           0.076427
Medu          0.398177
Fedu          0.431565
traveltime    0.481668
studytime     0.412313
failures      2.225319
famrel        0.227330
freetime      0.308725
goout         0.358098
Dalc          0.601441
Walc          0.562121
health        0.391147
absences      1.401873
G1            0.304266
G2            0.351086
G3            0.439881
dtype: float64

3-3-7 Scatter plot and correlation coefficient

fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
x = student_data_math['G1']
y = student_data_math['G3']
ax1.plot(x,y, 'o')
ax1.set_xlabel('G1-grade')
ax1.set_ylabel('G3-grade')
ax1.legend()
plt.grid(True)
plt.show()

Those who had a high G1-Grade also have a high G3-Grade. However, there are some people who have 0 G3-Grade. This is an outlier, but there are various reasons for it, and there is a debate about whether to exclude it. Figure_15-G1G3.png So, what about the number of days attended by people with G3-Grade 0, draw the following correlation diagram.

fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
x = student_data_math['G3']
y = student_data_math['absences']
ax1.plot(x,y, 'o')
ax1.set_xlabel('G3-grade')
ax1.set_ylabel('absences')
ax1.legend()
plt.grid(True)
plt.show()

The result is that people with a G3-Grade of 0 are absent 0. Something is wrong. Actually, I can imagine that I stopped halfway and did not count. Figure_15-G3absences.png Furthermore, the correlation between G1-Grade and the number of absentees is as follows. Figure_15-G1absences.png In the first place, at the time of G2-Grade, some people are 0 Grade. Figure_15-G1G2.png And even if you look at the correlation between G2-Grade and G3-Grade, you can see that some people have fallen to 0 Grade, and that the number of such people is gradually increasing. And it seems that the deceased are out of those with low scores. Therefore, it is more important to analyze various data than to rush to the conclusion with one graph. Figure_15-G2G3.png

3-3-7-1 Covariance

The definition formula is as follows

S_{xy}=\frac{1}{n}\Sigma_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})

That is, the diagonal term is the variance defined above. So what does the off-diagonal term mean? According to the reference, if $ x $ and $ y $ have an inherently linear relationship, the equation of the straight line derived by the least squares method is as follows. 【reference】 Linear regression analysis / least squares means covariance / correlation coefficient

y=\frac{S_{xy}}{\sigma^2_{x}}x + \bar y−\frac{S_{xy}}{\sigma^2_x}\bar x \\
When transformed,\\
\frac{y-\bar y}{\sigma_y}=\frac{S_{xy}}{\sigma_x\sigma_y}\frac{x-\bar x}{\sigma_x}

That is, the slope of the linear equation standardized by the standard deviation and the mean value is as follows.

r_{xy}=\frac{S_{xy}}{\sigma_x\sigma_y}

In other words, it is a value obtained by standardizing the covariance with the standard deviation, and this is the definition formula of the so-called correlation coefficient $ r_ {xy} $.

3-3-7-2 Correlation coefficient

Here, we will find the covariance and the correlation coefficient. The covariance is the off-diagonal and the diagonal is the variance of G1 and G3.

print(np.cov(student_data_math['G1'],student_data_math['G3']))
[[11.01705327 12.18768232]
 [12.18768232 20.9896164 ]]

The front is the correlation coefficient and the second term is the p-value.

print(sp.stats.pearsonr(student_data_math['G1'],student_data_math['G3']))
(0.801467932017414, 9.001430312277865e-90)

The correlation matrix is calculated below.

print(np.corrcoef(student_data_math['G1'],student_data_math['G3']))
[[1.         0.80146793]
 [0.80146793 1.        ]]

3-3-8 Draw histograms and scatter plots for all variables

Dalc; Weekday alcohol intake Walc; Weekend Alcohol Intake And draw a scatter plot to see if there is a correlation between the scores of G1 and G3. Result; seems unlikely

g = sns.pairplot(student_data_math[['Dalc','Walc','G1','G3']])
g.savefig('seaborn_pairplot_g.png')

seaborn_pairplot_g.png

【reference】 Create a pair plot diagram (scatter plot matrix) with Python, pandas, seaborn No correlation between Walc and G3 scores

print(np.corrcoef(student_data_math['Walc'],student_data_math['G3']))
[[ 1.         -0.05193932]
 [-0.05193932  1.        ]]

No variation for each group

print(student_data_math.groupby('Walc')['G3'].mean())
Walc
1    10.735099
2    10.082353
3    10.725000
4     9.686275
5    10.142857
Name: G3, dtype: float64

Exercise 3-1

              age        Medu        Fedu  traveltime  ...    absences          G1          G2          G3
count  649.000000  649.000000  649.000000  649.000000  ...  649.000000  649.000000  649.000000  649.000000
mean    16.744222    2.514638    2.306626    1.568567  ...    3.659476   11.399076   11.570108   11.906009
std      1.218138    1.134552    1.099931    0.748660  ...    4.640759    2.745265    2.913639    3.230656
min     15.000000    0.000000    0.000000    1.000000  ...    0.000000    0.000000    0.000000    0.000000
25%     16.000000    2.000000    1.000000    1.000000  ...    0.000000   10.000000   10.000000   10.000000
50%     17.000000    2.000000    2.000000    1.000000  ...    2.000000   11.000000   11.000000   12.000000
75%     18.000000    4.000000    3.000000    2.000000  ...    6.000000   13.000000   13.000000   14.000000
max     22.000000    4.000000    4.000000    4.000000  ...   32.000000   19.000000   19.000000   19.000000

Exercise 3-2

df =student_data_math.merge(student_data_por,left_on=['school','sex','age','address','famsize','Pstatus','Medu','Fedu','Mjob','Fjob','reason','nursery','internet'], right_on=['school','sex','age','address','famsize','Pstatus','Medu','Fedu','Mjob','Fjob','reason','nursery','internet'], suffixes=('_math', '_por'))
print(df.head())
  school sex  age address famsize Pstatus  ...  Walc_por  health_por absences_por G1_por G2_por G3_por
0     GP   F   18       U     GT3       A  ...         1           3            4      0     11     11
1     GP   F   17       U     GT3       T  ...         1           3            2      9     11     11
2     GP   F   15       U     LE3       T  ...         3           3            6     12     13     12
3     GP   F   15       U     GT3       T  ...         1           5            0     14     14     14
4     GP   F   16       U     GT3       T  ...         2           5            0     11     13     13

[5 rows x 53 columns]

Exercise 3-3

gm = sns.pairplot(df[['G1_math','G3_math','G1_por','G3_por']])
gm.savefig('seaborn_pairplot_gm.png')

Correlation between math and por seems to be high Variance seems to be smaller in por than in math seaborn_pairplot_gm.png It is also supported by the following results.

print(np.corrcoef(df['G1_math'],df['G3_math']))
[[1.        0.8051287]
 [0.8051287 1.       ]]
print(np.corrcoef(df['G3_math'],df['G3_por']))
[[1.         0.48034936]
 [0.48034936 1.        ]]

print(np.cov(df['G1_math'],df['G3_math']))
[[11.2169202  12.63919693]
 [12.63919693 21.9702354 ]]
print(np.cov(df['G3_math'],df['G3_por']))
[[21.9702354   6.63169394]
 [ 6.63169394  8.67560567]]

Chapter 3-4 Simple Regression Analysis

"Next to descriptive statistics, let's learn the basics of regression analysis." "Regression analysis is an analysis that predicts numbers .... I've graphed the student data above. From this scatter plot, I can see that G1 and G3 are likely to be related."

fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
ax1.plot(student_data_math['G1'],student_data_math['G3'],'o')
ax1.set_xlabel('G1_Grade')
ax1.set_ylabel('G3_Grade')
ax1.grid(True)
plt.show()

Figure_16-G1G3.png "In the regression problem, we assume a relational expression from the given data and find the coefficient that best fits the data. Specifically, we predict the G3 grade based on the G1 grade that we know in advance. That is, there is a target variable G3 (called the objective variable), and the variable G1 (called the explanatory variable) that explains it is used for prediction. In regression analysis, one explanatory variable and one explanatory variable are used. The former is called simple regression and the latter is called multiple regression analysis. In this chapter, we will explain simple regression analysis. "

3-4-1 Linear Simple Regression Analysis

"Here, we will explain how to solve the regression problem by a method called linear simple regression, which assumes that the output and input have a linear relationship in simple regression analysis."

import pandas as pd
from sklearn import linear_model

reg = linear_model.LinearRegression()
student_data_math = pd.read_csv('./chap3/student-mat.csv', sep =';')

x = student_data_math.loc[:,['G1']].values
y = student_data_math['G3'].values
reg.fit(x,y)
print('Regression coefficient;',reg.coef_)
print('Intercept;',reg.intercept_)
Regression coefficient;[1.10625609]
Intercept;-1.6528038288004616
fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
ax1.plot(student_data_math['G1'],student_data_math['G3'],'o')
ax1.plot(x,reg.predict(x))
ax1.set_xlabel('G1_Grade')
ax1.set_ylabel('G3_Grade')
ax1.grid(True)
plt.show()

Figure_16-G1G3predict.png

3-4-2 Coefficient of determination

R^2 = 1- \frac{\Sigma_{i=1}^{n}(y_i-f(x_i))^2}{\Sigma_{i=1}^{n}(y_i-\bar y)^2}

The above equation is called the coefficient of determination, and $ R ^ 2 = 1 $ is the maximum value, and the closer it is to 1, the better the model.

print('Coefficient of determination;',reg.score(x,y))
Coefficient of determination; 0.64235084605227

Comprehensive problem 3-2-1 Lorenz Curve

df0 = student_data_math[student_data_math['sex'].isin(['M'])]
df = df0.sort_values(by=['G1'])
df['Ct']=np.arange(1,len(df)+1)

x = df['Ct']
print(x)
y = df['G1'].cumsum()
print(y)
fig, (ax1) = plt.subplots(1, 1, figsize=(8,6))
ax1.plot(x/max(x),y/max(y))
ax1.set_xlabel('peoples')
ax1.set_ylabel('G1_Grade.cumsum')
ax1.grid(True)
plt.show()

248      1
144      2
164      3
161      4
153      5
      ...
113    183
129    184
245    185
42     186
47     187
Name: Ct, Length: 187, dtype: int32
248       3
144       8
164      13
161      18
153      23
       ...
113    2026
129    2044
245    2062
42     2081
47     2100
Name: G1, Length: 187, dtype: int64

M Figure_17-G1cumsumvspplesM.png F Figure_17-G1cumsumvspples.png reference M;G1 vs peaples Figure_17-G1vspeoplesM.png F;G1 vs peaples Figure_17-G1vspeoples.png

Recommended Posts

[Introduction to Data Scientists] Descriptive Statistics and Simple Regression Analysis ♬
Data analysis: Easily apply descriptive and inference statistics to CSV data in Python
[Introduction to Data Scientists] Basics of Probability and Statistics ♬ Probability / Random Variables and Probability Distribution
[Introduction to Data Scientists] Basics of Python ♬ Functions and classes
[Introduction to Data Scientists] Basics of Python ♬ Conditional branching and loops
[Introduction to Data Scientists] Basics of Python ♬ Functions and anonymous functions, etc.
[Introduction to Data Scientists] Basics of Python ♬
Introduction to Statistical Modeling for Data Analysis GLM Likelihood-Ratio Test and Test Asymmetry
An introduction to statistical modeling for data analysis
[Introduction to minimize] Data analysis with SEIR model ♬
Reading Note: An Introduction to Data Analysis with Python
I decided to do a simple regression analysis manually-1-
[Technical book] Introduction to data analysis using Python -1 Chapter Introduction-
Introduction to Simple Regression Analysis with Python (Comparison of 6 Libraries of Numerical Calculation/Computer Algebra System)
20200329_Introduction to Data Analysis with Python Second Edition Personal Summary
An introduction to statistical modeling for data analysis (Midorimoto) reading notes (in Python and Stan)
Simple regression analysis in Python
[Introduction to Python] Combine Nikkei 225 and NY Dow csv data
Introduction to Statistical Modeling for Data Analysis GLM Model Selection
[Introduction to Data Scientists] Basics of scientific calculation, data processing, and how to use graph drawing library ♬ Basics of Scipy
[Introduction to Data Scientists] Basics of scientific calculation, data processing, and how to use graph drawing library ♬ Basics of Pandas
[Introduction to Data Scientists] Basics of scientific calculation, data processing, and how to use graph drawing library ♬ Basics of Matplotlib
"Introduction to data analysis by Bayesian statistical modeling starting with R and Stan" implemented in Python
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part1-
Introduction to Data Analysis with Python P32-P43 [ch02 3.US Baby Names 1880-2010]
Introduction to Data Analysis with Python P17-P26 [ch02 1.usa.gov data from bit.ly]
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part2-
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part4-
Introduction to Statistical Modeling for Data Analysis Generalized Linear Models (GLM)
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part3-
First simple regression analysis in Python
[Introduction to Python3 Day 1] Programming and Python
Machine learning algorithm (simple regression analysis)
Predictive Statistics (Practice Simple Regression) Python
Data analysis for improving POG 3-Regression analysis-
Simple regression analysis implementation in Keras
Introduction to image analysis opencv python
Summary of probability distributions that often appear in statistics and data analysis
I tried fMRI data analysis with python (Introduction to brain information decoding)
[Introduction to Data Scientists] Basics of scientific calculation, data processing, and how to use the graph drawing library ♬ Environment construction
Starbucks Twitter Data Location Visualization and Analysis
[Introduction to Python3, Day 17] Chapter 8 Data Destinations (8.1-8.2.5)
[Introduction to Python3, Day 17] Chapter 8 Data Destinations (8.3-8.3.6.1)
Machine learning with python (2) Simple regression analysis
Introduction to Deep Learning ~ Convolution and Pooling ~
[Introduction to Python3 Day 19] Chapter 8 Data Destinations (8.4-8.5)
[Introduction to Python3 Day 18] Chapter 8 Data Destinations (8.3.6.2 to 8.3.6.3)
Dump SQLite3 data and migrate to MySQL
Only 8 Techniques to Pretend to be Data Scientists
Understanding data types and beginning linear regression
Compress python data and write to sqlite
[Introduction to AWS] Text-Voice conversion and playing ♪
Time series analysis # 6 Spurious regression and cointegration
Analysis for Data Scientists: Qiita Self-Article Summary 2020
[Python] From morphological analysis of CSV data to CSV output and graph display [GiNZA]
[Introduction to cx_Oracle] (Part 9) DB and Python data type mapping (version 8 or later)
Introduction to Quiz Statistics (1) -Mathematical analysis of question sentences to know the tendency of questions-
Introduction to Statistical Modeling for Data Analysis Expanding the range of applications of GLM
Introduction to Time Series Analysis ~ Seasonal Adjustment Model ~ Implemented in R and Python
An introduction to data analysis using Python-To increase the number of video views-
The first step to log analysis (how to format and put log data in Pandas)