José Unpingco - Python for Probability, Statistics, and Machine Learning [2016, PDF, ENG]

Contents

1 Getting Started with Scientific Python 1
1.1 Installation and Setup 3
1.2 Numpy 4
1.2.1 Numpy Arrays and Memory 6
1.2.2 Numpy Matrices 9
1.2.3 Numpy Broadcasting 10
1.2.4 Numpy Masked Arrays 12
1.2.5 Numpy Optimizations and Prospectus 12
1.3 Matplotlib 13
1.3.1 Alternatives to Matplotlib 15
1.3.2 Extensions to Matplotlib 16
1.4 IPython 16
1.4.1 IPython Notebook 18
1.5 Scipy 20
1.6 Pandas 21
1.6.1 Series 21
1.6.2 Dataframe 23
1.7 Sympy 25
1.8 Interfacing with Compiled Libraries 27
1.9 Integrated Development Environments 28
1.10 Quick Guide to Performance and Parallel Programming 29
1.11 Other Resources 32
References 32

2 Probability 35
2.1 Introduction 35
2.1.1 Understanding Probability Density 36
2.1.2 Random Variables 37
2.1.3 Continuous Random Variables 42
2.1.4 Transformation of Variables Beyond Calculus 45
2.1.5 Independent Random Variables 47
2.1.6 Classic Broken Rod Example 49
2.2 Projection Methods 50
2.2.1 Weighted Distance 53
2.3 Conditional Expectation as Projection 54
2.3.1 Appendix 60
2.4 Conditional Expectation and Mean Squared Error 60
2.5 Worked Examples of Conditional Expectation and Mean Square Error Optimization 64
2.5.1 Example 64
2.5.2 Example 68
2.5.3 Example 70
2.5.4 Example 73
2.5.5 Example 74
2.5.6 Example 77
2.6 Information Entropy 78
2.6.1 Information Theory Concepts 79
2.6.2 Properties of Information Entropy 81
2.6.3 Kullback-Leibler Divergence 82
2.7 Moment Generating Functions 83
2.8 Monte Carlo Sampling Methods 87
2.8.1 Inverse CDF Method for Discrete Variables 88
2.8.2 Inverse CDF Method for Continuous Variables 90
2.8.3 Rejection Method 92
2.9 Useful Inequalities 95
2.9.1 Markov’s Inequality 96
2.9.2 Chebyshev’s Inequality 97
2.9.3 Hoeffding’s Inequality 98
References 99

3 Statistics 101
3.1 Introduction 101
3.2 Python Modules for Statistics 102
3.2.1 Scipy Statistics Module 102
3.2.2 Sympy Statistics Module 103
3.2.3 Other Python Modules for Statistics 104
3.3 Types of Convergence 104
3.3.1 Almost Sure Convergence 105
3.3.2 Convergence in Probability 107
3.3.3 Convergence in Distribution 109
3.3.4 Limit Theorems 110
3.4 Estimation Using Maximum Likelihood 111
3.4.1 Setting Up the Coin Flipping Experiment 113
3.4.2 Delta Method 123
3.5 Hypothesis Testing and P-Values 125
3.5.1 Back to the Coin Flipping Example 126
3.5.2 Receiver Operating Characteristic 130
3.5.3 P-Values 132
3.5.4 Test Statistics 133
3.5.5 Testing Multiple Hypotheses 140
3.6 Confidence Intervals 141
3.7 Linear Regression 144
3.7.1 Extensions to Multiple Covariates 154
3.8 Maximum A-Posteriori 158
3.9 Robust Statistics 164
3.10 Bootstrapping 171
3.10.1 Parametric Bootstrap 175
3.11 Gauss Markov 176
3.12 Nonparametric Methods 180
3.12.1 Kernel Density Estimation 180
3.12.2 Kernel Smoothing 183
3.12.3 Nonparametric Regression Estimators 188
3.12.4 Nearest Neighbors Regression 189
3.12.5 Kernel Regression 193
3.12.6 Curse of Dimensionality 194
References 196

4 Machine Learning 197
4.1 Introduction 197
4.2 Python Machine Learning Modules 197
4.3 Theory of Learning 201
4.3.1 Introduction to Theory of Machine Learning 203
4.3.2 Theory of Generalization 207
4.3.3 Worked Example for Generalization/Approximation Complexity 209
4.3.4 Cross-Validation 215
4.3.5 Bias and Variance 219
4.3.6 Learning Noise 222
4.4 Decision Trees 225
4.4.1 Random Forests 232
4.5 Logistic Regression 234
4.5.1 Generalized Linear Models 239
4.6 Regularization 240
4.6.1 Ridge Regression 244
4.6.2 Lasso 248
4.7 Support Vector Machines 250
4.7.1 Kernel Tricks 253
4.8 Dimensionality Reduction 256
4.8.1 Independent Component Analysis 260
4.9 Clustering 264
4.10 Ensemble Methods 268
4.10.1 Bagging 268
4.10.2 Boosting 271
References 273
Index 275