content:
linear regression, Ridge, Lasso
Logistic Regression, Softmax
Kmeans, GMM, EM, Spectral Clustering
Dimensionality Reduction: PCA、LDA、Laplacian Eigenmap、 LLE、 Isomap（修改前面的blog）
SVM
ID3、C4.5
Apriori，FP
PageRank
minHash, LSH
Manifold Ranking，EMR

ok，廢話不多說。

## 1、Linear Regression

123 250
150 320
87 160
102 220
… …

f(xi)=∑m=1pwmxim w0=wTxi

f(x_i) = \sum_{m=1}^{p}w_m x_{im} w_0={w}^T{x_i}

J(w)=1n∑i=1n(yi−f(xi))2=1n∥y−Xw∥2

J(w)=\frac{1}{n}\sum_{i=1}^{n}(y_i – f(x_i))^2=\frac{1}{n}\|y-Xw\|^2

w^=(XTX)−1XTy

\hat{w}=(X^{T}X)^{-1}X^{T}y

y^=Xw^=X(XTX)−1XTy

\hat{y} = X\hat{w} = X(X^{T}X)^{-1}X^{T}y

Var(β^)=(XTX)−1σ2

Var(\hat{\beta}) = (X^{T}X)^{-1}\sigma^2

Var(β^)=(XTX)−1XTyytX(XTX)−1=(XTX)−1σ2

Var(\hat{\beta}) = (X^{T}X)^{-1}X^{T}yy^{t}X(X^{T}X)^{-1}=(X^{T}X)^{-1}\sigma^2

σ^2=1n−p−1∑i=1n(yi−y^i)2

\hat{\sigma}^2=\frac{1}{n-p-1}\sum_{ i=1}^{n}(y_i-\hat{y}_i)^2

E(σ^2)=E(1n−p−1∑i=1n(yi−y^i)2)=E(1n−p−1[y−X(XTX)−1XTy]T[y−X(XTX)−1XTy]）=E(1n−p−1yT[In−X(XTX)−1XT]y）=nσ2n−p−1−1n−p−1tr(X(XTX)−1XTyyT)=nσ2n−p−1−σ2n−p−1tr(X(XTX)−1XT)=nσ2n−p−1−(p 1)σ2n−p−1=σ2

\begin{array}{cc}
E(\hat{\sigma}^2)=E(\frac{1}{n-p-1}\sum_{ i=1}^{n}(y_i-\hat{y}_i)^2)\\
=E(\frac{1}{n-p-1}[y-X(X^{T}X)^{-1}X^{T}y]^T[y-X(X^{T}X)^{-1}X^{T}y]）\\
=E(\frac{1}{n-p-1}y^T[I_{n}-X(X^{T}X)^{-1}X^{T}]y）\\
=\frac{n\sigma^2}{n-p-1}-\frac{1}{n-p-1}\text{tr}(X(X^TX)^{-1}X^Tyy^T) \\
=\frac{n\sigma^2}{n-p-1}-\frac{\sigma^2}{n-p-1}\text{tr}(X(X^TX)^{-1}X^T) \\
=\frac{n\sigma^2}{n-p-1}-\frac{(p 1)\sigma^2}{n-p-1} \\
=\sigma^2\\
\end{array}