# Scaling / Normalization / Standardization

Yao Yao on October 8, 2018

## 2. Standardization

1. centering: $\mathbf{x} \mapsto \mathbf{x} - \operatorname{mean}(\mathbf{x})$
2. scaling by standard deviation: $\mathbf{x} \mapsto \frac{\mathbf{x}}{\operatorname{std}(\mathbf{x})}$

• $\mu$ 是 $\operatorname{mean}(\mathbf{x})$
• $\sigma^{2}$ 是 variance $\operatorname{var}(\mathbf{x})$
• $\sigma$ 是 standard deviation (也叫 unit variance) $\operatorname{std}(\mathbf{x})$

## 3. Normalization

### 3.1 Norm

Quote from Wikipedia: Norm (mathematics):

Given a vector space $V$ over a subfield $F$ of the complex numbers, a norm on $V$ is a nonnegative-valued scalar function $p: V \to [0,+\infty)$ with the following properties:

$\forall c \in F, \mathbf{u}, \mathbf{v} \in V$,

1. $p(\mathbf{u} + \mathbf{v}) ≤ p(\mathbf{u}) + p(\mathbf{v})$
2. $p(c \mathbf{v}) = \vert c \vert p(\mathbf{v})$
3. If $p(\mathbf{v}) = 0$ then $\mathbf{v} = \mathbf{0}$ is the zero vector.

• $L_1$ norm：$\Vert \mathbf{x} \Vert_1 = \sum_{i=1}^{n} \vert x_i \vert$
• $L_2$ norm：$\Vert \mathbf{x} \Vert_2 = \sqrt{\sum_{i=1}^{n} x_i^2}$
• max norm：$\Vert \mathbf{x} \Vert_{\infty} = \max(\vert x_1 \vert, \dots, \vert x_n \vert)$

### 3.2 Normalization Scenario 1: Algebra

• $\Vert \frac{\mathbf{v}}{\Vert \mathbf{v} \Vert_{1}} \Vert_1 = 1$
• $\Vert \frac{\mathbf{v}}{\Vert \mathbf{v} \Vert_{2}} \Vert_2 = 1$
• $\Vert \frac{\mathbf{v}}{\Vert \mathbf{v} \Vert_{\infty}} \Vert_{\infty} = 1$

• unit vector 的 norm 为 1，只能说明各项 $x_i$ 的绝对值是在 $[0, 1]$ 区间内

### 3.3 Normalization Scenario 2: Statistics

• standardization 是一种 normalization
• studentization 是一种 normalization
• min-max scaling 也是一种 normalization (而且它不符合我们的 scaling 定义)

### 3.4 The Problems with Normalization

• sklearn 的 standardization 要用 sklearn.preprocessing.StandardScaler

......:
......:
# https://github.com/scikit-learn/scikit-learn/blob/bac89c2/sklearn/preprocessing/data.py#L1564
if norm == 'l1':
norms = np.abs(X).sum(axis=1)
elif norm == 'l2':
norms = row_norms(X)
elif norm == 'max':
norms = np.max(X, axis=1)  # You call this max norm???
norms = _handle_zeros_in_scale(norms, copy=False)
X /= norms[:, np.newaxis]