Introduction
In the previous posts, We have met two linear basis functions: polynomial basis function and Fourier basis function. In this one, I will review and summary them and try making some comparison. The following basis functions are implemented in python code.
Polynomial Basis Function
$$
\phi_{poly} (x) = (\phi_{0} (x), …, \phi_{M - 1} (x))^T
$$
where
$$
\phi_{i} (x) = x^{i}, i \in \{0, …, M - 1\}
$$
Gaussian Basis Function
$$
\phi_{gaussian} (x) = (\phi_{0} (x), …, \phi_{M - 1} (x))^T
$$
where
$$
\phi_{i} (x) = \exp \left\{ - \frac{(x - \mu_{i})^{2}} {2 s^{2}} \right\}, i \in \{0, …, M - 1\}
$$
Fourier Basis Function
$$
\phi_{fourier} (x) = (\phi_{0} (x), …, \phi_{2M} (x))^T
$$
where
$$
\begin{align} \phi_{0} (x) & = 1 \\ \phi_{2n - 1} (x) & = \frac{\cos{(2 \pi nx)}} {n} \\ \phi_{2n} (x) & = \frac{\sin{(2 \pi nx)}} {n}, n \in \{1, …, M\} \end{align}
$$
\(\mu_{i}\) and \(s\) control the function’s location and scale (width).
Sigmoid Basis Function
$$
\phi_{sigmoid} (x) = (\phi_{0} (x), …, \phi_{M - 1} (x))^T
$$
where
$$
\begin{align} \phi_{i} (x) & = \sigma \left( \frac{x - \mu_{i}} {s} \right), i \in \{0, …, M - 1\}\\ \sigma (a) & = \frac{1} {1 + \exp (a)} \end{align}
$$
\(\mu_{i}\) and \(s\) control the function’s location and scale (slope).
Summary
[1] Python code
[2] Lectures
[3] Basis Function
[4] Bishop, Christopher M. Pattern recognition and machine learning. springer, 2006.