Merge branch 'master' of github.com:SheffieldML/GPy

GPy/kern/rational_quadratic.py

@@ -11,7 +11,7 @@ class rational_quadratic(kernpart):
 
     .. math::
 
-       k(r) = \sigma^2 \left(1 + \frac{r^2}{2 \ell^2})^{- \alpha} \ \ \ \ \  \\text{ where  } r^2 = (x-y)^2
+       k(r) = \sigma^2 \\bigg( 1 + \\frac{r^2}{2 \ell^2} \\bigg)^{- \\alpha} \ \ \ \ \  \\text{ where  } r^2 = (x-y)^2
 
     :param D: the number of input dimensions
     :type D: int (D=1 is the only value currently supported)
@@ -19,6 +19,8 @@ class rational_quadratic(kernpart):
     :type variance: float
     :param lengthscale: the lengthscale :math:`\ell`
     :type lengthscale: float
+    :param power: the power :math:`\\alpha`
+    :type power: float
     :rtype: kernpart object
 
     """
@@ -76,4 +78,3 @@ class rational_quadratic(kernpart):
 
     def dKdiag_dX(self,dL_dKdiag,X,target):
         pass
-    

doc/GPy.examples.rst

@@ -9,14 +9,6 @@ examples Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`BGPLVM_demo` Module
--------------------------
-
-.. automodule:: GPy.examples.BGPLVM_demo
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 :mod:`classification` Module
 ----------------------------
 
@@ -25,18 +17,18 @@ examples Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`oil_flow_demo` Module
+:mod:`dimensionality_reduction` Module
----------------------------
+--------------------------------------
 
-.. automodule:: GPy.examples.oil_flow_demo
+.. automodule:: GPy.examples.dimensionality_reduction
     :members:
     :undoc-members:
     :show-inheritance:
 
-:mod:`poisson` Module
+:mod:`non_gaussian` Module
----------------------
+--------------------------
 
-.. automodule:: GPy.examples.poisson
+.. automodule:: GPy.examples.non_gaussian
     :members:
     :undoc-members:
     :show-inheritance:
@@ -49,30 +41,6 @@ examples Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`sparse_GPLVM_demo` Module
--------------------------------
-
-.. automodule:: GPy.examples.sparse_GPLVM_demo
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-:mod:`sparse_GP_regression_demo` Module
----------------------------------------
-
-.. automodule:: GPy.examples.sparse_GP_regression_demo
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-:mod:`sparse_ep_fix` Module
----------------------------
-
-.. automodule:: GPy.examples.sparse_ep_fix
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 :mod:`tutorials` Module
 -----------------------
 
@@ -81,27 +49,3 @@ examples Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`uncertain_input_GP_regression_demo` Module
-------------------------------------------------
-
-.. automodule:: GPy.examples.uncertain_input_GP_regression_demo
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-:mod:`uncollapsed_GP_demo` Module
----------------------------------
-
-.. automodule:: GPy.examples.uncollapsed_GP_demo
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-:mod:`unsupervised` Module
---------------------------
-
-.. automodule:: GPy.examples.unsupervised
-    :members:
-    :undoc-members:
-    :show-inheritance:
-

doc/GPy.kern.rst

@@ -137,6 +137,14 @@ kern Package
     :undoc-members:
     :show-inheritance:
 
+:mod:`rational_quadratic` Module
+--------------------------------
+
+.. automodule:: GPy.kern.rational_quadratic
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 :mod:`rbf` Module
 -----------------
 

doc/GPy.models.rst

@@ -9,10 +9,10 @@ models Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`BGPLVM` Module
+:mod:`Bayesian_GPLVM` Module
---------------------
+----------------------------
 
-.. automodule:: GPy.models.BGPLVM
+.. automodule:: GPy.models.Bayesian_GPLVM
     :members:
     :undoc-members:
     :show-inheritance:

doc/GPy.rst

@@ -9,14 +9,6 @@ GPy Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`test_coreg` Module
-------------------------
-
-.. automodule:: GPy.test_coreg
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 Subpackages
 -----------
 

doc/GPy.testing.rst

@@ -0,0 +1,59 @@
+testing Package
+===============
+
+:mod:`bgplvm_tests` Module
+--------------------------
+
+.. automodule:: GPy.testing.bgplvm_tests
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`examples_tests` Module
+----------------------------
+
+.. automodule:: GPy.testing.examples_tests
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`gplvm_tests` Module
+-------------------------
+
+.. automodule:: GPy.testing.gplvm_tests
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`kernel_tests` Module
+--------------------------
+
+.. automodule:: GPy.testing.kernel_tests
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`prior_tests` Module
+-------------------------
+
+.. automodule:: GPy.testing.prior_tests
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`sparse_gplvm_tests` Module
+--------------------------------
+
+.. automodule:: GPy.testing.sparse_gplvm_tests
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`unit_tests` Module
+------------------------
+
+.. automodule:: GPy.testing.unit_tests
+    :members:
+    :undoc-members:
+    :show-inheritance:
+

doc/index.rst

@@ -10,8 +10,7 @@ For a quick start, you can have a look at one of the tutorials:
 * `Basic Gaussian process regression <tuto_GP_regression.html>`_  
 * `Interacting with models <tuto_interacting_with_models.html>`_
 * `A kernel overview <tuto_kernel_overview.html>`_ 
-* Advanced GP regression (Forthcoming)
+* `Writing new kernels <tuto_creating_new_kernels.html>`_
-* Writing kernels (Forthcoming)
 
 You may also be interested by some examples in the GPy/examples folder.
 

doc/tuto_creating_new_kernels.rst

@@ -0,0 +1,183 @@
+********************
+Creating new kernels
+********************
+
+We will see in this tutorial how to create new kernels in GPy. We will also give details on how to implement each function of the kernel and illustrate with a running example: the rational quadratic kernel. 
+
+Structure of a kernel in GPy
+============================
+
+In GPy a kernel object is made of a list of kernpart objects, which correspond to symetric positive definite functions. More precisely, the kernel should be understood as the sum of the kernparts. In order to implement a new covariance, the following steps must be followed
+
+    1. implement the new covariance as a kernpart object
+    2. update the constructors that allow to use the kernpart as a kern object
+    3. update the __init__.py file
+
+Theses three steps are detailed below.
+
+Implementing a kernpart object
+==============================
+
+We advise the reader to start with copy-pasting an existing kernel and to modify the new file. We will now give a description of the various functions that can be found in a kernpart object.
+
+**Header**
+
+The header is similar to all kernels::
+
+    from kernpart import kernpart
+    import numpy as np
+
+    class rational_quadratic(kernpart):
+
+**__init__(self,D, param1, param2, ...)**
+
+The implementation of this function in mandatory.
+
+For all kernparts the first parameter ``D`` corresponds to the dimension of the input space, and the following parameters stand for the parameterization of the kernel.
+
+The following attributes are compulsory: ``self.D`` (the dimension, integer), ``self.name`` (name of the kernel, string), ``self.Nparam`` (number of parameters, integer).::
+
+    def __init__(self,D,variance=1.,lengthscale=1.,power=1.):
+        assert D == 1, "For this kernel we assume D=1"
+        self.D = D
+        self.Nparam = 3
+        self.name = 'rat_quad'
+        self.variance = variance
+        self.lengthscale = lengthscale
+        self.power = power
+
+**_get_params(self)**
+
+The implementation of this function in mandatory.
+
+This function returns a one dimensional array of length ``self.Nparam`` containing the value of the parameters.::
+
+    def _get_params(self):
+        return np.hstack((self.variance,self.lengthscale,self.power))
+
+**_set_params(self,x)**
+
+The implementation of this function in mandatory.
+
+The input is a one dimensional array of length ``self.Nparam`` containing the value of the parameters. The function has no output but it updates the values of the attribute associated to the parameters (such as ``self.variance``, ``self.lengthscale``, ...).::
+
+    def _set_params(self,x):
+        self.variance = x[0]
+        self.lengthscale = x[1]
+        self.power = x[2]
+
+**_get_param_names(self)**
+
+The implementation of this function in mandatory.
+
+It returns a list of strings of length ``self.Nparam`` corresponding to the parameter names.::
+
+    def _get_param_names(self):
+        return ['variance','lengthscale','power']
+
+**K(self,X,X2,target)**
+
+The implementation of this function in mandatory.
+
+This function is used to compute the covariance matrix associated with the inputs X, X2 (np.arrays with arbitrary number of line (say :math:`n_1`, :math:`n_2`) and ``self.D`` columns). This function does not returns anything but it adds the :math:`n_1 \times n_2` covariance matrix to the kernpart to the object ``target`` (a :math:`n_1 \times n_2` np.array). This trick allows to compute the covariance matrix of a kernel containing many kernparts with a limited memory use.::
+
+    def K(self,X,X2,target):
+        if X2 is None: X2 = X
+        dist2 = np.square((X-X2.T)/self.lengthscale)
+        target += self.variance*(1 + dist2/2.)**(-self.power)
+
+**Kdiag(self,X,target)**
+
+The implementation of this function in mandatory.
+
+This function is similar to ``K`` but it computes only the values of the kernel on the diagonal. Thus, ``target`` is a 1-dimensional np.array of length :math:`n_1`.::
+
+    def Kdiag(self,X,target):
+        target += self.variance    
+
+
+**dK_dtheta(self,dL_dK,X,X2,target)**
+
+This function is required for the optimization of the parameters.
+
+Computes the derivative of the likelihood. As previously, the values are added to the object target which is a 1-dimensional np.array of length ``self.Nparam``. For example, if the kernel is parameterized by :math:`\sigma^2,\ \theta`, then :math:`\frac{dL}{d\sigma^2} = \frac{dL}{d K} \frac{dK}{d\sigma^2}` is added to the first element of target and :math:`\frac{dL}{d\theta} = \frac{dL}{d K} \frac{dK}{d\theta}` to the second.::
+
+    def dK_dtheta(self,dL_dK,X,X2,target):
+        if X2 is None: X2 = X
+        dist2 = np.square((X-X2.T)/self.lengthscale)
+
+        dvar = (1 + dist2/2.)**(-self.power)
+        dl = self.power * self.variance * dist2 * self.lengthscale**(-3) * (1 + dist2/2./self.power)**(-self.power-1)
+        dp = - self.variance * np.log(1 + dist2/2.) * (1 + dist2/2.)**(-self.power)
+
+        target[0] += np.sum(dvar*dL_dK)
+        target[1] += np.sum(dl*dL_dK)
+        target[2] += np.sum(dp*dL_dK)
+
+
+**dKdiag_dtheta(self,dL_dKdiag,X,target)**
+
+This function is required for BGPLVM, sparse models and uncertain inputs.
+
+As previously, target is an ``self.Nparam`` array and :math:`\frac{dL}{d Kdiag} \frac{dKdiag}{dparam}` is added to each element.::
+
+    def dKdiag_dtheta(self,dL_dKdiag,X,target):
+        target[0] += np.sum(dL_dKdiag)
+        # here self.lengthscale and self.power have no influence on Kdiag so target[1:] are unchanged
+
+**dK_dX(self,dL_dK,X,X2,target)**
+
+This function is required for GPLVM, BGPLVM, sparse models and uncertain inputs.
+
+Computes the derivative of the likelihood with respect to the inputs ``X`` (a :math:`n \times D` np.array). The result is added to target which is a :math:`n \times D` np.array.::
+
+    def dK_dX(self,dL_dK,X,X2,target):
+        """derivative of the covariance matrix with respect to X."""
+        if X2 is None: X2 = X
+        dist2 = np.square((X-X2.T)/self.lengthscale)
+
+        dX = -self.variance*self.power * (X-X2.T)/self.lengthscale**2 *  (1 + dist2/2./self.power)**(-self.power-1)
+        target += np.sum(dL_dK*dX)
+
+**dKdiag_dX(self,dL_dKdiag,X,target)**
+
+This function is required for BGPLVM, sparse models and uncertain inputs. As for ``dKdiag_dtheta``, :math:`\frac{dL}{d Kdiag} \frac{dKdiag}{dX}` is added to each element of target.::
+
+    def dKdiag_dX(self,dL_dKdiag,X,target):
+        pass
+
+**Psi statistics**
+
+The psi statistics and their derivatives are required for BGPLVM and GPS with uncertain inputs.
+
+The expressions of the psi statistics are:
+
+TODO
+
+For the rational quadratic we have:
+
+TODO
+
+Update the constructor 
+======================
+
+Once the required functions have been implemented as a kernpart object, the file GPy/kern/constructors.py has to be updated to allow to build a kernel based on the kernpart object.
+
+The following line should be added in the preamble of the file::
+
+    from rational_quadratic import rational_quadratic as rational_quadratic_part
+
+as well as the following block::
+
+    def rational_quadratic(D,variance=1., lengthscale=1., power=1.):
+        part = rational_quadraticpart(D,variance, lengthscale, power)
+        return kern(D, [part])
+
+
+Update initialization
+=====================
+
+The last step is to update the list of kernels imported from constructor in GPy/kern/__init__.py.
+
+
+