Gpytorch

**Important**: This release requires Python 3.7 (up from 3.6) and PyTorch 1.10 (up from 1.9)

New Features
- gpytorch.metrics module offers easy-to-use metrics for GP performance.(1870) This includes:
- gpytorch.metrics.mean_absolute_error
- gpytorch.metrics.mean_squared_error
- gpytorch.metrics.mean_standardized_log_loss
- gpytorch.metrics.negative_log_predictive_density
- gpytorch.metrics.quantile_coverage_error
- Large scale inference (using matrix-multiplication techniques) now implements the variance reduction scheme described in [Wenger et al., ICML 2022](https://arxiv.org/abs/2107.00243). (#1836)
- This makes it possible to use LBFGS, or other line search based optimization techniques, with large scale (exact) GP hyperparameter optimization.
- Variational GP models support online updates (i.e. “fantasizing new models). (1874)
- This utilizes the method described in [Maddox et al., NeurIPS 2021](https://papers.nips.cc/paper/2021/hash/325eaeac5bef34937cfdc1bd73034d17-Abstract.html)
- Improvements to gpytorch.priors
- New HalfCauchyPrior (1961)
- LKJPrior now supports sampling (1737)

Minor Features
- Add LeaveOneOutPseudoLikelihood for hyperparameter optimization (1989)
- The PeriodicKernel now supports ARD lengthscales/periods (1919)
- LazyTensors (A) can now be matrix multiplied with tensors (B) from the left hand side (i.e. B x A) (1932)
- Maximum Cholesky retries can be controlled through a setting (1861)
- Kernels, means, and likelihoods can be pickled (1876)
- Minimum variance for FixedNoiseGaussianLikelihood can be set with a context manager (2009)

Bug Fixes
- Fix backpropagation issues with KeOps kernels (1904)
- Fix broadcasting issues with lazily evaluated kernels (1971)
- Fix batching issues with PolynomialKernel (1977)
- Fix issues with PeriodicKernel.diag() (1919)
- Add more informative error message when train targets and the train prior distribution mismatch (1905)
- Fix issues with priors on ConstantMean (2042)

This release contains several bug fixes and performance improvements.

New Features
- Variational multitask models can output a single task per input (rather than all tasks per input) (1769)

Small fixes
- LazyTensorto method more closely matches the torch Tensor API (1746)
- Add type hints and exceptions to kernels to improve usability (1802)

Performance
- Improve the speed of fantasy models (1752)
- Improve the speed of solves and log determinants with KroneckerProductLazyTensor (1786)
- Prevent explicit kernel evaluation when expanding a LazyTensor kernel (1813)

Fixes
- Fix indexing bugs with kernels (1802, 1819, 1828)
- Fix cholesky bugs on CUDA (1848)
- Remove lines of code that generate warnings in PyTorch 1.9 (1835)

New features

- Add `gpytorch.kernels.PiecewisePolynomialKernel` (1738)
- Include ability to turn off diagonal correction for SGPR models (1717)
- Include ability to cast LazyTensor to half and float types (1726)


Performance improvements

- Specialty MVN log_prob method for Gaussians with sum-of-Kronecker covariances (1674)
- Ability to specify devices when concatenating rows of LazyTensors (1712)
- Improvements to LazyTensor symeig method (1725)


Bug fixes

- Fix to computing batch sizes of kernels (1685)
- Fix SGPR prediction when `fast_computations` flags are turned off (1709)
- Improve stability of `stable_qr` function (1714)
- Fix bugs with pyro integration for full Bayesian inference (1721)
- `num_classes` in `gpytorch.likelihoods.DirichletLikelihood` should be an integer (1728)

This release adds 2 new model classes, as well as a number of bug fixes:
- GPLVM models for unsupervised learning
- Polya-Gamma GPs for GP classification
In addition, this release contains numerous improvements to SGPR models (that have also been included in prior bug-fix releases).

New features
- Add example notebook that demos binary classification with Polya-Gamma augmentation (1523)
- New model class: Bayesian GPLVM with Stochastic Variational Inference (1605)
- Periodic kernel handles multi-dimensional inputs (1593)
- Add missing data gaussian likelihoods (1668)

Performance
- Speed up SGPR models (1517, 1528, 1670)

Fixes
- Fix erroneous loss for ExactGP multitask models (1647)
- Fix pyro sampling (1594)
- Fix initialize bug for additive kernels (1635)
- Fix matrix multiplication of rectangular ZeroLazyTensor (1295)
- Dirichlet GPs use true train targets not labels (1641)

Various bug fixes, including

- Use current PyTorch functionality (1611, 1586)
- Bug fixes to Lanczos factorization (1607)
- Fixes to SGPR model (1607)
- Various fixes to LazyTensor math (1576, 1584)
- SmoothedBoxPrior has a sample method (1546)
- Fixes to additive-structure models (1582)
- Doc fixes {1603)
- Fix to index kernel and LCM kernels (1608, 1592)
- Fixes to KeOps bypass (1609)

Fixes
- Simplify interface for 3+ layer DSPP models (1565)
- Fix marginal log likelihood calculation for exact Bayesian inference w/ Pyro (1571)
- Remove CG warning for small matrices (1562)
- Fix Pyro cluster-multitask example notebook (1550)
- Fix gradients for KeOps tensors (1543)
- Ensure that gradients are passed through lazily-evaluated kernels (1518)
- Fix bugs for models with batched fantasy observations (1529, 1499)
- Correct default `latent_dim` value for LMC variational models (1512)

New features
- Create `gpytorch.utils.grid.ScaleToBounds` utility to replace `gpytorch.utils.grid.scale_to_bounds` method (1566)
- Fix skip connections in Deep GP example (1531)
- Add fantasy point support for structured kernel interpolation models (1545)

Documentation
- Add default values to all gpytorch.settings (1564)
- Improve Hadamard multitask notebook (1537)

Performance
- Speed up SGPR models (1517, 1528)