Pyro-ppl

New features

- New Tutorials:
- [Deep Generative Modeling with Single Cell Data](http://pyro.ai/examples/scanvi.html)
- [Conditional Variational Autoencoder](http://pyro.ai/examples/cvae.html)
- [Forecasting with Dynamic Linear Models](http://pyro.ai/examples/forecasting_dlm.html)
- New distributions and transforms:
- [OrderedLogistic](http://docs.pyro.ai/en/latest/distributions.html#orderedlogistic) distribution and [OrderedTransform](http://docs.pyro.ai/en/latest/distributions.html#orderedtransform)
- [ConditionalMatrixExponential](http://docs.pyro.ai/en/latest/distributions.html#conditionalmatrixexponential) normalizing flow
- [ConditionalSplineAutoregressive](http://docs.pyro.ai/en/latest/distributions.html#conditionalsplineautoregressive) normalizing flow
- [contrib.forecast](http://docs.pyro.ai/en/latest/contrib.forecast.html#pyro.contrib.forecast.forecaster.Forecaster) now supports [HaarReparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.haar.HaarReparam), [DiscreteCosineReparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.discrete_cosine.DiscreteCosineReparam), and [poutine.trace](http://docs.pyro.ai/en/latest/poutine.html#pyro.poutine.handlers.trace) to record posterior samples of latent variables.
- [CompartmentalModel](http://docs.pyro.ai/en/latest/contrib.epidemiology.html#pyro.contrib.epidemiology.compartmental.CompartmentalModel) now supports a [.finalize()](http://docs.pyro.ai/en/latest/contrib.epidemiology.html#pyro.contrib.epidemiology.compartmental.CompartmentalModel.finalize) method to add likelihoods that couple states across time.
- Integration with [Funsor](https://github.com/pyro-ppl/funsor), an experimental intermediate language for probabilistic programming
- [pyro.contrib.funsor](http://docs.pyro.ai/en/latest/contrib.funsor.html) is a new backend for Pyro that aims to simplify the implementations of Pyro's most powerful inference engines. For details, see: [tutorial 1](http://pyro.ai/examples/contrib_funsor_intro_i.html), [tutorial 2](http://pyro.ai/examples/contrib_funsor_intro_ii.html), [example usage with `pyroapi`](https://github.com/pyro-ppl/pyro/blob/dev/examples/contrib/funsor/hmm.py)
- [poutine.collapse](http://docs.pyro.ai/en/latest/poutine.html#pyro.poutine.handlers.collapse) and [pyro.barrier](http://docs.pyro.ai/en/latest/primitives.html#pyro.primitives.barrier) provide experimental support for collapsing conjugate fragments within existing inference algorithms, using [Funsor](https://github.com/pyro-ppl/funsor) under the hood.

Breaking changes
- Require PyTorch 1.6.
- Drop support for Python 3.5; require Python 3.6+.
- Zero inflated distributions changed interface. 2643

Bug fixes & performance tweaks
- [pyro.factor](http://docs.pyro.ai/en/stable/primitives.html#pyro.primitives.factor) statements are now allowed in guides without warning. 2664
- Fix model-directed subsampling in autoguides. 2638
- Fix sample shape bug in [LKJCorrCholesky](http://docs.pyro.ai/en/stable/distributions.html#lkjcorrcholesky) distribution. 2617
- Speed up log-matmul-exp operations in discrete enumeration and [DiscreteHMM](http://docs.pyro.ai/en/stable/distributions.html#discretehmm). 2640
- Fix ``potential_fn`` issues in MCMC. 2591

New features

- A new [pyro.contrib.epidemiology](http://docs.pyro.ai/en/latest/contrib.epidemiology.html) module for discrete-state discrete-time stochastic compartmental models. #2426
- New tutorials on:
- [Normalizing flows](https://pyro.ai/examples/normalizing_flows_i.html)
- [Epidemiology](https://pyro.ai/examples/epi_intro.html)
- [PyroModule](https://pyro.ai/examples/modules.html)
- [MAP/MLE inference](https://pyro.ai/examples/mle_map.html)
- New transforms and normalizing flows:
- Monotonic rational quadratic [Spline](http://docs.pyro.ai/en/latest/distributions.html#spline) flow.
- A [SplineCoupling](http://docs.pyro.ai/en/latest/distributions.html#splinecoupling) flow.
- A [SplineAutogregressive](http://docs.pyro.ai/en/latest/distributions.html#splineautoregressive) flow.
- A [MatrixExponential](http://docs.pyro.ai/en/latest/distributions.html#matrixexponential) flow.
- Conditional versions of `AffineAutoregressive`, `Householder`, `NeuralAutogregressive`, `Spline`, and `GeneralizedChannelPermute` flows.
- A Haar wavelet [transform](http://docs.pyro.ai/en/latest/distributions.html#pyro.distributions.transforms.HaarTransform) and [reparameterizer](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.haar.HaarReparam).
- `Permute` and `AffineCoupling` can operate on a specific dimension with `dim` keyword argument 2472
- New distributions:
- [CoalescentTimes](http://docs.pyro.ai/en/latest/distributions.html#coalescenttimes), [CoalescentTimesWithRate](http://docs.pyro.ai/en/latest/distributions.html#coalescenttimeswithrate), and [CoalescentRateLikelihood](http://docs.pyro.ai/en/latest/contrib.epidemiology.html#pyro.distributions.CoalescentRateLikelihood) for coalescent processes in phylogenetics.
- [TruncatedPolyGamma](http://docs.pyro.ai/en/latest/distributions.html#pyro.distributions.TruncatedPolyaGamma).
- [ExtendedBinomial](http://docs.pyro.ai/en/latest/distributions.html#extendedbinomial) and [ExtendedBetaBinomial](http://docs.pyro.ai/en/latest/distributions.html#extendedbetabinomial) with relaxed support.
- [ImproperUniform](http://docs.pyro.ai/en/latest/distributions.html#improperuniform) for expressing factor graphs.
- Improvements to MCMC:
- structured mass matrix adaptation 2425 2473
- arrowhead precision matrix adaptation 2465
- support for randomized init strategies like [init_to_generated](http://pyro.ai/examples/epi_intro.html) #2417
- Improvements to SMC:
- systematic resampling 2488
- resampling based on effective sample size (ESS) 2486
- A new [SplitReparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.split.SplitReparam) allows different inference strategies to apply to different parts of a tensor random variable.
- A new initialization strategy [init_to_generated](http://docs.pyro.ai/en/latest/infer.autoguide.html#pyro.infer.autoguide.initialization.init_to_generated).
- A [RandomVariable](http://docs.pyro.ai/en/latest/contrib.randomvariable.html#pyro.contrib.randomvariable.random_variable.RandomVariable) container class to support method chaining syntax for transforming distributions 2448

Bug fixes

- Support sequential plates in RenyiELBO 2541
- Fixes to `AffineAutogregressive` 2504
- Fixes to `BatchNorm` `TransformModule` 2459
- Fixes to how some transforms handle parameters 2544
- Fixes to reraising logic that clean up error reporting during inference 2494
- [many other fixes](https://github.com/pyro-ppl/pyro/pulls?q=is%3Apr+is%3Aclosed+merged%3A%3E2020-04-06+merged%3A%3C%3D2020-07-18) to documentation and code

New features

- A new [Spline](http://docs.pyro.ai/en/latest/distributions.html#spline) transform which implements element-wise rational spline bijections of linear order.
- A new [ConditionalAffineCoupling](http://docs.pyro.ai/en/latest/distributions.html#conditionalaffinecoupling) transform which implements the affine coupling layer of RealNVP that conditions on an additional context variable.

Enhancements to the [pyro.contrib.forecast](http://docs.pyro.ai/en/stable/contrib.forecast.html) module

- Support drawing samples in batches.
- Add walltime to backtest to measure performance of model training and forecasting.
- Support more likelihood distributions: Geometric, NegativeBinomial, ZeroInflatedNegativeBinomial.

Bug fixes

- 2399 raises an error when HMC/NUTS is used for a model with subsampling.
- 2390 makes `PyroModule` compatible with `torch.nn.RNN`.
- 2388 allows unused params in CSIS inference.
- 2384 fixes some caching issues in calculation of `log_abs_det_jacobian` of TransformModules
- 2365 fixes a naming bug in `LocScaleReparam` whereby all loc-scale reparameterized sites shared a single centeredness parameter.
- 2355 makes `jit_compile=True` flag in HMC/NUTS work for models with `pyro.param` statements.

New features

- A new [AutoNormal](http://docs.pyro.ai/en/latest/infer.autoguide.html#pyro.infer.autoguide.AutoNormal) guide that supports data subsampling, thanks to patrickeganfoley.
- Data subsampling support for the [AutoDelta](http://docs.pyro.ai/en/latest/infer.autoguide.html#pyro.infer.autoguide.AutoDelta) guide.
- A new [pyro.subsample](http://docs.pyro.ai/en/latest/primitives.html#pyro.primitives.subsample) primitive to aide in subsampling.
- An [AutoNormalizingFlow](http://docs.pyro.ai/en/latest/infer.autoguide.html#pyro.infer.autoguide.AutoNormalizingFlow) autoguide.
- A new [pyro.contrib.forecast](http://docs.pyro.ai/en/latest/contrib.forecast.html) module for multivariate hierarchical heavy-tailed forecasting, together with three tutorials: [uivariate, heavy-tailed](http://pyro.ai/examples/forecasting_i.html), [state space models](http://pyro.ai/examples/forecasting_ii.html), and [hierarchical models](http://pyro.ai/examples/forecasting_iii.html).
- A tutorial on [Boosting Black Box Variational Inference](http://pyro.ai/examples/boosting_bbvi.html), thanks to lorenzkuhn, gideonite, sharrison5, and TNU-yaoy.
- A [NeuTraReparam](http://pyro.ai/examples/neutra.html) example.
- [PyroModule]()'s interface is now stable, no longer EXPERIMENTAL.
- Better validation for [GaussianHMM](http://docs.pyro.ai/en/latest/distributions.html#gaussianhmm) and [LinearHMM](http://docs.pyro.ai/en/latest/distributions.html#linearhmm) distributions.
- Miscellaneous new [GaussianHMM](http://docs.pyro.ai/en/latest/distributions.html#gaussianhmm) method to handle conjugacy.

Bug fixes

- 2345 remove pillow-simd dependency
- 2327 Make pyro.deterministic not warn when called outside of inference
- 2321 Support plates in RenyiELBO
- 2266 Fixes to transform handling in MCMC api

Patches [1.2.0](https://github.com/pyro-ppl/pyro/releases/tag/1.2.0) with the following bug fixes:
- Fix for MCMC with parallel chains using multiprocessing, where transforms to the latent sites' support was not being correctly stored.
- Other minor rendering related fixes for tutorials.

Misc changes

* Updated to PyTorch 1.4.0 and torchvision 0.5.0.
* Changed license from MIT to Apache 2.0 and removed Uber CLA as part of Pyro's move to the [Linux foundation](https://lfai.foundation).

Reparameterization

This release adds a new effect handler and a collection of strategies that reparameterize models to improve geometry. These tools are largely orthogonal to other inference tools in Pyro, and can be used with SVI, MCMC, and other inference algorithms.

* [poutine.reparam()](http://docs.pyro.ai/en/latest/poutine.html#pyro.poutine.handlers.reparam) is a new effect handler that transforms models into other models for which inference may be easier [(Gorinova et al. 2019)](https://arxiv.org/abs/1906.03028).
* [pyro.infer.reparam](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam) is a collection of reparameterization strategies following a standard [Reparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.reparam.Reparam) interface:
* [Decentering transforms](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.loc_scale) for location-scale families [(Gorinova et al. 2019)](https://arxiv.org/abs/1906.03028).
* [Transform unwrapping](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.transform) to deconstruct `TransformedDistribution`s.
* [Discrete Cosine transforms](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.discrete_cosine) for frequency-domain parameterizations (useful for inference in time series).
* Auxiliary variable methods for [Levy Stable](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.stable) and [StudentT](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.studentt) distributions.
* [Linear Hidden Markov Model](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.hmm) reparameterization, allowing a range of non-Gaussian HMMs to be treated as conditional Gaussian processes.
* [Neural Transport](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.neutra) uses SVI to learn the geometry of a model before drawing samples using HMC [(Hoffman et al. 2019)](https://arxiv.org/abs/1903.03704).
* A [tutorial](http://pyro.ai/examples/stable.html) on inference with [Levy Stable](http://docs.pyro.ai/en/latest/distributions.html#stable) distrubutions, demonstrating [StableReparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.stable.StableReparam), [DiscreteCosineReparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.discrete_cosine.DiscreteCosineReparam), and [EnergyDistance](http://docs.pyro.ai/en/latest/inference_algos.html#pyro.infer.energy_distance.EnergyDistance).


Other new features
* A [tutorial](http://pyro.ai/examples/dirichlet_process_mixture.html) on Dirichlet process mixture modeling, contributed by m-k-S
* Added a [LinearHMM](http://docs.pyro.ai/en/latest/distributions.html#linearhmm) distribution with an `.rsample()` method. This supports non-Gaussian noise such as [Levy Stable](http://docs.pyro.ai/en/latest/distributions.html#stable) and [StudentT](http://docs.pyro.ai/en/latest/distributions.html#studentt), but requires [reparameterization](http://docs.pyro.ai/en/latest/infer.reparam.html) for inference.
* Implemented a [GaussianHMM.rsample()](http://docs.pyro.ai/en/latest/distributions.html#pyro.distributions.GaussianHMM.rsample) method for drawing joint samples from a linear-Gaussian HMM.
* Added a [LowerCholeskyAffine](http://docs.pyro.ai/en/latest/distributions.html#lowercholeskyaffine) transform.
* [2264](https://github.com/pyro-ppl/pyro/pull/2264) improves speed and numerical stability of `MultivariateNormal` conversion from `scale_tril` to `precision`.

Bug fixes

- [2263](https://github.com/pyro-ppl/pyro/pull/2263) fixes MCMC api to allow implementations other than HMC and NUTS.
- [2244](https://github.com/pyro-ppl/pyro/pull/2244) fixes an `event_dim` issue in `ConditionedPlanar` flow.
- [2243](https://github.com/pyro-ppl/pyro/pull/2243) fixes a bug in `AffineCoupling`.
- [2227](https://github.com/pyro-ppl/pyro/issues/2227) fixes device placement of the `MultivariateStudentT.df` param.
- [2226](https://github.com/pyro-ppl/pyro/pull/2226) fixes an edge case bug in discrete enumeration.