Nbconvert

The following 18 authors contributed 144 commits -- Thank you all!

- Benjamin Ragan-Kelley
- Clayton A Davis
- DInne Bosman
- Doug Blank
- Henrique Silva
- Jeff Hale
- Lukasz Mitusinski
- M Pacer
- Maarten Breddels
- Madhumitha N
- Matthew Seal
- Paul Gowder
- Philipp A
- Rick Lupton
- Rüdiger Busche
- Thomas Kluyver
- Tyler Makaro
- WrRan

The full list of changes they made can be seen [on
GitHub](https://github.com/jupyter/nbconvert/issues?q=milestone%3A5.5+)

Significant Changes

Deprecations

Python 3.4 support was dropped. Many of our upstream libraries stopped
supporting 3.4 and it was found that serious bugs were being caught
during testing against those libraries updating past 3.4.

See [979](https://github.com/jupyter/nbconvert/pull/979) for details.

IPyWidget Support

Now when a notebook executing contains [Jupyter
Widgets](https://github.com/jupyter-widgets/ipywidgets/), the state of
all the widgets can be stored in the notebook's metadata. This allows
rendering of the live widgets on, for instance nbviewer, or when
converting to html.

You can tell nbconvert to not store the state using the
`store_widget_state` argument:


jupyter nbconvert --ExecutePreprocessor.store_widget_state=False --to notebook --execute mynotebook.ipynb


This widget rendering is not performed against a browser during
execution, so only widget default states or states manipulated via user
code will be calculated during execution. `%%javascript` cells will
execute upon notebook rendering, enabling complex interactions to
function as expected when viewed by a UI.

If you can't view widget results after execution, you may need to
select `File --> Trust Notebook` in the menu.

See [779](https://github.com/jupyter/nbconvert/pull/779), [#900](https://github.com/jupyter/nbconvert/pull/900), and [#983](https://github.com/jupyter/nbconvert/pull/983) for details.

Execute Preprocessor Rework

Based on monkey patching required in
[papermill](https://github.com/nteract/papermill/blob/0.19.1/papermill/preprocess.py)
the `run_cell` code path in the ExecutePreprocessor was reworked to
allow for accessing individual message parses without reimplementing the
entire function. Now there is a `process_message` function which take a
ZeroMQ message and applies all of its side-effect updates on the
cell/notebook objects before returning the output it generated, if it
generated any such output.

The change required a much more extensive test suite covering cell
execution as test coverage on the various, sometimes wonky, code paths
made improvements and reworks impossible to prove undamaging. Now
changes to kernel message processing has much better coverage, so future
additions or changes with specs over time will be easier to add.

See [905](https://github.com/jupyter/nbconvert/pull/905) and [#982](https://github.com/jupyter/nbconvert/pull/982) for details

Out Of Memory Kernel Failure Catches

When running out of memory on a machine, if the kernel process was
killed by the operating system it would result in a timeout error at
best and hang indefinitely at worst. Now regardless of timeout
configuration, if the underlying kernel process dies before emitting any
messages to the effect an exception will be raised notifying the
consumer of the lost kernel within a few seconds.

See [959](https://github.com/jupyter/nbconvert/pull/959), [#971](https://github.com/jupyter/nbconvert/pull/971), and [#998](https://github.com/jupyter/nbconvert/pull/998) for details

Latex / PDF Template Improvements

The latex template was long overdue for improvements. The default
template had a rewrite which makes exports for latex and pdf look a lot
better. Code cells in particular render much better with line breaks and
styling the more closely matches notebook browser rendering. Thanks
t-makaro for the efforts here!

See [992](https://github.com/jupyter/nbconvert/pull/992) for details

Comprehensive notes

New Features

- IPyWidget Support [779](https://github.com/jupyter/nbconvert/pull/779),
[900](https://github.com/jupyter/nbconvert/pull/900), and [#983](https://github.com/jupyter/nbconvert/pull/983)
- A new ClearMetadata Preprocessor is available
[805](https://github.com/jupyter/nbconvert/pull/805)
- Support for pandoc 2 [964](https://github.com/jupyter/nbconvert/pull/964)
- New, and better, latex template [992](https://github.com/jupyter/nbconvert/pull/992)

Fixing Problems

- Refactored execute preprocessor to have a process_message function
[905](https://github.com/jupyter/nbconvert/pull/905):
- Fixed OOM kernel failures hanging [959](https://github.com/jupyter/nbconvert/pull/959) and [#971](https://github.com/jupyter/nbconvert/pull/971)
- Fixed latex export for svg data in python 3 [985](https://github.com/jupyter/nbconvert/pull/985)
- Enabled configuration to be shared to exporters from script exporter
[993](https://github.com/jupyter/nbconvert/pull/993)
- Make latex errors less verbose [988](https://github.com/jupyter/nbconvert/pull/988)
- Typo in template syntax [984](https://github.com/jupyter/nbconvert/pull/984)
- Improved attachments +fix supporting non-unique names
[980](https://github.com/jupyter/nbconvert/pull/980)
- PDFExporter "output_mimetype" traitlet is not longer
'text/latex' [972](https://github.com/jupyter/nbconvert/pull/972)
- FIX: respect wait for clear_output [969](https://github.com/jupyter/nbconvert/pull/969)
- address deprecation warning in cgi.escape [963](https://github.com/jupyter/nbconvert/pull/963)
- Correct inaccurate description of available LaTeX template
[958](https://github.com/jupyter/nbconvert/pull/958)
- Fixed kernel death detection for executions with timeouts
[998](https://github.com/jupyter/nbconvert/pull/998):
- Fixed export names for various templates [1000](https://github.com/jupyter/nbconvert/pull/1000), [#1001](https://github.com/jupyter/nbconvert/pull/1001), and
[1001](https://github.com/jupyter/nbconvert/pull/1001):

Deprecations

- Dropped support for python 3.4 [979](https://github.com/jupyter/nbconvert/pull/979)
- Removed deprecated `export_by_name` [945](https://github.com/jupyter/nbconvert/pull/945)

Testing, Docs, and Builds

- Added tests for each branch in execute's run_cell method
[982](https://github.com/jupyter/nbconvert/pull/982)
- Mention formats in --to options more clearly
[991](https://github.com/jupyter/nbconvert/pull/991)
- Adds ascii output type to command line docs page, mention image
folder output [956](https://github.com/jupyter/nbconvert/pull/956)
- Simplify setup.py [949](https://github.com/jupyter/nbconvert/pull/949)
- Use utf-8 encoding in execute_api example [921](https://github.com/jupyter/nbconvert/pull/921)
- Upgrade pytest on Travis [941](https://github.com/jupyter/nbconvert/pull/941)
- Fix LaTeX base template name in docs [940](https://github.com/jupyter/nbconvert/pull/940)
- Updated release instructions based on 5.4 release walk-through
[887](https://github.com/jupyter/nbconvert/pull/887)
- Fixed broken link to jinja docs [997](https://github.com/jupyter/nbconvert/pull/997)

Thanks to the following 11 authors who contributed 57 commits.

- Benjamin Ragan-Kelley
- Carol Willing
- Clayton A Davis
- Daniel Rodriguez
- M Pacer
- Matthew Seal
- Matthias Geier
- Matthieu Parizy
- Rüdiger Busche
- Thomas Kluyver
- Tyler Makaro

Comprehensive notes

New Features

- Expose pygments styles [889](https://github.com/jupyter/nbconvert/pull/889)
- Tornado 6.0 support -- Convert proxy handler from callback to
coroutine [937](https://github.com/jupyter/nbconvert/pull/937)
- Add option to overwrite the highlight_code filter
[877](https://github.com/jupyter/nbconvert/pull/877)

Fixing Problems

- Mathjax.tpl fix for rendering Latex in html [932](https://github.com/jupyter/nbconvert/pull/932)
- Backwards compatibility for empty kernel names
[927](https://github.com/jupyter/nbconvert/pull/927) [#924](https://github.com/jupyter/nbconvert/pull/924)

Testing, Docs, and Builds

- DOC: Add missing language specification to code-block
[882](https://github.com/jupyter/nbconvert/pull/882)

Significant Changes

Deprecations

Python 3.3 support was dropped. The version of python is no longer
common and new versions have many fixes and interface improvements that
warrant the change in support.

See [843](https://github.com/jupyter/nbconvert/pull/843) for implementation details.

Changes in how we handle metadata

There were a few new metadata fields which are now respected in
nbconvert.

`nb.metadata.authors` metadata attribute will be respected in latex
exports. Multiple authors will be added with `,` separation against
their names.

`nb.metadata.title` will be respected ahead of `nb.metadata.name` for
title assignment. This better matches with the notebook format.

`nb.metadata.filename` will override the default
`output_filename_template` when extracting notebook resources in the
`ExtractOutputPreprocessor`. The attribute is helpful for when you want
to consistently fix to a particular output filename, especially when you
need to set image filenames for your exports.

The `raises-exception` cell tag
(`nb.cells[].metadata.tags[raises-exception]`) allows for cell
exceptions to not halt execution. The tag is respected in the same way
by [nbval](https://github.com/computationalmodelling/nbval) and other
notebook interfaces. `nb.metadata.allow_errors` will apply this rule for
all cells. This feature is toggleable with the `force_raise_errors`
configuration option. Errors from executing the notebook can be allowed
with a `raises-exception` tag on a single cell, or the `allow_errors`
configurable option for all cells. An allowed error will be recorded in
notebook output, and execution will continue. If an error occurs when it
is not explicitly allowed, a 'CellExecutionError' will be raised. If
`force_raise_errors` is True, `CellExecutionError` will be raised for
any error that occurs while executing the notebook. This overrides both
the `allow_errors` option and the `raises-exception` cell tags.

See [867](https://github.com/jupyter/nbconvert/pull/867), [#703](https://github.com/jupyter/nbconvert/pull/703), [#685](https://github.com/jupyter/nbconvert/pull/685),
[672](https://github.com/jupyter/nbconvert/pull/672), and [#684](https://github.com/jupyter/nbconvert/pull/684) for implementation changes.

Configurable kernel managers when executing notebooks

The kernel manager can now be optionally passed into the
`ExecutePreprocessor.preprocess` and the `executenb` functions as the
keyword argument `km`. This means that the kernel can be configured as
desired before beginning preprocessing.

This is useful for executing in a context where the kernel has external
dependencies that need to be set to non-default values. An example of
this might be a Spark kernel where you wish to configure the Spark
cluster location ahead of time without building a new kernel.

Overall the ExecutePreprocessor has been reworked to make it easier to
use. Future releases will continue this trend to make this section of
the code more inheritable and reusable by others. We encourage you read
the source code for this version if you're interested in the detailed
improvements.

See [852](https://github.com/jupyter/nbconvert/pull/852) for implementation changes.

Surfacing exporters in front-ends

Exporters are now exposed for front-ends to consume, including classic
notebook. As an example, this means that latex exporter will be made
available for latex 'text/latex' media type from the Download As
interface.

See [759](https://github.com/jupyter/nbconvert/pull/759) and [#864](https://github.com/jupyter/nbconvert/pull/864) for implementation changes.

Raw Templates

Template exporters can now be assigned raw templates as string
attributes by setting the `raw_template` variable.

python
class AttrExporter(TemplateExporter):
If the class has a special template and you want it defined within the class
raw_template = """{%- extends 'rst.tpl' -%}
{%- block in_prompt -%}
raw template
{%- endblock in_prompt -%}
"""

exporter_attr = AttrExporter()
output_attr, _ = exporter_attr.from_notebook_node(nb)
assert "raw template" in output_attr


See [675](https://github.com/jupyter/nbconvert/pull/675) for implementation changes.

New command line flags

The `--no-input` will hide input cells on export. This is great for
notebooks which generate "reports" where you want the code that was
executed to not appear by default in the extracts.

An alias for `notebook` was added to exporter commands. Now `--to ipynb`
will behave as `--to notebook` does.

See [825](https://github.com/jupyter/nbconvert/pull/825) and [#873](https://github.com/jupyter/nbconvert/pull/873) for implementation changes.

Comprehensive notes

New Features

- No input flag (`--no-input`) [825](https://github.com/jupyter/nbconvert/pull/825)
- Add alias `--to ipynb` for notebook exporter [873](https://github.com/jupyter/nbconvert/pull/873)
- Add `export_from_notebook` [864](https://github.com/jupyter/nbconvert/pull/864)
- If set, use `nb.metadata.authors` for LaTeX author line
[867](https://github.com/jupyter/nbconvert/pull/867)
- Populate language_info metadata when executing
[860](https://github.com/jupyter/nbconvert/pull/860)
- Support for `\mathscr` [830](https://github.com/jupyter/nbconvert/pull/830)
- Allow the execute preprocessor to make use of an existing kernel
[852](https://github.com/jupyter/nbconvert/pull/852)
- Refactor ExecutePreprocessor [816](https://github.com/jupyter/nbconvert/pull/816)
- Update widgets CDN for ipywidgets 7 w/fallback
[792](https://github.com/jupyter/nbconvert/pull/792)
- Add support for adding custom exporters to the "Download as" menu.
[759](https://github.com/jupyter/nbconvert/pull/759)
- Enable ANSI underline and inverse [696](https://github.com/jupyter/nbconvert/pull/696)
- Update notebook css to 5.4.0 [748](https://github.com/jupyter/nbconvert/pull/748)
- Change default for slides to direct to the reveal cdn rather than
locally [732](https://github.com/jupyter/nbconvert/pull/732)
- Use "title" instead of "name" for metadata to match the notebook
format [703](https://github.com/jupyter/nbconvert/pull/703)
- Img filename metadata [685](https://github.com/jupyter/nbconvert/pull/685)
- Added MathJax compatibility definitions [687](https://github.com/jupyter/nbconvert/pull/687)
- Per cell exception [684](https://github.com/jupyter/nbconvert/pull/684)
- Simple API for in-memory templates [674](https://github.com/jupyter/nbconvert/pull/674) [#675](https://github.com/jupyter/nbconvert/pull/675)
- Set BIBINPUTS and BSTINPUTS environment variables when making PDF
[676](https://github.com/jupyter/nbconvert/pull/676)
- If `nb.metadata.title` is set, default to that for notebook
[672](https://github.com/jupyter/nbconvert/pull/672)

Deprecations

- Drop support for python 3.3 [843](https://github.com/jupyter/nbconvert/pull/843)
- Default conversion method on the CLI was removed (`--to html` now
required)

Fixing Problems

- Fix api break [872](https://github.com/jupyter/nbconvert/pull/872)
- Don't remove empty cells by default [784](https://github.com/jupyter/nbconvert/pull/784)
- Handle attached images in html converter [780](https://github.com/jupyter/nbconvert/pull/780)
- No need to check for the channels already running
[862](https://github.com/jupyter/nbconvert/pull/862)
- Update `font-awesome` version for slides [793](https://github.com/jupyter/nbconvert/pull/793)
- Properly treat JSON data [847](https://github.com/jupyter/nbconvert/pull/847)
- Skip executing empty code cells [739](https://github.com/jupyter/nbconvert/pull/739)
- Ppdate log.warn (deprecated) to log.warning [804](https://github.com/jupyter/nbconvert/pull/804)
- Cleanup notebook.tex during PDF generation [768](https://github.com/jupyter/nbconvert/pull/768)
- Windows unicode error fixed, nosetest added to setup.py
[757](https://github.com/jupyter/nbconvert/pull/757)
- Better content hiding; template & testing improvements
[734](https://github.com/jupyter/nbconvert/pull/734)
- Fix Jinja syntax in custom template example. [738](https://github.com/jupyter/nbconvert/pull/738)
- Fix for an issue with empty math block [729](https://github.com/jupyter/nbconvert/pull/729)
- Add parser for Multiline math for LaTeX blocks
[716](https://github.com/jupyter/nbconvert/pull/716) [#717](https://github.com/jupyter/nbconvert/pull/717)
- Use defusedxml to parse potentially untrusted XML
[708](https://github.com/jupyter/nbconvert/pull/708)
- Fixes for traitlets 4.1 deprecation warnings [695](https://github.com/jupyter/nbconvert/pull/695)

Testing, Docs, and Builds

- A couple of typos [870](https://github.com/jupyter/nbconvert/pull/870)
- Add python_requires metadata. [871](https://github.com/jupyter/nbconvert/pull/871)
- Document `--inplace` command line flag. [839](https://github.com/jupyter/nbconvert/pull/839)
- Fix minor typo in `usage.rst` [863](https://github.com/jupyter/nbconvert/pull/863)
- Add note about local `reveal_url_prefix` [844](https://github.com/jupyter/nbconvert/pull/844)
- Move `onlyif_cmds_exist` decorator to test-specific utils
[854](https://github.com/jupyter/nbconvert/pull/854)
- Include LICENSE file in wheels [827](https://github.com/jupyter/nbconvert/pull/827)
- Added Ubuntu Linux Instructions [724](https://github.com/jupyter/nbconvert/pull/724)
- Check for too recent of pandoc version [814](https://github.com/jupyter/nbconvert/pull/814) [#872](https://github.com/jupyter/nbconvert/pull/872)
- Removing more nose remnants via dependencies.
[758](https://github.com/jupyter/nbconvert/pull/758)
- Remove offline statement and add some clarifications in slides docs
[743](https://github.com/jupyter/nbconvert/pull/743)
- Linkify PR number [710](https://github.com/jupyter/nbconvert/pull/710)
- Added shebang for python [694](https://github.com/jupyter/nbconvert/pull/694)
- Upgrade mistune dependency [705](https://github.com/jupyter/nbconvert/pull/705)
- add feature to improve docs by having links to prs
[662](https://github.com/jupyter/nbconvert/pull/662)
- Update notebook CSS from version 4.3.0 to 5.1.0
[682](https://github.com/jupyter/nbconvert/pull/682)
- Explicitly exclude or include all files in Manifest.
[670](https://github.com/jupyter/nbconvert/pull/670)

- MANIFEST.in updated to include `LICENSE` and `scripts/` when
creating sdist. [666](https://github.com/jupyter/nbconvert/pull/666)

Major features

Tag Based Element Filtering

For removing individual elements from notebooks, we need a way to signal
to nbconvert that the elements should be removed. With this release, we
introduce the use of tags for that purpose.

Tags are user-defined strings attached to cells or outputs. They are
stored in cell or output metadata. For more on tags see the [nbformat
docs on cell
metadata](https://nbformat.readthedocs.io/en/latest/format_description.html#cell-metadata).

**Usage**:

1. Apply tags to the elements that you want to remove.

For removing an entire cell, the cell input, or all cell outputs apply
the tag to the cell.

For removing individual outputs, put the tag in the output metadata
using a call like
`display(your_output_element, metadata={tags=[<your_tags_here>]})`.

_NB_: Use different tags depending on whether you want to remove the
entire cell, the input, all outputs, or individual outputs.

2. Add the tags for removing the different kinds of elements to the
following traitlets. Which kind of element you want to remove
determines which traitlet you add the tags to.

The following traitlets remove elements of different kinds:

- `remove_cell_tags`: removes cells
- `remove_input_tags`: removes inputs
- `remove_all_outputs_tag`: removes all outputs
- `remove_single_output_tag`: removes individual outputs

Comprehensive notes

- new: configurable `browser` in ServePostProcessor
[618](https://github.com/jupyter/nbconvert/pull/618)
- new: `--clear-output` command line flag to clear output in-place
[619](https://github.com/jupyter/nbconvert/pull/619)
- new: remove elements based on tags with `TagRemovePreprocessor`.
[640](https://github.com/jupyter/nbconvert/pull/640), [#643](https://github.com/jupyter/nbconvert/pull/643)
- new: CellExecutionError can now be imported from
`nbconvert.preprocessors` [656](https://github.com/jupyter/nbconvert/pull/656)
- new: slides now can enable scrolling and custom transitions
[600](https://github.com/jupyter/nbconvert/pull/600)
- docs: Release instructions for nbviewer-deploy
- docs: improved instructions for handling errors using the
`ExecutePreprocessor` [656](https://github.com/jupyter/nbconvert/pull/656)
- tests: better height/width metadata testing for images in rst & html
[601](https://github.com/jupyter/nbconvert/pull/601) [#602](https://github.com/jupyter/nbconvert/pull/602)
- tests: normalise base64 output data to avoid false positives
[650](https://github.com/jupyter/nbconvert/pull/650)
- tests: normalise ipython traceback messages to handle old and new
style [631](https://github.com/jupyter/nbconvert/pull/631)
- bug: mathjax obeys `\\(\\)` & `\\[\\]` (both nbconvert & pandoc)
[609](https://github.com/jupyter/nbconvert/pull/609) [#617](https://github.com/jupyter/nbconvert/pull/617)
- bug: specify default templates using extensions
[639](https://github.com/jupyter/nbconvert/pull/639)
- bug: fix pandoc version number [638](https://github.com/jupyter/nbconvert/pull/638)
- bug: require recent mistune version [630](https://github.com/jupyter/nbconvert/pull/630)
- bug: catch errors from IPython `execute_reply` and `error` messages
[642](https://github.com/jupyter/nbconvert/pull/642)
- nose completely removed & dependency dropped [595](https://github.com/jupyter/nbconvert/pull/595) [#660](https://github.com/jupyter/nbconvert/pull/660)
- mathjax processing in mistune now only uses inline grammar
[611](https://github.com/jupyter/nbconvert/pull/611)
- removeRegex now enabled by default on all TemplateExporters, does
not remove cells with outputs [616](https://github.com/jupyter/nbconvert/pull/616)
- validate notebook after applying each preprocessor (allowing
additional attributes) [645](https://github.com/jupyter/nbconvert/pull/645)
- changed COPYING.md to LICENSE for more standard licensing that
GitHub knows how to read [654](https://github.com/jupyter/nbconvert/pull/654)