Prompt-tool-kit

-----------------

Fixes:
- Bugfix in GrammarValidator and SentenceValidator.
- Don't leave the alternate screen on resize events.
- Use errors=surrogateescape, in order to handle mouse events in some
terminals.
- Ignore key presses in _InterfaceEventLoopCallbacks.feed_key when the CLI is in the done state.
- Bugfix in get_common_complete_suffix. Don't return any suffix when there are
completions that change whatever is before the cursor.
- Bugfix for Win32/Python2: use unicode literals: This crashed arrow navigation
on Windows.
- Bugfix in InputProcessor: handling of more complex key bindings.
- Fix: don't apply completions, if there is only one completion which doesn't
have any effect.
- Fix: correctly handle prompts starting with a newline in
prompt_toolkit.shortcuts.
- Fix: thread safety in autocomplete code.
- Improve styling for matching brackets. (Allow individual styling for the
bracket under the cursor and the other.)
- Fix in ShowLeadingWhiteSpaceProcessor/ShowTrailingWhiteSpaceProcessor: take
output encoding into account. (The signature had to change a little for
this.)
- Bug fix in key bindings: only activate Emacs system/open-in-editor bindings
if editing_mode is emacs.
- Added write_binary parameter to Vt100_Output. This fixes a bug in some cases
where we expect it to write non-encoded strings.
- Fix key bindings for Vi mode registers.

New features (**):
- Added shortcuts.confirm/create_confirm_application function.
- Emulate bracketed paste on Windows. (When the input stream contains multiple
key presses among which a newline and at least one other character, consider
this a paste event, and handle as bracketed paste on Unix.
- Added key handler for displaying completions, just like readline does.
- Implemented Vi guu,gUU,g~~ key bindings.
- Implemented Vi 'gJ' key binding.
- Implemented Vi ab,ib,aB,iB text objects.
- Support for ZeroWidthEscape tokens in prompt and token lists. Used to support
final shell integration.
- Fix: Make document.text/cursor_position/selection read-only. (Changing these
would break the caching causing bigger issues.)
- Using pytest for unit tests.
- Allow key bindings to have Keys.Any at any possible position. (Not just the
end.) This made it significantly easier to write the named register Vi
bindings, resulting in an approved start-up time.)
- Better feedback when entering multi-key key bindings in insert mode. (E.g.
when 'jj' would be mapped to escape.)
- Small improvement in key processor: allow key bindings to generate new key
presses.
- Handle ControlUp and ControlDown by default: move to the previous/next record
in the history.
- Accept 'char'/'get_char' parameters in FillControl.
- Added refresh_interval method to prompt() function.

Performance improvements:
- Improve the performance of test_callable_args: this should significantly
increase the start-up time.
- Start-up time for creating the Vi bindings has been improved significantly.

(**) Some small backwards-compatible features were allowed for this minor
release. After evaluating the impact/risk/work involved we concluded that
we could ship these in a minor release.

-----------------

Fixes:
- Adjust minimum completion menu width to match UIControl and Window class.
- Bugfix regarding weakref in InputProcessor.
- Fix for pypy3: bug in WeakValueDictionary.
- Correctly handle '0' key binding in Vi mode.
- Also load Vi bindings by default in Application if no registry has been given.
- Only go into selection mode if the current buffer is not empty.
- Close PipeInput after usage.
- Only use 16 colors in (Emacs) eterm-color.
- Bugfix in "xP Vi key binding.
- Bugfix in Vi { and } key binding.
- Fix: use correct token for Scrollbar in MultiColumnCompletionMenuControl.
- Handle negative values in translate_row_col_to_index.
- Handle decomposed unicode characters.
- Fixed Window.always_hide_cursor. (Parameter was ignored.)
- Fix in zz Vi key binding. (When render info is not available.)
- Fix in Document.get_cursor_up_position. (When an argument is given.)

New features:
- Separated `load_mouse_bindings`.
- Refactoring/simplification of the key bindings: better use of filters and
CLI.editing_mode.
- Added DummyOutput class and a few unit tests that test the whole CLI.
- Use the bisect module in Document._line_start_indexes instead of a custom
binary search. This should improve the performance.
- Stay in the same column when doing multiple up/down movements.
- Visual improvements:
* Implemented cursorcolumn, cursorline and colorcolumn.
* Only reserve menu space when `complete_while_typing=True` or when there are
completions to be displayed.
* Support for chaining tokens for combined styles. SelectedText will now
reverse the colors from the highlighting by default. Style
`Token.SelectedText` to set a fixed foreground/background.
Also for SearchMatch, we now use combined tokens.
* Support for dark gray on Windows.
* Default token for SystemToolbar and SearchToolbar.
* Display selection also on empty lines.
- Emacs key bindings improved:
* Recognize + handle ControlDelete key.
* Implemented meta-* and control-backslash key bindings.
- Vi key bindings improved:
* Handle inclusive and linewise motions properly.
* Fix g_ motion off by one character, and don't work when cursor is in
the trailing whitespace part of line.
* Make a(/a)/i(/i)/... motions. Find enclosing brackets instead of the next
bracket.
* Update N% motion according to vim behaviors.
* Fix | motion off by one character.
* ge/gE motions go to end of previous word, not start.
* Added Vi 'gm' key binding.
* Implemented 'gq' key binding in Vi mode. (Reshape text.)
* Vi operator/text object separation for key bindings.
* Added 'ap' (auto-paragraph) text object.
* Implemented Vi digraphs. ControlK will now insert a digraph.
* Implemented vi tilde_operator.
* Support named registers.
* Vi < and > key bindings became operators.
* Text objects and motions are now separate bindings.
* Improved copy/paste in Vi mode.

Backwards-incompatible changes:
- Don't reset the current buffer anymore by default in
CommandLineInterface.run(). Passing `reset_current_buffer=True` is now
required.
- Renamed MouseEventTypes to MouseEventType for consistency. The old name is
still valid, but deprecated.
- Refactoring of Callbacks. All events should now receive one argument, which
is the sender. (Further, Callback was renamed to Event.) This is mostly used
internally.
- Moved on_invalidate callback from CommandLineInterface to Application
- Renamed `PipeInput.send` to `PipeInput.send_text`. (Old deprecated name is
still kept as a valid alias.)
- Renamed SimpleLexer.default_token to SimpleLexer.token. (+
backwards-compatibility.)
- Refactoring of the filters: `ViStateFilter` has been deprecated. (Should not
be used anymore.) Use the filters, as defined in prompt_toolkit.filters.
- `editing_mode` is now a property of `CommandLineInterface`. This is replacing
the `vi_mode` parameter in `KeyBindingManager`.
- The default accept_action for the default Buffer in Application now becomes
IGNORE. This is a much more sensible default. Pass RETURN_DOCUMENT to get
the previous behaviour,
- Always expect an EventLoop instance in CommandLineInterface. Creating it in
__init__ caused a memory leak.

----------------

Fixes:
- Fix in Document.paste. (The screen was not updated after an undo of a paste.)
- Don't use deprecated inspect.getargspec on Python 3.
- Fixed reading input on Windows when input was piped in stdin.
- Use correct file descriptors for input/output in run_system_command.
- Always correctly split prompt in shortcuts.prompt. (Even when multiline=False)
- Correctly align right prompt to the top when the left prompt consists of
multiple lines.
- Correctly use Token.Transparent as default token for a TokenListControl.
- Fix in syntax synchronisation. (Better handle the case when no
synchronisation point was found.)
- Send SIGTSTP to the whole process group.
- Correctly raise on_buffer_changed on all text changes.
- Fix in regular_languages.GrammarLexer. (Fixes bug in ptipython syntax
highlighting.)

New features:
- Add support for additional readers to the Win32 event loop.
- Added on_render event.
- Carry the weight in layout dimensions to allow stretching.

----------------

Fixes:
- Set correct default color on Windows. (Gray instead of high intensity gray.)
- Reverse colors on Windows when foreground/background color have not been
specified.
- Correct handling of mouse events for FillControl.
- Take margin into account when calculating Window height. (Fixes bug in
multiline prompt.)
- Handle division by zero in UIContent.get_height_for_text.

----------------

Fixes:
- Correctly return result for mouse handler in TokenListControl.
- Bugfix in meta-backspace key binding. (Delete all whitespace before the
cursor, when there is only whitespace.)
- Bugfix in Vi gu, gU, g? and g~ key bindings (in selection mode).
- Correctly restore default console attributes on Windows.
- Disable bracketed paste support in ConEmu. (This was broken.)
- When an unknown exception is raised in `CommandLineInterface.run()`, don't
forget to redraw the CLI.

New features:
- Many performance improvements and better caching. (Especially in the
`Document` class.)
- Support for continuation tokens in `shortcuts.prompt` and
`shortcuts.create_prompt_layout`.
- Added `shortcuts.print_tokens` function for printing colored output.
- Sound bell when nothing was deleted.
- Added escape sequences for F1-F5 keys on the Linux console.
- Improved support for the Linux console. (Switch back to 16 colors.)
- Added F13-F24 input codes for xterm.
- Created prompt_toolkit.token. A custom Token implementation, that is
compatible with Pygments.token. (This way, Pygments becomes an optional
dependency. For many use cases, nothing except the Token class from Pygments
was used, so it was a bit overkill to install Pygments for only that.)
- Refactoring of prompt_toolkit.styles.
- `Float` objects got a `hide_when_covering_content` option.
- Implementation of RPROMPT, like ZSH: Added `get_rprompt_tokens` to
`create_prompt_layout`.
- Some improvements to the default style.
- Also handle Ctrl-R and Ctrl-S in Vi mode when searching.
- Added TabsProcessor: a tool to visualise tabs instead of displaying ^I.
- Give a better error message when trying to run in git-bash.
- Support for ANSI color names in style dictionaries.

- Big refactoring of the `Window` and `UIControl` classes. This should result
in huge performance improvements on big inputs. (While first, a document
could have 1,000 lines; now it can have about 100,000 lines on the same system.)

The Window and UIControl have been rewritten very much. Rather than each time
rendering the whole user control, we now only have to render the visible part.

Because of this, many pieces had to be rewritten:
- UIControls work differently. They return a `UIContent` instance that
consist of a collection of lines.
- All processors have been rewritten. (Their API changed as well, because
they process one line at a time.)
- Lexers work differently. `Lexer.lex_document` should now return a function
that returns the tokens for one line. PygmentsLexer has been optimized that
it becomes 'lazy', and it has optional syntax synchronisation. That means,
that the lexer doesn't have to start the lexing at the beginning of the
document. (Which would be slow for big documents.)

Backwards-incompatible changes:
- As mentioned above, the refactoring of `Window` and `UIControl` caused many
"internal" APIs to change. All custom `UIControl`, `Processor` and `Lexer`
classes have to be rewritten. However, for most applications this should not
be an issue. Especially, the `shortcuts.prompt` function is
backwards-compatible.
- `wrap_lines` became a property of `Window` instead of `BufferControl`.

----------------

Fixes:
- Made `max_render_postpone_time` configurable. The current default was bad.
(We should probably always draw the UI once every cycle of the event loop.)