Civis

Added
- Added a script for integration tests (smoke tests).

Fixed
- Added missing string formatting to a log emit in file multipart upload and
correct ordering of parameters in another log emit (217)

Changed
- Updated documentation with new information about predefined stacking
estimators (221)
- Updated CivisML 2.0 notebook (214)
- Reworded output of `civis notebooks new` CLI command (215)

Added
- Documentation updated to reflect CivisML 2.1 features (209)
- ``civis.io.dataframe_to_civis``, ``civis.io.csv_to_civis``, and ``civis.io.civis_file_to_table`` functions now support the `diststyle` parameter.
- New notebook-related CLI commands: "new", "up", "down", and "open".
- Additional documentation for using the Civis joblib backend (199)
- Documented additional soft dependencies for CivisML (203)

Changed
- Changed `ModelPipeline.train` default for `n_jobs` from 4 to `None`,
so that `n_jobs` will be dynamically calculated by default (203)
- Use "feather"-formatted files to send data from users to CivisML, if possible.
Require this when using ``pd.Categorical`` types, since CSVs require us to
re-infer column types, and this can fail. Using feather should also give a
speed improvement; it reads and writes faster than CSVs and produces smaller files (200).
- ``ModelFuture`` objects will emit any warnings which occurred during their
corresponding CivisML job (204)
- Removed line setting "n_jobs" from an example of CivisML prediction.
Recommended use is to let CivisML determine the number of jobs itself (211).
- Update maximum CivisML version to v2.1; adjust fallback logic such that users get
the most recent available release (212).

Fixed
- Restored the pre-v1.7.0 default behavior of the ``joblib`` backend by setting the ``remote_backend``
parameter default to 'sequential' as opposed to 'civis'. The default of 'civis' would launch additional
containers in nested calls to ``joblib.Parallel``. (205)
- If validation metadata are missing, ``ModelFuture`` objects will return ``None``
for metrics or validation metadata, rather than issuing an exception (208)
- Allowed callers to pass `index` and `encoding` arguments to the `to_csv` method through `dataframe_to_civis`.

Performance Enhancements
- ``civis.io.file_to_civis`` now uses additional file handles for multipart upload instead of writing to disk to reduce disk usage
- ``civis.io.dataframe_to_civis`` writes dataframes to disk instead of using an in memory buffer

Fixed
- Relaxed requirement on ``cloudpickle`` version number (187)
- Restore previous behavior of ``civis.io.civis_to_csv`` when using "compression='gzip'" (195)

Fixed
- Specify escape character in ``civis.io.read_civis_sql`` when performing parallel unload
- Issue uploading files in ``civis.io.file_to_civis``
- Revert performance enhancement that will change format of file produced by ``civis.io.civis_to_csv``

Changed
- Updated CivisML template ids to v2.0 (139)
- Optional arguments to API endpoints now display in function signatures.
Function signatures show a default value of "DEFAULT"; arguments will still
only be transmitted to the Civis Platform API when explicitly provided. (140)
- ``APIClient.feature_flags`` has been deprecated to avoid a name collision
with the feature_flags endpoint. In v2.0.0, ``APIClient.featureflags``
will be renamed to ``APIClient.feature_flags``.
- The following APIClient attributes have been deprecated in favor of the
attribute that includes underscores:
``APIClient.bocceclusters`` -> ``APIClient.bocce_clusters``
``APIClient.matchtargets`` -> ``APIClient.match_targets``
``APIClient.remotehosts`` -> ``APIClient.remote_hosts``
- ``civis.io.csv_to_civis`` and ``civis.io.dataframe_to_civis`` functions now use
``civis.io.file_to_civis`` and ``civis.io.civis_file_to_table`` functions instead
of separate logic
- ``civis.io.file_to_civis``, ``civis.io.csv_to_civis`` and ``civis.io.dataframe_to_civis``
now support files over 5GB
- Refactor internals of ``CivisFuture`` and ``PollableResult`` to centralize handling
of threads and ``pubnub`` subscription.
- Updated API specification and base resources to include all general
availability endpoints.
- Changed ``civis.io.file_to_civis`` and ``civis.io.civis_to_file`` to allow
strings for paths to local files in addition to just file/buffer objects.

Fixed
- Fixed parsing of multiword endpoints. Parsing no longer removes underscores
in endpoint names.
- In ``civis.futures.ContainerFuture``, return ``False`` when users attempt to cancel
an already-completed job. Previously, the object would sometimes give a ``CivisAPIError``
with a 404 status code. This fix affects the executors and joblib backend, which
use the ``ContainerFuture``.
- Tell ``flake8`` to ignore a broad except in a ``CivisFuture`` callback.
- Close open sockets (in both the ``APIClient`` and ``CivisFuture``) when they're no
longer needed, so as to not use more system file handles than necessary (173).
- Correct treatment of ``FileNotFoundError`` in Python 2 (176).
- Fixed parsing of endpoints containing hyphens. Hyphens are replaced with
underscores.
- Use ``civis.compat.TemporaryDirectory`` in ``civis.io.file_to_civis`` to be
compatible with Python 2.7
- Catch notifications sent up to 30 seconds before the ``CivisFuture`` connects.
Fixes a bug where we would sometimes miss an immediate error on SQL scripts (174).

Added
- Documentation updated to include new CivisML features (137).
- ``civis.resources.cache_api_spec`` function to make it easier to record the
current API spec locally (141).
- Autospecced mock of the ``APIClient`` for use in testing third-party code which
uses this library (141).
- Added `etl`, `n_jobs`, and `validation_data` arguments to
ModelPipeline.train (139).
- Added `cpu`, `memory`, and `disk` arguments to ModelPipeline.predict
(139).
- Added ``remote_backend`` keyword to the ``civis.parallel.make_backend_factory``
and ``civis.parallel.infer_backend_factory`` in order to set the joblib
backend in the container for nested calls to ``joblib.Parallel``.
- Added the PyPI trove classifiers for Python 3.4 and 3.6 (152).
- ``civis.io.civis_file_to_table`` function to import an existing Civis file
to a table
- ``civis.io.file_to_civis`` function will now automatically retry uploads to
the Civis Platform up to 5 times if is there is an HTTPError, ConnectionError
or ConnectionTimeout
- Additional documentation about the use case for the Civis joblib backend.
- Added a note about serializing ``ModelPipeline`` ``APIClient`` objects to the docstring.
- Added `civis notebooks download` command-line interface command to facilitate
downloading notebooks.

Performance Enhancements
- ``civis.io.file_to_civis`` now takes advantage of multipart uploads to chunk
files and perform I/O in parallel
- ``civis.io.civis_to_csv`` and ``civis.io.read_civis_sql`` will always request
data with gzip compression to reduce I/O. Also, they will attempt to fetch
headers in a separate query so that data can be unloaded in parallel
- ``civis.io.civis_to_csv`` with ``compression='gzip'`` currently returns a file
with no compression. In a future release, ``compression='gzip'`` will return a
gzip compressed file.

Changed
- Added explanatory text to CivisML_parallel_training.ipynb (126).

Fixed
- Added `ResourceWarning` for Python 2.7 (128).
- Added `TypeError` for multi-indexed dataframes when used as input to
CivisML (131).
- ``ModelPipeline.from_existing`` will warn if users attempt to recreate
a model trained with a newer version of CivisML, and fall back on the
most recent prediction template it knows of (134).
- Make the `PaginatedResponse` returned by LIST endpoints a full iterator.
This also makes the `iterator=True` parameter work in Python 2.
- When using ``civis.io.civis_to_csv``, emit a warning on SQL queries which
return no results instead of allowing a cryptic ``IndexError`` to surface (135).
- Fixed the example code snippet for ``civis.io.civis_to_multifile_csv``.
Also provided more details on its return dict in the docstring.
- Pinned down `sphinx_rtd_theme` and `numpydoc` in `dev-requirements.txt`
for building the documentation.

Added
- Jupyter notebook with demonstrations of use patterns and abstractions in the Python API client (127).