Changelogs » Civis

PyUp Safety actively tracks 266,461 Python packages for vulnerabilities and notifies you when to upgrade.



  - Bump minimum pubnub version to `4.1.12` (397)
  - In ``, ensure that data types are detected when table_columns are provided with no sql_types. Additionally, throw an error if some sql_types are provided and not others. (400)
  - Retain specific sql types when there are multiple input files and `table_columns` specified in `` (402)
  - Removed Python 3.5 support (404)
  - Updated list of base API resources to include `aliases`, `git_repos`, `json_values`, `services`, and `storage_hosts` so that they show up in the sphinx docs (406)
  - Update the API spec at `civis/tests/civis_api_spec.json` so that new endpoints are included (e.g., `/exports/files/csv`) (407)
  - Refactor file cleaning logic for `` (405)
  - Refactored retry logic to use tenacity package, added random jitter on retries, and retry on POST 429 and 503s. (401)
  - Fixed a workflows usage example in `docs/source/client.rst` that had an incorrect endpoint. (409)
  - Fixed a bug in parsing responses that included "update" as a key (e.g., in column information from `client.tables.get(...)`). (410)


  - Added support for Python 3.8 (391)
  - Fixed a bug in the CLI tool which caused failed commands to exit with a 0 exit status. (389)
  - Fixed some issues that the newly-released flake8 3.8 complained about, including a buggy print statement for logging in (394)
  - Fixed a bug when cancelling jobs while using the Civis joblib backend. (395)
  - Added additional detail to ``, ``, and ``'s docstrings on the primary key parameter. (388)
  - Made polling threads for Civis futures be daemon threads so that Python processes will shut down properly in Python 3.8 (391)
  - Removed deprecation warning on the `file_id` parameter of ``. The parameter name will be kept in v2. (360, 393)
  - Show tables of methods for each set of endpoints in the API Resources pages. (396)


  - Fixed a bug in the `ServiceClient` where the API root path was not passed when generating classes. (384)


  - Added `.outputs` method to retrieve outputs from `CivisFuture`
  objects. (381)
  - Added `table_columns` parameter to ``, ``, and `` (379)
  - Fixed/relaxed version specifications for click, jsonref, and jsonschema. (377)
  - Removed support for Python 2.7 and 3.4. (378)
  - No longer require ServiceClient to be instantiated to parse a
  service api spec. (382)


  - Suppressed FutureWarning from sklearn.externals.joblib. (375)


  - Add `civis jobs follow-log` and `civis jobs follow-run-log` CLI commands. (359)
  - Add documentation for follow-log CLI Commands to main docs. (367)
  - Fixed a bug related to duplicating parent job parameters when using `civis.parallel.infer_backend_factory`. (363)
  - Fixed crashing on NULL fields in `civis sql` CLI command. (366)
  - Fixed `hidden` parameter not getting used in ``. (364)
  - Fixed `effective_n_jobs` to account for `n_jobs=None`, which is a default for the LogisticsRegression in `sklearn=0.22.x`. (365)
  - Fixed crashing on NULL fields in `civis sql` CLI command (366)
  - Fixed a bug related to creating a ModelPipeline from a registered model. (369)
  - Fixed readme and to appease twine. (373)
  - Made repeated invocations of `civis.tests.create_client_mock` faster by caching the real APIClient that the mock spec is based on (371)


  - Fixed issue where client did not generate functions for deprecated API endpoints. (353)
  - Changed `ServiceClient` to raise `CivisAPIError`. (355)
  - Updated documentation language for CivisML version. (358)


  - Added method `get_storage_host_id` to the APIClient. (328)
  - Added debug logging to some `` functions. (325)
  - Added `ServiceClient` and `ServiceEndpoint` class. (343)
  - Added new arguments to `` to expose max_file_size parameter. (342)
  - Removed incorrect "optional" marker for the `sql` argument in I/O
  functions. (338)
  - Raise a more informative exception when calling `file_to_dataframe`
  on an expired file. (337)
  - `ModelPipeline.register_pretrained_model` should persist the user-supplied
  estimator object indefinitely. (331)
  - Fixed requirements.txt listing for `cloudpickle` -- `>=0.2`, not `<=0.2`. (323)
  - Fixed issue in `` when returning data that contains
  double quotes. (328)
  - Fixed issue with pyyaml version for Python 3.4 by requiring pyyaml version <=5.2
  - Updated cloudpickle and joblib dependencies. (349)
  - CivisML uses platform aliases instead of hard-coded template IDs. (341, 347)
  - CivisML versions and pre-installed packages are documented on Zendesk instead. (341)
  - Issue a `FutureWarning` on import for Python 2 and 3.4 users. (333,
  - Pass `headers` and `delimiter` to Civis API endpoint for cleaning files in ``. (334)
  - Issue a `FutureWarning` on import for Python 2 users. (333)
  - Update the Civis logo in the Sphinx documentation. (330)
  - Allow the `name` arg to be optional in ``. (324)
  - Refactor `` to use a new set of Civis API endpoints for cleaning and importing CSV files. (328)
  - Added new arguments to `` to expose additional functionality from new Civis API endpoints. (328)
  - Added new arguments from `` to `dataframe_to_civis` and `csv_to_civis` methods. (328)


  - Add CLI command "sql" for command line SQL query execution. (319)
  - Add helper function (run_template) to run a template given its id and return
  either the JSON output or the associated file ids. (318)
  - Add helper function to list CivisML models. (314)
  - Added helper functions to share CivisML models with users or groups,
  patterned after the existing API sharing endpoints. (315)
  - Allow the base URL of the CLI to be configured through the
  `CIVIS_API_ENDPOINT` environment variable, like the civis Python module. (312)
  - Allow the CLI log level to be configured with the `CIVIS_LOG_LEVEL`
  environment variable with the standard `logging` module levels.
  For example: `CIVIS_LOG_LEVEL=DEBUG civis users list-me` (312)
  - Allow users to access `civis.utils.run_job` after an `import civis`. (305)
  - `` and `` convenience functions.
  (262, 304)
  - Add the user's Python version to the User-Agent string. (255, 301)
  - Added a `last_response` parameter to the `APIClient` object. (153, 302)
  - The deprecate_param decorator can take multiple parameter names, to allow
  Python 2.7 compatibility for multiple deprecations. (311)
  - Added missing docs for `json_to_file` and `dataframe_to_file` (320).
  - Fix unintentional dependency on scikit-learn for `parallel` module tests. (245, 303)
  - Deprecate the `headers` parameter of `dataframe_to_civis` and always tell Civis
  whether the import has headers or not, rather than autodetecting. (263, 307)
  - Set `cloudpickle` requirements to <1.2 on Python v3.4. (309)
  - Fixed an issue in the CLI which prevented users from accessing GET /aliases/{id}
  and simultaneously generated a warning message. (298, 316)
  - Loosened version requirements of `pyyaml` to include `pyyaml<=5.99`. (293)
  - Loosened version requirement of `jsonref` to include `0.2` to fix a
  DeprecationWarning under Python 3.7. (295)
  - Changed pubnub version requirement in requirements.txt to match
  - Loosened version requirements of `click` to include v7 and `jsonschema`
  to include v3. (286, 300)
  - Surfaced `` in the Sphinx docs. (294)
  - Loosen `joblib` version requirement to include v0.13 and add code to
  the Civis joblib backend which newer versions of `joblib` can take
  advantage of. Also loosened version requirement on `cloudpickle` to
  include v1. (296, 299)
  - Run all tests in Ubuntu Xenial. (310)


  - `CivisFuture` has the `job_id` and `run_id` property attributes. (290)
  - Polling will treat `None` responses generated by spotty internet connections
  like responses with a non-DONE state. (289)


  - `get_table_id` will correctly handle quoted schema.tablename. (285)
  - Fix parsing of empty responses from run cancellation endpoints. (287)


  - Correct prediction template id for CivisML 1.0 (278)
  - `` checks for primary key before reading in
  data. (276)
  - Test for Python 3.7 compatibility (277)
  - Updated mock API specs (281)


  - `` will now retry on S3 connection errors (273)
  - Buffers will be reset appropriately on connection failures during
  `` (273)


  - `_stash_dataframe_as_csv` in `civis/ml/` now uses a `StringIO`
  object which has the `getvalue` method (required by `pandas` v0.23.1
  if a file-like object is passed into `df.to_csv`). (259)
  - `civis_to_multifile_csv` fully respects the `client` keyword argument
  - Added instructions in the README for adding an API key to a Windows 10
  - Configured Windows CI using AppVeyor. (258)
  - Coalesced `README.rst` and `index.rst`. (254)
  - joblib documentation has moved to readthedocs. (267)


  - Added more robust parsing for tablename parsing in io.  You may now
  pass in tables like schema."tablename.with.periods".
  - Adding in missing documentation for civis_file_to_table
  - Include JSON files with pip distributions (244)
  - Added flush to `civis_to_file` when passed a user-created buffer,
  ensuring the buffer contains the entire file when subsequently read.
  - Fix several tests in the `test_io` module (248)
  - Travis tests for Python 3.4 are now restricted to pandas<=0.20, the
  last version which supported Python 3.4 (249)
  - Added a utility function which can robustly split a Redshift schema name
  and table name which are presented as a single string joined by a "." (225)
  - Added docstrings for `civis.find` and `civis.find_one`. (224)
  - Executors in ``futures`` (and the joblib backend, which uses them) will now
  add "CIVIS_PARENT_JOB_ID" and "CIVIS_PARENT_RUN_ID" environment variables
  to the child jobs they create (236)
  - Update default CivisML version to v2.2. This includes a new function
  ``ModelPipeline.register_pretrained_model`` which allows users to train
  a model outside of Civis Platform and use CivisML to score it at scale (242, 247).
  - Added a new parameter ``dvs_to_predict`` to ````.
  This allows users to select a subset of a model's outputs for scoring (241).
  - Added `` to store results of a SQL query
  to a Civis file
  - Surfaced `civis.find` and `civis.find_one` in the Sphinx docs. (250)
  - Moved "Optional Dependencies" doc section to top of ML docs, and
  added clarifications for pre-defined models with non-sklearn
  estimators (238)
  - Switched to pip install-ing dependencies for building the documentation (230)
  - Added a merge rule for the changelog to .gitattributes (229)
  - Default to "all" API resources rather than "base".
  - Updated documentation on algorithm hyperparameters to reflect changes with
  CivisML v2.2 release (240)


  - Added a script for integration tests (smoke tests).
  - Added missing string formatting to a log emit in file multipart upload and
  correct ordering of parameters in another log emit (217)
  - Updated documentation with new information about predefined stacking
  estimators (221)
  - Updated CivisML 2.0 notebook (214)
  - Reworded output of `civis notebooks new` CLI command (215)


  - Documentation updated to reflect CivisML 2.1 features (209)
  - ````, ````, and ```` functions now support the `diststyle` parameter.
  - New notebook-related CLI commands: "new", "up", "down", and "open".
  - Additional documentation for using the Civis joblib backend (199)
  - Documented additional soft dependencies for CivisML (203)
  - Changed `ModelPipeline.train` default for `n_jobs` from 4 to `None`,
  so that `n_jobs` will be dynamically calculated by default (203)
  - Use "feather"-formatted files to send data from users to CivisML, if possible.
  Require this when using ``pd.Categorical`` types, since CSVs require us to
  re-infer column types, and this can fail. Using feather should also give a
  speed improvement; it reads and writes faster than CSVs and produces smaller files (200).
  - ``ModelFuture`` objects will emit any warnings which occurred during their
  corresponding CivisML job (204)
  - Removed line setting "n_jobs" from an example of CivisML prediction.
  Recommended use is to let CivisML determine the number of jobs itself (211).
  - Update maximum CivisML version to v2.1; adjust fallback logic such that users get
  the most recent available release (212).
  - Restored the pre-v1.7.0 default behavior of the ``joblib`` backend by setting the ``remote_backend``
  parameter default to 'sequential' as opposed to 'civis'. The default of 'civis' would launch additional
  containers in nested calls to ``joblib.Parallel``. (205)
  - If validation metadata are missing, ``ModelFuture`` objects will return ``None``
  for metrics or validation metadata, rather than issuing an exception (208)
  - Allowed callers to pass `index` and `encoding` arguments to the `to_csv` method through `dataframe_to_civis`.
  Performance Enhancements
  - ```` now uses additional file handles for multipart upload instead of writing to disk to reduce disk usage
  - ```` writes dataframes to disk instead of using an in memory buffer


  - Relaxed requirement on ``cloudpickle`` version number (187)
  - Restore previous behavior of ```` when using "compression='gzip'" (195)


  - Specify escape character in ```` when performing parallel unload
  - Issue uploading files in ````
  - Revert performance enhancement that will change format of file produced by ````


  - Updated CivisML template ids to v2.0 (139)
  - Optional arguments to API endpoints now display in function signatures.
  Function signatures show a default value of "DEFAULT"; arguments will still
  only be transmitted to the Civis Platform API when explicitly provided. (140)
  - ``APIClient.feature_flags`` has been deprecated to avoid a name collision
  with the feature_flags endpoint. In v2.0.0, ``APIClient.featureflags``
  will be renamed to ``APIClient.feature_flags``.
  - The following APIClient attributes have been deprecated in favor of the
  attribute that includes underscores:
  ``APIClient.bocceclusters`` -> ``APIClient.bocce_clusters``
  ``APIClient.matchtargets`` -> ``APIClient.match_targets``
  ``APIClient.remotehosts`` -> ``APIClient.remote_hosts``
  - ```` and ```` functions now use
  ```` and ```` functions instead
  of separate logic
  - ````, ```` and ````
  now support files over 5GB
  - Refactor internals of ``CivisFuture`` and ``PollableResult`` to centralize handling
  of threads and ``pubnub`` subscription.
  - Updated API specification and base resources to include all general
  availability endpoints.
  - Changed ```` and ```` to allow
  strings for paths to local files in addition to just file/buffer objects.
  - Fixed parsing of multiword endpoints. Parsing no longer removes underscores
  in endpoint names.
  - In ``civis.futures.ContainerFuture``, return ``False`` when users attempt to cancel
  an already-completed job. Previously, the object would sometimes give a ``CivisAPIError``
  with a 404 status code. This fix affects the executors and joblib backend, which
  use the ``ContainerFuture``.
  - Tell ``flake8`` to ignore a broad except in a ``CivisFuture`` callback.
  - Close open sockets (in both the ``APIClient`` and ``CivisFuture``)  when they're no
  longer needed, so as to not use more system file handles than necessary (173).
  - Correct treatment of ``FileNotFoundError`` in Python 2 (176).
  - Fixed parsing of endpoints containing hyphens.  Hyphens are replaced with
  - Use ``civis.compat.TemporaryDirectory`` in ```` to be
  compatible with Python 2.7
  - Catch notifications sent up to 30 seconds before the ``CivisFuture`` connects.
  Fixes a bug where we would sometimes miss an immediate error on SQL scripts (174).
  - Documentation updated to include new CivisML features (137).
  - ``civis.resources.cache_api_spec`` function to make it easier to record the
  current API spec locally (141).
  - Autospecced mock of the ``APIClient`` for use in testing third-party code which
  uses this library (141).
  - Added `etl`, `n_jobs`, and `validation_data` arguments to
  ModelPipeline.train (139).
  - Added `cpu`, `memory`, and `disk` arguments to ModelPipeline.predict
  - Added ``remote_backend`` keyword to the ``civis.parallel.make_backend_factory``
  and ``civis.parallel.infer_backend_factory`` in order to set the joblib
  backend in the container for nested calls to ``joblib.Parallel``.
  - Added the PyPI trove classifiers for Python 3.4 and 3.6 (152).
  - ```` function to import an existing Civis file
  to a table
  - ```` function will now automatically retry uploads to
  the Civis Platform up to 5 times if is there is an HTTPError, ConnectionError
  or ConnectionTimeout
  - Additional documentation about the use case for the Civis joblib backend.
  - Added a note about serializing ``ModelPipeline`` ``APIClient`` objects to the docstring.
  - Added `civis notebooks download` command-line interface command to facilitate
  downloading notebooks.
  Performance Enhancements
  - ```` now takes advantage of multipart uploads to chunk
  files and perform I/O in parallel
  - ```` and ```` will always request
  data with gzip compression to reduce I/O. Also, they will attempt to fetch
  headers in a separate query so that data can be unloaded in parallel
  - ```` with ``compression='gzip'`` currently returns a file
  with no compression. In a future release, ``compression='gzip'`` will return a
  gzip compressed file.


  - Added explanatory text to CivisML_parallel_training.ipynb (126).
  - Added `ResourceWarning` for Python 2.7 (128).
  - Added `TypeError` for multi-indexed dataframes when used as input to
  CivisML (131).
  - ``ModelPipeline.from_existing`` will warn if users attempt to recreate
  a model trained with a newer version of CivisML, and fall back on the
  most recent prediction template it knows of (134).
  - Make the `PaginatedResponse` returned by LIST endpoints a full iterator.
  This also makes the `iterator=True` parameter work in Python 2.
  - When using ````, emit a warning on SQL queries which
  return no results instead of allowing a cryptic ``IndexError`` to surface (135).
  - Fixed the example code snippet for ````.
  Also provided more details on its return dict in the docstring.
  - Pinned down `sphinx_rtd_theme` and `numpydoc` in `dev-requirements.txt`
  for building the documentation.
  - Jupyter notebook with demonstrations of use patterns and abstractions in the Python API client (127).


  - Catch unnecessary warning while importing xgboost in CivisML_parallel_training.ipynb (121)
  - Fixed bug where instantiating a new model via ``ModelPipeline.from_existing`` from an existing model with empty "PARAMS" and "CV_PARAMS" boxes fails (122).
  - Users can now access the ``ml`` and ``parallel`` namespaces from the base ``civis`` namespace (123).
  - Parameters in the Civis API documentation now display in the proper order (124).


  - Edited example for safer null value handling
  - Make ``pubnub`` and ``joblib`` hard dependencies instead of optional dependencies (110).
  - Retry network errors and wait for API rate limit refresh when using the CLI (117).
  - The CLI now provides a User-Agent header which starts with "civis-cli" (117)
  - Include ``pandas`` and ``sklearn``-dependent code in Travis CI tests.
  - Version 1.1 of CivisML, with custom dependency installation from remote git hosting services (i.e., Github, Bitbucket).
  - Added email notifications option to ``ModelPipeline``.
  - Added custom ``joblib`` backend for multiprocessing in the Civis Platform. Public-facing functions are ``make_backend_factory``, ``make_backend_template_factory``, and ``infer_backend_factory``. Includes a new hard dependency on ``cloudpickle`` to facilitate code transport.
  - Fixed a bug where the version of a dependency for Python 2.7 usage was incorrectly specified.
  - Non-seekable file-like objects can now be provided to ````. Only seekable file-like objects will be streamed.
  - The ```` no longer raises an exception if its model job is cancelled.
  - The CLI's API spec cache now expires after 24 hours instead of 10 seconds.


  - Fixed a bug where ``ModelFuture.validation_metadata`` would not source training job metadata for a ``ModelFuture`` corresponding to prediction job (90).
  - Added more locks to improve thread safety in the ``PollableResult`` and ``CivisFuture``.
  - Fix issue with Python 2/3 dependency management (89).


  - Fixed a bug which caused an exception to be set on all ``ModelFuture`` objects, regardless of job status (86).
  - Fixed a bug which made the ``ModelPipeline`` unable to generate prediction jobs for models trained with v0.5 templates (84).
  - Handle the case when inputs to ``ModelFuture`` are ``numpy.int64`` (or other non-``integer`` ints) (85).
  - Convert `` (Markdown) to `README.rst` (reStructuredText).


  - Retries to http request in ``get_swagger_spec`` to make calls to ``APIClient`` robust to network failure
  - Parameter ``local_api_spec`` to ``APIClient`` to allow creation of client from local cache
  - Clarify ```` docstring with a note about treatment of the index.
  - Added functions ````, ````, and ````.
  - Added ```` namespace with ``ModelPipeline`` interface to Civis Platform modeling capabilities.
  - Added ``examples`` directory with sample ``ModelPipeline`` code from ````.
  - Python 2.7 compatibility
  - Corrected the defaults listed in the docstring for ````.
  - Do not allow uploading of files greater than 5GB to S3 (58).
  - Revised example code of docstring of civis_to_file to use bytes when downloading file
  - Modified retry behavior so that 413, 429, or 503 errors accompanied by a "Retry-After" header will be retried regardless of the HTTP verb used.
  - Add CSV settings arguments to ```` function.
  - Refactored use of "swagger" language.  ``get_swagger_spec`` is now ``get_api_spec`` and ``parse_swagger`` is now ``parse_api_spec``.
  - Modified ``CivisFuture`` so if PubNub is disconnected, it will fall back to polling on a shorter interval.


API Changes
  - Deprecate ``api_key`` input to higher-level functions and classes in favor of an ``APIClient`` input. The ``api_key`` will be removed in v2.0.0. (46)
  - Improved threading implementation in ``PollableResult`` so that it no longer blocks interpreter shutdown.
  - Allow the base url of the API to be configured through the ``CIVIS_API_ENDPOINT`` environment variable. (43)
  - Decorator function for deprecating parameters (46)


  - `civis.futures.CivisFuture` for tracking future results
  Performance Enhancements
  - ```` will perform a streaming upload to Platform if the optional ``requests-toolbelt`` package is installed.
  - Replace all ``PollableResult`` return values with ``CivisFuture`` to reduce the number of API calls and increase speed


  - support for multifile csv exports
  - support for subscription based polling
  - functions use the "hidden" API option to keep jobs out of the UI. Deprecate the "archive" parameter in favor of "hidden".
  - now has a "hidden" parameter which defaults to True
  - expose `poller` and `poller_args` as public attributes in `PollableResults`
  - update to  `default_credential` to handle pagination in `credentials.list` endpoint.
  - miscellaneous documentation fixes
  - unexpected keyword arguments passed to `APIClient` methods now raise appropriate TypeError
  Performance Enhancements
  - Decrease time required to create client objects from ~0.6 seconds to ~150 us for all objects after the first in a session


  - reads/writes to/from memory instead of disk where appropriate
  - Minor documentation corrections
  - 204/205 responses now return valid Response object


  - Initial release