Django-storages

******************

S3
--

- Avoid misleading ``AWS_DEFAULT_ACL`` warning for insecure ``default_acl`` when
overridden as a class variable (`591`_)
- Propagate file deletion to cache when ``preload_metadata`` is ``True``,
(not the default) (`743`_, `749`_)
- Fix exception raised on closed file (common if using ``ManifestFilesMixin`` or
``collectstatic``. (`382`_, `754`_)

Azure
-----

- Pare down the required packages in ``extra_requires`` when installing the ``azure`` extra to only
``azure-storage-blob`` (`680`_, `684`_)
- Fix compatibility with ``generate_blob_shared_access_signature`` updated signature (`705`_, `723`_)
- Fetching a file now uses the configured timeout rather than hardcoding one (`727`_)
- Add support for configuring all blobservice options: ``AZURE_ENDPOINT_SUFFIX``,
``AZURE_CUSTOM_DOMAIN``, ``AZURE_CONNECTION_STRING``, ``AZURE_TOKEN_CREDENTIAL``.
See the docs for more info. Huge thanks once again to nitely. (`750`_)
- Fix filename handling to not strip special characters (`609`_, `752`_)


Google Cloud
------------

- Set the file acl in the same call that uploads it (`698`_)
- Reduce the number of queries and required permissions when ``GS_AUTO_CREATE_BUCKET`` is
``False`` (the default) (`412`_, `718`_)
- Set the ``predefined_acl`` when creating a ``GoogleCloudFile`` using ``.write``
(`640`_, `756`_)
- Add ``GS_BLOB_CHUNK_SIZE`` setting to enable efficient uploading of large files (`757`_)

Dropbox
-------

- Complete migration to v2 api with file fetching and metadata fixes (`724`_)
- Add ``DROPBOX_TIMEOUT`` to configure client timeout defaulting to 100 seconds
to match the underlying sdk. (`419`_, `747`_)

SFTP
----

- Fix reopening a file (`746`_)

.. _591: https://github.com/jschneier/django-storages/pull/591
.. _680: https://github.com/jschneier/django-storages/issues/680
.. _684: https://github.com/jschneier/django-storages/pull/684
.. _698: https://github.com/jschneier/django-storages/pull/698
.. _705: https://github.com/jschneier/django-storages/issues/705
.. _723: https://github.com/jschneier/django-storages/pull/723
.. _727: https://github.com/jschneier/django-storages/pull/727
.. _746: https://github.com/jschneier/django-storages/pull/746
.. _724: https://github.com/jschneier/django-storages/pull/724
.. _412: https://github.com/jschneier/django-storages/pull/412
.. _718: https://github.com/jschneier/django-storages/pull/718
.. _743: https://github.com/jschneier/django-storages/issues/743
.. _749: https://github.com/jschneier/django-storages/pull/749
.. _750: https://github.com/jschneier/django-storages/pull/750
.. _609: https://github.com/jschneier/django-storages/issues/609
.. _752: https://github.com/jschneier/django-storages/pull/752
.. _382: https://github.com/jschneier/django-storages/issues/382
.. _754: https://github.com/jschneier/django-storages/pull/754
.. _419: https://github.com/jschneier/django-storages/issues/419
.. _747: https://github.com/jschneier/django-storages/pull/747
.. _640: https://github.com/jschneier/django-storages/issues/640
.. _756: https://github.com/jschneier/django-storages/pull/756
.. _757: https://github.com/jschneier/django-storages/pull/757

******************

- Fix off-by-1 error in ``get_available_name`` whenever ``file_overwrite`` or ``overwrite_files`` is ``True`` (`588`_, `589`_)
- Change ``S3Boto3Storage.listdir()`` to use ``list_objects`` instead of ``list_objects_v2`` to restore
compatibility with services implementing the S3 protocol that do not yet support the new method (`586`_, `590`_)

.. _588: https://github.com/jschneier/django-storages/issues/588
.. _589: https://github.com/jschneier/django-storages/pull/589
.. _586: https://github.com/jschneier/django-storages/issues/586
.. _590: https://github.com/jschneier/django-storages/pull/590

****************

**Security**

- The ``S3BotoStorage`` and ``S3Boto3Storage`` backends have an insecure
default ACL of ``public-read``. It is recommended that all current users audit their bucket
permissions. Support has been added for setting ``AWS_DEFAULT_ACL = None`` and ``AWS_BUCKET_ACL =
None`` which causes all created files to inherit the bucket's ACL (and created buckets to inherit the
Amazon account's default ACL). This will become the default in version 1.10 (for ``S3Boto3Storage`` only
since ``S3BotoStorage`` will be removed in version 1.9, see below). Additionally, a warning is now
raised if ``AWS_DEFAULT_ACL`` or ``AWS_BUCKET_ACL`` is not explicitly set. (`381`_, `535`_, `579`_)

**Breaking**

- The ``AzureStorage`` backend and documentation has been completely rewritten. It now
depends on ``azure`` and ``azure-storage-blob`` and is *vastly* improved. Big thanks to nitely and all
other contributors along the way (`565`_)
- The ``.url()`` method of ``GoogleCloudStorage`` has been completely reworked. Many use
cases should require no changes and will experience a massive speedup. The ``.url()`` method no longer hits
the network for public urls and generates signed urls (with a default of 1-day expiration, configurable
via ``GS_EXPIRATION``) for non-public buckets. Check out the docs for more information. (`570`_)
- Various backends will now raise ``ImproperlyConfigured`` at runtime if their
location (``GS_LOCATION``, ``AWS_LOCATION``) begins with a leading ``/`` rather than silently
stripping it. Verify yours does not. (`520`_)
- The long deprecated ``GSBotoStorage`` backend is removed. (`518`_)

**Deprecation**

- The insecure default of ``public-read`` for ``AWS_DEFAULT_ACL`` and
``AWS_BUCKET_ACL`` in ``S3Boto3Storage`` will change to inherit the bucket's setting in version 1.10 (`579`_)
- The legacy ``S3BotoBackend`` is deprecated and will be removed in version 1.9.
It is strongly recommended to move to the ``S3Boto3Storage`` backend for performance,
stability and bugfix reasons. See the `boto migration docs`_ for step-by-step guidelines. (`578`_, `584`_)
- The long aliased arguments to ``S3Boto3Storage`` of ``acl`` and ``bucket`` are
deprecated in favor of ``bucket_name`` and ``default_acl`` (`516`_)
- The minimum required version of ``boto3`` will be increasing to ``1.4.4`` in
the next major version of ``django-storages``. (`583`_)

**Features**

- Add support for a file to inherit its bucket's ACL by setting ``AWS_DEFAULT_ACL = None`` (`535`_)
- Add ``GS_CACHE_CONTROL`` setting for ``GoogleCloudStorage`` backend (`411`_, `505`_)
- Add documentation around using django-storages with Digital Ocean Spaces (`521`_)
- Add support for Django 2.1 and Python 3.7 (`530`_)
- Make ``S3Boto3Storage`` pickleable (`551`_)
- Add automatic reconnection to ``SFTPStorage`` (`563`_, `564`_)
- Unconditionally set the security token in the boto backends (`b13efd`_)
- Improve efficiency of ``.listdir`` on ``S3Boto3Storage`` (`352`_)
- Add ``AWS_S3_VERIFY`` to support custom certificates and disabling certificate verification
to ``S3Boto3Storage`` (`486`_, `580`_)
- Add ``AWS_S3_PROXIES`` setting to ``S3Boto3Storage`` (`583`_)
- Add a snazzy new logo. Big thanks to reallinfo

**Bugfixes**

- Reset file read offset before passing to ``GoogleCloudStorage`` and ``AzureStorage`` (`481`_, `581`_, `582`_)
- Fix various issues with multipart uploads in the S3 backends
(`169`_, `160`_, `364`_, `449`_, `504`_, `506`_, `546`_)
- Fix ``S3Boto3Storage`` to stream down large files (also disallow `r+w` mode) (`383`_, `548`_)
- Fix ``SFTPStorageFile`` to align with the core ``File`` abstraction (`487`_, `568`_)
- Catch ``IOError`` in ``SFTPStorage.delete`` (`568`_)
- ``AzureStorage``, ``GoogleCloudStorage``, ``S3Boto3Storage`` and ``S3BotoStorage`` now
respect ``max_length`` when ``file_overwrite = True`` (`513`_, `554`_)
- The S3 backends now consistently use ``compresslevel=9`` (the Python stdlib default)
for gzipped content (`572`_, `576`_)
- Improve error message of ``S3Boto3Storage`` during an unexpected exception when automatically
creating a bucket (`574`_, `577`_)

.. _381: https://github.com/jschneier/django-storages/issues/381
.. _535: https://github.com/jschneier/django-storages/pull/535
.. _579: https://github.com/jschneier/django-storages/pull/579
.. _565: https://github.com/jschneier/django-storages/pull/565
.. _520: https://github.com/jschneier/django-storages/pull/520
.. _518: https://github.com/jschneier/django-storages/pull/518
.. _516: https://github.com/jschneier/django-storages/pull/516
.. _481: https://github.com/jschneier/django-storages/pull/481
.. _581: https://github.com/jschneier/django-storages/pull/581
.. _582: https://github.com/jschneier/django-storages/pull/582
.. _411: https://github.com/jschneier/django-storages/issues/411
.. _505: https://github.com/jschneier/django-storages/pull/505
.. _521: https://github.com/jschneier/django-storages/pull/521
.. _169: https://github.com/jschneier/django-storages/pull/169
.. _160: https://github.com/jschneier/django-storages/issues/160
.. _364: https://github.com/jschneier/django-storages/pull/364
.. _449: https://github.com/jschneier/django-storages/issues/449
.. _504: https://github.com/jschneier/django-storages/pull/504
.. _530: https://github.com/jschneier/django-storages/pull/530
.. _506: https://github.com/jschneier/django-storages/pull/506
.. _546: https://github.com/jschneier/django-storages/pull/546
.. _383: https://github.com/jschneier/django-storages/issues/383
.. _548: https://github.com/jschneier/django-storages/pull/548
.. _b13efd: https://github.com/jschneier/django-storages/commit/b13efd92b3bf3e9967b8e7819224bfcf9abb977e
.. _551: https://github.com/jschneier/django-storages/pull/551
.. _563: https://github.com/jschneier/django-storages/issues/563
.. _564: https://github.com/jschneier/django-storages/pull/564
.. _487: https://github.com/jschneier/django-storages/issues/487
.. _568: https://github.com/jschneier/django-storages/pull/568
.. _513: https://github.com/jschneier/django-storages/issues/513
.. _554: https://github.com/jschneier/django-storages/pull/554
.. _570: https://github.com/jschneier/django-storages/pull/570
.. _572: https://github.com/jschneier/django-storages/issues/572
.. _576: https://github.com/jschneier/django-storages/pull/576
.. _352: https://github.com/jschneier/django-storages/pull/352
.. _574: https://github.com/jschneier/django-storages/issues/574
.. _577: https://github.com/jschneier/django-storages/pull/577
.. _486: https://github.com/jschneier/django-storages/pull/486
.. _580: https://github.com/jschneier/django-storages/pull/580
.. _583: https://github.com/jschneier/django-storages/pull/583
.. _boto migration docs: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#migrating-boto-to-boto3
.. _578: https://github.com/jschneier/django-storages/pull/578
.. _584: https://github.com/jschneier/django-storages/pull/584

******************

* You can now specify the backend you are using to install the necessary dependencies using
``extra_requires``. For example ``pip install django-storages[boto3]`` (`417`_)
* Add additional content-type detection fallbacks (`406`_, `407`_)
* Add ``GS_LOCATION`` setting to specify subdirectory for ``GoogleCloudStorage`` (`355`_)
* Add support for uploading large files to ``DropBoxStorage``, fix saving files (`379`_, `378`_, `301`_)
* Drop support for Django 1.8 and Django 1.10 (and hence Python 3.3) (`438`_)
* Implement ``get_created_time`` for ``GoogleCloudStorage`` (`464`_)

.. _417: https://github.com/jschneier/django-storages/pull/417
.. _407: https://github.com/jschneier/django-storages/pull/407
.. _406: https://github.com/jschneier/django-storages/issues/406
.. _355: https://github.com/jschneier/django-storages/pull/355
.. _379: https://github.com/jschneier/django-storages/pull/379
.. _378: https://github.com/jschneier/django-storages/issues/378
.. _301: https://github.com/jschneier/django-storages/issues/301
.. _438: https://github.com/jschneier/django-storages/issues/438
.. _464: https://github.com/jschneier/django-storages/pull/464

******************

* Fix Django 1.11 regression with gzipped content being saved twice
resulting in empty files (`367`_, `371`_, `373`_)
* Fix the ``mtime`` when gzipping content on ``S3Boto3Storage`` (`374`_)

.. _367: https://github.com/jschneier/django-storages/issues/367
.. _371: https://github.com/jschneier/django-storages/pull/371
.. _373: https://github.com/jschneier/django-storages/pull/373
.. _374: https://github.com/jschneier/django-storages/pull/374

******************

* Files uploaded with ``GoogleCloudStorage`` will now set their appropriate mimetype (`320`_)
* Fix ``DropBoxStorage.url`` to work. (`357`_)
* Fix ``S3Boto3Storage`` when ``AWS_PRELOAD_METADATA = True`` (`366`_)
* Fix ``S3Boto3Storage`` uploading file-like objects without names (`195`_, `368`_)
* ``S3Boto3Storage`` is now threadsafe - a separate session is created on a
per-thread basis (`268`_, `358`_)

.. _320: https://github.com/jschneier/django-storages/pull/320
.. _357: https://github.com/jschneier/django-storages/pull/357
.. _366: https://github.com/jschneier/django-storages/pull/366
.. _195: https://github.com/jschneier/django-storages/pull/195
.. _368: https://github.com/jschneier/django-storages/pull/368
.. _268: https://github.com/jschneier/django-storages/issues/268
.. _358: https://github.com/jschneier/django-storages/pull/358