Pulpcore

===================
REST API
--------

Features
~~~~~~~~

- Added an option for automatically creating repositories on the fly during an import procedure. The
option is disabled by default. Enable it by setting the field ``create_repositories`` to ``True``
via the REST API.
:github:`1920`
- Content app now groups distributions in a directory structure on the landing page.
:github:`1951`
- Added RBAC protection to upload objects.
:github:`2362`
- New endpoint to list all Remote objects is now available at /pulp/api/v3/remotes/.
:github:`2530`
- ``HyperlinkRelatedFilter`` can now be filtered by object types and NULL values.

Repositories can now be filtered by Remotes.
:github:`2864`
- Introduced the ``with_content`` query parameter that filters distributions by the specified content
unit.
:github:`2952`
- Add a debug log to see where is file downloaded from.
:github:`3088`
- Introduces anonymous telemetry data posting to `<https://analytics.pulpproject.org/>`_. This is
enabled by default, and can be disabled by setting the ``TELEMETRY`` setting to ``False``. See the
:ref:`telemetry docs <analytics>` for more info on exactly what is posted along with an example.
:github:`3115`


Bugfixes
~~~~~~~~

- Fixed the value of the Content-Type header returned for .xml.gz files.
:github:`2811`
- Improve content app performance on head requests
:github:`2924`
- Use published relative paths for FS Exporter.
:github:`2933`
- Configured aiohttp to avoid rewriting redirect URLs, as some web servers (e.g. Amazon CloudFront) can be tempermental about the encoding of the URL.
:github:`2964`
- Fixed inaccurate 404 error message for content app.
:github:`2977`
- Fixed variable referenced before assignment error in ``django-admin dump-permissions``.
:github:`3011`
- Do not create telemetry TaskSchedule for production systems.
:github:`3015`
- Serialized orphan cleanup tasks with respect to each other to prevent them from failing.
:github:`3030`
- Fixed 500 error when 'range' header starts with a negative value for 'on-demand' content.
:github:`3052`
- Fixed bug where 'range' header with a start value greater than size of on-demand content would produce an incomplete response.
:github:`3054`
- Fixed a bug where Content-Length header value was wrong when on-demand content was requested with
a Range header that has an end value greater than the size of the content.
:github:`3055`
- Fixed a bug in the routing logic, where generic base class viewsets were served on actual urls.
:github:`3056`
- Fixed a bug in import code where all objects imported would also be added to the target repository
by their UUID. In case of a UUID-collision with content, unwanted content may have ended up being
in the next repository version.
:github:`3064`
- Fixed a bug that caused the import machinery to import the same content multiple times in a row.
:github:`3075`
- Limited access policy reset to viewsets with a default one. This will solve 500 errors when
trying to reset an access policy whose viewset name is repeated by an abstract base class.
Stopped reporting on unmodified access policies when migrating.
:github:`3080`
- Fixed another rare deadlock for high-concurrency/overlapping-content syncs.
:github:`3111`
- Fixed the progress report counter for imported content units.
:github:`3113`
- Moved telemetry setup to the pulpcore-worker startup sequence. This will prevent orm calls before
all apps are ready.
:github:`3122`


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- docs: Update the architecture diagram to reflect the fact that both API and workers talk to redis.
:github:`3000`
- Multiple updates to the PyPI (manual) install instructions.
:github:`3051`


Misc
~~~~

- :github:`2445`, :github:`2890`, :github:`3063`, :github:`3091`


Plugin API
----------

Features
~~~~~~~~

- Exposed the ``RepositoryResource`` class to enable plugin writers to customize the way of
importing/exporting of particular repository types. Repositories should be now a part of exported
resources to enable automatic creation of missing repositories.
:github:`1920`
- Added a global access condition ``has_upload_param_model_or_obj_perms`` to enforce permissions
on the upload parameter.
:github:`2362`
- Extended the interface of ``verify_signature`` as a new function ``gpg_verify`` to support file
like objects in addition to a file path and also return the ``python-gnupg`` ``verify`` object.
:github:`2930`
- Added new field ``info`` to the ``RepositoryVersion``. This will allow to store additional information for a specific version.
:github:`2998`
- Added `pulpcore.plugin.models.EncryptedTextField to plugin api.
:github:`3157`


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- Adds Master/Detail pattern overview and usage documentation to the Plugin writer docs.
:github:`2981`
- Documented the use of ``RolesMixin`` in the plugin writer concepts section.
:github:`3085`


Removals
~~~~~~~~

- Deprecated ``verify_signature`` in favor of ``gpg_verify`` for removal in 3.25.
:github:`2930`

===================
REST API
--------

Features
~~~~~~~~

- Added a repository filter to publications.
:github:`1912`
- The status API endpoint now shows the python package name that provides a given plugin.
:github:`1982`
- Queryset scoping can be customized by the user using the new field ``queryset_scoping`` on a
ViewSet's AccessPolicy.
:github:`2114`
- Enabled administrators to work with a customized GnuPG home directory and keyring during the
creation of a signing service. The introduced optional arguments ``--gnupghome`` and ``--keyring``
are available under the ``pulpcore-manager add-signing-service`` command.
:github:`2476`
- Added the setting ``REDIRECT_TO_OBJECT_STORAGE`` to allow using cloud storage with or without
redirecting urls.

Added support for sftp storage via the ``pulpcore.app.models.storage.PulpSFTPStorage`` class.
:github:`2537`
- Added more details to an error message that is shown when none of the allowed content checksums
hashers could be used.
:github:`2550`
- Add contains_permission query parameter to the roles API that allows clients to get back a list
of roles that have any permission in a list of permissions.
:github:`2715`
- Master Content endpoint, ``/pulp/api/v3/content/``, has a new access policy that allows any
authenticated user to view content. The endpoint now scopes the content based on repositories
the user can see.
:github:`2724`
- New AccessPolicies have been added to ContentGuard, Distribution, Publication, Repository,
and RepositoryVersions master ViewSets. Queryset scoping has been enabled for each ViewSet.
:github:`2725`
- New AccessPolicy for ContentRedirectContentGuard ViewSet has been added.
:github:`2726`
- Added dump-permissions management command to list deprecated permissions not yet translated into
roles. This is the only way to get to this information after the 3.20 release.
:github:`2741`
- Add ?for_object_type query parameter to Roles API that accepts an object HREF and returns a list
of roles that only contain permissions for the given object type.
:github:`2747`
- Add role description and permissions to group and user role serializer.
:github:`2765`


Bugfixes
~~~~~~~~

- Leading and trailing whitespace characters are no longer trimmed in passwords within remotes.
:github:`2068`
- Fixed generation of the redirect url to the object storage
:github:`2075`
- Taught PulpImport to stream imports rather than reading files into memory in one chunk.

This largely alleviates the memory-pressure that results from importing multiple
large repositories in parallel.
:github:`2307`
- Made the API root endpoint accessible for anonymous users once again.
:github:`2340`
- Removed il8n from the logs written so they will always show up in English for speedy resolution of
error messages. All user facing strings are still expected to be il8n.
:github:`2477`
- Replaced "//" with "/" in base_url when CONTENT_PATH_PREFIX is "" or "/".
:github:`2553`
- Fixed `does_batch` method in sync pipeline to allow waiting on content that is already resolved.
:github:`2557`
- Fixed OOM error after uploading large chunked files.
:github:`2573`
- Ensure downloader resets file on retry.
:github:`2576`
- Taught PulpImport to retry more than once in the event of creation-collisions.

This fixes a rare import-failure during high-concurrency, high-content-overlap imports.
:github:`2589`
- Improved the error message when HTTP proxies reject requests from Pulp.
:github:`2654`
- Fix ?ordering=role on user and group role apis so that it sorts results by role name.
:github:`2703`
- Add options to the role_util functions to make them work the same as guardian did.
:github:`2739`
- Fixed a bug that disallowed administrators to create a signing service via the pulpcore-manager
utility.
:github:`2798`
- Reduced duplicate SQL queries for ``AccessPolicy`` when accessing any view.
:github:`2802`
- Fixed docs regarding the default for orphan protection time.
:github:`2810`
- Started showing errors when users try to export remote artifacts.
:github:`2817`
- Restore multiple-retry logic for PulpImport.
:github:`2854`


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- Cleared out some of the paragraphs from the "Pull Request Walkthrough" section.
:github:`1852`
- Added a troubleshooting section that includes information on how to enable DEBUG logging.
:github:`1944`
- Removed some out of date references to Redmine (the previous issue tracker). We use Github Issues
now.
:github:`2642`
- Added a note about explicitly setting ``content_object`` to null when assigning model-level
permissions.
:github:`2758`
- Fixed ``extlinks`` use in docs to be Sphinx==5.0.0 compatible.
:github:`2782`
- Update installation instructions about "User and database configuration" for the Database setup to point to a matching Django documentation.
:github:`2877`


Removals
~~~~~~~~

- Removed the group permission endpoints ``api/v3/groups/:id/model_permissions/`` and
``api/v3/groups/:id/object_permissions/``. Permissions should be managed via roles exclusively.
:github:`2050`
- Removed django-guardian from the stack. The ``guardian.backends.ObjectPermissionBackend`` should
not be used anymore.
:github:`2051`
- Deprecated creation hook interface was removed. Creation hook need to be registered with the view
set by the plugin writer before being used. Creation hooks can no longer be added with the
deprecated name ``permission_assignments``.
:github:`2559`


Misc
~~~~

- :github:`2070`, :github:`2244`, :github:`2605`, :github:`2643`


Plugin API
----------

Features
~~~~~~~~

- Plugins are required to provide the ``python_package_name`` as a string attribute on their subclass
of ``PulpPluginAppConfig``.
:github:`1982`
- Exposed the method ``raise_for_unknown_content_units`` which raises ``ValidationError`` for content
units that were not found in the database.
:github:`2052`
- Plugins now have to enable default queryset scoping by setting the ``queryset_scoping`` field on the
AccessPolicy to ``{"function": "scope_queryset"}``.

Default queryset scoping behavior can be changed by supplying a new ``scope_queryset`` method.

Extra queryset scoping functions can be declared on plugin ViewSets and used by setting the
AccessPolicy's ``queryset_scoping`` field.
:github:`2114`
- DeclarativeArtifact now accepts a ``urls`` option which permits multiple URLs
to be provided for a single artifact. If multiple URLs are provided, the download
stage will try each of them in turn upon encountering failures.
:github:`2175`
- Exposed the function ``pulpcore.plugin.util.verify_signature`` for verifying signatures created
by signing services.
:github:`2476`
- Added ``pulpcore.plugin.content.ArtifactResponse`` to plugin API. Use this response to stream an
artifact from the object storage if redirecting is impossible.
:github:`2537`
- Queryset scoping is now performed when the ViewSet's AccessPolicy field ``scope_queryset`` is set to
a function on the ViewSet.

``NamedModelViewSet`` now has default scoping method ``scope_queryset`` that will scope the request
off of ``queryset_filtering_required_permission`` if present. If ViewSet is a master ViewSet then
scoping will be performed by calling each child's scoping method if present.
:github:`2723`
- Content ViewSets default ``scope_queryset`` method will scope based on repositories the user can see.
:github:`2724`
- Added the ability to specify an upload for the single shot upload serializer. This allows to
upload files in chunks and attach them with content in repositories without creating orphans.
:github:`2786`
- Added new access condition ``has_required_repo_perms_on_upload`` for RBAC plugins to use to require
users to specify a repository when uploading content. If not used when uploading content, non-admin
users will not be able to see their uploaded content if queryset scoping is enabled.
:github:`2796`


Bugfixes
~~~~~~~~

- Reworked the ordering framework to use django-filters.

Plugins should not declare filter-backends on viewsets.
:github:`2703`


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- Updated plugin writers RBAC guide to explain more roles and less permissions. Removed mentions of
django-guardian.
:github:`2463`
- Added docs on the expectation that all user-facing strings are i8ln wrapped with gettext, but log
messages are not.
:github:`2477`


Removals
~~~~~~~~

- The ``pulpcore.plugin.exceptions.MissingResource`` object has been removed. Instead let 404
errors propagate upwards for DRF to handle, or use the DRF exception ``NotFound``.
:github:`1812`
- Removed django-guardian from the stack. This includes the removal of ``AutoDeleteObjPermsMixin``
from the plugin api.
:github:`2051`
- Removed the ``custom_file_object`` argument to ``pulpcore.plugin.download.BaseDownloader``. Now all
downloaded data will be written to a random file in the current working directory. Further
customization of where downloaded data can be written to can be done through subclassing.
:github:`2137`
- Constructor signature of `DigestValidationError` and `SizeValidationError` has changed - the
"actual" and "expected" values are now required and "url" which was previously a positional
argument is now a keyword argument.
:github:`2244`
- The pulpcore.plugin.constants.API_ROOT has been removed. Use the ``V3_API_ROOT`` and
``V3_API_ROOT_NO_FRONT_SLASH`` settings instead.
:github:`2556`
- Plugins using the ``SingleArtifactContentUploadSerializer`` must place a super call when
overwriting ``deferred_validate``. They can only assume the existance of the ``Artifact`` in the
database, after this call.
:github:`2786`


Misc
~~~~

- :github:`2634`, :github:`2742`

===================
REST API
--------

Bugfixes
~~~~~~~~

- Taught PulpImport to stream imports rather than reading files into memory in one chunk.

This largely alleviates the memory-pressure that results from importing multiple
large repositories in parallel.
:github:`2307`
- Fixed `does_batch` method in sync pipeline to allow waiting on content that is already resolved.
:github:`2557`
- Ensure downloader resets file on retry.
:github:`2576`
- Taught PulpImport to retry more than once in the event of creation-collisions.

This fixes a rare import-failure during high-concurrency, high-content-overlap imports.
:github:`2589`
- Improved the error message when HTTP proxies reject requests from Pulp.
:github:`2654`
- Restore multiple-retry logic for PulpImport.
:github:`2854`
- Improve content app performance on head requests
:github:`2924`


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- Removed some out of date references to Redmine (the previous issue tracker). We use Github Issues
now.
:github:`2642`
- Fixed ``extlinks`` use in docs to be Sphinx==5.0.0 compatible.
:github:`2782`


Misc
~~~~

- :github:`2605`


Plugin API
----------

No significant changes.

===================
REST API
--------

Features
~~~~~~~~

- Content app now logs where it gets on-demand and streamed content from.
:github:`2059`
- Reclaim disk space can now accept ["*"] for ``repo_hrefs`` to specify all repositories for reclaim.
:github:`2065`
- Added a filter to allow searching for user roles by their description.
:github:`2276`
- Add swagger view and make OpenAPI human readable
:github:`2291`
- Adds a ``TASK_DIAGNOSTICS`` setting which will enable each task to write out diagnostic information
such as memory usage of the task to a data file in ``/var/tmp/pulp/<task_UUID>/``. This is disabled
by default.
:github:`2329`
- Added a ``/pulp/api/v3/distributions/`` endpoint to list all distributions.
:github:`2379`


Bugfixes
~~~~~~~~

- Added reason for 404 error when accessing distributions without a publication.
:github:`1910`
- Fixed validation order of required settings to occur before plugin settings are loaded.
:github:`1968`
- Fix delete repository version causing "duplicate key value violates unique constraint" error.
:github:`2047`
- Fixed two instances of Pulp not writing to the task worker's temporary directory.
:github:`2061`
- Reduced memory usage during tasks like sync by holding fewer objects in-memory unnecessarily.
:github:`2069`
- Fixed migration 0064_add_new_style_task_columns to purge extraneous ReservedResource and
TaskReservedResource entries, which could block sync and publish tasks post-upgrade.

Also taught the migration to bulk-update the Task changes. In large installations, this
should have a positive impact on the time it takes to apply the migration.
:github:`2101`
- Taught task-purge to process tasks in batches of 1000. This prevents large purges from using
large amounts of memory as a result of reading all the affected Tasks into memory at once.
:github:`2215`
- This fix prevents the lost track of a content removed version when deleting a repository version that deletes a content that is added back in the subsequent version, but deleted again in a later version.
:github:`2267`
- Added transactions around repository version operations to prevent data loss.
:github:`2268`
- Loosened the version-restrictions on PulpImport to only require X.Y matching.
:github:`2269`
- Fix a mistake in a previous migration which may have caused improperly encrypted remote fields.
:github:`2327`
- Fixed improper fields being listed in ``RepositoryVersion`` repair API.
:github:`2330`
- Fixes duplicate key error ``Key (content_artifact_id, remote_id)`` when creating ``RemoteArtifacts``
during syncs in pulp_container and possibly other plugins.
:github:`2381`
- Declared proper dependency on user model in migration 0040.
:github:`2403`
- Fixed a rare deadlock when sync'ing overlapping content in high-concurrency envs.
:github:`2420`
- Fixed a (rare) deadlock around bulk_update() during syncs with overlapping content.
:github:`2430`
- Fixed a bug where notifications to workers may go unnoticed. This may lead to idle workers while
there are tasks waiting.
:github:`2506`


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- Updates and revises docs on webserver based authentication.
:github:`2260`
- Adds docs on recording and building graphs from the memory data saved by the ``TASK_DIAGNOSTICS``
setting.
:github:`2329`


Removals
~~~~~~~~

- Removed the Django UI Admin site. It was added to provide RBAC permissions management before there
were APIs that could provide that. It was tech preview and now there are APIs for user and group
management, along with role and permission assignment. It is being removed because the direct DB
access it provides has caused some issues for users, especially since its not integrated with the
validation provided by Django Rest Framework, which Pulp uses.
:github:`2374`


Plugin API
----------

Features
~~~~~~~~

- Exposed the ``PulpRemoteUserAuthentication`` class to plugin writers. This will allow the use of
remote authentication methods when building protected endpoints.
:github:`2262`
- Added new global access conditions ``has_publication_param_model_or_obj_perms`` and
``has_repo_or_repo_ver_param_model_or_obj_perms`` for RBAC checks.
:github:`2364`
- Changed the ``reusable_conditions`` module configuration for access policies to being a list to
enable plugins to add custom modules to it.
:github:`2495`


Bugfixes
~~~~~~~~

- Adjusted the default size of the queues between pipelines to be 1 instead of 1000. The batchers in
the stage will still accumulate up to 500 (by default) items so batching is still in-effect there
where it matters.
:github:`2069`

====================
REST API
--------

No significant changes.


Plugin API
----------

No significant changes.


----

====================
REST API
--------

Bugfixes
~~~~~~~~

- Fixed a warning that gets raised when cache is enabled: ``RuntimeWarning: coroutine
'AsyncCache.delete' was never awaited``.
:github:`4967`


Plugin API
----------

No significant changes.


----