Scriptworker

----------------------

.. _added-73:

Added
~~~~~

- ``scriptworker.cot.verify`` now verifies the chain of trust for the
graph.
- ``scriptworker.exceptions.CoTError`` now marks chain of trust
validation errors.
- ``scriptworker.task.get_task_id``, ``scriptworker.task.get_run_id``,
``scriptworker.task.get_decision_task_id``,
``scriptworker.task.get_worker_type``
- ``scriptworker.log.contextual_log_handler`` for short-term logs
- added framework for new docs

.. _changed-70:

Changed
~~~~~~~

- config files are now yaml, to enable comments.
``config_example.json`` and ``cot_config_example.json`` have been
consolidated into ``scriptworker.yaml.tmpl``. ``context.cot_config``
items now live in ``context.config``.
- ``validate_artifact_url`` now takes a list of dictionaries as rules,
leading to more configurable url checking.
- ``scriptworker.cot`` is now ``scriptworker.cot.generate``. The
``get_environment`` function has been renamed to
``get_cot_environment``.
- ``scriptworker.gpg.get_body`` now takes a ``verify_sig`` kwarg.
- ``download_artifacts`` now takes ``valid_artifact_task_ids`` as a
kwarg.
- ``max_connections`` is now ``aiohttp_max_connections``
- scriptworker task definitions now expect an ``upstreamArtifacts``
list of dictionaries

.. _fixed-62:

Fixed
~~~~~

- docstring single backticks are now double backticks
- catch aiohttp exceptions on upload

.. _removed-30:

Removed
~~~~~~~

- removed all references to ``cot_config``
- removed the credential update, since puppet restarts scriptworker on
config change.

.. _section-120:

--------------------

.. _added-74:

Added
~~~~~

- ``gpg_lockfile`` and ``last_good_git_revision_file`` in config
- ``get_last_good_git_revision`` and ``write_last_good_git_revision``
now return the last good git revision, and write it to
``last_good_git_revision_file``, respectively.
- ``get_tmp_base_gpg_home_dir`` is a helper function to avoid
duplication in logic.
- ``rebuild_gpg_homedirs`` is a new entry point script that allows us
to recreate the gpg homedirs in a tmpdir, in a separate process
- ``is_lockfile_present``, ``create_lockfile``, and ``rm_lockfile`` as
helper functions for the two gpg homedir entry points.

.. _changed-71:

Changed
~~~~~~~

- ``sign_key``, ``rebuild_gpg_home_flat``, ``rebuild_gpg_home_signed``,
``build_gpg_homedirs_from_repo`` are no longer async.
- ``overwrite_gpg_home`` only keeps one backup.
- ``update_signed_git_repo`` now returns the latest git revision,
instead of a boolean marking whether the revision is new or not. This
will help avoid the scenario where we update, fail to generate the
gpg homedirs, and then stay on an old revision until the next push.
- ``update_logging_config`` now takes a ``file_name`` kwarg, which
allows us to create new log files for the ``rebuild_gpg_homedirs``
and ``create_initial_gpg_homedirs`` entry points.

.. _fixed-63:

Fixed
~~~~~

- ``build_gpg_homedirs_from_repo`` now waits to verify the contents of
the updated git repo before nuking the previous base gpg homedir.
- ``create_initial_gpg_homedirs`` now creates a logfile

.. _removed-31:

Removed
~~~~~~~

- ``rebuild_gpg_homedirs_loop`` is no longer needed, and is removed.

.. _section-121:

--------------------

.. _changed-72:

Changed
~~~~~~~

- logged the stacktrace if the ``main`` loop hits an exception. No
longer catch and ignore ``RuntimeError``, since it wasn’t clear why
that was put in.
- updated ``check_config`` to make sure taskcluster-related configs
match taskcluster requirements

.. _fixed-64:

Fixed
~~~~~

- changed the way the polling loop works: ``async_main`` is now a
single pass, which ``main`` calls in a ``while True`` loop. This
should fix the situation where polling was dying silently while the
git update loop continued running every 5 minutes.

.. _section-122:

--------------------

.. _fixed-65:

Fixed
~~~~~

- explicitly pass ``taskId`` and ``runId`` to ``claim_task``. There’s a
new ``hintId`` property that appears in ``message_info['task_info']``
that broke things.

.. _section-123:

--------------------

.. _added-75:

Added
~~~~~

- added ``git_key_repo_dir``, ``base_gpg_home_dir``, ``my_email``, and
``gpg_path`` to ``config_example.json``
- added ``cot_config_example.json``, ``cot_config_schema.json``, and
``scriptworker.config.get_cot_config`` for ChainOfTrust config
- added ``update_signed_git_repo``, ``verify_signed_git_commit``,
``build_gpg_homedirs_from_repo``, ``rebuild_gpg_homedirs_loop``, and
``create_initial_gpg_homedirs`` for gpg homedir creation and updates
in the background.
- added a background call to update the gpg homedirs in
``scriptworker.worker.async_main``
- added another entry point, ``create_initial_gpg_homedirs``, for
puppet to create the first gpg homedirs

.. _changed-73:

Changed
~~~~~~~

- default config filename is now ``scriptworker.json`` instead of
``config.json``
- moved ``scriptworker.config.get_context_from_cmdln`` out of
``scriptworker.worker.main``; now using argparse
- changed default ``sign_chain_of_trust`` to True
- ``scriptworker.gpg.sign_key``,
``scriptworker.gpg.rebuild_gpg_home_flat``, and
``scriptworker.gpg.rebuild_gpg_home_signed`` are now async, so they
can happen in parallel in the background
- renamed ``scriptworker.gpg.latest_signed_git_commit`` to
``scriptworker.gpg.verify_signed_git_commit_output``
- combined ``scriptworker.log.log_errors`` and
``scriptworker.log.read_stdout`` into
``scriptworker.log.pipe_to_log``
- added ``taskGroupId`` to the list of default valid ``taskId``\ s to
download from. This logic will need to change in version 0.9.0 due to
the new `chain of trust dependency traversal
logic <https://gist.github.com/escapewindow/a6a6944f51d4219d08284ededc65aa30>`__

.. _fixed-66:

Fixed
~~~~~

- added missing docstrings to the ``download_artifacts`` and
``download_file`` functions
- fixed coverage version in ``tox.ini py35-coveralls``
- ``sign_key`` now supports signing keys with multiple subkeys

.. _section-124:

--------------------

.. _added-76:

Added
~~~~~

- added ``DownloadError`` exception for ``download_file``
- added ``scriptworker.task.download_artifacts``
- added ``scriptworker.util.download_file``

.. _changed-74:

Changed
~~~~~~~

- ``DEFAULT_CONFIG``, ``STATUSES``, and ``REVERSED_STATUSES`` have
moved to ``scriptworker.constants``.
- ``list_to_tuple`` has been renamed ``freeze_values``, and also
converts dict values to frozendicts.

.. _section-125: