Baseplate

* Fix local span metrics observers registering child observers (592)

Changes

* Allow maximum mqueue size to be configured in trace and event publisher sidecars (553)

Bug Fixes

* Fix swapped limits on trace messages sent to trace publisher sidecars, this would have artificially limited the size of spans (537)

Changes

* Add pool_size and max_overflow to SQLAlchemy config parser (533)
* Configurables that take UNIX usernames/groups will now except numeric IDs as well. (535)

New feature: first fakes for testing

This release adds the first two fakes (of many to come) for your service to use in its test suite. The new fakes are [`FakeSecretsStore`](https://baseplate.readthedocs.io/en/v1.5.0/api/baseplate/lib/secrets.html#testing) and [`FakeFileWatcher`](https://baseplate.readthedocs.io/en/v1.5.0/api/baseplate/lib/file_watcher.html#testing). These fakes act like the real object but have no external dependencies and are easier to set up. This means you can test components using these classes without mocking, building your own fakes, or setting up costly external dependencies.

Here's an example of the fake secrets store in use:

python
>>> secrets = FakeSecretsStore({
... "secrets": {
... "secret/foo/bar": {
... "type": "versioned",
... "current": "hunter2",
... },
... },
... })
>>> secrets.get_versioned("secret/foo/bar")
VersionedSecret(previous=None, current=b'hunter2', next=None)


If you have commonly mocked/faked objects in your test suites you'd like to see included here please let us know in [the issue tracker](https://github.com/reddit/baseplate.py/issues)!

See 512 for more background and example uses.

Changes

- redis client: Add connection pool runtime metrics (494)
- redis client: Unlock `pubsub()` (494)
- experiments: Change `expose()`'s `name` parameter to `experiment_name` for consistency with `variant()` (492)
- kombu consumer: Add `worker_kwargs` to `KombuQueueConsumerFactory.new()` for passing parameters to the consumer (498)
- metrics: Don't emit "Would send metric" log messages unless explicitly enabled (508)
- Add appropriate `User-Agent` header to HTTP requests made by sidecars (516)
- sentry: Don't report "operational" errors which usually indicate transient conditions like connection timeouts (517)
- Add PID to startup message and log on shutdown as well (523)
- Make live data fetcher able to create parent directories (526)
- Allow trace sidecar to be configured via environment variables (527)
- experiments: Cache experiment configuration objects from request to request (509)
- edge context: Cache parsed public keys for authentication token validation (512)

Bug fixes

- pyramid: Keep server span open while streaming responses (489)
- Fix JSON logging of trace ID (493)
- requests client: Properly handle keyword arguments meant for `send()` (496, 530)
- kafka consumer: Don't crash on null Kafka events (497)
- baseplate-shell: Start, don't embed, IPython to fix handling of globals (507)
- queue_consumer: Start runtime monitor (514)
- queue_consumer: Handle `ServerTimeout` properly (515)

Upgrading

No special changes are required to upgrade to this version. As always, you can use [the baseplate.py upgrader](https://github.com/reddit/baseplate.py-upgrader/) to handle the drudgework.

New Features

Tagged statsd metrics

Baseplate.py now includes an optional new metrics observer that emits
InfluxStatsD tagged metrics. This makes metric paths more discoverable and
allows for much cleaner metric queries. Additionally, you can change the
default list of tags to further slice your metrics as needed in your
application.

See upgrade instructions below and 452 for history

JSON Structured Logging

The default format of log entries emitted by Baseplate.py applications is now
JSON. This allows a variety of metadata to be included in log entries without
cluttering up the main message.

[See the logging observer documentation][logging observer] for full details of the available
fields.

[logging observer]: https://baseplate.readthedocs.io/en/v1.4.0/api/baseplate/observers/logging.html

See upgrade instructions below and 430 and 466 for history

New healthcheck protocol

Baseplate.py now supports different kinds of health checks. Currently
supported healthcheck probe types are: readiness, liveness, and startup. This
can be used to fine tune when the infrastructure should route traffic to your
application and when it should restart an unresponsive application.

See upgrade instructions below and 460 and 468 for history

Instrumented HTTP client

Requests is a library for making HTTP requests. Baseplate.py now provides two
wrappers for Requests: the "external" client is suitable for communication with
third party, potentially untrusted, services; the "internal" client is suitable
for talking to first-party services and automatically includes trace and edge
context data in requests. Baseplate.py uses [Advocate] to prevent the external
client from talking to internal services and vice versa.

See [`baseplate.clients.requests` documentation] and 465 and 479 for history

[Advocate]: https://github.com/JordanMilne/Advocate
[`baseplate.clients.requests` documentation]: https://baseplate.readthedocs.io/en/v1.4.0/api/baseplate/clients/requests.html

Changes

* The [KombuProducer] client now passes a `secrets` parameter through to
`connection_from_config` (451)
* Baseplate.thrift now contains a standardized typedef for timestamp fields.
Use this in your services to communicate the timestamp format expected. (458)
* A new convenience function
[`baseplate.lib.datetime.epoch_milliseconds_to_datetime`][emtd] parses
milliseconds since the UNIX epoch into a datetime object (461).
* When using a pre-release of Thrift v0.14, the Thrift connection pool will now
test if the connection was closed by the remote side before returning a
connection to the application for use. This should reduce the frequency of
spurious RPC failures. (also in 1.3.5, 464)
* Many additions and changes to the documentation. Check out the new [Startup
Sequence] and [FAQ] pages. (467, 469, 471, 473)
* Unblock use of `lock()` on redis client wrapper (480)

[KombuProducer]: https://baseplate.readthedocs.io/en/v1.4.0/api/baseplate/clients/kombu.html#baseplate.clients.kombu.KombuProducer
[emtd]: https://baseplate.readthedocs.io/en/v1.4.0/api/baseplate/lib/datetime.html#baseplate.lib.datetime.epoch_milliseconds_to_datetime
[Startup Sequence]: https://baseplate.readthedocs.io/en/v1.4.0/guide/startup.html
[FAQ]: https://baseplate.readthedocs.io/en/v1.4.0/guide/faq.html

Bug fixes

* Fix bug where thrift pool gains an extra slot after the retry policy is
exhausted while trying to connect (also in 1.3.1, 447)
* The event publisher sidecar will no longer crash when events are rejected
because of validation errors (also in 1.3.1, 450)
* Fix extra connection slots being returned to thrift connection pool when
server timeout occurs while waiting for a slot (also in 1.3.2, 431)
* Properly handle base64-encoded edge context headers in Pyramid services (also
in 1.3.2, 449, 456)
* Fix breaking behavior introduced in 1.2 around server timeouts. Rather than
defaulting to a 10 second timeout if not specified in config, timeouts are
defaulted to off and a deprecation warning is emitted. Please do [configure
timeouts] for your service as they're an important mitigation for overloads
during production incidents. (also in 1.3.3, 446)

[configure timeouts]: https://baseplate.readthedocs.io/en/v1.4.0/api/baseplate/observers/timeout.html

Upgrading

This release should not contain any breaking changes but following are notes on
how to upgrade to take advantage of the new features.

Tagged statsd metrics

**For reddit employees, search "Baseplate Tags Guide (v1.4+)" in Confluence to
get a comprehensive upgrade guide or come to baseplate-tags-support in Slack.**

In your application's configuration file, remove `metrics.namespace`
configuration and add `metrics.tagging = true`. See [the tagged metrics
observer documentation][taggedmetrics] for full information.

[taggedmetrics]: https://baseplate.readthedocs.io/en/v1.4.0/api/baseplate/observers/tagged_statsd.html

JSON Structured Logging

By default, no changes should be necessary to get JSON formatted logs. However,
if your service is overriding the default [`logging.Formatter`] (e.g. [by
configuring it in your INI file][logging.config]) you may need to amend your
configuration. See [python-json-logger's documentation] for more information. Note that
Baseplate.py supplies a custom formatter subclass that you can find at `baseplate.lib.log_formatter.CustomJsonFormatter`.

[`logging.Formatter`]: https://docs.python.org/3/library/logging.html#formatter-objects
[logging.config]: https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig
[python-json-logger's documentation]: https://github.com/madzak/python-json-logger#using-a-config-file

New healthcheck protocol

To support the new healthcheck, you'll need to make a few small changes:

In your Thrift IDL, change the base your service extends to `BaseplateServiceV2`:

diff
--- a/myservice/myservice.thrift
+++ b/myservice/myservice.thrift

-service MyService extends baseplate.BaseplateService {
+service MyService extends baseplate.BaseplateServiceV2 {


Then add the new `request` parameter to your service's `is_healthy` method and
use its `probe` field:

diff
--- a/myservice/__init__.py
+++ b/myservice/__init__.py
-6,12 +6,15 import re
from typing import Dict
from typing import List
+from typing import Optional

from baseplate import Baseplate
from baseplate import RequestContext
from baseplate.clients.redis import RedisClient
from baseplate.frameworks.thrift import baseplateify_processor
from baseplate.lib import config
+from baseplate.thrift.ttypes import IsHealthyProbe
+from baseplate.thrift.ttypes import IsHealthyRequest
from thrift.Thrift import TProcessor

from .myservice_thrift import MyService
-45,8 +48,13 class Handler:
- def is_healthy(self, context: RequestContext) -> bool:
- return context.redis.ping()
+ def is_healthy(
+ self, context: RequestContext, request: Optional[IsHealthyRequest] = None
+ ) -> bool:
+ if request.probe == IsHealthyProbe.LIVENESS:
+ return True
+ else: treat all other probe types as READINESS
+ return context.redis.ping()