Important News
This release is a big leap into the future for Baseplate.py. We're dropping a lot of old baggage, taking advantage of new language features, and reorganizing for clarity.
* Python 3.6 and 3.7 are the only supported versions. Older versions will not work.
* All interfaces in the library are fully type annotated for use with [mypy](http://mypy-lang.org/) etc.
* The module structure has been reorganized to clarify the structure of the library.
If your service is on an older version of Python, you will need to upgrade to 3.6 or 3.7 first before upgrading Baseplate.py. Python 2.7 goes End of Life at the end of 2019 and 3.4 already went End of Life earlier this year. For services stuck on 2.7 in the meantime, we will continue to backport bugfixes and critical improvements to the 0.30.x series.
Because of these breaking changes, we've built a [tool you can use to automate most of the upgrade process](https://github.com/reddit/baseplate.py-upgrader/). See below for full upgrade instructions.
New Features
Simplified application wire-up
Two new functions on the main `Baseplate` object help simplify application wire-up. [`configure_observers`](https://baseplate.readthedocs.io/en/v1.0.0/api/baseplate/index.html#baseplate.Baseplate.configure_observers) sets up all the observers your application needs based on its config file in one quick call. [`configure_context`](https://baseplate.readthedocs.io/en/v1.0.0/api/baseplate/index.html#baseplate.Baseplate.configure_context) helps you set up all the clients and configuration variables your application needs in one place without having to deal with intermediate objects.
Before:
python
def make_processor(app_config):
baseplate = Baseplate()
baseplate.configure_logging()
metrics_client = metrics_client_from_config(app_config)
baseplate.configure_metrics(metrics_client)
redis = redis_pool_from_config(app_config)
baseplate.add_to_context("redis", redis)
...
After:
python
def make_processor(app_config):
baseplate = Baseplate()
baseplate.configure_observers(app_config)
baseplate.configure_context(app_config, {"redis": RedisClient()})
You can continue to use `add_to_context` with `configure_context`, they are compatible. Likewise, all the older `configure_{metrics,tracing,etc.}` functions are still available.
Datetime Helper Functions
There's now a new group of helper functions for working with `datetime` objects. The new functions are:
* `datetime_to_epoch_milliseconds`
* `datetime_to_epoch_seconds`
* `epoch_seconds_to_datetime`
* `get_utc_now`
See [the documentation](https://baseplate.readthedocs.io/en/v1.0.0/api/baseplate/lib/datetime.html) for more details.
Thanks to [mimichen226](https://github.com/mimichen226) for [implementation](https://github.com/reddit/baseplate.py/commit/f2656b5d19712a5055267e924b63841ebd83f683)!
Custom arguments to `baseplate-script`
The entry point called by `baseplate-script` can now take a second parameter in addition to the config dictionary. This parameter is a list of strings representing the command line arguments `baseplate-script` didn't itself understand. Your script can then use these arguments as it sees fit.
python
def run(app_config, args):
parser = argparse.ArgumentParser()
parser.add_argument("name")
args = parser.parse_args(args)
print(args.name)
console
$ baseplate-script printer.ini printer:run Hello
Hello
$ baseplate-script printer.ini printer:run Goodbye.
Goodbye.
See [the docs for `baseplate-script`](https://baseplate.readthedocs.io/en/v1.0.0/cli/script.html).
Thanks to [bradengroom](https://github.com/bradengroom) for [implementation](https://github.com/reddit/baseplate.py/commit/d79ddda5a566019055dd537e80822c932c911d9f)!
Reference Cycle Finder
Python generally uses reference counts to manage the lifetime of objects in memory. When objects reference eachother, they form a cycle that will live forever. Python also has a garbage collector to find and dispose of these cycles. The collector is bad for latency because it will occasionally pause your application. To reduce the pressure on the collector, it's important to find and root out reference cycles in your applications.
Baseplate.py now has a tool built in that uses the `objgraph` library to help you find reference cycles building up in your application. Make sure `objgraph` is installed (`pip install objgraph`), flag the feature on by adding `monitoring.gc.refcycle = /path/to/output/directory` to the server section of your config file, and then watch images pop out like this:
Places where arrows are pointing back up the tree are reference cycles. Fixing these will improve the performance of the Python GC.
This feature should only be used under supervision as it will surely impact response times itself. See [the documentation on `baseplate-serve` for more information](https://baseplate.readthedocs.io/en/v1.0.0/cli/serve.html#process-level-metrics).
Lazy-populated context object
Setting up all the configured attributes on the context object each request is now done lazily. This means that you don't pay the cost of having something attached to the context object unless you actually use it in that request. Depending on your application, this can have significant performance gains.
There's very little performance downside to this, so it should generally be win-win.
https://github.com/reddit/baseplate.py/pull/314
Changes
* **BREAKING:** Python 3.6 is now the minimum version supported.
* **BREAKING:** Many modules have been reorganized to clarify the structure of the library. Use the upgrade tool to automatically fix your imports.
* **BREAKING:** The Python 2 compatibility shim `baseplate.crypto.constant_time_compare` has been removed. Use `hmac.compare_digest` from the Python standard library instead. (This is handled automatically by the upgrader tool.)
* **BREAKING:** Support for v1 events has been removed. Migrate to v2 events.
* Support for `B3-` prefixed trace headers has been removed. We weren't using this so it should be transparent.
* Error messages from the secret fetcher daemon now tell you which secret caused an issue ([fff83b8](https://github.com/reddit/baseplate.py/commit/fff83b82fc56508ded73e9a483b48d14c359db09)).
* The SQLAlchemy and Thrift Pool clients now report connection pool statistics ([c7045c9](https://github.com/reddit/baseplate.py/commit/c7045c905a0ec2249a07e352f8dbceaf5ccd4303)).
* Setting the debug flag on a trace span now forces tracing regardless of upstream sampling. (Also backported to 0.30.5)
* Fixed various Python 3 deprecation warnings generated by Baseplate.
* Added `extras_require` with version specifications for the various integration libraries. You can install baseplate like `pip install baseplate[sqlalchemy]` to enforce these restrictions. See [setup.py](https://github.com/reddit/baseplate.py/blob/master/setup.py) for the full list.
* The documentation was refreshed and a new tutorial was added to help with initial learning.
* Code was modernized to Python 3.6+ and standardized linting was applied across the board.
* Several improvements to log output to reduce noise. ([800f44](https://github.com/reddit/baseplate.py/commit/800f448d2f1a96e890ec33e8c239ca7f97075b0a), [9b6176f](https://github.com/reddit/baseplate.py/commit/9b6176f11f975a40790d968afc9a36430b76ac69)).
* Counters can now be incremented via the span interface by calling `context.trace.incr_tag()`. This uses the observer system to be agnostic to what observers are available and makes your application more future proof. ([344](https://github.com/reddit/baseplate.py/pull/344))
Bug Fixes
* When manually calling `flush()` on a metrics batch, counters are now properly reset. ([500e9e7](https://github.com/reddit/baseplate.py/commit/500e9e7c0f5d297013e56211ad83c340de1e248c)).
* The `KombuConsumer` now has a max size on its internal queue to prevent runaway memory growth when messages come in faster than they can be processed ([9e38a078](https://github.com/reddit/baseplate.py/commit/9e38a078f90c28f586d94e69fa2c86acad3b9108)—also backported to 0.30.1).
* Fix a couple of bugs in memcached serialization helpers ([21fa734](https://github.com/reddit/baseplate.py/commit/21fa7341ec41cd0f534200e2e03ef09039de4109) and [4933b2](https://github.com/reddit/baseplate.py/commit/4933b2322d56372bbbd092ca8292b71d854a33c5#diff-f3758d3ef79fe4ad271efcf71d86a5e5L115)—also backported to 0.30.4).
* Some reference cycles causing unnecessary GC pressure within Baseplate itself were fixed ([76fd5c88](https://github.com/reddit/baseplate.py/commit/76fd5c88bacbd456a228c4c2743923f6936d3fc2)—also backported to 0.30.3).
* Graceful shutdown on SIGUSR2 (Einhorn) now works as intended and is implemented for SIGTERM (Kubernetes). ([a4144df](https://github.com/reddit/baseplate.py/commit/a4144dfb8b772b3d477da614eeb4145e1ee81273)—also backported to 0.30.1)
* The metrics client displays the "batch too big" message only when an error actually indicates the message was too big. ([5810d2](https://github.com/reddit/baseplate.py/commit/5810d2b65d8b09805729e6c173506a0fe23bc737)).
Upgrading
Because this version has many breaking mechanical changes, we've built an [upgrader tool](https://github.com/reddit/baseplate.py-upgrader/) that will automatically refactor your code as best as it can to get you ready for the new version. To use it, make sure you have Python 3.7 installed then install the tool:
pip3.7 install git+ssh://gitgithub.com/reddit/baseplate.py-upgrader
and run it on your service:
console
$ baseplate.py-upgrader .
Baseplate.py Upgrader
Upgrading .