Ddtrace

Prelude

Initial library support has been added for Python 3.11.

<div class="note">

<div class="title">

Note

</div>

Continuous Profiler and Dynamic Instrumentation are not yet compatible and must be disabled in order to use the library with Python 3.11. Support for them will be added in a future release. To track the status, see the [Support Python 3.11](https://github.com/DataDog/dd-trace-py/issues/4149) issue on GitHub.

</div>

Upgrade Notes

- The default propagation style configuration changes to `DD_TRACE_PROPAGATION_STYLE=tracecontext,datadog`. To only support Datadog propagation and retain the existing default behavior, set `DD_TRACE_PROPAGATION_STYLE=datadog`.
- tracer: support for Datadog Agent v5 has been dropped. Datadog Agent v5 is no longer supported since ddtrace==1.0.0. See <https://ddtrace.readthedocs.io/en/v1.0.0/versioning.html#release-support> for the version support.
- Python 3.11: Continuous Profiler and Dynamic Instrumentation must be disabled as they do not current support Python 3.11.
- The configured styles in `DD_TRACE_PROPAGATION_STYLE_EXTRACT` are now evaluated in order to specification. To keep the previous fixed evaluation order, set: `DD_TRACE_PROPAGATION_STYLE_EXTRACT=datadog,b3,b3 single header`.
- tracing: upgrades the default trace API version to `v0.5` for non-Windows systems. The `v0.5` trace API version generates smaller payloads, thus increasing the throughput to the Datadog agent especially with larger traces.
- tracing: configuring the `v0.5` trace API version on Windows machines will raise a `RuntimeError` due to known compatibility issues. Please see <https://github.com/DataDog/dd-trace-py/issues/4829> for more details.

Deprecation Notes

- propagation: Configuration of propagation style with `DD_TRACE_PROPAGATION_STYLE=b3` is deprecated and will be removed in version 2.0.0. Please use the newly added `DD_TRACE_PROPAGATION_STYLE=b3multi` instead.
- aws: The boto, botocore and aiobotocore integrations no longer include all API parameters by default. To retain the deprecated behavior, set the environment variable `DD_AWS_TAG_ALL_PARAMS=1`. The deprecated behavior and environment variable will be removed in v2.0.0.

New Features

- django: add configuration option to allow a resource format like <span class="title-ref">{method} {handler}.{url_name}</span> in projects with Django \<2.2.0
- django: Adds the `DD_DJANGO_INCLUDE_USER_NAME` option to toggle whether the integration sets the `django.user.name` tag.
- Added environment variable `DD_TRACE_PROPAGATION_STYLE` to configure both injection and extraction propagation styles. The configured styles can be overridden with environment variables `DD_TRACE_PROPAGATION_STYLE_INJECT` and `DD_TRACE_PROPAGATION_STYLE_EXTRACT`.
- tracing: This introduces `none` as a supported propagator for trace context extraction and injection. When `none` is the only propagator listed, the corresponding trace context operation is disabled. If there are other propagators in the inject or extract list, the none propagator has no effect. For example `DD_TRACE_PROPAGATION_STYLE=none`
- ASM: now http.client_ip and network.client.ip will only be collected if ASM is enabled.
- tracing: Adds support for W3C Trace Context propagation style for distributed tracing. The `traceparent` and `tracestate` HTTP headers are enabled by default for all incoming and outgoing HTTP request headers. The Datadog propagation style continue to be enabled by default.
- flask: Adds support for streamed responses. Note that two additional spans: `flask.application` and `flask.response` will be generated.
- profiling: Adds support for Python 3.11.
- tracer: added support for Python 3.11.

Bug Fixes

- ASGI: response headers are correctly processed instead of ignored
- Fix issue with `attrs` and `contextlib2` version constraints for Python 2.7.
- CGroup file parsing was fixed to correctly parse container UUID for PCF containers.
- ASM: Do not raise exceptions when failing to parse XML request body.
- ASM: fix a body read problem on some corner case where don't passing the content length makes wsgi.input.read() blocks.
- aws: We are reducing the number of API parameters that the boto, botocore and aiobotocore integrations collect as span tags by default. This change limits span tags to a narrow set of parameters for specific AWS APIs using standard tag names. To opt out of the new default behavior and collect no API parameters, set the environment variable `DD_AWS_TAG_NO_PARAMS=1`. To retain the deprecated behavior and collect all API parameters, set the environment variable `DD_AWS_TAG_ALL_PARAMS=1`.
- tracing: make `ddtrace.context.Context` serializable which fixes distributed tracing across processes.
- django: avoid `SynchronousOnlyOperation` when failing to retrieve user information.
- Remove `forbiddenfruit` as dependency and rollback `wrapt` changes where `forbiddenfruit` was called. IAST: Patch builtins only when IAST is enabled.
- httpx: Fixes an incompatibility from `httpx==0.23.1` when the `URL.raw` property is not available.
- Fix error in patching functions. `forbiddenfruit` package has conflicts with some libraries such as `asynctest`. This conflict raises `AttributeError` exception. See issue \4484.
- tracer: This fix resolves an issue where the rate limiter used for span and trace sampling rules did not reset the time since last call properly if the rate limiter already had max tokens. This fix resets the time since last call always, which leads to more accurate rate limiting.
- Ensure that worker threads that run on start-up are recreated at the right time after fork on Python \< 3.7.
- tracing: This fix resolves an issue where the `DD_SERVICE_MAPPING` mapped service names were not used when updating span metadata with the `DD_VERSION` set version string.
- wsgi: This fix resolves an issue where `BaseException` raised in a WSGI application caused spans to not be submitted.
- library injection: Pin the library version in the library injection image. Prior, the latest version of `ddtrace` would always be installed, regardless of the image version.
- Fix error in the agent response payload when the user disabled ASM in a dashboard using 1-click Remote Configuration.
- flask: add support for flask v2.3. Remove deprecated usages of `flask._app_ctx_stack` and `flask._request_ctx_stack`.
- The specification of `DD_TRACE_PROPAGATION_STYLE_EXTRACT` now respects the configured styles evaluation order. The evaluation order had previously been fixed and so the configured order was ignored.
- tracing: Ensures that encoding errors due to wrong span tag types will be logged. Previously, if non-text span tags were set, this resulted in v0.5 encoding errors to be output to `stderr` instead of to a logger.

Other Changes

- Kubernetes library injection: run commands as non-root user.
- tracing: The value of `ddtrace.constants.PID` has been changed from `system.pid` to `process_id`. All spans will now use the metric tag of `process_id` instead.
- tracing: The exception logged for writing errors no longer includes a long, unhelpful stack trace. The message now also includes the number of traces dropped and the number of retries attempted.

---

Prelude

Application Security Management (ASM) has added support for preventing attacks by blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click activation for ASM is currently in beta.

</div>

Dynamic instrumentation allows instrumenting a running service dynamically to extract runtime information that could be useful for, e.g., debugging purposes, or to add extra metrics without having to make code changes and re-deploy the service. See <https://ddtrace.readthedocs.io/en/stable/configuration.html> for more details.

Upgrade Notes

- Pin \[attrs\](<https://pypi.org/project/attrs/>) dependency to version `>=20` due to incompatibility with \[cattrs\](<https://pypi.org/project/cattrs/>) version `22.1.0`.
- Use `Span.set_tag_str()` instead of `Span.set_tag()` when the tag value is a text type as a performance optimizations in manual instrumentation.

New Features

- ASM: add support for one click activation using Remote Configuration Management (RCM). Set `DD_REMOTE_CONFIGURATION_ENABLED=true` to enable this feature.
- ASM: ip address collection will be enabled if not explicitly disabled and appsec is enabled.
- tracing: HTTP query string tagged by default in http.url tag (sensitive query strings will be obfuscated).
- Django: set <span class="title-ref">usr.id</span> tag by default if <span class="title-ref">request.user</span> is authenticated.
- Introduced the public interface for the dynamic instrumentation service. See <https://ddtrace.readthedocs.io/en/stable/configuration.html> for more details.
- Add `Span.set_tag_str()` as an alternative to the overloaded functionality of `Span.set_tag()` when the value can be coerced to unicode text.
- Enable `telemetry <Instrumentation Telemetry>` collection when tracing is enabled.

Bug Fixes

- ASM: only report actor.ip on attack.
- aioredis: added exception handling for <span class="title-ref">CancelledError</span> in the aioredis integration.
- CI Visibility: fixed AppVeyor integration not extracting the full commit message.
- Add iterable methods on TracedCursor. Previously these were not present and would cause iterable usage of cursors in DB API integrations to fail.
- Fix parsing of the `DD_TAGS` environment variable value to include support for values with colons (e.g. URLs). Also fixed the parsing of invalid tags that begin with a space (e.g. `DD_TAGS=" key:val"` will now produce a tag with label `key`, instead of `key`, and value `val`).
- opentracing: don't raise an exception when distributed tracing headers are not present when attempting to extract.
- sqlite3: fix error when using `connection.backup` method.
- Change dependency from `` backport_ipaddress` to ``ipaddress`. Only install`ipaddress\`\` for Python \< 3.7.
- gevent: disable gevent after fork hook which could result in a performance regression.
- profiling: restart automatically on all Python versions.
- profiling: fixes an issue with Gunicorn child processes not storing profiling events.
- wsgi: when using more than one nested wsgi traced middleware in the same app ensure wsgi spans have the correct parenting.

Other Changes

- tracing: add http.route tag to root span for Flask framework.

---

New Features

- graphene: add support for `graphene>=2`. [See the graphql documentation](https://ddtrace.readthedocs.io/en/stable/integrations.html#graphql) for more information.
- Add support for aiobotocore 1.x and 2.x.
- ASM: add user information to traces.
- ASM: collect http client_ip.
- ASM: configure the sensitive data obfuscator.
- ASM: Detect attacks on Pylons body.
- ASM: propagate user id.
- ASM: Support In-App WAF metrics report.
- Collect user agent in normalized span tag `http.useragent`.
- ASM: Detect attacks on XML body (for Django, Pylons and Flask).
- Adds support for Lambda profiling, which can be enabled by starting the profiler outside of the handler (on cold start).
- profiler: collect and export the class name for the wall time, CPU time and lock profiles, when available.
- add DD_PYMONGO_SERVICE configuration
- ASM: Redact sensitive query strings if sent in http.url.
- redis: track the connection client_name.
- rediscluster: add service name configuration with `DD_REDISCLUSTER_SERVICE`
- snowflake: add snowflake query id tag to `sql.query` span

Bug Fixes

- aiohttp_jinja2: use `app_key` to look up templates.
- ASM: (flask) avoid json decode error while parsing request body.
- ASM: fix Python 2 error reading WAF rules.
- ASM: reset wsgi input after reading.
- tracing: fix handling of unicode `_dd.origin` tag for Python 2
- tracing: fix nested web frameworks re-extracting and activating HTTP context propagation headers.
- requests: fix split-by-domain service name when multiple `` signs are present in the url
- profiling: internal use of RLock needs to ensure original threading locks are used rather than gevent threading lock. Because of an indirection in the initialization of the original RLock, we end up getting an underlying gevent lock. We work around this behavior with gevent by creating a patched RLock for use internally.
- profiler: Remove lock for data structure linking threads to spans to avoid deadlocks with the trade-off of correctness of spans linked to threads by stack profiler at a given point in time.
- profiling: fix a possible deadlock due to spans being activated unexpectedly.

---

New Features

- graphql: add tracing for `graphql-core>2`. See [the graphql documentation](https://ddtrace.readthedocs.io/en/stable/integrations.html#graphql) for more information.
- ASM: Detect attacks on Django body.
- ASM: Detect attacks on Flask request cookies
- ASM: Detect attacks on Django request cookies
- ASM: Detect attacks on Pylons HTTP query.
- ASM: Detect attacks on Pylons request cookies
- ASM: detect attacks on Pylons path parameters.
- ASM: Report HTTP method on Pylons framework
- ASM: Collect raw uri for Pylons framework.
- AppSec: collect response headers
- ASM: Detect attacks on Flask body.
- ASM: Detect attacks on path parameters
- The profiler now supports Windows.
- The profiler now supports code provenance reporting. This can be enabled by using the `enable_code_provenance=True` argument to the profiler or by setting the environment variable `DD_PROFILING_ENABLE_CODE_PROVENANCE` to `true`.

Bug Fixes

- flask: add support for `flask>=2.2.0`
- Fixed the environment variable used for log file size bytes to be `DD_TRACE_LOG_FILE_SIZE_BYTES` as documented.
- jinja2: fix handling of template names which are not strings.
- Fixed support for pytest-bdd 6.
- Fixes cases where a pytest test parameter object string representation includes the `id()` of the object, causing the test fingerprint to constantly change across executions.
- wsgi: ignore GeneratorExit Exception in wsgi.response spans
- wsgi: ensures resource and http tags are always set on <span class="title-ref">wsgi.request</span> spans.

Other Changes

- profiler: don't initialize the `AsyncioLockCollector` unless asyncio is
available. This prevents noisy logs messages from being emitted in Python 2.

- docs: Added troubleshooting section for missing error details in the root span of a trace.

---

New Features

- internal: Add support for Datadog trace tag propagation
- django: added `DD_DJANGO_INSTRUMENT_TEMPLATES=false` to allow tracing of Django template rendering.
- internal: Add sampling mechanism trace tag
- Add environment variables to write `ddtrace` logs to a file with `DD_TRACE_LOG_FILE`, `DD_TRACE_LOG_FILE_LEVEL`, and `DD_TRACE_FILE_SIZE_BYTES`
- Adds pytest-bdd integration to show more details in CI Visibility product.

Bug Fixes

- starlette: Add back removed `aggregate_resources` feature.
- fastapi: Add back removed `aggregate_resources` feature.
- aiomysql: fix `AttributeError: __aenter__` when using cursors as context managers.
- asgi, starlette, fastapi: Exclude background tasks duration from web request spans.
- asgi: set the `http.url` tag using the hostname in the request header before defaulting to the hostname of the asgi server.
- mypy: Avoid parsing redis asyncio files when type checking Python 2
- starlette: Add back removed `ddtrace.contrib.starlette.get_resource` and `ddtrace.contrib.starlette.span_modifier`.
- fastapi: Add back removed `ddtrace.contrib.fastapi.span_modifier`.
- internal: fix exception raised for invalid values of `DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH`.
- flask_caching: fix redis tagging after the v2.0 release.
- redis: create default Pin on asyncio client. Not having a Pin was resulting in no traces being produced for the async redis client.

Other Changes

- perf: don't encode default parent_id value.
- profiling: add support for protobuf \>=4.0.

---

Upgrade Notes

- The profiler `asyncio_loop_policy` attribute has been renamed to `asyncio_loop_policy_class` to accept a user-defined class. This guarantees the same asyncio loop policy class can be used process children.

New Features

- Add tracing support for `aiomysql>=0.1.0`.

- Add support for `grpc.aio`.

- botocore: allow defining error status codes for specific API operations.

See our `botocore` document for more information on how to enable this feature.

- ciapp: detect code owners of PyTest tests

- The memory profile collector can now entirely disabled with the `DD_PROFILING_MEMORY_ENABLED` environment variable.

- psycopg2: add option to enable tracing `psycopg2.connect` method.

See our `psycopg2` documentation for more information.

- Add asyncio support of redis ≥ 4.2.0

Bug Fixes

- Fixes deprecation warning for `asyncio.coroutine` decorator.

- internal: normalize header names in ASM

- profiling: implement `__aenter__` and `__aexit__` methods on `asyncio.Lock` wrapper.

- tracing: fix issue with `ddtrace-run` having the wrong priority order of tracer host/port/url env variable configuration.

- django,redis: fix unicode decode error when using unicode cache key on Python 2.7

- fastapi/starlette: when using sub-apps, formerly a call to `/sub-app/hello/{name}` would give a resource name of `/sub-app`. Now the full path `/sub-app/hello/{name}` is used for the resource name.

- sanic: Don't send non-500s error traces.

- pin protobuf to version `>=3,<4` due to incompatibility with version `4.21`.

- Fixes a performance issue with the profiler when used in an asyncio application.

- The profiler now copy all user-provided attributes on fork.

- pytest: Add note for disabling ddtrace plugin as workaround for side-effects

- Set required header to indicate top level span computation is done in the client to the Datadog agent. This fixes an issue where spans were erroneously being marked as top level when partial flushing or in certain asynchronous applications.

The impact of this bug is the unintended computation of stats for non-top level spans.

Other Changes

- The default number of events kept by the profiler has been reduced to decreased CPU and memory overhead.

---