Ddtrace

New Features

- tracing: Adds support for 128 bit trace ids for b3 and w3c distributing tracing headers.
- pytest: Adds the `DD_CIVISIBILITY_AGENTLESS_ENABLED` environment variable to configure the `CIVisibility` service to use an agent-less test reporting `CIVisibilityWriter`. Note that the `CIVisibility` service will use regular agent reporting by default.
- sci: Extracts and sends git metadata from environment variables `DD_GIT_REPOSITORY_URL`, `DD_GIT_COMMIT_SHA`, or from the python package specified in the `DD_MAIN_PACKAGE`. This feature can be disabled by setting `DD_TRACE_GIT_METADATA_ENABLED=False`.
- otel: Adds support for the [OpenTelemetry Tracing API](https://opentelemetry.io/docs/reference/specification/trace/api/). Please refer to the `docs <ddtrace.opentelemetry>` for more details.

Bug Fixes

- tracing: Ensure datadog headers propagate 128 bit trace ids when `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=False`
- aws_lambda: Fix AttributeError raised when `ddtrace.patch_all()`, or `ddtrace.patch(aws_lambda=True)`, is set on user handler.
- aws_lambda: Fix AttributeError raised when extracting context from arguments.
- aws_lambda: Fix AttributeError raised when callable handlers are traced.
- dynamic instrumentation: Fixed an issue with expressions in metric probes that prevented them from being evaluated.
- Prevent exceptions when autoreloading modules that directly or indirectly import ddtrace with the iPython autoreload extension.
- profiling: Corrects accounting of wall and CPU time for gevent tasks within the main Python thread.
- profiling: Fixed an issue with the memory collector where a segmentation fault could occur during shutdown.
- lib-injection: The ddtrace package is now provided via the Docker image rather than relying on a run-time `pip install`. This solves issues like containers blocking network requests, installation overhead during application startup, permissions issues with the install.

---

Deprecation Notes

- ASM: Several deprecated ASM constants that were added to the public API will be removed. This should not affect existing code as they were meant for private use only.

New Features

- tracing: Adds support for 128 bit trace ids. To generate and propagate 128 bit trace ids using Datadog distributed tracing headers set the following configuration: `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=True`. Support for B3 and W3C distributed tracing headers will be added in a future change.
- aiohttp: Add missing component meta tag to aiohttp server spans.
- redis: Adds tracing support for <span class="title-ref">redis.cluster.RedisCluster</span>.
- celery: Adds automatic tracing of the `celery.beat` scheduling service to the `celery` integration.
- kafka: Adds instrumentation support for `confluent-kafka>=1.7`. See the `confluent-kafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#kafka>` documentation for more information.
- dynamic instrumentation: introduced support for dynamic span probes.
- Adds source code integration with setuptools build metadata. This enables traces and profiles to be automatically tagged with git metadata to track deployments in Datadog.

Bug Fixes

- tracing: This fix resolves an issue where making a sampling decision before the `env` span tag had been set caused sample rate data from the Datadog Agent to be ignored.
- ASM: make `track_custom_event()` also set `appsec.events.<custom_event>.track` which was missing.
- django: Fixes an issue where `http.route` was only set if `use_handler_resource_format` and `use_legacy_resource_format` were set to `False`.
- tracing: This fix resolves an issue where a very long string as a span attribute would cause that span not to be delivered. It replaces string span attributes larger than DD_TRACE_WRITER_BUFFER_SIZE_BYTES (which as of this version defaults to 8388608) with a small string containing debug information and not containing any of the original attribute string.
- ASM: Resolves installation issues with compiling native code on Windows and unknown platforms.
- aws_lambda: Fixes a `RecursionError` which is raised when aws lambda signal handlers are wrapped infinitely. This caused lambdas to crash on startup.
- botocore: Fix TypeError raised by injecting trace context into Kinesis messages.
- dynamic instrumentation: Fix a bug where the dynamic instrumentation would stop injecting function probes after the first failed one.
- dynamic instrumentation: This change fixes a bug whereby probes that have been disabled/removed from the front-end would not be removed by the client library.
- futures: Resolves an issue that prevents tasks from being submitted to a thread pool executor when gevent is used (e.g. as a worker class for gunicorn or celery).
- propagation: This fix resolves an issue where previously W3C tracestate propagation could not handle whitespace. With this fix whitespace is now removed for incoming and outgoing requests.
- httplib: Fixes an issue with patching of http client upon import
- Ensure DD_REMOTE_CONFIGURATION_ENABLED environment variable disables remote config if set to False

Other Changes

- aws_lambda: Updates how <span class="title-ref">DD_APM_FLUSH_DEADLINE_MILLISECONDS</span> is used. Previously, we would set the deadline as the environment variable value, if set. Now, when the remaining time in an AWS Lambda invocation is less than <span class="title-ref">DD_APM_FLUSH_DEADLINE_MILLISECONDS</span>, the tracer will attempt to submit the current active spans and all finished spans. the value in the environment variable is used to subtract from the deadline. The default is still 100ms.

---

Prelude

Application Security Management (ASM) has added Django support for blocking malicious users using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Deprecation Notes

- dbapi: `ddtrace.ext.mongo.ROWS` is deprecated. Use `ddtrace.ext.db.ROWCOUNT` instead.

New Features

- starlette: Add http.route tag to `starlette.request` spans.
- fastapi: Add http.route tag to `fastapi.request` spans.
- ASM: Add support for one click blocking of user ids with the Django framework using Remote Configuration Management.
- ASM: This introduces the "suspicious request blocking" feature for Django and Flask.

Known Issues

- ASM: There is a known issue with the flask support for any rule blocking on `server.request.path_params`. The request will be correctly blocked but the client application will be receiving and processing the suspicious request. Possible workaround: use `server.request.uri.raw` instead, if you want the request to be blocked before entering the flask application.

Bug Fixes

- dbapi: The dbapi integration no longer assumes that a cursor object will have a rowcount as not all database drivers implement rowcount.

- dbm: Support sql queries with the type `byte`.

- elasticsearch: Omit large `elasticsearch.body` tag values that are
greater than 25000 characters to prevent traces from being too large to send.

- aws_lambda: This fix resolves an issue where existing signals were wrapped multiple times.

- profiling: Handles a race condition on process shutdown that would cause an error about a module not being started to occasionally appear in the logs.

- Fix for KeyError exceptions when when <span class="title-ref">ASM_FEATURES</span> (1-click activation) disabled all ASM products. This could cause 1-click activation to work incorrectly in some cases.
- ASM: Solve some corner cases where a Flask blocking request would fail because headers would be already sent.
- ASM: Solve the content-type not always being correct in blocking responses.
- ASM: Ensure the blocking responses have the following tags: <span class="title-ref">http.url</span>, <span class="title-ref">http.query_string</span>, <span class="title-ref">http.useragent</span>, <span class="title-ref">http.method</span>, <span class="title-ref">http.response.headers.content-type</span> and <span class="title-ref">http.response.headers.content-length</span>.
- ASM: fix memory leaks and memory corruption in the interface between ASM and the WAF library
- psycopg2: Fixes a bug with DSN parsing integration.

Other Changes

- remote_config: Change the level of remote config startup logs to debug.

---

Deprecation Notes

- gevent: `DD_GEVENT_PATCH_ALL` is deprecated and will be removed in the next major version. Gevent compatibility is now automatic and does not require extra configuration when running with `ddtrace-run`. If not using `ddtrace-run`, please import `ddtrace.auto` before calling `gevent.monkey.patch_all()`.

Bug Fixes

- aws_lambda: Resolves an exception not being handled, which occurs when no root span is found before a lambda times out.
- gevent: This fix resolves an incompatibility between ddtrace and gevent that caused threads to hang in certain configurations, for example the profiler running in a gunicorn application's gevent worker process.

Other Changes

- ASM: The list of headers for retrieving the IP when Application Security Management is enabled or the
<span class="title-ref">DD_TRACE_CLIENT_IP_ENABLED</span> environment variable is set has been updated. "Via" has been removed as it rarely contains IP data and some common vendor headers have been added. You can also set the environment variable <span class="title-ref">DD_TRACE_CLIENT_IP_HEADER</span> to always retrieve the IP from the header specified as the value.

---

Prelude

Application Security Management (ASM) has added Django support for blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Application Security Management (ASM) has added Flask support for blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Deprecation Notes

- grpc: Deprecates `ddtrace.contrib.grpc.constants.GRPC_PORT_KEY`. Use `ddtrace.ext.net.TARGET_PORT` instead.
- dbapi: `ddtrace.ext.sql.ROWS` is deprecated. Use `ddtrace.ext.db.ROWCOUNT` instead.
- cassandra: `ddtrace.ext.cassandra.ROW_COUNT` is deprecated. Use `ddtrace.ext.db.ROWCOUNT` instead.

New Features

- Enable traces to be sent before an impending timeout for `datadog_lambda>=4.66.0`. Use `DD_APM_FLUSH_DEADLINE` to override the default flush deadline. The default is the AWS Lambda function configured timeout limit.

- debugger: Add dynamic log probes to that generate a log message and optionally capture local variables, return value and exceptions

- tracing: Add support for enabling collecting of HTTP request client IP addresses as the `http.client_ip` span tag. You can set the `DD_TRACE_CLIENT_IP_ENABLED` environment variable to `true` to enable. This feature is disabled by default.

- ASM: add support for one click blocking of IPs with the Django framework using Remote Configuration Management.

- ASM: add support for one click blocking of IPs with the Flask framework using
Remote Configuration Management.

- ASM: also fetch loopback IPs if client IP fetching is enabled (either via ASM or DD_TRACE_CLIENT_IP_ENABLED).

- ASM: Enable ability to remotely activate and configure ASM features. To enable, check the Python Security page in your account. Note that this is a beta feature.

- profiling: Collects endpoint invocation counts.

- dynamic instrumentation: Python 3.11 is now supported.

- graphene: Adds support for Python 3.11.

- graphql: Adds support for Python 3.11.

- httpx: Add support for `httpx<0.14.0,>=0.9.0`.

- tracer/span: Add `Span.finish_with_ancestors` method to enable the abrupt
finishing of a trace in cases where the trace or application must be immediately terminated.

Known Issues

- remote config: There is a known issue with remote configuration management (RCM) when paired with gevent which can cause child processes to deadlock. If you are experiencing issues, we recommend disabling RCM with `DD_REMOTE_CONFIGURATION_ENABLED=false`. Note, this will disable one click activation for ASM.
- gunicorn: ddtrace-run does not work with gunicorn. To instrument a gunicorn application, follow the instructions [here](https://ddtrace.readthedocs.io/en/latest/integrations.html#gunicorn).

Bug Fixes

- fastapi: Previously, custom fastapi middlewares configured after application startup were not traced. This fix ensures that all fastapi middlewares are captured in the <span class="title-ref">fastapi.request</span> span.

- tracing: Pads trace_id and span_ids in b3 headers to have a minimum length of 16.

- Fix full stacktrace being sent to the log on remote config connection errors.

- httpx: Only patch `httpx.AsyncClient` for `httpx>=0.11.0`.

- tracing: This fix resolves an issue with the encoding of traces when using the v0.5 API version with the Python optimization option flag `-O` or the `PYTHONOPTIMIZE` environment variable.

- pylons: This fix resolves an issue where `str.decode` could cause critical unicode decode errors when ASM is enabled. ASM is disabled by default.

- gevent: This fix resolves incompatibility under 3.8\>=Python\<=3.10 between `ddtrace-run` and applications that depend on `gevent`, for example `gunicorn` servers. It accomplishes this by keeping copies that have not been monkey patched by `gevent` of most modules used by `ddtrace`. This "module cloning" logic can be controlled by the environment variable `DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE`. Valid values for this variable are "1", "0", and "auto". "1" tells `ddtrace` to run its module cloning logic unconditionally, "0" tells it never to run that logic, and "auto" tells it to run module cloning logic *only if* `gevent` is accessible from the application's runtime. The default value is "0".

- lib-injection: Use package versions published to PyPI to install the
library. Formerly the published image was installing the package from source using the tagged commit SHA which resulted in slow and potentially failing installs.

- profiler: Handles potential `AttributeErrors` which would arise while collecting frames during stack unwinding in Python 3.11.

- remote config: ensure proper validation of responses from the agent.

---

Upgrade Notes

- ASM: libddwaf upgraded to version 1.6.1 using a new library loading mechanism
- profiling: upgrades the profiler to support the `v2.4` backend API for profile uploads, using a new request format.

Deprecation Notes

- `DD_REMOTECONFIG_POLL_SECONDS` environment variable is deprecated and will be removed in v2.0. Please use `DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS` instead.

New Features

- CI Visibility: Add support for CI provider buddy.works

- The component tag has been added for all auto-instrumented spans. The value of the component tag is equal to the name of the integration that produced the span.

- tracing: Adds support for IPv6 agent hostnames for <span class="title-ref">DD_AGENT_HOST</span>.

- elasticsearch: Update `elasticsearch` integration to add support for `opensearch-py`. See [the elasticsearch documentation](https://ddtrace.readthedocs.io/en/stable/integrations.html#elasticsearch) for more information.

- ASM: one click activation enabled by default using Remote Configuration Management (RCM). Set `DD_REMOTE_CONFIGURATION_ENABLED=false` to disable this feature.

- ASM: New Application Security Events Tracking API, starting with the functions `track_user_login_success_event` and
`track_user_login_failure_event` for tracking user logins (it will also internally call `set_user`) and `track_custom_event` for any custom events. You can find these functions in the `ddtrace.appsec.trace_utils` module. Calling these functions will create new tags under the `appsec.events` namespace (`appsec.events.user.login` for logins) allowing you to track these events with Datadog. In the future this will be used to provide protection against account takeover attacks (ATO). Public documentation will be online soon.

- celery: Enhances context tags containing dictionaries so that their contents are sent as individual tags (issue \4771).

- tornado: Support custom error codes: <https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#custom-error-codes>.

- CI Visibility: Support reliably linking tests to the pipeline that executed them.

Known Issues

- profiling: There is currently a known performance regression issue with the profiler's code provenance feature. Note that this feature is disabled by default and will only be enabled if `DD_PROFILING_ENABLE_CODE_PROVENANCE` is set to true.

Bug Fixes

- This fix improves a cryptic error message encountered during some `pip install ddtrace` runs under pip versions \<18.
- dynamic instrumentation: remove unnecessary log line from application start up
- This fix removes unintended url parts in the `http.url` tag.
- botocore: Before this change, the botocore integration stripped newlines from the JSON string encoded in the data blob of Amazon Kinesis records. This change includes a terminating newline if it is present in the decoded data.
- profiling: This fix resolves an issue in Python 3.11 where a PyFrameObject strong reference count was not properly decremented in the stack collector.
- telemetry: This fix resolves an issue when we try to fetch `platform.libc_ver()` on an unsupported system.
- Fix for ValueError when `` is not present in network location but other part of the url.

Other Changes

- profiler: CPU overhead reduction.

---