Prelude
tracing: This release adds support for lazy sampling, essentially moving when we make a sampling decision for a trace to the latest possible moment. These include the following: 1. Before encoding a trace chunk to be sent to the agent 2. Before making an outgoing request via HTTP, gRPC, or a DB call for any automatically instrumented integration 3. Before running `os.fork()` For most users this change shouldn't have any impact on their traces, but it does allow for more flexibility in sampling (see `features` release note). It should be noted that if a user has application egress points that are not automatically instrumented, to other Datadog components (downstream instrumented services, databases, or execution context changes), and rely on the Python tracer to make the sampling decision (don't have an upstream service doing this), they will need to manually run the sampler for those traces, or use `HttpPropagator.inject()`. For more information please see the following: <https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#distributed-tracing> <https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#tracing-context-management>
New Features
- DSM: Adds base64 format for encoding and decoding DSM context hash.
- botocore: adds dsm payload size stats for botocore messaging services of kinesis, sqs and sns.
- botocore: Adds support to the bedrock integration for tagging input and output messages.
- langchain: This introduces support for `langchain==0.1.0`. Note that this does not have tracing support for deprecated langchain operations. Please follow the langchain upgrade [guide](https://python.langchain.com/docs/changelog/core) or the langchain integration :ref: <span class="title-ref">docs\<langchain\></span> to enable full tracing support.
- dramatiq: Adds automatic tracing of the `dramatiq` library.
- tracing: Added support for lazy sampling, the benefit of which is the ability to make a sampling decision using `DD_TRACE_SAMPLING_RULES` based on any span attribute (service, resource, tags, name)regardless of when the value for the attribute is set. This change is particularly beneficial for sampling on tags, since the vast majority of tags are set after the span is created. Since sampling was previously done at span creation time, this meant that those tags could not be used for sampling decisions.
- openai: Adds support for tagging streamed responses for completion and chat completion endpoints.
- profiling: implement an experimental stack sampling feature, which can be enabled by setting `DD_PROFILING_STACK_V2_ENABLED=true`. This new sampler should resolve segfault issues on Python 3.11 and later, while also decreasing the latency contribution of the profiler in many situations, and also improving the accuracy of stack-sampling data. This feature is currently only available on Linux using CPython 3.8 or greater. Requires `DD_PROFILING_EXPORT_LIBDD_ENABLED=true` to be set.
- botocore: Changes botocore aws kinesis contrib to set DSM pathway using extracted DSM context, if found, instead of always using a new pathway with default context.
- kafka: Adds tracing and DSM support for `confluent_kafka.Consumer.consume()`. Previously only <span class="title-ref">confluent_kafka.Consumer.poll</span> was instrumented.
Deprecation Notes
- tracing: Deprecates support for `ddtrace.contrib.asyncio.AsyncioContextProvider`. ddtrace fully support tracing across asyncio tasks. Asyncio no longer requires additional configurations.
- tracing: `tracer.sampler` is deprecated and will be removed in the next major version release. To manually sample please call `tracer.sample` instead.
- gevent: Deprecates `ddtrace.contrib.gevent.provider.GeventContextProvider`. Drops support for <span class="title-ref">gevent\<20.12.0</span> and <span class="title-ref">greenlet\<1.0</span>.
Bug Fixes
- Vulnerability Management for Code-level (IAST): Some native exceptions were not being caught correctly by the python tracer. This fix remove those exceptions to avoid fatal error executions.
- otel: Ensures that the last datadog parent_id is added to w3c distributed tracing headers generated by the OpenTelemetry API.
- ASM: This fix resolves an issue where a valid user may trigger a failed login event.
- ASM: always clear the DDWaf context at the end of the span to avoid gc-induced latency spikes at the end of some requests.
- ASM: This fix resolves an issue where django login failure events may send wrong information of user existence.
- CI Visibility: fixes an issue where git author or committer names containing commas (eg: "Lastname, Firstname") would not work (and log an error) due to the use of comma as a separator.
- propagation: This fix resolves an issue where the sampling decision-maker tag in tracestate propagation headers was clobbered by a default value.
- datastreams: Changed DSM processor error logs to debug logs for a statement which is retried. If all retries fail, the stack trace is included
- internal telemetry: Ensures heartbeat events are sent at regular intervals even when no other events are being sent.
- Fix an incompatibility between the handling of namespace module imports and parts of the functionalities of the standard library importlib module.
- internal: This fix resolves an issue where importing the `ddtrace.appsec._iast._patches` module would fail raising an ImportError
- internal: This fix resolves an issue where importing the `ddtrace.internal.peer_service` module would fail raising an ImportError
- langchain: Ensures langchain vision APIs are correctly instrumented
- Fix for the declaration of dependencies for the package.
- internal: This fix resolves an issue where importing the `ddtrace.contrib.botocore.services` module would fail raising an ImportError
- profiling: handle unexpected stack data to prevent the profiler from stopping.
- starlette: Fix a bug that crashed background tasks started from functions without a <span class="title-ref">\_\_name\_\_</span> attribute
- ASM: This fix resolves an issue where the asgi middleware could crash with a RuntimeError "Unexpected message received".
- ASM: This fix resolves an issue with Flask instrumentation causing CPU leak with ASM, API Security and Telemetry enabled.
- Vulnerability Management for Code-level (IAST): Addresses an issue where the IAST native module was imported even though IAST was not enabled.
- Vulnerability Management for Code-level (IAST): This fix addresses an issue where tainting objects may fail due to context not being created in the current span.
- Vulnerability Management for Code-level (IAST): This fix addresses an issue where AST patching would generate code that fails to compile, thereby preventing the application from starting correctly.
- Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where other subscript operations than `Load` were being unintentionally patched, leading to compilation errors for the patched module.
- Vulnerability Management for Code-level (IAST): Fixes an issue where an atexit handler could lead to a segmentation fault.
- Vulnerability Management for Code-level (IAST): This fix addresses an issue where a vulnerability would be reported at line 0 if we couldn't extract the proper line number, whereas the default line number should be -1.
- kafka: This fix resolves an issue where `None` messages from confluent-kafka could cause crashes in the Kafka integration.
- appsec: This fix resolves an issue in which the library attempted to finalize twice a context object used by the Application Security Management product.
- tracing: Removes `allow_false` argument from ddtrace samplers. `allow_false` allows datadog samplers to return a value that differs from the sampling decision, this behavior is not supported.
- profiling: This fixes a `free(): invalid pointer` error which would arise as a result of incorrectly linking the C++ runtime.
- starlette: Ensures correct URL tag is set for starlette v0.34.0 and above.
- structlog: Fixes error where multiple loggers would duplicate processors. Also adds processors injection when resetting to defaults.
---