Ddtrace

Upgrade Notes

- <div id="remove-unsupported-pylons">

This removes the `pylons` integration, which does not support Python 3.

</div>

Deprecation Notes

- aioredis: The aioredis integration is deprecated and will be removed in a future version. As an alternative to the aioredis integration, you can use the redis integration with redis\>=4.2.0.

New Features

- ASM: dependency telemetry metrics now will only report dependencies actually in use (imported) and will also report new imported modules periodically.

- ASM: This introduces Threat Monitoring and Blocking on FastAPI.
- IP Blocking and all input addresses are supported on requests and responses

\- Custom Blocking This does not contain user blocking specific features yet.

- tracing: Introduces support for OpenTracing Baggage Items with HTTP Propagation. Enable this support by `DD_TRACE_PROPAGATION_HTTP_BAGGAGE_ENABLED=true`. The `Context._set_baggage_item` and `Context._get_baggage_item` internal methods are provided for manual modifications to the Baggage Items. These API changes are subject to change.

- dynamic instrumentation: Add support for more built-in container types, such as `defaultdict`, `frozenset`, `OrderedDict` and `Counter`.

- Vulnerability Management for Code-level (IAST): Adds Python 3.12 compatibility

- Optionally lazy loads and disables Instrumentation Telemetry. Users must explicitly set "DD_INSTRUMENTATION_TELEMETRY_ENABLED=false".

- tracer: Add support for remotely setting the trace sample rate from the Datadog UI. This functionality is enabled by default when using `ddtrace-run` and library injection. To enable it when using the library manually, use `ddtrace.config.enable_remote_config()`.

Bug Fixes

- tracer: tag spans that have been sampled due to an Agent sampling configuration.
- lambda: This change disables the use of `multiprocessing.queue` in Lambda, because it is not supported in Lambda
- langchain: This fix resolves a crash that could occur during embedding when no embeddings are found.
- Fix a regression with the support for gevent that could have occurred if some products, like ASM, telemetry, were enabled.
- kafka: Resolves `TypeError` raised by serializing producers and deserializing consumers when the `message.key` tag is set on spans.
- dynamic instrumentation: Fix an issue that caused the instrumented application to fail to start if a non-standard module was imported.
- openai: This fix resolves an issue where tagging image inputs in the chat completions endpoint resulted in attribute errors.
- openai: This fix resolves an issue where requesting raw API responses from openai\>=1.0 resulted in attribute errors while tagging.
- profiling: Fix an issue that prevented threading locks from being traced when using gevent.
- profiling: Fix a segmentation fault with CPython 3.12 when sampling thread stacks.
- pylibmc: Fixes an issue where using `ddtrace-run` or `ddtrace.patch_all()` with `DD_TRACE_ENABLED=False` would break with get, gets, and get_multi operations on pylibmc Clients.
- tracing: This fix resolves an issue where concurrent mutations to the `context._meta` dict caused <span class="title-ref">RuntimeError: dictionary changed size during iteration</span>.
- django: Resolves `AttributeError` raised by traced `StreamingHttpResponse`.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where certain aspects incorrectly expected at least one argument, leading to an IndexError when none were provided. The solution removes this constraint and incorporates regression tests for stability assurance.
- Vulnerability Management for Code-level (IAST): Cookies vulnerabilities are only reported if response cookies are insecure.
- Vulnerability Management for Code-level (IAST): Fix propagation error on `.format` string method.
- requests: Updates the resource names of `requests.requests` spans to include the method and path of the request.
- propagation: This fix resolves an issue where a `Context` generated from extracted headers could lack a span_id or trace_id, leading `SpanLink` encoding errors.
- psycopg: This fix resolves an issue where a circular import of the psycopg library could cause a crash during monkeypatching.
- psycopg: This fix resolves an issue where exceptions originating from asynchronous Psycopg cursors were not propagated up the call stack.
- redis: This fix resolves an issue where the yaaredis and aredis integrations imported code from the redis integration, causing a circular import error.
- tracing: Resolves trace encoding errors raised when `DD_TRACE_API_VERSION` is set to `v0.5` and a BufferFull Exception is raised by the TraceWriter. This fix ensures span fields are not overwritten and reduces the frequency of 4XX errors in the trace agent.

Other Changes

- tracing: Upgrades the trace encoding format to v0.5. This change improves the performance of encoding and sending spans.

---

New Features

- propagation: 128-bit trace ids are now used by default for propagation. Previously the default was 64-bit. This change is backwards compatible with tracers that still use 64-bit trace ids and should not cause any breaking behavior.
- Adds DSM `pathway.hash` tag to spans when DSM is enabled. This allows traces from the instrumented service to show up in the DSM traces tab.
- propagation: When the tracer is configured to extract and inject `tracecontext`, the tracer will propagate the tracestate values from other vendors so long as the traceparent trace-id matches the first found trace context, regardless of propagator configuration order. To disable this behavior `DD_TRACE_PROPAGATION_EXTRACT_FIRST=true` can be set.
- opentelemetry: Map reserved OpenTelemetry attributes to Datadog span model.
- opentelemetry: datadog operation name from semantic conventions
- propagation: If a valid context is extracted from headers, and the following extracted trace context's `trace_id`s do not match the valid context's, then add a span link to the root span to represent the broken propagation.
- tracing: This change treats spans that terminated with `sys.exit(0)` as successful non-error spans.
- tracing: This introduces the `DD_TRACE_SPAN_TRACEBACK_MAX_SIZE` environment variable, allowing the maximum size of tracebacks included on spans to be configured.

Bug Fixes

- CI Visibility: fixes the fact that the GITHUB_SERVER_URL environment variable was not being sanitized for credentials
- dynamic instrumentation: Needs to update the pubsub instance when the application forks because the probe mechanism should run in the child process. For that, DI needs the callback as the method of an instance of Debugger, which lives in the child process.
- CI Visibility: Fixes an issue where a `ValueError` was raised when using different path drives on Windows
- Fixes an issue where ddtrace could not be installed from source when using `setuptools>=69` due to a change in the license field.
- tracing: Fixes an issue where the thread responsible for sending traces is killed due to concurrent dictionary modification.
- structlog: Fixes `TypeError` raised when ddtrace log processor is configured with a tuple
- Vulnerability Management for Code-level (IAST): Generates cookies vulnerabilities report if IAST is enabled. Before this fix, Cookies vulnerabilities were only generated if both IAST and Appsec were enabled.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where, at AST patching to replace code with IAST aspects, passing the original function/method as an extra parameter for accurate patching unintentionally triggers side effects in methods obtained from an expression (like `decode` in `file.read(n).decode()`), resulting in unexpected multiple calls to the expression (`file.read(n)` in the example).
- Vulnerability Management for Code-level (IAST): This fix eliminates some reference leaks and C-API usage when IAST reports a vulnerability and calls `get_info_frame`.
- kafka: This fix resolves an issue where calls to `confluent_kafka`'s `produce` method with `key=None` would cause an exception to be raised.
- tracing: This fix resolves an issue where ddtrace's signal handlers could cause Flask apps not to respond correctly to SIGINT.
- logging: A log handler is automatically added to the ddtrace logger upon ddtrace import, when not using ddtrace-run. This can lead to duplicate logging if users add additional loggers and do not explicitly modify the ddtrace logger. This fix adds a feature flag that can be used to toggle this behavior off `DD_TRACE_LOG_STREAM_HANDLER` which defaults to `true`.

---

Upgrade Notes

- The `wrapt` and `psutil` packages are vendored to help users avoid building these packages if wheels were not available for a given platform. This reverses a change released in v2.0.0.

New Features

- CI Visibility: adds ITR support for `unittest`
- CI Visibility: adds start/end line support for `pytest` test spans
- CI Visibility: adds start/end line source file data to `unittest` test spans
- aiohttp: This introduces basic tracing of streaming responses that stay open long after the <span class="title-ref">on_prepare</span> signal has been sent.
- CI Visibility: introduce pytest hooks for modifying the module, suite, and test naming logic
- CI Visibility: add support for AWS Codepipeline to CI env var gathering
- datastreams: this change adds message payload size metrics and aggregations for Kafka.
- structlog: Wraps get_logger function in order to add datadog injection processor regardless of configuration
- openai: This adds support for openai v1.
- Source Code: filters Git repo URLs from env vars and setuptools
- logbook: This introduces log correlation for the logbook library. Refer to `logbook-docs <ddtrace.contrib.logbook>` for more details.
- loguru: This introduces log correlation for the loguru library. Refer to `loguru-docs <ddtrace.contrib.loguru>` for more details.
- openai: This adds support for tagging function call arguments when using OpenAI's function calling feature.
- Adds ARM64 support for Single-Step instrumentation
- structlog: This introduces log correlation for the structlog library. Refer to `structlog-docs <ddtrace.contrib.structlog>` for more details.
- celery: Adds Python 3.11 and 3.12 support for the celery integration.

Known Issues

- ASM: fix a body read problem on some corner case where passing empty content length makes wsgi.input.read() blocks.

Bug Fixes

- Application Security Management (ASM): fix a body read error when `Transfer-Encoding: chunked` header is sent
- CI Visibility: fixes an issue where class-based test methods with the same name across classes would be considered duplicates, and cause one (or more) tests to be dropped from results, by adding `--ddtrace-include-class-name` as an optional flag (defaulting to false) to prepend the class name to the test name.
- CI Visibility: fixes a crash where the unittest integration would try to enable coverage when tests are run even if the Intelligent Test Runner is not enabled.
- data_streams: This fix resolves an issue where tracing would crash if a kafka client produced a message with no key or value.
- CI: fixes an issue which prevented the library from filtering user credentials for SSH Git repository URLs
- dynamic instrumentation: fix an issue that caused function probes on the same module to fail to instrument and be reported in the `ERROR` status in the UI if the module was not yet imported.
- Use a unique default service name across all the products provided by the library when one is not given via the configuration interface.
- sampling: This fix reverts a refactor which affected how the tracer handled the trace-agent's recommended trace sampling rates, leading to an unintended increase in traces sampled.
- tracing: Fixes a msgpack import error when `DD_TRACE_API` is set to `v0.5`
- fix(profiling): numeric type exception in memalloc When pushing allocation samples, an exception was being thrown due to a float being passed instead of an integer. We now cast the ceiled value to an integer.
- CI Visibility: fixes `unittest` data not being initialized properly
- CI Visibility: fixes an issue where just importing <span class="title-ref">unittest</span> enabled CIVisibility and potentially caused unexpected logs and API requests
- Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where custom functions or methods could be replaced by aspects with differing argument numbers, causing runtime errors as a result. Furthermore, it addresses a case during patching where the module is inadvertently passed as the first argument to the aspect.
- Vulnerability Management for Code-level (IAST): Fix potential string id collisions that could cause false positives with non tainted objects being marked as tainted.
- IAST: This fix resolves an issue where JSON encoder would throw an exception while encoding a tainted dict or list.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where SimpleJSON encoder would throw an exception while encoding a tainted dict or list.
- ASM: add support for psycopg2 adapt mechanism to LazyTaintList, preventing a ProgrammingError when using psycopg2 with IAST.
- tracing: This fix resolves an issue where unserializable tracer attributes caused crashes when `DD_TRACE_DEBUG` was set.
- This fix resolves an issue where `confluent_kafka`'s `SerializingProducer` and `DeserializingConsumer` classes were incorrectly patched, causing crashes when these classes are in use with Datadog patching.
- langchain: This fix resolves an issue with tagging pydantic <span class="title-ref">SecretStr</span> type api keys.
- lib injection: Fix permissions error raised when non-root users copy single step instrumentation files.
- redis: The Datadog Agent removes command arguments from the resource name. However there are cases, like compressed keys, where this obfuscation cannot correctly remove command arguments. To safeguard that situation, the resource name set by the tracer will only be the command (e.g. SET) with no arguments. To retain the previous behavior and keep arguments in the span resource, with the potential risk of some command arguments not being fully obfuscated, set `DD_REDIS_RESOURCE_ONLY_COMMAND=false`.

Other Changes

- tags: Previously `DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH` had a max limit setting of 512 characters. This change removes that limit but keeps the default at 512.

---

Prelude

The Datadog APM Python team is happy to announce the release of v2.0.0 of ddtrace. This release drops support for Python 2.7, 3.5, and 3.6. This release adds support for Python 3.12.

<div class="important">

<div class="title">

Important

</div>

If you are on version of Python not supported by v2, we will continue to maintain the ddtrace v1 with bug fixes.

</div>

<div class="note">

<div class="title">

Note

</div>

Before upgrading to v2.0.0, we recommend users install `ddtrace~=1.20.0` and enable deprecation warnings. All removals to the library interface and environment variables in v2 were deprecated in the 1.x release line.

</div>

<div class="note">

<div class="title">

Note

</div>

The changes to environment variables apply only to the configuration of the ddtrace library and not the Datadog Agent.

</div>

Upgrading summary

Functionality changes

The default logging configuration functionality of ddtrace has been changed to avoid conflicting with application logging configurations. `DD_CALL_BASIC_CONFIG` has been removed and the ddtrace logger will log to stdout by default, or a log file as specified using `DD_TRACE_LOG_FILE`.

Setting the environment variable `DD_TRACE_PROPAGATION_STYLE='b3'`, which previously enabled `b3multi` now enables `b3 single header`. `b3 single header` still works but is deprecated for `b3`. Simplified: `b3` used to enable `b3multi`, but now enables `b3 single header` to better align with Opentelemetry's terms.

Removed deprecated environment variables

These environment variables have been removed. In all cases the same functionality is provided by other environment variables and replacements are provided as recommended actions for upgrading.

| Variable | Replacement | Note |
|--------------------------------------------|--------------------------------------------|-----------------------------------------------------|
| `DD_GEVENT_PATCH_ALL` | None | `📝<remove-dd-gevent-patch-all>` |
| `DD_AWS_TAG_ALL_PARAMS` | None | `📝<remove-aws-tag-all-params>` |
| `DD_REMOTECONFIG_POLL_SECONDS` | `DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS` | `📝<rename-remote-config-poll-seconds>` |
| `DD_CALL_BASIC_CONFIG` | None | `📝<remove-basic-config>` |
| `DD_TRACE_OBFUSCATION_QUERY_STRING_PATERN` | `DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP` | `📝<remove-trace-obfuscation-query-string-pattern>` |

Removed deprecated library interfaces

These methods and module attributes have been removed. Where the same functionality is provided by a different public method or module attribute, a recommended action is provided for upgrading. In a few limited cases, because the interface was no longer used or had been moved to the internal interface, it was removed and so no action is provided for upgrading.

| Module | Method/Attribute | Note |
|-----------------------------------|---------------------------------|----------------------------------------------------|
| `ddtrace.constants` | `APPSEC_ENABLED` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_JSON` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_VERSION` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_ERRORS` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_LOADED` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_ERROR_COUNT` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_DURATION` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_DURATION_EXT` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_TIMEOUTS` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_VERSION` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_ORIGIN_VALUE` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_BLOCKED` | `📝<remove-appsec-private-constants>` |
| | `IAST_JSON` | `📝<remove-appsec-private-constants>` |
| | `IAST_ENABLED` | `📝<remove-appsec-private-constants>` |
| | `IAST_CONTEXT_KEY` | `📝<remove-appsec-private-constants>` |
| `ddtrace.contrib.fastapi.patch` | `span_modifier` | `📝<remove-fastapi-starlette-span-modifier>` |
| | `aggregate_resources` | `📝<remove-fastapi-starlette-aggregate-resources>` |
| `ddtrace.contrib.starlette.patch` | `span_modifier` | `📝<remove-fastapi-starlette-span-modifier>` |
| | `aggregate_resources` | `📝<remove-fastapi-starlette-aggregate-resources>` |
| | `get_resource` | `📝<remove-fastapi-starlette-span-modifier>` |
| `ddtrace.contrib.grpc.constants` | `GRPC_PORT_KEY` | `📝<remove-grpc-port-key>` |
| `ddtrace.ext.cassandra` | `ROW_COUNT` | `📝<remove-cassandra-row-count>` |
| `ddtrace.ext.mongo` | `ROWS` | `📝<remove-mongo-row-count>` |
| `ddtrace.ext.sql` | `ROWS` | `📝<remove-sql-row-count>` |
| `ddtrace.filters` | `TraceCiVisibilityFilter` | `📝<remove-trace-ci-visibility-filter>` |
| `ddtrace.tracer` | `DD_LOG_FORMAT` | `📝<remove-dd-log-format>` |

Upgrade Notes

- <div id="remove-dd-gevent-patch-all">

`DD_GEVENT_PATCH_ALL` is removed. There is no special configuration necessary to make ddtrace work with gevent if using ddtrace-run.

</div>

- <div id="remove-aws-tag-all-params">

`DD_AWS_TAG_ALL_PARAMS` is removed. The boto/botocore/aiobotocore integrations no longer collect all API parameters by default.

</div>

- <div id="rename-remote-config-poll-seconds">

`DD_REMOTECONFIG_POLL_SECONDS` is removed. Use the environment variable `DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS` instead.

</div>

- <div id="remove-appsec-private-constants">

`APPSEC_ENABLED`, `APPSEC_JSON`, `APPSEC_EVENT_RULE_VERSION`, `APPSEC_EVENT_RULE_ERRORS`, `APPSEC_EVENT_RULE_LOADED`, `APPSEC_EVENT_RULE_ERROR_COUNT`, `APPSEC_WAF_DURATION`, `APPSEC_WAF_DURATION_EXT`, `APPSEC_WAF_TIMEOUTS`, `APPSEC_WAF_VERSION`, `APPSEC_ORIGIN_VALUE`, `APPSEC_BLOCKED`, `IAST_JSON`, `IAST_ENABLED`, `IAST_CONTEXT_KEY` are removed. This should not affect existing code as these deprecated ASM constants were meant for private use only.

</div>

- <div id="remove-fastapi-starlette-span-modifier">

`ddtrace.contrib.starlette.get_resource`, `ddtrace.contrib.starlette.span_modifier`, and `ddtrace.contrib.fastapi.span_modifier` are removed. The starlette and fastapi integrations now provide the full route and not just the mounted route for sub-applications.

</div>

- <div id="remove-fastapi-starlette-aggregate-resources">

`ddtrace.contrib.starlette.config['aggregate_resources']` and `ddtrace.contrib.fastapi.config['aggregate_resources']` are removed. The starlette and fastapi integrations no longer have the option to `aggregate_resources`, as it now occurs by default.

</div>

- <div id="remove-grpc-port-key">

`ddtrace.contrib.grpc.constants.GRPC_PORT_KEY` is removed. Use `ddtrace.ext.net.TARGET_PORT` instead.

</div>

- <div id="remove-cassandra-row-count">

`ddtrace.ext.cassandra.ROW_COUNT` is removed. Use `ddtrace.ext.db.ROWCOUNT` instead.

</div>

- <div id="remove-mongo-row-count">

`ddtrace.ext.mongo.ROW_COUNT` is removed. Use `ddtrace.ext.db.ROWCOUNT` instead.

</div>

- <div id="remove-sql-row-count">

`ddtrace.ext.sql.ROW_COUNT` is removed. Use `ddtrace.ext.db.ROWCOUNT` instead.

</div>

- <div id="remove-trace-ci-visibility-filter">

`ddtrace.filters.TraceCiVisibilityFilter` is removed.

</div>

- <div id="remove-dd-log-format">

`ddtrace.tracer.DD_LOG_FORMAT` is removed. As an alternative, please follow the log injection formatting as provided in the [log injection docs](https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#update-log-format).

</div>

- <div id="remove-basic-config">

`DD_CALL_BASIC_CONFIG` is removed. There is no special configuration necessary to replace `DD_CALL_BASIC_CONFIG`. The ddtrace logger will log to stdout by default or additionally to a file specified by `DD_TRACE_LOG_FILE`.

</div>

- <div id="remove-trace-obfuscation-query-string-pattern">

`DD_TRACE_OBFUSCATION_QUERY_STRING_PATTERN` is removed. Use `DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP` instead.

</div>

New Features

- Adds support for Python 3.12.

Known Issues

- aiohttp: Python 3.12 is not supported.
- aiohttp-jinja: Python 3.12 is not supported.
- aiobotocore: Python 3.12 is not supported.
- asm: IAST for Python 3.12 is not supported.
- flask-caching: Python 3.12 is not supported.
- openai/langchain: Python 3.12 is not supported.
- opentelemetry-api: Python 3.12 is not supported.
- opentracing: Python 3.12 is not supported.
- pyramid: Python 3.12 is not supported.
- pynamodb: Python 3.12 is not supported.
- redis/redis-py-cluster: Python 3.12 is not supported.

---

Prelude

Vulnerability Management for Code-level (IAST) is now available in private beta. Use the environment variable `DD_IAST_ENABLED=True` to enable this feature.

New Features

- ASM: This introduces support for custom blocking actions of type redirect_request.
- data_streams: Adds public api `set_produce_checkpoint` and `set_consume_checkpoint`

Bug Fixes

- kafka: Resolves an issue where traced kafka connections were assigned a default timeout of 1 second. The default timeout in [Consumer.poll(...)](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.Consumer.poll) should be None.
- openai: This fix resolves an issue where errors during streamed requests resulted in unfinished spans.

---

New Features

- Adds the <span class="title-ref">db.row_count</span> tag to redis and other redis-like integrations. The tag represents the number of returned results.
- CI Visibility: adds test level visibility for [unittest](https://docs.python.org/3/library/unittest.html)
- ASM: Adds detection of insecure cookie vulnerabilities on responses.
- ASM: This introduces trusted IPs capabilities in the tracer, to allow specific IPs not to be blocked by ASM but still be monitored.
- ASM: This introduces a new capability to configure the blocking response of ASM. Users can change the default blocking response behavior or create new custom actions. Configuration of a custom blocking page or payload can still be provided by using <span class="title-ref">DD_APPSEC_HTTP_BLOCKED_TEMPLATE_JSON</span> and <span class="title-ref">DD_APPSEC_HTTP_BLOCKED_TEMPLATE_HTML</span> to change the static files used for the response body. The action block, that can be defined in the static rule file or via remote configuration, allows now to create new custom blocking actions with any status code for the response.
- The aiopg and aiomysql integrations no longer set the sql.query tag on query spans. This tag duplicated the value captured by the span resource. Users who want to send this query unobfuscated can use the tracer API to set tags on the query span.
- data_streams: Starts tracking Kafka lag in seconds.
- kafka: Adds support for the Kafka serializing producer and deserializing consumer.
- profiling: allow individual collectors to be disabled.
- tracing: This change introduces the `allow_false` keyword argument to `BaseSampler.sample()`, which defaults to `True`. `allow_false` controls the function's return value. If `allow_false` is `False`, the function will always return `True` regardless of the sampling decision it made. This is useful when `sample` is called only for its side effects, which can include setting span tags.

Known Issues

- There are known issues configuring python's builtin multiprocessing library when ddtrace is installed. To use the multiprocessing library with ddtrace ensure `DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE` is set to `True`.
- When running setup.py extensions with the CMake parameter "-j", it could potentially raise an out-of-memory error. If someone wants to expedite the ddtrace installation, they should manually set the "CMAKE_BUILD_PARALLEL_LEVEL" environment variable.

Bug Fixes

- ASM: avoid potentially unneeded import of the IAST native module.

- ASM: avoid potentially unneeded import of the IAST native module if setup doesn't build extensions correctly.

- data_streams: This fix resolves an issue where data stream context propagation would not propagate via SNS if raw message delivery was enabled.

- dynamic instrumentation: function duration measurements are now reported in milliseconds to match the expectation from the UI.

- dynamic instrumentation: fixed an issue that prevented line probes from being injected in some finally blocks.

- dynamic instrumentation: Fixed the programmatic API to ensure that the dynamic instrumentation service is fully enabled when `Dynamic Instrumentation.enable()` is called.

- dynamic instrumentation: fixed a bug that might have caused probe status to fail to update correctly.

- django: This fix resolves an issue where 'span.resource' would not include the endpoint when a Handler was interrupted, such as in the case of gunicorn worker timeouts.

- CI Visibility: fixes an issue where the Intelligent Test Runner would not work when in EVP proxy mode due to missing `X-Datadog-NeedsAppKey` header.

- CI Visibility: revert to using DD_CIVISIBILITY_ITR_ENABLED (instead of \_DISABLED) to conform with other tracers.

- profiling: fixed a bug that prevented profiles from being correctly correlated to traces in gevent-based applications, thus causing code hotspot and end point data to be missing from the UI.

- docs: Fix undefined variable reference in otel documentation

- CI Visibility: fixes that Python 2.7 test results were not visible in UI due to improperly msgpack-ed data

- ASM: This fix resolves an issue where <span class="title-ref">track_user_signup_event</span> and <span class="title-ref">track_custom_event</span> where not correctly tagging the span. This could lead to the loss of some events in the sampling.

- appsec: Fixes an issue where ddtrace.appsec is imported and assumed to be available in all deployments of ddtrace

- lib-inject: This fix resolves an issue where `libdl.so.2: cannot open shared object file: No such file or directory` errors occurred when the
injection image started.

- lib-injection: Resolves permissions errors raised when ddtrace packages are copied from the InitContainer to the shared volume.

- mariadb: This fix resolves an issue where MariaDB connection information objects not including the user or port caused exceptions to be raised.

- appsec: This fix resolves an issue in which the library attempted to finalize twice a context object used by the Application Security Management product.

- propagation: Prevent propagating unsupported non-ascii `origin` header values.

- pymongo: This upgrades the PyMongo integration to work with PyMongo versions 4.5.0 and above by choosing the root function of the integration on the basis of the PyMongo version.

- tracing: This fix resolves an issue where the <span class="title-ref">\_dd.p.dm</span> and <span class="title-ref">\_dd.\*\_psr</span> tags were applied to spans in ways that did not match their intended semantics, increasing the potential for metrics-counting bugs.

- ASM: This fix resolves issue where user information was only set in root span. Now span for user information can be selected.

- sqlalchemy: sqlalchemy rollbacks could previously cause intermittent deadlocks in some cases. To fix this `DD_TRACE_SPAN_AGGREGATOR_RLOCK` was introduced in 1.16.2 with the default as `False`. We are now changing the default to `True`.

Other Changes

- Adds a <span class="title-ref">get_version</span> method to each integration and updates the basic template for developing an integration to include this method. The <span class="title-ref">get_version</span> method returns the integration's package distribution version and is to be included in the APM Telemetry integrations payload.
- Add a <span class="title-ref">ddtrace_iast_flask_patch</span> function defined in <span class="title-ref">ddtrace.appsec.iast</span> to ensure that the main Flask <span class="title-ref">app.py</span> file is patched for IAST propagation. This function should be called before the <span class="title-ref">app.run()</span> call. You only need this if you have set <span class="title-ref">DD_IAST_ENABLED=1</span>. Only the main file needs to call this functions, other imported modules are automatically patched.
- docs: Fixes formatting in ddtrace docs.
- ASM: Improve default value of regex for query string obfuscation. Rename env var `DD_TRACE_OBFUSCATION_QUERY_STRING_PATTERN` to `DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP`.

---