Ddtrace

Prelude

Data Streams Monitoring (DSM) has added support for AWS Kinesis

**Breaking change** for CI Visibility: `test.suite` and `test.full_name` are changed, so any visualization or monitor that uses these fields is potentially affected.

Deprecation Notes

- `DD_CALL_BASIC_CONFIG` will be removed in the upcoming 2.0.0 release. As an alternative to `DD_CALL_BASIC_CONFIG`, you can call `logging.basicConfig()` to configure logging in your application.
- `DD_LOG_FORMAT` is deprecated and will be removed in 2.0.0. As an alternative, please follow the log injection formatting as provided in the [log injection docs](https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#update-log-format).

New Features

- CI Visibility: added tracing support for pytest-benchmark

- ASM: The vulnerability report now includes a feature to scrub potentially sensitive information. This scrubbing process looks for common patterns, and it can be further expanded using environment variables such as `DD_IAST_REDACTION_NAME_PATTERN` and `DD_IAST_REDACTION_VALUE_PATTERN`. See the [docs](https://ddtrace.readthedocs.io/en/stable/configuration.html#DD_IAST_REDACTION_ENABLED) for more information.

- DSM: Adds DSM support for AWS Kinesis. For information about DSM, see the [official documentation](https://docs.datadoghq.com/data_streams/). This change requires users to use botocore version 1.26.30 or later and update calls to Kinesis' <span class="title-ref">PutRecord</span>, <span class="title-ref">PutRecords</span>, and <span class="title-ref">GetRecords</span> calls with the StreamARN argument.

- pytest: This change introduces an option to the pytest plugin to disable ddtrace: `--no-ddtrace`

- CI visibility: Adds support for tracking repository URLs via the BITBUCKET_GIT_HTTP_ORIGIN environment variable

- CI visibility: Adds CodeFresh integration

- CI Visibility: Beta release of `pytest` support for the [Intelligent Test Runner](https://docs.datadoghq.com/continuous_integration/intelligent_test_runner/) .

- openai: `tiktoken` has been introduced as an optional package dependency to calculate the number of
tokens used in a prompt for a streamed completion or streamed chat completion. To enable this feature, install `ddtrace[openai]` or `tiktoken`. If `tiktoken` is not installed, the prompt token count will be continue to be estimated instead.

- Allows the use of a new backend for storing and exporting profiling data. This feature can be enabled for now by setting the DD_PROFILING_EXPORT_LIBDD_ENABLED environment variable to true. This should improve performance while decreasing memory overhead.

Known Issues

- sqlalchemy: sqlalchemy rollbacks can intermittently cause deadlocks in some cases. If experiencing this issue, set `DD_TRACE_SPAN_AGGREGATOR_RLOCK=True`. After testing and feedback we intend to make True the default value.

Bug Fixes

- CI Visibility: fixes an issue where the CIVisibility client would raise an exception if it was started in agentless mode without the DD_API_KEY set

- core: This fix moves `cmake` from `install_requires` to `setup_requires`.

- data_streams: This change fixes a bug in the Kafka & SQS integrations in which the Data Streams product code incorrect set timestamps for statistics. This led to all points being submitted for the same timestamp (the start of the application).

- dynamic instrumentation: handle null literal in conditions and expressions.

- dynamic instrumentation: fixed a bug that prevented span decoration probes from being received and instrumented.

- dynamic instrumentation: ensure that probes that fail to be instrumented because of invalid conditions/expressions are reported with status `ERROR` in the UI.

- CI Visibility: This fix solves an issue where the git unshallow command wasn't called

- tracing: Ensures health metrics are tagged with the correct values.

- CI Visibility: This fix resolves an issue where test skipping was not working properly.

- langchain: This fix resolves an issue where chat messages and embedding arguments
passed in as keyword arguments were not parsed correctly and resulted in an `ArgumentError`.

- langchain: This fix resolves an issue where `langchain.embeddings.HuggingFaceEmbeddings` embedding
methods, and `langchain.vectorstores.Milvus.similarity_search` were patched twice due to a nested class hierarchy in `langchain`.

- profiling: prevent deadlocks while recording events of different type.

- pytest: This fix resolves an issue where test modules could be non-existent, causing errors in the CI Visibility product.

- kafka: Resolves `UnicodeDecodeError` raised when kafka messages key contain characters that are not supported by UTF-8 encoding.

- lib-injection: Adds support for non-root run applications in containers.

- This fix resolves an issue causing span tags used by the Datadog backend not to be inherited by spans that exist in a different process from their parents.

Other Changes

- tracing: Previously the maximum size of a span tag was set to the full size of trace writer buffer (via DD_TRACE_WRITER_BUFFER_SIZE_BYTES). With this change the maximum size of span tags will be set to 10% of the size of the writer's buffer. This should decrease the frequency of encoding errors due to large span tags.

---

Prelude

Datadog has added support for automatically creating login success or failure events when a configured Django authentication backend is used. This will automatically fill the following tags in these cases:

> - <span class="title-ref">appsec.events.users.login.success.track</span>
> - <span class="title-ref">appsec.events.users.login.failure.track</span>
> - <span class="title-ref">appsec.events.users.login.success.\[email\|login\|username\]</span>
> - <span class="title-ref">appsec.events.users.login.failure.usr.exists</span>

New Features

- ASM: Add support for automatic user login events in Django.

- langchain: Adds integration with support for metrics, logs, and traces from LangChain requests.
See the `docs<langchain>` for more information.

- redis: Add support for Async RedisCluster.

Bug Fixes

- core: This fix removes the inclusion of our `benchmarks/` directory in the `ddtrace` wheels.
- internal: call `_fixupChildren` when retrieving `DDLogger`
- profiling: Fixed a regression whereby the profile exporter would not handle known request errors and asks the user to report an issue instead.
- profiling: Handles a race condition, which would occasionally throw an error, which would read `"RuntimeError: the memalloc module was not started."`
- CI visibility: fix version and step arguments gathering to enable plugin compatibility with pytest-bdd 6.1.x
- Fixed a bug that caused applications using gevent and cassandra to fail to start with the ddtrace-run command.
- tracing: This fix resolves a `google.protobuf` import error when module unloading.
- wsgi: This fix resolves an issues when trying to parse the `environ` property `HTTPS` as an HTTP header.
- Pin `cython<3` due to an incompatibility with `cython==3.0.0` and typing annotations in profiling code.
- telemetry: resolves issue with sending unnecessary duplicate logs

---

Prelude

Application Security Management (ASM) has added support for tracing subprocess executions.

Exception Debugging allows capturing debug information from exceptions attached to traces. The information about local variables and function arguments is displayed in the Error Tracking UI and augments the traceback data already collected.

New Features

- ASM: vulnerabilities related to insecure request cookies will be reported when `DD_APPSEC_ENABLED` is set to `true`.

- ASM: add support for tracing subprocess executions (like <span class="title-ref">os.system</span>, <span class="title-ref">os.spawn</span>, <span class="title-ref">subprocess.Popen</span> and others) and adding
information to a span names <span class="title-ref">command_execution</span> with the new type <span class="title-ref">system</span>. Currently we add the <span class="title-ref">cmd.exec</span> or <span class="title-ref">cmd.shell</span> tags to store the full command line (<span class="title-ref">cmd.shell</span> will be used when the command is run under a shell like with <span class="title-ref">os.system</span> or <span class="title-ref">Popen</span> with <span class="title-ref">shell=True</span>), <span class="title-ref">cmd.exit_code</span> to hold the return code when available, <span class="title-ref">component</span> which will hold the Python module used and the span <span class="title-ref">resource</span> will hold the binary used. This feature requires ASM to be activated using the <span class="title-ref">DD_APPSEC_ENABLED=True</span> configuration environment variable.

- botocore: Introduces environment variable `DD_BOTOCORE_INSTRUMENT_INTERNALS` that opts into tracing certain internal functionality.

- botocore: Added message attributes to Amazon Simple Queue Service spans to support data streams monitoring.

- exception debugging: Introduced the Exception Debugging feature that allows capturing debug information from exceptions attached to traces. This new feature can be enabled via the <span class="title-ref">DD_EXCEPTION_DEBUGGING_ENABLED</span>\` environment variable.

- openai: Adds support for metrics, logs, and traces for the models, edits, images, audio, files, fine-tunes, and
moderations endpoints. See [the docs](https://ddtrace.readthedocs.io/en/stable/integrations.html#openai) for more information.

- CI Visibility: Updates how pytest modules and test suites are reported. Modules names are now set to the fully qualified name, whereas test suites will be set to the file name.
Before this change: {"module": "tests", "suite":"my_module/tests/test_suite.py"} After this change: {"module": "my_module.tests", "suite": "test_suite.py"}

- core: Apply `DD_TAGS` to runtime metrics.

- kafka: Adds <span class="title-ref">messaging.kafka.bootstrap.servers</span> tag for the confluent-kafka producer configuration value found in <span class="title-ref">metadata.broker.list</span> or <span class="title-ref">bootstrap.servers</span>

- tracing: This reports the GRPC package name (optional) and service name in a single <span class="title-ref">rpc.service</span> tag

Bug Fixes

- botocore: This fix resolves an issue where ddtrace attempted to parse as URLs SQS QueueUrl attributes that were not well-formed URLs.
- psycopg: Resolves `TypeError` raised when an async cursor object is traced. This fix ensures <span class="title-ref">exc_type</span>, <span class="title-ref">exc_val</span>, and <span class="title-ref">exc_tb</span> are passed down to the wrapped object on <span class="title-ref">\_\_aexit\_\_</span>.
- Fixed an issue that prevented the library from working as expected when a combination of gevent and asyncio-based frameworks that rely on the functionalities of the ssl module is used.
- openai: Fixes the issue with `ImportError` of `TypedDict` from `typing` module in Python 3.7.
- openai: This fix resolves an issue where embeddings inputs were always tagged regardless of the configured prompt-completion sample rate.
- pytest: This fix resolves an issue where failures and non-skipped tests were not propagated properly when `unittest.TestCase` classes were used.
- Fixes an issue where harvesting runtime metrics on certain managed environments, such as Google Cloud Run, would cause ddtrace to throw an exception.
- graphql: `graphql.execute` spans are now marked as measured.
- tracing: This fix resolves an issue where negative trace ID values were allowed to propagate via Datadog distributed tracing HTTP headers.
- openai: Resolves some inconsistencies in logs generated by the image and audio endpoints, including filenames, prompts, and not logging raw binary image data.
- pymemcache: This fix resolves an issue where overriding span attributes on `HashClient` failed when `use_pooling` was set.
- This fix resolves an issue causing MyPy linting to fail on files that import ddtrace.
- The 1.15.0 version has a bug that arises when Remote Config receives both kinds of actions (removing target file configurations and loading new target file configurations) simultaneously, as the load action overrides the remove action. This error occurs if someone creates and removes Dynamic Instrumentation Probes rapidly, within a time interval shorter than the Remote Config interval (5s). To fix this issue, this update appends all new configurations and configurations to remove, and dispatches them at the end of the RC request.

---

New Features

- pyramid: Adds http.route tag to `pyramid.request` spans.
- data_streams: Add data streams core integration and instrument the confluent Kafka library with it. For more information, check out the docs, <https://docs.datadoghq.com/data_streams/>
- dynamic instrumentation: Added support for span decoration probes.

Bug Fixes

- ASM: This fix resolves an issue where the WAF rule file specified by DD_APPSEC_RULES was wrongly updated and modified by remote config.
- celery: Resolves an issue where hostname tags were not set in spans generated by `celery>4.0`.
- django: Resolves an issue where the resource name of django.request span did not contain the full name of a view when `DD_DJANGO_USE_HANDLER_RESOURCE_FORMAT=True`. This issue impacts `django>=4.0`.
- CI Visibility: This fix resolves the compatibility for Gitlab 16.0 deprecated urls
- openai: Resolves an issue where using an array of tokens or an array of token arrays for the Embeddings endpoint caused an AttributeError.
- profiling: Fixed an issue with gunicorn and gevent workers that occasionally caused an `AttributeError` exception to be raised on profiler start-up.
- psycopg: Fixes `ValueError` raised when dsn connection strings are parsed. This was fixed in ddtrace v1.9.0 and was re-introduced in v1.13.0.
- gunicorn: This fix ensures ddtrace threads do not block the master process from spawning workers when `DD_TRACE_DEBUG=true`. This issue impacts gunicorn applications using gevent and `python<=3.6`.

---

Prelude

profiling: Code provenance is a feature that enhances the "My code" experience in the Datadog UI by allowing the tracer to report packaging metadata about installed source files. This information is used to distinguish between user and third-party code.

New Features

- aws: Adds span tags for consistency with tags collected by Datadog for AWS metrics and logs.

- botocore: Adds the ability to control which botocore submodules will be patched.

- ASM: Send WAF metrics over telemetry

- pytest: This introduces test suite and module level visibility for the pytest integration. Pytest test traces will now include test session, test module, test suite, and test spans, which correlate to pytest session, pytest package, pytest module, and pytest test functions respectively.

- redis: Introducing redis command span tag max length configuration for `aioredis<aioredis>`, `aredis<aredis>`, `redis<redis>`, `rediscluster<rediscluster>`, and `yaaredis<yaaredis>` integrations.

- profiling: Code provenance is enabled by default.

- OpenAI: Add integration with support for metrics, logs and traces from
OpenAI requests. See [the docs](https://ddtrace.readthedocs.io/en/stable/integrations.html#openai) for more information.

Bug Fixes

- dependencies: Resolves an issue where ddtrace installs an incompatible version of cattrs when Python 3.6 is used.

- tracing: Resolves an issue where `DD_TRACE_<INTEGRATION>_ENABLED=False` could not be used to disable the following integrations when `ddtrace-run` was used: flask, django, bottle, falcon, and pyramid.

- asgi: Ensures `error.message` and `error.stack` tags are set when an exception is raised in a route.

- appsec: Fixes an encoding error when we are unable to cleanup the AppSec request context associated with a span.

- ASM: Fixes encoding error when using AppSec and a trace is partial flushed.

- CI Visibility: This fix resolves an issue where the tracer was doing extra requests if the `DD_CIVISIBILITY_ITR_ENABLED` env var was not set.

- CI Visibility: This fix resolves an issue where the API call would fail because it is reporting a null service name

- bootstrap: fixed an issue with the behavior of `ddtrace.auto` that could have caused incompatibilities with frameworks such as `gevent` when used as a programmatic alternative to the `ddtrace-run` command.

- django: Fixed a bug that prevented a Django application from starting with celery and gevent workers if `DJANGO_SETTINGS_MODULE` was not explicitly set.

- tracing: Fixes a cryptic encoding exception message when a span tag is not a string.

- ASM: fix extract_body for Django such that users of Django Rest Framework can still use custom parsers.

- flask: Remove patching for Flask hooks `app.before_first_request` and `bp.before_app_first_request` if Flask version \>= 2.3.0.

- gevent: Fix a bug that caused traceback objects to fail to pickle when using gevent.

- OpenAI: Resolved an issue where OpenAI API keys set in individual requests rather than as an environment variable caused an error in the integration.

- profiler: Fixed a bug that caused segmentation faults in applications that use protobuf as a runtime dependency.

- redis: Resolves an issue where the aioredis/aredis/yaaredis integrations cross-imported a helper method from the redis integration, which triggered redis patching before the redis integration was fully loaded.

- wsgi: Resolves an issue where accessing the `__len__` attribute on traced wsgi middlewares raised a TypeError

- django: Adds catch to guard against a ValueError, AttributeError, or NotImplementedError from being thrown when evaluating a django cache result for `db.row_count` tag.

- lib-injection: Ensure local package is installed. Previously the package
could still be pulled from the internet causing application slowdowns.

- kafka: Fixes `TypeError` raised when arbitrary keyword arguments are passed to `confluent_kafka.Consumer`

- profiler: Fix support for latest versions of protobuf.

- psycopg: Resolves an issue where an AttributeError is raised when `psycopg.AsyncConnection` is traced.

- sanic: Resolves `sanic_routing.exceptions.InvalidUsage` error raised when gevent is installed or `DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE` is set to True.

- elasticsearch: This fix resolves an issue where the tracer would throw an error when patching unsupported versions of elasticsearch (\> 8.0). Patching is now skipped if an unsupported version is detected.

Other Changes

- span: Increases the traceback limit in `error.stack` tags from 20 to 30
- aws_lambda: Logs warnings and exceptions on cold start only.

---

New Features

- psycopg: This release adds support for the new psycopg3 package. This new integration has all the same tracing functionality as the previous psycopg2-binary package, with added support for new methods including async connection and async cursor classes. The release also adds support for using Django\>=4.2 with psycopg3 integrated tracing.

Bug Fixes

- algoliasearch: This fix resolves an issue where non-text search query arguments caused Type Errors when being added as tags.

- ASM: fix calling <span class="title-ref">set_user</span> without a created span raising a <span class="title-ref">ValueError</span>.

- django: Adds fix for bug where Django cache return object throws an error if it does not implement `__bool__()`.

- kafka: Previously instantiating a subclass of kafka's Producer/Consumer classes would result in attribute errors due to patching the Producer/Consumer classes with an ObjectProxy. This fix resolves this issue by making the traced classes directly inherit from kafka's base Producer/Consumer classes.

- profiling: Fixed a regression in the memory collector that caused it to fail to cleanly re-initialize after a fork, causing error messages to be logged.

- logging: Ensure that the logging module can report thread information, such as thread names, correctly when a framework like gevent is used that requires modules cleanup.

- ASM: This fix resolves an issue where path parameters for the Flask framework were handled at response time instead of at request time for suspicious request blocking. This close a known issue opened in 1.10.0.

- lib-injection: Switch installation to install from included wheels. Prior,
the wheels were merged together which caused conflicts between versions of dependencies based on Python version.

- tracer: Handle exceptions besides `ImportError` when integrations are loaded.

Other Changes

- ASM: Add information about Application Security config values on <span class="title-ref">ddtrace-run --info</span>.
- otel: Fixes code formatting in api docs

---