Sqlalchemy

:released: April 6, 2021

.. change::
:tags: bug, sql, regression, oracle, mssql
:tickets: 6202

Fixed further regressions in the same area as that of :ticket:`6173` released in
1.4.5, where a "postcompile" parameter, again most typically those used for
LIMIT/OFFSET rendering in Oracle and SQL Server, would fail to be processed
correctly if the same parameter rendered in multiple places in the
statement.



.. change::
:tags: bug, orm, regression
:tickets: 6203

Fixed regression where a deprecated form of :meth:`_orm.Query.join` were
used, passing a series of entities to join from without any ON clause in a
single :meth:`_orm.Query.join` call, would fail to function correctly.

.. change::
:tags: bug, mypy
:tickets: 6147

Applied a series of refactorings and fixes to accommodate for Mypy
"incremental" mode across multiple files, which previously was not taken
into account. In this mode the Mypy plugin has to accommodate Python
datatypes expressed in other files coming in with less information than
they have on a direct run.

Additionally, a new decorator :func:`_orm.declarative_mixin` is added,
which is necessary for the Mypy plugin to be able to definifitely identify
a Declarative mixin class that is otherwise not used inside a particular
Python file.

.. seealso::

:ref:`mypy_declarative_mixins`


.. change::
:tags: bug, mypy
:tickets: 6205

Fixed issue where the Mypy plugin would fail to interpret the
"collection_class" of a relationship if it were a callable and not a class.
Also improved type matching and error reporting for collection-oriented
relationships.


.. change::
:tags: bug, sql
:tickets: 6204

Executing a :class:`_sql.Subquery` using :meth:`_engine.Connection.execute`
is deprecated and will emit a deprecation warning; this use case was an
oversight that should have been removed from 1.4. The operation will now
execute the underlying :class:`_sql.Select` object directly for backwards
compatibility. Similarly, the :class:`_sql.CTE` class is also not
appropriate for execution. In 1.3, attempting to execute a CTE would result
in an invalid "blank" SQL statement being executed; since this use case was
not working it now raises :class:`_exc.ObjectNotExecutableError`.
Previously, 1.4 was attempting to execute the CTE as a statement however it
was working only erratically.

.. change::
:tags: bug, regression, orm
:tickets: 6206

Fixed critical regression where the :meth:`_orm.Query.yield_per` method in
the ORM would set up the internal :class:`_engine.Result` to yield chunks
at a time, however made use of the new :meth:`_engine.Result.unique` method
which uniques across the entire result. This would lead to lost rows since
the ORM is using ``id(obj)`` as the uniquing function, which leads to
repeated identifiers for new objects as already-seen objects are garbage
collected. 1.3's behavior here was to "unique" across each chunk, which
does not actually produce "uniqued" results when results are yielded in
chunks. As the :meth:`_orm.Query.yield_per` method is already explicitly
disallowed when joined eager loading is in place, which is the primary
rationale for the "uniquing" feature, the "uniquing" feature is now turned
off entirely when :meth:`_orm.Query.yield_per` is used.

This regression only applies to the legacy :class:`_orm.Query` object; when
using :term:`2.0 style` execution, "uniquing" is not automatically applied.
To prevent the issue from arising from explicit use of
:meth:`_engine.Result.unique`, an error is now raised if rows are fetched
from a "uniqued" ORM-level :class:`_engine.Result` if any
:ref:`yield per <orm_queryguide_yield_per>` API is also in use, as the
purpose of ``yield_per`` is to allow for arbitrarily large numbers of rows,
which cannot be uniqued in memory without growing the number of entries to
fit the complete result size.


.. change::
:tags: usecase, asyncio, postgresql
:tickets: 6199

Added accessors ``.sqlstate`` and synonym ``.pgcode`` to the ``.orig``
attribute of the SQLAlchemy exception class raised by the asyncpg DBAPI
adapter, that is, the intermediary exception object that wraps on top of
that raised by the asyncpg library itself, but below the level of the
SQLAlchemy dialect.

.. changelog::

:released: April 2, 2021

.. change::
:tags: bug, sql, postgresql
:tickets: 6183

Fixed bug in new :meth:`_functions.FunctionElement.render_derived` feature
where column names rendered out explicitly in the alias SQL would not have
proper quoting applied for case sensitive names and other non-alphanumeric
names.

.. change::
:tags: bug, regression, orm
:tickets: 6172

Fixed regression where the :func:`_orm.joinedload` loader strategy would
not successfully joinedload to a mapper that is mapper against a
:class:`.CTE` construct.

.. change::
:tags: bug, regression, sql
:tickets: 6181

Fixed regression where use of the :meth:`.Operators.in_` method with a
:class:`_sql.Select` object against a non-table-bound column would produce
an ``AttributeError``, or more generally using a :class:`_sql.ScalarSelect`
that has no datatype in a binary expression would produce invalid state.


.. change::
:tags: bug, mypy
:tickets: sqlalchemy/sqlalchemy2-stubs/14

Fixed issue in mypy plugin where newly added support for
:func:`_orm.as_declarative` needed to more fully add the
``DeclarativeMeta`` class to the mypy interpreter's state so that it does
not result in a name not found error; additionally improves how global
names are setup for the plugin including the ``Mapped`` name.


.. change::
:tags: bug, mysql, regression
:tickets: 6163

Fixed regression in the MySQL dialect where the reflection query used to
detect if a table exists would fail on very old MySQL 5.0 and 5.1 versions.

.. change::
:tags: bug, sql
:tickets: 6184

Added a new flag to the :class:`_engine.Dialect` class called
:attr:`_engine.Dialect.supports_statement_cache`. This flag now needs to be present
directly on a dialect class in order for SQLAlchemy's
:ref:`query cache <sql_caching>` to take effect for that dialect. The
rationale is based on discovered issues such as :ticket:`6173` revealing
that dialects which hardcode literal values from the compiled statement,
often the numerical parameters used for LIMIT / OFFSET, will not be
compatible with caching until these dialects are revised to use the
parameters present in the statement only. For third party dialects where
this flag is not applied, the SQL logging will show the message "dialect
does not support caching", indicating the dialect should seek to apply this
flag once they have verified that no per-statement literal values are being
rendered within the compilation phase.

.. seealso::

:ref:`engine_thirdparty_caching`

.. change::
:tags: bug, postgresql
:tickets: 6099

Fixed typo in the fix for :ticket:`6099` released in 1.4.4 that completely
prevented this change from working correctly, i.e. the error message did not match
what was actually emitted by pg8000.

.. change::
:tags: bug, orm, regression
:tickets: 6171

Scaled back the warning message added in :ticket:`5171` to not warn for
overlapping columns in an inheritance scenario where a particular
relationship is local to a subclass and therefore does not represent an
overlap.

.. change::
:tags: bug, regression, oracle
:tickets: 6173

Fixed critical regression where the Oracle compiler would not maintain the
correct parameter values in the LIMIT/OFFSET for a select due to a caching
issue.


.. change::
:tags: bug, postgresql
:tickets: 6170

Fixed issue where the PostgreSQL :class:`.PGInspector`, when generated
against an :class:`_engine.Engine`, would fail for ``.get_enums()``,
``.get_view_names()``, ``.get_foreign_table_names()`` and
``.get_table_oid()`` when used against a "future" style engine and not the
connection directly.

.. change::
:tags: bug, schema
:tickets: 6146

Introduce a new parameter :paramref:`_types.Enum.omit_aliases` in
:class:`_types.Enum` type allow filtering aliases when using a pep435 Enum.
Previous versions of SQLAlchemy kept aliases in all cases, creating
database enum type with additional states, meaning that they were treated
as different values in the db. For backward compatibility this flag
defaults to ``False`` in the 1.4 series, but will be switched to ``True``
in a future version. A deprecation warning is raise if this flag is not
specified and the passed enum contains aliases.

.. change::
:tags: bug, mssql
:tickets: 6163

Fixed a regression in MSSQL 2012+ that prevented the order by clause
to be rendered when ``offset=0`` is used in a subquery.

.. change::
:tags: bug, asyncio
:tickets: 6166


Fixed issue where the asyncio extension could not be loaded
if running Python 3.6 with the backport library of
``contextvars`` installed.

.. changelog::

:released: March 30, 2021

.. change::
:tags: bug, misc

Adjusted the usage of the ``importlib_metadata`` library for loading
setuptools entrypoints in order to accommodate for some deprecation
changes.


.. change::
:tags: bug, postgresql
:tickets: 6099

Modified the ``is_disconnect()`` handler for the pg8000 dialect, which now
accommodates for a new ``InterfaceError`` emitted by pg8000 1.19.0. Pull
request courtesy Hamdi Burak Usul.


.. change::
:tags: bug, orm
:tickets: 6139

Fixed critical issue in the new :meth:`_orm.PropComparator.and_` feature
where loader strategies that emit secondary SELECT statements such as
:func:`_orm.selectinload` and :func:`_orm.lazyload` would fail to
accommodate for bound parameters in the user-defined criteria in terms of
the current statement being executed, as opposed to the cached statement,
causing stale bound values to be used.

This also adds a warning for the case where an object that uses
:func:`_orm.lazyload` in conjunction with :meth:`_orm.PropComparator.and_`
is attempted to be serialized; the loader criteria cannot reliably
be serialized and deserialized and eager loading should be used for this
case.


.. change::
:tags: bug, engine
:tickets: 6138

Repair wrong arguments to exception handling method
in CursorResult.

.. change::
:tags: bug, regression, orm
:tickets: 6144

Fixed missing method :meth:`_orm.Session.get` from the
:class:`_orm.ScopedSession` interface.


.. change::
:tags: usecase, engine
:tickets: 6155

Modified the context manager used by :class:`_engine.Transaction` so that
an "already detached" warning is not emitted by the ending of the context
manager itself, if the transaction were already manually rolled back inside
the block. This applies to regular transactions, savepoint transactions,
and legacy "marker" transactions. A warning is still emitted if the
``.rollback()`` method is called explicitly more than once.

.. changelog::

:released: March 25, 2021

.. change::
:tags: bug, orm
:tickets: 6069

Fixed a bug where python 2.7.5 (default on CentOS 7) wasn't able to import
sqlalchemy, because on this version of Python ``exec "statement"`` and
``exec("statement")`` do not behave the same way. The compatibility
``exec_()`` function was used instead.

.. change::
:tags: sqlite, feature, asyncio
:tickets: 5920

Added support for the aiosqlite database driver for use with the
SQLAlchemy asyncio extension.

.. seealso::

:ref:`aiosqlite`

.. change::
:tags: bug, regression, orm, declarative
:tickets: 6128

Fixed regression where the ``.metadata`` attribute on a per class level
would not be honored, breaking the use case of per-class-hierarchy
:class:`.schema.MetaData` for abstract declarative classes and mixins.


.. seealso::

:ref:`declarative_metadata`

.. change::
:tags: bug, mypy

Added support for the Mypy extension to correctly interpret a declarative
base class that's generated using the :func:`_orm.as_declarative` function
as well as the :meth:`_orm.registry.as_declarative_base` method.

.. change::
:tags: bug, mypy
:tickets: 6109

Fixed bug in Mypy plugin where the Python type detection
for the :class:`_types.Boolean` column type would produce
an exception; additionally implemented support for :class:`_types.Enum`,
including detection of a string-based enum vs. use of Python ``enum.Enum``.

.. change::
:tags: bug, reflection, postgresql
:tickets: 6129

Fixed reflection of identity columns in tables with mixed case names
in PostgreSQL.

.. change::
:tags: bug, sqlite, regression
:tickets: 5848

Repaired the ``pysqlcipher`` dialect to connect correctly which had
regressed in 1.4, and added test + CI support to maintain the driver
in working condition. The dialect now imports the ``sqlcipher3`` module
for Python 3 by default before falling back to ``pysqlcipher3`` which
is documented as now being unmaintained.

.. seealso::

:ref:`pysqlcipher`


.. change::
:tags: bug, orm
:tickets: 6060

Fixed bug where ORM queries using a correlated subquery in conjunction with
:func:`_orm.column_property` would fail to correlate correctly to an
enclosing subquery or to a CTE when :meth:`_sql.Select.correlate_except`
were used in the property to control correlation, in cases where the
subquery contained the same selectables as ones within the correlated
subquery that were intended to not be correlated.

.. change::
:tags: bug, orm
:tickets: 6131

Fixed bug where combinations of the new "relationship with criteria"
feature could fail in conjunction with features that make use of the new
"lambda SQL" feature, including loader strategies such as selectinload and
lazyload, for more complicated scenarios such as polymorphic loading.

.. change::
:tags: bug, orm
:tickets: 6124

Repaired support so that the :meth:`_sql.ClauseElement.params` method can
work correctly with a :class:`_sql.Select` object that includes joins
across ORM relationship structures, which is a new feature in 1.4.


.. change::
:tags: bug, engine, regression
:tickets: 6119

Restored the :class:`_engine.ResultProxy` name back to the
``sqlalchemy.engine`` namespace. This name refers to the
``LegacyCursorResult`` object.

.. change::
:tags: bug, orm
:tickets: 6115

Fixed issue where a "removed in 2.0" warning were generated internally by
the relationship loader mechanics.


.. changelog::

:released: March 19, 2021

.. change::
:tags: bug, orm, dataclasses
:tickets: 6093

Fixed issue in new ORM dataclasses functionality where dataclass fields on
an abstract base or mixin that contained column or other mapping constructs
would not be mapped if they also included a "default" key within the
dataclasses.field() object.


.. change::
:tags: bug, regression, orm
:tickets: 6088

Fixed regression where the :attr:`_orm.Query.selectable` accessor, which is
a synonym for :meth:`_orm.Query.__clause_element__`, got removed, it's now
restored.

.. change::
:tags: bug, engine, regression

Restored top level import for ``sqlalchemy.engine.reflection``. This
ensures that the base :class:`_reflection.Inspector` class is properly
registered so that :func:`_sa.inspect` works for third party dialects that
don't otherwise import this package.


.. change::
:tags: bug, regression, orm
:tickets: 6086

Fixed regression where use of an unnamed SQL expression such as a SQL
function would raise a column targeting error if the query itself were
using joinedload for an entity and was also being wrapped in a subquery by
the joinedload eager loading process.


.. change::
:tags: bug, orm, regression
:tickets: 6092

Fixed regression where the :meth:`_orm.Query.filter_by` method would fail
to locate the correct source entity if the :meth:`_orm.Query.join` method
had been used targeting an entity without any kind of ON clause.


.. change::
:tags: postgresql, usecase
:tickets: 6982

Rename the column name used by a reflection query that used
a reserved word in some postgresql compatible databases.

.. change::
:tags: usecase, orm, dataclasses
:tickets: 6100

Added support for the :class:`_orm.declared_attr` object to work in the
context of dataclass fields.

.. seealso::

:ref:`orm_declarative_dataclasses_mixin`

.. change::
:tags: bug, sql, regression
:tickets: 6101

Fixed issue where using a ``func`` that includes dotted packagenames would
fail to be cacheable by the SQL caching system due to a Python list of
names that needed to be a tuple.


.. change::
:tags: bug, regression, orm
:tickets: 6095

Fixed regression where the SQL compilation of a :class:`.Function` would
not work correctly if the object had been "annotated", which is an internal
memoization process used mostly by the ORM. In particular it could affect
ORM lazy loads which make greater use of this feature in 1.4.

.. change::
:tags: bug, sql, regression
:tickets: 6097

Fixed regression in the :func:`_sql.case` construct, where the "dictionary"
form of argument specification failed to work correctly if it were passed
positionally, rather than as a "whens" keyword argument.

.. change::
:tags: bug, orm
:tickets: 6090

Fixed regression where the :class:`.ConcreteBase` would fail to map at all
when a mapped column name overlapped with the discriminator column name,
producing an assertion error. The use case here did not function correctly
in 1.3 as the polymorphic union would produce a query that ignored the
discriminator column entirely, while emitting duplicate column warnings. As
1.4's architecture cannot easily reproduce this essentially broken behavior
of 1.3 at the ``select()`` level right now, the use case now raises an
informative error message instructing the user to use the
``.ConcreteBase._concrete_discriminator_name`` attribute to resolve the
conflict. To assist with this configuration,
``.ConcreteBase._concrete_discriminator_name`` may be placed on the base
class only where it will be automatically used by subclasses; previously
this was not the case.


.. change::
:tags: bug, mypy
:tickets: sqlalchemy/sqlalchemy2-stubs/2

Fixed issue in MyPy extension which crashed on detecting the type of a
:class:`.Column` if the type were given with a module prefix like
``sa.Integer()``.


.. changelog::

:released: March 17, 2021

.. change::
:tags: bug, orm, regression
:tickets: 6066

Fixed regression where producing a Core expression construct such as
:func:`_sql.select` using ORM entities would eagerly configure the mappers,
in an effort to maintain compatibility with the :class:`_orm.Query` object
which necessarily does this to support many backref-related legacy cases.
However, core :func:`_sql.select` constructs are also used in mapper
configurations and such, and to that degree this eager configuration is
more of an inconvenience, so eager configure has been disabled for the
:func:`_sql.select` and other Core constructs in the absence of ORM loading
types of functions such as :class:`_orm.Load`.

The change maintains the behavior of :class:`_orm.Query` so that backwards
compatibility is maintained. However, when using a :func:`_sql.select` in
conjunction with ORM entities, a "backref" that isn't explicitly placed on
one of the classes until mapper configure time won't be available unless
:func:`_orm.configure_mappers` or the newer :func:`_orm.registry.configure`
has been called elsewhere. Prefer using
:paramref:`_orm.relationship.back_populates` for more explicit relationship
configuration which does not have the eager configure requirement.


.. change::
:tags: bug, mssql, regression
:tickets: 6058

Fixed regression where a new setinputsizes() API that's available for
pyodbc was enabled, which is apparently incompatible with pyodbc's
fast_executemany() mode in the absence of more accurate typing information,
which as of yet is not fully implemented or tested. The pyodbc dialect and
connector has been modified so that setinputsizes() is not used at all
unless the parameter ``use_setinputsizes`` is passed to the dialect, e.g.
via :func:`_sa.create_engine`, at which point its behavior can be
customized using the :meth:`.DialectEvents.do_setinputsizes` hook.

.. seealso::

:ref:`mssql_pyodbc_setinputsizes`

.. change::
:tags: bug, orm, regression
:tickets: 6055

Fixed a critical regression in the relationship lazy loader where the SQL
criteria used to fetch a related many-to-one object could go stale in
relation to other memoized structures within the loader if the mapper had
configuration changes, such as can occur when mappers are late configured
or configured on demand, producing a comparison to None and returning no
object. Huge thanks to Alan Hamlett for their help tracking this down late
into the night.



.. change::
:tags: bug, regression
:tickets: 6068

Added back ``items`` and ``values`` to ``ColumnCollection`` class.
The regression was introduced while adding support for duplicate
columns in from clauses and selectable in ticket 4753.


.. change::
:tags: bug, engine, regression
:tickets: 6074

The Python ``namedtuple()`` has the behavior such that the names ``count``
and ``index`` will be served as tuple values if the named tuple includes
those names; if they are absent, then their behavior as methods of
``collections.abc.Sequence`` is maintained. Therefore the
:class:`_result.Row` and ``LegacyRow`` classes have been fixed
so that they work in this same way, maintaining the expected behavior for
database rows that have columns named "index" or "count".

.. change::
:tags: bug, orm, regression
:tickets: 6076

Fixed regression where the :meth:`_orm.Query.exists` method would fail to
create an expression if the entity list of the :class:`_orm.Query` were
an arbitrary SQL column expression.


.. change::
:tags: bug, orm, regression
:tickets: 6052

Fixed regression where calling upon :meth:`_orm.Query.count` in conjunction
with a loader option such as :func:`_orm.joinedload` would fail to ignore
the loader option. This is a behavior that has always been very specific to
the :meth:`_orm.Query.count` method; an error is normally raised if a given
:class:`_orm.Query` has options that don't apply to what it is returning.

.. change::
:tags: bug, orm, declarative, regression
:tickets: 6054

Fixed bug where user-mapped classes that contained an attribute named
"registry" would cause conflicts with the new registry-based mapping system
when using :class:`.DeclarativeMeta`. While the attribute remains
something that can be set explicitly on a declarative base to be
consumed by the metaclass, once located it is placed under a private
class variable so it does not conflict with future subclasses that use
the same name for other purposes.



.. change::
:tags: bug, orm, regression
:tickets: 6067

Fixed regression in :meth:`_orm.Session.identity_key`, including that the
method and related methods were not covered by any unit test as well as
that the method contained a typo preventing it from functioning correctly.


.. changelog::