Sqlalchemy

:released: December 18, 2020

.. change::
:tags: bug, oracle
:tickets: 5784
:versions: 1.4.0b2

Fixed regression which occurred due to :ticket:`5755` which implemented
isolation level support for Oracle. It has been reported that many Oracle
accounts don't actually have permission to query the ``v$transaction``
view so this feature has been altered to gracefully fallback when it fails
upon database connect, where the dialect will assume "READ COMMITTED" is
the default isolation level as was the case prior to SQLAlchemy 1.3.21.
However, explicit use of the :meth:`_engine.Connection.get_isolation_level`
method must now necessarily raise an exception, as Oracle databases with
this restriction explicitly disallow the user from reading the current
isolation level.

.. changelog::

:released: December 17, 2020

.. change::
:tags: bug, orm
:tickets: 5774
:versions: 1.4.0b2

Added a comprehensive check and an informative error message for the case
where a mapped class, or a string mapped class name, is passed to
:paramref:`_orm.relationship.secondary`. This is an extremely common error
which warrants a clear message.

Additionally, added a new rule to the class registry resolution such that
with regards to the :paramref:`_orm.relationship.secondary` parameter, if a
mapped class and its table are of the identical string name, the
:class:`.Table` will be favored when resolving this parameter. In all
other cases, the class continues to be favored if a class and table
share the identical name.

.. change::
:tags: sqlite, usecase
:tickets: 5685

Added ``sqlite_with_rowid=False`` dialect keyword to enable creating
tables as ``CREATE TABLE … WITHOUT ROWID``. Patch courtesy Sean Anderson.

.. change::
:tags: bug, sql
:tickets: 5691

A warning is emitted if a returning() method such as
:meth:`_sql.Insert.returning` is called multiple times, as this does not
yet support additive operation. Version 1.4 will support additive
operation for this. Additionally, any combination of the
:meth:`_sql.Insert.returning` and :meth:`_sql.ValuesBase.return_defaults`
methods now raises an error as these methods are mutually exclusive;
previously the operation would fail silently.


.. change::
:tags: bug, mssql
:tickets: 5751

Fixed bug where a CREATE INDEX statement was rendered incorrectly when
both ``mssql-include`` and ``mssql_where`` were specified. Pull request
courtesy Adiorz.

.. change::
:tags: bug, postgresql, mysql
:tickets: 5729
:versions: 1.4.0b2

Fixed regression introduced in 1.3.2 for the PostgreSQL dialect, also
copied out to the MySQL dialect's feature in 1.3.18, where usage of a non
:class:`_schema.Table` construct such as :func:`_sql.text` as the argument
to :paramref:`_sql.Select.with_for_update.of` would fail to be accommodated
correctly within the PostgreSQL or MySQL compilers.


.. change::
:tags: bug, mssql
:tickets: 5646

Added SQL Server code "01000" to the list of disconnect codes.


.. change::
:tags: usecase, postgresql
:tickets: 5604
:versions: 1.4.0b2

Added new parameter :paramref:`_postgresql.ExcludeConstraint.ops` to the
:class:`_postgresql.ExcludeConstraint` object, to support operator class
specification with this constraint. Pull request courtesy Alon Menczer.

.. change::
:tags: bug, mysql, reflection
:tickets: 5744
:versions: 1.4.0b2

Fixed issue where reflecting a server default on MariaDB only that
contained a decimal point in the value would fail to be reflected
correctly, leading towards a reflected table that lacked any server
default.


.. change::
:tags: bug, orm
:tickets: 5664

Fixed bug in :meth:`_query.Query.update` where objects in the
:class:`_ormsession.Session` that were already expired would be
unnecessarily SELECTed individually when they were refreshed by the
"evaluate"synchronize strategy.

.. change::
:tags: usecase, oracle
:tickets: 5755

Implemented support for the SERIALIZABLE isolation level for Oracle
databases, as well as a real implementation for
:meth:`_engine.Connection.get_isolation_level`.

.. seealso::

:ref:`oracle_isolation_level`

.. change::
:tags: mysql, sql
:tickets: 5696

Added missing keywords to the ``RESERVED_WORDS`` list for the MySQL
dialect: ``action``, ``level``, ``mode``, ``status``, ``text``, ``time``.
Pull request courtesy Oscar Batori.

.. change::
:tags: bug, orm
:tickets: 5737
:versions: 1.4.0b2

Fixed bug involving the ``restore_load_context`` option of ORM events such
as :meth:`_ormevent.InstanceEvents.load` such that the flag would not be
carried along to subclasses which were mapped after the event handler were
first established.



.. change::
:tags: bug, sql
:tickets: 5656

Fixed structural compiler issue where some constructs such as MySQL /
PostgreSQL "on conflict / on duplicate key" would rely upon the state of
the :class:`_sql.Compiler` object being fixed against their statement as
the top level statement, which would fail in cases where those statements
are branched from a different context, such as a DDL construct linked to a
SQL statement.


.. change::
:tags: mssql, sqlite, reflection
:tickets: 5661

Fixed issue with composite primary key columns not being reported
in the correct order. Patch courtesy fulpm.

.. changelog::

:released: October 12, 2020

.. change::
:tags: bug, orm
:tickets: 4428

An :class:`.ArgumentError` with more detail is now raised if the target
parameter for :meth:`_query.Query.join` is set to an unmapped object.
Prior to this change a less detailed ``AttributeError`` was raised.
Pull request courtesy Ramon Williams.

.. change::
:tags: bug, mysql
:tickets: 5568

The "skip_locked" keyword used with ``with_for_update()`` will emit a
warning when used on MariaDB backends, and will then be ignored. This is
a deprecated behavior that will raise in SQLAlchemy 1.4, as an application
that requests "skip locked" is looking for a non-blocking operation which
is not available on those backends.



.. change::
:tags: bug, engine
:tickets: 5599

Fixed issue where a non-string object sent to
:class:`_exc.SQLAlchemyError` or a subclass, as occurs with some third
party dialects, would fail to stringify correctly. Pull request
courtesy Andrzej Bartosiński.

.. change::
:tags: bug, sql
:tickets: 5644

Fixed issue where the ``pickle.dumps()`` operation against
:class:`_expression.Over` construct would produce a recursion overflow.

.. change::
:tags: postgresql, usecase
:tickets: 4392

The psycopg2 dialect now support PostgreSQL multiple host connections, by
passing host/port combinations to the query string. Pull request courtesy
Ramon Williams.

.. seealso::

:ref:`psycopg2_multi_host`

.. change::
:tags: bug, mysql
:tickets: 5617

Fixed bug where an UPDATE statement against a JOIN using MySQL multi-table
format would fail to include the table prefix for the target table if the
statement had no WHERE clause, as only the WHERE clause were scanned to
detect a "multi table update" at that particular point. The target
is now also scanned if it's a JOIN to get the leftmost table as the
primary table and the additional entries as additional FROM entries.


.. change::
:tags: bug, postgresql
:tickets: 5518

Adjusted the :meth:`_types.ARRAY.Comparator.any` and
:meth:`_types.ARRAY.Comparator.all` methods to implement a straight "NOT"
operation for negation, rather than negating the comparison operator.

.. change::
:tags: bug, pool
:tickets: 5582

Fixed issue where the following pool parameters were not being propagated
to the new pool created when :meth:`_engine.Engine.dispose` were called:
``pre_ping``, ``use_lifo``. Additionally the ``recycle`` and
``reset_on_return`` parameter is now propagated for the
:class:`_engine.AssertionPool` class.

.. change::
:tags: bug, ext, associationproxy
:tickets: 5541, 5542

An informative error is now raised when attempting to use an association
proxy element as a plain column expression to be SELECTed from or used in a
SQL function; this use case is not currently supported.


.. change::
:tags: bug, sql
:tickets: 5618

Fixed bug where an error was not raised in the case where a
:func:`_sql.column` were added to more than one :func:`_sql.table` at a
time. This raised correctly for the :class:`_schema.Column` and
:class:`_schema.Table` objects. An :class:`_exc.ArgumentError` is now
raised when this occurs.

.. change::
:tags: bug, orm
:tickets: 4589

Fixed issue where using a loader option against a string attribute name
that is not actually a mapped attribute, such as a plain Python descriptor,
would raise an uninformative AttributeError; a descriptive error is now
raised.



.. change::
:tags: mysql, usecase
:tickets: 5462

Adjusted the MySQL dialect to correctly parenthesize functional index
expressions as accepted by MySQL 8. Pull request courtesy Ramon Williams.

.. change::
:tags: bug, engine
:tickets: 5632

Repaired a function-level import that was not using SQLAlchemy's standard
late-import system within the sqlalchemy.exc module.


.. change::
:tags: change, mysql
:tickets: 5539

Add new MySQL reserved words: ``cube``, ``lateral`` added in MySQL 8.0.1
and 8.0.14, respectively; this indicates that these terms will be quoted if
used as table or column identifier names.

.. change::
:tags: bug, mssql
:tickets: 5592

Fixed issue where a SQLAlchemy connection URI for Azure DW with
``authentication=ActiveDirectoryIntegrated`` (and no username+password)
was not constructing the ODBC connection string in a way that was
acceptable to the Azure DW instance.

.. change::
:tags: bug, postgresql
:tickets: 5520

Fixed issue where the :class:`_postgresql.ENUM` type would not consult the
schema translate map when emitting a CREATE TYPE or DROP TYPE during the
test to see if the type exists or not. Additionally, repaired an issue
where if the same enum were encountered multiple times in a single DDL
sequence, the "check" query would run repeatedly rather than relying upon a
cached value.


.. change::
:tags: bug, tests
:tickets: 5635

Fixed incompatibilities in the test suite when running against Pytest 6.x.


.. changelog::

:released: August 17, 2020

.. change::
:tags: usecase, py3k
:tickets: 5357

Added a ``**kw`` argument to the :meth:`.DeclarativeMeta.__init__` method.
This allows a class to support the :pep:`487` metaclass hook
``__init_subclass__``. Pull request courtesy Ewen Gillies.


.. change::
:tags: bug, sql
:tickets: 5470

Repaired an issue where the "ORDER BY" clause rendering a label name rather
than a complete expression, which is particularly important for SQL Server,
would fail to occur if the expression were enclosed in a parenthesized
grouping in some cases. This case has been added to test support. The
change additionally adjusts the "automatically add ORDER BY columns when
DISTINCT is present" behavior of ORM query, deprecated in 1.4, to more
accurately detect column expressions that are already present.

.. change::
:tags: usecase, mysql
:tickets: 5481

The MySQL dialect will render FROM DUAL for a SELECT statement that has no
FROM clause but has a WHERE clause. This allows things like "SELECT 1 WHERE
EXISTS (subquery)" kinds of queries to be used as well as other use cases.


.. change::
:tags: bug, mssql, sql
:tickets: 5467

Fixed bug where the mssql dialect incorrectly escaped object names that
contained ']' character(s).

.. change::
:tags: bug, reflection, sqlite, mssql
:tickets: 5456

Applied a sweep through all included dialects to ensure names that contain
single or double quotes are properly escaped when querying system tables,
for all :class:`.Inspector` methods that accept object names as an argument
(e.g. table names, view names, etc). SQLite and MSSQL contained two
quoting issues that were repaired.

.. change::
:tags: bug, mysql
:tickets: 5411

Fixed an issue where CREATE TABLE statements were not specifying the
COLLATE keyword correctly.

.. change::
:tags: bug, datatypes, sql
:tickets: 4733

The ``LookupError`` message will now provide the user with up to four
possible values that a column is constrained to via the :class:`.Enum`.
Values longer than 11 characters will be truncated and replaced with
ellipses. Pull request courtesy Ramon Williams.

.. change::
:tags: bug, postgresql
:tickets: 5476

Fixed issue where the return type for the various RANGE comparison
operators would itself be the same RANGE type rather than BOOLEAN, which
would cause an undesirable result in the case that a
:class:`.TypeDecorator` that defined result-processing behavior were in
use. Pull request courtesy Jim Bosch.



.. change::
:tags: bug, mysql
:tickets: 5493

Added MariaDB code 1927 to the list of "disconnect" codes, as recent
MariaDB versions apparently use this code when the database server was
stopped.

.. change::
:tags: usecase, declarative, orm
:tickets: 5513

The name of the virtual column used when using the
:class:`_declarative.AbstractConcreteBase` and
:class:`_declarative.ConcreteBase` classes can now be customized, to allow
for models that have a column that is actually named ``type``. Pull
request courtesy Jesse-Bakker.

.. change::
:tags: usecase, orm
:tickets: 5494

Adjusted the workings of the :meth:`_orm.Mapper.all_orm_descriptors`
accessor to represent the attributes in the order that they are located in
a deterministic way, assuming the use of Python 3.6 or higher which
maintains the sorting order of class attributes based on how they were
declared. This sorting is not guaranteed to match the declared order of
attributes in all cases however; see the method documentation for the exact
scheme.



.. change::
:tags: bug, sql
:tickets: 5500

Fixed issue where the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature would not take effect when the :meth:`_schema.Sequence.next_value`
function function for a :class:`_schema.Sequence` were used in the
:paramref:`_schema.Column.server_default` parameter and the create table
DDL were emitted.

.. changelog::

:released: June 25, 2020

.. change::
:tags: bug, sqlite
:tickets: 5395

Added "exists" to the list of reserved words for SQLite so that this word
will be quoted when used as a label or column name. Pull request courtesy
Thodoris Sotiropoulos.

.. change::
:tags: bug, mssql
:tickets: 5366, 5364

Refined the logic used by the SQL Server dialect to interpret multi-part
schema names that contain many dots, to not actually lose any dots if the
name does not have bracking or quoting used, and additionally to support a
"dbname" token that has many parts including that it may have multiple,
independently-bracketed sections.



.. change::
:tags: bug, mssql, pyodbc
:tickets: 5346

Fixed an issue in the pyodbc connector such that a warning about pyodbc
"drivername" would be emitted when using a totally empty URL. Empty URLs
are normal when producing a non-connected dialect object or when using the
"creator" argument to create_engine(). The warning now only emits if the
driver name is missing but other parameters are still present.

.. change::
:tags: bug, mssql
:tickets: 5373

Fixed issue with assembling the ODBC connection string for the pyodbc
DBAPI. Tokens containing semicolons and/or braces "{}" were not being
correctly escaped, causing the ODBC driver to misinterpret the
connection string attributes.

.. change::
:tags: usecase, orm
:tickets: 5326

Improve error message when using :meth:`_query.Query.filter_by` in
a query where the first entity is not a mapped class.

.. change::
:tags: sql, schema
:tickets: 5324

Introduce :class:`.IdentityOptions` to store common parameters for
sequences and identity columns.

.. change::
:tags: usecase, sql
:tickets: 5309

Added a ".schema" parameter to the :func:`_expression.table` construct,
allowing ad-hoc table expressions to also include a schema name.
Pull request courtesy Dylan Modesitt.

.. change::
:tags: bug, mssql
:tickets: 5339

Fixed issue where ``datetime.time`` parameters were being converted to
``datetime.datetime``, making them incompatible with comparisons like
``>=`` against an actual :class:`_mssql.TIME` column.

.. change::
:tags: bug, mssql
:tickets: 5359

Fixed an issue where the ``is_disconnect`` function in the SQL Server
pyodbc dialect was incorrectly reporting the disconnect state when the
exception message had a substring that matched a SQL Server ODBC error
code.

.. change::
:tags: bug, engine
:tickets: 5326

Further refinements to the fixes to the "reset" agent fixed in
:ticket:`5326`, which now emits a warning when it is not being correctly
invoked and corrects for the behavior. Additional scenarios have been
identified and fixed where this warning was being emitted.


.. change::
:tags: usecase, sqlite
:tickets: 5297

SQLite 3.31 added support for computed column. This change
enables their support in SQLAlchemy when targeting SQLite.

.. change::
:tags: bug, schema
:tickets: 5276

Fixed issue where ``dialect_options`` were omitted when a
database object (e.g., :class:`.Table`) was copied using
:func:`.tometadata`.

.. change::
:tags: bug, sql
:tickets: 5344

Correctly apply self_group in type_coerce element.

The type coerce element did not correctly apply grouping rules when using
in an expression

.. change::
:tags: bug, oracle, reflection
:tickets: 5421

Fixed bug in Oracle dialect where indexes that contain the full set of
primary key columns would be mistaken as the primary key index itself,
which is omitted, even if there were multiples. The check has been refined
to compare the name of the primary key constraint against the index name
itself, rather than trying to guess based on the columns present in the
index.

.. change::
:tags: change, sql, sybase
:tickets: 5294

Added ``.offset`` support to sybase dialect.
Pull request courtesy Alan D. Snow.

.. change::
:tags: bug, engine
:tickets: 5341

Fixed issue in :class:`.URL` object where stringifying the object
would not URL encode special characters, preventing the URL from being
re-consumable as a real URL. Pull request courtesy Miguel Grinberg.

.. change::
:tags: usecase, mysql
:tickets: 4860

Implemented row-level locking support for mysql. Pull request courtesy
Quentin Somerville.

.. change::
:tags: change, mssql
:tickets: 5321

Moved the ``supports_sane_rowcount_returning = False`` requirement from
the ``PyODBCConnector`` level to the ``MSDialect_pyodbc`` since pyodbc
does work properly in some circumstances.

.. change::
:tags: change, examples

Added new option ``--raw`` to the examples.performance suite
which will dump the raw profile test for consumption by any
number of profiling visualizer tools. Removed the "runsnake"
option as runsnake is very hard to build at this point;

.. change::
:tags: bug, sql
:tickets: 5353

Added :meth:`.Select.with_hint` output to the generic SQL string that is
produced when calling ``str()`` on a statement. Previously, this clause
would be omitted under the assumption that it was dialect specific.
The hint text is presented within brackets to indicate the rendering
of such hints varies among backends.


.. change::
:tags: usecase, orm
:tickets: 5198

Added a new parameter :paramref:`_orm.query_expression.default_expr` to the
:func:`_orm.query_expression` construct, which will be appled to queries
automatically if the :func:`_orm.with_expression` option is not used. Pull
request courtesy Haoyu Sun.

.. changelog::

:released: May 13, 2020

.. change::
:tags: bug, oracle
:tickets: 5246

Some modifications to how the cx_oracle dialect sets up per-column
outputtype handlers for LOB and numeric datatypes to adjust for potential
changes coming in cx_Oracle 8.


.. change::
:tags: bug, orm
:tickets: 5288

Fixed bug where using :func:`.with_polymorphic` as the target of a join via
:meth:`.RelationshipComparator.of_type` on a mapper that already has a
subquery-based with_polymorphic setting that's equivalent to the one
requested would not correctly alias the ON clause in the join.

.. change::
:tags: bug, oracle, performance
:tickets: 5314

Changed the implementation of fetching CLOB and BLOB objects to use
cx_Oracle's native implementation which fetches CLOB/BLOB objects inline
with other result columns, rather than performing a separate fetch. As
always, this can be disabled by setting auto_convert_lobs to False.

As part of this change, the behavior of a CLOB that was given a blank
string on INSERT now returns None on SELECT, which is now consistent with
that of VARCHAR on Oracle.


.. change::
:tags: usecase, postgresql
:tickets: 5265

Added support for columns or type :class:`_sqltypes.ARRAY` of :class:`.Enum`,
:class:`_postgresql.JSON` or :class:`_postgresql.JSONB` in PostgreSQL.
Previously a workaround was required in these use cases.


.. change::
:tags: schema
:tickets: 4138

Add ``comment`` attribute to :class:`_schema.Column` ``__repr__`` method.

.. change::
:tags: bug, orm
:tickets: 5303

Fixed issue in the area of where loader options such as selectinload()
interact with the baked query system, such that the caching of a query is
not supposed to occur if the loader options themselves have elements such
as with_polymorphic() objects in them that currently are not
cache-compatible. The baked loader could sometimes not fully invalidate
itself in these some of these scenarios leading to missed eager loads.


.. change::
:tags: bug, engine
:tickets: 5326

Fixed fairly critical issue where the DBAPI connection could be returned to
the connection pool while still in an un-rolled-back state. The reset agent
responsible for rolling back the connection could be corrupted in the case
that the transaction was "closed" without being rolled back or committed,
which can occur in some scenarios when using ORM sessions and emitting
.close() in a certain pattern involving savepoints. The fix ensures that
the reset agent is always active.


.. change::
:tags: bug, orm
:tickets: 5304

Modified the internal "identity set" implementation, which is a set that
hashes objects on their id() rather than their hash values, to not actually
call the ``__hash__()`` method of the objects, which are typically
user-mapped objects. Some methods were calling this method as a side
effect of the implementation.


.. change::
:tags: usecase, postgresql
:tickets: 5266

Raise an explicit :class:`.exc.CompileError` when adding a table with a
column of type :class:`_sqltypes.ARRAY` of :class:`.Enum` configured with
:paramref:`.Enum.native_enum` set to ``False`` when
:paramref:`.Enum.create_constraint` is not set to ``False``

.. change::
:tags: bug, schema
:tickets: 5298

Fixed issue where an :class:`.Index` that is deferred in being associated
with a table, such as as when it contains a :class:`.Column` that is not
associated with any :class:`.Table` yet, would fail to attach correctly if
it also contained a non table-oriented expression.


.. change::
:tags: change, firebird
:tickets: 5278

Adjusted dialect loading for ``firebird://`` URIs so the external
sqlalchemy-firebird dialect will be used if it has been installed,
otherwise fall back to the (now deprecated) internal Firebird dialect.

.. change::
:tags: bug, mssql, reflection
:tickets: 5255

Fix a regression introduced by the reflection of computed column in
MSSQL when using the legacy TDS version 4.2. The dialect will try
to detect the protocol version of first connect and run in compatibility
mode if it cannot detect it.

.. change::
:tags: bug, mssql, reflection
:tickets: 5271

Fix a regression introduced by the reflection of computed column in
MSSQL when using SQL server versions before 2012, which does not support
the ``concat`` function.

.. change::
:tags: bug, orm
:tickets: 5269

An informative error message is raised when an ORM many-to-one comparison
is attempted against an object that is not an actual mapped instance.
Comparisons such as those to scalar subqueries aren't supported;
generalized comparison with subqueries is better achieved using
:meth:`~.RelationshipProperty.Comparator.has`.


.. change::
:tags: usecase, orm
:tickets: 5262

Added an accessor :attr:`.ColumnProperty.Comparator.expressions` which
provides access to the group of columns mapped under a multi-column
:class:`.ColumnProperty` attribute.


.. change::
:tags: bug, schema
:tickets: 5316

A warning is emitted when making use of the :attr:`.MetaData.sorted_tables`
attribute as well as the :func:`_schema.sort_tables` function, and the
given tables cannot be correctly sorted due to a cyclic dependency between
foreign key constraints. In this case, the functions will no longer sort
the involved tables by foreign key, and a warning will be emitted. Other
tables that are not part of the cycle will still be returned in dependency
order. Previously, the sorted_table routines would return a collection that
would unconditionally omit all foreign keys when a cycle was detected, and
no warning was emitted.


.. change::
:tags: orm, usecase
:tickets: 5237

Introduce :paramref:`_orm.relationship.sync_backref` flag in a relationship
to control if the synchronization events that mutate the in-Python
attributes are added. This supersedes the previous change :ticket:`5149`,
which warned that ``viewonly=True`` relationship target of a
back_populates or backref configuration would be disallowed.

.. changelog::