Auto Generating Migrations
===========================

Alembic can view the status of the database (pointed to by ``sqlalchemy.url`` in
your ``alembic.ini`` file using the *current* schema) and compare against the
table metadata in the application (your ORM which defines the *proposed* schema),
generating the "obvious" migrations based on a comparison.  This is achieved
using the ``--autogenerate`` option to the ``alembic revision`` command, which
places so-called *candidate* migrations into our new migrations file.  We
review and modify these by hand as needed, then proceed normally.

To use autogenerate, we first need to modify our ``env.py`` so that it gets access
to a table metadata object that contains the target.  Suppose our application
has a :ref:`declarative base <sqla:declarative_toplevel>`
in ``myapp.mymodel``.  This base contains a :class:`~sqlalchemy.schema.MetaData` object which
contains :class:`~sqlalchemy.schema.Table` objects defining our database.  We make sure this
is loaded in ``env.py`` and then passed to :meth:`.EnvironmentContext.configure` via the
``target_metadata`` argument.   The ``env.py`` sample script used in the
generic template already has a
variable declaration near the top for our convenience, where we replace ``None``
with our :class:`~sqlalchemy.schema.MetaData`.  Starting with::

    # add your model's MetaData object here
    # for 'autogenerate' support
    # from myapp import mymodel
    # target_metadata = mymodel.Base.metadata
    target_metadata = None

we change to::

    from myapp.mymodel import Base
    target_metadata = Base.metadata

.. note::

  The above example refers to the **generic alembic env.py template**, e.g.
  the one created by default when calling upon ``alembic init``, and not
  the special-use templates such as ``multidb``.   Please consult the source
  code and comments within the ``env.py`` script directly for specific
  guidance on where and how the autogenerate metadata is established.

If we look later in the script, down in ``run_migrations_online()``,
we can see the directive passed to :meth:`.EnvironmentContext.configure`::

    def run_migrations_online():
        engine = engine_from_config(
                    config.get_section(config.config_ini_section), prefix='sqlalchemy.')

        with engine.connect() as connection:
            context.configure(
                        connection=connection,
                        target_metadata=target_metadata
                        )

            with context.begin_transaction():
                context.run_migrations()

We can then use the ``alembic revision`` command in conjunction with the
``--autogenerate`` option.  Suppose
our :class:`~sqlalchemy.schema.MetaData` contained a definition for the ``account`` table,
and the database did not.  We'd get output like::

    $ alembic revision --autogenerate -m "Added account table"
    INFO [alembic.context] Detected added table 'account'
    Generating /path/to/foo/alembic/versions/27c6a30d7c24.py...done

We can then view our file ``27c6a30d7c24.py`` and see that a rudimentary migration
is already present::

    """empty message

    Revision ID: 27c6a30d7c24
    Revises: None
    Create Date: 2011-11-08 11:40:27.089406

    """

    # revision identifiers, used by Alembic.
    revision = '27c6a30d7c24'
    down_revision = None

    from alembic import op
    import sqlalchemy as sa

    def upgrade():
        ### commands auto generated by Alembic - please adjust! ###
        op.create_table(
        'account',
        sa.Column('id', sa.Integer()),
        sa.Column('name', sa.String(length=50), nullable=False),
        sa.Column('description', sa.VARCHAR(200)),
        sa.Column('last_transaction_date', sa.DateTime()),
        sa.PrimaryKeyConstraint('id')
        )
        ### end Alembic commands ###

    def downgrade():
        ### commands auto generated by Alembic - please adjust! ###
        op.drop_table("account")
        ### end Alembic commands ###

The migration hasn't actually run yet, of course.  We do that via the usual ``upgrade``
command.   We should also go into our migration file and alter it as needed, including
adjustments to the directives as well as the addition of other directives which these may
be dependent on - specifically data changes in between creates/alters/drops.

.. _autogenerate_detects:

What does Autogenerate Detect (and what does it *not* detect?)
--------------------------------------------------------------

The vast majority of user issues with Alembic centers on the topic of what
kinds of changes autogenerate can and cannot detect reliably, as well as
how it renders Python code for what it does detect.     It is critical to
note that **autogenerate is not intended to be perfect**.   It is *always*
necessary to manually review and correct the **candidate migrations**
that autogenerate produces.   The feature is getting more and more
comprehensive and error-free as releases continue, but one should take
note of the current limitations.

Autogenerate **will detect**:

* Table additions, removals.
* Column additions, removals.
* Change of nullable status on columns.
* Basic changes in indexes and explicitly-named unique constraints
* Basic changes in foreign key constraints

Autogenerate can **optionally detect**:

* Named CHECK constraint additions and removals.  This is **off by default**
  and must be enabled explicitly; the detection is name-based only and has
  significant limitations.  See :ref:`autogenerate_check_constraints` below.

* Change of column type.  This will occur by default unless the parameter
  :paramref:`.EnvironmentContext.configure.compare_type` is set to ``False``.
  The default implementation will reliably detect major changes,
  such as between :class:`.Numeric` and :class:`.String`, as well as
  accommodate for the types generated by SQLAlchemy's "generic" types such as
  :class:`.Boolean`.   Arguments that are shared between both types, such as
  length and precision values, will also be compared.   If either the metadata
  type or database type has **additional** arguments beyond that of the other
  type, these are **not** compared, such as if one numeric type featured a
  "scale" and other type did not, this would be seen as the backing database
  not supporting the value, or reporting on a default that the metadata did not
  specify.

  The type comparison logic is fully extensible as well; see
  :ref:`compare_types` for details.

* Change of server default.  This will occur if you set
  the :paramref:`.EnvironmentContext.configure.compare_server_default`
  parameter to ``True``, or to a custom callable function.
  This feature works well for simple cases but cannot always produce
  accurate results.  The Postgresql backend will actually invoke
  the "detected" and "metadata" values against the database to
  determine equivalence.  The feature is off by default so that
  it can be tested on the target schema first.  Like type comparison,
  it can also be customized by passing a callable; see the
  function's documentation for details.

Autogenerate **can not detect**:

* Changes of table name.   These will come out as an add/drop of two different
  tables, and should be hand-edited into a name change instead.
* Changes of column name.  Like table name changes, these are detected as
  a column add/drop pair, which is not at all the same as a name change.
* Anonymously named constraints.  Give your constraints a name,
  e.g. ``UniqueConstraint('col1', 'col2', name="my_name")``.  See the section
  :doc:`naming` for background on how to configure automatic naming schemes
  for constraints.
* Special SQLAlchemy types such as :class:`~sqlalchemy.types.Enum` when generated
  on a backend which doesn't support ENUM directly - this because the
  representation of such a type
  in the non-supporting database, i.e. a CHAR+ CHECK constraint, could be
  any kind of CHAR+CHECK.  For SQLAlchemy to determine that this is actually
  an ENUM would only be a guess, something that's generally a bad idea.
  To implement your own "guessing" function here, use the
  :meth:`sqlalchemy.events.DDLEvents.column_reflect` event
  to detect when a CHAR (or whatever the target type is) is reflected,
  and change it to an ENUM (or whatever type is desired) if it is known that
  that's the intent of the type.  The
  :meth:`sqlalchemy.events.DDLEvents.after_parent_attach`
  can be used within the autogenerate process to intercept and un-attach
  unwanted CHECK constraints.

Autogenerate can't currently, but **will eventually detect**:

* Some free-standing constraint additions and removals may not be supported,
  including PRIMARY KEY, EXCLUDE; these are not necessarily implemented
  within the autogenerate detection system and also may not be supported by
  the supporting SQLAlchemy dialect.
* Sequence additions, removals - not yet implemented.

.. _autogenerate_check_constraints:

Detecting CHECK Constraints
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. versionadded:: 1.19.0

.. versionchanged:: 1.19.2  This feature is no longer enabled by default.
   See the warning below for the rationale.

Autogenerate includes optional, **off by default** support for detecting the
addition and removal of **named** CHECK constraints.  The detection is
**name-based only**: a metadata-side CHECK constraint and a reflected CHECK
constraint that share the same name are always presumed to be equivalent,
regardless of any difference in their expression text.

The feature is implemented as a built-in
:ref:`plugin <alembic.plugins.toplevel>` named
``alembic.ext.checkconstraint_byname``.  Unlike Alembic's other built-in
autogenerate plugins, it is named outside of the ``alembic.autogenerate.``
namespace and is therefore **not** part of the ``alembic.autogenerate.*``
wildcard; it must be named explicitly in the
:paramref:`.EnvironmentContext.configure.autogenerate_plugins` list in order
to be used::

    context.configure(
        # ...
        autogenerate_plugins=[
            "alembic.autogenerate.*",
            "alembic.ext.checkconstraint_byname",
        ]
    )

The plugin was named ``alembic.autogenerate.checkconstraint_byname`` in
versions 1.19.0 and 1.19.1.  That name continues to be accepted, both to
enable the plugin and within a ``"~"`` exclusion, so an ``env.py`` written
against those releases requires no change.

.. warning::

    This feature has significant limitations and should be enabled only for
    schemas that fit the narrow set of conditions it can handle.   Before
    turning it on, note all of the following:

    * **Expression changes are never detected.**  Two CHECK constraints that
      share a name are presumed equal even if their SQL expressions differ
      entirely.  A migration that alters the condition of an existing
      constraint will not be generated.

    * **Every CHECK constraint in the target metadata must be named.**  A
      CHECK constraint with no name, whether written directly as
      ``CheckConstraint("x > 5")`` or left unnamed by a naming convention
      that does not produce a name for it, is skipped on the metadata side.
      Backends such as PostgreSQL nonetheless assign a name to every
      constraint in the database, so the reflected constraint matches
      nothing in the metadata and is reported as a **spurious removal**.
      This includes CHECK constraints generated by types such as
      :class:`~sqlalchemy.types.Enum` and :class:`~sqlalchemy.types.Boolean`
      that have ``create_constraint=True``; assign a ``name`` to these
      types to make the CHECK constraints named.

      .. versionchanged:: 1.19.2  CHECK constraints that are "type bound" are
          no longer ignored on the metadata side, so that normal name-based
          matching can occur.


    * **Not all backends reflect CHECK constraints**, in which case the
      plugin silently does nothing.

    Unlike unique constraints, which fall back to matching on column names
    when the metadata side is unnamed, there is no comparable fallback
    available for CHECK constraints.   Matching on the expression instead of
    the name is not feasible, because dialects are free to normalize,
    reformat, or otherwise rewrite the original DDL text.   PostgreSQL, for
    example, reflects ``x != y`` as ``x <> y``, ``x BETWEEN 1 AND 10`` as
    ``x >= 1 AND x <= 10``, and ``s IN ('a', 'b')`` as
    ``s::text = ANY (ARRAY['a'::character varying, ...])``.  Comparing that
    text against the metadata expression would report a change on nearly
    every constraint.

Given the above, the feature is most appropriate for a schema in which all
CHECK constraints are explicitly named, ideally via a naming convention (see
:doc:`naming`).   As always, the candidate migrations that autogenerate
produces must be reviewed by hand.

Notable 3rd-party libraries that extend the built-in Alembic autogenerate functionality
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* `alembic-utils <https://github.com/olirice/alembic_utils>`_
  A library that adds autogenerate support for PostgreSQL functions, views, triggers, etc.
* `alembic-postgresql-enum <https://pypi.org/project/alembic-postgresql-enum>`_
  A library that adds autogenerate support for creation, alteration and deletion of Enums in PostgreSQL.

Autogenerating Multiple MetaData collections
--------------------------------------------

The ``target_metadata`` collection may also be defined as a sequence
if an application has multiple :class:`~sqlalchemy.schema.MetaData`
collections involved::

    from myapp.mymodel1 import Model1Base
    from myapp.mymodel2 import Model2Base
    target_metadata = [Model1Base.metadata, Model2Base.metadata]

The sequence of :class:`~sqlalchemy.schema.MetaData` collections will be
consulted in order during the autogenerate process.  Note that each
:class:`~sqlalchemy.schema.MetaData` must contain **unique** table keys
(e.g. the "key" is the combination of the table's name and schema);
if two :class:`~sqlalchemy.schema.MetaData` objects contain a table
with the same schema/name combination, an error is raised.

.. _autogenerate_include_hooks:

Controlling What to be Autogenerated
------------------------------------

The autogenerate process scans across all table objects within
the database that is referred towards by the current database connection
in use.

The list of objects that are scanned in the target database connection include:

* The "default" schema currently referred towards by the database connection.

* If the :paramref:`.EnvironmentContext.configure.include_schemas` is set to
  ``True``, all non-default "schemas", which are those names returned by the
  :meth:`~sqlalchemy.engine.reflection.Inspector.get_schema_names` method of
  :class:`~sqlalchemy.engine.reflection.Inspector`.    The SQLAlchemy document
  :ref:`sqla:schema_table_schema_name` discusses the concept of a
  "schema" in detail.

* Within each "schema", all tables present are scanned using the
  :meth:`~sqlalchemy.engine.reflection.Inspector.get_table_names` method of
  :class:`~sqlalchemy.engine.reflection.Inspector`.

* Within each "table", most sub-objects of the each
  :class:`~sqlalchemy.schema.Table` construct are scanned, including columns
  and some forms of constraints.  This process ultimately involves the use of
  methods on :class:`~sqlalchemy.engine.reflection.Inspector` including
  :meth:`~sqlalchemy.engine.reflection.Inspector.get_columns`,
  :meth:`~sqlalchemy.engine.reflection.Inspector.get_indexes`,
  :meth:`~sqlalchemy.engine.reflection.Inspector.get_unique_constraints`,
  :meth:`~sqlalchemy.engine.reflection.Inspector.get_foreign_keys` (as of this
  writing, CHECK constraints and primary key constraints are not yet included).

.. seealso::

    :ref:`sqla:schema_table_schema_name` - in depth introduction to how
    SQLAlchemy interprets schema names

    :ref:`sqla:postgresql_schema_reflection` - important notes specific to the
    PostgreSQL database

Omitting Schema Names from the Autogenerate Process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As the above set of database objects are typically to be compared to the contents of
a single :class:`~sqlalchemy.schema.MetaData` object, particularly when the
:paramref:`.EnvironmentContext.configure.include_schemas` flag is enabled
there is an important need to filter out unwanted "schemas", which for some
database backends might be the list of all the databases present.   This
filtering is best performed using the :paramref:`.EnvironmentContext.configure.include_name`
hook, which provides for a callable that may return a boolean true/false
indicating if a particular schema name should be included::

    def include_name(name, type_, parent_names):
        if type_ == "schema":
            # note this will not include the default schema
            return name in ["schema_one", "schema_two"]
        else:
            return True

    context.configure(
        # ...
        include_schemas = True,
        include_name = include_name
    )

Above, when the list of schema names is first retrieved, the names will be
filtered through the above ``include_name`` function so that only schemas
named ``"schema_one"`` and ``"schema_two"`` will be considered by the
autogenerate process.

In order to include **the default schema**, that is, the schema that is
referred towards by the database connection **without** any explicit
schema being specified, the name passed to the hook is ``None``.  To alter
our above example to also include the default schema, we compare to
``None`` as well::

    def include_name(name, type_, parent_names):
        if type_ == "schema":
            # this **will* include the default schema
            return name in [None, "schema_one", "schema_two"]
        else:
            return True

    context.configure(
        # ...
        include_schemas = True,
        include_name = include_name
    )

Omitting Table Names from the Autogenerate Process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The :paramref:`.EnvironmentContext.configure.include_name` hook is also
most appropriate to limit the names of tables in the target database
to be considered.  If a target database has many tables that are not
part of the :class:`~sqlalchemy.schema.MetaData`, the autogenerate process
will normally assume these are extraneous tables in the database to be
dropped, and it will generate a :meth:`.Operations.drop_table` operation
for each.  To prevent this, the :paramref:`.EnvironmentContext.configure.include_name`
hook may be used to search for each name within the
:attr:`~sqlalchemy.schema.MetaData.tables` collection of the
:class:`~sqlalchemy.schema.MetaData` object and ensure names
which aren't present are not included::

    target_metadata = MyModel.metadata

    def include_name(name, type_, parent_names):
        if type_ == "table":
            return name in target_metadata.tables
        else:
            return True

    context.configure(
        # ...
        target_metadata = target_metadata,
        include_name = include_name,
        include_schemas = False
    )

The above example is limited to table names present in the default schema only.
In order to search within a :class:`~sqlalchemy.schema.MetaData` collection for
schema-qualified table names as well, a table present in the non
default schema will be present under a name of the form
``<schemaname>.<tablename>``.   The
:paramref:`.EnvironmentContext.configure.include_name` hook will present
this schema name on a per-tablename basis in the ``parent_names`` dictionary,
using the key ``"schema_name"`` that refers to the name of the
schema currently being considered, or ``None`` if the schema is the default
schema of the database connection::

    # example fragment

    if parent_names["schema_name"] is None:
        return name in target_metadata.tables
    else:
        # build out schema-qualified name explicitly...
        return (
            "%s.%s" % (parent_names["schema_name"], name) in
            target_metadata.tables
        )

However more simply, the ``parent_names`` dictionary will also include
the dot-concatenated name already constructed under the key
``"schema_qualified_table_name"``, which will also be suitably formatted
for tables in the default schema as well with the dot omitted.  So the
full example of omitting tables with schema support may look like::

    target_metadata = MyModel.metadata

    def include_name(name, type_, parent_names):
        if type_ == "schema":
            return name in [None, "schema_one", "schema_two"]
        elif type_ == "table":
            # use schema_qualified_table_name directly
            return (
                parent_names["schema_qualified_table_name"] in
                target_metadata.tables
            )
        else:
            return True

    context.configure(
        # ...
        target_metadata = target_metadata,
        include_name = include_name,
        include_schemas = True
    )

The ``parent_names`` dictionary will also include the key ``"table_name"``
when the name being considered is that of a column or constraint object
local to a particular table.

The :paramref:`.EnvironmentContext.configure.include_name` hook only refers
to **reflected** objects, and not those located within the target
:class:`~sqlalchemy.schema.MetaData` collection.   For more fine-grained
rules that include both :class:`~sqlalchemy.schema.MetaData` and reflected
object, the :paramref:`.EnvironmentContext.configure.include_object` hook
discussed in the next section is more appropriate.

Omitting Based on Object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The :paramref:`.EnvironmentContext.configure.include_object` hook provides
for object-level inclusion/exclusion rules based on the :class:`~sqlalchemy.schema.Table`
object being reflected as well as the elements within it.   This hook can
be used to limit objects both from the local :class:`~sqlalchemy.schema.MetaData`
collection as well as from the target database.   The limitation is that when
it reports on objects in the database, it will have fully reflected that object,
which can be expensive if a large number of objects will be omitted.  The
example below refers to a fine-grained rule that will skip changes on
:class:`~sqlalchemy.schema.Column` objects that have a user-defined flag
``skip_autogenerate`` placed into the :attr:`~sqlalchemy.schema.Column.info`
dictionary::

    def include_object(object, name, type_, reflected, compare_to):
        if (type_ == "column" and
            not reflected and
            object.info.get("skip_autogenerate", False)):
            return False
        else:
            return True

    context.configure(
        # ...
        include_object = include_object
    )



Comparing and Rendering Types
------------------------------

The area of autogenerate's behavior of comparing and rendering Python-based type objects
in migration scripts presents a challenge, in that there's
a very wide variety of types to be rendered in scripts, including those
part of SQLAlchemy as well as user-defined types.   A few options
are given to help out with this task.

.. _autogen_module_prefix:

Controlling the Module Prefix
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When types are rendered, they are generated with a **module prefix**, so
that they are available based on a relatively small number of imports.
The rules for what the prefix is is based on the kind of datatype as well
as configurational settings.   For example, when Alembic renders SQLAlchemy
types, it will by default prefix the type name with the prefix ``sa.``::

    Column("my_column", sa.Integer())

The use of the ``sa.`` prefix is controllable by altering the value
of :paramref:`.EnvironmentContext.configure.sqlalchemy_module_prefix`::

    def run_migrations_online():
        # ...

        context.configure(
                    connection=connection,
                    target_metadata=target_metadata,
                    sqlalchemy_module_prefix="sqla.",
                    # ...
                    )

        # ...

In either case, the ``sa.`` prefix, or whatever prefix is desired, should
also be included in the imports section of ``script.py.mako``; it also
defaults to ``import sqlalchemy as sa``.


For user-defined types, that is, any custom type that
is not within the ``sqlalchemy.`` module namespace, by default Alembic will
use the **value of __module__ for the custom type**::

    Column("my_column", myapp.models.utils.types.MyCustomType())

The imports for the above type again must be made present within the migration,
either manually, or by adding it to ``script.py.mako``.

The above custom type has a long and cumbersome name based on the use
of ``__module__`` directly, which also implies that lots of imports would
be needed in order to accommodate lots of types.  For this reason, it is
recommended that user-defined types used in migration scripts be made
available from a single module.  Suppose we call it ``myapp.migration_types``::

    # myapp/migration_types.py

    from myapp.models.utils.types import MyCustomType

We can first add an import for ``migration_types`` to our ``script.py.mako``::

    from alembic import op
    import sqlalchemy as sa
    import myapp.migration_types
    ${imports if imports else ""}

We then override Alembic's use of ``__module__`` by providing a fixed
prefix, using the :paramref:`.EnvironmentContext.configure.user_module_prefix`
option::

    def run_migrations_online():
        # ...

        context.configure(
                    connection=connection,
                    target_metadata=target_metadata,
                    user_module_prefix="myapp.migration_types.",
                    # ...
                    )

        # ...

Above, we now would get a migration like::

  Column("my_column", myapp.migration_types.MyCustomType())

Now, when we inevitably refactor our application to move ``MyCustomType``
somewhere else, we only need modify the ``myapp.migration_types`` module,
instead of searching and replacing all instances within our migration scripts.

.. _autogen_render_types:

Affecting the Rendering of Types Themselves
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The methodology Alembic uses to generate SQLAlchemy and user-defined type constructs
as Python code is plain old ``__repr__()``.   SQLAlchemy's built-in types
for the most part have a ``__repr__()`` that faithfully renders a
Python-compatible constructor call, but there are some exceptions, particularly
in those cases when a constructor accepts arguments that aren't compatible
with ``__repr__()``, such as a pickling function.

When building a custom type that will be rendered into a migration script,
it is often necessary to explicitly give the type a ``__repr__()`` that will
faithfully reproduce the constructor for that type.   This, in combination
with :paramref:`.EnvironmentContext.configure.user_module_prefix`, is usually
enough.  However, if additional behaviors are needed, a more comprehensive
hook is the :paramref:`.EnvironmentContext.configure.render_item` option.
This hook allows one to provide a callable function within ``env.py`` that will fully take
over how a type is rendered, including its module prefix::

    def render_item(type_, obj, autogen_context):
        """Apply custom rendering for selected items."""

        if type_ == 'type' and isinstance(obj, MySpecialType):
            return "mypackage.%r" % obj

        # default rendering for other objects
        return False

    def run_migrations_online():
        # ...

        context.configure(
                    connection=connection,
                    target_metadata=target_metadata,
                    render_item=render_item,
                    # ...
                    )

        # ...

In the above example, we'd ensure our ``MySpecialType`` includes an appropriate
``__repr__()`` method, which is invoked when we call it against ``"%r"``.

The callable we use for :paramref:`.EnvironmentContext.configure.render_item`
can also add imports to our migration script.  The :class:`.AutogenContext` passed in
contains a datamember called :attr:`.AutogenContext.imports`, which is a Python
``set()`` for which we can add new imports.  For example, if ``MySpecialType``
were in a module called ``mymodel.types``, we can add the import for it
as we encounter the type::

    def render_item(type_, obj, autogen_context):
        """Apply custom rendering for selected items."""

        if type_ == 'type' and isinstance(obj, MySpecialType):
            # add import for this type
            autogen_context.imports.add("from mymodel import types")
            return "types.%r" % obj

        # default rendering for other objects
        return False

The finished migration script will include our imports where the
``${imports}`` expression is used, producing output such as::

  from alembic import op
  import sqlalchemy as sa
  from mymodel import types

  def upgrade():
      op.add_column('sometable', Column('mycolumn', types.MySpecialType()))


.. _compare_types:

Comparing Types
^^^^^^^^^^^^^^^^

The default type comparison logic will work for SQLAlchemy built in types as
well as basic user defined types.   This logic is enabled by default.
It can be disabled by setting the
:paramref:`.EnvironmentContext.configure.compare_type` to ``False``::

    context.configure(
        # ...
        compare_type = False
    )

.. versionchanged:: 1.12.0 The default value of
   :paramref:`.EnvironmentContext.configure.compare_type` has been changed
   to ``True``.

.. note::

   The default type comparison logic (which is end-user extensible) currently
   (as of Alembic version 1.4.0) works by comparing the generated SQL for a
   column. It does this in two steps-

   * First, it compares the outer type of each column such as ``VARCHAR``
     or ``TEXT``. Dialect implementations can have synonyms that are considered
     equivalent, this is because some databases support types by converting them
     to another type. For example, NUMERIC and DECIMAL are considered equivalent
     on all backends, while on the Oracle backend the additional synonyms
     BIGINT, INTEGER, NUMBER, SMALLINT are added to this list of equivalents

   * Next, the arguments within the type, such as the lengths of
     strings, precision values for numerics, the elements inside of an
     enumeration are compared. If BOTH columns have arguments AND they are
     different, a change will be detected. If one column is just set to the
     default and the other has arguments, Alembic will pass on attempting to
     compare these. The rationale is that it is difficult to detect what a
     database backend sets as a default value without generating false
     positives.

Alternatively, the :paramref:`.EnvironmentContext.configure.compare_type`
parameter accepts a callable function which may be used to implement custom type
comparison logic, for cases such as where special user defined types
are being used::

    def my_compare_type(context, inspected_column,
                metadata_column, inspected_type, metadata_type):
        # return False if the metadata_type is the same as the inspected_type
        # or None to allow the default implementation to compare these
        # types. a return value of True means the two types do not
        # match and should result in a type change operation.
        return None

    context.configure(
        # ...
        compare_type = my_compare_type
    )

Above, ``inspected_column`` is a :class:`sqlalchemy.schema.Column` as
returned by
:meth:`sqlalchemy.engine.reflection.Inspector.reflect_table`, whereas
``metadata_column`` is a :class:`sqlalchemy.schema.Column` from the
local model environment.  A return value of ``None`` indicates that default
type comparison to proceed.


.. _post_write_hooks:

Applying Post Processing and Python Code Formatters to Generated Revisions
---------------------------------------------------------------------------

Revision scripts generated by the ``alembic revision`` command can optionally
be piped through a series of post-production functions which may analyze or
rewrite Python source code generated by Alembic, within the scope of running
the ``revision`` command.   The primary intended use of this feature is to run
code-formatting tools such as `Black <https://black.readthedocs.io/>`_ or
`autopep8 <https://pypi.org/project/autopep8/>`_, as well as custom-written
formatting and linter functions, on revision files as Alembic generates them.
Any number of hooks can be configured and they will be run in series, given the
path to the newly generated file as well as configuration options.

The post write hooks, when configured,  run against generated revision files
regardless of whether or not the autogenerate feature was used.

.. note::

    Alembic's post write system is partially inspired by the `pre-commit
    <https://pre-commit.com/>`_ tool, which configures git hooks that reformat
    source files as they are committed to a git repository.  Pre-commit can
    serve this role for Alembic revision files as well, applying code
    formatters to them as they are committed.  Alembic's post write hooks are
    useful only in that they can format the files immediately upon generation,
    rather than at commit time, and also can be useful for projects that prefer
    not to use pre-commit.

.. _post_write_hooks_config:

Basic Post Processor Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The template samples for ``alembic.ini`` as well as ``pyproject.toml`` for
applicable templates now include commented-out configuration
illustrating how to configure code-formatting tools, or other tools like linters
to run against the newly generated file path.  Example from an alembic.ini file::

    [post_write_hooks]

    # format using "black"
    hooks=black

    black.type = console_scripts
    black.entrypoint = black
    black.options = -l 79 REVISION_SCRIPT_FILENAME

The same example configured in a pyproject.toml file would look like::

    [[tool.alembic.post_write_hooks]]

    # format using "black"
    name = "black"
    type = "console_scripts"
    entrypoint = "black"
    options = "-l 79 REVISION_SCRIPT_FILENAME"

Above, we configure ``hooks`` to be a single post write hook labeled
``"black"``.   Note that this label is arbitrary.   We then define the
configuration for the ``"black"`` post write hook, which includes:

* ``type`` - this is the type of hook we are running.  Alembic includes
  three hook runners:

  * ``"console_scripts"``, which is specifically a Python function that uses
    ``subprocess.run()`` to invoke a separate Python script against the revision file;
  * ``"exec"``, which uses ``subprocess.run()`` to execute an arbitrary binary; and
  * ``"module"``, which uses ``subprocess.run()`` to invoke a Python module directly.

  For a custom-written hook function, this configuration variable would
  refer to the name under which the custom hook was registered; see the
  next section for an example.

.. versionadded:: 1.12 added new ``exec`` runner

.. versionadded:: 1.16.3 added new ``module`` runner


The following configuration option is specific to the ``"console_scripts"``
hook runner:

* ``entrypoint`` - the name of the `setuptools entrypoint <https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points>`_
  that is used to define the console script.   Within the scope of standard
  Python console scripts, this name will match the name of the shell command
  that is usually run for the code formatting tool, in this case ``black``.

The following configuration option is specific to the ``"exec"`` hook runner:

* ``executable`` - the name of the executable to invoke.  Can be either a
  bare executable name which will be searched in ``$PATH``, or a full pathname
  to avoid potential issues with path interception.

The following options are supported by both ``"console_scripts"`` and ``"exec"``:

* ``options`` - a line of command-line options that will be passed to
  the code formatting tool.  In this case, we want to run the command
  ``black /path/to/revision.py -l 79``.  By default, the revision path is
  positioned as the first argument.  In order specify a different position,
  we can use the ``REVISION_SCRIPT_FILENAME`` token as illustrated by the
  subsequent examples.

  .. note:: Make sure options for the script are provided such that it will
     rewrite the input file **in place**.  For example, when running
     ``autopep8``, the ``--in-place`` option should be provided::

        [post_write_hooks]
        hooks = autopep8
        autopep8.type = console_scripts
        autopep8.entrypoint = autopep8
        autopep8.options = --in-place REVISION_SCRIPT_FILENAME

* ``cwd`` - optional working directory from which the code processing tool is run.

When running ``alembic revision -m "rev1"``, we will now see the ``black``
tool's output as well::

  $ alembic revision -m "rev1"
    Generating /path/to/project/versions/481b13bc369a_rev1.py ... done
    Running post write hook "black" ...
  reformatted /path/to/project/versions/481b13bc369a_rev1.py
  All done! ✨ 🍰 ✨
  1 file reformatted.
    done

Hooks may also be specified as a list of names, which correspond to hook
runners that will run sequentially.  As an example, we can also run the
`zimports <https://pypi.org/project/zimports/>`_ import rewriting tool (written
by Alembic's author) subsequent to running the ``black`` tool.   The
alembic.ini configuration would be as follows::

    [post_write_hooks]

    # format using "black", then "zimports"
    hooks=black, zimports

    black.type = console_scripts
    black.entrypoint = black
    black.options = -l 79 REVISION_SCRIPT_FILENAME

    zimports.type = console_scripts
    zimports.entrypoint = zimports
    zimports.options = --style google REVISION_SCRIPT_FILENAME

The equivalent pyproject.toml configuration would be::

    # format using "black", then "zimports"

    [[tool.alembic.post_write_hooks]]
    name = "black"
    type="console_scripts"
    entrypoint = "black"
    options = "-l 79 REVISION_SCRIPT_FILENAME"

    [[tool.alembic.post_write_hooks]]
    name = "zimports"
    type="console_scripts"
    entrypoint = "zimports"
    options = "--style google REVISION_SCRIPT_FILENAME"

When using the above configuration, a newly generated revision file will
be processed first by the "black" tool, then by the "zimports" tool.

Alternatively, one can run pre-commit itself as follows::

  [post_write_hooks]

  hooks = pre-commit

  pre-commit.type = console_scripts
  pre-commit.entrypoint = pre-commit
  pre-commit.options = run --files REVISION_SCRIPT_FILENAME
  pre-commit.cwd = %(here)s

(The last line helps to ensure that the ``.pre-commit-config.yaml`` file
will always be found, regardless of from where the hook was called.)

.. _post_write_hooks_custom:

Writing Custom Hooks as Python Functions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The previous section illustrated how to run command-line code formatters,
through the use of a post write hook provided by Alembic known as
``console_scripts``.  This hook is in fact a Python function that is registered
under that name using a registration function that may be used to register
other types of hooks as well.

To illustrate, we will use the example of a short Python function that wants
to rewrite the generated code to use tabs instead of four spaces.   For simplicity,
we will illustrate how this function can be present directly in the ``env.py``
file.   The function is declared and registered using the :func:`.write_hooks.register`
decorator::

    from alembic.script import write_hooks
    import re

    @write_hooks.register("spaces_to_tabs")
    def convert_spaces_to_tabs(filename, options):
        lines = []
        with open(filename) as file_:
            for line in file_:
                lines.append(
                    re.sub(
                        r"^(    )+",
                        lambda m: "\t" * (len(m.group(1)) // 4),
                        line
                    )
                )
        with open(filename, "w") as to_write:
            to_write.write("".join(lines))

Our new ``"spaces_to_tabs"`` hook can be configured in alembic.ini as follows::

  [alembic]

  # ...

  # ensure the revision command loads env.py
  revision_environment = true

  [post_write_hooks]

  hooks = spaces_to_tabs

  spaces_to_tabs.type = spaces_to_tabs


When ``alembic revision`` is run, the ``env.py`` file will be loaded in all
cases, the custom "spaces_to_tabs" function will be registered and it will then
be run against the newly generated file path::

  $ alembic revision -m "rev1"
    Generating /path/to/project/versions/481b13bc369a_rev1.py ... done
    Running post write hook "spaces_to_tabs" ...
    done

.. _alembic_check:

Running Alembic Check to test for new upgrade operations
--------------------------------------------------------

When developing code it's useful to know if a set of code changes has made any
net change to the database model, such that new revisions would need to be
generated. To automate this, Alembic provides the ``alembic check`` command.
This command will run through the same process as
``alembic revision --autogenerate``, up until the point where revision files
would be generated, however does not generate any new files. Instead, it
returns an error code plus a message if it is detected that new operations
would be rendered into a new revision, or if not, returns a success code plus a
message.   When ``alembic check`` returns a success code, this is an indication
that the ``alembic revision --autogenerate`` command would produce only empty
migrations, and does not need to be run.

``alembic check`` can be worked into CI systems and on-commit schemes to ensure
that incoming code does not warrant new revisions to be generated.  In
the example below, a check that detects new operations is illustrated::


    $ alembic check
    FAILED: New upgrade operations detected: [
      ('add_column', None, 'my_table', Column('data', String(), table=<my_table>)),
      ('add_column', None, 'my_table', Column('newcol', Integer(), table=<my_table>))]

by contrast, when no new operations are detected::

    $ alembic check
    No new upgrade operations detected.


.. versionadded:: 1.9.0

.. note::  The ``alembic check`` command uses the same model comparison process
   as the ``alembic revision --autogenerate`` process.  This means parameters
   such as :paramref:`.EnvironmentContext.configure.compare_type`
   and :paramref:`.EnvironmentContext.configure.compare_server_default`
   are in play as usual, as well as that limitations in autogenerate
   detection are the same when running ``alembic check``.
