{"id":15973736,"url":"https://github.com/c-bata/alembic-tmp-table-issue","last_synced_at":"2025-08-30T15:14:21.157Z","repository":{"id":60351124,"uuid":"542529244","full_name":"c-bata/alembic-tmp-table-issue","owner":"c-bata","description":null,"archived":false,"fork":false,"pushed_at":"2022-09-29T04:06:29.000Z","size":15,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-10T01:46:19.351Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/sqlalchemy/alembic/discussions/1092","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c-bata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-28T10:19:10.000Z","updated_at":"2022-09-29T08:42:04.000Z","dependencies_parsed_at":"2023-01-19T00:31:11.957Z","dependency_job_id":null,"html_url":"https://github.com/c-bata/alembic-tmp-table-issue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Falembic-tmp-table-issue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Falembic-tmp-table-issue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Falembic-tmp-table-issue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Falembic-tmp-table-issue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c-bata","download_url":"https://codeload.github.com/c-bata/alembic-tmp-table-issue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208404,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-07T21:06:54.203Z","updated_at":"2025-04-04T16:21:21.736Z","avatar_url":"https://github.com/c-bata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# alembic-tmp-table-issue\n\n## How to reproduce the issue\n\n### Setup Python Environment\n\n```\n$ python3.9 -m venv venv\n$ source venv/bin/activate\n$ pip install alembic sqlalchemy\n```\n\n### Create `foo` Table\n\n```\ndef upgrade() -\u003e None:\n    # Create table\n    op.create_table(\n        \"foo\",\n        sa.Column(\"foo_id\", sa.Integer(), nullable=False),\n        sa.Column(\"bar\", sa.String(length=512), nullable=False),\n        sa.PrimaryKeyConstraint(\"foo_id\"),\n    )\n\n    # Insert Dummy Data\n    op.bulk_insert(...)\n```\n\n```\n$ alembic upgrade 90823bd3809e\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO  [alembic.runtime.migration] Running upgrade  -\u003e 90823bd3809e, create foo table\n```\n\n### Run a schema migration and a data migration\n\nThe migration script is like this:\n\n```\ndef upgrade() -\u003e None:\n    # Schema Migration\n    with op.batch_alter_table(\"foo\") as batch_op:\n        batch_op.alter_column(\n            \"bar\",\n            nullable=True,\n        )\n\n    # Write a data migration here like the following pattern.\n    # https://alembic.sqlalchemy.org/en/latest/cookbook.html#conditional-migration-elements\n    ...\n    raise Exception(\"An exception is raised during the data migration\")\n```\n\nThe script raises `Exception(...)` deliberately after quit from the `batch_alter_table(\"foo\")` context manager.\nSo the `alembic upgrade head` will be failed like the following:\n\n```\n$ alembic upgrade head\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO  [alembic.runtime.migration] Running upgrade 90823bd3809e -\u003e 8e32f2af1bd2, Add schema \u0026 data migration\nTraceback (most recent call last):\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/bin/alembic\", line 8, in \u003cmodule\u003e\n    sys.exit(main())\n  ...\n  File \"/Users/c-bata/sandbox/alembic-sandbox/batchop/versions/8e32f2af1bd2_add_schema_data_migration.py\", line 35, in upgrade\n    raise Exception(\"An exception is raised during data migration\")\nException: An exception is raised during data migration\n```\n\nThis is an expected behavior. However, the script cannot be rerun even if the data migration is fixed to avoid the exception since there is `_alembic_tmp_foo` table.\n\n```\n$ sqlite3 alembic.db\nSQLite version 3.37.0 2021-12-09 01:34:53\nEnter \".help\" for usage hints.\nsqlite\u003e .tables\n_alembic_tmp_foo  alembic_version   foo\n```\n\n```\n$ alembic upgrade head\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO  [alembic.runtime.migration] Running upgrade 90823bd3809e -\u003e 8e32f2af1bd2, Add schema \u0026 data migration\nTraceback (most recent call last):\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1900, in _execute_context\n    self.dialect.do_execute(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py\", line 736, in do_execute\n    cursor.execute(statement, parameters)\nsqlite3.OperationalError: table _alembic_tmp_foo already exists\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/bin/alembic\", line 8, in \u003cmodule\u003e\n    sys.exit(main())\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/config.py\", line 590, in main\n    CommandLine(prog=prog).main(argv=argv)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/config.py\", line 584, in main\n    self.run_cmd(cfg, options)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/config.py\", line 561, in run_cmd\n    fn(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/command.py\", line 322, in upgrade\n    script.run_env()\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/script/base.py\", line 569, in run_env\n    util.load_python_file(self.dir, \"env.py\")\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py\", line 94, in load_python_file\n    module = load_module_py(module_id, path)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py\", line 110, in load_module_py\n    spec.loader.exec_module(module)  # type: ignore\n  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 883, in exec_module\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 241, in _call_with_frames_removed\n  File \"/Users/c-bata/sandbox/alembic-sandbox/batchop/env.py\", line 79, in \u003cmodule\u003e\n    run_migrations_online()\n  File \"/Users/c-bata/sandbox/alembic-sandbox/batchop/env.py\", line 73, in run_migrations_online\n    context.run_migrations()\n  File \"\u003cstring\u003e\", line 8, in run_migrations\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/runtime/environment.py\", line 853, in run_migrations\n    self.get_context().run_migrations(**kw)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/runtime/migration.py\", line 623, in run_migrations\n    step.migration_fn(**kw)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/batchop/versions/8e32f2af1bd2_add_schema_data_migration.py\", line 28, in upgrade\n    with op.batch_alter_table(\"foo\") as batch_op:\n  File \"/Users/c-bata/.pyenv_x64/versions/3.10.4/lib/python3.10/contextlib.py\", line 142, in __exit__\n    next(self.gen)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/operations/base.py\", line 381, in batch_alter_table\n    impl.flush()\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/operations/batch.py\", line 159, in flush\n    batch_impl._create(self.impl)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/operations/batch.py\", line 448, in _create\n    op_impl.create_table(self.new_table)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/ddl/impl.py\", line 354, in create_table\n    self._exec(schema.CreateTable(table))\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/ddl/impl.py\", line 195, in _exec\n    return conn.execute(construct, multiparams)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1380, in execute\n    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py\", line 80, in _execute_on_connection\n    return connection._execute_ddl(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1472, in _execute_ddl\n    ret = self._execute_context(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1943, in _execute_context\n    self._handle_dbapi_exception(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 2124, in _handle_dbapi_exception\n    util.raise_(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py\", line 208, in raise_\n    raise exception\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1900, in _execute_context\n    self.dialect.do_execute(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py\", line 736, in do_execute\n    cursor.execute(statement, parameters)\nsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table _alembic_tmp_foo already exists\n[SQL:\nCREATE TABLE _alembic_tmp_foo (\n\tfoo_id INTEGER NOT NULL,\n\tbar VARCHAR(512),\n\tPRIMARY KEY (foo_id)\n)\n\n]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)\n```\n\n## What's happens\n\nI checked SQL queries by changing the logger level of `sqlalchemy.engine`.\n\n```\n# alembic.ini\n[logger_sqlalchemy]\nlevel = WARN   # -\u003e Change to DEBUG\nhandlers =\nqualname = sqlalchemy.engine\n```\n\nThen I got the following log.\n\n```\n$ alembic upgrade head\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO  [sqlalchemy.engine.Engine] PRAGMA main.table_info(\"alembic_version\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('cid', 'name', 'type', 'notnull', 'dflt_value', 'pk')\nDEBUG [sqlalchemy.engine.Engine] Row (0, 'version_num', 'VARCHAR(32)', 1, None, 1)\nINFO  [sqlalchemy.engine.Engine] SELECT alembic_version.version_num\nFROM alembic_version\nINFO  [sqlalchemy.engine.Engine] [generated in 0.00006s] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('version_num',)\nDEBUG [sqlalchemy.engine.Engine] Row ('90823bd3809e',)\nINFO  [sqlalchemy.engine.Engine] BEGIN (implicit)\nINFO  [alembic.runtime.migration] Running upgrade 90823bd3809e -\u003e 8e32f2af1bd2, Add schema \u0026 data migration\nINFO  [sqlalchemy.engine.Engine] PRAGMA main.table_xinfo(\"foo\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('cid', 'name', 'type', 'notnull', 'dflt_value', 'pk', 'hidden')\nDEBUG [sqlalchemy.engine.Engine] Row (0, 'foo_id', 'INTEGER', 1, None, 1, 0)\nDEBUG [sqlalchemy.engine.Engine] Row (1, 'bar', 'VARCHAR(512)', 1, None, 0, 0)\nINFO  [sqlalchemy.engine.Engine] SELECT sql FROM  (SELECT * FROM sqlite_master UNION ALL   SELECT * FROM sqlite_temp_master) WHERE name = ? AND type = 'table'\nINFO  [sqlalchemy.engine.Engine] [raw sql] ('foo',)\nDEBUG [sqlalchemy.engine.Engine] Col ('sql',)\nDEBUG [sqlalchemy.engine.Engine] Row ('CREATE TABLE foo (\\n\\tfoo_id INTEGER NOT NULL, \\n\\tbar VARCHAR(512) NOT NULL, \\n\\tPRIMARY KEY (foo_id)\\n)',)\nINFO  [sqlalchemy.engine.Engine] PRAGMA main.foreign_key_list(\"foo\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('id', 'seq', 'table', 'from', 'to', 'on_update', 'on_delete', 'match')\nINFO  [sqlalchemy.engine.Engine] PRAGMA temp.foreign_key_list(\"foo\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('id', 'seq', 'table', 'from', 'to', 'on_update', 'on_delete', 'match')\nINFO  [sqlalchemy.engine.Engine] SELECT sql FROM  (SELECT * FROM sqlite_master UNION ALL   SELECT * FROM sqlite_temp_master) WHERE name = ? AND type = 'table'\nINFO  [sqlalchemy.engine.Engine] [raw sql] ('foo',)\nDEBUG [sqlalchemy.engine.Engine] Col ('sql',)\nDEBUG [sqlalchemy.engine.Engine] Row ('CREATE TABLE foo (\\n\\tfoo_id INTEGER NOT NULL, \\n\\tbar VARCHAR(512) NOT NULL, \\n\\tPRIMARY KEY (foo_id)\\n)',)\nINFO  [sqlalchemy.engine.Engine] PRAGMA main.index_list(\"foo\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('seq', 'name', 'unique', 'origin', 'partial')\nINFO  [sqlalchemy.engine.Engine] PRAGMA temp.index_list(\"foo\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('seq', 'name', 'unique', 'origin', 'partial')\nINFO  [sqlalchemy.engine.Engine] PRAGMA main.index_list(\"foo\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('seq', 'name', 'unique', 'origin', 'partial')\nINFO  [sqlalchemy.engine.Engine] PRAGMA temp.index_list(\"foo\")\nINFO  [sqlalchemy.engine.Engine] [raw sql] ()\nDEBUG [sqlalchemy.engine.Engine] Col ('seq', 'name', 'unique', 'origin', 'partial')\nINFO  [sqlalchemy.engine.Engine] SELECT sql FROM  (SELECT * FROM sqlite_master UNION ALL   SELECT * FROM sqlite_temp_master) WHERE name = ? AND type = 'table'\nINFO  [sqlalchemy.engine.Engine] [raw sql] ('foo',)\nDEBUG [sqlalchemy.engine.Engine] Col ('sql',)\nDEBUG [sqlalchemy.engine.Engine] Row ('CREATE TABLE foo (\\n\\tfoo_id INTEGER NOT NULL, \\n\\tbar VARCHAR(512) NOT NULL, \\n\\tPRIMARY KEY (foo_id)\\n)',)\nINFO  [sqlalchemy.engine.Engine]\nCREATE TABLE _alembic_tmp_foo (\n\tfoo_id INTEGER NOT NULL,\n\tbar VARCHAR(512),\n\tPRIMARY KEY (foo_id)\n)\n\n\nINFO  [sqlalchemy.engine.Engine] [no key 0.00004s] ()\nINFO  [sqlalchemy.engine.Engine] INSERT INTO _alembic_tmp_foo (foo_id, bar) SELECT foo.foo_id, foo.bar\nFROM foo\nINFO  [sqlalchemy.engine.Engine] [generated in 0.00005s] ()\nINFO  [sqlalchemy.engine.Engine]\nDROP TABLE foo\nINFO  [sqlalchemy.engine.Engine] [no key 0.00003s] ()\nINFO  [sqlalchemy.engine.Engine] ALTER TABLE _alembic_tmp_foo RENAME TO foo\nINFO  [sqlalchemy.engine.Engine] [no key 0.00003s] ()\nINFO  [sqlalchemy.engine.Engine] ROLLBACK\nTraceback (most recent call last):\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/bin/alembic\", line 8, in \u003cmodule\u003e\n    sys.exit(main())\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/config.py\", line 590, in main\n    CommandLine(prog=prog).main(argv=argv)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/config.py\", line 584, in main\n    self.run_cmd(cfg, options)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/config.py\", line 561, in run_cmd\n    fn(\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/command.py\", line 322, in upgrade\n    script.run_env()\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/script/base.py\", line 569, in run_env\n    util.load_python_file(self.dir, \"env.py\")\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py\", line 94, in load_python_file\n    module = load_module_py(module_id, path)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py\", line 110, in load_module_py\n    spec.loader.exec_module(module)  # type: ignore\n  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 883, in exec_module\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 241, in _call_with_frames_removed\n  File \"/Users/c-bata/sandbox/alembic-sandbox/batchop/env.py\", line 79, in \u003cmodule\u003e\n    run_migrations_online()\n  File \"/Users/c-bata/sandbox/alembic-sandbox/batchop/env.py\", line 73, in run_migrations_online\n    context.run_migrations()\n  File \"\u003cstring\u003e\", line 8, in run_migrations\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/runtime/environment.py\", line 853, in run_migrations\n    self.get_context().run_migrations(**kw)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/venv/lib/python3.10/site-packages/alembic/runtime/migration.py\", line 623, in run_migrations\n    step.migration_fn(**kw)\n  File \"/Users/c-bata/sandbox/alembic-sandbox/batchop/versions/8e32f2af1bd2_add_schema_data_migration.py\", line 43, in upgrade\n    raise Exception(\"An exception is raised during the data migration\")\nException: An exception is raised during the data migration\n```\n\nAlembic issues `ROLLBACK` statement due to an exception.\nHowever, SQLite3 does not seems to be able to rollback `CREATE TABLE _alembic_tmp_foo ...`.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-bata%2Falembic-tmp-table-issue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc-bata%2Falembic-tmp-table-issue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-bata%2Falembic-tmp-table-issue/lists"}