今天遇到一个关于MySQL求助的问题,修改表结构时遇到“ERROR 1050 (42S01): table xxx already exits"
mysql> ALTER TABLE DAY_BOOK_REPORT ADD UNIT_PRICE_PCS DOUBLE(12,2) DEFAULT NULL;
ERROR 1050 (42S01): TABLE 'INVGSP/#SQL-IB379' ALREADY EXISTS
mysql>
检查了后,发现表DAY_BOOK_REPORT确实不存在字段UNIT_PRICE_PCS,但是给表加字段时就报这个错误,遂咨询了一下他具体的操作过程,反馈是当时在做大量数据更新,然后给这个表增加字段时,突然报“DB connect fail”, 登录MySQL服务器检查发现MySQL服务已经挂了,MySQL版本为5.6.20-enterprise-commercial-advanced-log,检查错误日志,发现有下面错误信息:
2018-03-31 23:29:16 7f09c1830700 InnoDB: Error: Write to file ./INVOICE/#sql-ib379.ibd failed at offset 600834048.
InnoDB: 1048576 bytes should have been written, only 446464 were written.
InnoDB: Operating system error number 0.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
15:29:16 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=120
max_threads=151
thread_count=6
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68245 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x9ac95e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f09c182fe10 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x946155]
/usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x6a58c8]
/lib64/libpthread.so.0[0x3a6b60f710]
/usr/sbin/mysqld[0xa45a2b]
/usr/sbin/mysqld[0xa50f5a]
/usr/sbin/mysqld[0x9e1afd]
/usr/sbin/mysqld[0x9e55a5]
/usr/sbin/mysqld[0x96aec5]
/usr/sbin/mysqld[0x7790a5]
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P24st_ha_create_informationP10TABLE_LISTP10Alter_infojP8st_orderb+0x1e54)[0x77b204]
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x4a5)[0x87fab5]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3d4f)[0x72aa4f]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x318)[0x72de48]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x11b6)[0x72f7f6]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xd7)[0x7310a7]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x116)[0x6f8856]
/usr/sbin/mysqld(handle_one_connection+0x45)[0x6f8935]
/usr/sbin/mysqld(pfs_spawn_thread+0x126)[0xb153e6]
/lib64/libpthread.so.0[0x3a6b6079d1]
/lib64/libc.so.6(clone+0x6d)[0x3a6b2e89dd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f095e93b2e0): is an invalid pointer
Connection ID (thread ID): 4237691
Status: NOT_KILLED
从错误提示看,MySQL在往./INVGSP/#sql-ib379.ibd文件写入数据时,遇到了错误,但是最终写入成功(InnoDB: Operating system error number 0.),按错误日志里面的信息提示排查问题:
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
最终检查发现MySQL数据文件所在的分区已经爆了,看错误提示,很有可能是因为空间问题,导致MySQL进程Crash掉了,而MySQL在ALTER TABLE操作过程中崩溃,那么最终可能会在InnoDB表空间中生成一个孤立的中间表(orphaned intermediate table)。 其实#sql-ib379.ibd就是在修改DAY_BOOK_REPORT时,由于MySQL进程Crash掉后生成的孤立中间表。检查如下所示:
mysql> show variables like '%innodb_file_per_table%';
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| innodb_file_per_table | ON |
+-----------------------+-------+
1 row in set (0.00 sec)
mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE NAME LIKE '%#sql%';
+----------+--------------------+------+--------+-------+-------------+------------+---------------+
| TABLE_ID | NAME | FLAG | N_COLS | SPACE | FILE_FORMAT | ROW_FORMAT | ZIP_PAGE_SIZE |
+----------+--------------------+------+--------+-------+-------------+------------+---------------+
| 650 | INVOICE/#sql-ib379 | 1 | 65 | 636 | Antelope | Compact | 0 |
+----------+--------------------+------+--------+-------+-------------+------------+---------------+
1 row in set (0.04 sec)
mysql>
官方文档https://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html关于孤立中间表的介绍如下:
Orphan Intermediate Tables
If MySQL exits in the middle of an in-place ALTER TABLE operation (ALGORITHM=INPLACE), you may be left with an orphan intermediate table that takes up space on your system. This section describes how to identify and remove orphan intermediate tables.
Intermediate table names begin with an #sql-ib prefix (e.g., #sql-ib87-856498050). The accompanying .frm file has an #sql-* prefix and is named differently (e.g., #sql-36ab_2.frm).
To identify orphan intermediate tables on your system, you can view Table Monitor output or query INFORMATION_SCHEMA.INNODB_SYS_TABLES. Look for table names that begin with #sql. If the original table resides in a file-per-table tablespace, the tablespace file (the #sql-*.ibd file) for the orphan intermediate table should be visible in the database directory.
找到对应的frm文件(这里是#sql-71a_40a97b.frm ),然后将其命名为#sql-ib379.frm(数据文件为#sql-ib379.ibd), 然后删除表(对应的文件会删除)即可解决上面这个问题。
# mv "#sql-71a_40a97b.frm" "#sql-ib379.frm"
mysql> DROP TABLE `#mysql50##sql-ib379`
-> ;
Query OK, 0 rows affected (0.11 sec)
mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE NAME LIKE '%#sql%';
Empty set (0.01 sec)
mysql>
个人还测试了网上另外一种方法,就是首先先删除#sql开头的这些文件,然后拷贝源表数据到备份表,接着删除原表,最后将备份表重命名为源表。添加相关索引。这种方法也能解决这个问题。
mysql> show index from DAY_BOOK_REPORT;
mysql> create table DAY_BOOK_REPORT_BK as select * from DAY_BOOK_REPORT;
mysql> drop table DAY_BOOK_REPORT;
mysql> rename table DAY_BOOK_REPORT_BK to DAY_BOOK_REPORT;
mysql>ALTER TABLE DAY_BOOK_REPORT ADD INDEX INDEX_NAME (column_list) --根据实际情况输入具体字段
mysql>ALTER TABLE DAY_BOOK_REPORT ADD UNIQUE (column_list) --根据实际情况输入具体字段
mysql>ALTER TABLE DAY_BOOK_REPORT ADD PRIMARY KEY (column_list) --根据实际情况输入具体字段
参考资料:
https://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html