Giter Site home page Giter Site logo

Comments (8)

ddcw avatar ddcw commented on August 13, 2024 1

你好，我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件，使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况
./python a.py ../project.ibd Traceback (most recent call last): File "a.py", line 28, in sdiinfo = sdi_page_data(bdata) File "a.py", line 22, in sdi_page_data sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())] IndexError: list index out of range
sdi:
[root@aaaaa ~]# more project.json ["ibd2sdi" , { "type": 1, "id": 1773, "object": { "mysqld_version_id": 80034, "dd_version": 80023, "sdi_version": 80019,

我在我的8.0.34环境测试也是可以的... -_- 从报错来看, 第4页不是sdi page或者不完整. 请使用如下python3代码验证下呢

import sys
with open(sys.argv[1],'rb') as f:
	for i in range(16):
		bdata = f.read(16384)
		page_type = int.from_bytes(bdata[24:26],'big')
		page_type = 'SDI_PAGE' if page_type == 17853 else page_type
		print(f'PAGE_NO: {i}  \tPAGE_TYPE:{page_type}')

我上面遇到的报错就是因为第4页不是SDI PAGE。对于升级上来的Mysql 8的ibd文件SDI PAGE位置不固定。

对于5.7升级到8.0的ibd文件, sdi page确实不在第四页. 已修复(参考ibd2sdi.cc). 请下载最新版本测试下呢

from ibd2sql.

ddcw avatar ddcw commented on August 13, 2024

python3 main.py --ddl /mnt/d/temp/sys_menus_admin.ibd Traceback (most recent call last): File "/data/source/ibd2sql-main/main.py", line 100, in print('\n',innodb_sdi.sdi(filename).get_ddl(),'\n') File "/data/source/ibd2sql-main/innodb_sdi.py", line 43, in get_ddl dic_info = self.get_dic() File "/data/source/ibd2sql-main/innodb_sdi.py", line 132, in get_dic unzbdata = zlib.decompress(self.bdata[offset+33:offset+33+dzip_len]) zlib.error: Error -3 while decompressing data: incorrect header check

目前只支持 mysql 8.0 的数据文件, 请确认下数据库版本信息

from ibd2sql.

andkylele avatar andkylele commented on August 13, 2024

使用ibd2sdi解析ibd文件出来的结果：
["ibd2sdi"
,
{
"type": 1,
"id": 290,
"object":
{
"mysqld_version_id": 80021,
"dd_version": 80021,
"sdi_version": 80019,
"dd_object_type": "Table",
"dd_object": {
"name": "sys_menus_admin",
"mysql_version_id": 80021,

from ibd2sql.

ddcw avatar ddcw commented on August 13, 2024

使用ibd2sdi解析ibd文件出来的结果： ["ibd2sdi" , { "type": 1, "id": 290, "object": { "mysqld_version_id": 80021, "dd_version": 80021, "sdi_version": 80019, "dd_object_type": "Table", "dd_object": { "name": "sys_menus_admin", "mysql_version_id": 80021,

从ibd2sdi解析结果来看是mysql 8.0.21版本的Ibd文件, 我也解析了下8.0.21版本的ibd文件是正常的

12:03:42 [root@ddcw21 ibd2sql_v0.2]#python3 main.py  --ddl /data/mysql_3352/mysqldata/db1/t1.ibd

 CREATE Table IF NOT EXISTS `db1`.`t1`(
`id` int NOT NULL ,
`name` varchar(200) DEFAULT NULL ,
PRIMARY KEY (`id`)
)ENGINE=InnoDB ; 

12:03:45 [root@ddcw21 ibd2sql_v0.2]#/soft/mysql_3352/mysqlbase/mysql/bin/ibd2sdi /data/mysql_3352/mysqldata/db1/t1.ibd | head -10
["ibd2sdi"
,
{
	"type": 1,
	"id": 352,
	"object":
		{
    "mysqld_version_id": 80021,
    "dd_version": 80021,
    "sdi_version": 80019,
12:04:16 [root@ddcw21 ibd2sql_v0.2]#

那么这个报错的原因可能为:

  1. python版本问题. 本次测试的是python3.6
  2. sdi page 解析有问题
  3. page_size不为16384 select @@innodb_page_size;
    可以使用如下python代码进行验证第二个可能(暴力解析第4页)
import sys,zlib,json
# 参考https://cloud.tencent.com/developer/article/2272297

def sdi_page_data(bdata):
	_sdi_offset = []
	isok = False
	for i in range(120,16384):
		if isok:
			break
		for j in range(i,16384):
			if isok:
				break
			try:
				_test = zlib.decompress(bdata[i:j])
				_sdi_offset.append(i)
				i += len(_test)
				if len(_sdi_offset) == 2:
					isok = True #break 2
				break
			except:
				pass
	sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())]
	return sdi_info

with open(sys.argv[1],'rb') as f:
	f.seek(3*16384,0)
	bdata = f.read(16384)
	sdiinfo = sdi_page_data(bdata)
	print(sdiinfo)
(venv) 12:15:39 [root@ddcw21 ei]#python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd
[{'mysqld_version_id': 80021, 'dd_version': 80021, 'sdi_version': 80019, 'dd_object_type': 'Tablespace', 'dd_object': {'name': 'db1/t1', 'comment': '', 'options': 'encryption=N;', 'se_private_data': 'flags=16417;id=2;server_version=80021;space_version=1;state=normal;', 'engine': 'InnoDB', 'engine_attribute': '', 'files': [{'ordinal_position': 1, 'filename': './db1/t1.ibd', 'se_private_data': 'id=2;'}]}}, {'mysqld_version_id': 80021, 'dd_version': 80021, 'sdi_version': 80019, 'dd_object_type': 'Table', 'dd_object': {'name': 't1', 'mysql_version_id': 80021, 'created': 20231011040312, 'last_altered': 20231011040312, 'hidden': 1, 'options': 'avg_row_length=0;encrypt_type=N;key_block_size=0;keys_disabled=0;pack_record=1;stats_auto_recalc=0;stats_sample_pages=0;', 'columns': [{'name': 'id', 'type': 4, 'is_nullable': False, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 1, 'ordinal_position': 1, 'char_length': 11, 'numeric_precision': 10, 'numeric_scale': 0, 'numeric_scale_null': False, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': True, 'default_value_null': False, 'srs_id_null': True, 'srs_id': 0, 'default_value': 'AAAAAA==', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': 'interval_count=0;', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 2, 'column_type_utf8': 'int', 'elements': [], 'collation_id': 33, 'is_explicit_collation': False}, {'name': 'name', 'type': 16, 'is_nullable': True, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 1, 'ordinal_position': 2, 'char_length': 600, 'numeric_precision': 0, 'numeric_scale': 0, 'numeric_scale_null': True, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': False, 'default_value_null': True, 'srs_id_null': True, 'srs_id': 0, 'default_value': '', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': 'interval_count=0;', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 1, 'column_type_utf8': 'varchar(200)', 'elements': [], 'collation_id': 33, 'is_explicit_collation': False}, {'name': 'DB_TRX_ID', 'type': 10, 'is_nullable': False, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 2, 'ordinal_position': 3, 'char_length': 6, 'numeric_precision': 0, 'numeric_scale': 0, 'numeric_scale_null': True, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': False, 'default_value_null': True, 'srs_id_null': True, 'srs_id': 0, 'default_value': '', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': '', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 1, 'column_type_utf8': '', 'elements': [], 'collation_id': 63, 'is_explicit_collation': False}, {'name': 'DB_ROLL_PTR', 'type': 9, 'is_nullable': False, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 2, 'ordinal_position': 4, 'char_length': 7, 'numeric_precision': 0, 'numeric_scale': 0, 'numeric_scale_null': True, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': False, 'default_value_null': True, 'srs_id_null': True, 'srs_id': 0, 'default_value': '', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': '', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 1, 'column_type_utf8': '', 'elements': [], 'collation_id': 63, 'is_explicit_collation': False}], 'schema_ref': 'db1', 'se_private_id': 1059, 'engine': 'InnoDB', 'last_checked_for_upgrade_version_id': 0, 'comment': '', 'se_private_data': '', 'engine_attribute': '', 'secondary_engine_attribute': '', 'row_format': 2, 'partition_type': 0, 'partition_expression': '', 'partition_expression_utf8': '', 'default_partitioning': 0, 'subpartition_type': 0, 'subpartition_expression': '', 'subpartition_expression_utf8': '', 'default_subpartitioning': 0, 'indexes': [{'name': 'PRIMARY', 'hidden': False, 'is_generated': False, 'ordinal_position': 1, 'comment': '', 'options': 'flags=0;', 'se_private_data': 'id=143;root=4;space_id=2;table_id=1059;trx_id=1568;', 'type': 1, 'algorithm': 2, 'is_algorithm_explicit': False, 'is_visible': True, 'engine': 'InnoDB', 'engine_attribute': '', 'secondary_engine_attribute': '', 'elements': [{'ordinal_position': 1, 'length': 4, 'order': 2, 'hidden': False, 'column_opx': 0}, {'ordinal_position': 2, 'length': 4294967295, 'order': 2, 'hidden': True, 'column_opx': 2}, {'ordinal_position': 3, 'length': 4294967295, 'order': 2, 'hidden': True, 'column_opx': 3}, {'ordinal_position': 4, 'length': 4294967295, 'order': 2, 'hidden': True, 'column_opx': 1}], 'tablespace_ref': 'db1/t1'}], 'foreign_keys': [], 'check_constraints': [], 'partitions': [], 'collation_id': 33}}]

from ibd2sql.

guminhu avatar guminhu commented on August 13, 2024

你好，我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件，使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况

./python a.py ../project.ibd
Traceback (most recent call last):
File "a.py", line 28, in
sdiinfo = sdi_page_data(bdata)
File "a.py", line 22, in sdi_page_data
sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())]
IndexError: list index out of range

sdi:

[root@aaaaa ~]# more project.json
["ibd2sdi"
,
{
"type": 1,
"id": 1773,
"object":
{
"mysqld_version_id": 80034,
"dd_version": 80023,
"sdi_version": 80019,

from ibd2sql.

ddcw avatar ddcw commented on August 13, 2024

你好，我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件，使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况

./python a.py ../project.ibd Traceback (most recent call last): File "a.py", line 28, in sdiinfo = sdi_page_data(bdata) File "a.py", line 22, in sdi_page_data sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())] IndexError: list index out of range

sdi:

[root@aaaaa ~]# more project.json ["ibd2sdi" , { "type": 1, "id": 1773, "object": { "mysqld_version_id": 80034, "dd_version": 80023, "sdi_version": 80019,

我在我的8.0.34环境测试也是可以的... -_-
从报错来看, 第4页不是sdi page或者不完整. 请使用如下python3代码验证下呢

import sys
with open(sys.argv[1],'rb') as f:
	for i in range(16):
		bdata = f.read(16384)
		page_type = int.from_bytes(bdata[24:26],'big')
		page_type = 'SDI_PAGE' if page_type == 17853 else page_type
		print(f'PAGE_NO: {i}  \tPAGE_TYPE:{page_type}')

from ibd2sql.

andkylele avatar andkylele commented on August 13, 2024

你好，我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件，使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况
./python a.py ../project.ibd Traceback (most recent call last): File "a.py", line 28, in sdiinfo = sdi_page_data(bdata) File "a.py", line 22, in sdi_page_data sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())] IndexError: list index out of range
sdi:
[root@aaaaa ~]# more project.json ["ibd2sdi" , { "type": 1, "id": 1773, "object": { "mysqld_version_id": 80034, "dd_version": 80023, "sdi_version": 80019,

我在我的8.0.34环境测试也是可以的... -_- 从报错来看, 第4页不是sdi page或者不完整. 请使用如下python3代码验证下呢

import sys
with open(sys.argv[1],'rb') as f:
	for i in range(16):
		bdata = f.read(16384)
		page_type = int.from_bytes(bdata[24:26],'big')
		page_type = 'SDI_PAGE' if page_type == 17853 else page_type
		print(f'PAGE_NO: {i}  \tPAGE_TYPE:{page_type}')

我上面遇到的报错就是因为第4页不是SDI PAGE。对于升级上来的Mysql 8的ibd文件SDI PAGE位置不固定。

from ibd2sql.

guminhu avatar guminhu commented on August 13, 2024

可以了，感谢大佬！

from ibd2sql.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.