twindb / undrop-for-innodb Goto Github PK

View Code? Open in Web Editor NEW

376.0 376.0 145.0 1.16 MB

TwinDB data recovery toolkit for MySQL/InnoDB

Home Page: https://twindb.com

License: GNU General Public License v2.0

Makefile 0.18% C 91.71% C++ 3.49% Shell 0.82% Lex 0.60% Yacc 1.87% Ruby 0.58% Puppet 0.20% SWIG 0.55% Pascal 0.01%

undrop-for-innodb's People

Contributors

Stargazers

Watchers

Forkers

wwwbjqcom ycg liuqian1990 aimdotsh jiayifei skymyyang saup007 qiangleizang zhishutech tom2jack minhnkt smudkey ayushghosh mpl0045 cothee jrelo jions7ihj yuanddqiao 10oo01sir zhl463 jyotie aspros-zhong dbnski nettedfish ohsdba dowa2013 cloudufull r66wen jessebaomihua hoozone chinlung hahaguai jhx1008 hien picassio shitouxiaogege dngv drlimd1987 luhouhua lamxops xiaoshudian01 sirsepp romston zhangzaki pushpen radtek elliottfdh kevin6386 chrisxiao tffy as10128 suzitai sageyan woihzh iqbalme haroldyong heisenbergdo 1newstar ardgbani ca4ti cloudwafs maxsteciuk livenobin bashirshallah jafm1611 vic868 odasm martowu bmildren gczheng ehb54 h2co3bearcard tools-env gaoyi34201 xnohat ibringit hajro92 sunyifeng dobetter413 wqfgithub vswb iomarcovalente pmacontrol michaelcoburn hmilkovi dotmanila dongzhang1937 yogspuuu rscpass leeworker swmwlm xt0207 angeltgc521 randing80s atkins126 albert-kam qt-pay gpf199541 yemo-memeda andreescocard

undrop-for-innodb's Issues

c_parser input sql with KEY_BLOCK_SIZE

If a table has the KEY_BLOCK_SIZE value set, c_parser fails the data dump

eg:

CREATE TABLE `settings` (
  `name` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
  `value` varchar(2048) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
  PRIMARY KEY (`name`) KEY_BLOCK_SIZE=1024
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC;

error: ld returned 1 exit status / undefined reference to `make_longlong'

😢

cc -D_FILE_OFFSET_BITS=64 -Wall -g -O3 -pipe -std=gnu89  -I./include  sql_parser.o c_parser.o tables_dict.o print_data.o check_data.o -o c_parser -pthread -lm
print_data.o: In function `guess_datetime_format':
/usr/src/repositories/undrop-for-innodb/print_data.c:35: undefined reference to `make_longlong'
/usr/src/repositories/undrop-for-innodb/print_data.c:35: undefined reference to `make_longlong'
print_data.o: In function `print_datetime':
/usr/src/repositories/undrop-for-innodb/print_data.c:85: undefined reference to `make_longlong'
print_data.o: In function `get_uint_value':
/usr/src/repositories/undrop-for-innodb/print_data.c:224: undefined reference to `make_ulonglong'
/usr/src/repositories/undrop-for-innodb/print_data.c:221: undefined reference to `make_ulonglong'
/usr/src/repositories/undrop-for-innodb/print_data.c:222: undefined reference to `make_ulonglong'
/usr/src/repositories/undrop-for-innodb/print_data.c:223: undefined reference to `make_ulonglong'
print_data.o: In function `get_int_value':
/usr/src/repositories/undrop-for-innodb/print_data.c:251: undefined reference to `make_ulonglong'
print_data.o:/usr/src/repositories/undrop-for-innodb/print_data.c:251: more undefined references to `make_ulonglong' follow
print_data.o: In function `guess_datetime_format':
/usr/src/repositories/undrop-for-innodb/print_data.c:35: undefined reference to `make_longlong'
print_data.o: In function `print_datetime':
/usr/src/repositories/undrop-for-innodb/print_data.c:85: undefined reference to `make_longlong'
print_data.o: In function `get_int_value':
/usr/src/repositories/undrop-for-innodb/print_data.c:251: undefined reference to `make_ulonglong'
/usr/src/repositories/undrop-for-innodb/print_data.c:251: undefined reference to `make_ulonglong'
print_data.o: In function `get_uint_value':
/usr/src/repositories/undrop-for-innodb/print_data.c:224: undefined reference to `make_ulonglong'
/usr/src/repositories/undrop-for-innodb/print_data.c:223: undefined reference to `make_ulonglong'
/usr/src/repositories/undrop-for-innodb/print_data.c:222: undefined reference to `make_ulonglong'
print_data.o:/usr/src/repositories/undrop-for-innodb/print_data.c:221: more undefined references to `make_ulonglong' follow
collect2: error: ld returned 1 exit status
Makefile:55: recipe for target 'c_parser' failed
make: *** [c_parser] Error 1

does this tool support recovering data from json fields?

Excuse me, does this tool support recovering data from json fields? I use "./c_parser" to parse json error reporting structure exceptions. If I change the json column to longblob, it can be parsed, but the result is a string of abnormal characters, such as' 02110037000501005030005040005050050050070005080005090050A00050B00050C00050E000510000511000514000515000516005160051600 '. Can I solve this problem?

stream_parser

Hi stream_parser has no support for mysql 8.11,innodb file, the latest version of mysql's innodb checksum calculation method has changed.
not found page type
FIL_PAGE_TYPE_LOB_INDEX 22
FIL_PAGE_TYPE_LOB_DATA 23
FIL_PAGE_TYPE_LOB_FIRST 24
FIL_PAGE_TYPE_ZLOB_FIRST 25
FIL_PAGE_TYPE_ZLOB_DATA 26
FIL_PAGE_TYPE_ZLOB_INDEX 27
FIL_PAGE_TYPE_ZLOB_FRAG 28
FIL_PAGE_TYPE_ZLOB_FRAG_ENTRY 29

c_parser error

Hi,

there is an error when there is 2 single quote in the DDL script when you try to recover data

in a column comment
in a table comment
in a enum definition

ie :

create table _test
(
a varchar(10) comment 'abcd''xyz',
b enum('aaa', 'bbb''bb')
) engine=innodb comment='aaa''vvv';

This is a valid mysql create statement, but this throw an error in c_parser.

Marc.

c_parser with special characters

Hi, when i tried to use c_parser, it ignore special characters and add some strange characters to data
Examples:
PROMOÇÕES showing PROMO��ES
after some data, shows �\0\0\0\0�
instead float numbers, shows 0�

is it possible to read special characters?

install: target './bin' is not a directory make: *** [Makefile:68: install] Error 1

run sudo make install get this error

install -s stream_parser c_parser innochecksum_changer ./bin
install: target './bin' is not a directory
make: *** [Makefile:68: install] Error 1

mysql 5.7 json

First of all thank you for this package! I'm restoring a database and so far it's working fine.

I do have a question about mysql5.7. I have some json fields which hold data. I changed the field datatype to longtext because the c_parser could not understand the json field. The data is intact except I have fields like "\0�\04\0�\0�\0�\0�\0�\0\%\0ennl�Transportation�Transportation".

Do you have any idea if I can restore the information in the json column?

Kind regards,

Rogier

Segmentation fault

@akuzminsky when running ./c_parser to see the command help it just displays

root@server:/home/undrop-for-innodb# ./c_parser
Segmentation fault

run c_parser print Killed

Hi, when I execute c_parser, It print Killed。How can I solve this problem?

[root@VM_0_11_centos undrop-for-innodb]# ./c_parser -4f pages-ibdata1/FIL_PAGE_INDEX/0000000000000001.page -t dictionary/SYS_TABLES.sql > dumps/default/SYS_TABLES 2> dumps/default/SYS_TABLES.sql
Killed
[root@VM_0_11_centos undrop-for-innodb]# ./recover_dictionary.sh 
Generating dictionary tables dumps... ./recover_dictionary.sh: line 33: 29484 Killed                  ./c_parser -4f $pages -t dictionary/$t.sql > dumps/default/$t 2> dumps/default/$t.sql

MYSQL v8.x

I have been running these tools over a mysql 8.x files, and having issues. It seems 8.x changed enough for these tools not to work? Has anyone had any success with versions other than 5.x

Get Wrong data, wrong datetime

I restore the data everything runs normally but the output is not correct, the string columns are empty, the date column is wrong, the column interger is invalid (much larger than the original value).

Data recovered:

-- Page id: 4, Format: COMPACT, Records list: Valid, Expected records: (29 29)

4E4441344E47	4A68597A4A685A	customers	5562594712738876793	""	""	""	""	NULL	""	NULL	NULL	""	1834117454	NULL	-1304008625	1140850688	0	907935744	3355488334	NULL	"2028-02-17 13:06:06"	NULL	"2006-05-12 00:45:51"	NULL

5932457A5A44	6B7A597A593059	customers	5578361700692413746	""	""	""	""	NULL	""	NULL	NULL	""	1464756559	NULL	-682002598	1191182336	0	756940800	2147585102	NULL	NULL	"7667-78-26 73:85:09"	NULL	1232686423

5A57466B4F47	5934596A646C4F	customers	5572196507250344313	""	""	""	""	NULL	""	NULL	NULL	""	1196845645	NULL	-362517938	1828716544	0	706609152	1879262287	"2014-10-24 22:16:41"	"2026-06-29 02:34:39"	"9780-06-36 43:06:69"	NULL	1232686386

4F4449305A54	45324D3251335A	customers	5634651267177669736	""	""	""	""	NULL	""	NULL	NULL	""	1464432718	NULL	-950638514	838860800	0	807272448	2566857807	NULL	NULL	NULL	NULL	1413820762

4E6D4A684D6D	56694D4451774F	customers	5639704627130944105	""	""	""	""	NULL	""	NULL	NULL	""	1465281113	NULL	-361994150	1459617792	0	689840128	671133773	NULL	NULL	NULL	NULL	NULL
4E6D59305A57	45334E6D5A6959	customers	5639710064342487415	""	""	""	""	NULL	""	NULL	NULL	""	843738447	NULL	-683313318	1828716544	0	1025377537	16642137	"1996-10-08 04:47:05"	"2035-01-05 20:51:10"	NULL	NULL	NULL

5A5749344E6D	5A6B5A5459304E	customers	5640561949848066424	""	""	""	""	NULL	""	NULL	NULL	""	1146714457	NULL	-362517170	1409286144	0	1092485121	553580633	"2016-05-28 17:56:14"	"2006-04-29 02:22:01"	"6719-78-84 79:13:73"	"2025-01-05 16:13:22"	1366969978

5A6A51775A54	42684F54526D4D	customers	5650427060031411509	""	""	""	""	NULL	""	NULL	NULL	""	1782675790	NULL	-362318002	1409286144	0	673054720	1744886861	NULL	NULL	NULL	"2014-11-18 18:54:01"	1413577050

5A6A6B7A4F47	59784D6A56694F	customers	5651262375335713385	""	""	""	""	NULL	""	NULL	NULL	""	1414100814	NULL	-733907123	838860800	0	572399616	939557978	NULL	"2014-11-19 17:09:30"	"3990-85-45 22:88:77"	"2015-04-17 07:01:49"	1517112938

4D5749345A54	4D354D6D45355A	customers	5651263728231139705	""	""	""	""	NULL	""	NULL	NULL	""	1834037593	NULL	-1302696871	1409286144	0	773718016	2298309978	NULL	"2028-02-14 12:12:45"	NULL	NULL	1196765774

4F4467354F47	4D314D7A63784D	customers	5651268130589401976	""	""	""	""	NULL	""	NULL	NULL	""	1834300238	NULL	-1302697639	1828716544	0	924712960	3489773657	"2014-10-27 23:01:17"	NULL	NULL	"2014-10-18 03:08:46"	NULL

4E7A45794F57	52684F5749774D	customers	5651271171428737841	""	""	""	""	NULL	""	NULL	NULL	""	1147893581	NULL	-1001818278	1140850688	0	991821824	4026610777	"2016-06-01 11:45:02"	NULL	"3448-21-47 56:79:51"	NULL	NULL

596A45324D44	6B314D6A466C5A	customers	5644208218152655152	""	""	""	""	NULL	""	NULL	NULL	""	1145727065	NULL	-731434918	1828716544	0	874381312	3103739983	NULL	NULL	"9800-79-42 14:74:17"	"2015-12-13 18:41:08"	NULL

5A54426B4D44	526D5A6A417A59	customers	5645056753197662516	""	""	""	""	NULL	""	NULL	NULL	""	2051618137	NULL	-682807207	2046820352	0	975045632	3909012558	NULL	"2028-02-19 15:26:54"	"2520-38-43 29:96:63"	NULL	NULL

593245334E6A	45794D5745344D	customers	5711761916011637552	""	""	""	""	NULL	""	NULL	NULL	""	1146695757	NULL	-94162355	2046820352	0	1075707905	419171406	NULL	NULL	NULL	NULL	NULL

5A544A694D54	46694E4441354E	customers	5711766610612217651	""	""	""	""	NULL	""	NULL	NULL	""	1414163034	NULL	-731616177	1191182336	0	958267392	3758164046	"2016-05-25 12:17:03"	"2006-04-16 00:58:50"	"3322-72-51 75:63:78"	NULL	NULL

4D5449314F54	64685A5459795A	customers	5712623933343417721	""	""	""	""	NULL	""	NULL	NULL	""	1146172750	NULL	-363235505	1409286144	0	857604096	2952857678	"2014-10-18 03:13:13"	NULL	"4233-27-06 33:73:57"	"2006-10-15 22:01:24"	NULL

59544D7A4E47	55324E32466C4E	customers	5712632733714830957	""	""	""	""	NULL	""	NULL	NULL	""	1145203534	NULL	-362598066	1409286144	0	891158528	3237901389	"1996-09-29 19:28:10"	"2006-04-25 08:20:14"	NULL	"2006-04-16 00:58:38"	NULL

595463784E54	51325A47566A4D	customers	5716269905311188584	""	""	""	""	NULL	""	NULL	NULL	""	2053663053	NULL	-733058214	1778384896	0	824049664	2684557401	"2026-07-02 20:53:29"	"2028-02-07 17:17:51"	"2077-68-28 84:28:30"	"2011-01-17 03:00:04"	NULL

4F54557A5A57	49794D3259775A	customers	5716275660552028535	""	""	""	""	NULL	""	NULL	NULL	""	1784886094	NULL	-313628083	2046820352	0	723386368	2030020685	"2026-07-11 23:03:26"	NULL	"3389-99-03 97:39:78"	NULL	1165646404

5A6A646B4E32	49795A545A6A59	customers	5717137656006530355	""	""	""	""	NULL	""	NULL	NULL	""	1785166170	NULL	-312903346	1191182336	0	1058930689	285133903	"2006-04-16 00:54:21"	NULL	NULL	NULL	NULL

4E5755774E54	4A694F44526D59	customers	6427298000233187636	""	""	""	""	NULL	""	NULL	NULL	""	1834580314	NULL	-950438067	1828716544	0	1042153473	150983759	NULL	"2016-06-10 14:12:14"	NULL	NULL	NULL

4F4459304D7A	4D334E445A684F	customers	6443039652372629880	""	""	""	""	NULL	""	NULL	NULL	""	1415805517	NULL	-950699699	2046820352	0	840828160	2818594895	NULL	"2007-12-01 17:11:38"	"9304-56-12 50:74:71"	"2008-11-29 21:19:00"	NULL

5A5468685A44	4E694D6A426D4E	customers	6443042693025785720	""	""	""	""	NULL	""	NULL	NULL	""	1197046351	NULL	-732272038	1409286144	0	1140850688	0	NULL	NULL	"0000-00-00 00:00:00"	"1970-01-01 08:00:00"	0

4E47526C4D47	4E68597A55344F	customers	6436855994480083319	""	""	""	""	NULL	""	NULL	NULL	""	1415212365	NULL	-94799271	1459617792	0	941490176	3640588367	NULL	NULL	NULL	"2006-04-19 06:38:01"	NULL

59544179597A	677A4D32517A4E	customers	6437694977702384235	""	""	""	""	NULL	""	NULL	NULL	""	2052666702	NULL	-682020274	2046820352	0	1109262337	687640666	NULL	"2007-11-28 16:09:34"	"8481-28-94 02:45:38"	"2024-11-12 20:38:53"	1315654996

4D324A694F47	4A6C4E475A694E	customers	6437700402244899122	""	""	""	""	NULL	""	NULL	NULL	""	1785228634	NULL	-733399729	1409286144	0	1008599040	4160794702	NULL	NULL	"3872-23-31 75:22:70"	NULL	NULL

4E4467335A6D	4E6B4D5463324F	customers	6505253056628084332	""	""	""	""	NULL	""	NULL	NULL	""	1196845389	NULL	-999527335	1409286144	0	1126039553	821948505	"2026-07-04 23:14:06"	NULL	"5072-45-86 92:41:22"	"2025-07-05 05:48:36"	1315523946

4E324E6C5954	59794D4441784F	customers	6515954856686474546	""	""	""	""	NULL	""	NULL	NULL	""	1464757070	NULL	-1002018470	1459617792	0	790495232	2432538702	"1996-09-29 02:15:25"	NULL	NULL	NULL	1415866447

-- Page id: 4, Found records: 29, Lost records: NO, Leaf page: YES

Table structure file
FRM file
IBD file

I run MySQL 5.7 on Windows Server 2012 R2 build 9600.
Run undrop-for-innodb on Ubuntu 20.4

Please help me!

datetime column problem

Hello, I try to restore a table with two datetime columns, one column restores the correct data, but the other is wrong,
correct column "2022-09-14 12:31:20.0" wrong column "9265-80-81 12:64:00",
Why?

Need a windows version

Need a windows version without the need for git etc, just an executable.

Documentation site is off

Any ideas on when will https://twindb.com/ be back ?

Thanks for the tool. Question

I really appreciate what you have done here. I have used stream-parser on an IBD table-only file, and it returned 3 index page-files. I used c-parser on each one. Only the first had valid data. The second and third were completely garbled (the field data was garbled). I used the -6 option, and both the -U and -D options.

Can you explain why sometimes the output is garbled?

Could you update the README or wiki documentation to explain the difference between the c_parser -U and -D options? The latter results in more corrupted output.

In my runs, I see a lot of errors like:

-- #####CannotOpen_./0000000368541696.page;
-- print_field_value_with_external(): open(): No such file or directory

Here is the table structure. Perhaps this provides insights?

CREATE TABLE `xxxx` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `book_datetime` datetime DEFAULT NULL,
  `start_datetime` datetime DEFAULT NULL,
  `end_datetime` datetime DEFAULT NULL,
  `notes` mediumtext COLLATE utf8mb4_german2_ci,
  `hash` mediumtext COLLATE utf8mb4_german2_ci,
  `is_unavailable` tinyint(4) DEFAULT '0',
  `id_users_provider` int(11) DEFAULT NULL,
  `id_users_customer` int(11) DEFAULT NULL,
  `id_services` int(11) DEFAULT NULL,
  `id_google_calendar` mediumtext COLLATE utf8mb4_german2_ci,
  `appointment_type` varchar(40) COLLATE utf8mb4_german2_ci DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `id_users_customer` (`id_users_customer`),
  KEY `id_services` (`id_services`),
  KEY `id_users_provider` (`id_users_provider`),
) ENGINE=InnoDB AUTO_INCREMENT=8869 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_german2_ci;

(constraints removed)

c_parser outputting strange numbers in the SQL file

Hi Aleksandr,

After trying to extract the SQL I am getting some strange numbers in the tabs. Any idea why this is happening and what I can do to try and resolve it please?

Page id: 827904, Format: COMPACT, Records list: Valid, Expected records: (41 41)
0000273A0000 4245B3B00700 27C0321D0C5768 tbl_requests 1635000436 "ype of venue are you looking for? <strong class="service_value">Hall

Do you have a preferred venue location

Building on FreeBSD

FYI - To build undrop on FreeBSD you have to use gmake (GNU make) and following changes are required:

diff --git a/include/my_config.h b/include/my_config.h
index 6c56449..c1c9a56 100644
--- a/include/my_config.h
+++ b/include/my_config.h
@@ -42,7 +42,7 @@
 
 /* Define to 1 if you have <alloca.h> and it should be used (not on Ultrix).
    */
-#define HAVE_ALLOCA_H 1
+#undef HAVE_ALLOCA_H
 
 /* Builds Archive Storage Engine */
 /* #undef HAVE_ARCHIVE_DB */
@@ -893,10 +893,10 @@
 /* #undef HAVE_UCHAR */
 
 /* system headers define uint */
-#define HAVE_UINT 1
+#undef HAVE_UINT
 
 /* system headers define ulong */
-#define HAVE_ULONG 1
+#undef HAVE_ULONG
 
 /* Define to 1 if you have the <unistd.h> header file. */
 #define HAVE_UNISTD_H 1
@@ -1083,7 +1083,7 @@
 #define SYSTEM_TYPE "unknown-linux-gnu"
 
 /* Whether we build for Linux */
-#define TARGET_OS_LINUX 1
+#undef TARGET_OS_LINUX
 
 /* Define if you want to have threaded code. This may be undef on client code
    */

Innodb Page Compression

It seems one of the ibd files I'm looking to recover (I altered the table thinking I was doing a "create like", so some columns got dropped) has a mix of compressed pages and uncompressed. I'm able to extract X number or rows that I can visibility see in the ibd file that I believe was before I set the page compression on a year ago. However everything since then is not being pulled out, I believe its because of this page compression, I see in the 36mb file a bunch of compressed looking text (believe zlib) however the extracted data results in 5.5mb or so

Does this tool support decompressing pages? Will it?

recipe for target 'steam_parser' failed

Hi.
I have ubuntu x64, installed all pre-requisites and try to compile with 'make' with root access.
This comes up

cc -D_FILE_OFFSET_BITS=64 -Wall -g -03 -pipe -I./include -c steam_parser.c
cc -D_FILE_OFFSET_BITS=64 -Wall -g -03 -pipe -I./include .pthread -lm stream_parser.o -o stream_parser
stream_parser.o: in the function 'process_ibfile":
/home/ubuntu/undrop-forinnodb/stream_parser.c:387: riferimento non definito a "process_ibpage"
collect2: error: Id returned 1 exit status
Makefile:31: recipe for target 'stream_parser' failed
make: *** {stream_parser] Error 1

documents are not available

Hi , I tried to read the mentioned documents in this url
https://twindb.com/recover-innodb-table-after-drop-table-innodb/
but the site is unreachable , how can I access the website and documents ?

deleted_records_only

when looking for deleted records only, why we consider all pages are not valid?(in c_parser.c, line 512)