Giter Site home page Giter Site logo

seunmatt / mysql-backup4j Goto Github PK

View Code? Open in Web Editor NEW
122.0 9.0 69.0 169 KB

mysql-backup4j is a library for programmatically exporting mysql databases and sending the zipped dump to email, Amazon S3, Google Drive or any other cloud storage of choice

License: MIT License

Java 100.00%
java mysql database-management database mysql-backup4j

mysql-backup4j's Introduction

mysql-backup4j

SeunMatt

Maven Central

mysql-backup4j is a library for programmatically exporting mysql databases and sending the zipped dump to email, Amazon S3, Google Drive or any other cloud storage of choice

It gives the developer access to the generated zip file and the generated SQL query string for use in other part of the application.

It also provides a method for importing the SQL exported by the tool - programmatically.

Installation

The artifact is available on Maven Central and can be added to the project's pom.xml:

<dependency>
    <groupId>com.smattme</groupId>
    <artifactId>mysql-backup4j</artifactId>
    <version>1.3.0</version>
</dependency>

The latest version can be found here

Usage

The minimum configuration required for the library is the database name, username and password.

However, if you want the backup file to be sent to your email automatically after backup, you must provide email configurations as well.

//required properties for exporting of db
Properties properties = new Properties();
properties.setProperty(MysqlExportService.DB_NAME, "database-name");
properties.setProperty(MysqlExportService.DB_USERNAME, "root");
properties.setProperty(MysqlExportService.DB_PASSWORD, "root");
properties.setProperty(MysqlExportService.DB_HOST, "localhost");
properties.setProperty(MysqlExportService.DB_PORT, "3306");
        
//properties relating to email config
properties.setProperty(MysqlExportService.EMAIL_HOST, "smtp.mailtrap.io");
properties.setProperty(MysqlExportService.EMAIL_PORT, "25");
properties.setProperty(MysqlExportService.EMAIL_USERNAME, "mailtrap-username");
properties.setProperty(MysqlExportService.EMAIL_PASSWORD, "mailtrap-password");
properties.setProperty(MysqlExportService.EMAIL_FROM, "[email protected]");
properties.setProperty(MysqlExportService.EMAIL_TO, "[email protected]");

//optional email configs
properties.setProperty(MysqlExportService.EMAIL_SSL_PROTOCOLS, "TLSv1.2");
properties.setProperty(MysqlExportService.EMAIL_SMTP_AUTH_ENABLED, "true");
properties.setProperty(MysqlExportService.EMAIL_START_TLS_ENABLED, "true");

//set the outputs temp dir
properties.setProperty(MysqlExportService.TEMP_DIR, new File("external").getPath());

MysqlExportService mysqlExportService = new MysqlExportService(properties);
mysqlExportService.export();

Calling mysqlExportService.export(); will export the database and save the dump temporarily in the configured TEMP_DIR

If an email config is supplied, the dump will be sent as an attachment. Finally, when all operations are completed the temporary dir is cleared and deleted.

If you want to get the generated backup file as a Java File object, you need to specify this property as part of the configuration:

//...
properties.setProperty(MysqlExportService.PRESERVE_GENERATED_ZIP, "true");
properties.setProperty(MysqlExportService.PRESERVE_GENERATED_SQL_FILE, "true");

and then you can call this method:

File file = mysqlExportService.getGeneratedZipFile();

Finally, let's say for some reason you want the generated SQL string you can do this:

String generatedSql = mysqlExportService.getGeneratedSql();

Other parameters are:

properties.setProperty(MysqlExportService.ADD_IF_NOT_EXISTS, "true");
properties.setProperty(MysqlExportService.JDBC_DRIVER_NAME, "com.mysql.cj.jdbc.Driver");
properties.setProperty(MysqlExportService.JDBC_CONNECTION_STRING, "jdbc:mysql://localhost:3306/database-name?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&useSSL=false");

They are explained in a detailed manner in this tutorial

Importing a Database

To import a database, you need to use the ImportService like so:

String sql = new String(Files.readAllBytes(Paths.get("path/to/sql/dump/file.sql")));

boolean res = MysqlImportService.builder()
        .setDatabase("database-name")
        .setSqlString(sql)
        .setUsername("root")
        .setPassword("root")
        .setHost("localhost")
        .setPort("3306")
        .setDeleteExisting(true)
        .setDropExisting(true)
        .importDatabase();
        
assertTrue(res);

First get SQL as a String and then pass it to the import service with the right configurations.

Alternatively, you can also use the .setJdbcConnString(jdbcURL) method on the import service.

e.g.

boolean res = MysqlImportService.builder()
                .setSqlString(generatedSql)
                .setJdbcConnString("jdbc:mysql://localhost:3306/backup4j_test?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&useSSL=false")
                .setUsername("db-username")
                .setPassword("db-password")
                .setDeleteExisting(true)
                .setDropExisting(true)
                .importDatabase();

setDeleteExisting(true) will delete all data from existing tables in the target database.

While setDropExisting(true) will drop the table.

Supplying false to these functions will disable their respective actions.

NOTE: The import service is only guaranteed to work with SQL files generated by the export service of this library

CHANGELOG

v1.2.1 - Raises a new runtime exception MysqlBackup4JException if the required properties are not configured

Author

Seun Matt smattme.com with 💚

Contributions and Support

If you want to create a new feature, though not compulsory, but it will be helpful to reach out to me first before proceeding.

To avoid a scenario where you submit a PR for an issue that someone else is working on already.

Tutorials / Articles

mysql-backup4j's People

Contributors

antwpapak avatar delbono-smeup avatar seunmatt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mysql-backup4j's Issues

Empty database after import

Hi,

I was happy to find your tool to import a mysql dump, however after running the database is still empty?

Code I used:

    val ret = MysqlImportService.builder()
      .setJdbcConnString(container.jdbcUrl)
      .setUsername(container.username)
      .setPassword(container.password)
      .setDatabase(DATABASE_NAME)
      .setSqlString(dbDump)
      .importDatabase()

Not sure if user error or bug...

UTF-8 Persian/Arabic support

How can I back from DB when data is in Persian/Arabic?
currently, my backup contains bunch of ???? instead of real data.

bug

bug:
connection = MysqlBaseService.connectWithURL(username, username,
jdbcConnString, jdbcDriver);

Import fails when using a database user with empty password

This is happening because of the validation that checks that
the password is not null and empty on line 145 of MysqlImportService

Example code to reproduce

String sql = new String(Files.readAllBytes(Paths.get("external/test_output_file_name.sql")));
        MysqlImportService res = MysqlImportService.builder();
        res.setJdbcDriver("com.mysql.cj.jdbc.Driver");
        res.setJdbcConnString("jdbc:mysql://localhost:3306/backup4j_test?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&useSSL=false");
        res.setSqlString(sql);
        res.setUsername("smatt");
        res.setPassword("smatt");
        res.setDropExisting(true);
        assertTrue(res.importDatabase());

[Feature Request] Avoid calling clearTempFiles() by default while export()

Use Case:
We need to be able to grab the exported files(specifically the zip) use them in some process (example send them to an SFTP server) and then call clearTempFiles() to delete the files.

Right now this is not possible as clearTempFiles() is call by default while exporting and if i set PRESERVE_GENERATED_ZIP i can use the zip file as i want but then when i called clearTempFiles() the files are not cleaned.

This will give more flexibility. So please take into account.

Code of JDBC_CONNECTION_STRING can not work well for MySQL8's jdbc url

Problem

Cause the jdbc url of MySql8 sometimes is like "jdbc:mysql://localhost:3306/test?characterEncoding=utf-8&useSSL=true&serverTimezone=Asia/Shanghai"

at MysqlExportService.java:385 database = jdbcURL.substring(jdbcURL.lastIndexOf("/") + 1, jdbcURL.indexOf("?"));, the value of jdbcURL.lastIndexOf("/") is larger than the value of jdbcURL.indexOf("?"),then it will cause String index out of range

sometimes can export Blob data but cannot import them

problem

I using Blob/TinyBlob/MediumBlob in MySql8. Sometimes i can export data from database but i cannot import them when read Blod data from file, java.sql will throw exception because MysqlExportService.java_Line:258 will convert binary data to string and some strange characters in the string which will cause the subscripts to cross the bounds.

I think we need to add judgment sentence when the type is LONGVARBINARY and save data like 0x00001abc maybe can solve this problem

javax.mail.MessagingException: Could not convert socket to TLS;

image

javax.mail.MessagingException: Could not convert socket to TLS;
nested exception is:
javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
at com.sun.mail.smtp.SMTPTransport.startTLS(SMTPTransport.java:1652)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:534)
at javax.mail.Service.connect(Service.java:313)
at javax.mail.Service.connect(Service.java:172)
at javax.mail.Service.connect(Service.java:121)
at javax.mail.Transport.send0(Transport.java:190)
at javax.mail.Transport.send(Transport.java:120)
at com.smattme.EmailService.sendMail(EmailService.java:149)
at com.smattme.MysqlExportService.export(MysqlExportService.java:298)
at com.atcollection.dashboard.setting.Setting.lambda$emailBackup$0(Setting.java:205)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)

Getting java.sql.SQLException when calling export method

I'm getting java.sql.SQLException: Operation not allowed for a result set of type ResultSet.TYPE_FORWARD_ONLY when calling export method.
This method seems to finish successfully. I got the email message with attached zip file, but its content has only 'CREATE' statements, there aren't 'INSERT' statements.

Reimporting a single mysql table containing a longblob column throws exceptions

I created a MySQL database containing a single table with a primary key column, some describing meta data columns and a longblob column. Then I added 3 records containing the meta data. Finally I used MySQL Workbench to upload the images by using its "Load Value From File..." function. Using the MySQL Workbench Value Editor I was able to display the Images correctly. After that I used the MysqlBackup4JIntegrationTest class to verify both export and import of the database. While export ran without complications the import threw exceptions.

I noticed the recommendation to store the images hexadecimally encoded but I have to handle a customer database in which all blobs are already filled up. So I tried to setup my test database nearly the same way like the customer did it.

Hope you can help me by solving the issue.

Perhaps in MysqlExportService the method getDataInsertStatement should should handle blobs as hexadecimally encoded byte arrays rather than handle them as strings?

Here I uploaded my sample dump:

mysqlbackup4j.dump.zip

Not able to configure additional SMTP properties

EmailService receive some properties like mail.smtp.auth, mail.smtp.starttls.enable, mail.smtp.host, mail.smtp.port, mail.smtp.ssl.trust but is not possible to setup neither mail.smtp.ssl.protocols nor mail.smtp.starttls.required.

Backslashes in text fields are lost after backup and restore

Backslashes in String fields are lost after backup and restore.

Original text field content: "data\with\backslash";

mysql-backup4j produces an insert statement like this:
INSERT INTO [...] VALUES ('data\with\backslash')

After restore, the field contains the value "datawithbackslash". This is because MySQL requires backslashes in SQL to be escaped.

How to upload on any cloud?

Can you please show the information or code to upload on any cloud? Because in your code its not mentioned.

I'm getting Invalid login or password? but, I put my right credentials.. What is the reason?

image

Stack Trace:

javax.mail.AuthenticationFailedException: 535 5.7.0 Invalid login or password

at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:648)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:583)
at javax.mail.Service.connect(Service.java:313)
at javax.mail.Service.connect(Service.java:172)
at javax.mail.Service.connect(Service.java:121)
at javax.mail.Transport.send0(Transport.java:190)
at javax.mail.Transport.send(Transport.java:120)
at com.smattme.EmailService.sendMail(EmailService.java:149)

Dropping tables with foreign key constraints fails

When trying to restore a database that has foreign key constraints - with dropExisting enabled - the MysqlImportService returns e.g.:
Cannot drop table 'address' referenced by a foreign key constraint 'FKditu6lr4ek16tkxtdsne0gxib' on table 'users'.

Probably needed to disable foreign key checks before executing the drop.

clearTempFiles(false) not available anymore in version 1.2.1 wich is the latest.

          > This is already possible. Set the property `MysqlExportService.PRESERVE_GENERATED_ZIP` to `true`.
//...
properties.setProperty(MysqlExportService.PRESERVE_GENERATED_ZIP, "true");
MysqlExportService mysqlExportService = new MysqlExportService(properties);
mysqlExportService.export();
File file = mysqlExportService.getGeneratedZipFile();

You can read more about it here https://smattme.com/posts/how-to-backup-mysql-database-programmatically-using-mysql-backup4j/

This is not available anymore in version 1.2.1 wich is the latest.

You can verify from the master branch also the method doesnt take any parameter: https://github.com/SeunMatt/mysql-backup4j/blob/master/src/main/java/com/smattme/MysqlExportService.java#L469

From the link you provided the boolean parameter was indeed supported for 1.0.1

Was there any reason to remove it? Otherwise it is a regression and you might want to add it back

Originally posted by @albilu in #33 (comment)

unknown file encoding

Hi
when i want to resore backup file in workbench it give me this message unknown file encoding it say its not utf8
its work fine in macos but in windows it has this issue
how can i fix this issue
thanks

Split insert query during export when they exceed a configurable length

When exporting a large datebase, the generated insert sql statements may exceed the system limits. Trying to execute the query on MariaDB triggers the following error :
SQL Error (1153): Got a packet bigger than 'max_allowed_packet' bytes

Increasing the 'max_allowed_packet' value is a workaround, hower, adding an "insert size" limit, like in HeidiSQL, would be a better way to solve the issue by splitting the insert statement in several ones.

A default 0 value for this limit would keep the current behaviour.

java.sql.BatchUpdateException: Not a valid escape sequence

I have some java object in database, and export works fine but on import I'm getting an error:

Exception in thread "main" java.sql.BatchUpdateException: Not a valid escape sequence: {DÊû>
°�²Ë?<�sTXKC«��̱��0b?²�7áý|!Àkn¢¾�;7¥Gó?µF�+?s§Â­ô?_�nÇ¥?2�Í?vsìÃíâO*

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.