rust-db / refinery Goto Github PK
View Code? Open in Web Editor NEWPowerful SQL migration toolkit for Rust.
License: MIT License
Powerful SQL migration toolkit for Rust.
License: MIT License
See also #70 (comment)
Running tests in refinery/
seem to be only configured for Travis, but according to badges on the README CI is done with CircleCI.
afair, when we started developing refinery, refinery_cli
used functions from refinery
that were not meant to be public, since refinery_cli
also has to be an independent crate , we moved them to refinery_migrations
.
We have since pub'ed migrate_from_config
, so that reason doesn't seem to be legit anymore.
from a quick glance at refinery_cli
it seems the only function that is still used without being doc'ed and oficially pub'ed is find_migration_filenames
, maybe we can also make it pub and document it to be able to be used with migrate_from_config
, and then merge refinery_migrations
into refinery
?
for a 0.2 I think then we could:
I'm going to preface this with I'm a rust newb. I am trying to build a server app using actix/sqlx and I'm using a custom config struct and all I want to do is pass a postgres url to configure refinery - but there doesn't seem to be a way to do that.
How can I run migrations in a selected schema postgresql? Always it runs the migrations in the public schema and I don't see how can I choose it in the documentation.
Update:
Even if I choose schema with barrel, doesn't work
I get the following compiler error while trying to compile a "hello world" project using mysql_async:
Compiling refinery-migrations v0.2.0
error: expected an item keyword
--> C:\Users\####\.cargo\registry\src\github.com-1ecc6299db9ec823\refinery-migrations-0.2.0\src\config.rs:259:21
|
259 | let url = build_db_url("mysql", &config);
| ^^^
error: aborting due to previous error
error: could not compile `refinery-migrations`.
To learn more, run the command again with --verbose.
Please, find attached a minimal example that shows the error. Doing a cargo build (with up to date stable rust) produces the given error.
Hi, sorry if this already answered here somewhere, but I am curious if support for mongodb and other NoSql databases is on the roadmap or if I should be looking at another tool for doing mongo migrations for my rust micro-services.
Thanks!
Currently, performing migrations does not return any information. It may be useful to return information about the migrations that were successfully applied during a run.
My use case here is simply that I'd like to see what, if anything, was changed as a result of running a migration. At present I manually query the refinery_schena_history for and after applying migrations in order to find out what changed.
My project used to have self-written migration manager, now I'm trying to figure out if I can explain to refinery how my migration version control is organized. In other words, before refinery
retrieves previously applied migration versions, I need to check if database uses old fashion version control and if it does, retrieve applied versions in old-fashion way and make refinery
think that those migrations have been applied as if refinery
had applied them by itself.
Diesel_migrations exposes trait MigrationConnection so I can simply wrap it and make it know if some migrations have been applied earlier. In contrast, refinery has Query trait (which is subset of MigrationConnection) but hides it.
I think idea of having MigrationConnection is good enough to reuse.
I've been searching around for some way to manage my database migrations in Rust. I'm considering using Diesel, but Diesel only supports defining migrations in SQL. I anticipate wanting to make migrations which would be difficult or impractical to represent in SQL alone - modifying blobs, making complex format changes, etc. - so that is a non-starter for me.
Refinery's README says that Refinery's design is based on Flyway and it supports defining migrations in Rust code instead of SQL, which looked promising. Flyway's document for Java-based migrations says explicitly that this kind of use case is what they're targeted at:
Java-based migrations are a great fit for all changes that can not easily be expressed using SQL.
These would typically be things like
- BLOB & CLOB changes
- Advanced bulk data changes (Recalculations, advanced format changes, …)
However, it doesn't seem Refinery's "migrations in a Rust module" support gets me what I need. For example, see Flyway's getting started tutorial for Java-based migrations: https://flywaydb.org/getstarted/java. Implementing this tutorial in the same way does not seem to be possible in Refinery, as the migration would need access to the database connection. (The specific modification in the example is trivial to implement in SQL, but imagine calculating the updated name was an operation that was impractical in SQL.)
Are there any plans to support more complex, manually coded migrations in Refinery?
Hi,
Currently refinery seems to fail to build under stable-x86_64-pc-windows-gnu
and nightly-x86_64-pc-windows-gnu
.
stable-x86_64-pc-windows-msvc
on the other hand builds refinery fine.
Error message:
Compiling refinery v0.2.1
error: could not compile `refinery`.
Caused by:
process didn't exit successfully: `rustc
--crate-name refinery
--edition=2018 'C:\Users\almetica\.cargo\registry\src\github.com-1ecc6299db9ec823\refinery-0.2.1\src\lib.rs'
--error-format=json
--json=diagnostic-rendered-ansi,artifacts
--crate-type lib
--emit=dep-info,metadata,link -C debuginfo=2
--cfg 'feature="default"'
--cfg 'feature="postgres"' -C metadata=b4b0923c9635f1a1 -C extra-filename=-b4b0923c9635f1a1
--out-dir 'C:\Users\almetica\Documents\Development\almetica\almetica\target\debug\deps' -L 'dependency=C:\Users\almetica\Documents\Development\almetica\almetica\target\debug\deps'
--extern 'barrel=C:\Users\almetica\Documents\Development\almetica\almetica\target\debug\deps\libbarrel-c14e7acdbe063085.rmeta'
--extern 'refinery_macros=C:\Users\almetica\Documents\Development\almetica\almetica\target\debug\deps\refinery_macros-fdbe15d768c175f7.dll'
--extern 'refinery_migrations=C:\Users\almetica\Documents\Development\almetica\almetica\target\debug\deps\librefinery_migrations-ff4312ab516cca44.rmeta'
--cap-lints allow -C target-cpu=native -L 'native=C:\Users\almetica\.cargo\registry\src\github.com-1ecc6299db9ec823\winapi-x86_64-pc-windows-gnu-0.4.0\lib'`
(exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)
warning: build failed, waiting for other jobs to finish...
error: build failed
Maybe rust-lang/rust#33434 is related?
Edit: --release
works fine. So only debug build is affected. I will try to pinpoint the exact cause further.
Hello,
I'm a Rust beginner, and I've tried to set this crate up to do migrations on my database. Unfortunately, I'm getting the following error:
error[E0599]: no method named `run` found for struct `refinery::Runner` in the current scope
--> wopplebloxd/src/db/migrations.rs:22:40
|
22 | embedded::migrations::runner().run(&mut connection);
| ^^^ method not found in `refinery::Runner`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0599`.
For context, here's my migrations.rs
file:
use super::definitions::{ Connection };
// use rusqlite::Connection;
mod embedded {
use refinery::embed_migrations;
embed_migrations!("wopplebloxd/migrations");
}
pub struct SqliteMigrator;
impl Default for SqliteMigrator {
fn default() -> SqliteMigrator {
SqliteMigrator {
}
}
}
impl SqliteMigrator {
pub fn migrate(&self, connection : &mut Connection) {
embedded::migrations::runner().run(&mut connection);
}
}
....which references definitions.rs
:
pub type Pool = r2d2::Pool<r2d2_sqlite::SqliteConnectionManager>;
pub type Connection = r2d2::PooledConnection<r2d2_sqlite::SqliteConnectionManager>;
I'm using r2d2 & r2d2-sqlite, as I'm building a web app with actix. If I don't do that, then I get errors because the plain-old rusqlite::Connection
can't be shared across multiple threads (which are spawned automatically by actix).
I also have a migrations
folder that contains a single .sql
file called V1__initial.sql
.
Hey there,
the README
currently states that
If you are using a driver that is not yet supported, namely SQLx you can run migrations providing a Config instead of the connection type, as Config impl's Migrate. You will still need to provide the postgres/mysql/rusqlite driver as a feature for Runner::run and tokio-postgres/mysql_async for Runner::run_async.
Having never used this crate i can't quite make sense of that passage. I see that you could provide a Config
to the Runner::run
method. But i'm not sure how to initialize it, considering i want to use SQLx with SQLite. And how can i pass the the rusqlite driver along with the Config
?
An example, or some pointers, of how to use refinery alongside SQLx and SQLite would be greatly appreciated.
From the readme/Rollback section:
refinery's design is based on flyway and so, shares its perspective on undo/rollback migrations. To undo/rollback a migration,
you have to generate a new one and write specifically what you want to undo.
However according to this, undo is actually supported by Flyway.
Personally, I don't see how it is possible to switch between different branches of my project and work with the same database if refinery does not allow you to rollback changes.
My proposal is it to:
Runner
struct has a migrations
property, but it is not public. Would be nice to expose that, and maybe even some convenience methods:
get_last_known_migration()
are_all_migrations_applied()
Hello, and thanks for this crate!
I was wondering whether it's on the roadmap to support running migrations up to a specific version, rather than all the way. If not, I'd be curious to know what your stance on the matter is (particularly because I am fairly inexperienced at the DB layer, so any insight is very welcome)!
Hi,
I assume I'm doing something wrong, but Refinery doesn't seem to want to apply my migrations.
jkaye@jkaye-linux:~/git/wanderlust$ cat refinery.toml
[main]
db_type = "Postgres"
db_host = "localhost"
db_port = "5432"
db_user = "postgres"
db_pass = "..."
db_name = "postgres"
jkaye@jkaye-linux:~/git/wanderlust$ ls migrations/
1_create_users.sql
jkaye@jkaye-linux:~/git/wanderlust$ cat migrations/1_create_users.sql
create table external_auth_source
(
external_auth_source_id serial primary key,
source_name varchar(255) not null,
constraint ux_external_auth_source_name unique(source_name)
);
create table traveler
(
traveler_id serial primary key,
external_auth_source_id int not null foreign key references external_auth(external_auth_source_id),
external_auth_id varchar(255) not null,
constraint ux_traveler_external_auth unique(external_auth_source_id, external_auth_id)
);
jkaye@jkaye-linux:~/git/wanderlust$ refinery migrate files
schema history table is empty, going to apply all migrations
no migrations to apply
Any ideas?
migrate_from_config
was left out private again on #43.
Should it be re-publicated again? If so, should also find_migration_files
be public?
I'd like to log "Applying X" before each migration starts, to be able to know what's going on. As far as I know, currently, you only know once everything has been done.
The README.md
implies that the project is licensed only under MIT:
License
This project is licensed under the MIT license.
But the Cargo.toml
manifests suggest it's dual licensed under MIT or Apache-2.0:
[package]
name = "refinery"
version = "0.2.0"
authors = ["Katharina Fey <[email protected]>", "João Oliveira <[email protected]>"]
license = "MIT OR Apache-2.0"
I think this should be clarified. If dual license was the intent, I'd follow the recommendations at https://rust-lang.github.io/api-guidelines/necessities.html#crate-and-its-dependencies-have-a-permissive-license-c-permissive
Because refinery config does not support database params, my current code looks like this:
let url = Url::parse(&db_url).context("invalid DATABASE_URL")?;
let mut db = Config::new(ConfigDbType::Postgres);
if !url.username().is_empty() {
db = db.set_db_user(url.username());
}
if let Some(password) = url.password() {
db = db.set_db_user(password);
}
if let Some(host) = url.host_str() {
db = db.set_db_host(host);
}
if let Some(port) = url.port() {
db = db.set_db_port(&port.to_string());
}
let mut db_name = url.path().trim_start_matches('/').to_string();
if let Some(query) = url.query() {
db_name.push('?');
db_name.push_str(query);
// ^^^^^ this causes the database url to be correct,
// but is results in failure to connect over TLS
}
if !db_name.is_empty() {
db = db.set_db_name(&db_name);
}
Cargo.toml
[dependencies]
refinery = { version = "0.3.0", features = ["postgres"] }
Error
Error: `could not connect to database`, `error performing TLS handshake: no TLS implementation configured`
Caused by:
0: error performing TLS handshake: no TLS implementation configured
1: no TLS implementation configured
Hello,
First, thanks for the library.
I've tried to use this method with a postgresql database to start migration:
let mut db_config = Config::new(ConfigDbType::Postgres).set_db_path(env::var("postgres_pool").unwrap().as_str());
My env variable is with an url format. I've also tried key value format but i could not connect to my database. Here is the error message:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Connection("could not connect to database", Error { kind: Config, cause: Some("host missing") }), report: None }',
Am I using set_db_path
method correctly? If not, is there a way to pass url or key value postgres path to refinery Config?
Thanks !
Nice project, I tried to use it with one of my projects but stumbled across a limitation in the refinery_cli. It only allows a single statement in any .sql
file. In my case I want to create multiple tables but it fails with the error
Error:
error applying migration V2__demo
,database error: ERROR: cannot insert multiple commands into a prepared statement
.sql
file:
create table roles (
role varchar(32) PRIMARY Key,
);
create table powers (
role_id SERIAL PRIMARY KEY,
);
refinery::Runner
docs don't include the Runner::run
and Runner::run_async
functions.
I noticed that docs metadata was commented out in Cargo.toml
:
# [package.metadata.docs.rs]
# features = ["postgres", "mysql", "sqlite", "extras"]
# no-default-features = true
was this intentional?
Also, the links for embedded_migrations
and include_migration_mods
are broken in https://docs.rs/refinery/0.2.0/refinery/index.html.
Is it possible to add a database URL environment variable as a fallback if the config file is missing?
The DefaultHasher
intentionally omits details of what algorithm is used:
The internal algorithm is not specified, and so it and its hashes should not be relied upon over releases.
sha1
is probably a better choice.
Hey! So, today I stumbled about possible improvement to how migration loading is handled. Currently, when using the sql-files based approach, refinery does not report SQL files in the migrations directory that do not conform to the naming convention. It wont say a word about having something like "V1_name" instead of "V1__proper_name" (bad version prefix). Maybe this is a possible improvement to make?:)
I was planning to try refinery_cli for some postgres work, but noticed the crate depends on libsqlite3 whether or not I want to use sqlite.
I didn't have it installed on my system at the time, so the build failed. I thought I'd mention it here as a feature request. Seems like others may have specific dbs in mind and will also want to pick and choose.
Thanks!
Hi,
It's pretty common to use time based version numbers for migration files (e.g. migrations of Active Record or Diesel). The current implementation of refinery is only using the column type INT4 for the version column, which is too small to hold the current time based on a YEAR-MONTH-DAY-HOUR-MINUTE basis.
Is there any special reason not to use INT8 for the version column? The rust code seems to use a i32, so a switch to i64 should be possible.
(I'm using a postgres database).
While investigating an int conversion issue when using CockroachDB, I noticed that migrations didn't work at all on the master branch:
INFO refinery_migrations::traits::r#async > no migrations to apply
I'll see this even after dropping all the tables, including the schema migrations table.
This seems to be a regression from the version on crates.io. I didn't see a branch for the older version but was able to dig out the release commit for 0.2
. I applied the int conversion patch to confirm that the old version works as expected (and it does).
First, 💯 for the project!
The refinery_cli
creates a refinery.toml
config file when running refinery setup
-- however, refinery migrate
expects the file to be called Refinery.toml
As a passing comment, in the cli readme, the examples use refinery_cli
, but the executable seems to be called refinery
when installing as suggested with cargo install refinery_cli
.
waiting on prisma/tiberius#28
I am getting
Error: `error asserting migrations table`, `db error: ERROR: no database specified`
when refinery
tries to create the migrations table. What database should the migrations
table be created in?
This is on linux, against CockroachDB (using postgres driver) -- I suspect the latter is more pedantic than postgres about specifying a database (but it's just a theory).
Currently refinery depends on rusqlite 0.18, while the latest version is 0.21. Because rusqlite links to a native library, it is not possible to have multiple versions of it in the same build. Consider upgrading to the latest version of rusqlite.
Hello,
Adding refinery 0.3.0 to Cargo.toml on windows 10 crashes rustc 1.47.0 (18bf6b4f0 2020-10-07).
I don't know if this is known and do not know how to report this kind of issue to rust-lang.
Edit: Problem still happens with 0.4.0
Django and other more mature ORMs such as TypeORM can automatically generate migrations based on the data structure. This increases productivity and frees users from having to keep DB layout and code data structures in sync.
In Enterprise settings, there are tooling that take the declarative database management concepts even further for more robust database management: https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/data-tier-applications?view=sql-server-2017
However for simple use cases the Django style automigrations will suffice.
Hello, I seem to have similar problem as #115.
➤ exa migrations/
V1__initial.sql V2__serverid.sql
➤ env RUST_LOG=info cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.13s
Running `target/debug/neomason-discord-bot`
[2020-07-11T18:21:28Z INFO refinery_core::traits] schema history table is empty, going to apply all migrations
[2020-07-11T18:21:28Z INFO refinery_core::traits::sync] applying migration: V1__initial
...
Despite having two migrations to run, refinery only run one. I tried looking at RUST_LOG=debug
output, but it didn't provide any additional output.
I also assume I'm doing something wrong, but my file seems to be named correctly.
Do you have any suggestions?
I'm getting an error like this:
error[E0277]: the trait bound `rusqlite::Connection: refinery::MigrateGrouped<'_>` is not satisfied
--> src/sqldata.rs:61:24
|
61 | migrations::runner().run(&mut conn).map_err(|e| e.into())
| ^^^ the trait `refinery::MigrateGrouped<'_>` is not implemented for `rusqlite::Connection`
error[E0277]: the trait bound `rusqlite::Connection: refinery::Migrate` is not satisfied
--> src/sqldata.rs:61:24
|
61 | migrations::runner().run(&mut conn).map_err(|e| e.into())
| ^^^ the trait `refinery::Migrate` is not implemented for `rusqlite::Connection`
What should I do to get these traits into scope for rusqlite::Connection?
In my Cargo.toml I added these (I'm also adding barrel)
barrel = { version = "0.6.2", features = ["sqlite3"] }
refinery = { version = "0.1.10", feature = ["rusqlite"] }
And in the file where I'm getting the error, I added some new 'use' stmts:
use barrel::backend::Sqlite;
use barrel::{types, Migration};
use migrations;
use refinery::embed_migrations;
use refinery::{Migrate, MigrateGrouped};
use rusqlite::{params, Connection};
And the actual migration code looks like this:
mod embedded {
use refinery::embed_migrations;
embed_migrations!("src/migrations");
}
pub fn dbinit(dbfile: &Path) -> Result<(), Box<dyn Error>> {
let conn = Connection::open(dbfile)?;
migrations::runner().run(&mut conn).map_err(|e| e.into())
}
I have a PR open on this project with all my changes:
I'd like for it to be possible to change the table name (and possibly the schema name) of the migrations table.
This could potentially be accomplished by passing a config argument to the runner
api; or alternatively by allowing optional arguments in the "include" macros.
When depending on rusqlite 0.24 and refinery 0.3, theres a version conflict of the underlying libsqlite3-sys crate version. The dependency of rusqlite = {version = ">= 0.23, < 0.25", optional = true}
in refinery-core does not seem to fix this.
Cargo.toml
# ...
[dependencies]
rusqlite = { version = "^0.24.1", features = ["uuid", "chrono", "serde_json", "load_extension"] }
refinery = { version = "^0.3.0", features = ["rusqlite"] }
# ...
Error
error: failed to select a version for `libsqlite3-sys`.
... required by package `rusqlite v0.23.0`
... which is depended on by `refinery-core v0.3.1`
... which is depended on by `refinery v0.3.0`
... which is depended on by mylib v0.2.2 (/home/rene/git/mylib)`
versions that meet the requirements `^0.18.0` are: 0.18.0
the package `libsqlite3-sys` links to the native library `sqlite3`, but it conflicts with a previous package which links to `sqlite3` as well:
package `libsqlite3-sys v0.20.1`
... which is depended on by `mylib v0.2.2 (/home/rene/git/mylib)`
failed to select a version for `libsqlite3-sys` which could resolve this conflict
Does refinery have an option to support r2d2_sqlite for connection pooling?
The specific error I'm getting is as follows:
]error[E0277]: the trait bound `&mut r2d2::PooledConnection<r2d2_sqlite::SqliteConnectionManager>: refinery_migrations::traits::sync::Query<std::vec::Vec<refinery_migrations::AppliedMigration>>` is not satisfied
--> wopplebloxd/src/db/migrations.rs:23:44
|
23 | embedded::migrations::runner().run(&mut connection);
| ^^^^^^^^^^^^^^^ the trait `refinery_migrations::traits::sync::Query<std::vec::Vec<refinery_migrations::AppliedMigration>>` is not implemented for `&mut r2d2::PooledConnection<r2d2_sqlite::SqliteConnectionManager>`
|
= note: required because of the requirements on the impl of `refinery::Migrate` for `&mut r2d2::PooledConnection<r2d2_sqlite::SqliteConnectionManager>`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0277`.
error: could not compile `wopplebloxd`.
To learn more, run the command again with --verbose.
I'm having to use connection pooling here because I'm setting up a web server with actix, and I'm storing my db connection in my global state object, which is access by multiple threads I assume that actix is spawning.
refinery 0.2.1
depends on refinery-migrations ^0.2.2
. refinery-migrations 0.2.2
depends on rusqlite ^0.21
, which depends on libsqlite3-sys ^0.17
. refinery-migrations 0.2.3
upgraded from rusqlite ^0.21
to rusqlite ^0.23
, which also upgraded libsqlite-3sys
to ^0.18
.
The refinery-migrations
update breaks builds for people who use other crates which currently depend on a different version of libsqlite3-sys
, like current sqlx-core
using ^0.17
. The rusqlite
upgrade should've been done in a new major version (0.3
) of refinery-migrations
and refinery
.
I would appreciate some help with my poor understanding of the type system...
let mut client = db_pool.get().await?.deref_mut().deref_mut();
let report = embedded::migrations::runner()
.run_async(&mut client)
.await?;
using deadpool with tokio-postgres and I get the following compile error (that I do not understand, really :)
the trait `refinery::AsyncMigrate` is not implemented for `&mut tokio_postgres::client::Client`
I'm just starting with Rust and hope this is not some dumb user error, but I'm confident I did the same as code in docs and tests.
I couldn't check if tests work though because I could not make them run locally as well (See also #71).
cargo run --bin migrations
Compiling barrel v0.6.6-alpha.0 (https://github.com/belak/barrel#504e8b7d)
Compiling database_core v0.1.0 (/home/user/source/bwu_app/database_core)
error[E0277]: the trait bound `postgres::client::Client: refinery_core::traits::sync::Query<std::vec::Vec<refinery_core::runner::AppliedMigration>>` is not satisfied
--> database_core/src/mig.rs:16:30
|
16 | migrations::runner().run(&mut client).unwrap();
| ^^^^^^^^^^^ the trait `refinery_core::traits::sync::Query<std::vec::Vec<refinery_core::runner::AppliedMigration>>` is not implemented for `postgres::client::Client`
|
= note: required because of the requirements on the impl of `refinery_core::traits::sync::Migrate` for `postgres::client::Client`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0277`.
error: could not compile `database_core`.
To learn more, run the command again with --verbose.
active toolchain
----------------
nightly-x86_64-unknown-linux-gnu (default)
rustc 1.43.0-nightly (dd6718796 2020-03-16)
[dependencies]
barrel = { git="https://github.com/belak/barrel", features = ["pg"]} # contains a PR that fixes a build error
postgres = "0.17.2"
refinery = { git = "https://github.com/rust-db/refinery" }
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.