frioux / dbix-class-deploymenthandler Goto Github PK
View Code? Open in Web Editor NEWHome Page: https://metacpan.org/pod/DBIx::Class::DeploymentHandler
Home Page: https://metacpan.org/pod/DBIx::Class::DeploymentHandler
The change from database_version to initial_version may cause issues in existing deployments - we had a client where we needed an override for the version result table, which meant that we needed to provide our own core class to pull in the override.
It may be worth adding either a handles on initial_version
, or a deprecation notice on database_version
in HandlesVersioning, or at least mentioning it in the Changes file that the database_version
attribute on VersionHandlers has been renamed.
The commit that brought in this change was fc54898 for reference
This seems wonky because if I deploy a version I should be on that version - but I also understand that deploy is handled directly by the HandlesDeploy
. But I'm working with more than one schema, so if I use install
again it dies.
I feel like I want to call this an issue but my only real problem is that it's odd that the user has to know to deploy and then manually add the version, where it would seem sensible that a deployment would be a procedure that adds the version for you.
Seems like one of three things would be nice:
deploy
when we install
(a la #50) but that might break a bunch of stuffdeploy
so it adds the versioninstall
if version storage is already installed, but just skip __VERSION
I defined up a (non virtual) view in a DBIC schema using DBIx::Class::ResultSource::View.
I was rather hoping that DBIC-DH would magically add the view definition to the upgrade/downgrade SQL sets.
Although the PostgreSQL/deploy/... file did contain the CREATE VIEW
the PostgreSQL/upgrade/... file did not mention the view
Checked this with Postgres and SQLite drivers.
For a ->prepare_upgrade to work correctly, a ->prepare_deploy must be called first.
This creates deploy ddl's along with the yml's which is contrary to good advice regarding using just one deploy (version 1) for installation and walking through the upgrade scripts to yield the requested version.
Currently the code is very DBIC centric. You must connect with a DBIC schema at the very least. We should factor out the "runner" part of the DM so that people can slot in whatever they want. So maybe a spartan user will only support .sql files run with a raw $dbh, or maybe someone special wants to use Rose::DB or DBIDM instead of DBIC for .pl migrations (probably a bad idea unless they have an ::SL replacement, but still.)
on clean system I get next error when run prepare
command:
Can't load producer 'SQL::Translator::Producer::YAML' : Error loading SQL::Translator::Producer::YAML as SQL::Translator::Producer::YAML : Can't locate YAML.pm in @INC (you may need to install the YAML module)
My cpanfile
is:
requires 'Carton';
requires 'Mojolicious';
requires 'Mojolicious::Plugin::PODRenderer';
requires 'DBD::Pg'; # by Database
requires 'DBIx::Class', '==0.082841';
requires 'DBIx::Class::Migration';
As the title says, the sqlt_deploy-hook can end up getting called twice, once for the versions tables only it seems and once for everything else.
Under 5.19.4, @{ foo {"bar"} } is always parsed as @foo{bar} -- in previous version of perl, it was parsed as either that or @{ ( foo{"bar"} ) }, depending on whenever there was a newline after the '@{', and on whenever you were inside of an eval STRING or not.
DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator uses '@{\nfunc {} args}', which now parses as a syntax error; The attached patch should fix it.
diff --git a/lib/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm b/
index 021ee2b..71b6429 100644
--- a/lib/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm
+++ b/lib/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm
@@ -486,10 +486,10 @@ sub _sql_from_yaml {
my @sql;
my $actual_file = $self->$from_file($version);
- for my $yaml_filename (@{
+ for my $yaml_filename (@{(
DlogS_trace { "generating SQL from Serialized SQL Files: $_" }
(ref $actual_file?$actual_file:[$actual_file])
- }) {
+ )}) {
my $sqlt = SQL::Translator->new({
add_drop_table => 0,
parser => 'SQL::Translator::Parser::YAML',
DBIx::Class::DeploymentHandler version 999 required--this is only version 0.002219.
This is perl 5, version 22, subversion 2 (v5.22.2)
I am maintaining an old project where I stupidly used "user" as a table name. this makes things unnecessarily painful because its a reserved word. but i have to stick with it.
I manually quoted the table name in ...
#db_upgrades/PostgreSQL/upgrade/57-58/001-auto.sql
DROP TABLE "user" CASCADE;
but now I get this error, possibly not related to the quoting stuff.
i follow the same procedure as in the past but something appears to have changed
david@nbdt:~/dev/EPPlication$ ./script/database.pl --cmd upgrade
NOTICE: drop cascades to constraint user_role_fk_role_id on table user_role
NOTICE: drop cascades to constraint test_tag_fk_tag_id on table test_tag
NOTICE: drop cascades to 4 other objects
DETAIL: drop cascades to constraint step_fk_test_id on table step
drop cascades to constraint job_fk_config_id on table job
drop cascades to constraint job_fk_test_id on table job
drop cascades to constraint test_tag_fk_test_id on table test_tag
NOTICE: drop cascades to 2 other objects
DETAIL: drop cascades to constraint job_fk_user_id on table job
drop cascades to constraint user_role_fk_user_id on table user_role
NOTICE: drop cascades to constraint step_result_fk_job_id on table step_result
failed to run Perl in /home/david/dev/EPPlication/script/../lib/EPPlication/Util/../../../db_upgrades/_common/upgrade/57-58/002_add_filter_param.pl: DBIx::Class::Schema::source(): Can't find source for Step at /home/david/dev/EPPlication/script/../lib/EPPlication/Util/../../../db_upgrades/_common/upgrade/57-58/002_add_filter_param.pl line 16
You are seeing this error because the DBIx::Class::ResultSource in your
migration script called "Step" is not part of the schema that ::Schema::Loader
has inferred from your existing database.
To help you debug this issue, here's a list of the actual sources that the
schema available to your migration knows about:
DbixClassDeploymenthandlerVersion
DBIx::Class::Storage::TxnScopeGuard::DESTROY(): A DBIx::Class::Storage::TxnScopeGuard went out of scope without explicit commit or error. Rolling back. at /home/david/perl5/perlbrew/perls/perl-5.22.2/lib/site_perl/5.22.2/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 321
DBIx::Class::Storage::TxnScopeGuard::DESTROY(): A DBIx::Class::Storage::TxnScopeGuard went out of scope without explicit commit or error. Rolling back. at /home/david/perl5/perlbrew/perls/perl-5.22.2/lib/site_perl/5.22.2/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 321
my upgrade script
#!/usr/bin/env perl
use strict;
use warnings;
use 5.010;
use EPPlication::Util::SchemaUpgradeHelper qw/ get_params set_params add_param change_param /;
use
DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator::ScriptHelpers
'schema_from_schema_loader';
schema_from_schema_loader(
{ naming => { ALL => 'v8', force_ascii => 1 } },
sub {
my ( $schema, $versions ) = @_;
my $step_rs = $schema->resultset('Step')->search( { type => 'PrintVars' } );
say "Add filter param.";
while ( my $step = $step_rs->next ) {
say "Upgrade for step " . $step->id;
my $params = get_params($step);
add_param( $params, 'filter', '' );
set_params($step, $params);
}
}
);
The 001-auto-__VERSION.sql looks like the following. (Oracle limits object names to 30 characters, so i changed the table name as described in your cookbook)
--
-- Created by SQL::Translator::Producer::Oracle
-- Created on Wed Jun 12 10:41:11 2013
--
;
--
-- Table: dbix_class_dh_versions
--;
CREATE SEQUENCE sq_dbix_class_dh_versions_id;
CREATE TABLE dbix_class_dh_versions (
id number NOT NULL,
version varchar2(50) NOT NULL,
ddl clob,
upgrade_sql clob,
PRIMARY KEY (id),
CONSTRAINT u_dbix_class_dh_versions_dbix_ UNIQUE (version)
);
CREATE OR REPLACE TRIGGER ai_dbix_class_dh_versions_id
BEFORE INSERT ON dbix_class_dh_versions
FOR EACH ROW WHEN (
new.id IS NULL OR new.id = 0
)
BEGIN
SELECT sq_dbix_class_dh_versions_id.nextval
INTO :new.id
FROM dual;
END;
;
The Problem is in the parsing. Currently your split-method removes the BEGIN and push the END; Block in an seperate SQL Chunk.
This changes are working for me:
sub _split_sql_chunk {
# Oracle Producer
# keep END's in Trigger
my @sql = map { split /(?<!END);\n(?!END;)/i, $_ } @_;
for ( @sql ) {
# strip transactions
s/^(?:BEGIN TRANSACTION|COMMIT).*//mgi;
# Oracle Producer
# keep BEGIN's in Trigger
s/^(?<!\)\n)BEGIN//mgi;
#...
We have some custom code that extends this and uses DBIx::Class::DeploymentHandler::WithApplicatorDumple directly. This change has broken things, since Moose does not consider this a role.
Why not use MooX::Role::Parameterized?
If I have a declaration such as
package My::Schema 5 {
# ...
}
I get an error like
Attribute (schema_version) does not pass the type constraint because: Schema version (currently '5') must be a string at reader DBIx::Class::DeploymentHandler::Dad::schema_version
If I change the declaration to be
package My::Schema {
our $VERSION = 5;
# ...
}
it all works well.
(sorry for the terseness)
I'm using ignore_ddl because I quite like the idea of my repository and history not being littered with MySQL/SQLite 1/2/3/etc directories. I've also got my schemaloader and deployment settings to a point where the generated SQL is pretty much identical to what was loaded initially, and so far I don't need the deployment files.
However, I would like DH to make use of the common/deploy/_any directory, but it won't load with ignore_ddl turned on. I'm not sure if this is by design, or a bug. I've narrowed it down to lib/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm:
sub deploy {
my $self = shift;
my $version = (shift @_ || {})->{version} || $self->schema_version;
log_info { "deploying version $version" };
my $sqlt_type = $self->storage->sqlt_type;
my $sql;
if ($self->ignore_ddl) {
$sql = $self->_sql_from_yaml({},
'_ddl_protoschema_deploy_consume_filenames', $sqlt_type
);
}
return $self->_run_sql_and_perl($self->_ddl_schema_consume_filenames(
$sqlt_type,
$version,
), $sql, [$version]);
}
$sqlt_type is 'MySQL' in my case, which is then fed to:
sub __ddl_consume_with_prefix {
my ($self, $type, $versions, $prefix) = @_;
my $base_dir = $self->script_directory;
my $main = dir( $base_dir, $type );
my $common =
dir( $base_dir, '_common', $prefix, join q(-), @{$versions} );
my $common_any =
dir( $base_dir, '_common', $prefix, '_any' );
my $dir;
if (-d $main) {
$dir = dir($main, $prefix, join q(-), @{$versions})
} else {
if ($self->ignore_ddl) {
return []
} else {
croak "$main does not exist; please write/generate some SQL"
}
}
Here it'll look for -d "./share/migrations/MySQL" and won't find that directory, where it then will quietly exit. Thus it will not search for any files in my _common directory and obviously won't run them. I haven't yet found a way to cleanly and easily override the behaviour here ... can possibly achieve it through my own deploy method though.
This is a bit of a pain, because that's what currently allows me to rebuild my database from scratch. I do have code that can load the files for me, though the exact behaviour of DH would be hard to mimic ... and I'd much rather have DH do it for me.
So sorry for being verbose, but the question is really short: is not running _any (pun intended) DDL by design or a bug, and is it something I'm just going to have to work around in some way?
Thank you,
Lianna
See also #67. We have a fairly straightforward subclass of DeploymentHandler::Dad. This code works as expected up through 0.002223, and begins breaking in 0.002224 with SQL splitting error
.
This broke further in 0.002225 because of the issue in #67, which was supposedly fixed by 0.002229. But it's also broken in 0.002229, with this error:
'DBIx::Class::DeploymentHandler::WithReasonableDefaults' requires the methods 'prepare_downgrade' and 'prepare_upgrade' to be implemented....[snip]
But my class does implement these, by virtue of doing the DeployMethod::SQL::Translator role. For reference, my subclass looks like this:
extends 'DBIx::Class::DeploymentHandler::Dad';
has initial_version => (
is => 'ro',
lazy => 1,
default => sub { $_[0]->database_version },
);
with (
'DBIx::Class::DeploymentHandler::WithApplicatorDumple' => {
interface_role => 'DBIx::Class::DeploymentHandler::HandlesDeploy',
class_name => 'DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator',
delegate_name => 'deploy_method',
attributes_to_assume => [qw(schema schema_version version_source)],
attributes_to_copy => [qw(
ignore_ddl databases script_directory sql_translator_args force_overwrite
)],
},
'DBIx::Class::DeploymentHandler::WithApplicatorDumple' => {
interface_role => 'DBIx::Class::DeploymentHandler::HandlesVersionStorage',
class_name => 'DBIx::Class::DeploymentHandler::VersionStorage::Standard',
delegate_name => 'version_storage',
attributes_to_assume => ['schema'],
attributes_to_copy => [qw(version_source version_class)],
},
# This is the role we're subbing out.
'DBIx::Class::DeploymentHandler::WithApplicatorDumple' => {
interface_role => 'DBIx::Class::DeploymentHandler::HandlesVersioning',
class_name => 'MyApp::Schema::DeploymentHandler::Monotonic::Next',
delegate_name => 'version_handler',
attributes_to_assume => [qw( initial_version schema_version to_version )],
attributes_to_copy => [qw( last_numeric next_version )],
},
);
with 'DBIx::Class::DeploymentHandler::WithReasonableDefaults';
...
You'll note that this is basically exactly the same code as was in DeploymentHandler.pm prior to the conversion to Moo. I tried changing the with
there to use MooX::Role::Parameterized::With (which, gross, but fine if it works), and I get a whole other pile of Moose garbage:
Found unknown argument 'index' in the has declaration for 'ignore_ddl' in class DBIx::Class::DeploymentHandler::WithApplicatorDumple at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/darwin-2level/Moose/Meta/Attribute.pm line 87.
Moose::Meta::Attribute::new("Moose::Meta::Attribute", "ignore_ddl", "isa", Type::Tiny=HASH(0x7fe3f403de00), "index", 0, "reader", "ignore_ddl", ...) called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/darwin-2level/Moose/Meta/Attribute.pm line 105
Moose::Meta::Attribute::interpolate_class_and_new("Moose::Meta::Attribute", "ignore_ddl", "definition_context", HASH(0x7fe3f78f1a10), "reader", "ignore_ddl", "is", "ro", ...) called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/darwin-2level/Moose/Meta/Class.pm line 724
Moose::Meta::Class::_process_new_attribute(Moose::Meta::Class=HASH(0x7fe3f7920d50), "ignore_ddl", "definition_context", HASH(0x7fe3f78f1a10), "reader", "ignore_ddl", "is", "ro", ...) called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/darwin-2level/Moose/Meta/Class.pm line 717
Moose::Meta::Class::_process_attribute(Moose::Meta::Class=HASH(0x7fe3f7920d50), "ignore_ddl", "definition_context", HASH(0x7fe3f78f1a10), "reader", "ignore_ddl", "is", "ro", ...) called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/darwin-2level/Moose/Meta/Class.pm line 581
Moose::Meta::Class::add_attribute(Moose::Meta::Class=HASH(0x7fe3f7920d50), "ignore_ddl", "definition_context", HASH(0x7fe3f78f1a10), "reader", "ignore_ddl", "is", "ro", ...) called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/darwin-2level/Moose.pm line 74
Moose::has(Moose::Meta::Class=HASH(0x7fe3f7920d50), "ignore_ddl", "reader", "ignore_ddl", "is", "ro", "default", undef, ...) called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/darwin-2level/Moose/Exporter.pm line 419
Moose::has(MooX::Role::Parameterized::Proxy=HASH(0x7fe3f6eac940), "ignore_ddl", "reader", "ignore_ddl", "is", "ro", "default", undef, ...) called at /Users/michael/code/hub/DBIx-Class-DeploymentHandler/DBIx-Class-DeploymentHandler-0.002229/lib/DBIx/Class/DeploymentHandler/WithApplicatorDumple.pm line 31
DBIx::Class::DeploymentHandler::WithApplicatorDumple::__ANON__(HASH(0x7fe3f7bc7630), MooX::Role::Parameterized::Proxy=HASH(0x7fe3f6eac940)) called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/MooX/Role/Parameterized.pm line 48
MooX::Role::Parameterized::apply("DBIx::Class::DeploymentHandler::WithApplicatorDumple", HASH(0x7fe3f7bc7630), "target", "Topicbox::Schema::DeploymentHandler") called at /Users/michael/.plenv/versions/5.26.1/lib/perl5/site_perl/5.26.1/MooX/Role/Parameterized/With.pm line 21
MooX::Role::Parameterized::With::import("MooX::Role::Parameterized::With", "DBIx::Class::DeploymentHandler::WithApplicatorDumple", HASH(0x7fe3f7bc7630), "DBIx::Class::DeploymentHandler::WithApplicatorDumple", HASH(0x7fe3f7bc63e0), "DBIx::Class::DeploymentHandler::WithApplicatorDumple", HASH(0x7fe3f7bc8350))
At which point I sort of gave up and said "this is stupid, I'm filing an issue." I suspect this breakage ultimately comes from the switch from MooseX::Role::Parameterized to MooX::Role::Parameterized (which is marked as experimental), but have not fully confirmed that yet.
I have comaint on this module, and I have mostly not been paying attention because I've been busy with other things, but I had occasion to happen across it tonight when a coworker said "hey why is this broken?" And I have nothing to say other than "beats me!"
It's really frustrating to have this module, which is pretty heavily depended on, break backcompat, especially with no mention of it in the changes file. This breakage seems largely due to the switch from Moose to Moo, which was done for seemingly no reason. (And not to mention, we've dropped support for perl 5.8 while doing so.) Maybe there was a good reason, and I don't happen to remember seeing it come across my email; if so, I apologize for being so harsh.
In the meantime, I think I'm going to have to pin my deps at 0.002223.
described here: jjn1056/DBIx-Class-Migration#108
DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator writes files without binmode
, which means they contain platform-specific newlines. \r\n
in the case of windows. This means if MySQL on linux tries to process these, all it sees is broken .sql
files.
#46 has a fix but no tests yet.
In lib/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm the _split_sql_chunk relies on lines ending with ; and then \n to split on. If your lines aren't formatted that way they get left as a big blob of text.
This isn't too big a deal when dealing with Postgres, chucking through multiple statements is fine. With SQLite however that means that the first statement is the only thing executes.
This bit me because I used a pg_dump to export insert statements, then removed the pg specific stuff to produce a deployment file for SQLite. It turns out that that exports DOS newlines (\r\n) and therefore the statements weren't split, causing the SQLite to silently ignore most of the statements.
SQL::Translator::Diff::schema_diff(): producer_args is deprecated -- it does not go straight to the producer, it goes to the internal sqlt object. Please use sqlt_args, which reflects how it's used at DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 533
https://github.com/dbsrgits/sql-translator/blob/master/Changes#L4
Just a small renaming.
From email I wrote:
There are two ways to do it. The first, arguably more pure way, is
with the _preprocess_schema directory. See docs about that in
https://metacpan.org/pod/DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator#Directory-Specification, https://metacpan.org/pod/SQL::Translator::Schema#get_table, and https://metacpan.org/pod/SQL::Translator::Schema::Table#add_index
Basically you could do something like
_preprocess_schema/0-1/001-add-foo-index.pl:
sub {
my ($from, $to) = @_;
$to->get_table('Foo')->add_index(name => 'idx_name', fields => ['name']);
}
If you support multiple DB types this is probably the most clean way
to do it.
Honestly though, I just write the DDL for the index by hand and put it
in SQLServer/upgrade/1-2/001-add-foo-index.sql
Remove this line: https://metacpan.org/source/FREW/DBIx-Class-DeploymentHandler-0.002204/lib/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm#L709 and instead change https://metacpan.org/source/FREW/DBIx-Class-DeploymentHandler-0.002204/lib/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm#L716 to
my $yml = $sqlt->translate($self->schema);
This will solve the "weird warning" problem without requiring a new DBIC version, and will make you forward compatible with the next DBIC which will warn on use of 'package'.
The technical explanation (in case you haven't seen it): Perl5/DBIx-Class@b3f891879
The first thing _run_sql_array
in DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator
does during the deploy stage:
$sql = [ _split_sql_chunk( @$sql ) ];
And _split_sql_chunk
executes:
my @sql = map { split /;\n/, $_ } @_;
This naive splitter which strips all semicolons doesn't work well with slightly more complex SQL, e.g. a MySQL trigger example:
sub sqlt_deploy_hook {
my ( $self, $sqlt_table ) = @_;
$sqlt_table->schema->add_trigger({
name => 'foo',
perform_action_when => 'BEFORE',
database_events => [qw(INSERT)],
fields => [qw(name)],
on_table => $sqlt_table->name,
schema => $sqlt_table->schema,
scope => 'row',
action => q#
IF ( 1 = 1 ) THEN
SET NEW.name = 'foo';
END IF;
#,
});
};
While deploying, the SQL is split just before the END IF, the semicolon is removed from the SQL (that first part is then fed to $dbh->do
) and MySQL correctly returns a syntax error on the statement.
I don't see an obvious way yet to solve it properly with a patch for DH; in my subclassed ::DeployMethod::SQL::Translator
class I've overridden _run_sql_array
to skip calling _split_sql_chunk
entirely (which isn't a method and thus not overridable), and this works around the problem for me. I haven't seen any adverse effects yet from leaving out the SQL splitting, though no doubt it'll have some effect somewhere.
In Debian we are currently applying the following patch to DBIx-Class-DeploymentHandler.
We thought you might be interested in it too.
The patch is located at http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libdbix-class-deploymenthandler-perl.git;a=blob;f=debian/patches/pod-whatis.patch;hb=HEAD
Feel free to improve the entries :)
add missing whatis entries to various PODs
Origin: vendor
Author: gregor herrmann <[email protected]>
Last-Update: 2014-05-11
Thanks in advance,
gregor herrmann, Debian Perl Group
This is where the bug happens:
perl5/lib/perl5/DBIx/Class/DeploymentHandler/Dad.pm
30: ref($version) ? $version->numify : $version;
Instead this should be done in Perl v5.36.0 to keep compatibility.
30: ref($version) ? int($version->numify) : $version;
I am using perl v5.36.0.
An upgrade changed a column from integer to text.
The corresponding downgrade SQL was something like:
ALTER TABLE mytable ALTER COLUMN mycol TYPE integer;
In Postgres, tt should be:
ALTER TABLE mytable ALTER COLUMN mycol TYPE integer USING (my_col::integer);
Is this an SQL::Translator issue or a DBICDH issue?
I'm getting an error while trying to move this SQL to a DBIC::Migration
https://github.com/mysociety/fixmystreet/blob/master/db/schema.sql#L21
DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator::deploy(): DBI Exception: DBD::Pg::db do failed: ERROR: unterminated quoted string at or near "' declare today date"
LINE 1: ...ction ms_current_timestamp() returns timestamp as ' declare ...
^ at /var/www/fixmystreet.127.0.0.1.xip.io/fixmystreet/local/lib/perl5/x86_64-linux-gnu-thread-multi/Moose/Meta/Method/Delegation.pm line 110
As far as I can tell, the problem seems to be _split_sql_chunk
in DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm
which chunks by ";" (which are present within the body of the function.)
Presumably this should instead be using something like Text::Balanced?
When working on my triggers migration I found I could not just add the sql to an sql file as somewhere along the line the statements were getting split in the wrong places.
The following is the sql I was attempting to issue:
-- Add triggers to maintain sync between list_material_ratings table and list_materials table:;
CREATE FUNCTION add_rating() RETURNS trigger AS $add_rating$
BEGIN
IF NEW."type" = 'like' THEN
UPDATE "list_materials" SET "likes" = (SELECT COUNT(*) FROM "list_material_ratings" WHERE "list" = NEW."list" AND "material" = NEW."material" AND "type" = 'like') WHERE "list" = NEW."list" AND "material" = NEW."material";
END IF;
IF NEW."type" = 'dislike' THEN
UPDATE "list_materials" SET "dislikes" = (SELECT COUNT(*) FROM "list_material_ratings" WHERE "list" = NEW."list" AND "material" = NEW."material" AND "type" = 'dislike') WHERE "list" = NEW."list" AND "material" = NEW."material";
END IF;
RETURN NULL;
END;
$add_rating$ LANGUAGE plpgsql;
;
CREATE FUNCTION remove_rating() RETURNS trigger AS $remove_rating$
BEGIN
IF OLD."type" = 'like' THEN
UPDATE "list_materials" SET "likes" = (SELECT COUNT(*) FROM "list_material_ratings" WHERE "list" = OLD."list" AND "material" = OLD."material" AND type = 'like') WHERE "list" = OLD."list" AND "material" = OLD."material";
END IF;
IF OLD."type" = 'dislike' THEN
UPDATE "list_materials" SET "dislikes" = (SELECT COUNT(*) FROM "list_material_ratings" WHERE "list" = OLD."list" AND "material" = OLD."material" AND type = 'dislike') WHERE "list" = OLD."list" AND "material" = OLD."material";
END IF;
RETURN NULL;
END;
$remove_rating$ LANGUAGE plpgsql;
;
CREATE TRIGGER list_materials_like AFTER INSERT ON list_material_ratings
FOR EACH ROW EXECUTE PROCEDURE add_rating();
;
CREATE TRIGGER list_materials_like AFTER DELETE ON list_material_ratings
FOR EACH ROW EXECUTE PROCEDURE remove_rating();
Which resulted in:
failed to run SQL in /home/rebus/rebus-list/share/migrations/PostgreSQL/upgrade/17-18/002-triggers.sql: DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator::try {...} (): DBI Exception: DBD::Pg::db do failed: ERROR: unterminated dollar-quoted string at or near "$add_ratin
g$ BEGIN IF NEW."type" = 'like' THEN UPDATE "list_materials" SET "likes" = (SELECT COUNT(*) FROM "list_material_ratings" WHERE "list" = NEW."list" AND "material" = NEW."material" AND "type" = 'like') WHERE "list" = NEW."list" AND "material" = NEW."material" END IF"
LINE 1: CREATE FUNCTION add_rating() RETURNS trigger AS $add_rating$...
^ at /home/rebus/rebus-list/local/lib/perl5/x86_64-linux/Moose/Meta/Method/Delegation.pm line 110
(running line 'CREATE FUNCTION add_rating() RETURNS trigger AS $add_rating$ BEGIN IF NEW."type" = 'like' THEN UPDATE "list_materials" SET "likes" = (SELECT COUNT(*) FROM "list_material_ratings" WHERE "list" = NEW."list" AND "material" = NEW."material" AND "type" = 'like') WHERE "li
st" = NEW."list" AND "material" = NEW."material" END IF') at /home/rebus/rebus-list/local/lib/perl5/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 243.
DBIx::Class::Storage::TxnScopeGuard::DESTROY(): A DBIx::Class::Storage::TxnScopeGuard went out of scope without explicit commit or error. Rolling back. at /home/rebus/rebus-list/local/lib/perl5/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 281
DBIx::Class::Storage::TxnScopeGuard::DESTROY(): A DBIx::Class::Storage::TxnScopeGuard went out of scope without explicit commit or error. Rolling back. at /home/rebus/rebus-list/local/lib/perl5/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 281
The closing
this commit introduces the bug
when for upgrade script DROP TABLE is generated
Now my upgrade scripts look like:
--- a/share/migrations/PostgreSQL/upgrade/20-21/001-auto.sql
+++ b/share/migrations/PostgreSQL/upgrade/20-21/001-auto.sql
@@ -4,6 +4,7 @@
BEGIN;
;
+DROP TABLE "ips" CASCADE;
CREATE TABLE "ips" (
"id" serial NOT NULL,
"subnet_id" integer NOT NULL,
@@ -16,6 +17,7 @@ CREATE TABLE "ips" (
CREATE INDEX "ips_idx_subnet_id" on "ips" ("subnet_id");
;
+DROP TABLE "localities" CASCADE;
CREATE TABLE "localities" (
"id" serial NOT NULL,
"name" character varying(64) NOT NULL,
In Debian we are currently applying the following patch to
DBIx-Class-DeploymentHandler.
We thought you might be interested in it too.
Description: fix a spelling mistake
Origin: vendor
Author: gregor herrmann <[email protected]>
Last-Update: 2015-10-17
The patch is tracked in our Git repository at
https://anonscm.debian.org/cgit/pkg-perl/packages/libdbix-class-deploymenthandler-perl.git/plain/debian/patches/spelling.patch
Thanks for considering,
gregor herrmann,
Debian Perl Group
prepare_install
runs prepare_deploy
and then prepare_version_storage_install
, which implies the concept of deployment does not include the concept of version storage install.
It follows that deploy
should skip the __VERSION
DDL, or the __VERSION
DDL should not be in the same directory as the 001
st deploy - and then install
itself should first discover the __VERSION
DDL and then run the first deployment.
The current setup makes it impossible to have multiple schemata in the same database. I've created a HandlesVersionStorage implementation that stores the class name of the schema being installed as well as the version, but I can't use it, because every schema contains a __VERSION
DDL and so I can't currently install the second one.
The full API of this is pending, but the gist is that the DeployMethod currently is in charge of where it gets the migrations, which is Not Great. Factoring out a MigrationStorage will allow us to easily get migrations from somewhere else, for example version control.
One of the interesting things that this can do for us is that the two different options right now for MigrationStorage can become subclasses or reimplementations of the MS instead of the DM. So for example, the first basic option is ignore_ddl
, which really means any DDL that SQLT would have generated, ignore and regenerate. Well what would happen now is that ignore_ddl uses a special MS class that takes a coderef that knows how to "get" deployments from SQLT and skip the ones that would have been generated.
The next option is the ::Deprecated DM, which is more or less a straight subclass as it currently is, with the benefit that the actual DM will now stay exactly the same.
Another goal of this is that the MigrationStorage should be "queryable" in that the user can ask it what Migrations it has stored etc.
_load_sandbox dynamically creates a package to run a perl script, naming it after the full path of the script. Paths can easily exceed perl's limitation on identifier names, resulting in the Identifier too long
error.
Hi
Trying to setup my Catalyst app to use this to make updates to the DB simpler when I put them live. I have the following (part script taken from [the Catalyst intro in the manual].(https://metacpan.org/pod/distribution/DBIx-Class-DeploymentHandler/lib/DBIx/Class/DeploymentHandler/Manual/CatalystIntro.pod)
my $dh = DH->new({
schema => $schema,
script_directory => "$FindBin::Bin/../dbicdh",
databases => 'MySQL',
});
sub install {
$dh->prepare_install;
$dh->install;
}
When calling this, I get the following error:
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Data too long for column 'ddl' at row 1 [for Statement "INSERT INTO dbix_class_deploymenthandler_versions ( ddl, version) VALUES ( ?, ? )" with ParamValues: 0='SET foreign_key_checks=0
CREATE TABLE `dbix_class_deploymenthandler_versions` ( `id` integer NOT NULL auto_increment, `version` varchar(50) NOT NULL, `ddl` text NULL, `upgrade_sql` text NULL, PRIMARY KEY (`id`), UNIQUE `dbix_class_deploymenthandler_versions_version` (`version`) )
SET foreign_key_checks=1SET foreign_key_checks=0
I believe this is because of limitations on the TEXT datatype in MySQL / MariaDB. The 001-auto-__VERSION.sql file generated from $db->prepare_install
gives this:
CREATE TABLE `dbix_class_deploymenthandler_versions` (
`id` integer NOT NULL auto_increment,
`version` varchar(50) NOT NULL,
`ddl` text NULL,
`upgrade_sql` text NULL,
PRIMARY KEY (`id`),
UNIQUE `dbix_class_deploymenthandler_versions_version` (`version`)
);
Presumably from the table definition in DBIx::Class::DeploymentHandler::VersionStorage::Standard::VersionResult, which doesn't look like it can be overridden:
my $table = 'dbix_class_deploymenthandler_versions';
__PACKAGE__->table($table);
__PACKAGE__->add_columns (
id => {
data_type => 'int',
is_auto_increment => 1,
},
version => {
data_type => 'varchar',
# size needs to be at least
# 40 to support SHA1 versions
size => '50'
},
ddl => {
data_type => 'text',
is_nullable => 1,
},
upgrade_sql => {
data_type => 'text',
is_nullable => 1,
},
);
However, my project is fairly large and the 001-auto.sql file gives the following stats:
I think I need to somehow change the definition so that ddl (and potentially upgrade_sql) are LONGTEXT fields, but I'm unsure how to do this - especially given that LONGTEXT is specific to MySQL and eventually I want to be able to install this on other database types as well - is this possible please?
Many thanks in advance!
This module doesn't work to bring an old DB up to speed.
[j@eagle bin]$ perl upgrade_db.pl
Connecting to 'dbi:SQLite:dbname=/x/eng/testbedN,rtp/performance_hpa/perfqa_10g/DTB2/hosts_info3.sqlite'
INSTALL UNVERSIONED
[j@eagle bin]$
[j@eagle bin]$ perl upgrade_db.pl
Connecting to 'dbi:SQLite:dbname=/x/eng/testbedN,rtp/performance_hpa/perfqa_10g/DTB2/hosts_info3.sqlite'
UPGRADE
No previous schema file found (sql/_source/deploy/1/001-auto.yml) at /u/perfqa/perl5/perlbrew/perls/perl-5.14.0-threaded/lib/site_perl/5.14.0/x86_64-linux-thread-multi/Moose/Meta/Method/Delegation.pm line 110.
failed to run SQL in sql/SQLite/upgrade/1-2/001-auto.sql: DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator::try {...} (): DBI Exception: DBD::SQLite::db do failed: table host already exists at /u/perfqa/perl5/perlbrew/perls/perl-5.14.0-threaded/lib/site_perl/5.14.0/x86_64-linux-thread-multi/Moose/Meta/Method/Delegation.pm line 110
(running line 'CREATE TABLE host ( id INTEGER PRIMARY KEY NOT NULL, hostname text NOT NULL, os_build text, os_type text, named_reservation_info_id int, type text NOT NULL, cluster_mode text, cluster_label text, FOREIGN KEY (named_reservation_info_id) REFERENCES named_reservation_info(id) ON DELETE CASCADE ON UPDATE CASCADE )') at /u/perfqa/perl5/perlbrew/perls/perl-5.14.0-threaded/lib/site_perl/5.14.0/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 243.
DBIx::Class::Storage::TxnScopeGuard::DESTROY(): A DBIx::Class::Storage::TxnScopeGuard went out of scope without explicit commit or error. Rolling back. at /u/perfqa/perl5/perlbrew/perls/perl-5.14.0-threaded/lib/site_perl/5.14.0/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 281
DBIx::Class::Storage::TxnScopeGuard::DESTROY(): A DBIx::Class::Storage::TxnScopeGuard went out of scope without explicit commit or error. Rolling back. at /u/perfqa/perl5/perlbrew/perls/perl-5.14.0-threaded/lib/site_perl/5.14.0/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 281
[j@eagle bin]$
Using the version_storage trick here and the upgrade portion here
I came across this one whilst attempting to use db schema's (as in what the db vendor means) with deployment handler.
In the DBIC Connection call I have an 'on_connect_do => [ SET search_path TO my_schema ] setup for my postgres connection. This works well for the txn_do calls as we use the dbic handle so our search path is always set.. however when we move onto the perl scripts the SET is not run adn therefore the dbic schema that is 'dumped' is not relating the the right database schema (In schema::loader there is a different arguament for this 'db_schema => my_schema').
I'd like to be able to set this for all my scripts rather than specify it per perl script.. much like 'sql_translator_args' can be.
Instead of having the silly deprecated storage and deploy method, we should instead have a script that migrates an old storage or set of scripts to the new format, so people can just enjoy the new setup.
Method _split_sql_chunk
breaks SQL at what it thinks is a statement delimiter,
and removes what it thinks is a transaction start.
Consider this MySQL code:
DELIMITER $$
CREATE PROCEDURE `hello_world`
BEGIN
SELECT 'Hello World!';
END$$
DELIMITER ;
Changing the delimiter is required for this to work but DH has no idea about that.
And BEGIN
is not necessarily the start of a transaction.
When using a schema in PostgreSQL, schema_from_schema_loader
doesn't find any sources. This is run on Perl 5.14.2 (with perlbrew) and the following versions of the involved Perl modules:
Module | Installed | Latest on CPAN |
---|---|---|
DBD::Pg | 3.4.1 | 3.4.1 |
DBI | 1.631 | 1.631 |
DBIx::Class | 0.08270 | 0.08270 |
DBIx::Class::DeploymentHandler | 0.002211 | 0.002211 |
DBIx::Class::Schema::Loader | 0.07042 | 0.07042 |
SQL::Translator | 0.11020 | 0.11020 |
[DBICDH] [info] installing version 1
[DBICDH] [info] deploying version 1
[DBICDH] [debug] Running SQL from sql/PostgreSQL/deploy/1/001-auto-__VERSION.sql
[DBICDH] [debug] Running SQL from sql/PostgreSQL/deploy/1/001-auto.sql
[DBICDH] [debug] Running Perl from sql/_common/deploy/1/100-add-data.pl
[DBICDH] [debug] schema_from_schema_loader generated the following sources: [
"DbixClassDeploymenthandlerVersions",
"Artist"
]
[DBICDH] [debug] Adding database version 1
[DBICDH] [debug] Running SQL from sql/PostgreSQL/deploy/1/001-auto-__VERSION.sql
[DBICDH] [debug] Running SQL from sql/PostgreSQL/deploy/1/001-auto.sql
[DBICDH] [debug] Running Perl from sql/_common/deploy/1/100-add-data.pl
[DBICDH] [debug] schema_from_schema_loader generated the following sources: []
DBIx::Class::Schema::source(): Can't find source for Artist at sql/_common/deploy/1/100-add-data.pl line 17
You are seeing this error because the DBIx::Class::ResultSource in your
migration script called "Artist" is not part of the schema that ::Schema::Loader
has inferred from your existing database.
To help you debug this issue, here's a list of the actual sources that the
schema available to your migration knows about:
Frankly I don't know if the problem is within DBIC::S::L
or DBIC::DH
, but since I'm trying to use DH
I'll report the problem here. It look like there is an issue passing the ->connect
parameters from DH
to S::L
, but I can't imaging why it differs because the exact same parameters are used (no changes in the script).
The ResultSource looks like this (a simplified DBIC::Class
example):
package MyApp::Schema::Result::Artist;
use base qw/DBIx::Class::Core/;
__PACKAGE__->table('artist');
__PACKAGE__->add_columns(
artistid => { data_type => 'integer', is_auto_increment => 1, auto_nextval => 1 },
name => { data_type => 'varchar' },
);
__PACKAGE__->set_primary_key('artistid');
1;
With the following data insertion in sql/_common_deploy/1/100-add_data.pl
:
#!perl
use strict;
use warnings;
use DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator::ScriptHelpers
'schema_from_schema_loader';
schema_from_schema_loader({ naming => 'v4' }, sub {
my $schema = shift;
my $versions = shift;
my @artists = ('Michael Jackson', 'Eminem');
$schema->resultset('Artist')->find_or_create({
name => $_
}) for @artists;
});
The script run is is this one:
#!/usr/bin/env perl
use lib 'lib/';
use DBIx::Class::DeploymentHandler;
use MyApp::Schema;
my $s = MyApp::Schema->connect('dbi:Pg:dbname=test', 'foo', '', {});
my $dh = DBIx::Class::DeploymentHandler->new({
schema => $s,
sql_translator_args => { add_drop_table => 0 },
script_dir => './sql',
force_overwrite => 1,
});
$dh->prepare_install;
$dh->install;
The user/databases are created like this:
$ createdb test
$ createuser foo
and
GRANT ALL ON DATABASE test TO foo
Where a schema is used, this is additionally added:
CREATE SCHEMA bar;
ALTER SCHEMA bar OWNER TO foo;
ALTER ROLE foo SET search_path TO bar;
The docs for DBIx::Class::DeploymentHandler::DeployMethod::SQL::Translator say:
*initialize* Gets run before the deploy is deployed. Has the same structure as the deploy subdirectory as well...
But I cannot see where this call is made, and my perl scripts stored in either MySQL/initialize/1 or _common/initialize/1 do not get run.
If I add a call to $self->initialize({ version=> $version });
into the Dad.pm install()
method, then they do get called.
Is that call missing, or do I have to do something else to get initialise()
to be called to run any Perl install scripts?
(I can fork+pull request if that would help...)
Try deploying this:
INSERT INTO test_strings (content_text)
VALUES
(
'This text is fine',
'This gets #truncated'
)
;
I get this error:
LINE 1: ...ings (content_text) VALUES ( 'This text is fine', 'This gets...
^ at (eval 816) line 17
(running line 'INSERT INTO test_strings (content_text) VALUES ( 'This text is fine', 'This gets )') at /opt/perl-5.28.1/lib/site_perl/5.28.1/DBIx/Class/DeploymentHandler/DeployMethod/SQL/Translator.pm line 172.
with
0.002230
We build our applications out of Catalyst components, each of which may pull in a different model from a different DB package with its own DBIC schema in it.
As it stands, we can only use DeploymentHandler for one of them, because the version table doesn't have a way of specifying which Schema class the version number represents.
It would be great to be able to use DH for all of our schemata in the same database.
I use App::DH and DBIC-DeploymentHandler to create migration files, currently for SQLite and PostgreSQL. After I've made my application enforce PRAGMA foreign_keys = on
for SQLite I found that the generated SQL is invalid. For complex table changes a temporary table is created like this:
CREATE TEMPORARY TABLE mytable_temp_alter (
-- copy columns
FOREIGN KEY ( mycolumn_id ) REFERENCES othertable(id)
);
This is invalid SQL.
I could not find that detail in the SQLite docs. The only page that explicitly discusses this that I found is http://sqlite.1065341.n5.nabble.com/Foreign-keys-amp-TEMPORARY-tables-td92306.html
You can see here that this is just ignored by default and throws an error if PRAGMA foreign_keys
is on:
temp.
schema$ sqlite3 -init /dev/null
-- Loading resources from /dev/null
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> CREATE TABLE first (id);
sqlite> CREATE TEMPORARY TABLE second (first_id REFERENCES first(id) );
sqlite> INSERT INTO second VALUES(42);
sqlite>
$ sqlite3 -init /dev/null
-- Loading resources from /dev/null
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> PRAGMA foreign_keys = on;
sqlite> CREATE TABLE first (id);
sqlite> CREATE TEMPORARY TABLE second (first_id REFERENCES first(id) );
sqlite> INSERT INTO second VALUES(42);
Error: no such table: temp.first
sqlite>
$ sqlite3 -init /dev/null
-- Loading resources from /dev/null
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> CREATE TABLE first (id);
sqlite> CREATE TEMPORARY TABLE second (first_id REFERENCES main.first(id) );
Error: near ".": syntax error
For SQLite I see no other solution that to just skip the FKs for the temporary table. The new main table will have FKs again and if PRAGMA foreign_keys
is on they will be checked during insertion.
Module should have a $VERSION
:
$ perl -MDBIx::Class::DeploymentHandler\ 99
DBIx::Class::DeploymentHandler does not define $DBIx::Class::DeploymentHandler::VERSION--version check failed.
BEGIN failed--compilation aborted.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.