Comments (14)
As always happens, just after opening this issue i have managed to solve it 🤦🏻
I'm assuming this is a compatability issue between Diesel and MySql 8.3.
I solved this by:
- Installing the previous version of mysql using
brew install [email protected]
- Locate the directory containing the
libmysqlclient
files. For me this was/opt/homebrew/Cellar/mysql\@8.0/8.0.36/lib/
- Install
diesel_cli
and point it at that directory for the client lib:MYSQLCLIENT_LIB_DIR=/opt/homebrew/Cellar/mysql\@8.0/8.0.36/lib/ cargo install diesel_cli --no-default-features --features=mysql --force
from diesel.
@clouds56 Thanks for investigating. That's really helpful ❤️
I've marked this as release blocker now, as we should resolve this before issuing a new release.
Given the current state of the mysqlclient-sys crate I would expect that this is unfortunately a bit more work than "just" updating the bindings.
We should likely setup a CI there as first step. After that we likely need to check if the new bindings also work with older versions otherwise that would be problematic.
I'm generally not a big fan of only generating the bindings at build time as that requires a dependency on bindgen, which in turn requires libclang, which might be problematic on windows.
(Help by all of this is appreciated.)
from diesel.
Thanks for opening this issue and adding these details. I would say it is expected that diesel works with the latest libmysqlclient version, therefore that's likely a bug. So I reopen this issue.
Can you verify a detail: You can fix this by just using a older libmysqlclient version with the newer database? (So client 8.0, database 8.3)
from diesel.
I also faced this issue when re-installing MySQL on MacOS. I first saw the below overflow/underflow error when querying a table that had not been touched in a long time:
DeserializationError("Numeric overflow/underflow occurred")
And finally came across the "Using unsupported buffer type: 253 (parameter: 1)" error when trying to list migrations, which pointed me to this issue.
I had to run cargo clean
but your solution did the trick:
brew remove mysql
brew install [email protected]
MYSQLCLIENT_LIB_DIR=/opt/homebrew/Cellar/mysql\@8.0/8.0.36/lib/
cargo clean
cargo install diesel_cli --version 2.1.1 --no-default-features --features "mysql" --force
diesel migration list
Thank you @watsom27 you're a lifesaver - this was doing my head in!
from diesel.
Share some investigation here (note: the line number might offset by 2-3 lines since I added some debug print).
last_error_number 2036
for test in tests::queryable_by_name::struct_with_no_table
(at ./tests/queryable_by_name.rs:70:16)
// failed on this query
sql_query("SELECT 1 AS foo, 2 AS bar").get_result(conn);
downside I've tracked down to StatementIterator
, which might be corrupt as early as
// the metadata returned by this api is not accessible from rust
let metadata = stmt.metadata(); // underlying delegate to `ffi::mysql_stmt_result_metadata`
// when trying to print fields of underlying raw pointer
println!("field_count: {}", (&unsafe { *result_ptr }).field_count); // this should be `2`, but `0` printed.
// when trying to print all fields via, segment faults with `(signal: 11, SIGSEGV: invalid memory reference)`
println!("fields: {:?}", metadata.fields());
by comparing st_mysql_res
in mysqlclient-sys-0.2.5, I've add some comment to help alignment.
pub struct st_mysql_res {
// uint64_t row_count;
pub row_count: my_ulonglong,
// MYSQL_FIELD *fields;
pub fields: *mut MYSQL_FIELD,
// struct MYSQL_DATA *data;
pub data: *mut MYSQL_DATA,
// MYSQL_ROWS *data_cursor;
pub data_cursor: *mut MYSQL_ROWS,
// unsigned long *lengths; /* column lengths of current row */
pub lengths: *mut ::std::os::raw::c_ulong,
// MYSQL *handle; /* for unbuffered reads */
pub handle: *mut MYSQL,
// const struct MYSQL_METHODS *methods;
pub methods: *const st_mysql_methods,
// MYSQL_ROW row; /* If unbuffered read */
pub row: MYSQL_ROW,
// MYSQL_ROW current_row; /* buffer to current row */
pub current_row: MYSQL_ROW,
// struct MEM_ROOT *field_alloc;
pub field_alloc: MEM_ROOT, // This field is embedded in rust
// unsigned int field_count, current_field;
pub field_count: ::std::os::raw::c_uint,
pub current_field: ::std::os::raw::c_uint,
// bool eof; /* Used by mysql_fetch_row */
pub eof: my_bool,
// bool unbuffered_fetch_cancelled;
pub unbuffered_fetch_cancelled: my_bool,
// enum enum_resultset_metadata metadata;
// void *extension;
pub extension: *mut ::std::os::raw::c_void,
}
and corresponding C struct (from mysql master
, 8.3.0)
typedef struct MYSQL_RES {
uint64_t row_count;
MYSQL_FIELD *fields;
struct MYSQL_DATA *data;
MYSQL_ROWS *data_cursor;
unsigned long *lengths; /* column lengths of current row */
MYSQL *handle; /* for unbuffered reads */
const struct MYSQL_METHODS *methods;
MYSQL_ROW row; /* If unbuffered read */
MYSQL_ROW current_row; /* buffer to current row */
// ------------------------
// NOTE here in C struct, the MEM_ROOT is a pointer
// ------------------------
struct MEM_ROOT *field_alloc;
unsigned int field_count, current_field;
bool eof; /* Used by mysql_fetch_row */
/* mysql_stmt_close() had to cancel this result */
bool unbuffered_fetch_cancelled;
// ------------------------
// NOTE here in C struct, a new field not in `st_mysql_res`
// ------------------------
enum enum_resultset_metadata metadata;
void *extension;
} MYSQL_RES;
Note that the difference in alignment of MYSQL_RES
is not critical since we use ffi::mysql_num_fields
and ffi::mysql_fetch_fields
to access fields.
diesel/diesel/src/mysql/connection/stmt/metadata.rs
Lines 18 to 28 in a5c82df
But the difference (size) between st_mysql_field
and MYSQL_FIELD
would cause the error since we treat the return value as an array of MysqlFieldMetadata
, or say &[st_mysql_field]
.
pub struct st_mysql_field {
// char *name;
pub name: *mut ::std::os::raw::c_char,
// char *org_name;
pub org_name: *mut ::std::os::raw::c_char,
// char *table;
pub table: *mut ::std::os::raw::c_char,
// char *org_table;
pub org_table: *mut ::std::os::raw::c_char,
// char *db;
pub db: *mut ::std::os::raw::c_char,
// char *catalog;
pub catalog: *mut ::std::os::raw::c_char,
// /* THIS FIELD IS NOT PRESENT IN C STRUCT */
pub def: *mut ::std::os::raw::c_char,
// unsigned long length;
pub length: ::std::os::raw::c_ulong,
// unsigned long max_length;
pub max_length: ::std::os::raw::c_ulong,
// unsigned int name_length;
pub name_length: ::std::os::raw::c_uint,
// unsigned int org_name_length;
pub org_name_length: ::std::os::raw::c_uint,
// unsigned int table_length;
pub table_length: ::std::os::raw::c_uint,
// unsigned int org_table_length;
pub org_table_length: ::std::os::raw::c_uint,
// unsigned int db_length;
pub db_length: ::std::os::raw::c_uint,
// unsigned int catalog_length;
pub catalog_length: ::std::os::raw::c_uint,
// /* THIS FIELD IS NOT PRESENT IN C STRUCT */
pub def_length: ::std::os::raw::c_uint,
// unsigned int flags;
pub flags: ::std::os::raw::c_uint,
// unsigned int decimals;
pub decimals: ::std::os::raw::c_uint,
// unsigned int charsetnr;
pub charsetnr: ::std::os::raw::c_uint,
// enum enum_field_types type;
pub type_: enum_field_types,
// void *extension;
pub extension: *mut ::std::os::raw::c_void,
}
I still can't see how specifying MYSQLCLIENT_LIB_DIR
would bypass this.
from diesel.
Can you verify a detail: You can fix this by just using a older libmysqlclient version with the newer database? (So client 8.0, database 8.3)
Correct, the 8.0 libmysqlclient
seems to work with the 8.3 MySql docker image. Although I haven't done a full in depth test, but it creates the database and runs the migrations.
from diesel.
That really sounds like something changed fundamentally in the mysql client headers at the c side. In turn that means we likely need to update the generated bindings for that + adjust some code in diesel. Unfortunately I do currently not have the capacity to do all of that on my own, so I need to mark this as accepting contributions and let it sit here till someone spends the time to perform the necessary steps. (Given that this seems to happen on a minor mysql release I highly question their approach of handling breaking changes…)
from diesel.
@weiznich I would be happy to have a go at this, but having looked at the codebase I have no idea where to start.
If you could point me in the general direction of the code this change would involve, or if there is any developer documentation I will happily take a look.
Thanks
from diesel.
@watsom27 It's totally fine to ask for help where to start and what to try. I know that the codebase is large and complex, so you are not expected to solve this completely on your own.
I would likely start by running at these tests with the new libmysqlclient version.
diesel/diesel/src/mysql/connection/bind.rs
Line 736 in ad8ef47
Given the error message in the bug report I would expect that just running them will produce some similar error. That should give use more information about which type is affected. Starting from there I would likely check the header files of the new and old libmysqlclient version to check if the relevant bind type for that sql type has changed. Maybe add some println statements in there and see how the values change between versions. If we have more information about which type is affected and why that happens we can discuss ways how to solve that without breaking compatibility with the old versions.
from diesel.
Maybe we should updating bindings for mysqlclient-sys, sgrif/mysqlclient-sys#24,
or not use a fix binding, instead we could generate one on build.
from diesel.
Hi Apple fans,
Thanks for @clouds56 investigation.
mysql 8.0.1 Remove my_bool and then added MYSQL_TYPE_INVALID
, MYSQL_TYPE_BOOL
and MYSQL_TYPE_TYPED_ARRAY
fields for enum_field_types, so I just change mysql connection for macos mysqlclient 8.3.
And I also use rust-bindgen to autogen the bindings.rs
for macos, so there is no need to maintain the bindings_os.rs any more.
Thanks,
Leslie Zhai
from diesel.
@xiangzhai Thanks for looking into this. This is likely a good starting point, but I fear it won't be as easy as just adjusting the code to work with the new version (only) as we also need to support older versions. I'm not sure yet how to do that.
As for auto generating the bindings: I would consider accepting a PR to mysqlclient-sys
that adds that behind a feature flag. I do not want to have that enabled by default as it requires libclang to be present on the build system, which might not be the case on most windows systems.
from diesel.
Hi @weiznich
You are welcome!
I am considering about LIBMYSQL_VERSION and MYSQL_VERSION_ID to get the VERSION_ID
and also check whether or not it is MySQL or MariaDB via MYSQL_SERVER_VERSION in the mysqlclient-sys side. Then it is able to cover the new version and also support the older versions:
let (version_id, is_mysql, is_mariadb) = ffi::get_libmysql_version_id();
assert!(0 != version_id, "Invalid MySQL Version ID");
if is_mysql {
...
}
if is_mariadb {
...
}
I just wrapper the my_bool and mysql_ssl_mode because there is no mysql_ssl_mode for MariaDB. Then it is able to still use mysql_ssl_mode
in diesel side:
use mysqlclient_sys::mysql_ssl_mode;
...
But there is ugly FALSE to support my_bool
. so how to fix it :P
#[cfg(target_os = "linux")]
static FALSE: ffi::my_bool = 0;
#[cfg(target_os = "macos")]
static FALSE: ffi::my_bool = false;
Could you review the path to give some comments please?
Thanks,
Leslie Zhai
from diesel.
it requires libclang to be present on the build system, which might not be the case on most windows systems.
Thanks,
Leslie Zhai
from diesel.
Related Issues (20)
- Unable to group by date HOT 1
- Regression: LIKE queries on nullable fields in MySQL HOT 5
- Windows CI is failing to install postgres HOT 2
- Can not use .eq(Decimal_field + xxx) HOT 4
- Closing a sqlite connection does not clear the -wal and -shm file HOT 4
- Execute single-statement migrations without batch_execute HOT 6
- Linking error for diesel_cli in Alpine despite libraries being installed HOT 5
- Option field check in diesel_derives in incomplete
- oid caching leads to broken queries after schema changes HOT 2
- The diesel 2.1.5 version cannot close the tls connection HOT 7
- Diesel-Schema not supporting Postgres-SCHEMA HOT 1
- `skip_insertion` results in a compiler error HOT 4
- table with more than one primary_key can not use on_conflict().do_update() HOT 1
- Make empty changeset updates using `.optional()` , either ignore or bypass errors `QueryBuilderError` s.
- Rust-analyzer fails to autocomplete Match with diesel errors (Enum) HOT 4
- Migration Not Generating Correct Schema for Custom Types HOT 3
- Investigate new nightly warning about a soundness hole in the type system
- `diesel print-schema > foo.rs` fails without error message if database url not set HOT 1
- Using diesel CLI with `exclude_custom_type_definitions` has unexpected outcomes HOT 1
- [2.2.0 regression] mysqlclient-sys does not provide bundled bindings for libmysqlclient 21. Consider using the `buildtime_bindgen` feature or contribute bindings to the crate HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from diesel.