Giter Site home page Giter Site logo

liquidweb / woocommerce-custom-orders-table Goto Github PK

View Code? Open in Web Editor NEW
475.0 65.0 53.0 1.61 MB

Store WooCommerce order data in a custom table for improved performance.

License: GNU General Public License v3.0

PHP 94.90% Shell 5.10%
woocommerce woocommerce-plugin woocommerce-extension ecommerce wordpress wordpress-plugin woocommerce-orders

woocommerce-custom-orders-table's Introduction

WooCommerce Custom Orders Table

Build Status Coverage Status

This plugin improves WooCommerce performance by introducing a custom table to hold all of the most common order information in a single, properly-indexed location.

Current State of the plugin

WooCommerce core is working on a custom order tables feature. The plan for the WooCommerce custom order table. More information can be found in this testing post. Call for early testing – custom order table migrations

Background

WooCommerce 3.0 introduced the notion of CRUD (Create, Read, Update, and Delete) interfaces in a move to unify the way WooCommerce data is stored and retrieved. However, orders are still stored as custom post types within WordPress, with each piece of order information (billing address, shipping address, taxes, totals, and more) being stored in post meta.

In fact, WooCommerce will typically create over 40 separate post meta entries for every single order. If your store receives even 10 orders a day, that means 400 new rows in the table every single day!

The larger the post meta table grows, the longer queries will take to execute, potentially slowing down queries (and thus page load times) for you and your visitors.

WooCommerce Custom Orders Table uses the WooCommerce CRUD design to save order data into a single, flat table that's optimized for WooCommerce queries; one order means only one new row, with minimal performance impact.

Requirements

WooCommerce Custom Orders Table requires WooCommerce 3.5.1 or newer.

If you're looking to migrate existing order data, you'll need to have the ability to run WP-CLI commands in your WooCommerce environment.

Migrating order data

After installing and activating the plugin, you'll need to migrate orders from post meta into the newly-created orders table.

The easiest way to accomplish this is via WP-CLI, and the plugin ships with three commands to help:

Counting the orders to be migrated

If you'd like to see the number of orders that have yet to be moved into the orders table, you can quickly retrieve this value with the count command:

$ wp wc orders-table count

Migrate order data from post meta to the orders table

The migrate command will flatten the most common post meta values for WooCommerce orders into a flat database table, optimized for performance.

$ wp wc orders-table migrate

Orders are queried in batches (determined via the --batch-size option) in order to reduce the memory footprint of the command (e.g. "only retrieve $size orders at a time"). Some environments may require a lower value than the default of 100.

Please note that migrate will delete the original order post meta rows after a successful migration. If you want to preserve these, include the --save-post-meta flag!

Options

--batch-size=<size>
The number of orders to process in each batch. Default is 100 orders per batch.
Passing `--batch-size=0` will disable batching.
--save-post-meta
Preserve the original post meta after a successful migration. Default behavior is to clean up post meta.

Copying data from the orders table into post meta

If you require the post meta fields to be present (or are removing the custom orders table plugin), you may rollback the migration at any time with the backfill command.

$ wp wc orders-table backfill

This command does the opposite of migrate, looping through the orders table and saving each column into the corresponding post meta key. Be aware that this may dramatically increase the size of your post meta table!

Options

--batch-size=<size>
The number of orders to process in each batch. Default is 100 orders per batch.
Passing `--batch-size=0` will disable batching.

Contributing

If you're interested in contributing to the development of the plugin or need to report an issue, please see the contributing guidelines for the project.

woocommerce-custom-orders-table's People

Contributors

alchemyunited avatar boogah avatar bswatson avatar coenjacobs avatar crstauf avatar jb510 avatar lukecav avatar pmgarman avatar ranss avatar schlessera avatar stevegrunwell avatar szepeviktor avatar uvlabs avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

woocommerce-custom-orders-table's Issues

Duplicate Entry for 'order_key' Due To Partially MIgrated Row

Describe the bug
This issue is still occurring. This is not happening on the PK of "order_id", but the "UNIQUE" column of 'order_key'.

To Reproduce

  1. Try WP CLI migration
  2. Some orders migrate
  3. Fails with error "Error: A database error occurred while migrating order XXXXXX: Duplicate entry '' for key 'order_key'."

Expected behavior
Orders should be migrated completely, thus eliminating an empty 'order_key' field.

Versions

  • WordPress version - 4.7.9
  • WooCommerce version - 3.4.3
  • Plugin version - 1.0.0 (Beta 3)

Additional context
Adding the --debug flag showed something interesting. These are the last two lines in the terminal:
Debug: Order ID 153226 has been migrated. (2.114s)
Debug: Order ID 153224 has been migrated. (2.118s)
Error: A database error occurred while migrating order 153222: Duplicate entry '' for key 'order_key'.

However, if I look at the data for order_id 153224 in the custom orders table, only the "order_id" column is filled. All other columns are empty.

All previous rows before 153224 migrated fully with all columns having valid values.

So, since 153224 didn't migrate any data other than the order_id, the next row will fail on initial insertion because the "order_key" column must be UNIQUE. Ergo, 153222 failed.

Looking at the data in the posts and postmeta tables for both 153226(last fully migrated order) and 153224(first order missing all data other than order ID), I can see no difference. Post meta rows existing for 153226 exist for 153224. Rows that should have "meta_value"s for each do. Both rows in the posts table show similar. All this to say that both orders should be treated identically by the migration code, and I see no reason that 153224 was reported by the migration as successful even with all but one column in the custom orders table empty.

Diving deeper, this might have something to do with the $changes being empty in the update_post_meta method. That's the only block where if an order ID already exists in the custom orders table, to get the changes of the post meta for the order and update the row with that data. If $changes is empty, the update is not performed.

Hopefully this info will help you all in some way. I'll be happy to provide more, if needed.

Reports totals calculate still on postmeta values

In WooCommerce reporting, the totals are still calculated based on postmeta fields. The public function get_order_report_data( $args = array() ) method on the class WC_Admin_Report are not designed yet (I believe) to take these calculations to a separate table, instead of postmeta.

Steps to reproduce:

  1. Create a new order
  2. Set the order to completed so the order is taken into account with reporting
  3. Flush transients to make sure resulting report is not cached.
  4. See the reports totals are not updated with new order

Extra verification steps

  1. Run the CLI backfill command
  2. Flush transients again to kill cached reports
  3. See the reports totals are now updated with the new order totals

Approach

My best guess is that this needs WooCommerce core enhancements, but I have to dive in a little deeper to make sure this is the case.

customer_user_agent column not large enough for migration

The current size of the customer_user_agent column is not large enough. The longest user agent string I have in my database is 284 characters. When this value is too long, $wpdb->insert() returns false and my migration won't finish because those rows are never migrated.

I have 32 such records in my database. It appears that social media apps with in-app browsers (Facebook and Instagram) are including their own customizations to the user agent string, making these grow quite large. I've included some examples.

Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E302 [FBAN/FBIOS;FBAV/183.0.0.41.81;FBBV/119182652;FBDV/iPhone8,4;FBMD/iPhone;FBSN/iOS;FBSV/11.3.1;FBSS/2;FBCR/Sprint;FBID/phone;FBLC/en_US;FBOP/5;FBRV/0]

Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG-SM-G930A Build/R16NW; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/69.0.3497.100 Mobile Safari/537.36 [FB_IAB/Orca-Android;FBAV/185.0.0.31.96;]

Mozilla/5.0 (Linux; Android 8.1.0; Nexus 5X Build/OPM3.171019.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 Mobile Safari/537.36 Instagram 43.0.0.10.97 Android (27/8.1.0; 420dpi; 1080x1794; LGE/google; Nexus 5X; bullhead; bullhead; en_US; 105842053)

I'm not sure what size you'd want to increase to, so I didn't put together a pull request, but I will if a maintainer would like to provide guidance.

Versions

  • WordPress version 4.9.3
  • WooCommerce version 3.4.6
  • Plugin version 1.0.0 RC

48554 orders were migrated, with 571 skipped.

Warning: A database error occurred while migrating order 55305, skipping: Duplicate entry '' for key 'order_key'.
Eroare bază de date WordPress Duplicate entry '' for key 'order_key' pentru interogarea INSERT INTO `_tkwoocommerce_orders` (`order_id`, `order_key`, `customer_id`, `payment_method`, `payment_method_title`, `transaction_id`, `customer_ip_address`, `customer_user_agent`, `created_via`, `date_completed`, `date_paid`, `cart_hash`, `billing_index`, `billing_first_name`, `billing_last_name`, `billing_company`, `billing_address_1`, `billing_address_2`, `billing_city`, `billing_state`, `billing_postcode`, `billing_country`, `billing_email`, `billing_phone`, `shipping_index`, `shipping_first_name`, `shipping_last_name`, `shipping_company`, `shipping_address_1`, `shipping_address_2`, `shipping_city`, `shipping_state`, `shipping_postcode`, `shipping_country`, `discount_total`, `discount_tax`, `shipping_total`, `shipping_tax`, `cart_tax`, `total`, `version`, `currency`, `prices_include_tax`) VALUES ('24585', '', '0', '', '', '', '', '', '', NULL, NULL, '', '          ', '', '', '', '', '', '', '', '', '', '', '', '        ', '', '', '', '', '', '', '', '', '', '0', '0', '0', '0', '0', '0', '', '', 'no') efectuată de include('phar:///usr/local/bin/wp/php/boot-phar.php'), include('phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/wp-cli.php'), WP_CLI\bootstrap, WP_CLI\Bootstrap\LaunchRunner->process, WP_CLI\Runner->start, WP_CLI\Runner->_run_command_and_exit, WP_CLI\Runner->run_command, WP_CLI\Dispatcher\Subcommand->invoke, call_user_func, WP_CLI\Dispatcher\CommandFactory::WP_CLI\Dispatcher\{closure}, call_user_func, WooCommerce_Custom_Orders_Table_CLI->migrate, WooCommerce_Custom_Orders_Table_CLI->get_order, wc_get_order, WC_Order_Factory::get_order, WC_Abstract_Order->__construct, WC_Data_Store->read, Abstract_WC_Order_Data_Store_CPT->read, WC_Order_Data_Store_Custom_Table->read_order_data, WC_Order_Data_Store_Custom_Table->populate_from_meta, WC_Order_Data_Store_Custom_Table->update_post_meta
Eroare bază de date WordPress Duplicate entry '' for key 'order_key' pentru interogarea INSERT INTO `_tkwoocommerce_orders` (`order_id`, `order_key`, `customer_id`, `payment_method`, `payment_method_title`, `transaction_id`, `customer_ip_address`, `customer_user_agent`, `created_via`, `date_completed`, `date_paid`, `cart_hash`, `billing_index`, `billing_first_name`, `billing_last_name`, `billing_company`, `billing_address_1`, `billing_address_2`, `billing_city`, `billing_state`, `billing_postcode`, `billing_country`, `billing_email`, `billing_phone`, `shipping_index`, `shipping_first_name`, `shipping_last_name`, `shipping_company`, `shipping_address_1`, `shipping_address_2`, `shipping_city`, `shipping_state`, `shipping_postcode`, `shipping_country`, `discount_total`, `discount_tax`, `shipping_total`, `shipping_tax`, `cart_tax`, `total`, `version`, `currency`, `prices_include_tax`) VALUES ('24585', '', '0', '', '', '', '', '', '', NULL, NULL, '', '          ', '', '', '', '', '', '', '', '', '', '', '', '        ', '', '', '', '', '', '', '', '', '', '0', '0', '0', '0', '0', '0', '', '', 'no') efectuată de include('phar:///usr/local/bin/wp/php/boot-phar.php'), include('phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/wp-cli.php'), WP_CLI\bootstrap, WP_CLI\Bootstrap\LaunchRunner->process, WP_CLI\Runner->start, WP_CLI\Runner->_run_command_and_exit, WP_CLI\Runner->run_command, WP_CLI\Dispatcher\Subcommand->invoke, call_user_func, WP_CLI\Dispatcher\CommandFactory::WP_CLI\Dispatcher\{closure}, call_user_func, WooCommerce_Custom_Orders_Table_CLI->migrate, WC_Data_Store->__call, WC_Order_Data_Store_Custom_Table->populate_from_meta, WC_Order_Data_Store_Custom_Table->update_post_meta
Warning: A database error occurred while migrating order 24585, skipping: Duplicate entry '' for key 'order_key'.

What this mean ?

check that order_key is not empty in update_post_meta()

This is just another of those duplicate order_key issues. if order_key is empty (for example on an order that has not been fully instantiated yet, such as a new order from the admin panel), you may get an empty value back for $order->get_order_key(). If this occurs, the update function should return, rather than attempting to update or insert.

Customer notes are not being updated

Thank you so much for this fantastic plugin, this is what we after - We have a billion rows in postmeta table already and it growing fast. So we have to find a solution to solve this issue.

We have start tried your beta plugin this morning. However, we've been notice that we can't update "Customer provided note" on orders page. If you did so, the note will all gone. Can you please look into this?

Many thanks.

Regards,
Zack

Drop autoloader from distributed plugin

Considering the plugin is only a few files and the plugin bootstrapping issues have been fixed (see #22), there's no reason to ship the plugin with a Composer-generated, PHP 5.2-compatible autoloader.

Autoloaders are great (when done correctly), but when we're building a small autoloader to then bundle with the distributed plugin, it becomes a liability (especially without a build script for building releases). Instead, the necessary files should be included manually (or an autoload process should be written into the plugin, using spl_autoload().

Stall during migration

I tried this on a site on my local machine, I started with "10984 orders to be migrated" and then ran the migration, it however got stuck at 100%. I cancelled it and checked the count, it now says "159 orders to be migrated" but every time I try the migration again it is stuck on 100%.

Also all the orders are still in the wp_posts database, so I guess it does not delete the data until it finishes? Which it never does.

Using Storefront theme, only WooCommerce and this plugin active.

Reports totals calculate still on postmeta values

In WooCommerce reporting, the totals are still calculated based on postmeta fields. The public function get_order_report_data( $args = array() ) method on the class WC_Admin_Report are not designed yet (I believe) to take these calculations to a separate table, instead of postmeta.

Steps to reproduce:

  1. Create a new order
  2. Set the order to completed so the order is taken into account with reporting
  3. Flush transients to make sure resulting report is not cached.
  4. See the reports totals are not updated with new order

Extra verification steps

  1. Run the CLI backfill command
  2. Flush transients again to kill cached reports
  3. See the reports totals are now updated with the new order totals

Approach

My best guess is that this needs WooCommerce core enhancements, but I have to dive in a little deeper to make sure this is the case.

Support seller_id which is used by vendor plugins

Dokan and WC Vendors plugins will allow vendors to sell products on a WooCommerce store, they both use seller_id to set the vendor relationship on a order, similar to the way customer_id is used to set the customer relationship to an order, allow WooCommerce Custom Orders Table to support seller_id so that it might be used on stores using Dokan or WC Vendors.

Invalid billing email address

Describe the bug
Migration encountered an email address with an invalid character (>); not sure how that happened, so not sure if there's even anything to fix, but thought better to report than not.

To Reproduce
Steps to reproduce the behavior:

  1. Set _billing_email on an order with an invalid character.
  2. Initiate migration via the CLI.
  3. See error

Expected behavior
To be determined.

Versions

Screenshots
screen shot 2018-11-09 at 12 31 12 pm

Any statistics on improvements?

Great idea, and great implementation. This is something I have said WC should have done years ago.

I was wondering if you have any statistics on the magnitude of the improvement that can be gained on small sites and large sites? It will certainly cut down on the numbers of queries, but I suspect these are queries that are already well optimised by indexes and would not affect the shop front end performance (or maybe they do?).

I'm just trying to quantify the gains compared to, say, a shop with many thousands of products but relatively few orders. There are also tables which we have found (in the past at least) optimised very poorly for the types of use that WC puts them to, such as the order update notes which go into the comments table with appallingly bad indexes, or sessions in the options table which are incredibly difficult to clean out when the table grows from its designed several-hundred rows to hundreds of thousands.

Sorry - reads a bit like whataboutism that - not intended. This is a great first step and it would be great to see some evidence on what types of site it will work well on, and what types it may not have any measurable impact on.

Errors when running the migration WC-CLI command

Verified orders to be migrated using wp wc-order-table count
There are 176612 orders to be migrated.

When running wp wc-order-table migrate

Getting a number of these errors on the migration

Warning: Unable to retrieve order with ID 474228
Order Data Migration 13 % [=====> ] 0:47 / 5:47Exception caught in get_order. Invalid billing email address. Args: Array
(
[0] => 474228
)

Migration ran for 20mins, then killed the process and it did not complete.

WooCommerce 3.3.4 is active on that site.

Bump "WC tested up to" value

Ahead of release, the "WC tested up to" value in the main plugin file (along with any other applicable metadata) should be updated to reflect the latest information.

Clear post meta following the CLI migration

As pointed out in #57, the post meta isn't currently being cleared following a migration to the custom order table. Since the backfill command exists to "undo" a migration to the custom order table, this meta should be deleted once a migration is successful.

As some people may want to preserve legacy meta data, a --no-delete-post-meta (or similar) flag should be available for the migrate command, which will prevent the meta from being removed.

Infinite loop during migrate script

I am running this on a store with 120k orders, the progress bar goes all the way to 100% but then it endlessly loops. Checking the top command shows PHP and MySQL processes continue to run so it seems to get stuck. Is there a --debug flag or something similar to help diagnose this?

I am happy to provide SSH root access to help figure this out :)

Datetime columns for date_completed, date_paid

Another spin-off from #48, we should consider changing the date_completed and date_paid columns to use MySQL's DATETIME column type rather than a Unix timestamp.

From the original ticket:

Converting these to datetime rather than keeping timestamp in a varchar field; because if these fields are converted to datetime then we can use MySQL inbuilt date function to generate various kind of weekly, monthly, or yearly reports. DateTime is also human readable rather than timestamp.
Also renaming the field to completed_at_gmt, paid_at_gmt to avoid confusing or adding a comment there the datetime is in GMT timezone.

PHP Fatal error: Can't use function return value in write context

Describe the bug
When I trying to run command 'wp wc orders-table count' it throws me 'PHP Fatal error: Can't use function return value in write context in /var/www/vhosts/cmj.ro/woo.cmj.ro/wp-content/plugins/woocommerce-custom-orders-table-develop/includes/class-woocommerce-custom-orders-table-cli.php on line 120'

Versions

  • WordPress version 4.9.8
  • WooCommerce version 3.4.5
  • Plugin version ["develop @ fb5724c"]

Risky test for CLITest::test_migrate_output_when_items_were_skipped()

During CI, we're often running into the following failure:

  1. CLITest::test_migrate_output_when_items_were_skipped
    Expected to only see two orders in the custom table.
    Failed asserting that 1 matches expected 2.
    /home/travis/build/liquidweb/woocommerce-custom-orders-table/tests/test-cli.php:267

Unfortunately, we're only seeing the issue occasionally β€”Β re-running the CI job often enables the test to pass. That means this test may be behaving improperly, which needs adjusted.

Use background processing if a store has under a set number of orders

Currently the only method to migrate existing orders out is to use the WP-CLI commands.
https://github.com/liquidweb/woocommerce-custom-orders-table#migrating-order-data

WooCommerce core uses
https://github.com/A5hleyRich/wp-background-processing
woocommerce/includes/libraries/wp-background-process.php
woocommerce/includes/class-wc-background-updater.php

To run certain db migrations as a background process, maybe build in the option to enable order migrations to be run from the wp-admin UI. But would need to explain that running the process can cause processing issues on the live site and that you would want to test the order migrations on a staging site first. Also if a higher order count exists on a site, then going with the WP-CLI command methods may work better in those cases and set a lower --batch-size when running the migrate command.

Delete Argument on Migrate Command

First off, thanks for the work you've put into building this excellent plugin.

I have noticed that in the order data store, in the populate_from_meta method there is a parameter ($delete) that if set will delete the order postmeta from the postmeta table after it inserts it in the new orders table. However on the cli migrate command there isn't any way to set this parameter when populate_from_meta is called.

Is there any negative that would come from deleting the existing postmeta after it has been moved into the new table? As is, the plugin will definitely improve all the searches and queries that will now hit the new orders table. However, leaving all the data in the postmeta table will still affect any other postmeta queries (for instance the ones associated with other post types).

If there aren't any negative drawbacks, I think a pretty simple solution would be to add a argument to the cli command that would set that parameter.

Thanks!

Database table indexes

The wp_woocommerce_orders table created by the plugin does not currently include any indexes beyond a primary key, which means it's very quick to write but will become increasingly slower to query against as the table grows.

PHP Parse error: syntax error, unexpected '<<' (T_SL)

When I trying to run command 'wp wc orders-table count' it throws me 'PHP Parse error: syntax error, unexpected '<<' (T_SL) in /var/www/vhosts/cmj.ro/woo.cmj.ro/wp-content/plugins/woocommerce-custom-orders-table-develop/includes/class-woocommerce-custom-orders-table-cli.php on line 120
Parse error: syntax error, unexpected '<<' (T_SL) in /var/www/vhosts/cmj.ro/woo.cmj.ro/wp-content/plugins/woocommerce-custom-orders-table-develop/includes/class-woocommerce-custom-orders-table-cli.php on line 120'

Versions

WordPress version 4.9.8
WooCommerce version 3.4.5
Plugin version ["develop @ 60d4581"]
PHP Version 7.1.23

@stevegrunwell can you look at this ?

Order count is the same after migrating the orders over using the WP-CLI command

Current dev branch is installed and active.
WooCommerce 3.3.5
WordPress 4.9.6
wp plugin install https://github.com/liquidweb/woocommerce-custom-orders-table/archive/develop.zip --activate --force

Ran
wp wc-order-table count
There are 168934 orders to be migrated.

All 168934 orders were migrated using wp wc-order-table migrate, which showed as complete.

But when running wp wc-order-table count to verify all orders have migrated it showed
There are 168934 orders to be migrated.

So it looks like the orders did not get migrated.

Normalize plugin name

Right now, the plugin is referred to as both "WooCommerce Order Tables" and "WooCommerce Custom Order Tables". This usage should be normalized ahead of an official 1.0.0 release.

shop_order CPT still created in wp_posts table

This plugin sounded like it would be a lifesaver in solving the issue of migrating databases from staging to production without affecting shop orders. The plugin works great but an entry is still created in the wp_posts table (I'm assuming for backwards compatibility).

Is there any likelihood of an update which completely removes orders from the standard wp tables so enable a smooth migration path? Thanks in advance.

Cheers!

Optimizing WooCommerce Order Table

I have few suggestions regarding wp_woocommerce_orders table

  1. billing_address_1, shipping_address_1
    Increasing the size of these field from varchar(200) to varchar(255), because there are many people who do not use billing_address_2/shipping_address_2 and only keep the address in a single filed and hide the second one, So making it to 255 will ensure no address truncating.

  2. billing_postcode, shipping_postcode
    Decreasing the size from varchar(100) to varchar(20) because according to the wiki the maximum character a postal code could have is 8 character so this field size can be reduced.

  3. discount_total, shipping_total, total, discount_tax, shipping_tax, cart_tax
    Making all the money value storing field as decimal(10,2) instead of varchar because when the varchar field is ordered by ASC or DESC it treats the amount as a string and gives the preference to the first character and end's up ordering as mentioned in below two screenshots; So to get it correct one need to typecast the varchar to int/decimal and do the ordering which obviously affects the query execution time. But this won't happen if the fields are in decimal.

2018-03-09_0022

2018-03-09_0027

  1. billing_country, shipping_country
    Converting these fields from varchar(100) to char(2) for ISO 3166-1 Alpha-2 or char(3) for ISO 3166-1 Alpha-3
    As these are fixed value and will never change so it can be stored in char, it will have 2 advantages firstly storage and secondary faster query execution when using country in where or group by clause (source: https://dba.stackexchange.com/a/2643/125935)

  2. date_completed, date_paid
    Converting these to datetime rather than keeping timestamp in a varchar field; because if these fields are converted to datetime then we can use MySQL inbuilt date function to generate various kind of weekly, monthly, or yearly reports. DateTime is also human readable rather than timestamp.
    Also renaming the field to completed_at_gmt, paid_at_gmt to avoid confusing or adding a comment there the datetime is in GMT timezone.

  3. prices_include_tax
    Converting it from varchar(3) to tinyint(1) because integer search is much for master then string search. The field will be auto-understandable but still, comment can be added as 0: Excluding tax, 1: Excluding tax.

  4. Making all the files default value as NULL rather than keeping 0 or blank.

Here is the full table structure that I think might enhanced query performance.

--
-- Table structure for table `wp_woocommerce_orders`
--

CREATE TABLE `wp_woocommerce_orders` (
  `order_id` bigint(20) UNSIGNED NOT NULL,
  `order_key` varchar(100) NULL DEFAULT NULL,
  `customer_id` bigint(20) UNSIGNED NOT NULL,
  `billing_index` varchar(255) NOT NULL,
  `billing_first_name` varchar(100) NULL DEFAULT NULL,
  `billing_last_name` varchar(100) NULL DEFAULT NULL,
  `billing_company` varchar(100) NULL DEFAULT NULL,
  `billing_address_1` varchar(255) NULL DEFAULT NULL,
  `billing_address_2` varchar(200) NULL DEFAULT NULL,
  `billing_city` varchar(100) NULL DEFAULT NULL,
  `billing_state` varchar(100) NULL DEFAULT NULL,
  `billing_postcode` varchar(20) NULL DEFAULT NULL,
  `billing_country` char(2) NULL DEFAULT NULL,
  `billing_email` varchar(200) NULL DEFAULT NULL,
  `billing_phone` varchar(50) NULL DEFAULT NULL,
  `shipping_index` varchar(255) NOT NULL,
  `shipping_first_name` varchar(100) NULL DEFAULT NULL,
  `shipping_last_name` varchar(100) NULL DEFAULT NULL,
  `shipping_company` varchar(100) NULL DEFAULT NULL,
  `shipping_address_1` varchar(255) NULL DEFAULT NULL,
  `shipping_address_2` varchar(200) NULL DEFAULT NULL,
  `shipping_city` varchar(100) NULL DEFAULT NULL,
  `shipping_state` varchar(100) NULL DEFAULT NULL,
  `shipping_postcode` varchar(20) NULL DEFAULT NULL,
  `shipping_country` char(2) NULL DEFAULT NULL,
  `payment_method` varchar(100) NULL DEFAULT NULL,
  `payment_method_title` varchar(100) NULL DEFAULT NULL,
  `discount_total` decimal(10,2) DEFAULT NULL,
  `discount_tax` decimal(10,2) DEFAULT NULL,
  `shipping_total` decimal(10,2) DEFAULT NULL,
  `shipping_tax` decimal(10,2) DEFAULT NULL,
  `cart_tax` decimal(10,2) DEFAULT NULL,
  `total` decimal(10,2) DEFAULT NULL,
  `version` varchar(16) NOT NULL,
  `currency` char(3) NULL DEFAULT NULL,
  `prices_include_tax` tinyint(1) UNSIGNED NOT NULL DEFAULT '0' COMMENT '0: Excluding tax, 1: Including tax',
  `transaction_id` varchar(200) NOT NULL,
  `customer_ip_address` varchar(40) NOT NULL,
  `customer_user_agent` varchar(200) NOT NULL,
  `created_via` varchar(200) NOT NULL,
  `completed_at` datetime DEFAULT NULL COMMENT 'in GMT',
  `paid_at` datetime DEFAULT NULL COMMENT 'in GMT',
  `cart_hash` varchar(32) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

--
-- Indexes for table `wp_woocommerce_orders`
--
ALTER TABLE `wp_woocommerce_orders`
  ADD PRIMARY KEY (`order_id`),
  ADD UNIQUE KEY `order_key` (`order_key`),
  ADD KEY `customer_id` (`customer_id`),
  ADD KEY `order_total` (`total`) USING BTREE;

Numeric pricing columns

In #48, @raunak-gupta pointed out the following:

Making all the money value storing field as decimal(10,2) instead of varchar because when the varchar field is ordered by ASC or DESC it treats the amount as a string and gives the preference to the first character and end's up ordering as mentioned in below two screenshots; So to get it correct one need to typecast the varchar to int/decimal and do the ordering which obviously affects the query execution time. But this won't happen if the fields are in decimal.

While there are larger implications to changing the way values are stored (and not something I want to delay version 1.0.0 any further), this is still definitely something that should be considered for a future version. We'll also want to consider the implications when storing prices as decimal/float values rather than integers (e.g. 100 == $1.00).

Ensure WooCommerce reports can access data

As noted in the beta release notes, the plugin is currently designed to store + retrieve orders from the custom database table, but that doesn't guarantee that the data will be used when WooCommerce is generating reports.

We need to look at how WooCommerce is retrieving information during report generation and make any necessary adjustments to ensure order data is being evaluated.

Fatal error in migration process

During the migration process I am getting a Fatal error;

Fatal error: Call to a member function get_data_store() on boolean in /Users/zscott/httpdocs/zanerobe.local/wp-content/plugins/woocommerce-custom-orders-table-1.0.0-beta.3/includes/class-woocommerce-custom-orders-table-cli.php on line 96

I have tracked the problem down to the call to wc_get_order() in WooCommerce_Custom_Orders_Table_CLI::migrate().

If an there is an error with an order, wc_get_order() will return false, an unchecked condition. In my particular case it is failing because the order has an invalid email address (its an old order so I'm not suprised).

WooCommerce dashboard status widget uses wc_get_order_types so the default status widget will not get order count correctly

WordPress 4.9 running

Installed plugin via
--- Directory - /var/www/wordpress/wp-content/plugins/woocommerce-order-tables/
/composer install

--- Then activated via Wordpress Plugin(s).
Made an order; and no databases were created; so if it was meant to at the start, they did not.

Debian 8
PHP 5.6
MySQL 5.5

Refactor WP-CLI command

There are a number of PHP_CodeSniffer issues and with the WP-CLI class; with some of the changes coming in #26, this file needs some attention to ensure we're keeping things DRY and working as expected.

  • Add tests around existing behavior
  • Leverage existing functionality within the plugin, rather than redefining everything
  • Ensure code passes PHP_CodeSniffer
  • Update documentation

Convert prices_include_tax to Boolean

Another recommendation out of #48, simplify the prices_include_tax column to be a simple tinyint(1) column, where 1 indicates that prices include tax and 0 indicates that they do not. This will enable MySQL to search using integers, which are faster than strings.

Check that "order_key" is not empty on migration

Describe the bug
When running the migration, any order in the database with an empty "order_key" meta field will produce the following error:

"Error: A database error occurred while migrating order XXXXXX: Duplicate entry '' for key 'order_key'"

To Reproduce
Steps to reproduce the behavior:

  1. Have an order with an '_order_key' in the postmeta table with an empty value.
  2. Attempt WP CLI migration
  3. See error

Expected behavior
Since the new orders table has a "UNIQUE" index on the order_key column, a check needs to be performed to ensure the '_order_key' meta is valid.

This is similar to #64, but this should check the value in the meta value, not if the order is already migrated.

Versions

  • WordPress version - 4.7.9
  • WooCommerce version - 3.4.3
  • Plugin version - 1.0.0-beta3

Duplicate entry during WP-CLI order migration command

Ran into this issue during a orders migration wp wc-order-table migrate --allow-root

There are 19228 orders to be migrated.
Order Data Migration 49 % [===================> ] 1:03 / 2:06Error: A database error occurred while migrating order 30525: Duplicate entry '' for key 'order_key'.
root@default:/home/s9/html# There are 19228 orders to be migrated.
There: command not found
root@default:/home/s9/html# Order Data Migration 49 % [===================> ] 1:03 / 2:06Error: A database error occurred while migrating order 30525: Duplicate entry '' for key 'order_key'.

Data transformation

Continuing the work from #48, we should be looking into storing data in the most sensible format for the data type rather than strictly storing data as strings (which is what WooCommerce has historically had to do, thanks to the structure of the WordPress post meta table.

For example, the prices_include_tax database column is derived from the _prices_include_tax post meta value; historically, WooCommerce has stored this value as a string (e.g. "yes" or "no"), then casts it to a proper Boolean value via the wc_string_to_bool() helper function.

Rather than simply storing the value of that column as a string (varchar(3)), a more performant way to handle it would be a one-byte, tiny integer (tinyint(1)) column, which consists of either a 1 (true) or 0 (false).

WooCommerce itself doesn't need to know how it's stored in the database table (the whole point of its CRUD API), as long as it's able to retrieve information from the data store in a format that it expects.

Possible approach

At its core, the WooCommerce CRUD API is designed for this very use-case: it shouldn't matter how the data is stored under the hood, as long as it comes out in a way that WooCommerce can handle.

As of #52, we have (at least) two data stores we need to worry about, both pulling from the same table: Orders and Order Refunds. Each has their own custom data store, and while they share a common relation in Abstract_WC_Order_Data_Store_CPT, it's not a direct ancestor.

This would be an ideal case for PHP traits, enabling us to define the appropriate transformations once and apply them to both of our data stores. Unfortunately, traits are only available for PHP 5.4 and higher, while WooCommerce officially supports 5.2 and above.

We can work around this by creating something of a pseudo-trait, comprised of a class full of static methods; within our data stores, we'll outsource the logic to these methods. PHP interfaces would be helpful to define a contract that each data store must satisfy, but it's still a rather ugly hack.

Timing

I'd love to get working on this sooner rather than later, but there are a few things I'd like to see in WooCommerce core, first (outlined below). As a result, I don't think all of these enhancements are feasible for a 1.0.0 release, though the plugin does have a method for pushing schema updates down the road (hooray for planning!).

Better test coverage

I've been working on submitting PRs to WooCommerce core, focusing on areas that involve orders and order data but lacked [sufficient] test coverage.

Since the WooCommerce Custom Orders Table plugin test suite includes the WooCommerce core test suite, a test added to WooCommerce helps not only core but also the custom tables plugin; the more test coverage we can get in WooCommerce, the more confidently we can start manipulating data.

Better usage of the CRUD API

A fair portion of the plugin as it exists today is focused around filtering WooCommerce post meta queries for orders (another strong case for extensive tests) and overriding methods that are explicitly using get_post_meta() instead of the CRUD API.

As WooCommerce continues to evolve, more and more functionality should (and is) moving towards leveraging the CRUD APIs. Until such time that all database interactions are handled through the data store, however, there will still be a need to [very carefully] filter database queries.

Error when started migrating orders

I got this error

wp wc-order-table migrate --batch=3000 --page=1

PHP Warning:  array_key_exists() expects parameter 2 to be array, null given in /home/domain/staging/wp-content/plugins/imagemagick-engine/imagemagick-engine.php on line 211
Warning: array_key_exists() expects parameter 2 to be array, null given in /home/domain/staging/wp-content/plugins/imagemagick-engine/imagemagick-engine.php on line 211
WordPress database error Table 'domain.wp_woocommerce_orders' doesn't exist for query 
            SELECT COUNT(1)
            FROM wp_posts p
            LEFT JOIN wp_woocommerce_orders o ON p.ID = o.order_id
            WHERE p.post_type IN ('shop_order,shop_order_refund')
            AND o.order_id IS NULL
            ORDER BY p.post_date DESC
         made by include('phar:///usr/local/bin/wp/php/boot-phar.php'), include('phar:///usr/local/bin/wp/php/wp-cli.php'), WP_CLI\bootstrap, WP_CLI\Bootstrap\LaunchRunner->process, WP_CLI\Runner->start, WP_CLI\Runner->_run_command_and_exit, WP_CLI\Runner->run_command, WP_CLI\Dispatcher\Subcommand->invoke, call_user_func, WP_CLI\Dispatcher\CommandFactory::WP_CLI\Dispatcher\{closure}, call_user_func, WC_Custom_Order_Table_CLI->migrate, WC_Custom_Order_Table_CLI->count
0 orders to be migrated.
Order Data Migration  100% [=======================================================================================================================================================================] 0:00 / 0:00
0 orders processed in 0 batches.

Make the plugin mu-ready

Thanks for the great plugin
I did not use wp-admin for plugin activation, advanced users often use mu-plugin folder instead, so I suggest you did not use the activation hook for the installation, instead you can make a script to check if tables already exists.

Refunded orders not getting migrated to custom table

There is an issue with the migration of refunded orders to the custom table. I am running the wp wc-order-table migrate command and it is silently skipping the refunded orders, while other orders are updated just fine.

I have done a little debugging and it appears the call to populate_from_meta() on the data store is failing, in WooCommerce_Custom_Orders_Table_CLI::migrate().

This line specifically - $result = $order->get_data_store()->populate_from_meta( $order );

That call is returning NULL which is fails the error check on the line below, so there is no error reporting at all when this happens.

Consideration for this separate table being used as read-only "posting" table

Just getting to this area of a project ourselves and will do some testing but I wonder if you would consider the use of the separate custom table as a read-only accounts posting table?

Given the nature of this being a plugin, it would make for a natural extension for large-scale users only - that didn't lose backward compatibility with other plugins already referring to the existing post tables.

One would also expect that use of this plugin would then only be entertained by those absolutely needing this and mostly likely the resources to work on their other plugin compatibilities.

So - "Open" orders would remain in post tables and "Completed" orders would be archived to this separate orders table - therefore incorporation accounting principles for open and complete contractual statuses by design.

This then meets the aim of keeping the posts tables lighter and faster to query for reporting and analytics but also means that a reliance on plugins hardwired to posts can remain simply by having the ability to re-open an order - sending it back to the posts table if it needs working on or some other interaction.

Fatal error: Call to a member function get_data_store()...

Hello,
I tried the migrate command with the latest beta (3). About half way through, the process stopped with "Killed" being returned.

Trying the wp wc-order-table migrate command again to continue the process returned an error part way through.

"Fatal error: Call to a member function get_data_store() on boolean in /home/PATH-TO-SITE/wp-content/plugins/woocommerce-custom-orders-table/includes/class-woocommerce-custom-orders-table-cli.php on line 96"

Trying the command again results in the same error as above.

Running the backfill command after this also returns "Killed" part way through.

Anything I can do to remedy this?

P.S. Thank you for this project. Our wp_postmeta table as become big and is slowing down the orders backend, big time.

Check_admin_referer returning false

Describe the bug
I'm using nonce to validate access to one of my page. It's been working fine. But since I install WooCommerce Custom Orders Table, the check_admin_referer always returning false / die. When I remove check_admin_referer, I can load my page normally again. Not sure if this is caused by this plugin, but since the only changes I made was installing this plugin, I have no idea where else to look.

To Reproduce
I'm using this code, sorry if a bit mixed up:

$ajax_nonce = wp_create_nonce( "foobar" );
$fake_ajax_url = wp_nonce_url( admin_url( 'admin-ajax.php?action=woocommerce_dropshippers_order_details&order_id=' . get_the_ID()), 'woocommerce_dropshippers_order_details' );

And to check the nonce I use this statement:

if( empty( $_GET['action'] ) || ! is_user_logged_in() || !check_admin_referer( $_GET['action'] ) ) {
		wp_die( __( 'You do not have sufficient permissions to access this page.','woocommerce-dropshippers' ) );
	}
	else{ load the page }

The response that I get from the page is "The link you followed has expired. Please try again."

Expected behavior
The page loads normally with check_admin_referer returning true.

Versions

  • WordPress version 4.9.8
  • WooCommerce version 3.5.2
  • Plugin version 1.0.0-rc1
  • PHP version 7.2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.