Fetch tables metadata takes too long

[datanymizer]

Fetch tables metadata takes too long

[martingehrke]

I see SELECT pg_catalog.pg_get_serial_sequence ( ? ) millions of times on the db

[angryjohnnie]

Fetch tables metadata... takes more time than a full database dump.

I came here today to say exactly this. I have 20-something schemas. Each with about 50 tables. The metadata fetch takes over an hour. the actual db dumping part takes probably 10 min to a local disk.

TY for the tool, it does the thing, I just wish it were quicker.

[evgeniy-r]

Hello! I think that it is a some kind of bug, we will check this.

[martingehrke]

Let me know how I can help

…

On Thu, Feb 3, 2022, 10:55 AM Evgeniy Rashchepkin ***@***.***> wrote: Hello! I think that it is some kind of bug, we will check this. — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSHM2YM422LQT53SIGTUZKQO7ANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[evgeniy-r]

I have some ideas, could you please provide the database schema (the dump without data)?

[martingehrke]

angryjohnnie <https://github.com/angryjohnnie>, are you able to easily? this is a production database, I can't just share our schema, sorry.

…

On Fri, Feb 4, 2022 at 8:09 AM Evgeniy Rashchepkin ***@***.***> wrote: I have some ideas, could you please provide the database schema (the dump without data)? — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSBZSADUPV35GKTIRJTUZPFXNANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[angryjohnnie]

angryjohnnie https://github.com/angryjohnnie, are you able to easily? this is a production database, I can't just share our schema, sorry.
…

im not sure what our policy is regarding sharing that schema. I can see if I can find out though.

[angryjohnnie]

It looks like I am not allowed to share our schema either, unfortunately.

[evgeniy-r]

OK, then we will try to fix this without these schemas.

[martingehrke]

if you have branch or changes you would like to test, I can run it and supply logs

…

On Mon, Feb 7, 2022 at 11:50 AM Evgeniy Rashchepkin < ***@***.***> wrote: OK, then we will try to fix this without these schemas. — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSCGX7ZZZHVLPOSMG2TUZ7Z3TANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[marmik18]

I have a similar kind of issue, I have 4 schemas and around 20 tables in each of the schema. I also tried passing filter args using the cli <PG_DUMP_ARGS>. But it still takes more than 20 mins to fetch the metadata.

[evgeniy-r]

Can anybody run dumping with the tracing branch and copy the output here? I can build a binary if required.

[martingehrke]

I can if you can build a binary.

…

On Sun, Feb 13, 2022 at 9:53 AM Evgeniy Rashchepkin < ***@***.***> wrote: Can anybody run dumping with the tracing <#153> branch and copy the output here? I can build a binary if required. — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSGW6MYEV3CG7BXNPWDU27AXJANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[evgeniy-r]

Platform? (win/linux/mac - Darwin or ARM)

[martingehrke]

sorry, linux amd64

…

On Tue, Feb 15, 2022 at 8:58 AM Evgeniy Rashchepkin < ***@***.***> wrote: Platform? (win/linux/mac - Darwin or ARM) — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSAQ4N7TRXUNFPKVYMTU3JLX7ANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[evgeniy-r]

pg_datanymizer.zip

[martingehrke]

pg_datanymizer_debug_build --config ../config.yaml [...] Inspecting tables ... Inspecting completed in 4 seconds Inspecting table dependencies... and it has been stuck at the stage for hour+

…

On Tue, Feb 15, 2022 at 9:52 AM Evgeniy Rashchepkin < ***@***.***> wrote: pg_datanymizer.zip <https://github.com/datanymizer/datanymizer/files/8072135/pg_datanymizer.zip> — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSBGZJ57KXUMKQZ6MG3U3JSDLANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[evgeniy-r]

Thank you very much, I will look into it...

[evgeniy-r]

Try the updated version, please.
pg_datanymizer.zip

[martingehrke]

this one worked! $ time ../bin/pg_datanymizer_new_debug --config ../config.yaml [...] Fetching tables metadata... Inspecting tables ... Inspecting completed in 59 seconds Inspecting table dependencies... Inspecting completed in 0 seconds Processing table dependencies... Processing completed in 0 seconds Fetching completed in 59 seconds Sorting tables... Sorting completed in 0 seconds Prepare data scheme... [1 / 373] Prepare to dump table: public.cities

…

On Wed, Feb 16, 2022 at 1:20 PM Evgeniy Rashchepkin < ***@***.***> wrote: Try the updated version, please. pg_datanymizer.zip <https://github.com/datanymizer/datanymizer/files/8082457/pg_datanymizer.zip> — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSDQCPANUA76LTNS3GDU3PTFTANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[evgeniy-r]

@martingehrke, can you check this one too please?
pg_datanymizer.zip

[martingehrke]

sorry for the late reply. this one works as well; Fetching tables metadata... Inspecting tables ... Inspecting completed in 1 minute Inspecting table dependencies... Inspecting completed in 0 seconds Processing table dependencies... Processing completed in 0 seconds Fetching completed in 1 minute Sorting tables... Sorting completed in 0 seconds Prepare data scheme...

…

On Sun, Feb 20, 2022 at 9:53 AM Evgeniy Rashchepkin < ***@***.***> wrote: @martingehrke <https://github.com/martingehrke>, can you check this one too please? pg_datanymizer.zip <https://github.com/datanymizer/datanymizer/files/8104212/pg_datanymizer.zip> — Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWRLSF5ELWY52GZ3LQY4KLU4D553ANCNFSM5NNBFGCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>