[Bug]: date format changed from input to output

[logoshot]

[Bug]: date format changed from input to output

[logoshot]

I tried to print data from to_json, found that the time value is right before to_json, so the datetime value is changed during to_json.

    @staticmethod
    def to_jsonl(dataset, export_path, num_proc=1, **kwargs):
        """
        Export method for json/jsonl target files.

        :param dataset: the dataset to export.
        :param export_path: the path to store the exported dataset.
        :param num_proc: the number of processes used to export the dataset.
        :param kwargs: extra arguments.
        :return:
        """
        print(dataset['time'])
        dataset.to_json(export_path, force_ascii=False, num_proc=num_proc)

[zhijianma]

I tried to print data from to_json, found that the time value is right before to_json, so the datetime value is changed during to_json.

    @staticmethod
    def to_jsonl(dataset, export_path, num_proc=1, **kwargs):
        """
        Export method for json/jsonl target files.

        :param dataset: the dataset to export.
        :param export_path: the path to store the exported dataset.
        :param num_proc: the number of processes used to export the dataset.
        :param kwargs: extra arguments.
        :return:
        """
        print(dataset['time'])
        dataset.to_json(export_path, force_ascii=False, num_proc=num_proc)

The image bellow is the content after exporting dataset with iso in my local machine, and we can find the values of text remain unchaned , but with a ios format. Please check your local time and the Python dependencies, such as datasets, pandas, and pyarrow.

[logoshot]

I install these package with command pip install -v -e .[all], so I think it is the default version, could you help me check your version, here is mine:

pandas 2.0.0
datasets 2.11.0
pyarrow 14.0.1

[zhijianma]

'2023-10-13 16:06:31'

This maybe a bug of pyarrow from v13.0.
I have tested from v11.0 to v14.0 with date_format iso, here is my results:

• v11.0.0 and v12.0.0 write the right text 2023-10-13T16:06:31.000 to the jsonl file.
• v13.0.0 and v14.0.0 write the wrong text 1970-01-01T00:00:01.697 to the jsonl file.
  So , we suggest you to downgrade pyarrw to v12.0.0 .

[github-actions]

This issue is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this issue will be closed in 3 day.

[github-actions]

Close this stale issue.