Content duplicates

[juliushaertl]

Content duplicates

[max-nextcloud]

@bugspencor thanks for your heads up. Let me ask you some questions first before I reply to yours...

Questions

• Are you seeing this with folder descriptions only (Readme.md) or also with other files?
• Do you have a file with duplicate content that has not been saved to the file yet?
  Could you make sure the file does not contain private content and provide us with a database excerpt for that file?
  You'd need to run the following query on your database:
  SELECT * FROM oc_text_steps WHERE document_id=723
  (replacing 723 with the file-id in question)

Answering your questions

In which Nextcloud versions is that bug fixed? I still see a similar symptom in Nextcloud 27.1.10 with Text 3.8.0 (I first noticed it after the update to 27.1.9, but it might have been there already).

The fixes are included in #5543 and were merged in 27.1.9.

Are there any manual steps required or helpful to resolve the issue and get into a clean state?

You can clear all the current editing sessions with

occ text:reset [file-id] 

Editing sessions are long running to allow reconnects when users open their laptop after a while again.

What I see that sometimes when I download the file the content is correct (only once). But content displayed in the editor (and above the file listing) is duplicate.

That state will be fixed by running the above command for the file in question

Sometimes the duplicate content is saved to the file.

In this case the command won't make any difference.

[max-nextcloud]

Other code path to check

We discussed this today and it looks like this code path is indeed the culprit:

1. When all sessions are closed the document table entry is removed but the yjs file remains in case someone reopens their laptop and wants to continue the session.
2. Someone creates a new session and the document is loaded from the markdown ( first if block).
3. Next person comes around before the doc has been autosaved. Now a document entry exists again - therefore fresh session is false and the content of the yjs state file is send out instead.
4. The two clients communicate and combine the content they both loaded via different mechanisms
5. content duplicate.

[mejo-]

This should hopefully be fixed now with all the pieces from #5476 (comment)

[max-nextcloud]

• add an occ task to clean up all sessions
• repair step that does this with next release
• store document state after initialization
  maybe only send content when initializing a doc without any steps rather than without document state 💭

[juliushaertl]

Looks a bit suspicious, maybe this is related to the file opening process in some way.

FYI @max-nextcloud @mejo-

[juliushaertl]

Just an idea, with some questions that might be worth to investigate:

• Could there be a race condition when multiple sessions initialize a session where we initially set the content of the y.js doc which then could lead to two steps being pushed with the initial document source?
  • Probably worth to check here https://github.com/nextcloud/text/blob/enh/vite/src/components/Editor.vue#L540
  • Can we check on the backend side if the step that is being pushed is such a initial one (so we could just skip any second)?

[mejo-]

I recently encountered this as well (on 27.1.6). Unfortunately I didn't have time to debug in that very moment. In case it happens again, probably a backup of the steps in the database could be helpful, right?

[netzpolitikorg]

We can confirm, at least with 28.0.2, probably also 28.0.3.

[juliushaertl]

Managed to reproduce by:

• Open a document
• Manually drop the entry from oc_text_documents
• Reload the page

Steps seem to be still in the steps file and then reapply on top of the new base document

[juliushaertl]

A fix for a possible case is in #5470 however I currently cannot see other code paths that could lead to this

[juliushaertl]

Other code path to check

text/lib/Service/ApiService.php

Lines 139 to 159 in 9412ae1

	if ($freshSession) {
	$this->logger->debug('Starting a fresh editing session for ' . $file->getId());
	$documentState = null;
	$content = $this->loadContent($file);
	} else {
	$this->logger->debug('Loading existing session for ' . $file->getId());
	$content = null;
	try {
	$stateFile = $this->documentService->getStateFile($document->getId());
	$documentState = $stateFile->getContent();
	} catch (NotFoundException $e) {
	$this->logger->debug('State file not found for ' . $file->getId());
	$documentState = ''; // no state saved yet.
	// If there are no steps yet we might still need the content.
	$steps = $this->documentService->getSteps($document->getId(), 0);
	if (empty($steps)) {
	$this->logger->debug('Empty steps, loading content for ' . $file->getId());
	$content = $this->loadContent($file);
	}
	}
	}

[bugspencor]

In which Nextcloud versions is that bug fixed? I still see a similar symptom in Nextcloud 27.1.10 with Text 3.8.0 (I first noticed it after the update to 27.1.9, but it might have been there already).

Are there any manual steps required or helpful to resolve the issue and get into a clean state?

What I see that sometimes when I download the file the content is correct (only once). But content displayed in the editor (and above the file listing) is duplicate. Sometimes the duplicate content is saved to the file. I experimented with moving appdata_xyz/text/documents/ and after scan-app-data (IIRC) some files showed the content three times.

[bugspencor]

@max-nextcloud thanks for the comment. To answer your questions:

• this happens with other non-Readme.md files, too
• I have one example with the default Documents/Readme.md (I did not find another non-Readme.md which is not saved with duplicate content and does not contain private information, if that's required let me know, then I'll search more thoroughly):

# mariadb nextcloud -B -e "SELECT * FROM oc_text_steps WHERE document_id=433709"
id	document_id	session_id	data	version
4526	433709	1110	["AAKUAgEKzKaluAUABwEHZGVmYXVsdAMHaGVhZGluZwcAzKaluAUABgQAzKaluAUBCURvY3VtZW50cygAzKaluAUABWxldmVsAX0BKADMpqW4BQACaWQBfygAzKaluAUABHV1aWQBf4fMpqW4BQADCXBhcmFncmFwaAcAzKaluAUOBgQAzKaluAUPek5leHRjbG91ZCB3b3JrcyB3ZWxsIHdpdGggYWxsIHRoZSBjb21tb24gZG9jdW1lbnQgZm9ybWF0cy4gWW91IGNhbiBldmVuIGNvbGxhYm9yYXRlIHdpdGggb3RoZXJzIG9uIE9EVCBhbmQgTWFya2Rvd24gZmlsZXMhh8ympbgFDgMJcGFyYWdyYXBoAA==","AAJVAQLMpqW4BYsBqMympbgFDAF3C2gtZG9jdW1lbnRzqMympbgFDQF3JDU1NDFlMTFiLTEzYjEtNGJlNy1hMzc2LTFjOWZhOWJiZGJmMQHMpqW4BQEMAg=="]	2147483647
8047	433709	2626	["AALsAQEKAAAHAQdkZWZhdWx0AwdoZWFkaW5nBwAAAAYEAAABCURvY3VtZW50cygAAAAFbGV2ZWwBfQEoAAAAAmlkAX8oAAAABHV1aWQBf4cAAAMJcGFyYWdyYXBoBwAADgYEAAAPek5leHRjbG91ZCB3b3JrcyB3ZWxsIHdpdGggYWxsIHRoZSBjb21tb24gZG9jdW1lbnQgZm9ybWF0cy4gWW91IGNhbiBldmVuIGNvbGxhYm9yYXRlIHdpdGggb3RoZXJzIG9uIE9EVCBhbmQgTWFya2Rvd24gZmlsZXMhhwAOAwlwYXJhZ3JhcGgA","AAJIAQLM3ZGPDgCoAAwBdwtoLWRvY3VtZW50c6gADQF3JGE2OTMzYWI3LTY0NjktNDRhYS04ZjM0LTBlNGU4YjgzYjdkMwEAAQwC"]	2147483647
8048	433709	2626	["AAIrAQHM3ZGPDgKozKaluAWLAQF3DmgtZG9jdW1lbnRzLS0xAcympbgFAYsBAQ=="]	2147483647
8051	433709	2638	["AALsAQEKAAAHAQdkZWZhdWx0AwdoZWFkaW5nBwAAAAYEAAABCURvY3VtZW50cygAAAAFbGV2ZWwBfQEoAAAAAmlkAX8oAAAABHV1aWQBf4cAAAMJcGFyYWdyYXBoBwAADgYEAAAPek5leHRjbG91ZCB3b3JrcyB3ZWxsIHdpdGggYWxsIHRoZSBjb21tb24gZG9jdW1lbnQgZm9ybWF0cy4gWW91IGNhbiBldmVuIGNvbGxhYm9yYXRlIHdpdGggb3RoZXJzIG9uIE9EVCBhbmQgTWFya2Rvd24gZmlsZXMhhwAOAwlwYXJhZ3JhcGgA","AAJIAQLd8JrOCwCoAAwBdwtoLWRvY3VtZW50c6gADQF3JGM5NzljMmRhLWUyNGUtNGZjYi04YmRlLWYzYmUxZmIwZmE1ZAEAAQwC"]	2147483647
8052	433709	2641	["AALsAQEKAAAHAQdkZWZhdWx0AwdoZWFkaW5nBwAAAAYEAAABCURvY3VtZW50cygAAAAFbGV2ZWwBfQEoAAAAAmlkAX8oAAAABHV1aWQBf4cAAAMJcGFyYWdyYXBoBwAADgYEAAAPek5leHRjbG91ZCB3b3JrcyB3ZWxsIHdpdGggYWxsIHRoZSBjb21tb24gZG9jdW1lbnQgZm9ybWF0cy4gWW91IGNhbiBldmVuIGNvbGxhYm9yYXRlIHdpdGggb3RoZXJzIG9uIE9EVCBhbmQgTWFya2Rvd24gZmlsZXMhhwAOAwlwYXJhZ3JhcGgA","AAJHAQK15YMNAKgADAF3C2gtZG9jdW1lbnRzqAANAXckNWM0NWI4MjEtYTc1MC00OGIyLWEzZjAtMzI3M2M1MmJhYTI1AQABDAI="]	2147483647

Some other information, maybe there's something relevant:

• I see that duplicate behaviour in our test instance, too
• In both instances I have the Versions app enabled
• It seems the problem can be easily reproduced (above example was from a random user, and seen with both Readme.mds we tried).
• Like said in my previous comment I already tried several occ commands to cleanup. I also enabled the Plain Text Editor once and tried to configure it for other filetypes, something like that: https://help.nextcloud.com/t/plain-text-editor-for-all-files/133291 Unfortunately I have not documented it well. Currently the Plain Text Editor is disabled

I have not yet issued any occ text:reset [file-id] commands.

Please let me know if you need anymore info.

[max-nextcloud]

• It seems the problem can be easily reproduced

Could you try to reproduce this problem from scratch - i.e. creating a new file or uploading one and then creating the problem from there?

I suspect this is due to editing sessions still being around from before the update. However if you can still create the problem with a new file that could not be the case.

If you can reproduce with new files... could you create a screen recording of the steps so i can try and reproduce myself?

Thanks a lot for your support in tracking this down.

[bugspencor]

I could not reproduce the problem with a new file with my account and another user's account.

Probably I can reproduce the problem with other users with existing files. Can I check the session state and their files before logging in and reproducing the issues?

[juliushaertl]

In addition maybe we could actually add a timestamp or version identifier to the steps table so we could identify old versions in the future more easily.

[mejo-]

Related bugreport in Collectives: nextcloud/collectives#1270