nextcloud / previewgenerator Goto Github PK

View Code? Open in Web Editor NEW

442.0 17.0 53.0 405 KB

Nextcloud app to do preview generation in the background.

Home Page: https://apps.nextcloud.com/apps/previewgenerator

License: GNU Affero General Public License v3.0

PHP 100.00%

nextcloud previews

previewgenerator's Introduction

Preview Generator

Nextcloud app that allows admins to pre-generate previews. The app listens to edit events and stores this information. Once a cron job is triggered it will start preview generation. This means that you can better utilize your system by pre-generating previews when your system is normally idle and thus putting less load on your machine when the requests are actually served.

The app does not replace on demand preview generation so if a preview is requested before it is pre-generated it will still be shown.

How to install

Install directly from within your Nextcloud from the app store
Clone this repository into your Nextcloud app folder

How to use the app

Install the app
Enable the app
Run ./occ preview:generate-all once after installation.
Add a (system) cron job for ./occ preview:pre-generate

I run it every 10 minutes

Known issues

The app does not work with encryption enabled

How does the app work

Listen to events that a file has been written or modified and store it in the database
On cron run request previews for the files that have been written or modified

If a preview already exists at step 2 then requesting it is really cheap. If not it will be generated. Depending on the sizes of the files and the hardware you are running on the time this takes can vary.

Commands

preview:generate-all [--path=PATH ...] [user_id ...]

Loop over all files and try to generate previews for them. If one or multiple user ids are supplied it will just loop over the files of those users. You can also limit the generation to one or more paths using --path="/[username]/files/[folder path]", e.g. --path="/alice/files/Photos". Note that all given user_ids are ignored if at least one path is specified.

preview:pre-generate

Do the actual pregeneration. This means only for new or modified files (since the app was enabled or the last pregeneration was done).

FAQ

Why can't I use webcron?

Preview generation can be a very long running job. Thus we need a system that does not time out.

I get "Command already running"

Yes this happens when the pre-generate command crashes. No worries the lock will be released after 30 minutes of inactivity from the app. So go grab a cookie.

What if I'm using Nextcloud All In One?

Follow these instructions

I don't want to generate all the preview sizes

This is possible since version 1.0.8. Just set the correct values via the command line

./occ config:app:set --value="64 256 1024" previewgenerator squareSizes
./occ config:app:set --value="64 256 1024" previewgenerator widthSizes
./occ config:app:set --value="64 256 1024" previewgenerator heightSizes

This will only generate:

square previews of: 64x64, 256x256 and 1024x1024
aspect ratio previews with a width of: 64, 256 and 1024
aspect ratio previews with a height of: 64, 256 and 1024

Note:

preview sizes are always a power of 4.
The smallest size is 64
The max size is determined by your preview settings in config.php

I get "PHP Fatal error: Allowed memory size of X bytes exhausted"

You need to increase the memory allowance of PHP, by default it is 128 MB. You do that by changing the memory_limit in the php.ini file.

If you use the docker container you need set the environment variable PHP_MEMORY_LIMIT instead.

I want to skip a folder and everything in/under it

Add an empty file with the name .nomedia in the folder you wish to skip. All files and subfolders of the folder containing .nomedia will also be skipped.

I want to reset/regenerate all previews

WARNING: This is not supported but it has been confirmed to work by multiple users. Proceed at your own risk. Always keep backups around.

Remove the folder your-nextcloud-data-directory/appdata_*/preview
Optional: change parameters preview_max_x and preview_max_y in config.php (e.g., to 512), and change the previewgenerator app parameters heightSizes, squareSizes and widthSizes as per the README (or better yet, to a low value each, e.g. 512, 256 and 512 respectively)
Run occ files:scan-app-data (this will reset generated previews in the database)
Run occ preview:generate-all [user-id] (this will run very fast if you did step 2)

previewgenerator's People

Contributors

Stargazers

Watchers

Forkers

akhenaton adduxa noudandi icewind1991 oliverpa77 sndrr lomion0815 seik lukerepko biva seanpandrew phief cronekorkn chy-causer bakkegaard aeltawela wrestrtdr adnoh 0x00031337 rcm7 nachoparker yennor doblanko daveaross apmole hashworks gargeburt ctfhacker cwhipple christophwurst flyingpersian fakezeta anpavlov alexsat hiro5id ikogan nssatlantis markuman fenvarien meonkeys hammer065 proletarius101 michaelepope quinnzipse florencebiree dotlambda akiiino jl102 alexgaspar guieco asherbond

previewgenerator's Issues

mp3 file (attached) that triggers an exception during pre-generate

Hello,
I manually ran "generate all", then I manually ran pre-generate, and I got the following exception:

php -f /var/www/nextcloud/occ preview:pre-generate
An unhandled exception has been thrown:
Error: Call to a member function file_get_contents() on null in /var/www/nextcloud/lib/private/Files/Filesystem.php:719
Stack trace:
#0 /var/www/nextcloud/lib/private/legacy/image.php(597): OC\Files\Filesystem::file_get_contents('/var/www/nextcl...')
#1 /var/www/nextcloud/lib/private/Preview/MP3.php(80): OC_Image->loadFromFile('/var/www/nextcl...')
#2 /var/www/nextcloud/lib/private/Preview/MP3.php(64): OC\Preview\MP3->getNoCoverThumbnail()
#3 /var/www/nextcloud/lib/private/Preview/GeneratorHelper.php(54): OC\Preview\MP3->getThumbnail('/files/Document...', 2048, 2048, false, Object(OC\Files\View))

I'm running Nextcloud 11.0.2 (stable), with defaults apps and previewgenerator activated. previewgenerator 1.0.3.

After debugging a while, I found the file that triggers the exception, it's a mp3 file from a chinese book I bought, here it is:

6440_p105_Bon.zip

Thanks for the great pluging !

Why is this in the app store and not in the default install?

@rullzer seems valuable for everyone, no? Or does it require some dependencies only available on certain systems, or admin interference?

Skip failed file instead of cancelling the whole job

I get this:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 5742 bytes) in /lib/private/legacy/image.php on line 538

not a big deal if a thumbmnail can't be generated, but I think it just kills the whole job, which should be avoided.

I am afraid of running out of inodes

The preview pictures are located in ~/nextcloud_data/appdata_XXXXXXX/preview
Each file has his own subfolder for the preview pictures in there.

The count of subfolders in a folder is limited by the inodes of the system. So if you have really lots of files, the chance to run out of inodes is growing. The result would be that you can't ls -al anymore in the preview folder.

I would suggest that the subfolders for the preview pictures are not stored in one huge folder, rather divided into a certain count of subfolders which contains subfolders as well.

Similar like the cache system of Squid proxy.

squid -z
ls -al /var/spool/squid

occ preview:pre-generate don't stop after hours of execution

Hi,

after 2 days that the process runs for hours, I have start the command with verbose flag:
occ preview:pre-generate --force -vvv
it don't stop and seems analyze continuously and circularly all the images.

preview:generate error

$ sudo -u oc_user php5 /www/owncloud/occ preview:generate

**** Unrecoverable error in xref!
Error: /rangecheck in resolveR
Operand stack:
--dict:3/3(L)-- --dict:4/4(L)-- --dict:4/4(L)-- --dict:4/4(L)-- 4252440 1050 z▒▒▒
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval--
false 1 %stopped_push 1910 1 3 %oparray_pop 1909 1 3 %oparray_pop 1893 1 3 %oparray_pop --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval-- %array_continue --nostringval-- --nostringval-- --nostringval-- %array_continue --nostringval-- --nostringval-- --nostringval-- %array_continue --nostringval-- --nostringval-- --nostring
val-- %array_continue --nostringval-- --nostringval-- --nostringval--
Dictionary stack:
--dict:1174/1684(ro)(G)-- --dict:1/20(G)-- --dict:82/200(L)-- --dict:82/200(L)-- --dict:109/127(ro)(G)-- --dict:293/300(ro)(G)-- --dict:21/31(
L)-- --dict:5/10(L)--
Current allocation mode is local
GPL Ghostscript 9.06: Unrecoverable error, exit code 1
Error: /typecheck in defineresource
Operand stack:
--dict:10/19(L)-- false DefaultGray DeviceGray ColorSpace DefaultGray DeviceGray
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval--
false 1 %stopped_push 1910 1 3 %oparray_pop 1909 1 3 %oparray_pop 1893 1 3 %oparray_pop --nostringval-- --nostringval-- 2
1 1 --nostringval-- %for_pos_int_continue --nostringval-- --nostringval-- --nostringval-- 1862 5 7 %oparray_pop defineresource %errorexec_pop --nostringval--
Dictionary stack:
--dict:1175/1684(ro)(G)-- --dict:1/20(G)-- --dict:82/200(L)-- --dict:82/200(L)-- --dict:109/127(ro)(G)-- --dict:293/300(ro)(G)-- --dict:25/31(
L)-- --dict:16/20(ro)(G)--
Current allocation mode is local
Last OS error: No such file or directory
GPL Ghostscript 9.06: Unrecoverable error, exit code 1
Error: /undefined in ▒d
Operand stack:

Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1910 1 3 %oparray_pop 1909 1 3 %oparray_pop 1893 1 3 %oparray_pop 1787 1 3 %oparray_pop --nostringval-- %errorexec_pop .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval--
Dictionary stack:
--dict:1174/1684(ro)(G)-- --dict:0/20(G)-- --dict:77/200(L)--
Current allocation mode is local
Current file position is 5
GPL Ghostscript 9.06: Unrecoverable error, exit code 1
PHP Fatal error: imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Unsupported marker type 0xb9 in /www/owncloud/lib/private/legacy/image.php on line 538

Use a background job (instead of cron)

Be sure to add './occ preview:pre-generate' to a cronjob

generate-all ignores global log level

Hello!

The "generate-all" command seems to ignore the global log level of Nextcloud.

Steps to reproduce

root@red:htdocs/config# grep level config.php -H
config.php:  'loglevel' => 2,

Today I've got "php occ preview:generate-all" running in a screen.
At the same time I was sorting and moving tons of pictures around (but I was connected via Samba, so previewgenerator should not be triggered).
I noticed some log entries in htdocs/data/nextcloud.log with log level INFO.
I finished sorting my pics.
There are still INFO-messages logged. See screenshot.

Expected

No logging into htdocs/data/nextcloud.log

Actual

Logging into logfile, despite loglevel set to 2 (2=warning, 1=info)

Screenshots

INFO messages:

Used previewgenerator-command:

Since I'm not doing anything right now (except for the generator that is still running), I conclude that this must be the preview generator.
I double checked this and I saw that IMG_2990 was logged around the very same time it was processed by the command. (The screenshots were taken at close but different times, so you cannot see that.)
Only thing is that the logging app is "admin_audit". I hope that bug is right in this project. If not, I appreciate for help by directing me to the right way.

Command already running

I get this message from cron for about 2 days now... every 15 minutes... i'm getting fat if I eat more cockies... :) what is causing this and how can I fix it?

Q: Why does it abort when encryption is enabled

Hi,

The idea of the app is very good.
why can't previewgenerator work if encryption is enabled?
To make Nextcloud secure, https is the good option...

Thanks for the answer

Error while running preview:generate-all

after few hours i get

root@omv:/var/www/nextcloud# sudo -uowncloud php ./occ preview:generate-all PHP Fatal error: imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Unsupported marker type 0x6f in /var/www/nextcloud/lib/private/legacy/image.php on line 538

nexcloud v11.0.0
previewgenerator v1.0.1
php v5.6.29

Error on occ preview:pre-generate

occ preview:pre-generate command generates an exception:

An unhandled exception has been thrown:
Error: Call to a member function file_get_contents() on null in /var/www/kolosowscy.cloud/htdocs/lib/private/Files/Filesystem.php:719
Stack trace:
#0 /var/www/kolosowscy.cloud/htdocs/lib/private/legacy/image.php(597): OC\Files\Filesystem::file_get_contents('/var/www/koloso...')
#1 /var/www/kolosowscy.cloud/htdocs/lib/private/Preview/MP3.php(80): OC_Image->loadFromFile('/var/www/koloso...')
#2 /var/www/kolosowscy.cloud/htdocs/lib/private/Preview/MP3.php(64): OC\Preview\MP3->getNoCoverThumbnail()
#3 /var/www/kolosowscy.cloud/htdocs/lib/private/Preview/GeneratorHelper.php(54): OC\Preview\MP3->getThumbnail('/files/Dokument...', 2048, 2048, false, Object(OC\Files\View))
#4 /var/www/kolosowscy.cloud/htdocs/lib/private/Preview/Generator.php(155): OC\Preview\GeneratorHelper->getThumbnail(Object(OC\Preview\MP3), Object(OC\Files\Node\File), 2048, 2048)
#5 /var/www/kolosowscy.cloud/htdocs/lib/private/Preview/Generator.php(108): OC\Preview\Generator->getMaxPreview(Object(OC\Files\SimpleFS\SimpleFolder), Object(OC\Files\Node\File), 'audio/mpeg')
#6 /var/www/kolosowscy.cloud/htdocs/lib/private/PreviewManager.php(182): OC\Preview\Generator->getPreview(Object(OC\Files\Node\File), 32, 32, true, 'fill', 'audio/mpeg')
#7 /var/www/kolosowscy.cloud/htdocs/apps/previewgenerator/lib/Command/PreGenerate.php(158): OC\PreviewManager->getPreview(Object(OC\Files\Node\File), 32, 32, true)
#8 /var/www/kolosowscy.cloud/htdocs/apps/previewgenerator/lib/Command/PreGenerate.php(181): OCA\PreviewGenerator\Command\PreGenerate->processFile(Object(OC\Files\Node\File))
#9 /var/www/kolosowscy.cloud/htdocs/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
#10 /var/www/kolosowscy.cloud/htdocs/apps/previewgenerator/lib/Command/PreGenerate.php(150): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
#11 /var/www/kolosowscy.cloud/htdocs/apps/previewgenerator/lib/Command/PreGenerate.php(113): OCA\PreviewGenerator\Command\PreGenerate->processRow(Array)
#12 /var/www/kolosowscy.cloud/htdocs/apps/previewgenerator/lib/Command/PreGenerate.php(91): OCA\PreviewGenerator\Command\PreGenerate->startProcessing()
#13 /var/www/kolosowscy.cloud/htdocs/3rdparty/symfony/console/Command/Command.php(256): OCA\PreviewGenerator\Command\PreGenerate->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/kolosowscy.cloud/htdocs/3rdparty/symfony/console/Application.php(818): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/kolosowscy.cloud/htdocs/3rdparty/symfony/console/Application.php(186): Symfony\Component\Console\Application->doRunCommand(Object(OCA\PreviewGenerator\Command\PreGenerate), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/kolosowscy.cloud/htdocs/3rdparty/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /var/www/kolosowscy.cloud/htdocs/lib/private/Console/Application.php(169): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /var/www/kolosowscy.cloud/htdocs/console.php(90): OC\Console\Application->run()
#19 /var/www/kolosowscy.cloud/htdocs/occ(11): require_once('/var/www/koloso...')
#20 {main}

Previews going to appdata folder and not being used

Command runs fine and builds a large cache of previews in the appdata_xxxx/preview folder, but none of these files appear to be used by nextcloud, and the gallery app makes on-demand previews in the user/thumbnails folder instead of using the previews generated by this app. Have I configured something incorrectly?

Link to app store in repo website

This repository's website (as seen on the default GitHub view) should probably link to https://apps.nextcloud.com/apps/previewgenerator.

Pre-generate issue

Hey, thanks for this app!

When running any command I get this error:

root@cloud:~# sudo -u www-data php /var/www/nextcloud/occ preview:pre-generate

                                                                                                                      
  [OC\Encryption\Exceptions\DecryptionFailedException]                                                                
  Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.  
                                                                                                                      

preview:pre-generate

PHP Fatal error:  Uncaught Error: Access to undeclared static property: OC\Files\Filesystem::$normalizedPathCache in /var/www/nextcloud/lib/private/Files/Filesystem.php:806
Stack trace:
#0 /var/www/nextcloud/lib/private/Files/View.php(2018): OC\Files\Filesystem::normalizePath('/usr/files/Ph...')
#1 /var/www/nextcloud/lib/private/Files/View.php(1156): OC\Files\View->unlockFile('/files/Photos/M...', 1)
#2 [internal function]: OC\Files\View->OC\Files\{closure}()
#3 /var/www/nextcloud/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(109): call_user_func(Object(Closure))
#4 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#5 {main}
  thrown in /var/www/nextcloud/lib/private/Files/Filesystem.php on line 806

Anything to do with decryption?

Fatal error running preview generator

When i execute the command: "occ preview:pre-generate" i am getting the follwoing error:

imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Invalid JPEG file structure: two SOI markers in /lib/private/legacy/image.php on line 538

Not sure what it means but i comes everytime when i run the preview generator and it seems that because of it no more previews are beeing generated.

unrecoverable error instead of error with notice

Hello,
when I run:
/var/www/nextcloud# sudo -u www-data /usr/bin/php -f ./occ preview:"doesn't matter"
it goes thrue some image that generates error and script ends with log:
imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Unsupported marker type 0x28 at /var/www/nextcloud/lib/private/legacy/image.php#538
"0x28" suggest problem with file but:

sometimes it's better to know :) ie which file is problematic.
it would be nice if preview will inform about corrupted jpeg and go straigt, but not die.
best regards
Krzysztof

new option --force

Currently there is a timeout of 30min.
If, e.g. a preview generation fails with a fatal exception, the user would have to wait.
With a new option "--force" this timeout could be bypassed.

Use relative paths

I recently renamed my data directory from /data/owncloud to /data/nextcloud. This had the unfortunate consequence of making all the previously generated previews invalid and forcing me to re-generate them all (which is quite unpleasant since it takes days...). I guess that the paths are coded as full paths, therefore making such as data directory rename quite unpleasant. If paths were stored relative to the data directory root, then the previews would still be valid and lots of time would be saved...

Remote (or offline) preview generations

Don't get me wrong: I was waiting for this app for a long time, so many thanks for making it real!

It is now time to ask for more 😉

In my use case, my server is a not-so-powerful Atom CPU that is not really able to generate a lot of previews at once in a reasonable time. However, I have my desktop that have full access to the source files (in my case, via NFS), so it can theoretically do the job in less time. Is this possible in the current design? Maybe the algorithm needs an access to the database?

Missing check about running processes

There is no lockfile, flag or other control that indicates that preview generation is still running. Thus, when running ./occ preview:pre-generate it will just start to do the same tasks. Depending on the activity on the instance, one run of ./occ preview:pre-generate can already take a long time. So there is the chance that previews for each file are generated twice or multiple files (and making them processes run even longer).

How to see if it works?

Hello,

how I can see if it works? When I do it manually (or with crontab) is there no output.

Edit1:

I tried it now with

sudo -u www-data php -f /var/www/occ preview:pre-generate -vvv

still no output. I'm doing something wrong? I'm using Ubuntu 16.04 with PHP7 with Nextcloud 11.

Greetings
Roger

Preview Generator 1.0.1 → occ preview:pre-generate runs forever on certain PDF file

No warnings in terminal, no messages in nextcloud.log or Apache log.

Example of PDF file and more info:

https://help.nextcloud.com/t/preview-generator-1-0-1-runs-forever-app-is-not-usable-anymore/8481

Preview Generator 1.0.1 → Filename truncated in nextcloud.log

NC 11.0.1 on Ubuntu 16.04.1

The occ command preview:generate-all truncates the filenames in nextcloud.log, therefore debugging is impossible at the moment.

Example

sudo -u www-data php occ preview:generate-all

tail -f nextcloud.log

MP3->getThumbnail('\/files\/Archiv\/K...', 2048, 2048, false, O

Detailed info

https://help.nextcloud.com/t/preview-generate-all-not-working/8190/4

php fatal error

I'm not 100% sure, but I think that error started appearing after the last update of the plugin. The version I'm using is: 1.0.5
I've got a cronjub which runs:
/var/www/html/cloud/occ preview:pre-generate

it then generates following error message.

PHP Fatal error: Call to a member function file_get_contents() on null in /var/www/html/cloud/lib/private/Files/Filesystem.php on line 719

The last few months it was running well. But as I'm said, it happened shortly after the last update, but I can't say right now if directly, or if maybe some files were uploaded which generate that error.

Generate custom resolutions

It's to be good to generate previews with custom resolution.
It's maybe something like that
occ preview:generate-all 200x100
syntax it's only example - need to be tuned.

Preview Generator 1.0.1 → crash on mp3 file without cover image

The app stops working in case of mp3 without a cover image.

No way to debug it because the filename is truncated in nextcloud.log (see #19)

More info:
https://help.nextcloud.com/t/preview-generate-all-not-working/8190

problems adding con job "pre-generate" with webcron

Hey,
maybe a stupid question, but how can I add

To enable pre-generation of previews you must add ./occ preview:pre-generate to a system cron job

the generation of thumbnails, if I only can use the "Webcron" functionality of nextcloud.

Thanks for any hint

PHP Fatal error on generation

I'm recieving this error while generating a preview of an Android panorama image.

PHP Fatal error: imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Unsupported marker type 0x93 in /home/cloud/public_html/lib/private/legacy/image.php on line 538

I don't need to support this file types, but it would be great, if the generator creation will not be aborted on any kind of error.

Preview Generator 1.0.1 → big JPG file → occ preview:pre-generate → killed

sudo -u www-data php occ preview:pre-generate
↓
Getötet

That's all what the app displays. No log in nextcloud.log or Apache as well. Just "killed".

It comes from a big jpg file which is created by Hugin panorama editor.

The preview generator app should parse the image size before processing it and ignore the preview generation if the image is too big.

How are the limitations at the moment and how can i adjust them?

Include external storage?

According to the desciption

Listen to events that a file has been writen or modified and store it in the database

an tests with my NextCloud installation it seems, that preview:generate-all does not include external storage. Is that right?

If so: It would be a nice feature for future versions, to generate previews for external storage content (e.g. with an additional trigger like ./occ preview:generate-all [include-external]).
I've added my photo collection to my NextCloud (runnig on a Raspberry PI 3) via SMB...and the on-demand generation speed of the thumbnail pics is not really satisfying...

Thanks a lot in advance!

what happens for images on external storage shared by all users

In my NC setup I have an external storage /path/to/my/photos setup for all users. When previewgenerator generates image caches, will it generate one for each user or will it only generate one cache and share it for all the users?

Preview pregeneration takes ages since update.

Hi,
I have a cron job with this script to run preview pre-generation twice a day, but since update it takes days to pre generate something, almost as long as full preview generation.

I have load around 5 new jpegs per day and 9653 at all.

Full preview generation takes around 3 days. But it is better, because it does not works before:)

Why it takes so long?
From the logs:

more nextcloud.log | grep Preview
{"app":"/var/www/nextcloud/occ preview:pre-generate","message":"+++ Preview generation is Completed.    Time: 91595s +++","level":1,"time":"2017-03-20T02:26:36+01:00"}
{"app":"/var/www/nextcloud/occ preview:pre-generate","message":"+++ Starting Cron Preview generation +++","level":1,"time":"2017-03-20T13:00:02+01:00"}
{"app":"/var/www/nextcloud/occ preview:pre-generate","message":"+++ Preview generation is Completed.    Time: 800s +++","level":1,"time":"2017-03-20T13:13:22+01:00"}
{"app":"/var/www/nextcloud/occ preview:pre-generate","message":"+++ Starting Cron Preview generation +++","level":1,"time":"2017-03-21T01:00:01+01:00"}
{"app":"/var/www/nextcloud/occ preview:pre-generate","message":"+++ Preview generation is Completed.    Time: 106071s +++","level":1,"time":"2017-03-22T06:27:52+01:00"}

Question about how the app works

Hi, I love the concept of this app but I'm bit confused how this app works.
I have created a test user (username: photos.db), uploaded 800MB folder full of photos.
then initiate the scan:

sudo -u www php /www/nextcloud/occ preview:generate-all photos.db

When that finished the appdata/preview folder was 500MB in size - that's nearly as big as the original folder. Furthermore when I login to nextcloud and switch to galery view I cannot see any speed improvment in the preview generation.

Am I missing something here?

README

This repo could use a README :)

Error while running preview:pre-generate

Hi All,
just after few seconds after I run
sudo -u www-data php /var/www/nextcloud/occ preview:pre-generate
I had an error:
PHP Fatal error: imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Unsupported marker type 0xba in /var/www/nextcloud/lib/private/legacy/image.php on line 538

My system is:
PHP 7.0.13
mysql 5.7.17
Nexcloud 11.0.1

Error: Call to a member function file_get_contents() on null in .../lib/private/Files/Filesystem.php:719

Hello Dev, ive encountered an Error which i cant solve on my own.
Im using Nextcloud 11.0.2 and php 7.0.8-0ubuntu0.16.04.3 with apache2 as the webserver.

When i start your tool from my terminal, it runs for a few seconds (no outputs are made to the console so im assuming it does at least something), and then it throws an exception. CPU usage is slightly higher than normal.

Thanks for your help, and if i can assist you further, feel free to ask!

Sorry for the missreferences, my log contained hashtags, which function i didnt knew...
The stacktrace:

An unhandled exception has been thrown:
Error: Call to a member function file_get_contents() on null in /var/www/mycloud/lib/private/Files/Filesystem.php:719
Stack trace:
0 /var/www/mycloud/lib/private/legacy/image.php(597): OC\Files\Filesystem::file_get_contents('/var/www/myclou...')
1 /var/www/mycloud/lib/private/Preview/MP3.php(80): OC_Image->loadFromFile('/var/www/myclou...')
2 /var/www/mycloud/lib/private/Preview/MP3.php(64): OC\Preview\MP3->getNoCoverThumbnail()
3 /var/www/mycloud/lib/private/Preview/GeneratorHelper.php(54): OC\Preview\MP3->getThumbnail('/files_trashbin...', 256, 256, false, Object(OC\Files\View))
4 /var/www/mycloud/lib/private/Preview/Generator.php(155): OC\Preview\GeneratorHelper->getThumbnail(Object(OC\Preview\MP3), Object(OC\Files\Node\File), 256, 256)
5 /var/www/mycloud/lib/private/Preview/Generator.php(108): OC\Preview\Generator->getMaxPreview(Object(OC\Files\SimpleFS\SimpleFolder), Object(OC\Files\Node\File), 'audio/mpeg')
6 /var/www/mycloud/lib/private/PreviewManager.php(182): OC\Preview\Generator->getPreview(Object(OC\Files\Node\File), 32, 32, true, 'fill', 'audio/mpeg')
7 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(158): OC\PreviewManager->getPreview(Object(OC\Files\Node\File), 32, 32, true)
8 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(181): OCA\PreviewGenerator\Command\PreGenerate->processFile(Object(OC\Files\Node\File))
9 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
10 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
11 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
12 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
13 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
14 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
15 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(183): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
16 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(150): OCA\PreviewGenerator\Command\PreGenerate->processFolder(Object(OC\Files\Node\Folder))
17 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(113): OCA\PreviewGenerator\Command\PreGenerate->processRow(Array)
18 /var/www/mycloud/apps/previewgenerator/lib/Command/PreGenerate.php(91): OCA\PreviewGenerator\Command\PreGenerate->startProcessing()
19 /var/www/mycloud/3rdparty/symfony/console/Command/Command.php(256): OCA\PreviewGenerator\Command\PreGenerate->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
20 /var/www/mycloud/3rdparty/symfony/console/Application.php(818): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
21 /var/www/mycloud/3rdparty/symfony/console/Application.php(186): Symfony\Component\Console\Application->doRunCommand(Object(OCA\PreviewGenerator\Command\PreGenerate), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
22 /var/www/mycloud/3rdparty/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
23 /var/www/mycloud/lib/private/Console/Application.php(169): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
24 /var/www/mycloud/console.php(90): OC\Console\Application->run()
25 /var/www/mycloud/occ(11): require_once('/var/www/myclou...')

Nextcloud Box 'nextcloud.occ preview:pre-generate' ERROR

I'm getting this error when i give the 'nextcloud.occ preview:pre-generate' command:

ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.

Only generate sizes that is used in gallery and files app.

Hi,
It would be much needed to generate only the sizes that are really needed.
I checked a random image and it had 21 previews. No wonder there is over 6 million previews on my servers, and my oc_filecache table is 3.2GB.

I browsed some random folders in the gallery app, and it looks like the gallery app only uses previews with heigths like 256, 512 and 2048. The files app uses 32x32 for the smallest preview and the sidebar preview was 768x512. At least some of the 21 generated previews is not ever needed.

Any thoughts?

Preview Generator 1.0.1 → PDF file → occ preview:pre-generate does not display filename on warning

Instead of displaying the filename it is displaying the producer → no way to debug :-(

Here the PDF file in question

https://mega.nz/#!9A0E2BYA!9OG26578lm03HqaBHvqE8xtUPHWmwlWTAHeSfkizLfA

error message i am not able to understand

I downloaded and installed your app from the appstore inside NC 11RC1. Everything worked fine even the preview:pre-generate command worked perfectly. I tried the generate-all feature and that was the error message I got.

occ preview:generate-all

An unhandled exception has been thrown:
Error: Call to a member function getPath() on null in /var/www/magicbroccoli.de/nextcloud/lib/private/Preview/GeneratorHelper.php:72
Stack trace:
#0 /var/www/magicbroccoli.de/nextcloud/lib/private/Preview/GeneratorHelper.php(53): OC\Preview\GeneratorHelper->getViewAndPath(NULL)
#1 /var/www/magicbroccoli.de/nextcloud/lib/private/Preview/Generator.php(155): OC\Preview\GeneratorHelper->getThumbnail(Object(OC\Preview\MP3), Object(OC\Files\Node\File), 1024, 1024)
#2 /var/www/magicbroccoli.de/nextcloud/lib/private/Preview/Generator.php(108): OC\Preview\Generator->getMaxPreview(Object(OC\Files\SimpleFS\SimpleFolder), Object(OC\Files\Node\File), 'audio/mpeg')
#3 /var/www/magicbroccoli.de/nextcloud/lib/private/PreviewManager.php(182): OC\Preview\Generator->getPreview(Object(OC\Files\Node\File), 32, 32, true, 'fill', 'audio/mpeg')
#4 /var/www/magicbroccoli.de/nextcloud/apps/previewgenerator/lib/Command/Generate.php(167): OC\PreviewManager->getPreview(Object(OC\Files\Node\File), 32, 32, true)
#5 /var/www/magicbroccoli.de/nextcloud/apps/previewgenerator/lib/Command/Generate.php(155): OCA\PreviewGenerator\Command\Generate->parseFile(Object(OC\Files\Node\File))
#6 /var/www/magicbroccoli.de/nextcloud/apps/previewgenerator/lib/Command/Generate.php(153): OCA\PreviewGenerator\Command\Generate->parseFolder(Object(OC\Files\Node\Folder))
#7 /var/www/magicbroccoli.de/nextcloud/apps/previewgenerator/lib/Command/Generate.php(142): OCA\PreviewGenerator\Command\Generate->parseFolder(Object(OC\Files\Node\Folder))
#8 /var/www/magicbroccoli.de/nextcloud/apps/previewgenerator/lib/Command/Generate.php(96): OCA\PreviewGenerator\Command\Generate->generateUserPreviews(Object(OC\User\User))
#9 /var/www/magicbroccoli.de/nextcloud/lib/private/User/Manager.php(410): OCA\PreviewGenerator\Command\Generate->OCA\PreviewGenerator\Command\{closure}(Object(OC\User\User))
#10 /var/www/magicbroccoli.de/nextcloud/apps/previewgenerator/lib/Command/Generate.php(97): OC\User\Manager->callForSeenUsers(Object(Closure))
#11 /var/www/magicbroccoli.de/nextcloud/3rdparty/symfony/console/Command/Command.php(256): OCA\PreviewGenerator\Command\Generate->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/magicbroccoli.de/nextcloud/3rdparty/symfony/console/Application.php(818): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/magicbroccoli.de/nextcloud/3rdparty/symfony/console/Application.php(186): Symfony\Component\Console\Application->doRunCommand(Object(OCA\PreviewGenerator\Command\Generate), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/magicbroccoli.de/nextcloud/3rdparty/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/magicbroccoli.de/nextcloud/lib/private/Console/Application.php(169): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/magicbroccoli.de/nextcloud/console.php(90): OC\Console\Application->run()
#17 /var/www/magicbroccoli.de/nextcloud/occ(11): require_once('/var/www/magicb...')
#18 {main}

I am using Nextcloud 11RC1 on a debian mashine with apache2 and php7.0.14 installed.

./occ preview:generate-all [OC\Encryption\Exceptions\DecryptionFailedException] Fails

If i do the command ./occ preview:generate-all, i get this error:
[OC\Encryption\Exceptions\DecryptionFailedException]
Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.

I have deleted all shared links and encyption is all disabled. What can I do?

Nextcloud: "12.0.0.16"

{
    "id": "ocysxstxaaf8",
    "items": [
        [
            "server",
            "version",
            "12.0.0.16"
        ],
        [
            "server",
            "code",
            "other"
        ],
        [
            "server",
            "enable_avatars",
            "yes"
        ],
        [
            "server",
            "enable_previews",
            "yes"
        ],
        [
            "server",
            "memcache.local",
            "\\OC\\Memcache\\APCu"
        ],
        [
            "server",
            "memcache.distributed",
            "none"
        ],
        [
            "server",
            "asset-pipeline.enabled",
            "yes"
        ],
        [
            "server",
            "filelocking.enabled",
            "yes"
        ],
        [
            "server",
            "memcache.locking",
            "none"
        ],
        [
            "server",
            "debug",
            "no"
        ],
        [
            "server",
            "cron",
            "cron"
        ],
        [
            "php",
            "version",
            "7.1.4"
        ],
        [
            "php",
            "memory_limit",
            1073741824
        ],
        [
            "php",
            "max_execution_time",
            3600
        ],
        [
            "php",
            "upload_max_filesize",
            17179869184
        ],
        [
            "database",
            "type",
            "mysql"
        ],
        [
            "database",
            "version",
            "5.7.18"
        ],
        [
            "database",
            "size",
            417939456
        ],
        [
            "apps",
            "EagleEye",
            "disabled"
        ],
        [
            "apps",
            "activity",
            "2.5.1"
        ],
        [
            "apps",
            "activitylog",
            "disabled"
        ],
        [
            "apps",
            "audioplayer",
            "disabled"
        ],
        [
            "apps",
            "bookmarks",
            "disabled"
        ],
        [
            "apps",
            "calendar",
            "disabled"
        ],
        [
            "apps",
            "checksum",
            "disabled"
        ],
        [
            "apps",
            "comments",
            "1.2.0"
        ],
        [
            "apps",
            "contacts",
            "disabled"
        ],
        [
            "apps",
            "dav",
            "1.3.0"
        ],
        [
            "apps",
            "delete_confirmation",
            "disabled"
        ],
        [
            "apps",
            "documents",
            "disabled"
        ],
        [
            "apps",
            "encryption",
            "1.6.0"
        ],
        [
            "apps",
            "external",
            "2.0.0"
        ],
        [
            "apps",
            "federatedfilesharing",
            "1.2.0"
        ],
        [
            "apps",
            "federation",
            "1.2.0"
        ],
        [
            "apps",
            "files",
            "1.7.2"
        ],
        [
            "apps",
            "files_antivirus",
            "disabled"
        ],
        [
            "apps",
            "files_downloadactivity",
            "disabled"
        ],
        [
            "apps",
            "files_embeddedvideo",
            "disabled"
        ],
        [
            "apps",
            "files_epubviewer",
            "disabled"
        ],
        [
            "apps",
            "files_external",
            "1.3.0"
        ],
        [
            "apps",
            "files_locking",
            "disabled"
        ],
        [
            "apps",
            "files_mv",
            "disabled"
        ],
        [
            "apps",
            "files_pdfviewer",
            "1.1.1"
        ],
        [
            "apps",
            "files_reader",
            "1.0.4"
        ],
        [
            "apps",
            "files_sharing",
            "1.4.0"
        ],
        [
            "apps",
            "files_texteditor",
            "2.4.1"
        ],
        [
            "apps",
            "files_trashbin",
            "1.2.0"
        ],
        [
            "apps",
            "files_versions",
            "1.5.0"
        ],
        [
            "apps",
            "files_videojs-sublime",
            "disabled"
        ],
        [
            "apps",
            "files_videoplayer",
            "1.1.0"
        ],
        [
            "apps",
            "files_videoviewer",
            "disabled"
        ],
        [
            "apps",
            "files_videoviewerplus",
            "disabled"
        ],
        [
            "apps",
            "firstrunwizard",
            "2.1"
        ],
        [
            "apps",
            "flowupload",
            "disabled"
        ],
        [
            "apps",
            "folderplayer",
            "disabled"
        ],
        [
            "apps",
            "gallery",
            "17.0.0"
        ],
        [
            "apps",
            "galleryplus",
            "disabled"
        ],
        [
            "apps",
            "html5_videoplayer",
            "disabled"
        ],
        [
            "apps",
            "libreonline",
            "disabled"
        ],
        [
            "apps",
            "logreader",
            "2.0.0"
        ],
        [
            "apps",
            "lookup_server_connector",
            "1.0.0"
        ],
        [
            "apps",
            "mail",
            "disabled"
        ],
        [
            "apps",
            "music",
            "disabled"
        ],
        [
            "apps",
            "nextant",
            "disabled"
        ],
        [
            "apps",
            "nextcloud_announcements",
            "1.1"
        ],
        [
            "apps",
            "notifications",
            "2.0.0"
        ],
        [
            "apps",
            "ocdownloader",
            "disabled"
        ],
        [
            "apps",
            "ownbackup",
            "disabled"
        ],
        [
            "apps",
            "password_policy",
            "1.2.2"
        ],
        [
            "apps",
            "pinit",
            "disabled"
        ],
        [
            "apps",
            "previewgenerator",
            "1.0.6"
        ],
        [
            "apps",
            "provisioning_api",
            "1.2.0"
        ],
        [
            "apps",
            "rainloop",
            "disabled"
        ],
        [
            "apps",
            "richdocuments",
            "disabled"
        ],
        [
            "apps",
            "search_lucene",
            "disabled"
        ],
        [
            "apps",
            "serverinfo",
            "1.2.0"
        ],
        [
            "apps",
            "sharebymail",
            "1.2.0"
        ],
        [
            "apps",
            "storagecharts2",
            "disabled"
        ],
        [
            "apps",
            "survey_client",
            "1.0.0"
        ],
        [
            "apps",
            "systemtags",
            "1.2.0"
        ],
        [
            "apps",
            "templateeditor",
            "0.2"
        ],
        [
            "apps",
            "theming",
            "1.3.0"
        ],
        [
            "apps",
            "twofactor_backupcodes",
            "1.1.0"
        ],
        [
            "apps",
            "updatenotification",
            "1.2.0"
        ],
        [
            "apps",
            "updater",
            "disabled"
        ],
        [
            "apps",
            "user_external",
            "0.4"
        ],
        [
            "apps",
            "user_webdavauth",
            "disabled"
        ],
        [
            "apps",
            "workflowengine",
            "1.2.0"
        ],
        [
            "stats",
            "num_files",
            282365
        ],
        [
            "stats",
            "num_users",
            3
        ],
        [
            "stats",
            "num_storages",
            55
        ],
        [
            "stats",
            "num_storages_local",
            17
        ],
        [
            "stats",
            "num_storages_home",
            3
        ],
        [
            "stats",
            "num_storages_other",
            35
        ],
        [
            "stats",
            "num_comments",
            0
        ],
        [
            "stats",
            "num_comment_markers",
            0
        ],
        [
            "stats",
            "num_systemtags",
            0
        ],
        [
            "stats",
            "num_systemtags_mappings",
            0
        ],
        [
            "files_sharing",
            "num_shares",
            13
        ],
        [
            "files_sharing",
            "num_shares_user",
            0
        ],
        [
            "files_sharing",
            "num_shares_groups",
            0
        ],
        [
            "files_sharing",
            "num_shares_link",
            13
        ],
        [
            "files_sharing",
            "num_shares_link_no_password",
            13
        ],
        [
            "files_sharing",
            "num_fed_shares_sent",
            0
        ],
        [
            "files_sharing",
            "num_fed_shares_received",
            0
        ],
        [
            "files_sharing",
            "permissions_3_1",
            "10"
        ],
        [
            "files_sharing",
            "permissions_3_15",
            "3"
        ],
        [
            "encryption",
            "enabled",
            "no"
        ],
        [
            "encryption",
            "default_module",
            "yes"
        ]
    ]
}

root@snappy-nextcloud:/var/snap/nextcloud/current/nextcloud/extra-apps# nextcloud.occ preview:pre-generate
The process control (PCNTL) extensions are required in case you want to interrupt long running commands - see http://php.net/manual/en/book.pcntl.php

                                                                                                                   
  [OC\Encryption\Exceptions\DecryptionFailedException]                                                             
  Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with y  
  ou.                                                                                                              
                                                                                                                   

preview:pre-generate

Let me know if there's any other info I can send you, and thanks for your help.