AppImage very large

[deltachat]

AppImage very large

[ralphtheninja]

We're down to 56Mb now, so I think this can be closed. We could spend endless time to optimize more but I don't think it will give us that much more bang for the buck.

https://github.com/deltachat/deltachat-desktop/releases/tag/v1.2.2

[okdistribute]

Here’s a PR on another project that reduces build size using electron-builder without two package json (look at package.json) https://github.com/cabal-club/cabal-desktop/pull/85/files

[ralphtheninja]

I suspect this is due to node_modules/ containing development modules. We might want to split up the app using the Two package.json structure. Simply put, move everything related to the app into an app folder and keep all development related dependencies in the root package.json.

[ralphtheninja]

@Karissa Do you think the two package.json structure is the way to go? I could take a stab at this.

[okdistribute]

I have never tried that — it seems like it should be unecessary to do that perhaps it’s the easiest to implement. I hear people complain about the two package json structure and there could be another way..

[ralphtheninja]

If I understand this correctly you need to use yarn? https://yarnpkg.com/lang/en/docs/selective-version-resolutions/

I hear people complain about the two package json structure

What are they complaining about?

[ralphtheninja]

I spent some time investigating what's causing this size. Below are my findings:

Prepare

Skip generating .deb file to increase speed of test. Also it's not relevant to the test.

I did a quick check on the folders and they are fairly small:

$ du -sh src/ es5/ build/ bin/ images/ _locales/ static/
144K    src/
164K    es5/
64K     build/
16K     bin/
300K    images/
212K    _locales/
2,0M    static/

It must be the node_modules/ folder that's causing the big size.

test command

I used the following command for testing size.

rm -rf node_modules/ && npm i && rm -rf dist/ && npm run dist; ls -al dist/*.AppImage

The last ; is because npm run dist fails due to missing github credentials.

devDependencies

First lets try if it's a problem with devDependencies being included.

0. size of .AppImage file: 214518973 bytes
1. remove standard -> 214518649 bytes (-324 bytes)
2. remove nodemon -> 214518789 bytes (+140 bytes)
3. remove babel-eslint -> 214519372 bytes (+583 bytes)
4. remove buble (install globally instead) -> 214518986 bytes (-386 bytes)
5. remove depcheck (not used) -> 214518885 bytes (+/- whatever bytes)

I think it's fairly clear that devDependencies isn't affecting the resulting image at all, since removing some dependencies adds size to the resulting binary. Also, this is in the magnitude of hundreds of bytes, which is uninteresting. Apparently electron-builder handles this in a good way.

Lets move on to dependencies.

dependencies

Lets also revert all devDependencies and start from scratch.

0. size of .AppImage file: 214518973
1. remove deltachat-node -> 200066198 (-14452725), apparently has a bit of effect but "only" 14Mb, there must be something else taking up more space (we might be able to reduce this size as well)
2. remove react -> 200065813 (-385), i.e. 0 effect
3. remove react-dom -> 200066045 (+232), i.e. 0 effect
4. remove react-intl -> 198946736 (-1119309), 1Mb so not a lot
5. remove run-parallel -> 198942483 (-4253), hardly noticable effect
6. remove lodash.merge -> 198938535 (-3948), hardly noticable effect
7. remove intl-tel-input -> 198491122 (-447413), ~450k so not a lot
8. remove insert-css -> 198491286 (+164), i.e. 0 effect
9. remove emoji-panel -> 165022652 (-33468634), ~32Mb so quite a bit, but still a lot to go (this should/could be optimized)
10. remove debounce -> 165017952 (-4700), hardly noticable effect
11. remove conversations -> (errors due to missing react and react-dom so putting them back) -> 54294609, so roughly 110Mb effect (highest prio)
12. remove application-config -> 54290421 (-4188), hardly noticable effect
13. remove @blueprint/core -> 51215169 (-3075252), ~3Mb so not a lot
14. remove arch -> 51211202 (-3967), hardly noticable effect

conclusion

We should look further into optimizing the following dependencies and see what's causing this. And in this order.

1. conversations (110Mb)
2. emoji-panel (32Mb)
3. deltachat-node (14Mb)

[ralphtheninja]

@Karissa Side note: I went through the cabal-desktop process and it's not really different from ours, since I only used npm to build the cabal desktop app and still produces a good sized binary, which lead me to this dependency quest to get more information.

[ralphtheninja]

Jumped into conversations a bit.

$ npm i --production
$ du -sh node_modules/
121M    node_modules/

Used Disk Usage Analyzer:

It seems the emoji-datasource module is the culprit here.

Checking conversations where this is used:

$ git grep -n -C3 emoji-datasource`

So, it seems conversations isn't using emoji-datasource directly. How about indirectly? (ag is the_silver_searcher)

$ ag -a emoji-datasource

There seems to be no usage of emoji-datasource whatsover. But conversations depend on emoji-js. So somewhere around here we should make a decision. Maybe fork emoji-js and just remove all that crap? :)

It's basically just a bunch of .png files:

$ ls -al node_modules/emoji-datasource/
total 108992
drwxr-xr-x  2 lms lms     4096 sep 29 17:59 .
drwxr-xr-x 24 lms lms     4096 sep 29 17:59 ..
-rw-r--r--  1 lms lms     3445 nov 17  2017 CHANGES.md
-rw-r--r--  1 lms lms   983922 nov 17  2017 emoji.json
-rw-r--r--  1 lms lms  1629054 nov 17  2017 emoji_pretty.json
-rw-r--r--  1 lms lms     1080 apr  6  2017 LICENSE
-rw-r--r--  1 lms lms     2042 sep 29 17:59 package.json
-rw-r--r--  1 lms lms     4832 nov 17  2017 README.md
-rw-r--r--  1 lms lms  1611272 nov 17  2017 sheet_apple_16.png
-rw-r--r--  1 lms lms  2287847 nov 17  2017 sheet_apple_20.png
-rw-r--r--  1 lms lms  4774502 nov 17  2017 sheet_apple_32.png
-rw-r--r--  1 lms lms 13693323 nov 17  2017 sheet_apple_64.png
-rw-r--r--  1 lms lms  1460924 nov 17  2017 sheet_emojione_16.png
-rw-r--r--  1 lms lms  2038219 nov 17  2017 sheet_emojione_20.png
-rw-r--r--  1 lms lms  4025880 nov 17  2017 sheet_emojione_32.png
-rw-r--r--  1 lms lms  8962564 nov 17  2017 sheet_emojione_64.png
-rw-r--r--  1 lms lms  1500486 nov 17  2017 sheet_facebook_16.png
-rw-r--r--  1 lms lms  2086626 nov 17  2017 sheet_facebook_20.png
-rw-r--r--  1 lms lms  4169966 nov 17  2017 sheet_facebook_32.png
-rw-r--r--  1 lms lms 10785774 nov 17  2017 sheet_facebook_64.png
-rw-r--r--  1 lms lms  1454327 nov 17  2017 sheet_google_16.png
-rw-r--r--  1 lms lms  2008386 nov 17  2017 sheet_google_20.png
-rw-r--r--  1 lms lms  3843575 nov 17  2017 sheet_google_32.png
-rw-r--r--  1 lms lms  9462647 nov 17  2017 sheet_google_64.png
-rw-r--r--  1 lms lms  1559060 nov 17  2017 sheet_messenger_16.png
-rw-r--r--  1 lms lms  2196115 nov 17  2017 sheet_messenger_20.png
-rw-r--r--  1 lms lms  4503752 nov 17  2017 sheet_messenger_32.png
-rw-r--r--  1 lms lms 12502765 nov 17  2017 sheet_messenger_64.png
-rw-r--r--  1 lms lms  1274797 nov 17  2017 sheet_twitter_16.png
-rw-r--r--  1 lms lms  1733472 nov 17  2017 sheet_twitter_20.png
-rw-r--r--  1 lms lms  3268949 nov 17  2017 sheet_twitter_32.png
-rw-r--r--  1 lms lms  7693547 nov 17  2017 sheet_twitter_64.png

[hpk42]

interesting analysis, thanks. The signal/desktop for Mac btw is 92MB -- maybe that's indicate of where we could get to? And maybe their build-process reveals optimization potential?

…

On Sat, Sep 29, 2018 at 09:34 -0700, Lars-Magnus Skog wrote: Jumped into `conversations` a bit. ``` $ npm i --production $ du -sh node_modules/ 121M node_modules/ ``` Used `Disk Usage Analyzer`:  It seems the `emoji-datasource` module is the culprit here. Checking `conversations` where this is used: ``` $ git grep -n -C3 emoji-datasource` ```  So, it seems `conversations` isn't using `emoji-datasource` _directly_. How about indirectly? (`ag` is `the_silver_searcher`) ``` $ ag -a emoji-datasource ```  There seems to be _no_ usage of `emoji-datasource` whatsover. But `conversations` depend on `emoji-js`. So somewhere around here we should make a decision. Maybe fork `emoji-js` and just remove all that crap? :) It's basically just a bunch of `.png` files: ``` $ ls -al node_modules/emoji-datasource/ total 108992 drwxr-xr-x 2 lms lms 4096 sep 29 17:59 . drwxr-xr-x 24 lms lms 4096 sep 29 17:59 .. -rw-r--r-- 1 lms lms 3445 nov 17 2017 CHANGES.md -rw-r--r-- 1 lms lms 983922 nov 17 2017 emoji.json -rw-r--r-- 1 lms lms 1629054 nov 17 2017 emoji_pretty.json -rw-r--r-- 1 lms lms 1080 apr 6 2017 LICENSE -rw-r--r-- 1 lms lms 2042 sep 29 17:59 package.json -rw-r--r-- 1 lms lms 4832 nov 17 2017 README.md -rw-r--r-- 1 lms lms 1611272 nov 17 2017 sheet_apple_16.png -rw-r--r-- 1 lms lms 2287847 nov 17 2017 sheet_apple_20.png -rw-r--r-- 1 lms lms 4774502 nov 17 2017 sheet_apple_32.png -rw-r--r-- 1 lms lms 13693323 nov 17 2017 sheet_apple_64.png -rw-r--r-- 1 lms lms 1460924 nov 17 2017 sheet_emojione_16.png -rw-r--r-- 1 lms lms 2038219 nov 17 2017 sheet_emojione_20.png -rw-r--r-- 1 lms lms 4025880 nov 17 2017 sheet_emojione_32.png -rw-r--r-- 1 lms lms 8962564 nov 17 2017 sheet_emojione_64.png -rw-r--r-- 1 lms lms 1500486 nov 17 2017 sheet_facebook_16.png -rw-r--r-- 1 lms lms 2086626 nov 17 2017 sheet_facebook_20.png -rw-r--r-- 1 lms lms 4169966 nov 17 2017 sheet_facebook_32.png -rw-r--r-- 1 lms lms 10785774 nov 17 2017 sheet_facebook_64.png -rw-r--r-- 1 lms lms 1454327 nov 17 2017 sheet_google_16.png -rw-r--r-- 1 lms lms 2008386 nov 17 2017 sheet_google_20.png -rw-r--r-- 1 lms lms 3843575 nov 17 2017 sheet_google_32.png -rw-r--r-- 1 lms lms 9462647 nov 17 2017 sheet_google_64.png -rw-r--r-- 1 lms lms 1559060 nov 17 2017 sheet_messenger_16.png -rw-r--r-- 1 lms lms 2196115 nov 17 2017 sheet_messenger_20.png -rw-r--r-- 1 lms lms 4503752 nov 17 2017 sheet_messenger_32.png -rw-r--r-- 1 lms lms 12502765 nov 17 2017 sheet_messenger_64.png -rw-r--r-- 1 lms lms 1274797 nov 17 2017 sheet_twitter_16.png -rw-r--r-- 1 lms lms 1733472 nov 17 2017 sheet_twitter_20.png -rw-r--r-- 1 lms lms 3268949 nov 17 2017 sheet_twitter_32.png -rw-r--r-- 1 lms lms 7693547 nov 17 2017 sheet_twitter_64.png ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #41 (comment)

[ralphtheninja]

@hpk42 We can go much lower than that if we tweak these dependencies a bit. Around 50-60Mb should be doable.

[hpk42]

great!

[r10s]

very cool :)