Comments (7)
here is a really rough pass. it errors on some modules, but it processes enough to do some testing.
https://github.com/JamesHabben/iLEAPP/tree/dynamicreport-dataarrays
the health - heart rate
module gets 23.5k records from josh's public image. previous html file was around 10-11mb. using this branch, that file cuts down to around 6mb. on the larger file, my browser was timing out with all the data. with this branch, i get 12 loops of the timer circle, and the data is all loaded. once it loads, the pages are actually quite responsive when moving around the pages. its also quite quick when using the sorts. its just the initial load that takes some time.
@Johann-PLW can you try this against your larger data set to see if it helps?
from ileapp.
i am less concerned about the size of the html file itself. this is more about the browser being able to load and work with the large amount of data. we could do some other things to do some file reduction, but doing something like gzip compression might actually add overhead to the browser making it worse. if this branch code doesnt work to load all of your large data set, we may need to explore other approaches of breaking that data up into segments.
from ileapp.
If this branch code doesnt work to load all of your large data set, we may need to explore other approaches of breaking that data up into segments.
That's something I've actually been thinking about, perhaps grouping a large amount of data by year, day, month or hour.
With the work you've already done on the HTML report, we could display less info on the screen and additional details that are redundant in the form of a tooltip.
I'll do some tests with my dataset with the code of your 'dynamicreport-dataarray' branch and let you know.
from ileapp.
@JamesHabben
Unfortunately, it doesn't work with my personal dataset (encrypted backup of an iPhone 13 mini with iOS 17.0.3)
The heart rate query matches 1028115 records.
The previous generated HTML file was 240.9 MB, the new one (with your updated code) is 235.1 MB.
The web browsers (Safari & Chrome) are still unresponsive after 10 minutes trying to load the data.
The number of steps query matches 493272 records.
The previous generated HTML file was 66.7 MB, the new one is bigger with 81.5 MB
Both web browsers are unresponsive.
Tests were conducted on a MacBook Pro 2019 - 2,4 GHz Intel Core i9 8 cores - 32 GB RAM with macOS 13.5.1
Web browsers are Safari 6.6 (18615.3.12.11.2) et Google Chrome 119.0.6045.159
from ileapp.
oof. not sure why heart rate didnt reduce more, and frustrated at steps increase. i can reduce some of that using less text in the structure, but i dont think that will make much different in the browser loading this data set. what do you think about sampling the data on the python side? 1mil records is a lot of data and will be hard to incorporate processing it in a broader framework like this. i wonder if we can find a framework that can do some time based sampling, averaging, anomoly highlight, and pass a reduced set of data to the browser.
from ileapp.
@Johann-PLW What's the time range and frequency of your heart beat data? If we did some summary of data, say every 15 mins, how many records would that reduce? Might have to adjust based on the frequency.
We can provide typical summary numbers like minimum, maximum, average, mean, etc. and if the user wants to investigate in more detail, then TSV output is available.
While typing this though, I wanted to do some math. I think hourly summary periods really might need to be the one.
Here are my calcs:
1. Hourly Records:
• 1 record per hour
• 24 records per day
• In 3 years: = 26,280 records
• In 5 years: = 43,800 records
• In 10 years: = 87,600 records
2. Half-hourly Records:
• 2 records per hour
• 48 records per day
• In 3 years: = 52,560 records
• In 5 years: = 87,600 records
• In 10 years: = 175,200 records
3. Every 15 Minutes:
• 4 records per hour
• 96 records per day
• In 3 years: = 105,120 records
• In 5 years: = 175,200 records
• In 10 years: = 350,400 records
from ileapp.
@JamesHabben
I have records since April 2015 and the frequency depends of my activity:
- up to 15 per minute
- 10 ~ 150 per hour
- 100 ~ 1000 per day
- 10k ~ 20k per month
I think we could also remove some columns like Device and Manufacturer
![image](https://private-user-images.githubusercontent.com/27498476/283207149-bcad747d-e170-4516-91c9-b630e3b4721a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTE1MjEwNTYsIm5iZiI6MTcxMTUyMDc1NiwicGF0aCI6Ii8yNzQ5ODQ3Ni8yODMyMDcxNDktYmNhZDc0N2QtZTE3MC00NTE2LTkxYzktYjYzMGUzYjQ3MjFhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzI3VDA2MjU1NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWIwODdlOWYzOGM2MWRjYmYwM2Y3YmVmOGFhZTQ5NjJlMWJhMWM0NzIwODUwNzhhY2VhM2RhMTU2ZDIxNzhlYzcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Q721wrOb1exoH6W6vRjw04M1PIN4zy4tp3op_lH5lRs)
As the device and/or software used to collect the data, and the timezone are very repetitive, could we use an array to store once the information and display it in all records?
from ileapp.
Related Issues (20)
- python-magic-bin not available for ARM Mac HOT 10
- Artifact Info Structure Update HOT 15
- Refactor Report Icons HOT 2
- iLEAPP GUI Reference to ALEAPP HOT 4
- Datetime Import HOT 50
- Apple Watch support HOT 1
- photosDbedif Issue HOT 9
- Dynamic Report Data Types HOT 33
- Artifact Report File Name
- Alarms Module Updates HOT 2
- iTunesBackupInfo Module Search Pattern HOT 5
- protonMail Module Search Pattern HOT 1
- telegramMessages Module Search Pattern HOT 1
- ilap_artifacts.py HOT 1
- lastbuild module not executed first in the CLI HOT 25
- GUI and iTunes Backup Assumption HOT 12
- Column order issue with Sync.db parser HOT 3
- GUI Plugin Scroll HOT 7
- Mac and Linux builds HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ileapp.