emranbm/sls-for-iot

Master of Science Project, at Sharif University of Technology

Shared Local data Storage for IoT

A design for IoT devices to share their storage across a local network
It's my Master of Science project at Sharif university of technology.

Roadmap

Here's my roadmap to get the project done. It's not fixed and is subject to change if required.

• Define the project
• Get the acceptance of the proposal
• Create a dockerized simulation environment
• Design and semi-implement the shared storage
  • Save file
  • Read file
  • Delete file
  • List files
• Test and Benchmark
  • Run FogBed simulation without SLS and note metrics (HDD usage, CPU usage, Memory usage, etc.)
    • Design and implement measurement script.
  • Run FogBed simulation with different scenarios of using SLS
• Revise the design and implementation, if needed
• Write down the paper
• Create presentation
• Defend the dissertation
• Enjoy the rest of your non-academic life!
  (Is it going to check ever? I hope so!)

Assumptions and Relaxed Constraints

For the sake of simplicity and staying minimal, some assumptions are made by design. Here is a list of them.

1. Files are immutable. i.e. it's not possible to rewrite a file content; instead, it should be deleted first and created back with the new contents.
2. The order of messages (of a specific type) sent from a single source to a single destination is guaranteed.
3. Files don't save in sub-directories but in the root of each client's (virtual) storage. i.e. it's not possible to save a file somewhere like sub/dir/file.txt, but file.txt. (See issue #7 for more details)

Enhancement Opportunities

There are too many (if not infinite) cases to get optimized. Here is a memory of ones faced.

1. Optimize protocol: More efficient serialization protocols can be used instead of JSON. e.g. protobuf
2. Compress contents: File contents can be compressed before transmission.
   But it should be tested whether it's worth doing computation over IO.
3. Optimize heartbeats: Consider any message from clients as a heartbeat. So the client can send the heartbeats less frequently.
4. Don't keep content in memory: Currently, the SDK holds file contents in memory before saving. It should also support other forms like file handles, etc.
5. Remove SLS manager (Done!): SLS manager can be omitted! Its only usage is the clients' book-keeping, which can be handled between the clients themselves. i.e. clients can subscribe for the heartbeats in a general channel and keep track of the others health.
6. Optimize local operations: Don't loop back content through the network when it should be handled locally.
7. Disk-based FileInfoRepo: Keep file infos in disk instead of memory.

Future Work

1. Provide file overwrite.
2. Allow using sub-directories (#7)
3. Add ability to divide huge files into chunks to be saved in multiple devices.
4. Use expireAt field to enhance the usage of shared storage as a cache.

sls-for-iot's Issues

Make build and acceptance test parallel

For the sake of faster CI feedback, it's good to make the build and acceptance tests parallel since they seem technically independent.

Support for saving files to non-existing directories.

Currently, the file path can't have a parent directory (e.g. some/where/file.txt) and it's forced to be on the root (e.g. file.txt). And worse, the sdk doesn't raise an appropriate error on such situation and just hangs on saving.

Ability to pass a file to client to save

It should be possible to pass a file to the client to be saved in SLS.

Don't keep unused file content in memory

Sometimes it has happened that a FileObject is up-casted to FileInfo; in such situations, the FileObject.content is absolutely not needed but is kept in memory in the lifetime of that FileInfo, even if FileInfo has no content field.
There should be a solution to handle such situations and leave the content alone to be garbage-collected.

Replace 'diskusage' package

diskusage package has a non deterministic behavior when calling diskusage.check("some-path/"); it sometimes randomly rejects with the below error:

Error: ENOENT, No such file or directory '././clients-storages/main'
    at check (/home/emran/dev/sls-for-iot/simulation/sls-sdk/node_modules/diskusage/index.js:23:21)
    at /home/emran/dev/sls-for-iot/simulation/sls-sdk/node_modules/diskusage/index.js:10:5
    at new Promise (<anonymous>)
    at Object.exports.check (/home/emran/dev/sls-for-iot/simulation/sls-sdk/node_modules/diskusage/index.js:9:10)
    at Function.<anonymous> (/home/emran/dev/sls-for-iot/simulation/sls-sdk/src/Utils.ts:6:46)
    at Generator.next (<anonymous>)
    at /home/emran/dev/sls-for-iot/simulation/sls-sdk/src/Utils.ts:8:71
    at new Promise (<anonymous>)
    at __awaiter (/home/emran/dev/sls-for-iot/simulation/sls-sdk/src/Utils.ts:4:12)
    at Function.getSelfClientInfo (/home/emran/dev/sls-for-iot/simulation/sls-sdk/src/Utils.ts:16:16)
    at AutoSyncedClientRepo.<anonymous> (/home/emran/dev/sls-for-iot/simulation/sls-sdk/src/AutoSyncedClientRepo.ts:35:46)
    at Generator.next (<anonymous>)
    at /home/emran/dev/sls-for-iot/simulation/sls-sdk/src/AutoSyncedClientRepo.ts:8:71
    at new Promise (<anonymous>)
    at __awaiter (/home/emran/dev/sls-for-iot/simulation/sls-sdk/src/AutoSyncedClientRepo.ts:4:12)
    at AutoSyncedClientRepo.handlePing (/home/emran/dev/sls-for-iot/simulation/sls-sdk/src/AutoSyncedClientRepo.ts:37:16)
    at Timeout._onTimeout (/home/emran/dev/sls-for-iot/simulation/sls-shared-utils/MqttSubscribeManager.ts:30:21)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7) {
  errno: 2,
  code: 'ENOENT',
  path: '././clients-storages/main',
  syscall: 'statvfs'
}

It's better to use check-disk-space package, instead.

Rename 'name' in FileObj

Rename name field of FileObj to path.

Use logger in other sls-shared-utils component

Use logger in the sls-shared-utils component as well as the sls-sdk.