stephensli/mavic

Mavic is a CLI application designed to download direct images found on selected reddit subreddits.

Hi, I'm Stephen, a passionate backend software engineer from the UK. I take great care in the architecture and code quality of the things I build. When I have the time, I am also an open-source enthusiast and maintainer, you can find me dabbling with projects throughout Github.

---

Cars (Compile & Run Sandbox) 🚙 - stephensli/cars

This is currently the main project I'm working on when I have some spare time. Cars is a sandbox environment to allow execution of untrusted code with fixed time, and memory with support for output testing. It runs on all the major queue services, and data stores with a focus on container execution.

mavic's Issues

Use fan-in, fan-out for image downloading

Support Paging Past 100

Is your feature request related to a problem? Please describe.
Currently the limit of images per sub-reddit is 100 due to the code not paging past past 1 page which can be at mast 100 items.

Describe the solution you'd like
Support paging past the 100 page limit on the subreddit.

Describe alternatives you've considered
N/A

Additional context
N/A

Execute all subreddits concurrently, currently its done sequential

Fix Race Condition

Describe the bug
There is a data race condition within the progress bar adjustment of the max value. This is because the total is always increased if the total number of images increase, but all metadata is downloaded at once which would result in a bad total.

To Reproduce
Steps to reproduce the behavior:
Run the application with the race flag.

example: go run -race cmd/mavic/main.go -l 10 -p --output ./pictures -f cute pics meme

Expected behavior
There should be no data race conditions regarding the process bar otherwise it could produce mixed results in the expected output of the progression.

Additional Information

WARNING: DATA RACE
Read at 0x00c0002120b0 by goroutine 12:
  github.com/schollz/progressbar/v3.(*ProgressBar).GetMax()
      /home/stephen.lineker-miller/source_personal/Mavic/vendor/github.com/schollz/progressbar/v3/progressbar.go:528 +0x2d9
  github.com/mambadev/mavic/internal/scraper.Scraper.downloadMetadata()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:206 +0x1b5
  github.com/mambadev/mavic/internal/scraper.Scraper.downloadRedditMetadata·dwrap·4()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:227 +0xd0

Previous write at 0x00c0002120b0 by goroutine 9:
  github.com/schollz/progressbar/v3.(*ProgressBar).ChangeMax64()
      /home/stephen.lineker-miller/source_personal/Mavic/vendor/github.com/schollz/progressbar/v3/progressbar.go:548 +0x31a
  github.com/schollz/progressbar/v3.(*ProgressBar).ChangeMax()
      /home/stephen.lineker-miller/source_personal/Mavic/vendor/github.com/schollz/progressbar/v3/progressbar.go:540 +0x30e
  github.com/mambadev/mavic/internal/scraper.Scraper.downloadMetadata()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:206 +0x1b5
  github.com/mambadev/mavic/internal/scraper.Scraper.downloadRedditMetadata·dwrap·4()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:227 +0xd0

Goroutine 12 (running) created at:
  github.com/mambadev/mavic/internal/scraper.Scraper.downloadRedditMetadata()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:227 +0x216
  github.com/mambadev/mavic/internal/scraper.Scraper.Start·dwrap·1()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:90 +0x97

Goroutine 9 (finished) created at:
  github.com/mambadev/mavic/internal/scraper.Scraper.downloadRedditMetadata()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:227 +0x216
  github.com/mambadev/mavic/internal/scraper.Scraper.Start·dwrap·1()
      /home/stephen.lineker-miller/source_personal/Mavic/internal/scraper/scraper.go:90 +0x97

Desktop (please complete the following information):

• OS: Linux
• Version: 0.2.0