0x416c616e/clone_all

Clone all repos from a given GitHub user

clone_all's Issues

account for 404

fix loop bug

Instead of downloading page 1, page 2, page 3, page 4, etc. it downloads page 1, then page 2, then page 1, then page 2, etc. indefinitely and doesn't break out of the loop either.

Make a "download all" option

Maybe keep it in this repo, maybe make a separate repo. But make it so someone can just download repos instead of cloning them.

11 add reset functionality

maybe instead of reset.sh, I can add reset functionality to the clone_all.py program, which will be a command line argument, like so:

clone_all.py --reset

it will delete any html files in html/, subdirectories in the repos/ folder, etc.

4. parse html and look for no repo string

"doesn’t have any public repositories yet."

7. if #6 string is within the HTML file, proceed with getting next page

Stub: print "there is a next page"

copy this repo's asio.py to the asio repo

and when the clone_all program is more or less finished, then pin this issue and make it known that the asio repo is where all future asio.py changes will be made, so this repo's asio.py will be downstream, not upstream

12 use the github repo links to clone all the repos to the repos/ directory

test asio.py functions

don't need to finish all the to-do stuff, but you do need to make sure the existing things work, and then copy the asio.py from this repo into the asio repo

fix UnicodeDecodeError when parsing emojis etc in git repo pages

maybe change file open to utf-8

might want to eventually make separate search functions in asio for utf8, ascii, and binary

6. parse HTML file and look for "next repo" string (which may or may not be present)

"&tab=repositories">Next"

make issues for rest of design doc

get username from command line arg

3. use requests to download the first repo file in html/ subfolder

delete all repo_page_X.html from html after cloning all repos

cleans it up a bit

not related to the reset script, which deletes repos in addition to html pages

the html pages are really only temporarily needed, so there's no point in keeping them after all the links are enumerated and then the repos are cloned

5. if #4 string is not found, exit program

it means there are no repos to clone

10 find repo links from html

after all repo pages have been downloaded, like repo_page_1.html, repo_page_2.html, repo_page_3.html, etc. then parse them again and search for the links to the repos

use a stack, push link strings onto the stack when enumerating, and pop them off when cloning (in a later step, not now)

2. build URL for repo page

9. if there are next pages, download them all and loop