bestform/souparchive

SOUP ARCHIVE

Usage:

./souparchive -user YOURUSERNAME

This will save all the entries in your soup.io rss feed in the archive folder.

Subsequent calls will remember already saved items so you can run this script as a cron job to continiously archive your soup.io feed.

Keep in mind that at its current state the script will not keep track of the ordering of the files. Nor will it save two different files with the same name as different entries. (PRs welcome)

souparchive's Issues

Source points to a local folder called github.com

Steps to reproduce:

• clone github repo to /opt/souparchive
• cd to /opt/souparchive
• make freebsd

Expected result:

souparchive builds

Actual result:
main.go:12:2: cannot find package "github.com/bestform/souparchive/db" in any of: /usr/local/go/src/github.com/bestform/souparchive/db (from $GOROOT) ($GOPATH not set) main.go:13:2: cannot find package "github.com/bestform/souparchive/feed" in any of: /usr/local/go/src/github.com/bestform/souparchive/feed (from $GOROOT) ($GOPATH not set) main.go:14:2: cannot find package "github.com/bestform/souparchive/fetch" in any of: /usr/local/go/src/github.com/bestform/souparchive/fetch (from $GOROOT) ($GOPATH not set) main.go:15:2: cannot find package "github.com/bestform/souparchive/host" in any of: /usr/local/go/src/github.com/bestform/souparchive/host (from $GOROOT) ($GOPATH not set)

Creating a symlink there as a workaround made it build successfully as far as I could see

Archiver should check for existence of both guid and actual file

Steps to Reproduce:

• Archive a soup
• Delete some of the downloaded files
• Archive the soup again

Expected Result:

• The missing files are retrieved again

Actual Result:

• Missing files are not fetched again since they are still referenced in the guids.json
• Deleting the guids.json leads to re-fetching all images no matter if they exist already

SIGSEGV everytime when trying to archive soup

Steps to reproduce

./souparchive_freebsd_64 -user proligde

Expected result:

Soup gets archived

Actual result:

`
Saving http://asset-4.soupcdn.com/asset/12328/1566_40b7.gif...
Saving http://asset-c.soupcdn.com/asset/12308/5581_c86b_400.jpeg...
Saving http://asset-2.soupcdn.com/asset/12328/3610_2d2c_400.jpeg...
Saving http://asset-5.soupcdn.com/asset/12327/4447_54b0_400.jpeg...
Saving http://asset-e.soupcdn.com/asset/12310/7497_ebed_400.jpeg...
Saving http://asset-f.soupcdn.com/asset/12308/2904_f7d5_400.jpeg...
Saving http://asset-d.soupcdn.com/asset/12327/5523_db23_400.jpeg...
Saving http://asset-4.soupcdn.com/asset/12094/7593_4a15_400.gif...
Saving http://asset-8.soupcdn.com/asset/12036/6798_8674_400.jpeg...
Saving http://asset-4.soupcdn.com/asset/12311/4145_49ca_400.gif...
Saving http://asset-9.soupcdn.com/asset/12327/4430_9c9c.gif...
Saving http://asset-d.soupcdn.com/asset/12022/2025_da54_400.gif...
Saving http://asset-6.soupcdn.com/asset/12269/9412_60e8_400.png...
Saving ...
Saving http://asset-a.soupcdn.com/asset/12312/0882_a4fe_400.jpeg...
Saving http://asset-b.soupcdn.com/asset/12327/8636_bdf1_400.jpeg...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x4d7176]

goroutine 14 [running]:
panic(0x662640, 0xc4200120d0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/bestform/souparchive/fetch.(*defaultHttpClient).Get(0x7fc348, 0x0, 0x0, 0xc4201a2c00, 0x8, 0xb)
/usr/local/go/src/github.com/bestform/souparchive/fetch/fetch.go:34 +0x76
github.com/bestform/souparchive/fetch.Fetch(0x0, 0x0, 0x0, 0x0, 0xc4200568c0, 0x49, 0xc42015e7a0, 0x1b, 0xc4200a4800, 0x16c7, ...)
/usr/local/go/src/github.com/bestform/souparchive/fetch/fetch.go:67 +0x189
main.main.func1(0xc42019a6b0, 0x0, 0x0, 0x0, 0x0, 0xc4200568c0, 0x49, 0xc42015e7a0, 0x1b, 0xc4200a4800, ...)
/opt/souparchive/main.go:77 +0x13e
created by main.main
/opt/souparchive/main.go:83 +0x540
`

File retriever should check for valid HTTP status code

Steps to Reproduce:

• Archive your soup

Expected Result:

• All images are saved

Actual Result:

• More than 50% of the images seem corrupted and can't be opened.

Note:

The files (in this case images) are in fact not corrupted but contain a HTML page stating that there was an error. They seem to be sent with a HTTP Status Code 5xx (502 in this case) that should prevent the retriever from persisting the response to the file system.