Comments (6)
Just an addendum to "additional context": something that would make me use goscrape over WebScrapBook would be the possibility to run it locally and download the pages in batch instead of one per one.
Thanks for the feature request, cookie support and a new version will be released tomorrow. For the batching please create a new feature request with a desired usage example
from goscrape.
Just an addendum to "additional context": something that would make me use goscrape over WebScrapBook would be the possibility to run it locally and download the pages in batch instead of one per one.
from goscrape.
Just an addendum to "additional context": something that would make me use goscrape over WebScrapBook would be the possibility to run it locally and download the pages in batch instead of one per one.
Thanks for the feature request, cookie support and a new version will be released tomorrow. For the batching please create a new feature request with a desired usage example
Well, unless you think it would be cool to have it, I don't think a built-in batch function would be needed right now, since one can use a foreach loop (or the apply
command) for iterating over URLs.
I'm really busy lately, but I will try to sketch something like a built-in batch function that reads from a text file if you guys are willing to.
from goscrape.
By the way, it would be great if a list of embed streams (video and/or audio) could be generated and kept as a .txt indicating which name can one download locally to replace in the backup. @cornelk
from goscrape.
By the way, it would be great if a list of embed streams (video and/or audio) could be generated and kept as a .txt indicating which name can one download locally to replace in the backup. @cornelk
one issues per feature/bug report please otherwise this will get lost the moment this issue will be closed
from goscrape.
one issues per feature/bug report please otherwise this will get lost the moment this issue will be closed
Sure, I will be opening it right now.
from goscrape.
Related Issues (20)
- Add serving of downloaded website by a http server
- how to set cookie
- how to set cookie
- Continue scraping after a 404 error
- Handle trailing slash at end as duplicate by default
- Filter fragments at the end of URLs
- error on install HOT 4
- Basic Auth Not Working HOT 3
- Not grabbing all images HOT 3
- Failure to scraping data URI HOT 1
- any plan to keep this uptodate? HOT 2
- Attachment(s) download HOT 7
- Handle img scrset attributes
- Inline CSS not parsed
- background attribute on body tag is not handled
- Runs out of memory on big scrapes
- Avoid too long filenames
- Create a text file containing references of media in the scrapped website and the filename that one can use to replace it locally
- HTML style attributes are not parsed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goscrape.