Comments (4)
I wonder if adding a check for a zero size file in the download_file
function will skip the directory objects. It seems like they will still be created by this line dest_file_path.parent.mkdir(parents=True, exist_ok=True)
Perhaps another if block after if not self.is_safe_file_to_download(obj.key, dest_directory):
?
Something like if obj.size == 0:
and skipping the download.
from s3scanner.
I'm a little confused about what the issue is you're running into. Does the bucket contain empty directories? If it's possible could you send me the name of the bucket for testing purposes? My email is listed on the contact page of my website if you don't want to post the bucket name here.
from s3scanner.
Sure, the bucket name is claycloud. Warning, it is around 28 GB.
At the end, I did get many files downloaded. The problem was that several directories (containing files) were not downloaded because empty dummy files were created earlier in the process. The second error had files in that directory skipped because that dummy file existed.
from s3scanner.
Thank you for your bug report. I have just merged the pull request for complete re-write of this tool in Go so this issue is likely no longer applicable.
I would encourage you to try out the new version either by checking out the main
branch or downloading the new release once I tag it. If you encounter errors with the new version, please open a new issue.
from s3scanner.
Related Issues (20)
- Ignore dependabot patches for AWS library
- Add package versions to readme HOT 1
- Add workflow to bump homebrew formula
- Add Scaleway support
- Optimize “bucket exists” checks
- Add Oracle provider HOT 1
- Add Vultr provider
- Add Alibaba provider
- Add Lumen (CenturyLink) provider
- winget-releaser failed to open PR
- Add IBM Cloud provider
- Add Tencent Cloud provider
- Add Wasabi provider HOT 1
- Add Huawei Cloud provider
- Add OVH provider HOT 1
- blackarch linux HOT 3
- Custom provider with vhost address style does not work HOT 3
- flag provided but not defined: -endpoint-url HOT 2
- Add Cloudflare R2 provider
- Move and consolidate docs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s3scanner.