Saving tweets with SaveToFilePipeline creates too many files

[jonbakerfish]

Saving tweets with SaveToFilePipeline creates too many files

[jonbakerfish]

Did you try using mongodb to save your data? All you need is to install mongodb and change the pipeline.
Besides, the mongodb will use the ID of user/tweet as index and ensure they are unique. see pipelines.py

BTW, you can use mongo-hacker for better UX when looking into the data.

[ckingdev]

I looked at the that file plenty, but that doesn't change the fact that the default pipeline is broken and there's no documentation. If you read what I wrote, I have my problem solved. I was offering to clean up the code and make a pull request. I don't need mongo to store this, I'd just have to pull everything out and dump it to disk as json anyway.

[grandimam]

@ckingdev - default pipeline is broken is not entirely correct, if you understood how scrapy worked. And one of the reason why DB interaction is simpler because of not having this overhead of storing data in files.

However, to make your work a bit simpler - you could have just removed the pipeline flow i.e removed yield at line numbers: 151 & 161 with file insertion logic. This would have worked for your specific use case.

Introducing this logic in the package entirely is not suggested as the DB interaction is more elegant and this design should be encouraged.