Option to source quotes directly from CSV instead of importing

[jameshnsears]

Option to source quotes directly from CSV instead of importing

[jameshnsears]

Hi - thanks for taking the time to raise this issue.

OK - just to help me establish things... at a high level the app has two databases: the "Internal" database containing around twenty thousand quotes; and, an "External" database that starts out empty and into which, via pressing a button in the app, you import some quotations from a .csv file.

The app is kind of, at least for the moment, built with the expectation that the .csv file will contain multiple quotes. For example, I know of one user who has created a .csv with almost three thousand bible quotes:

• https://raw.githubusercontent.com/jameshnsears/QuoteUnquote/main/app/src/androidTest/assets/EmanuelKebede.csv

It would certainly be possible to create, inside the app, a scheduled import feature (with time and file name / location on the device). However, If it was me, from my limited understanding of the issue, I would collate the quotations from bible.com and maintain a .csv file that I occasionally (perhaps only once), manually, import into the app.

I've looked at https://www.bible.com/verse-of-the-day and it seems to follow a particular web address format:

• https://www.bible.com/verse-of-the-day
• https://www.bible.com/bible/59/HEB.4.12.ESV
• https://www.bible.com/bible/1/GEN.1.ESV
• https://help.youversion.com/l/en/article/7kxvdc0duc-edit-verse-of-the-day-subscriptions-on-bible-com#additional_information

At time of writing the verse from the first two links is identical

Just an idea, but might it be possible for you to "scrape" bible.com (the middle two links in particular) using something like https://scrapy.org/ to construct a .csv file with possibly every verse from the specified bible?

Let me know what you think.

[swishkin]

Thanks for your reply!

I understand the design QuoteUnquote has of containing multiple quotes in a database, and so (whether by philosophy or design), it isn't an ideal solution for my use case; however, it's among the best options I've found so far.

It would be possible to import a CSV containing the entire Bible, but (as QuoteUnquote is now) the verses would merely be randomized or selected sequentially. Instead, I'd like it to show the same verse as Bible.com, as I've think their chosen verse is often topical/relevant for the particular day.

Another possible complication (even if QuoteUnquote could somehow sync with Bible.com), is that sometimes Bible.com has more than one verse as the "verse" of the day.

As for scraping the data, I've already put together a bash script that does that part of it, using lynx, awk, curl, the ESV API, jq, and sed. I know it could be improved, and that it could probably be done much more efficiently with other tools. However, I designed it to be run on my phone, so it could be run by a cron job every morning, alter a simple text file, and that be put on my homescreen somehow.

Another method I've considered is to use ImageMagick to produce an image and Termux to set the wallpaper automatically, again via cron.

Using QuoteUnquote is preferable to this because you've produced a quality experience and clean solution for simply putting some text on your homescreen. I deeply appreciate that regardless of whether it can serve my needs in this case.

[jameshnsears]

OK - your response has clarified things.

I will look into changing the Configuration > Quotations > Database screen. Such that the External card has either a Poll / Watch switch. When selected the app will check to see if the .csv has changed, if so it will reload the .csv into the database and the app widget will get updated automatically.

The fact that the .csv has only one entry, shouldn't matter.

What do you think to this possible stopgap solution?

[swishkin]

From what you describe, it sounds like this would meet the need quite well, and I appreciate your willingness to try to help make this happen!

[jameshnsears]

Hi - just a quick update.

I've coded a possible solution, it seems to work - at least with manual testing.

I still need to write some tests to cover this new code; as well as do some housekeeping for bumping some dependencies.

I'll update this ticket when I've released 4.14.0 - probably sometime next week, though there's a reasonable chance it might be this week.

[jameshnsears]

OK - not good news I'm afraid.

My solution works on Android 10 and earlier, but not from Android 11 onwards. This is because more recent versions of Android have gradually restricted file access.

It's not possible for me to produce a version of the app that targets Android 10 - in such a compatible way - as the app uses libraries that won't build against such an "early" version of Android.

I'm not sure if your using Android 10. If you are then there is some good news, as I'm keeping the code I've written in place. Meaning a Watch switch is present in the External database import card and a background process will update the widget if the .csv is updated.

I apologise for raising your hopes. I don't feel like I've wasted any of my time though, as I've learnt a few things and also managed to improve the code in a couple of ways.

I will close this issue once 4.14.0 has been released.

[jameshnsears]

4.14.0 has been released to Google Play & F-Droid - and should be published within a few days, if not sooner.

I'll close this ticket now but please feel free to contact me directly if you have anything more to say :-)

[jameshnsears]

Hi - out of interest I, today, emailed YouVersion, the makes of the bible.com app. To see if there is a legitimate way to intact with their systems such that I can retrieve VOTD.

As a side note I noticed that the bible.com Android app features a VOTD App Widget - it's pretty basic, but it might help you.

[jameshnsears]

FYI in the next regular release I've decided to extend the Quotations > Database tab. Such that it can Scrape a URL using Xpaths to query the scraped data for a Quote and Source.

The attached screenshot is WIP taken from Android 10 emulator.

This effort is separate from anything resulting from YouVersion request feedback.

[swishkin]

Thanks for the updates! Unfortunately, I am running Android 13, but I look forward to testing your planned changes for scraping!

Edit: Regarding the Bible.com/YouVersion app, I chose to stop using it due to its many extraneous features, i.e. social media, commentary, video content being included in an app which (ideally) should allow me to focus on the Bible itself.

[jameshnsears]

Hi - I've released 4.15.0: it includes the ability to scrape a web page for a quotation and source.

Hopefully this is a good enough solution until you find something more appropriate.

For VOTD I've used the following three parameters, which you'll need to paste into the app:

https://www.bible.com/verse-of-the-day
//[@id="votd_wrapper"]/div/div/div[1]/div[2]/p[1]
//[@id="votd_wrapper"]/div/div/div[1]/div[2]/p[2]

During testing I noticed that if I keep all prior responses (there is a new switch in the UI), and not just the latest, then sometimes the app will build up a history of multiple versions of the "same" quotation (received on the same day). The text, and source, can differ slightly. Even the language (maybe due to my locale) can differ. In such a case, if the latest response matches a prior response then the app will "jump" to the prior (until a new latest is received) - the app only stores unique values in its database.

I've also noticed that sometimes VOTD is not always syntactically correct - i.e. it has a closing " but not an opening one. The app is just reflecting what's on the web page.

Please let me know if/when you spot a problem, or if any fine tuning might be needed.

Thanks for raising this issue as I've learnt some new things along the way, and improved some code elsewhere.

I'll close this ticket in a few days after the release is made available in store.

[jameshnsears]

Hi - I'm going to close this issue now. The latest version - 4.15.2 - of the released code hasn't yet made it to F-Droid (which can be a bit slow) but it's been in Google Play for a few days.

The absolute latest version of this code - 4.15.3 - isn't released just yet, though its very close to being so: I made some improvements that lightly touch on the XPath functionality.

Although I'm happy that the XPath functionality works, I've today noticed that the VOTD XPaths above no longer work: YouVersion (who never got back to me) have changed their VOTD web page layout.

With new XPaths the VOTD page can be scraped. The problem is that they have split the quotation into two XPath nodes. And the code I wrote only takes the first node.

I could change Quote Unquote code to cater for this, however I'm not planning to. This is because the XPath functionality I've put in is meant to apply to generic scenarios rather than to a specific one (which is liable to the whims of the endpoint owner).

One possible answer to this might be that you still use termux to scrape, use avweb server on android to publish, and get the app widget to present the data. Just an idea!

Thanks for raising this issue though as its led to lots of interesting things as as result.