Make RSS feed validation part of CI

[artem-zinnatullin]

Make RSS feed validation part of CI

[sockeqwe]

It would be super awesome to have a ci that not only validates rss feed, but would automate all the things. Something like: 1. i make a new branch, i.e with name Episode10, containing just the show nodes markdown file and push this branch on github. 2. CI kicks in and validates show nodes, then generates: - rss feed.xml - update README - create milestone - create issue for discussion - validate all generated files - long term goal: auto editing of recorded files 3. Create new branch from Epsiode 10 branch and commit and push all CI generated branches back and creates a pull request. 4. Accept pull request and delete branch or reject pull request and start whole process again by with editing shownodes markdown file which the triggers ci and so on :) Maybe instead of mark down shownodes we could trigger step 1 by writing a tiny dsl (which basically represents shownodes) in kotlin and commit and push this dsl file instead of shownodes markdown file... Just some random ideas... Artem Zinnatullin :slowpoke: <[email protected]> schrieb am Di., 11. Apr. 2017, 16:45:

…

Currently we don't have any CI for the project, but we should since we do everything through GitHub PRs. As a first CI step we can add rss feed xml validation as suggested here #61 (comment) <#61 (comment)> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#63>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAjnrgCxQ0ylMtavXuQzhqeAOM903D7Gks5ru5IWgaJpZM4M6M9-> .

[sockeqwe]

So I have played around with the idea of auto generating and validating stuff for podcast here: https://github.com/sockeqwe/gitpodcaster

@artem-zinnatullin do you think this is going in the right direction and we should investigate further on this approach?
Summary: idea is to define episodes as kotlin classes in a typesafe manner incl. show notes (as markdown). Workflow is currently the following:

1. Record new episode
2. Create new branch for new episode (forked from master)
3. Create kotlin file for new episode with all episode data (guest, show notes, etc.). Commit and push this new file.
4. Create pull request
5. CI (travis) kicks in and evaluates file and generates feed.rss and shownotes.md and updates readme.md file. Travis will push those changes (generated files) back to same pull request branch, so that the auto generated / updated stuff can be reviewed as part of the pull requeat review.
6. Other cohost reviews pull request (after CI completed) and merges into master (which means publishing) or requests changes.

Advantages of this approach:

• automatic validation of Feed etc. Actually I would like to implement further checks like checking url's, auto download mp3 file of the episode to check for valid url and extract information like length (minutes / seconds) and size (bytes) for feed.rss
• fully visible which changes has been made by CI and also reviewable in pull request.
• changes by host can be made easily. I.e. fixing typo, because all files will be regenerated from scratch (not appending i.e. to feed.rss file). Just edit epsiode file, commit and push it.
• "hotfixes" can be applied directly by hosts on master branch
• Since everything is generated on the fly on each change, whole system is stateless which is nice and avoid struggling with "previous artifacts"

Disadvantages:

• feed.rss have to be generated by scanning all episodes file. There is no "appending" option (or at least O can't think of a good idea how to implement such a mechanism). This means that also all urls also of old episodes etc. will be validated on each pull request while generating feed which seems like wasting time and resources. Maybe this can be fixed somehow (Im not sure how yet, maybe assigning id for each epsiode and keep track of latest episode and only validate that one, but seems error prone). Maybe we can detect via git which file has been changed to determine which epsiode should be evaluated (may require a naming convention for episode file in kotlin)
• Automatically create github issue for discussion via CI is possible, however renaming of episodes might be challenging but still possible tho if keeping track of episodes (i.e. via episode id) works somehow.
• after merged to master we also want to create a git tag and upload mp3 file to release section. This should be done by CI (automate all the things). However, that is not super easy because requires of keeping track which episode has been released. Might work by providing additional info for CI as commit message (like the id of the episode that has been released so that CI knows which episode should be uploaded to release section and which git tag).
• I would also like to generate a small website (i.e. github page with jekyll) which might requires some "after merged to master action" as described in point above (git tag). I think such a little website would help a lot to reach mor listeners (i would like to add a very simple js web player to listen to the episode directly) and makes it easier to link to our podcast or a concrete episode (i.e. reddit, android weekly, etc.). Not sure how this could be implemented with our desired git flow / review process (maybe we don't want to review generated html at all).

While I really would like to automate all this things and integrate it with the git flow / pull request / review , I'm not sure if the current prototype implementation is going in the right direction. Unfortunately, I can't think of a better implementation right now. Any thoughts / ideas?

[artem-zinnatullin]

Sorry that it took me so long to answer, here is the feedback:

I've cloned the repo and reviewed code, in general, I like the implementation, especially knowing our hugest constraint — podcast hosting as a git repo on GitHub.

I was mostly thinking about one thing — commiting autogenerated files into the repo, which is usually a wrong thing to do.

So I was looking into ways to migrate hosting of our rss feed files to some real hosting but since we can't change GitHub response for a particular file to a 301 HTTP redirect we only can automatically migrate iTunes listeners (and those who use apps that implement iTunes new-feed-url directive).

So knowing all that I think your implementation is good enough (we'll do some PRs haha) and should be used for new episodes! 🎉

[sockeqwe]

commiting autogenerated files into the repo, which is usually a wrong thing to do.

Agreed, this feels wrong. Maybe we should use master branch to only "host" generated files and a "develop" branch where we commit our episode kotlin files. Then we deploy this to master.
However, not sure how this could work exactly with pull request or to be more precise: when do we generate files? After the pull requst has accepted and merged to develop? So workflow like this:

1. Create branch episodeXY (fork from develop branch)
2. Create kotlin file for episodeXY, commit and create pull request for episodeXY. Target branch of pull request is develop.
3. Our tool validates kotlin file for episodeXY, but doesn't generate files
4. After reviewing and validation, pull request get merged on "develop" branch
5. Our tool listens for commits on develop branch and generates files from it which then will be pushed to master branch

[artem-zinnatullin]

Another option is to use this git repo as production deployed by git pushes with full history of what was deployed and another repo for just Kotlin source files.

So whenever we put a tag in another repo it launches CI that deploys generated files to this repo!

[sockeqwe]

Sounds like a good idea! Artem Zinnatullin :slowpoke: <[email protected]> schrieb am Sa., 10. Juni 2017, 15:42:

…

Another option is to use this git repo as production deployed by git pushes with full history of what was deployed and another repo for just Kotlin source files. So whenever we put a tag in another repo it launches CI that deploys generated files to this repo! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAjnrv-rilhdOAdh9zJcH-1CJlejuja4ks5sCp08gaJpZM4M6M9-> .