opengovaus/ausbills

🇦🇺 This is a package for obtaining Bills from Australian parliaments.

ausbills's Issues

Repository is cluttered

Typically, I like to follow the format as seen in this repo and the article here.

README.rst
LICENSE
setup.py
requirements.txt
sample/__init__.py
sample/core.py
sample/helpers.py
docs/conf.py
docs/index.rst
tests/test_basic.py
tests/test_advanced.py

The goal of this issue is to determine what we can change to declutter the repository.

Beginning with:

• Move .gitignore into the top-level and populate it with GitHub's default Python project .gitignore as used here.
• Move example.py into /docs
• Move /ausbills/test.py to /test/test.py

Other things that seem to have been committed incidentally/don't have a purpose that I am aware of and should be considered for deletion:

• bills_out.json
• /dist
• /build/lib/ausbills
• /ausbills.egg-info

@KipCrossing Please school me or approve of this

Check warning: Bad data 'href'

Bad data 'href'  -  Coronavirus Economic Response Package (Payments and Benefits) 2020

Functional vs Object Oriented

Would it make more sense to go with an Object-Oriented approach to this package?

For instance, instead of or returning the data as dicts and lists, have objects such as a bill object and an upper_bills object for instance. Each bill object, may have objects such as bill.title and bill.sumary.

Federal scraper does not get bill/chamber progress

Describe the bug
The federal scraper lacks the functionality included in the new WA & ACT scrapers to get the actual progress of a bill in a readable format, e.g First chamber, 2nd reading:

types.BillProgress.FIRST, types.ChamberProgress.SECOND_READING

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

[Feature Request] Scrape Petitions

Need to investigate how viable it is to scrape petitions such as:

https://www.aph.gov.au/petition_list

Would be nice to include this in ausbills.

UI Alignment

We need to get some UI alignment between all the parliament scrapers. We should have a workshop for this.

Common names

We should try and get common names between all on them. For example, Senate vs LA; we could just use Upper

Common Objects

We already use Bill but should do more. Max has some more in the WA scraper

Common Errors

This will help the user to debug and handle errors in their own way

Typing

We should def use more typing to help the user. Common classes will help with this

Note, I'm not loving the fact that the user has to use the .value property with Maybe Just/Nothing (pymonad). We should abstract this away from the user when possible. ie return(result.value)

"." in dict key is not good for DB

Rename 'Act No.' to 'Act No' - without the .

Dates dissapear after use with Bill() object

Solution:

change:

https://github.com/KipCrossing/Aus-Bills/blob/9a5d9a7101a8dd3e61a74bc326970994578d8d92/ausbills/federal_parliment.py#L131

to:

_all_bills = all_bills.copy()

Dependabot released

We're still using Dependabot-preview but Dependabot has since been released, otherwise known as Dependabot 2.0

The biggest change here is that Dependabot is now built into GitHub and is managed on the repo settings instead of the Dependabot website.

To update
Head to the Dependabot website, login, view the relevant repository that Dependabot is monitoring and follow the instructions.

Wrong URL in setup.py

In line 37

    url="https://github.com/KipCrossing/Aus-Bills-Discord-Bot",

Should be https://github.com/KipCrossing/Aus-Bills

Irrelevant TAS topic

The TAS topic is meant for tool-assisted speedrunning, not Tasmania. This tag should be removed from this repository.

Not sure what licence to use

I just went with MIT but some insight would be nice.

https://choosealicense.com/

bs4 not installing

Traceback (most recent call last):
  File "update_bills_db.py", line 2, in <module>
    from ausbills.federal_parliment import all_bills, Bill
  File "/home/os/.local/lib/python3.6/site-packages/ausbills/federal_parliment.py", line 1, in <module>
    from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'

The Victorian extractor throws a KeyError on the bill progress stage

To reproduce: run the Victorian parliament test and see for yourself

Extra html with the em_link (Federal)

Describe the bug
Looks like its only happening with the em

text_link_doc
https://parlinfo.aph.gov.au/parlInfo/download/legislation/bills/r6536_aspassed/toc_word/20039b01.docx;fileType=application%2Fvnd%2Eopenxmlformats%2Dofficedocument%2Ewordprocessingml%2Edocument

em_link_doc
<a href="https://parlinfo.aph.gov.au/parlInfo/download/legislation/ems/r6536_ems_aee2e2b9-203e-46ce-a59d-d3f2e374d6de/upload_word/734686.docx;fileType=application%2Fvnd.openxmlformats-officedocument.wordprocessingml.document"><img alt="Word Format" src="/images/template/icons/doc-word.png"/></a>

text_link_pdf
https://parlinfo.aph.gov.au/parlInfo/download/legislation/bills/r6536_aspassed/toc_pdf/20039b01.pdf;fileType=application%2Fpdf

em_link_pdf
<a href="https://parlinfo.aph.gov.au/parlInfo/download/legislation/ems/r6536_ems_aee2e2b9-203e-46ce-a59d-d3f2e374d6de/upload_pdf/734686.pdf;fileType=application%2Fpdf"><img alt="PDF Format" src="/images/template/icons/doc-pdf.png"/></a>

text_link_html
https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Id:%22legislation/bills/r6536_aspassed/0000%22

em_link_html
<a href="https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Id%3A%22legislation%2Fems%2Fr6536_ems_aee2e2b9-203e-46ce-a59d-d3f2e374d6de%22" rel="noopener noreferrer" target="_blank"><img alt="HTML Format" src="/images/template/icons/doc-html.png"/></a>

@connorhsm could you please look into this? Otherwise, I'll look at it later, it's just a little low on my todo list rn.

To Reproduce
Steps to reproduce the behavior:

from ausbills.federal_parliment import get_all_bills, Bill


for bill in get_all_bills():
    bill_obj = Bill(bill_id=bill["id"])
    b_data = bill_obj.data
    print(b_data)

[Feature Request] Pull the Outline of the explanatory memorandum

Problem
Not enough useful information in app to persuade users to engage with the bill

Solution
Pull the outline of the explanatory memorandum

Portfolio value is all one string unformatted.

Portfolio value is all one string unformatted.

eg: 'Communications,CyberSafetyandtheArts'

Can this be fixed to "Communications, Cyber Safety and the Arts"