okfn-brasil/perfil-politico

A platform for profiling public figures in Brazilian politics

perfil-politico's Issues

Add CI to check pull-requests

• flake8
• isort
• coverage

Add endpoint to check people that have already won an election

[Suggestion] Add gender support for labels and text

Since we already have the candidates gender, it would be nice to print the text accordingly or use a gender neutral approach when necessary:

Swap the order between running and initial setup on README.md

I think in README.md the order of running and initial setup should be swapped.
When $ docker-compose up is runned first the API is not made available at localhost:8000, so I had to run $ docker-compose run django ./manage.py migrate and after that run $ docker-compose up.
Talking with other people in the chat during the sprint, they had a similar issue.

Add endpoint to search for a person

Add data on candidates wealth/heritage

Add data about bills and its authors (from the federal legislative)

A possible tool to generate the CSV to be imported here is the Raspador Legislativo.

A documentação do argumento `clean-previous-data` não está correta | The documentation of the `clean-previous-data` argument is not correct

Português

A documentação do argumento clean-previous-data do comando base atualmente não corresponde ao funcionamento do argumento (remover todos os objetos existentes).

perfil-politico/perfil/core/management/commands/__init__.py

Lines 196 to 204 in f59132f

	parser.add_argument(
	"clean-previous-data",
	default=False,
	nargs="?",
	help=(
	"Creates politicians for all affiliations "
	"regardless if the politician exists or not."
	),
	)

English

The documentation of the base command's clean-previous-data argument currently isn't related to the argument's actual function (delete all existing objects).

perfil-politico/perfil/core/management/commands/__init__.py

Lines 196 to 204 in f59132f

	parser.add_argument(
	"clean-previous-data",
	default=False,
	nargs="?",
	help=(
	"Creates politicians for all affiliations "
	"regardless if the politician exists or not."
	),
	)

[Otimização][spike] Investigar otimização do comando load_affiliations

Problema
O comando load_afilliations leva muito tempo para completar.

Descrição da tarefa
Explorar formas de otimização do comando e propor uma estratégia para implementar a otimização do comando

Critérios de aceite

• Documentar a investigação nesta issue
• Criar uma nova issue propondo uma estratégia de otimização para o comando load_affiliations

Get a new unique identifier for `Person` model

Since 'unique' fields are complicated to gather. A combination of name-birthday could be considered a "good enough" unique field for our dataset. since this could be a sensitive information, we can transform this slug into a hash.

This would facilitate working with difficult databases and use a lot of information that are not
rightly gathered by some sources.

load_people not working after CSVSplitter

Add endpoint to list candidates that have already run elections

Endpoint to list candidates that have not run for any previous election
Endpoint to list candidates that have run for any previous election

[Raspador Legislativo][spike] Discutir com cuducos solução para o crawler do senado

A spider dos dados do senado no projeto do Raspador Legislativo não está funcionando.
O objetivo dessa task é conversar com @cuducos para aprender sobre o projeto do Raspador Legislativo e a partir daí criar as tasks para corrigir a spider do senado.

Translate documentation to Portuguese (pt-br)

Add endpoint to filter by coalition

Add endpoint to filter by Party

Add pagination to endpoint of search candidates by state

Add filter by sum of assets in each election

Use the ORM to get most recent affiliation and his assets for each politician

Solve these TODO#load_affiliations and TODO#load_assets

perfil-politico/perfil/core/management/commands/load_affiliations.py

Lines 43 to 59 in 2d18841

	def politicians_from_affiliation():
	# TODO use the ORM (get most recent affiliation for each `voter_id`)
	sql = """
	SELECT core_affiliation.*
	FROM core_affiliation
	INNER JOIN (
	SELECT voter_id, MAX(started_in) AS started_in
	FROM core_affiliation
	GROUP BY voter_id
	) AS most_recent
	ON most_recent.voter_id = core_affiliation.voter_id
	WHERE status = 'R';
	"""
	yield from (
	Politician(current_affiliation=affiliation)
	for affiliation in Affiliation.objects.raw(sql).iterator()
	)

perfil-politico/perfil/core/management/commands/load_assets.py

Lines 50 to 65 in 88ad9b7

	@staticmethod
	def assets_per_politician_per_year():
	# TODO use the ORM?
	Row = namedtuple("Row", ("politician_id", "year", "value"))
	sql = """
	SELECT
	core_candidate.politician_id,
	core_candidate.year,
	SUM(core_asset.value) AS total
	FROM core_asset
	INNER JOIN core_candidate
	ON core_candidate.id = core_asset.candidate_id
	WHERE core_candidate.politician_id IS NOT NULL
	GROUP BY core_candidate.politician_id, core_candidate.year
	ORDER BY core_candidate.year DESC
	"""

Is it possible to use the ORM in these cases? Will the change turn it more readable?

[Otimização] Otimizar o comando load_affiliations

Foi verificado na issue #185 que o comando load_affiliations pode ser otimizado usando Subqueries na criação dos políticos

Critérios de aceite:

• Refatorar o comando para utilizar Subqueries ao invés de uma query para cada Político a ser criado
• Rodar testes de performance para garantir que o comando está rodando mais rápido
• PR mergeada no master
• Carregar o csv diretamente para o banco (opcional)

Add tests to endpoints

Add code climate.

@cuducos for adding codeclimate you just have to add the github app.

like:

1. Open codeclimate.com
2. Login on it.
3. Select "Add repository"

4. Add the repo you want to.

5. Have a look at you repository on codeclimate

6. You can install codeclimate app on the github repository by going to the tab of settings

[Documentação] Documentar fontes de dados

O README cita as fontes de dados, porém não está explícito (ou poderia ficar mais claro), onde encontrá-las.
Aqui está a linha que explica como conseguir os .csvs
https://github.com/okfn-brasil/perfil-politico/blame/main/README.md#L52

Add an Entity Relationship Diagram (ERD) in the readme.md

I think would be nice to add an ERD in the readme.md. This will make easier to see how the data are relationed in the database, also it makes easier to know what data is available in the API.

[Raspador Legislativo] Corrigir spider do senado

A spider do senado do Raspador Legislativo não está funciondo. Ela deve ser corrigida para podermos atualizar os dados de bills do Perfil Político.

Critério de aceite

• PR para corrigir a spider do senado mergeada no master do Raspador Legislativo

[Integração de dados] Implementar live-update do comando load_bills

Para atualizar o banco de dados de produção sem perder as referências ao IDs existentes, é necessário implementar e documentar como é feita essa rotina de live-update dos projetos de lei.

Critérios de aceite:

• Verificar necessidade de adicionar novas subrotinas para update do comando load_bills, caso necessário, tê-las implementadas e testadas
• Documentar procedimento para realização do update do banco no README

Get relatives of candidates

Get people that are relatives, to find third-party relationships

Add endpoint to list asset evolution trough the year to each candidate

Improve README with details of project

Add data about congresspeople biography (and legislative activity)

Add endpoint of specific candidate

How to obtain the data described on README?

This describes the importation of data. Where do the data come from?

New data of Affiliation with Null values on electoral_section and type has float

When try load new data from affiliation from Brasil.io using docker-compose run django python manage.py load_affiliations /mnt/data/filiacao.csv get somes errors:

ValueError: invalid literal for int() with base 10: ''

After remove some rows with nulls values on electoral_section filed , got a second error:

ValueError: invalid literal for int() with base 10: '83.0'

Model from Affiliation expect a Integer Field not null

perfil-politico/perfil/core/models.py

Line 96 in 88ad9b7

electoral_section = models.IntegerField()

but new data contains null values and electoral_section are float.

Debug Election and Donation data

As mentioned in #42:

• Election model has ~3 million records
• Donation CSV file has ~7 million lines
• However we're able to identify and ~80 thousand donations

So I think this smells like a bug to be investigated when #42 is merged.

Script to populate Neo4J based on Mongo

Notebooks having a failure of kernel when used with docker

When reading a large dataset (>500mb), the kernel of the notebook running on our docker-compose keeps failing. Running the same notebook locally raises no errors.

Verify that the dependency update is working

Update from PR #129

Add data on past campaign donations from companies

Discussion: Where the data should come from?

I think a valid discussion we could have is where the data should come from.
Currently some data comes from brasil.io website and they are not updated frequently. So if the idea behind this project is to have something that is updated with certain frequency maybe would be interesting to have a scrip to get the data from TSE website and make it available in the cloud.

Clarification on #142

PR #142 is a bit awkward to me. The description has no context or purpose for the changes, which break the tests:

• The description says it's a documentation related PR, but it actually changed code too
• Test expects the redirect removed in the code edits from the PR
• The new code does not follow Black style

Surely I can fix that, but before I would like to know more about the context of these changes.

My point is: as tests weren't updated, code edits were not mentioned or justified, and no one in the PR discussion seemed to worry about the CI being red I am not sure if this new state is intentional. Therefore I don't know the direction the fix should go:

• Should the tests be updated to match the code state after the PR being merged?
• Or should the code edited in the PR be reverted (keeping only the docs update)?

cc people involved in that PR @adorilson @dehatanes @BrunaNayara @julianyraiol @sergiomario
also cc @giuliocc

Add data about votes in the legislative

Requires a qualitative selection of relevant topics they could have voted in (cc @ncortezrj)

Add endpoint to filter by profession/career

Update the declared assets of politicians

It's been two years since the last update on the assets declared by politicians to the TSE in 2018.

Update party affiliation of all 2018 election candidates

Many candidates visible on the platform have their party bonds out of date.

• Since 2018 political parties have changed their names.
• During this period people changed or were expelled from parties.

[Integração de dados] Verificar funcionamento do Raspador Legislativo

A atualização dos projetos de lei precisa dos dados do Raspador Legislativo. É necessário verificar se ainda está funcionando a integração dos dados do projeto com o Perfil.

Critérios de aceite:

• Análise da integração do Raspador Legislativo realizada e documentada nessa issue
• Caso algum problema ocorra, ter as tarefas necessárias para corrigí-los elaboradas
• Caso nenhum problema ocorra, comando load_bills testado e funcionando

Add data from companies in which candidates are partners

A possible roadmap:

• Filter companies from the partners list that have a candidate (or homonym) as a partner
• Scrap extra information (CNAE and city/state) from Receita, SERPRO or ReceitaWS

Add data from past elections

Running tests with pipenv

Traceback (most recent call last):
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/bin/pytest", line 11, in <module>
    sys.exit(main())
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/_pytest/config/__init__.py", line 55, in main
    config = _prepareconfig(args, plugins)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/_pytest/config/__init__.py", line 180, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/callers.py", line 196, in _multicall
    gen.send(outcome)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/_pytest/helpconfig.py", line 89, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/_pytest/config/__init__.py", line 612, in pytest_cmdline_parse
    self.parse(args)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/_pytest/config/__init__.py", line 777, in parse
    self._preparse(args, addopts=addopts)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/_pytest/config/__init__.py", line 739, in _preparse
    early_config=self, args=args, parser=self._parser
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/callers.py", line 201, in _multicall
    return outcome.get_result()
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/pytest_django/plugin.py", line 246, in pytest_load_initial_conftests
    dj_settings.DATABASES
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/django/conf/__init__.py", line 106, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/amadeucavalcantefilho/Developer/perfil/api/api/settings.py", line 20, in <module>
    MONGO_URL = config('MONGO_URL')
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/decouple.py", line 197, in __call__
    return self.config(*args, **kwargs)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/decouple.py", line 85, in __call__
    return self.get(*args, **kwargs)
  File "/Users/amadeucavalcantefilho/.local/share/virtualenvs/perfil-hF4nD9Ih/lib/python3.7/site-packages/decouple.py", line 70, in get
    raise UndefinedValueError('{} not found. Declare it as envvar or define a default value.'.format(option))
decouple.UndefinedValueError: MONGO_URL not found. Declare it as envvar or define a default value.

I don't know whether I have be running mongo while runing test.

load_people is failing when importing a new CSV

When a new CSV is imported using load_people the scripts fails because there are duplicated person to be added.

Add sample data set and a documentation about it

Add a sample data set to be used in dev env and avoid the need to download a big amount of data.

We need too a documentation about CVS format used in load_candidates and other commands.