security-force-monitor/fmtrpt_data

A dataset on unclassified training activities for non-US security forces arranged and funded by the United States Department of State and Department of Defence between 2001 and 2021. Sources, scraping, cleaning and publishing toolset included. Search and explore the data at: https://trainingdata.securityforcemonitor.org

fmtrpt_data's Issues

Angola and Benin data in 2019-2020 report are corrupted in source

Page 101 of the FY 2019-2020 report (here) has a corrupted page, which contains an empty table:

Larger:

The page would have contained data that shows the end of Angola data and the start of Benin data. As this is missing, our scraper will not know when Angola data ends and Benin data starts, so all the Benin data will be classed as Angola data. IN the updated dataset we have removed this data until it is resolved in the source data.

Add table of contents to README

It's become a rather long document, and could do a navigational aid.

ISO codes missing from entries for Kosovo and Fiji in 2018-2019 data

Some countries do not have ISO code assigned to them. You can see this in this query:

https://trainingdata.securityforcemonitor.org/state-department-data?sql=select%0D%0Acountry%2Ccount%28country%29%0D%0Afrom%0D%0A++training_data%0D%0Awhere%0D%0A++iso_3166_1+IS+NULL%0D%0Agroup+by+%0D%0Acountry%0D%0A

select
country,count(country)
from
  training_data
where
  iso_3166_1 IS NULL
group by 
country

country	count(country)
	0
Fiji IMET - FY 2006 DoS Training	39
Indochina	1
Kosovo	1283

Need to:

• Find appropriate ISO codes for the entries
• Figure out the parsing problem with Fiji
• Update and re-run group-and-clean script