lsflk/gig-scripts

Sample crawlers and importers for GIG

Agency Jekyll theme

Agency theme based on Agency bootstrap theme

How to use

###Portfolio

Portfolio projects are in '/_posts'

Images are in '/img/portfolio'

###About

Images are in '/img/about/'

###Team

Team members and info are in '_config.yml'

Images are in '/img/team/'

Demo

View this jekyll theme in action here

========= For more details, read documentation

gig-scripts's Issues

fix outdated crawlers and write tests to check if a crawler is outdated

The Island Crawler is out of date

differentiate primary ministries from the ministry names

ministry names in the gazette consist of multiple ministries combined together. For eg. Ministry of A & B, Ministry of D,E,F and G. try to extract primary ministries such as Ministry of A, Ministry of B, Ministry of D, etc. So that a link can be established between the ministries even though the ministries get dissolved from time to time

User case - Test budget data against normalised names

Related Articles and articles list is empty for kavuda entities

Import budget and tender data if user test is successful

Error notifications for crawlers

Crawlers crash due to various server and network issues. So we have to restart the servers. Currently there is no notification system to get alerts when a server is not working.

Wireframe design

Wireframe design drawing on the crowds ideas (sketch)

Develop a prototype and share with the public again to get feedback.

crawlers change requesting normalizer based on entity type. i.e : Location, Person, Organization

All ministries are not imported for orgchart

Cross-check primary key list

Do we need to manually cross check if the noramlised names are accurate/representative? What I mean by manually is looking over a spreadsheet of the selected primary keys to see more need to be added.

UI - Crowd source design sprint

Crowd source - test our ideas from our in-house design sprint as well as as the crowd for ideas on how they would like to use it.

When the org ends its serving period, How to keep the entity suppressed from chart

• how to filter entity by regime (end date attribute)

• if entity ends and restart by the same name, no issue

• if new entity starts with a different name and changes name to old one???

• should the entity be restarted

• start with a fresh entity

• if starting with a fresh entity, name duplication might occur

• if entity is not restarted at all

User case - Test tender data against normalised names

Call crawlers for different news sources in separate threads

Problem

Currently, the crawlers for different news sources are called serially (@umayangag please verify this claim - it was based on a rough look at the code). This means that a crash on one source will stop the remaining sources being crawled.

Solution

Call crawlers for different news sources in separate threads

GIG-Scripts/kavuda/crawl.go

Lines 16 to 20 in f37a4a8

	crawl(ada_derana.AdaDeranaDecoder{})
	crawl(ceylon_today.CeylonTodayDecoder{})
	crawl(daily_mirror.DailyMirrorDecoder{})
	crawl(daily_news.DailyNewsDecoder{})
	crawl(the_island.TheIslandDecoder{})

Develop the front-end platform

Based on the final set of feedback on the wireframe prototype, develop the site.

Note that this will need to be meshed with the LDF website i.e About LDF etc Zaithoon and I are working on the content for that separately.

Digitisation of MoF budget est. data PDFs

Test OCR. If that doesn't work then we may need to resort to a more time intensive options of copy/paste.

Ideally need to have all the data on GIG by 21 March - in case there are researchers and journalists who may wish to use the data for analysis prior to the elections.

crawl wikipedia to fill empty entities created while crawling news articles

EIA Public Participation Alerts geotagged plus automated email generation for subscribers

Vanguard Pty Ltd has agreed to provide an EIA public comments alert data as a CSR activity. These notices are only published in newspapers at the moments.

This will be time sensitive activity, so I need to be able to enter the data and have it immediately pop up on the LDF SL map.

Also need to create an automated email subscription - so that those signed up will get a link to the announcement.

There will likely be only one alert per week.

orgchart crawlers automate csv data cleaning process

UI - In-house design sprint

Org chart + budget + tender data + RTI archive user interface - conceptualsing what we want the UI to deliver to the public

Plus UI for Kavuda.lk and how it will be integrated onto the main LDF platform.

We can then use this as the basis to launch the crowdsource UI design sprint.

Tag all SOEs & highlight the 55 strategically important SOEs on the org chart

1.Let's tag all the SOEs (527) as well as SOEs have been identified by MoF as being of strategic importance (55). I have shared the excel spreadsheet with you

2.I have also included a column indicated which sector (as per the MoF report the 55 SOEs were grouped into 16 sectors). Is there a way to select the sector and get a summary of all the strategically important SOEs on our org chart?
Department of Public Enterprises - Performance Report 2017: http://www.treasury.gov.lk/documents/63940/182153/PED+PR+-+2017++%28E%29.pdf/b85cb9f4-1ebf-4214-a211-23bc6e5521e7

Advocata report notes:

• 527 SOEs
• Only 10.4% report financials
• 55 of the 527 are recognised as strategically important SOEs by the Treasury

Remarks made by Hon. Mangala Samaraweera, published 5 July 2018
"During 2017, your 55 public ventures alone have recorded a total turnover of nearly Rs. 1800 billion which was almost 13.22 percent of the GDP. Out of the 55 SOBs 39 recorded a net profit amounted to Rs. 136 billion while 16 made net losses amounted to Rs. 87 billion in 2017.
The total asset base of SOBs grew by 13.6 percent in 2017 over 2016 and it accounted for almost 56.8 percent of the GDP. "
http://treasury.gov.lk/article/-/article-viewer-portlet/render/view/remarks-made-by-hon-mangala-samaraweera-minister-of-finance-and-media-at-the-conference-for-the-heads-of-the-state-owned-enterprises-soes-at-the-hilto