bdecon / us-chartbook Goto Github PK
View Code? Open in Web Editor NEWAutomated PDF containing charts, tables, maps, and analysis of the US economy
Automated PDF containing charts, tables, maps, and analysis of the US economy
Add a new chart with the employment rate by state
Fundamental to the new chartbook is having meaningful text. I can't manually intervene in the process often, so I want a simple way to decide which text to show and to create the text correctly.
A basic option so far has been:
"In latest_date
, series
((contributed/subtracted) latest_value
(to/from)/did not contribute to) GDP growth, (compared to a (contribution/subtraction) of 3yr_ma_value
percentage points/and did not contribute) on average over the past three years."
But that won't work for series other than contribution to growth and also is a bit confusing and probably not always helpful.
What I eventually want is to evaluate the latest value compared to its time series and use that evaluation to determine which text to generate. Specifically, I am interested in short- medium- and long-term developments and figuring out when they are meaningful.
Simple examples to target would be:
latest_value
percentage points/was unchanged)"latest_value
percentage points (higher/lower)/unchanged)"latest_value
"Some more advanced options:
Ideally, for some series, calculate the latest value and the last time it was that high/low. If that was more than X ago then use that line of text "(highest/lowest) since last_time_date
"
Some cases have large recent changes: "an (increase/decrease) of change_value
percentage points since reference_date
"
Deciding when to use a different word, for example: near 0 change could be "virtually unchanged", or very large changes "a large (increase/decrease) of ..."
To efficiently update the code, I want to run one command. This means I'll need to have a structured way to store the various individual chunks of code that update individual datasets or charts. Right now, there isn't really a structure to this.
Add text to describe chart showing NILF changes since March 2001.
Can either create a txt file with the info or try to get pgfplots to give me the data in the format I want. The latter is preferable but might not be feasible.
Figure out a better way to add the recession bars.
Add one para to cover the major developments in retail sales data by category of store.
Indicate on chartbook with mark/text when data shown in a chart was released in the last 7 days.
Python script could generate tex code that creates an x tick mark with the latest available date.
Ideally, I want to have the option to use a serif font in the chartbook. I tried to get this to work but didn't have any luck.
GDP by industry will be released for 2019 Q2 in mid October. This section was produced relatively manually and needs to be fixed/sorted at some point.
Data collection structured by report/release.
Data processing--separate charts and text processing. Each chart must be of a identified chart type which has its own function. Charts each have their own notebook cell and name. Text handled based on chartbook section/topic/use cases to allow for more flexibility.
When the latest quarter is available for some charts but not others, the chartbook should indicate that the charts are showing additional information.
Create a template for adding bubble and scatter plots to the chartbook.
Create a template for each chart type that can be filled by python.
A calendar that stores dates of new releases/updates could drive the chartbook data updates.
Both the text above the chart and the content of the chart currently involve manually interventions that can be automated.
Add basic text to capture employment by occupation data.
Add paragraph overview of investment from each sector (contribution to GDP from fixed investment).
The other category in US wealth is huge. It also is complicated because it handles the foreign sector balance. Try to make better sense of what is going on and include a paragraph to describe it.
For example bold if abs(gdp_contrib) > 2
Write some code that returns a dictionary of text for a given datetime.
Add summary paragraph for corporate profits
Look into better ways to retrieve data from the Fed. I'm fairly sure the current code will not work to retrieve updates.
It's not clear to me what people mean when they say the US economy is shifting to services. On one hand, clearly production of goods has been outsourced relative to the past.
But also, there's a shift towards spending a much larger share of the national income on health and education. Yet the health indicators aren't improving and while increased education does increase wages, wages haven't been going up as quickly as in the past.
Convert the code that generates the age maps to a loop that produces both. Currently, each must be created manually.
I've been inconsistent in applying price adjustments, often using the PCE deflator when my general preference is for the CPI-U. At some point I will need to address this, and be more consistent in which deflator is used.
Capital income: proprietor, interest and dividends, rent
Pretty sure the meaningful concept here will be private gross investment minus depreciation and not including inventories or inventory valuation adjustments. Need to research it a bit.
Running the chartbook with only one half of the first section complete is already slow and in need of help from externalizing the graphics. Will need to figure out a good way to do this.
Be clear about API requests and results.
Show the age 25+ population by education, over time.
Full-time, part-time, and second job share of population by detailed occupation, from the CPS. (OCCD)
Use CPS ASEC to replicate the classic chart from Matt Bruenig.
Add chart showing AHE for total and production/non-supervisory.
The use of csv files for storing API returned data will not scale well and doesn't allow for storing metadata.
I'll need to figure out a structured way to store the API returned data and metadata. In past experience, a pickle file would probably work but SQL would give me a lot of powerful options. For example, I would eventually like to build in some features that tell me whether data has been updated recently and that also prevents a situation where good data is overwritten by empty or broken data.
A key consideration, though, needs to be ease of use later on. The current csv approach makes it very easy to get a variable into memory.
Notebooks are not the answer for the final version of the chartbook. I need to move each data source's code into a self-contained .py file. Try this out with industrial production.
Current text doesn't make sense--it shows a fall in rented units but a positive contribution to total. Look into and fix.
Current code would lose data / break if API request fails. Ideally I should store the data locally or keep a backup on GitHub. Look into options.
Use the table 7 data to replicate SPM figure 8, which shows how many people were removed from poverty by various programs.
Optimize code in the contributions to GDP loop and add comments throughout. Convert some repetitive code to functions.
Getting duplicate data from the sql database for a few series. I suspect the BEA tables that were not updated by the advance estimate get duplicated in the database.
I'd like to know how expenses have been changing for US businesses. No idea on the source but ideally would want to know: labor, materials, capital, tax, and financing costs.
Add para on developments with USD exchange with EUR, GBP, JPY, and CAD.
Linking charts and tables to references would benefit from some structure/identification system beyond the name of the chart. Perhaps each chart can have a number like C1.1, C1.2, ..., C2.1, ... based on the section number. Tables would be T1.1...
The key would be automating the generation of each number and the link to the legend and references.
Several charts use CPS data, so create a function that reads the files in the CPS data directly and identifies the most recent month.
Create a new template so that I can easily add horizontal bar charts where the values are stacked
Add several charts to the personal income section. The obvious ones are for welfare and capital income by source. But more importantly try to make one good chart showing the distribution, possibly sourced from the CPS ASEC. Look at some past examples from Matt Bruenig, who has some of the clearest thinking on the topic.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.