raenonx/madison-metro-sim

Madison Metro System simulator. (UW Madison 2020 Fall CS 638 Project)

madison-metro-sim's Issues

Bus travel time comparison before/after stop manipulation

Continue from #8. Blocked by #25.

• DO NOT group the stops when calculating this because traveling in a different direction produces different results.

Find stops without any routes

Some stops currently seem to have no assigned routes. Create a map reporting these "dangling" stops.

Known stops without assigned routes (92 WKD)

• #2640 S Allen & Commonwealth (SB)
• #2969 S Allen & Commonwealth (NB)

Improve stop grouping algorithm

As stated in the report of 2020/09/21, some stops located on different streets should be grouped.

Group these stops to get a better idea of the stop density, could also potentially bring convenience to #8.

Things to do before commit

The reason of why we are doing this is because that we don't want to write codes that the others cannot understand or run. If such happens, we need to spend additional time on fixing these, which holds back our process. If we can ensure these things won't happen before it happens and it also won't take too much additional time, why aren't we doing that?

Things to make sure

Check that you've installed pylint, pydocstyle and pytest.

These should be installed if you do exactly what was stated in README.

If not, run this:

pip install -r requirements-dev.txt

Commands to run before commit

pydocstyle msnmetrosim
pylint msnmetrosim
pytest

pydocstyle checks if you have the consistent documentation format comply with PEP-257.

pylint checks if you have any coding style issues.

pytest runs all the tests. For creating test cases, refer to #10.

Notes

If you're interested in automating these things, here are the ways you can do:

• Try git hook
• Write a script (batch script for Windows; bash script for MacOS) and run it before you commit

Bus route plot to clearly show the overlapped routes

As of a338fc7, it's not clear to see whether a bus route is overlapped or not. Find a way to clearly indicate that.

Possible solutions

Not limited to these, but could be the starting point.

Parallel lines

For this, we may need to modify how shape data was being loaded, and apply a small offset to it.

Line with mixed colors

For this, we may need to find some scripts or builtin class to create a similar effect like this.

An alternate way of this is to have the line acts like it's combined with multiple color blocks.

Index of features to implement

• Remove/Add/Combine stops
• New bus stop passenger flow
• Budget check #23
• Bus frequency
• Automatically find the optimal solution

For features that require hourly data

#6 #13

If hourly data is unavailable and unprojectable, reduce the priority on the features that require hourly data first, and try to create a simple interface later for manually entering the customized input.

Budget comparison before/after stop manipulation

Continue from #8.

Needs to find the budget data first.

Let's Discuss Here! (Check #25)

Discuss in Group

• I'm thinking about how we should store our result
  • In my humble opinion:
    • We can store them into a csv/shp file which has the following columns (in doing so, we can store the data we want (fewer columns, less distraction); in addition, this creates more flexibility for us in the future since we can add some newly extracted data to this file.

    • Approach 1

      • write to each route file like route2.csv, route3.csv, route4.csv, ..., route84.csv, and put them in a folder
      • (this is what I prefer)

    • Approach 2

      • stores them all in one file
      • (may be convenient to just open one file and increase runtime(? but I think we could do this when had all stops sorted)

Columns!

• **Desired Columns** (more to be added)

  Route#.csv

  • route number
  • stop_id
  • stop_lat
  • stop_lon
  • directions
  • distance_from_last_stop
  • travel_time_from_last_stop
  • number_of_boarding
  • (frequency too?)
  • etc.

Technologies could be used

Folium

Interactive map. May need to embed the generated HTML into another index page.

• Documentation
• Mapping Points with Folium
  • Third-party tutorial
• Folium Icon

Things to confirm with Tyler for week 1

Make an appointment for office hour with Tyler

• Ask for data that contains hourly ridership of each route (from Meghan?)
  • #6
• Ask about what kinds of plot are feasible for the topic we have
  • @RaenonX : depends on how our progress goes and what kind of research we did.
• Ask if it's possible for us to present our plots after we built our simulator
  • @RaenonX confirmed with Tyler: present data plots produced during the development of the sim. These plots will be the major part of our weekly reports. We can also use our sim to generate some reports.

git commands

Concepts

You are working at your local computer. The code is also stored remotely at the origin.

You pull the code from the server / origin; you push your code from the local to the origin.

The most recent commit at the current branch is HEAD.

It's recommended to either commit or stash before you execute any of these commands except git commit, so you may save time on solving conflictions (if any) and/or reduce the risk of losing your work.

I personally recommends PyCharm as it provides some robust git controls. Despite of this, there are still many cases that you will find terminal/cmd/shell/bash useful than the IDE. So I will say, get an IDE for commit, solving conflictions; use terminal for other things.

• Create a JetBrains account with school email, and apply for the educational license, then you can use most of their IDEs for FREE.
• Also, if you are trying to play with the commands around, you won't need to be afraid of losing your works, because you can use its Local History feature to restore your code to almost any point of time. (I still recommend you to create a dummy history to play this fire though)

Basic Commands

Commit

git commit -m "YOUR_MESSAGE"

• Refer to #7 to see the preferred commit message format.
• Don't contain a lot of changes in a single commit. Once you've done implementing a feature and passed the tests, commit it.
• Do things listed in #11 to reduce the need of amending your commits (will be mentioned later)

Push

git push

• Push your code to the origin.
• Before I automated the process in #11, I used to commit some locally without pushing it. Once I think that I am ready to go without any errors and possible changes, I will push all of them at once.
  • If you automated the process in #11, you may not need to stage the commit locally.
  • git push pushes all not-yet-pushed commits.

Pull

git pull

• Pull the code from the origin.
• Recommended before you start your work to avoid wasting time on fixing conflicts.
• If you're using PyCharm, you can download this plugin called GitToolBox to auto fetch for you.
  • The auto-fetch mentioned above runs git fetch, which only check and download the progress of the origin, instead of overwriting your code, which git pull does. You can check this link for a better/clearer/detailed explanation.

Advanced Commands

Force push

git push -f

• Force update your commit history at the origin. Once you've executed this, the commit history at the remote side will be completed overwritten by your local commit history. Be careful when you execute this!

Rebase interactively

git rebase -i <REF>

<REF> can be (not limited to):

• HEAD~#: count of commits counting from and including the head.

  • For example, HEAD~2 means 2 commits from the head and including the head.

• <SHA>: commit hash. Preferably a short commit with 7 or 8 characters would be sufficient.

  • For example, 3508c81 will start the rebase from the initial commit of this repo.

• After you executed this message, your default text editor will pop up, check the comments inside.

  • The commits are listed with the oldest at the top.
  • Only commit hashes matter. If you apply squash (merge commits) or reword (edit commit message) command, you will be prompt to enter the new commit message once the rebasing process reaches there.
  • That is, if you want to squash the commits (merge 2 to be 1), the first line will be pick command, and the second line will be squash command.

Amend commit

git commit --amend

• This is similar to squash in interactive rebase git rebase -i, except that you don't need that complicating process for merging commits under this situation.
• Amend your previous commit, which means your new change will be merged into the last commit.

  For example, say you have the following changes and the commit message:
  
  ADD - New feature
  =================
  + feature.py
  + feature_base.py
  
  If I added a file called feature_adv.py and did some changes in feature.py, and it should be in the same commit of ADD - New feature:
  M feature.py
  + feature_adv.py
  
  Then once I execute this command, the change will then be
  
  ADD - New feature
  =================
  + feature.py
  + feature_base.py
  + feature_adv.py

• Commit message can be changed after you executed the command.
• This command is especially useful if I accidentally commit something that is a halfway work and I've done it and want to commit my whole work as one commit.
• Kinda alternative to the above. I can commit and push my halfway work as a "backup," then to amend and force push the commit once I've complete my works.

Rebase

git rebase <BRANCH_NAME>

• This might be used if you checked out from master at the 1st commit, but a commit was made to the master branch, and you haven't done working on your branch, but you want to synchronize the change on master into your dev branch.
  • Execute git rebase master if this happens.
    • Be sure that your working tree is clean (no files changed, the current status is as same as the origin) before you execute this to minimize the time on solving conflictions if any.
    • If you want to save your work, you may use PyCharm's shelf function or use git stash.
    • An IDE is recommended under this case (PyCharm again!) because you will find it easier to merge your changes when you're solving conflictions if any.

Usual Workflow

Modify the commit that you've already pushed

(This could happen quite frequently)

Scenario

You pushed a commit, and found that despite you passed the tests locally, GitHub Actions tests did not pass, and it's clear that you are the one responsible for it. After you applied some fixes, you want to amend the pushed commit.

Command

git commit --amend
git push -f

Explanation

• You, as usual, amend the commits locally first. (git commit --amend)
• Then, because that your local history differed from the one on the origin, you want to overwrite it, because your local version of the commit history is the real one. (git push -f)
• If you simply push it (git push), you'll find that you need to either commit your changes first then pull, then push, or to stash your changes first, then pull, then push. You'll find that eventually you are creating 2 commits instead.

Squash old commits

Scenario

You find that after you committed A, B and C, A and B are actually the same thing, and it should be squashed into A. However, all of these commits were already being pushed to the origin.

Command

git rebase -i HEAD~3

Then after your text editor pops up with the rebasing commands, change it from:

pick aaaaaaa A
pick aaaaaab B
pick aaaaaac C

to:

pick aaaaaaa A
squash aaaaaab B
pick aaaaaac C

Save the file, close the editor, then the text editor will pop up again, prompting you for the new commit message of aaaaaaa with something similar to this:

# 1st commit message

A

# 2nd commit message

B

Change it to what you want. Assuming we are going to change the new commit message as "AAAAA":

AAAAA

Save it then close the editor.

The rebase will then continue.

After you completed the rebase, run this:

git push -f

Explanation

• You initiate an interactive rebase, and rewrite the storyline of the commit. (git rebase -i)
• You typed in the new outcome you desired (changes in 2 pop ups of the text editor)
• Then, because that your local history differed from the one on the origin, you want to overwrite it, because your local version of the commit history is the real one. (git push -f)
• If you simply push it (git push), you'll find that you need to either commit your changes first then pull, then push, or to stash your changes first, then pull, then push. You'll find that eventually you are creating 2 commits instead.

Ability to select the routes to be displayed

Find a way to let to user select which routes to be displayed.

One solution is to add layers for each route, so the user can toggle it.

Get the hourly passenger flow data of each stops

Check with Megan.

Also, try to find hourly pedestrian flow, traffic flow data.

Smart stops deploying algorithm

Blocked by #22 (travel time), #23 (budget).

Develop an algorithm/small app to find the optimal stop deployment plan.

Factors to consider

Accessibility

• Travel time (shorter the better)
• Distance between the agent's location and the bus stop (lower the better)
• Cost for a bus pass
• Frequency

Operation costs:

• Budget
• Gas cost per trip
• Cost for each trip
• Optimal price for a one-time bus pass
• Driver salaries/scheduling...etc

Combine stops and measure the accessibility

Description

Try to merge the stops and observe the accessibility difference between before and after the merge.

Things may need to do

• Profile and optimize distance calculation code.

Find hourly pedestrian and traffic data

Continue from #6

If not found, halt on the features #2 that require hourly data for now.

Merge same stops with different direction to be the same

For example, Aberg & Loftsgordan (EB) and Aberg & Loftsgordan (WB) should be merged into Aberg & Loftsgordan.

Doing so, the stop removal feature to implement in #8 could be slightly easier.

Also, the rendered map will not have too many clustered stops which are essentially the same.

Possible workarounds

• Utilize the function of finding the stops within a certain range, and create a new stop at the center of these stops.
  • Better to create a new model and controller for these stops.
  • Try to plot a map once this is completed.
• Utilize primary_street and cross_location at C14 ([13]) and C16 ([15]) to group.
  • This could be faster because less calculation needed.
  • Need to do a sanity check to the data to ensure that there are no weird glitches in these fields.

Things to consider

• Need to create API and fields for accessing and storing the stops under the merged stops, because the ridership data correspond to these stops, not the merged one.
  • Ridership data could be summed for future access like approximating the time for heading to the stops.

Note

• Some stops were not grouped but essentially the same. For example, E Gorham St. and E Johnson St. is the same, but because they are single-way streets, bus stops/routes were being forced to separate, so they are counted as 2 stops instead of 1.

Commit message

Please follow the format at your best to save your future self some time.

Format

<PREFIX> - [ISSUE_ID] <MESSAGE>

Things inside <> is required; [] is optional.

Prefix

Prefix Abbr.	Prefix meaning	Description	Example
ADD	Add	You've added something.	New feature implemented.
FIX	Fix	You've fixed something.	A bug was being fixed.
IMP	Improvement	You've improved something.	Optimized a piece of code.
UPD	Update	You've updated something.	Refactored a piece of code or updated the README.md file.
DEL	Delete	You've deleted something.	Deleted a piece of temporary code.
OTH	Other	You've done something that cannot be categorized as any of the above.	Initial commit.

To clarify, the difference between IMP and UPD is that IMP brings some positive effects to the code (presumably); UPD only brings changes to the code (possibly positive, but not that explicit).

Message

Would be better to be < 50 words. However, if there's something that cannot be briefly described, it's fine to go over 50 words. Just don't be too much. You'll want to read the commit history as a story containing a lot of short sentences.

If your commit relates to an issue, mention it.

Examples

Added a new map-generating function.

ADD - new map-generating function

Fixed a bug in stop data parsing, which originally reads the data at column 3 as column 4.

FIX - Misread in stop data parsing process

Implemented a feature listed in issues as #87, which is to incorporate ML to automatically create an optimal route map for Madison.

ADD - #87 create optimal route for Madison

A bug was found after you published #87 which actually creates the worst route map.

FIX - #87 created worst map instead of the best map

Notes

• Additional reads: How to Write a Git Commit Message (& why it matters)
  • I am not doing the best practice according to this because I think that it might be too hard for us to suddenly adopt it. Also, it would be time-consuming for learning this, possibly losing the focus of what should we do now in the limited time.

• If you want to fix the error yielded by Github Actions, please amend the commit, squash it, then force push it to your dev branch. I believe that we don't want to see a commit history containing full of fixes that actually only do a small fix of a task. For how to do this, check #10.

Process stop data into a table

Continue of #22.

Description

Manually process the data into stop sequence data for easier access.

Example data table

route number	stop_id	stop_lat	stop_lon	direction	distance_from_last_stop	travel_time_from_last_stop	number_of_boarding
2	1234	56.78	90.12	S	0.12	12	50
2	5678	90.12	34.56	N	0.34	34	100

Columns to consider to be added:

• Frequency

Data storage method

1. One file for each route, example file name: Route#.csv

• Saves file loading time

2. One file for all routes

• Increases loading time, in-program sorting & parsing

About creating test cases

We use pytest for this project. Here is its documentation.

Rules of thumb

Every test related things will be prefixed with test_ for auto test discovery.

• Every test case will be a module-level function starting with test_.
• Every file containing the test cases will be named with the prefix test_.
• Every module containing the test case files will be named with the prefix test_.

For the module to be a module (and to be discovered), you need to have an __init__.py inside the module folder, regardless if it's an empty file (and it usually will be empty, except that you want to inject something).

Ability to remove a stop and show the projected result

Exploitable metrics

• Passenger Travel Time (to stops)
  • Create random agents and measure their traffic time from the place around the removed stop to the random destination.
  • Could replace "random agents" of the above with corresponding population data to increase the accuracy of the simulation. Reference
• Bus Travel Time #22
  • Estimate the chance of bus stopping at a certain stop to estimate the bus traveling speed.
  • Too many stops will increase the traveling time of a single trip for the bus.
• Budget Change #23
  • Need to find the budget data first
• Passenger Flow
  • If we can somehow get the hourly flow data, then this would be implementable. (#6 @yayen-lin)

Notes

• We need to determine how we will use the application (via CLI or Flask), then to implement this.
  • It's best to be able to compare the before and the after, so Flask might be preferred. However, it takes more time to implement than CLI (usually).

Population density interpolation plot

Secondary data reference

Madison Metro Annual Report

• Link

FTA

• Link
• glossary
• summary and trend

Transit Development Plan 13' - 17'

• pdf

Performance Measurement Report in 16'

• pdf

Fares

• Fares

(From Tyler)

• Human Transit
• Human Transit (AMZN)

Extra

• Bus stop spacing and location

Primary data reference

Bus Stops and Routes Info

View datasets

Metro Transit Ridership by Route

• Link
• Download shp
• Download csv

View columns

* OBJECTID (Unique ID)
* route_id (Number)
* route_short_name (Number)
* route_url (Text)
* Shape (Geometry) 
* Shape.STLength() (Number)

Metro Transit Ridership by Stop

• Link
• Download shp
• Download csv

View columns

* DotSize (Number)
* IntersectionID (Number)
* Lat (Number)
* Lon (Number)
* OBJECTID (Unique ID)
* Saturday (Number)
* Saturday_1_79 (Number)
* Shape (Geometry)
* StopDescription (Text)
* StopID (Number)
* Sunday (Number)
* Sunday_1_79 (Number)
* Weekday (Number)
* Weekday_1_79 (Number)
* WkdBdPerSvc (Number)
* WkdSvc (Number)

Metro Transit Ridership by Intersection

• Link
• Download shp
• Download csv

View columns

* DotSize (Number)
* InterDescription (Text)
* IntersectionID (Number)
* Lat (Number)
* Lon (Number)
* OBJECTID (Unique ID)
* Saturday (Number)
* Saturday_1 - 79 (Number)
* Shape (Geometry)
* Sunday (Number)
* Sunday_1 - 79 (Number)
* Weekday (Number)
* Weekday_1 - 79 (Number)
* WkdBdPerSvc (Number)
* WkdCash (Number)
* WkdSvc (Number)

Metro Transit Bus Stops

• Link
• Download shp
• Download csv

View columns

* address_range (Number)
* agency_id (Text)
* Bench (Number)
* cardinal_direction (Number)
* cross_location (Text)
* jurisdiction_id (Text)
* Light (Number)
* location_type (Number)
* OBJECTID (Unique ID)
* parent_station (Number)
* PK_MD_WE (Text)
* primary_street (Text)
* Reaitime (Text)
* relative_position (Number)
* Route (Text)
* Route_Dir (Text)
* Schedule (Number)
* Shape (Geometry)
* Shelter (Number)
* stop_code (Text)
* stop_desc (Text)
* stop_id (Number)
* stop_lat (Number)
* stop_lon (Number)
* stop_name (Text)
* wheelchair_boarding (Number)

Metro Transit Bus Routes

• Link
• Download shp
• Download csv

View columns

 * OBJECTID (Unique ID)
 * route_id  (Number)
 * route_short_name (Number)
 * route_url (Text)
 * Shape (Geometry )
 * Shape.STLength() (Number)

Daily Boardings Based on Routes and Bus Stops

View datasets

Metro Transit Ridership by Route Weekday

• Link
• Download shp
• Download csv

View columns

* DailyBoardings (Number)
* DotSize (Number)
* Lat (Number)
* Lon (Number)
* OBJECTID (Unique ID)
* Route (Number)
* Shape (Geometry)
* StopID (Number)

Metro Transit Ridership by Route Saturday

• Link
• Download shp
• Download csv

View columns

* DailyBoardings (Number)
* DotSize (Number)
* Lat (Number)
* Lon (Number)
* OBJECTID (Unique ID)
* Route (Number)
* Shape (Geometry)
* StopID (Number)

Metro Transit Ridership by Route Sunday

• Link
• Download shp
• Download csv

View columns

* DailyBoardings (Number)
* DotSize (Number)
* Lat (Number)
* Lon (Number)
* OBJECTID (Unique ID)
* Route (Number)
* Shape (Geometry)
* StopID (Number)

Pedestrian and Traffic Flow

View datasets

Pedestrian Flow

• Link
• daily.zip
  • more details see #13
• hourly.zip
  • more details see #13

City of Madison Traffic Flow

• Link
• City_of_Madison_Traffic_Flow.zip
  • more details see #13

View columns

* Day (Text)
* Direction (Text)
* Month (Text)
* OBJECTID (Unique ID)
* Station (Text)
* Time_Period (Text)
* Volume (Number)
* Year (Number)

Upper State St. Pedestrian Count

• Link
• Upper_State_Street_Pedestrian_Counts.zip
  • more details see #13

Traffic Flow Map

• Link
• Download shp
• Download csv

View columns

* AWT Count (Number)
* AWT_Yr (Number)  
* Count Year (Text)
* mslink (Number)
* OBJECTID (Unique ID)
* Shape (Geometry) 
* Shape.STLength() (Number)  
* SOURCE (Text)
* STATION (Text)
* Street Name (Text)

Other

GTFS Data

• Google Transit API Overview
• Link
• Download

"Syllabus"

Link

Class method implementation order

Reference: https://stackoverflow.com/a/36967648/11571888.

---

• Interface
• Properties
• Magic Methods
• Public Methods
  • Higher order methods at the top
• Class Methods
• Static Methods

Plots for travel time analysis

continue from #22

Desired Plot Designed (so far)

3 Plots For One Route

• Weekday, Weekend, Holiday

Line Plot Part:

Travel time from one stop to another

• This website can give us an intuition on travel time for each route.
  • e.g. route 2 seems to average 51 mins from one end to the other.

Split by 9 published stops

• We can look for each section and estimate the travel time between each stop using distances.
  1. Use haversine formula or euclidean distance because we're not looking at a very large region.
  2. Apply element-wise operation on Lat and Lon columns to optimize runtime.
  3. Then, we can look for:
     • stops that are too close to one another to improve on time-traveling of that route.
     • stops that overlap with other routes and have a less number of boarding to reduce traffic.

Bar Plot Part:

Calculate the number of passenger boarding

1. Use files from #1 (Daily Boardings Based on Routes and Bus Stops)
   • this will give us the average number of boarding per day
2. This link to obtain how many bus launches per day
3. Then we can estimate the number of boarding per bus launch
   • however, ofc we want to be more precise, but I haven't had an idea about how to manipulate the Traffic Flow data to distribute the number of boarding.
4. Approximating Boarding Time
   • Approximating how much time should be added to our Travel Time Line Plot
     • since if a stop has a larger passenger boarding, it'll take more time for that bus to take off.
     • I'm thinking + 30sec for a bus from slow down -> stop at a stop -> take off and speed up to normal speed
     • +3-5sec per passenger boarding with some noise (range from -1 to 1) added to it
     • (ignore cases of wheelchair boarding)

Last Step: Run Simulation For Our Approximation

• Since there are approximations when making our Bar Plot Part, we should run simulations, say 20, 50, or 100 times to get an average of boarding time and so on.