A scraper and reddit bot for the website MountainProject.com
This repository has three main parts:
The MountainProjectAPI (a 3rd-party API for MountainProject.com): https://github.com/derekantrican/MountainProject/tree/master/MountainProjectAPI
The MountainProjectDBBuilder: https://github.com/derekantrican/MountainProject/tree/master/MountainProjectDBBuilder
The MountainProjectBot (a reddit bot connecting MountainProject.com and the climbing subreddits): https://github.com/derekantrican/MountainProject/tree/master/MountainProjectBot
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
For https://www.mountainproject.com/route/107170320/the-mountaineers-route the bot is not parsing the "3rd" from the grade section
Every now and then, the bot makes an incorrect auto-reply. Recently this has happened twice. We could extend the "Monitored Comments" function to check for this. We should check for two things:
It is free, has C# examples, and is a xml-based database
C# examples: https://github.com/BaseXdb/basex/tree/master/basex-api/src/main/c%23
BaseX for Dummies: http://files.basex.org/publications/dummies/BaseX_for_dummies.pdf (also includes a "how to set up as a server to access from the outside")
Problems parsing special characters. For instance, the name of this route, Geflügelfrikadelle, is recognized as "Geflügelfrikadelle"
This goes for routes with characters like Japanese, etc too
Sometimes the DBBuilder hits an HTTP exception. For instance:
EXCEPTION MESSAGE: One or more errors occurred.
INNER EXCEPTION: System.Net.WebException: An exception occurred during a WebClient request. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
at System.Net.WebClient.DownloadBitsReadCallbackState(DownloadBitsState state, IAsyncResult result)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at MountainProjectAPI.Utilities.<GetHtmlDocAsync>d__6.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Utilities.cs:line 108
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at MountainProjectAPI.Parsers.<ParseRouteAsync>d__6.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 166
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 91
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
STACK TRACE: at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout)
at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
at MountainProjectDBBuilder.Program.BuildDB() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectDBBuilder\Program.cs:line 213
The suggestion would be to add a try-catch with retry (maybe with a limit of 5 or 10 retries) to the Utilities.GetHtmlDocAsync method
Every time we run the DBBuilder, we run through every route and every area on MountainProject. This is bad and might be putting some heavy load on MountainProject. Alternatively, we can just update the recently changed items. MountainProject actually has an RSS feed of recently changed items: https://www.mountainproject.com/rss/new?selectedIds=105708956&routes=on&areas=on
Title says it all
Title says it all
Add to the Google form a sort of "blacklist my username (make it so the bot won't reply to my comments)". Also add the functionality to the bot (should read from a text file on every iteration like RepliedTo so I don't have to make a new version to add another blacklisted user
"Route Guide" should not be listed as a parent
Currently the bot only gives the best matching result. Maybe, where the bot says I found the following info it could do something like I found the following info (there were 57 other results) Where "there were 57 other results" could link to some place (pastebin? something else?) with all the other results
The bot and dbbuilder should both have unhandled exception handling that - at a minimum - spits out the exception details to a file
In this comment the user mentions that they would like to see route grade conversions in the /u/ClimbingRouteBot. Maybe we can add a link to a page with conversions or something (not sure if I want to try to put the conversions directly in the comments as it will take up space)
Investigate https://github.com/sirkris/Reddit.NET to see if it's any faster than RedditSharp
Would be nice to show an "online/offline" status on the GitHub page based on whether the bot is running or not (maybe it hit an error and crashed, maybe the internet is down, etc). We can track this by sending a sort of "heartbeat/ping" every time we check for new comments
for instance: !MountainProject Moonstone Arizona should return https://www.mountainproject.com/v/moonstone/105960200 rather than all the other instances of "Moonstone" (other "keywords" could be "route" or "area" to restrict results to a route or area)
The text Helm’s Deep will return no results whereas the text Helm's Deep will return some results. This is because one is a right single quotation mark and the other is an apostrophe. Need to figure out how to handle this
The bot should be able to interpret some hard-coded nicknames. For instance:
Issue with parsing mixed grades (they're being recognized as a range).
Example: 5.11c/6c+
On this comment the text East Ridge returned a result for Northeast Ridges and Valleys (because it had a higher popularity value). Maybe we should filter results by popularity, then by closest string match?
In this comment the bot correctly shares the route, but it is missing the Aid grade of "A3" (https://www.mountainproject.com/route/108170851). This should also be added to the bot's comment
"Popular Climbs" should have hyperlinks to those climbs and have the rating next to them
If someone goes and finds the Mountain Project link themselves, the bot should:
(FYI in the future you can call me by saying`!MountainProject ___________`)
...typical bot response....
(if step 4 fails, then do nothing)
For example, if someone posts the link https://www.mountainproject.com/route/106085043/send-me-on-my-way then the bot should respond with
(FYI you can call me by saying `!MountainProject Send Me on My Way`)
Send Me on My Way
Grade: 5.9-, 75 ft
Rating: 3.1/5
Located in Bruise Brothers Wall, Kentucky
https://www.mountainproject.com/route/106085043/send-me-on-my-way
------------------
Feedback | Donate | GitHub | FAQ
With #23 we added the functionality to respond to MountainProject links. The condition, was that we would only respond if !MountainProject {title} would have returned the same result. I think we should make the following changes;
!MountainProject {title} would not return the same result, we will simply not add the (FYI in the future... line. (And, in the future when people are using the bot a lot, we can remove this line altogether)Nine Gallon Buckets [90 ft, 2 pitches]
Type: Sport
Grade: 5.10c (YDS)
Rating: 3.4/4
Located in Smith Rock, Oregon
https://www.mountainproject.com/route/105800412/nine-gallon-buckets
5 Gallon Buckets [60 ft]
Type: Sport
Grade: 5.8 (YDS)
Rating: 3.2/4
Located in Smith Rock, Oregon
https://www.mountainproject.com/route/105789060/5-gallon-buckets
-------------------------------------------------------------------------------
Feedback | FAQ | Operators | GitHub | Donate
/u/ClimbingRouteBot comments with the "Protection" section. Maybe we can do this also? I don't want it to take up too much space. Maybe we can just parse the important stuff like "# bolts", etc
DBBuilder should run automatically about once a week to generate a new xml. This should be done via the Windows Task Scheduler calling a .bat file.
In order to truly take advantage of this, the bot also needs to either re-generate the data from the xml (maybe once a day) or restart every day. The restart option might also fix any issues if the bot crashes one day, it will still start up again the next day on it's own (if the "restart" is done via Windows Task Scheduler not via the bot itself)
In addition to /r/climbing, the bot should be active on /r/ClimbingPorn and any other climbing subreddits
It might be useful to add some sort of "command line" parameters. Like !MountainProject Send Me On My Way -grade:YDS (would give the route grade in YDS or other systems) or !MountainProject Send Me On My Way -description would also give the description in the response
Parameters/Flags:
-description: bot response also includes the "Description" section from MountainProject-gettingthere: bot response also includes the "Location" (Routes) or "Getting There" (Areas) section from MountainProject-protection bot response also includes the "Protection" section from MountainProject (not valid for Areas or some routes)-grade:YDS (or Hueco)/French/Ewbanks (or Australia or NZ)/UIAA/ZA (or South Africa)/British(or UK) outputs the grade in that system (YDS is the default. "Hueco" should attempt to get the Hueco grade and, if it does not exist, get the YDS grade because it seems a good number of boulder route grades on MP are classified as YDS and not Hueco)-route or -area confines the search to only routes or areas-location:{area name} search results are limited to only routes/areas which are underneath the specified area (we should also change the bot's "nothing found" response to something like I couldn't find anything for "Send Me On My Way" in "Exit 38" for a search of !MountainProject Send Me On My Way -location:Exit 38). Another important note here is that the "area name" could have spaces or hyphens in it so we need to account for thatOn this comment the name of the boulder is the "Landjäger Boulder". I do not think a search for "Landjager" would have returned the same thing. But we should allow that
Implement some sort of progress indicator.
Where:
Where to get the "total":
MountainProject actually lists the total number of routes at https://www.mountainproject.com/route-guide
I haven't found a place where they list the total # of areas, but this might be good enough
Sometimes a route name uses a number (eg "5 gallon buckets") and sometimes it uses the "number word" (eg "Nine gallon buckets"). We should make it so a search for "9 gallon buckets" is the same as a search for "Nine gallon buckets"
Should only get search query after the keyword.
For example This is an example !MountainProject deception crags should just search for "deception crags"
Instead of an email, follow this tutorial to make a Google Form that will use a url parameter with the comment id
This reply is pretty scarce because there are no popular routes listed on MountainProject. Maybe instead we can come up with some ourselves based on the popularlity listing for each route
The /u/ClimbingRouteBot does this by looking for ".........{route name} {YDS grade}.........." in the title (https://old.reddit.com/r/climbing/comments/an8ylw/onsight_of_junkyard_511a_sanford_wall_foster/efsimuy/). I think we can do better
UKClimbing.com seems to be the biggest database of crags for the UK (but also seems to have data for all over the world). Maybe we can support this?
A search of !MountainProject East Ridge -location:Wolfs Head resulted in I could not find anything for "East Ridge " in "Wolfs Head". Please use the Feedback button below if you think this is a bug
What it should return: https://www.mountainproject.com/route/105848762/east-ridge
We get the name of areas from the "left side panel" of the parent area. The reason we do this is because the area name (in the large header) often contains the words "_______ Rock Climbing" or "_________ Climbing". The left side panel does not contain these words. But sometimes it can contain other random strings. For instance, on this page the areas are listed with the prefixes "A:", "B:", etc. We should either try to clean up these strings from the left side panel or get the header string and remove the "Rock Climbing", "Climbing", or etc strings.
Here's another example of random strings in area names
And here's another one (note the "14 -" in the header)
This comment quotes a reply from before and the bot had previously responded (until I deleted the response). That is because the quote included the MP link, meaning that the bot responded to the same MP link multiple times.
We should have some sort of extra step where we filter out anything that matches a regex like >[^\n\r]*mountainproject\.com to avoid things like this
Sometimes people edit a comment (for instance !MountainProject Angel Dnyo could be edited to correct spelling !MountainProject Angel Dyno) or delete comments if the bot gave the wrong response. Maybe we could monitor comments for the next 30 minutes or so and edit the response based on changes to the comment (ie if the parent comment is edited, we could edit our reply to match the edit or if it is deleted we could delete our reply)
Instead of just using the current grade, we should show multiple grades. Like:
Grade: 5.9- | 5c | VI
(or something like that)
Some other thoughts:
-grade operator maybe we should list that one first in the list
Release the MountainProjectAPI project as a nuget package
For instance, in https://www.reddit.com/r/climbing/comments/715awf/red_rock_season_is_back_cruisin_up_yak_crack_511c/ the user mentioned that they are climbing "Yak Crack". While there are some results with that name, the real result is actually "Yaak Crack". We should implement fuzzy query matching so that "Yaak Crack" would also come up in the results list.
It seems like there's plenty of times where there's a boulder that has multiple routes, but the boulder also has the same name as one of the routes.
Examples:
https://www.mountainproject.com/area/106657514/alexisizer
https://www.mountainproject.com/area/111818525/the-checkerboard-boulder
https://www.mountainproject.com/area/105867832/waimea
In these cases, if there is a sub route that has the same name as the area (and we haven't forced the "-area" operator) then we should prioritize that over the area
If someone uses the -location operator, that location should be referenced in the response.
For instance:
!MountainProject Send Me On My Way -location:Red River Gorge response should say "Located in Red River Gorge, Kentucky" not "Located in Bruise Brothers Wall, Kentucky" (which is what it currently does)
"Australia" the continent is the same as "Australia" the country. So we should have a special exception for results located in Australia to not filter them the same as other countries (see this comment where the location probably should have read "Sublime Point, Australia")
See this thread. We should see if instead of returning location text as "[immediate parent], [state]" we can return "[second parent], [state]"
MPBot needs an extra definition of blacklist level: bot can reply to my comments and MP links but NOT to my posts (eg dpotter)
Currently the program is very slow. One way to drastically improve speed is to make methods async
Parse route height as a new variable and allow it to be converted into meters
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.