This project forked from fabacab/fetlife-aslsearch
Tampermonkey user script offering an interface to perform pseudo-automatic searches of the FetLife.com user base filtered by age, sex, location, and role.
Currently Fetlife ASLsearch reborn uses Tampermonkey scrupting. It might be worth investigating turning it into extentions.
TODO
Moving fetlife ASL search into an extention gives us some extra options.
One of which is spoofing the referer header which would solve #15. Another one maybe implementing some sort of verification of the plugin/extention, to prevent users sending corrupting scraper data.
Currently one can select columns to display before search and afterwards using "datatables, colvis". selecting tables beforehand leads to strange behavior of colvis
TODO/BUG
Besides the above its redundant with colvis working as it should and leads to needless codebloat. Keep It Simple Stupid.
Please enter the steps to reproduce the bug or behaviour:
Location scraped wrong before its send to the extended search database.
The current (newish) format fetlife uses is:
/p/Country/Region/Locality
Where not every region has a locality.
When there is no locality set, the current scraper saves the country as region.
Isn't tested yet with firefox, got reports it might not be showing up, needs testing.
It includes A LOT of links to personal donation pages and homepages, those needs to be cleaned A LOT
The readme is kinda bloated.
Some content is not needed on the frontpage and should thus be moved to the wiki
Legacy search is not working, it doesn't have priority but should either be removed or fixed.
Some pages (for example users at a certain location) are not getting scraped
BUG
It seems the classes and ID's are different, it seems there are two wholly different "fetlife" formats. The second format has no scraper code
Please enter the steps to reproduce the bug or behaviour:
1.
2.
3.
With the transfer to SQL also comes the need to transfer the current database.
Although we currently have a 4,5 milion entry database within SQL, the resolution (number of colums filled) is a lot higher and it's a lot more up-to-date with the freshly scraped data
TODO
Clients that have manualy copy-pasted the plugin, do not auto-update the plugin
BUG
Please enter the steps to reproduce the bug or behaviour:
Colvis can be used to decide which columns to hide. Thats usefull.
TODO
at 0.5.6 default visibility is decided by the settings referenced in #39 as those are removed default visibility should be set using datatables colvis to prevent showing less used data
Please enter the steps to reproduce the bug or behaviour:
1.
2.
3.
The currently used colvis plugin for datatables has been depricated for ages now. Needs to be replaced with its replacement "buttons"
Currently location search can only use one location (either 1 city, 1 region or 1 country) without comma's. It would be nice if it supports the whole fetlife way of writing down location "City, Region, Country" and subsets of that.
As FAADE has been removed, it would be prudent to remove the "report" feature as well.
Print was not giving a nice printer popup and what did came out (pre 0.5.5 ) was ugly
Column name for city and province/state is wrong.
BUG
Please enter the steps to reproduce the bug or behaviour:
There is a warning in the code about paying members having high chances of being rapists. Personal discrimination based on what organisations you financially support, is NOT okey and might be illegal under some jurisdictions.
FAADE is already removed due to being illegal in A LOT of jurisdictions.
On chrome Fetlife seems to slowdown-freeze on certain pages when running ASLSearch.
I am not fully sure ASLSearch is the culprid, but this has to be noted down.
Pages where it happens:
Pages that are fine:
It seems google is going to block more mixed content.
This might create a need to move the frontend from a webserver to the plugin. I cant be sure our current iframe workaround is going to keep working
TODO
Currently errors when searching are not reflected in the actual UI.
TODO/Feature?
When searching is done, it would be nice if the "please wait" would be removed, the bar blinking red (instead of the green when results come in) and an error message being displayed.
Please enter the steps to reproduce the bug or behaviour:
Currently everything is hosted on google.
This needs to be moved to a node.js/sql server.
I'm currently working on getting a server to run it on.
Sadly enough this takes time and money.
ETA of 0.6.0 going live is between August and December
TODO
This is more of a "private-ish" note and although no one can do much about it i'm afraid, I do take care to be transparent about it.
Done:
Currently working on:
Low priority:
Currently the Extended search features has been move to a link due to fetlife not allowing (=blocking) the use of Iframes. We should try to find a way to get that back in.
Currently the user pictures are not showing in the search results.
Possibly due to some security changes done by fetlife.
A lot of the code is not commented correctly or at all.
This should be checked and corrected.
Location and Role being case sensitive is odd at best. No reason for it to be case sensitive.
Data storage for the server side is currently a mess of google sheets documents (no this is not a joke :P ). This should be migrated to a more platform friendly solution (and keep kinkdata out of google.
It would be neat having decent PR and issue templates and having the guidelines and licences fixed.
When opening ASL search, it disfigures the fetlife website (primarily it moves the top bar of fetlife down).
While it is working now, this should look WAY nicer.
Old server seems to be either down, broken or outdated.
New server needs to be setup before non-bot search will start working again!
Previous maintainer never got export buttons to work. Would be nice getting those in.
I removed some of the most annoying donation spam already.
But it need more love
The plugin should just have one request for donations at all (maybe with two buttons).
Current live version doesnt seem to show nickname
BUG
Possibly due to hiding of the avatar
Please enter the steps to reproduce the bug or behaviour:
The "filling" loadingbar has been replaced with a "pre filled" bar including the text "please wait". (see #27 )
However, some of the code doing the filling remains and needs to be cleaned
Sleazy and Greasy fork can use an auto-updated markdown. But the current Readme features too much content, those sources should just be a "info" page, not the whole readme
TODO
Also it seems pictures don't go well with Sleazy and Greazy fork.
Please enter the steps to reproduce the bug or behaviour:
1.
2.
3.
Loading bar does not represent actual search progress and it confuses the crap out of people.
It should either show the actual progress or be a different loading bar (with a small bar going round in a big empty loading bar, like the old windows "wait" bar).
Dear Ornias,
I have an idea.
Since this script didn't work for me, and I'm a developer myself, how about we make a separate searchable database of fetlife accounts?
Basically, the tampermonkey script would only parse the fetlife pages that the user opens and scrapes all the information that the user sees about other users - page URL, age, avatar URL, sex, kink, etc and then POST to a remote database.
Then, the actual search would be against that database (without crawling fetlife). It would be super-fast and almost accurate.
Fetlife will not be happy about this, but we could host this on some bulletproof hosting.
I'm a PHP/Mysql developer and can help with the backend.
Currently it would be possible to flood the API with requests.
While it should be able to handle significant API requests, it should not be floodable.
Best would be two limits:
Limiting searches to 2 or 3 per minute per user would be fine.
The POST API could get more load, but also closes the connection way faster. I think 25 per 5 minutes would be prudent
This would not prevent against DDOS however, DDOS protection could be added by limiting the connections to the SQL server and using decent networking solutions. That way the service would still go down, but with minimal damage to other services.
Suggested plugin for per-user rate limiting:
https://www.npmjs.com/package/express-rate-limit
TODO
Input validation and sanitization is a mess, at some needed spots it isn't there and in others its halfway done. This is a security issue
TODO
For PUSHed scrape results we should:
For GET requests we should:
When switching tabs while searching, the search seems to have issues completing sometimes.
Some sources of user data are not scraped. It would be nice if we could scrape more data with every pageview.
Feature
For example: replies in a thread, posts on a wall, or friend avatars to the left of a profle
Please enter the steps to reproduce the bug or behaviour:
On the 4,4 milioen row test dataset, it is possible to still create very slow queries. This should be prevented, as it also puts load on te server and facilitates abuse.
BUG
Most users would not be affected by such heavy queries, UX expectations are that people would almost always enter a location, gender, age-range or role. This would in itself ensure people at least hit a index.
On the User side of things a Timeout would occur after (at time of writhing) 20 seconds. However this does not cancel the executed SQL query on the server side.
While it would always be possible to DOS a system if willingly, users should be prevented to enter index-less search queries. We can do this by enforcing the use of at least gender and age.
We can also prevent abuse by stacking such queries by limiting the number of concurrent connections per IP, how this could be done in practice is still up for debate.
Please enter the steps to reproduce the bug or behaviour:
Currently some HTML or JS files also contains JS, CSS and/or HTML, this should be cleanly seperated in the future. The plugin might be the exception when it comes to HTML in a JS file, but should be structured more cleanly. Even in that case CSS isn't needed in a JS file.
The arrival of the Node.js based server, also comes the possiblity for local hosting and local development environments. A clear wiki article on how to set it all up should be made.
TODO
TODO/feature
Currently strings are not actually checked themselves. For example "shit" could be send as a country, and "dfrdefhydfh" as an url. We should sanitize those. Country could use the known country list used by the search form and maybe this could be merged with #26
The hosting platform for the server side is bound to google scripts.
For now this is fine, but should be rewritten to be more compatible with alternative hosting solutions. This also makes CI integration easier.
Project frontpage is a tad ugly (aka: Not 2019 material)
Feature
Please enter the steps to reproduce the bug or behaviour:
When you click a hyperlink on a nickname in the searchresults you get a google error page instead of the fetlife profile
BUG
Please enter the steps to reproduce the bug or behaviour:
1.
2.
3.
Group member lists are not getting scraped, it stops at the name of the first member
BUG
Please enter the steps to reproduce the bug or behaviour:
Currently avatars on discussions and profiles (wallposts, friendlists etc) get scraped. Avatars on personal homepages do not
BUG/TODO
The way those are setup is pretty close, except those on the homepage are inside a table and for some reason jquery is throwingup trying to process the same content inside a table.
It would be nice to scrape those, MOAR data.
Please enter the steps to reproduce the bug or behaviour:
It is unclear how to use the expressions and the mysql wildcard search since the example is not working
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.
