tgxn/lemmy-explorer

Instance and Community Explorer for Lemmy

lemmy-explorer's Issues

Crawler to Re-Try Failures

Failed instance polls should result in a re-try 1 hour later.

If they fail again they should be added to a "failed connect" table and excluded from jobs.

Instance Lists

Create Lists

• Blocked/Banned Servers (servers that appear is banned federation servers)

• Allowed (servers that appear in the allowed servers)

• Default Servers (servers that appear to be trusted/big/good moderation/no spam)

Feature request: Setting for Lemmy instance to open

Currently the lemmy communities open on their respective instances upon clicking. It would be great if user was able to provide their instance of choice to open all communities in, so that upon clicking on [email protected] the website would open with URL users-lemmy-instance.tld/c/[email protected] rather than lemmy.server/c/community

Happy to create a PR for that today or tomorrow.

Communites affected by the remote moderator bug are not showing in search results.

Hi, the following bug is making it so affected communities are not showing in search results.

LemmyNet/lemmy#3075

For example, the community https://lemmy.ml/c/ps5 no longer shows in search results as the site now returns a 404.

Until the bug is fixed in the next backend update, this community can only be found at https://lemmy.ml/c/[email protected]

Other communities are affected by this as well.

dont re-scan other known fediverse servers

like mastodon etc

Communities vanishing from the browser

Hi!

There seems to be an issue with the community browser where some communities get removed although they still exist on the instance.

I am not sure if this problem is with the specific Lemmy instance, the specific community or the lemmyverse browser and I am not sure how to troubleshoot it.

The community is feddit.de/c/cozygames.
It definitely did appear on the browser before.

Other communities from the feddit instance are still able to be found with your browser. On feddit I can find the community when I look for "local" or "all" communities.

Thank you so much and I hope this is posted to the correct place. :)

Lemmy Server Location

Ideally ability to filter communities by Country/Server Location would be ideal

Data Ideas:

• Perform IP Lookup on Server (CF/CDN can get classisifed as "Global"
• Perform ping from crawler - Would only give "distance from Australia"
• Use data from Feddit Explorer - They have time basic ping response times available.
• Use a service/lambda to ping servers from multiple locations

Save Filters/Options to Browser localStorage

should re-hydrate these settings when page is reloaded

[Feature] "Change View" Button to switch to an advanced Datatable

Add Image Credit

Logo made by Andy Cuccaro (@andycuccaro) under the CC-BY-SA 4.0 license.

Bug: Pagination doesn't hide last page data before loading next page

This causes the images to not be replaced immediately, only after the new ones load.
Less than ideal.

Bug: Crawler Data Normalization

• BaseUrl's should be transformed to lower case, spaces removed.

• should have no @ - strip if anything before
  

• Before scanning url, ensure it has no spaces, and no https:// and is not * and aklso not blank.
  
  

Certain communities does not show up on search bar

Hi
I tried to search certain communities like https://lemmy.ml/c/bitwarden or https://lemmy.world/c/nvidia . But, it does not show up. Is it a bug?

banner caching

they are pretty slow to load from all these federated servers...

should process and cache the banner

• download banner,m proces to correct dimensions
• minify to jpeg etc
• store ???

[Feature Request] Filter instances by country/language

Like title says, new filters for country or language

Export JSON or CSV

Is there a way to export data (instances or communities lists) to a JSON or other format file?

Communities should be primary page

not instances

Community data payload is too large

Hello
I tried to search the community (https://lemmyverse.net/communities). But, it stays on this screen and I cannot search anything. Is it a bug?

Missing kbin communities?

Hey folks,

I searched for "squaredcircle" and got:

This seems to be missing the SquaredCircle community on kbin: https://sh.itjust.works/c/[email protected].

Any idea why it didn't pop up? I have 'selected all' instances. Do kbin communities not show up?

Bug: Search results don't reset to 1st page

If you are not on page 1 (e.g. Browse to page 2 of community results) and then enter a term into the search box which produces less than 100 results, then the screen just shows a blank result set. You can navigate back to page 1 to see the results, but it's not immediately obvious to the user what the problem is.

Store Instance (Community?) Languages

Allow filtering by Languages

Overview Page (Instance/Community)

like this one https://the-federation.info/platform/73

Improve the display of communities with no banner

Community Card needs to show more/different details when an instance has no banner image.

https://lemmy.tgxn.net/comment/27851
https://lemmy.tgxn.net/comment/27886

Maybe also show more of the description or additional stats?

[Feature] Filter should have debounce

Add debounce to the filter search so it doesn't try to update every added character while the user is still typing.

looks like some bad data in federated instance lists...

Provide an option to filter communities by instance (with both whitelist and blacklist modes)

For example, have a searchable list containing all instances, with each list item able to be selected. Another drop-down or checkbox allows the user to choose whether to use the selected instances as a whitelist or blacklist.

Provide an option to filter communities to show only NSFW communities

Trending communities

It would be cool if there was something like a trending section. Trending communities could be the communities that have increased activity in the last x days. Or something like that.

Lemmy JS SDK Integration

Allow Configuring Instance/Username/Password to determine user subscriptions and allow users to one-click subscribe on their instance.

Error when filtering instances or communities

I had to open the site in incognito mode because I couldn't access it anymore because of this bug.

As soon as I start typing in the filter input field the website turns completely white (blank) and I get this error in the console:

Uncaught ReferenceError: retinstancesurn is not defined
    e https://lemmyverse.net/bundle.js:5
    T https://lemmyverse.net/bundle.js:5
    T https://lemmyverse.net/bundle.js:5
    Ki https://lemmyverse.net/bundle.js:2
    useMemo https://lemmyverse.net/bundle.js:2
    Sb https://lemmyverse.net/bundle.js:5
    xi https://lemmyverse.net/bundle.js:2
    Ls https://lemmyverse.net/bundle.js:2
    Ml https://lemmyverse.net/bundle.js:2
    yu https://lemmyverse.net/bundle.js:2
    gu https://lemmyverse.net/bundle.js:2
    vu https://lemmyverse.net/bundle.js:2
    au https://lemmyverse.net/bundle.js:2
    ou https://lemmyverse.net/bundle.js:2
    S https://lemmyverse.net/bundle.js:2
    Y https://lemmyverse.net/bundle.js:2

As soon as I got this error I was unable to revert to the site without filtering. Deleting cookies didn't help. Only opening it in an incognito tab.

Additional Crawler to Update Old Instances

Need to track when instances and communities were last updated and schedule jobs to update them.

[Feature Request] Suggest communities for user

This project has a great opportunity to help with one of the most requested features for Lemmy.

Many users are saying that there should be a way to somehow combine same-name communities across instances.

• Add new section to website that has a field for @[email protected] or https://instance.tld/u/username
• get / scrape communities that user is subscribed to
• return list of communities with the same name on other known instances.
• (option to hide/show already subscribed-to communities)

Something like this would be an awesome tool that would greatly help with discoverability.

page for "choose best server"

allow people to quickly navigate to an appropriate server they could join

use methods to determine if it's "good"

• ranking in federation (not appearing on any known blocked instances)
• ranking by active users (not top 3 by users?)
• randomize suggested server for each visitor

Public API

Hello! 👋 great work on Lemmy Explorer! This is really more of a question/request than an issue, but I was curious if the API is able/allowed to be used within other apps (ie as the search functionality in a lemmy client)?

Negative search terms in community search

e.g. a search like "cats -dogs" will show communities that have "cats" in their searchable fields, filtering out communities that have "dogs" in their searchable fields.

[Feature Request] Display/grab lemmy version in instance search

It would be nice to see the version of lemmy running on an instance to gain a general idea of the maintainer's interest level.

Sustainable Data Distribution

I want to publish:

• Redis data dump
• data/**.json exports

Somewhere other than Git, as they don't need long-term versioning.

I also don't really want people pulling data from CloudFront.

Currently thinking an S3 bucket for job storage, and publishing the dumps to GitHub Artifacts or Releases?

[Feature Request] Add Universal Link to open remote communities in a user's home instance

Add a Universal Link option in the format of /c/[email protected] so that communities will automatically open in the user's home instance.

Currently there are 2 options on a community listing:

1. A url to the community's home instance
2. A quick copy function for ![email protected]

The first is great for locating a community's source, but bad if the community is located on an instance other than your home instance.

The second is a remedy for the first's issue, allowing a user to copy the community string so they can place it in their home instance's search field

My recommended 3rd option would skip over the copy/paste step mandatesld by option no.2 , and allow the user to immediately open the remote community in their home instance. After which, the user can just press subscribe.

Lemmy instance not showing on list

Hello! I stood up a Lemmy instance on June 25th. I’ve been checking for a few days and it hasn’t shown up here yet. Is there anything I need to do on the server for the crawler to discover it? Instance name is techy.news. Thank you!

[Feature Request] Allow selecting of a "home" instance and show appropriate URLs to other communities

One of the biggest issues with Lemmy right now is that it's not easy to browse communities from other instances and subscribe to them in one-click. More often than not you can only browse an instance's communities directly from that instance, in which case you have to copy the [email protected] identifier and search for it on your home instance. It's quite a chore.

This site is wonderful for searching and browsing other instances and communities, but it suffers the same problem - you have to do the copy->paste->search dance to join the communities displayed.

Ideally, you should be able to pick whatever your home instance is (Or even just type in the base URL) and all of the various urls will be displayed as the https://my.home.instance/c/[email protected] format instead of just the [email protected] format.

It'd be a simple change but make finding and joining new communities 2 clicks instead of several.

Mobile UX (Android/iPhone Testing)

https://reddthat.com/comment/49831

Appears to have issues on mobile.

I also noticed issues with formatting on Android (Fields need to be responsive)

[Feature Request] "bot-infested" instance detection

There is currently a huge amount of bots signing up on smaller instances, especially instances with no captcha + no e-mail verification.

It would be quite useful to be able to detect and filter for such instances on lemmyverse.net, perhaps by checking for discrepancies between user counts and post counts? Or maybe by checking for instances with massive user growths but without a similar growth in post count?

NSFW communities no longer returned from `/community/list`

It looks like that since lemmynsfw.com upgraded to 0.18.0, I noticed it no longer returns NSFW in the community list.

I adjusted the data exporter to output nsfw communities from >24 hours ago, as an interim till there is a query param to support this. LemmyNet/lemmy#3363

Can we flag https://lemmit.online/ as as suspicious instance?

Can we flag https://lemmit.online/ as a suspicious instance? It's an instance run by a bot and it mass reposts stuff from reddit which is basically spam. I don't want to see communites from this instance while searching for sublemmies because sublemmies on this instance are VERY LOW quality and you won't get any engagement when you do interact with those communities by commenting on stuff.

Or could we at least get an option to block choosen instances by ourselves so user can decide what he doesn't want to see?

Allow filtering for communities which are officially endorsed failovers from reddit

This way someone who wants to find which community a subreddit migrated to, they can easily pinpoint it.

Official lists are being compiled in places like: https://www.quippd.com/writing/2023/06/15/unofficial-subreddit-migration-list-lemmy-kbin-etc.html

Also allow filtering by reddit community. So I can search for "/r/piracy" and find [email protected]

Show "Last Updated"

• For overall site (when was the json updated) ("Last Crawled")
• For the Individual Instances/Communities ("Data Age")

Sorting/Scoring System For Instances

Discussed in https://github.com/tgxn/lemmy-explorer/discussions/23

^{Originally posted by tgxn June 14, 2023}
Because we need to determine if an instance is "good" there needs to be a way to score each instance based on data we have about it.

Currently, my thinking/implementation looks at the lists of federated sites, and scores each instance based on the amount of other instances that refer to it (in the linked, allowed and blocked lists).

Scoring is applied by the following rules:

Instances

      let score = 0;
      if (linkedFederation[siteBaseUrl]) {
        score += linkedFederation[siteBaseUrl];
      }
      if (allowedFederation[siteBaseUrl]) {
        score += allowedFederation[siteBaseUrl] * 2;
      }
      if (blockedFederation[siteBaseUrl]) {
        score -= blockedFederation[siteBaseUrl] * 10;
      }

Communities

Uses the same base score as instances, and then adjusts based on a posts per subscriber metric.

      let score = 0;
      if (linkedFederation[siteBaseUrl]) {
        score += linkedFederation[siteBaseUrl];
      }
      if (allowedFederation[siteBaseUrl]) {
        score += allowedFederation[siteBaseUrl] * 2;
      }
      if (blockedFederation[siteBaseUrl]) {
        score -= blockedFederation[siteBaseUrl] * 10;
      }

      // also score based subscribers
      score = score * community.counts.subscribers;

These rules are obviously not ideal, as I'd need to run some more analysis to determine if they are tuned correctly.

I'm also thinking that it might be worthwhile to log an "uptime" or "first seen" score also to determine if it's been around/up for a while.

Additional Overview Page

• Show Totals (Instances/Communities)
• Show All Fediverse service counts
• Show Lemmy Version Breakdown
• Show Total Comments/Posts across all instances

Home instance doesn't work with kbin

When selecting kbin.social as a Home Instance, the generated links are not working, as kbin operates with /m/ instead of /c/ in the URL. So for example, the default generated https://kbin.social/**c**/[email protected] will not work, it needs to be generated as https://kbin.social/**m**/[email protected]

Out: Roll-up the total communities for each instance for display on the instance view