neocities/neocities

Neocities.org - the web site. Yep, the backend is open source!

NOTE: THIS IS NOT FOR NEOCITIES SUPPORT! Any issues filed not related to the source code itself will be closed. For support please contact: https://neocities.org/contact

Neocities.org

The web site for Neocities! It's open source. Want a feature on the site? Send a pull request!

Getting Started

Neocities can be quickly launched in development mode with Vagrant. Vagrant builds a virtual machine that automatically installs everything you need to run Neocities as a developer. Install Vagrant, then from the command line:

vagrant up --provision

vagrant ssh
bundle exec rackup -o 0.0.0.0

Now you can access the running site from your browser: http://127.0.0.1:9292

Want to contribute?

If you'd like to fix a bug, or make an improvement, or add a new feature, it's easy! Just send us a Pull Request.

1. Fork it (https://github.com/neocities/neocities/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

neocities's Issues

Follows and Followers detailed site view/browse page

A lot of sites have started following other sites they like:

https://neocities.org/site/steampunkdream

We should make a page that shows the full list of followers with large images, so it's easier to surf them. This should probably be implemented using the /browse view.

Design Neocities shirt

Per #15, we need a shirt design.

Github webhook deploys

Allow users to deploy sites using github repositories via web hooks.

WebDAV on Windows 7 may not be working

A user has reported that they were having trouble mounting WebDAV on their Windows 7 machine.

I tried to create the new network drive, and entered
https://neocities.org/webdav
as the network address to use.  Windows tried to connect to that address for ~15 seconds, and then reported that "The folder you entered does not appear to be valid."

I've tried debugging this for a while, but so far without any luck.  I can connect to the WebDav address using my browser, but not by adding a network location or mapping a network drive.

Relevant links:
http://www.onemetric.com.au/Documentation/Mounting-A-WebDAV-Share-Windows-7
https://social.technet.microsoft.com/forums/windows/en-US/67c361c0-ed6a-4e82-81df-da149c96492b/webdav
http://support.microsoft.com/kb/2560598

Mapping a network drive didn't fix this either, as the third article suggests.

Build Vagrantfile for development

This is the genesis of #24.

Build a Vagrantfile that gets the Neocities site running for development usage in a vagrant box.

Site browse algorithm ideas

Copied from old repo:

The trick to sites like Reddit and Hacker News is that they have a good algorithm for ranking content:

http://amix.dk/blog/post/19574
http://amix.dk/blog/post/19588

I would like to develop an algorithm for NeoCities site ranking, in particular for the browse page: http://neocities.org/browse.

I'm not sure what the best approach is here, and to be honest, I really suck at algorithms. We have a very unique situation, and solving it will require a pretty unique algorithm. Also, there's a lot of subjectivity here. So I wanted to post this and see if anyone in the community could come up with a default browse algorithm that would work well for NeoCities web site browsing/highlighting.

Currently, we just show the last updated sites. It's not ideal because it punishes sites that don't update all the time (it rewards quantity over quality), and there are already a few people that are gaming it using a script that regularly re-uploads the same HTML to the site. I expect the gaming problem to get progressively worse as the site expands.

It's a bit different than Reddit and Hacker News, in the context that posts there are time-based, and slowly disappear from the front page based on how old they are. We need to come up with an algorithm that doesn't neccessarily punish sites for being old (on the contrary, we want to better promote sites that aren't just "flashes in the pan"), shows sites that are trending in popularity, but is sufficiently dynamic that it doesn't give them a permanent placement on the top so that other sites have a chance to be seen too.

I have the following information I can potentially derive from:

• When site was created
• When site was updated
• How big the index.html is
• How much disk space the site is using
• How many hits the site has gotten
• How many files the site has
• Whether the site provided an email on creation or not
• When files were created
• When files were updated

Longer term, more information will be available. We are currently working to build out more social functionality on the site, which will allow users to follow/like sites, comment on changes, and make it much easier to share things. It will add this information to the pool:

• Number of follows a site has
• The date those follows occurred at
• How many comments a site change has received
• How many likes a site update has gotten, and when those likes occurred
• How many times a site has been shared
• How many tags a site has
• How popular those tags are
• The last time a site user logged in

Ideas/formulas welcome. CCing @amix on this because this is clearly something he thinks about a lot, and could potentially have some advice/pointers/formulas/book-recommendations to contribute!

To make things interesting, I'm offering a 0.5 BTC bounty (or the same amount in USD via Paypal or Venmo) for any user that contributes an algorithm that becomes our default algorithm (the "special sauce" sorting). You will also receive credit for developing the algorithm somewhere, perhaps even on the bottom of the browse page.

And of course, the algorithm implementation will be open source because the site is open source. Feel free to send pull requests if you'd prefer to do it that way, though it probably makes more sense to start with a formula. Thanks!

RESPONSES:

@noelwelsh:

This is going to be a bit of a drive-by comment because I don't have a great deal of time. Apologies in advance.

Firstly, I think you need to define the problem more precisely. I would define it thus: "the algorithm should return a list of sites ordered according to the likelihood of the user finding them engaging." Very quickly you need to decide if you are personalizing this list per-user or not. I'm assuming you're not.

With this definition we can go places because:

1. There is a huge literature on measuring engagement; and
2. There is a huge literature on generating rankings.

For the former we can start by looking here: http://www.dcs.gla.ac.uk/~mounia/ (There's lots more work in the field. I just happen to have some familiarity with her work.) For the latter searching for "submodular diversity ranking" will return a lot of papers with complicated algorithms. Yay!

The number one problem I see is that you don't have sufficient information to get a useful measurement of engagement. If you don't have useful feedback you can't learn if what you're suggesting is any good. At the minimum you want to measure clickthrough. Better would be time-on-page (possibly normalised by page length). Solve this problem, even poorly, and we can them look at using this to generate rankings.

As for generating rankings, while the submodular optimisation approach has some very nice properties (e.g. it maintains diversity in the suggestions) there are simpler methods to start with. Let's say we have some measure of engagement. We can put a confidence interval on this measure using standard maths. Then just display the sites that have the higher upper bound on the confidence interval. You should decay the measure over time, widening the confidence interval, to allow for changing interests.

Hope that's useful.

@robsheldon:

I left a comment on HN (https://news.ycombinator.com/item?id=7538491) with a straightforward, naive approach that should work OK. Includes pseudocode.

@amix:

I don't have that much time to devise an algorithm, but I would recommend reading "Programming Collective Intelligence": http://shop.oreilly.com/product/9780596529321.do It has a ton of great examples that you can get inspired by.

I would also recommend keeping the algorithm simple and testing it on real data. If you look at both Reddit's and HN's algorithm they are really, really simple, but quite effective.

These are my 2 cents, sorry that I could not be of a bigger help.

Improve subscription upgrade error message

When upgrading, if a card has been cancelled, there was a user that believed the upgrade was broken, and then realized their debit card was revoked by the bank.

I am discussing with him to figure out what the behavior was so we can try to improve it. If the debit card is rejected, we need to be better about telling people that (but it's possible that it's as good as it gets - I'll follow up).

Implement spam management admin console

Right now, the "spam detector" is a word list that almost never detects any spam and sends an email to the contact address.

What we really need is a console for reported spam that quickly lets us go through reported and detected sites as an extension of the administration console.

I'm going to make the SPAM detector not send an email, but instead flag for the spam admin console.

Less noisy following/followers on news/site feed

On sites with a lot of follows/followers:

https://neocities.org/site/steampunkdream

We should condense the news feed so it doesn't take up the whole thing. Perhaps instead of a big list of follows/followers, we have something like "This user followed 20 sites, click here to view them" or something like that.

Inotify based web site files replication

Currently, our backup for web sites works as follows:

Every hour, a backup server rsyncs the primary filesystem to it over SSH.
Every week and month, an internal rsync is performed to syncronize the backed up files to a weekly and monthly copy.

This leaves a one hour loss window for backups. Which isn't too bad, but it also prevents us from being able to scale the fileserver to multiple copies for the purposes of distributing web site load and minimizing downtime when a server goes down. We need something more real-time in order to do load balancing.

There are a lot of great implementations of inotify for ruby and node.js. This allows us to use the operating system to discover when files change using an event-driven system, and then order a real-time synchronization of files to replica servers, providing changes nearly instantly. Using this, we can emulate most of the properties of a hot replica without having to resort to crazy custom filesystems that aren't designed for what we need (Gluster, for example, has reliability properties that make it nonideal for our usage).

More information on inotify is here:

http://en.wikipedia.org/wiki/Inotify

Git push deploys

Allow users to "git push" deploys to Neocities.

Ability to "delete" a site

There is currently no way to delete a site. The easiest way right now is just to delete your site files (a solution that did not appease the OCD among us), but now that there's site profiles there's a different context and this is a vaild complaint.

Emails for social notifications

Right now we send emails when comments are created on a site profile.

This needs to be extended to include comment comments and follows.

There needs to be an email settings control to manage this if people want to turn them off.

Backup day

Go through backups, check them, add redundancy.

I'm working on this today.

Code of Conduct

At some point, we may want to consider having some sort of code of conduct or anti-harassment policy. There would need to be one set of guidelines for site comments and conduct within the company/open source project, and a looser set of guidelines for individual sites since those are basically the user's own space.

• Let people know that if they ever feel unsafe they can contact us or flag the offending account
• Acknowledge that this can be a serious problem for social networks and tech organizations
• Point out that online threats and websites dedicated to tormenting an individual are illegal by US state laws
• Balance our existing open company and free speech goals with pro-diversity goals

Related to issue #18

Make this repository the parent repository

I think the reason why our check-ins to this project don't show up in our github contributions may be because this project is a fork.
https://help.github.com/articles/why-are-my-contributions-not-showing-up-on-my-profile/

I think we can fix this If we delete the original repository and specify this project as the parent.
https://help.github.com/articles/what-happens-to-forks-when-a-repository-is-deleted-or-changes-visibility/

Make life harder for SEO spammers

SEO spam is picking up, and it's only going to get worse. Unless we want to become full-time SEO spam filterers (I know people that work full time at companies doing this - we do NOT want to go down this route), we need to be smarter about preventing this sort of activity on Neocities in order to be efficient.

So I'm introducing two proposed changes to site policy:

• Email addresses will need to be verified before users can be listed on the browse page, since almost all of the SEO sites are created with bad email addresses. This adds a level of technical complexity to the login process that will increase the difficulty of automating this process.
• Sites will require at least two "site changes" across two days before being listed on the browse page.

Not ideal, but when you're small, you need to make concessions like this.

Web Site Tipping

Tipping allows users and ideally anonymous donors (see below) to send money to sites they like on Neocities.

We will be using either Stripe Marketplace or Balanced for this, deciding on the features, pricing, fraud detection abilities and the usefulness of their client library.

The tip page will be accessed in the form of /site/:username/tip, which is a generic page that is attached to the site's Neocities profile. To mitigate fraud, there will be a clearly printed note on the site that the tipping page is not for purchasing anything and that they should not expect to receive anything in return for the tip, and that we will not be liable for any such claims. That way, someone doesn't start a "buy stuff" store using Neocities as a fraud channel, and we keep it to tips only.

Fraud is a serious concern. When someone uses a stolen credit card to send a tip, that charge will be reversed and a ~$25 fee will be assessed to Neocities (thanks, credit card industry, we love you too). In order to mitigate this risk and pay gateway fees, we will need to collect a percentage of the tip for Neocities internally, which I'm thinking will be around 20% initially but hopefully lower in the future once we get a large enough pool to mitigate fraud risk. We may also need to extend the amount of time until sites receive a tip to one month or longer, to allow for the charge to churn a bit. We will also need to limit the amount that is possible to tip to a low number ($50 per month per card?).

Bitcoin support would be nice, Paypal support would also be nice. And then maybe a comedy Alipay option or something. But I want to focus on banks/cards first. The second thing to support is probably Paypal, since that gives the tipping international capability.

This is a higher-ish priority, but I'm not labeling it high priority for now.

Use Vagrant to provision server infrastructure

When I was in Oakland, I met some awesome hackers that had come up with a great system for using Vagrant to deploy code to both development and production using shell scripts to do the configuration.

Implementing this will allow us to come up with a cohesive deploy structure that makes it easy for people to quickly get a Neocities stack running, and fulfills our goal of providing an "open source" backend infrastructure for public inspection.

Here's my lousy notes from the week:

Vagrant
make a vagrant_files dir with shell scripts in it
DEBIAN_FRONTEND=noninteractive turns off the ncurses derpiness
debconf-set-selections <<< 'mysql-server-5.5 mysql-server/root password again password blahblahputsomethinghere'

investigate `vagrant push`,find the blog post about it and production

System for managing DMCA takedown requests

Right now we can ban sites, but not really remove files on a site. This will probably be needed for managing DMCA takedown requests.

We have not yet received a DMCA takedown request, but you know, it's gonna happen.

Refactor password reset

It doesn't work very well, and fails for multiple email address from legacy sites. Refactor to reset for specific sites.

It also just turns the emailed token into a password. A better practice would be to allow the user to set a new password right away.

User name can contain spaces

On the account settings page (and during sign up process) site/username description says "cannot contain spaces" but the name can successfully be changed to one with spaces.

Visiting the URL of a site with spaces in the name url-encodes the spaces, doesn't give a 404 or other error, and never loads.

Implement two word tags

In UX observation, users get frustrated when not allowed to create tags with two words.

Restore Postgres secondary replication

An mistake was made in the configuration of the retention of WAL archives on the primary database, where the secondary database was down for long enough that we lost the ability to resynchronize the secondary database because the files were pruned too quickly.

The consequence is that we currently only have our SQL dumps to rely on incase of a catastrophic server failure, which could lose up to a day's worth of (non web site) data, such as account signups.

Unfortunately, it appears that this will require us to take the site down for maintenance to restore the database from a dump.

The wal archive retention has been turned up signifigantly to fix this, but it doesn't help us in this situation, as far as I understand.

Send password reset bug for bad email on site

[Neocities Error] Sequel::ValidationFailed: email An email address is required., email A valid email address is required.

POST /send_password_reset

Directories not supported

Directories cannot be created or uploaded. Additionally, attempting to drag-and-drop a folder into the upload area produces an unfriendly error.

Use case:
Doing local dev and making use of 'images', 'css' and 'js' folders as is best practice results in a site which is incompatible with upload to Neocities :(

Apostrophes not being escaped properly

@mythril reports:

Problem on the neocities file manager: if there is an apostrophe in a file name (drag and dropped some files) the delete function does not work via the web interface

It drops with a syntax error

It looks like the apostrophe is not being escaped

Right here, i think, though I'm not familiar with ruby:
https://github.com/neocities/neocities/blob/master/views/dashboard.erb#L153

Uploading an empty file is not possible

Description:
Uploading an empty file results in a "successful" upload, but the file is not actually uploaded.

To reproduce:

• Create an empty file "styles.css"
• Upload the file to the site

Has not been tested for other file types but was noted for empty css files.

Use case:
Uploading a "starter page" during our workshops includes a skeleton HTML page and a linked (empty) stylesheet. Uploading both files so that students can start coding directly in Neocities is not possible.

Workaround:
Add a comment at the top of the otherwise empty file before uploading.

Additional pricing tiers

While the current $5 plan seems like a good basic option for upgrading one's Neocities space while supporting the site, we should add some higher-priced tiers. I'm sort of intending these for tech community members who don't necessarily need hosting but want to sponsor the idea of free hosting for everybody else.

Tier ideas:

• $25/month - Everything in supporter, plus t-shirt (may not be practical to offer for international users)
• $100/month - Everything in $25 tier, plus (optionally) name or company name on homepage

Considerations:

• We could say they'll ship in 4-6 weeks to help with t-shirt cost flow, give us a better idea of demand, and remove t-shirt doneness as a dependency for announcing these tiers
• Sign up process needs to be updated to take shipping address, t-shirt size, maybe a checkbox for those who want to opt out of receiving physical items
• Victoria needs to do final tweaks on vector logo to prepare for vendor
• Could repurpose Cat Bus and Fat Cat names/graphics for these tiers
• If we can decide on t-shirt color, Victoria can make a little t-shirt graphic to put in the page. The plans section should probably be redesigned... maybe to reflect that these are donation tiers, not hosting plans... well, this will require some thought from me. Support page should communicate, "Help us bring free hosting to everyone! (and get cool nerdy shirt in the process)"
• T-shirt color decision probably depends on vendor decision. Kyle had a vendor in mind (which I've lost track of), but if we want this to be really easy we may want to look into a company that will not only print, but also pack/ship the shirts for us. Mac companies like BuyOlympia. (I've talked to them in the past - their minimum was 72 shirts, which would include women's sizes. I could get a quote)
• Ideally a sticker would be offered with all plans. Mac companies like stickermule which does awesome velvety die-cut stickers. This complicates order fulfillment though. Is there a vendor that is awesome with both shirts and stickers? EDIT: Actually, I think stickers complicate things too much. Maybe we hand them out at conferences later, or maybe I just get one made for myself lol
• For people who don't like subscriptions, I feel we could offer the t-shirt with one-time $100 donations as well
• Get testimonials from educators, etc so we can better explain the benefits of the project
• Should really have a section on the homepage for this... but different from what we had planned before. Less about pricing plans and more about why to support us.

Surf bar does not currently work due to iframe restrictions

We created this beautiful search capability for sites, but unfortunately it doesn't work on many sites.

The problem is that iframes are now strictly controlled by remote resources, which will prevent browsers from accessing them if they are sourced from an iframe.

There is a possible solution here that I don't yet foresee. The one idea I had in mind was to write some sort of proxy service that filters and processes sites for surf mode, which can then be used in this way. But there are logistical and security issues here I'm not sure I fully understand yet.

More research is needed on this. It would be awesome to bring it back.

Further reading:
http://stackoverflow.com/questions/7289139/why-are-iframes-considered-dangerous-and-a-security-risk
https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options

Tag browse does not show your own site in results after initial creation

When you have not updated your site yet, searching for a tag which your site has does not display it.

It is correct to not display it for other users until they have created a site, but they should be able to see their own.

Liability Insurance

We need liability insurance, specifically for legal issues.

Though under law we are safe regarding copyright issues as long as we remove content in time, there has been an incident I've learned about from conversation with someone in the Bay Area recently that is tied to activism with internet issues.

The story goes like this: an accuser sent a DMCA takedown notice to a hosting provider or a wiki, and even though the provider took it down in time, they just sued the hosting provider anyways. I was also informed in that specific case, it turned out that the accuser didn't even have a legal claim to the alleged copyrighted work (derp).

The case was thrown out by the judge, and the judge ordered the plaintiff to pay the defendant's court expenses. The problem is that they don't just write a check right away. In the mentioned case, the defendant had to take the plaintiff to collections and I believe they mentioned that they are still waiting to be compensated.

Until the plaintiff compensates us for expenses, we would be stuck with an upwards of $20,000 hole in our finances, which would require me to seek external funds, as we only have around $15,000 allocated to Neocities at the moment. Obviously if we were to lose such a case, the consequences would be worse.

Legal liability insurance allows us to plug this hole. In the event of a dumb situation like this, we will be able to protect ourselves.

I am requesting information from the insurance group in this situation, and am going to consult with a few people to try to find an insurance group that is familiar with these types of problems and is comfortable giving us insurance at a reasonable price (I've been told it's not too bad and we can probably afford it).

If anyone reading this has any expertise, suggestions or advice here, it's very welcome. I'm learning about this as I go along.

Investigate more robust messaging system

Right now we're using Redis pubsub, which is nice and simple, but also has a single point of failure.

A nice decentralized distributed messaging queue system could be better here.

NSQ: http://nsq.io/overview/design.html

Show pricing chooser after initial signup

When users create an account, they should be directed to an initial one-time page that asks them if they would like to become a supporter, and they choose a free or supporter button, and then it continues to the news feed once they're finished.

The hope is that this increases clickthrough to supporter.

Very high priority.

Integrate Paypal for supporter subscriptions

Barf. It needs to be done, international ppl use it.

Site banning from browse for admins

Streamline removing sites by allowing banning directly from the browse screen with admin users.

Prepare for Neocities announcement

We launched the major upgrade to Neocities, but have not yet made the big announcement.

Most of the things I wanted to do are completed, though I might want to fix the replication problem before we do.

@violasong I wanted to ping you for your list of changes you'd like to make before we do the announcement, so we can setup a task list (https://github.com/blog/1375%0A-task-lists-in-gfm-issues-pulls-comments) and be able to work on it.

Remove SVG font mention?

SVG fonts are being less supported, & don't have any real advantages over WOFF[2].

I don't think support should be removed, just not encouraged to help make sites more future-proof.

Gamification

Get a badge for things like:

• Updating site x times
• Posting x comments/likes
• Getting x site visits
• Getting x followers
• Being a featured site
• A streak of x days of visits to neocities
• A streak of x days of updates
• Successfully flagging bad sites
• Sharing a neocities link on twitter or facebook
• Contributing to github project
• Eventually we could track Neocities signups originating from a certain user's site, and reward that user for it
• Eventually, completing stages of the tutorial

Related to #34, as an email could be sent for any of these

Remove site profiles, remove follows/followers when site is banned

Add additional information for WebDAV users

Email search: [Neocities Contact]: WebDAV Authentication

Web Editor is awkward

The online code editor is a set height, which requires the user to scroll to save their work on smaller screens.

Suggestion:
Have always-visible save buttons and an editor window which resizes to fit the user's screen.

Bonus:
Enable keyboard save shortcut.

Better email outreach

Right now we just have a welcome email.

It would be nice to provide more of a path for interacting with the end user. Examples:

• An email notifying the user if they are now granted commenting privileges.
• An email congratulating them for their first 1,000 visitors (slideshare does this, and I love it).

Many more possibilities here, feel free to throw some ideas in!

Further reading:
https://www.sendwithus.com/resources/guide/

Stats page needs to compensate for fees

The stats page does not compensate for Stripe fees when calculating our revenue. The total revenue should be after-fee, not the raw input.

Issue uploading .js files (zepto.min.js, touch.js)

Hi, I have been testing successfully the site with a free account (eibolmooc), and I have found out that sometimes uploading fails with certain files (.js, .png). It has happened with a free account, and I was uploading to a folder some files that were already in the root folder. I test either upload button and drag & drop feature with same results.

I also tried those to upload the same files to my supporter account (eibol), and did not experience these issues.

Thanks for checking
Best regards
Abel

Build replacement for Pingdom

We're using Pingdom, however with the current price plan ($13/mo) we can only test one site. Which means we can't use it to check the health on any of our other servers. The step up plan only gives you 4 more and costs $42, which is way too much for us.

Basically we just need a program that tries to hit the site every once in a while and complains at us when it can't reach the site, that runs on a cheap instance somewhere offsite.

I would entertain using a different service if we can find a more affordable one, however I think it might just be best to roll our own at this point.

Emergency flagging for abusive content that requires prompt attention

Instead of having reporting of abusive information mixed in with all the contact information, there should be a way to escalate priority.

A radio selector might work here. Bad example:

What is the nature of your contact?

() - General Information
() - Support issue, bug or technical problem
() - Abusive / hostile / phishing
() - DMCA Takedown

It could also just be a checkbox: "[x] This is an emergency situation".

Claim @neocities on Twitter

Submit trademark report https://support.twitter.com/forms/trademark

In the report, we should ask if Twitter can just rename @neocitiesweb to @neocities so that it keeps all our current followers/tweets. (There's also a hackier way to make this happen.)

Contact form should send messages to @violasong too

so I can soak in the ambient feedback~

Better 'Site of the Day' functionality

Unified featured sites queue that automatically posts one site per day to homepage, social networks, notifies the user (both via email and the site), promotes site on browse page, and maybe gives some kind of special badge.