communitysolidserver/communitysolidserver

An open and modular implementation of the Solid specifications

Community Solid Server

The Community Solid Server is open software that provides you with a Solid Pod and identity. This Pod acts as your own personal storage space so you can share data with people and Solid applications.

As an open and modular implementation of the Solid specifications, the Community Solid Server is a great companion:

• 🧑🏽 for people who want to try out having their own Pod

• 👨🏿‍💻 for developers who want to quickly create and test Solid apps

• 👩🏻‍🔬 for researchers who want to design new features for Solid

And, of course, for many others who like to experience Solid.

⚡ Running the Community Solid Server

Use Node.js 18.0 or up and execute:

npx @solid/community-server

Now visit your brand new server at http://localhost:3000/!

To persist your pod's contents between restarts, use:

npx @solid/community-server -c @css:config/file.json -f data/

Find more ways to run the server in the documentation.

🔧 Configure your server

Substantial changes to server behavior can be achieved via JSON configuration files. The Community Solid Server uses Components.js to specify how modules and components need to be wired together at runtime.

Recipes for configuring the server can be found at CommunitySolidServer/recipes.

Examples and guidance on custom configurations are available in the config folder, and the configurations tutorial. There is also a configuration generator.

👩🏽‍💻 Developing server code

The server allows writing and plugging in custom modules without altering its base source code.

The 📗 API documentation and the 📓 user documentation can help you find your way. There is also a repository of 📚 comprehensive tutorials

📜 License

The Solid Community Server code is copyrighted by Inrupt Inc. and imec and available under the MIT License.

🎤 Feedback and questions

Don't hesitate to start a discussion or report a bug.

There's also a Matrix-based, CSS-focused chat

Learn more about Solid at solidproject.org.

communitysolidserver's Issues

How do I get involved?

You mentioned on the Solid call that you would be ready to accept contributors soon, probably in the next couple weeks. How can I get involved once you're ready? Is there any documentation or getting started guides I can start reading now? Do you need someone to write onboarding guides for contributors and developers?

AsyncHandlers' canHandle function should return a boolean

Otherwise the function name is just silly

Support HEAD requests

HEAD requests are not supported yet, so we should implement a handler.
Possibly similar or derived from SimpleGetOperationHandler.

Verify FileResourceStore on Windows

Currently all FileResourceStore tests work on Windows, due to the usage of path.posix, as was fixed in b489d69, but I should have a deeper look to see if it actually works when writing real files.

RDF to RDF content negotiation

Add max line length to linter

Distinguish between read and write locks

We current lock read and write in the same way (#28).

However, it is perfectly fine for multiple read operations to happen simultaneously; as long as no write operation happens.
It's only when a resource is being written to, that other reads and writes should be queued.

Inspiration in https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock

Implement a Switching and SPARQL-based ResourceStore

As a followup to the file-based ResourceStore of #46, we also want to be able to support SPARQL-based backends.
Data would be sent to and retrieved from a SPARQL endpoint by converting incoming requests to SPARQL queries.

We also want some sort of SwitchingResourceStore class (potentially abstract with separate specific implementation) that can write data to different stores based on specific parameters of the incoming requests.
Specifically in this case we would want to send triple data to the SPARQL store,
and binary data to a different store (such as a file based store).
But this implementation needs to be generic enough so any type of requirement can be used for any number of stores.
I'm not sure yet how much of this can be generalized, so that part might be quite small compared to the actual implementations.

Practical notes:

• It is important to make sure no functionality of the file store is duplicated. All functionality that is required by both stores should be provided by a separate external class.
• The SPARQL store should only support incoming internal/quads data and can reject all other types.
• Don't write SPARQL queries as strings, use a library like SPARQL.js.
• One way the switching store could work would be to try to convert incoming data to internal/quads using the currently existing converters. If that fails the data should be written to the other store.
• The SPARQL store should only support patches of the sparql-update kind.
• The graph value of the quads can be used to identify to which resource the triples in the store belong.
• Container triples will need to be instantiated in the store to keep track of container contents.
• To test the store with an actual SPARQL endpoint, a BlazeGraph docker container can potentially be used.

Ensure the global executable works with a basic config

Handling output status codes

There needs to be a way the ResponseWriter can return the status codes described here. Currently this gets lost somewhere between the execution and the output. One solution would be to include the HTTP verb in the ResponseDescription. Not sure yet if that would cover all cases. Another option would be to use the same solution as for errors, as described in #6 and include status codes already in ResponseDescription subclasses (but that would require the same discussion as there).

Can't update @typescript-eslint

The latest version of @typescript-eslint/eslint-plugin throws an error when using the eslint-config-es settings. Reason being that the input that is allowed for a rule was changed. Apparently eslint-config-es does not update regularly so until that is fixed we should stay on the version we currently have. This issue is mostly to remind us about this should we get errors when updating all dependencies (and to check later on if it is fixed).

Server currently gives UnsupportedHttpError: Only string or binary results are supported

$ npm start

> @solid/[email protected] start /Users/ruben/Documents/UGent/Solid/solid-community-server
> node ./bin/server.js -p 3000

Running at http://localhost:3000/
UnsupportedHttpError: Only string or binary results are supported.
UnsupportedHttpError: Only string or binary results are supported.
    at SimpleResponseWriter.canHandle (/Users/ruben/Documents/UGent/Solid/solid-community-server/src/ldp/http/SimpleResponseWriter.js:17:23)
    at SimpleResponseWriter.handleSafe (/Users/ruben/Documents/UGent/Solid/solid-community-server/src/util/AsyncHandler.js:17:20)
    at AuthenticatedLdpHandler.handle (/Users/ruben/Documents/UGent/Solid/solid-community-server/src/ldp/AuthenticatedLdpHandler.js:50:35)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async /Users/ruben/Documents/UGent/Solid/solid-community-server/src/server/ExpressHttpServer.js:23:17

The above error occurs when sending a curl localhost:3000.

There are two issues here:

• the error itself
• the fact that the error is returned with Content-Type: text/html, opening up the possibility for injecting malicious strings into the body; should be text/plain.

Create constants file for URIs

Could perhaps follow something like https://github.com/solid/solid-namespace/blob/master/index.js where you can import { acl } from 'uris' and then acl('Write') or so. Or even more fluent and acl.Write.

Responses have no trailing newline

Responses should always end with a newline.
Otherwise, requests via curl look like this:

rtaelman@zimmer2 ~ 🔷  curl -H 'Accept: text/turtlee' http://localhost:3000/
UnsupportedHttpError: No handler supports the given input: [Can only convert from application/ld+json,application/json,text/html,application/xhtml+xml,application/xml,text/xml,image/svg+xml,application/rdf+xml,application/n-quads,application/trig,application/n-triples,text/turtle,text/n3 to internal/quads., Can only convert from internal/quads to text/turtle.].
UnsupportedHttpError: No handler supports the given input: [Can only convert from application/ld+json,application/json,text/html,application/xhtml+xml,application/xml,text/xml,image/svg+xml,application/rdf+xml,application/n-quads,application/trig,application/n-triples,text/turtle,text/n3 to internal/quads., Can only convert from internal/quads to text/turtle.].
    at CompositeAsyncHandler.findHandler (/Users/rtaelman/Documents/UGent/nosync/community-server/src/util/CompositeAsyncHandler.js:77:15)
    at process._tickCallback X PUT -H "Content-Type: text/turtle" -d '<ex:s> <ex:p> <ex:o>' ick.js:68:7)rtaelman@zimmer2 ~ 🔷  curl -H 'Accept: text/turtlee' http://localhost:3000/
rtaelman@zimmer2 ~ 🔷  curl -H 'Accept: text/turtle' http://localhost:3000/
<ex:s> <ex:p> <ex:o>rtaelman@zimmer2 ~ 🔷

Call build in prepare step

After installation and before publishing, the TS compiler should be called. This can be done via this: "prepare": "npm run build".

Potential problem with locking and data streams

Talking about lazy data streams in #18 made me think that we might not be doing locking correctly currently. The lock is released when the Representation object has been fully constructed, but that object could potentially contain a data stream (linked to a file for example), which could be modified before it is read (if someone else appends to that file for example). Unless we start forcing stores to return streams that can not be modified in any way (but then you would probably lose the lazy benefits).

How do ACL files and URL hierarchy work

Is it still assumed that the corresponding acl file of a container is stored in url + ".acl"? If not, some that link between container and acl file would need to be stored.

Are the acl files still based on the hierarchy of the URLs? That is, to find the relevant acl file, do we split the URL on slashes, and keep going to the URL with one part less until we find an acl file? Since we are also going to support non-filebased resource stores that is what it comes down to.

A response from a solid server should also return a link to the corresponding acl file (and potentially also a metadata file?). How would the ResponseWriter be aware of those? Maybe the ResponseDescription interface needs extra fields for this (but then multiple classes would have to edit this object)? Other classes already write headers to the HttpResponse? Something in the RepresentationMetadata? There are many options here, also based on the answers on the questions above.

Chain converters

We need some way to be able to chain representation converters (e.g. rdf -> quad, quad -> other rdf) so they can be combined for more possible results.

Implement a file-based ResourceStore

We need a ResourceStore that makes use of the file system backend, similarly to how the current Solid server operates. There currently already is a basic in-memory store, but note that several shortcuts are taken there that should not be copied.

Some of the required features:

• Read the developer notes. Specifically the architecture document describes how ResourceStores should function.
• The interface lists each of the required functions.
• Containers should be represented by folders, while resources as the files in those folders.
• Metadata about a container can be stored in a .metadata file in that folder. Similarly, metadata about a (binary) file can be stored in ${filename}.metadata.
• The CRUD operations on the resources should mostly correspond to those on files. E.g., getting a resource should return a file stream, deleting a resource should delete the file, etc.

Things to take into account:

• Make sure the correct libraries are used for handling file paths and URLs, since the first are handled differently by Windows and Unix systems, while the second are handled the same.
• The SimpleResourceStore does some data conversions between Quads and text/turtle. In general ResourceStores should not do this since the plan is to have specific stores to handle the data conversion (to prevent repetition), so simply supporting BinaryRepresentation for in- and output is sufficient.
• Conditions don't have to be supported in the first implementation, but take into account that this will need to be added at some point later.
• The modifyResource function should simply throw an error so it can be handled by an external PatchingStore.
• Make sure there are no unexpected errors being thrown due to e.g. files not existing.

SPARQL PATCH causes hanging request

The following will cause the server request to hang without any observable errors client-side or server-side:

$ curl -X PUT -H "Content-Type: text/turtle" \
  -d '<ex:s> <ex:p> <ex:o>.' \
  http://localhost:3000/myfile.ttl
$ curl -X PATCH -H 'Content-Type: application/sparql-update' \
  -d 'INSERT DATA { <ex:s2> <ex:p2> <ex:o2> }' \
  http://localhost:3000/myfile.ttl

Implement conditional requests

All resource store functions accept an optional Conditions parameter, but currently it's not really defined how that object gets there. Should this be part of the Operation interface?

Also regarding ETags, how would a resource store expose those. Would that be an extra field in the RepresentationMetadata?

Travis CI - Branch check does not execute

When committing to a pull request (#52 in this case) the travis tests are executed automatically except of the Travis CI - Branch one.

I already gave Travis access to my fork via travis-ci.com.
At this site it says one check is passed but there's no mention of a second check anywhere else.

Possible things that come to my mind:

1. Commits were done on a fork of the repo while the PR is on the this repo.
2. There are still changes requested.
3. I pushed my commits to a different branch that's not master on my fork.

How to best query internal triples

In multiple situations it is going to be necessary to query the internal triples (e.g., acl files). N3.js can be used for single triple pattern queries but more complex queries might be necessary. Not sure yet which tool would be best for this or if something custom would have to be made.

Write README with context

Investigate creating new interface for file/sparql store backends

We should look into generalizing ResourceStore behaviour. E.g. if one of them rejects setting a Representation at a certain URI because there already is a container there, we want this to be consistent across stores. One idea is to have a DataResourceStore which handles all this behaviour and takes as input a DataAccessor (names totally up for change). DataAccessors would then have functions such as hasResource, createContainer, setResource, etc. which can be used to implement the behaviour. In the case of a file data accessor for example, hasResource would be implemented by fs.lstat, while in the case of a sparql backend it would be an ASK query.

The main question is if this can be done (cleanly) without losing any functionality. We would have to investigate which functions would all be needed and to make sure there is no drastic change in efficiency because of this.

Some potential issues I though of (but there probably are plenty more):

• Probably need a different function to write a Representation and its metadata since metadata is always going to be a Quad array while a Representation can be anything (unless there is some way to also handle this in the DataResourceStore).
• SPARQL stores only accept Quad streams as Representation, so there still needs to be a way to handle cases like this.
• PATCH/modifyResource calls probably still have to be sent directly to the accessor since in the case of SPARQL these get sent directly to the endpoint.

Enable `strict` in typescript settings

Correctly parse URLs

There are many things to keep into account when parsing URLs. This library seems to take most situations into account so we could use that one (would need typings though). Potentially to be expanded with extra options for forcing certain parts.

Check LDP spec for missing features

Should go through the entire LDP spec to see which required features are still missing, such as the Accept-Post header described here.

Initialization versus runtime config

While setting up our dependency injection, I ran into some problems in the way some config options are being passed around.

Concretely, starting the server from the CLI allows you to pass a port. This port determines the baseUrl. This baseUrl is passed into the constructors of several components.
The problem here is that this makes it impossible to have a clean standalone dependency injection config to instantiate components, since we depend on values from the CLI, which are only available after component instantiation.

If we would make a cleaner separation of initialization config (what goes in the config file), and runtime config (what is passed from the CLI), we should be able to solve this.

Concretely, what I would propose is to create a new RuntimeConfig component that can be passed to other components during dependency injection. While DI is going on, this component will not contain any data yet. When DI is done, and the server is started, it can be populated with data derived from the CLI, such as port, baseUrl, ... Other components can then read this data from RuntimeConfig at runtime.

The important thing to take into account here is that RuntimeConfig is passed around at initialization time, but can only be read at runtime.

Of course, we could still define default values in RuntimeConfig via our config file. This would mean that we could define a port in our config file, but if we pass one via the CLI, it would be overridden.

For reference, LDF server does something similar with UrlData

Any thoughts before I implement this?

Getting data with no content-type from a ResourceStore

How the SimpleResourceStore currently works is to look at the type value of the input RepresentationPreferences to know what kind of representation to return (this corresponds to the incoming Accept header). This is not possible if for some reason we would need a javascript object as representation (e.g., a stream of Quads), since those have no corresponding mediatype. Currently the SimpleResourceStore returns a stream of Quads if the type preference is missing, but this does make some assumptions and might become a problem for non-quad data, or if we have multiple javascript object types.

Implement metadata parser

Once #65 has been implemented we need to add onre or more handlers to actually parse the incoming HTTP headers to metadata (also important for integration testing.

How to handle content-types in file based stores

Currently when calling getRepresentation on a file store, the content-type is based on the file extension. Currently there are no checks on this in the store itself though, so when PUTting a resource the path doesn't need to end in an extension corresponding to its content-type, and similarly to the slug when POSTing. If we keep this behaviour we'll have to decide if we want to error in those cases or add the extension ourselves. I haven't checked yet what the current Solid implementation does but should probably copy that.

On the other hand, I'm a fan of storing the content-type in the metadata so you're not forced to have an extension. That would mean that every file would have a metadata file though, which would increase the number of files.

Use readable-stream library to have consistent streams

This would reduce the issues there currently are due to for example Node 10 missing some stream features.

https://www.npmjs.com/package/readable-stream

Container questions

Some questions about containers. I think we already discussed some of it but it's not bad to have it written down here.

A container should contain triples referencing its children. These need to be updated after deleting a child resource. Should this be the responsability of the resource store or is this another place to have a specific store with a decorator pattern?

Related to the previous: whoever is responsible for this needs to have a way to find the parent of a resource, I'm thinking this might be a separate class/function entirely, since the Acl handler is also going to need this?

A file based resource store can sort of break from the above by simply checking which files are present in a folder to generate the membership triples (although the parent function is still needed for acl then). But where is the other data besides the membership data stored of the container in such case?

If I understand the LDP spec correctly here, a client has to add a Link header describing the type when doing the POST, thereby identifying if they're adding a resource or creating a new resource. Does this mean we should reject all POST requests without that header? And how strict do we have to be on the contents on this and subsequent requests (e.g., doing a PUT to a container with body <> a ldp:Resource.).

What do we want to do when deleting a container? It's not required to delete the children but it could cause a bunch of invisible resources. Might still be needed to find a large binary file somewhere so it doesn't take up a lot of space without the user knowing.

Have more extensive integration tests

We currently already have some integration tests but we need a much more extensive testing suite as many cases are not covered yet. Things such as the FileResourceStore and the LockingResourceStore are not even covered yet.

One part of this would be having some easy way to set up a server config and run all tests on this config (TypeScript files exporting a HttpServer?). When executing npm run test all those tests should then be run on all those configs. The idea would then also be that /bin/server.ts loads one of these configs (potentially with a new CLI option to choose the config). This way we guarantee that its config has been tested.

On the other hand, there should be tests that fail on some configs and succeed on others (specifically for authentication this can be relevant), so this needs to be taken into account.

It's important that these tests are extensive and cover all potential edge cases since several of these components have not been tested together yet. These should also include more extensive "stories" with multiple requests, including simultaneous requests to test the locking mechanisms. Some reusable mechanism to easily create many of these stories would thus be helpful.

One important difference with the current FileResourceStore tests is also that actual files should be written, so the fs library should not be mocked. But this should be done in such a way that everything is always cleaned up. To have an initial setup there can be a folder of pre-defined files that get copied to a temporary folder when tests are run, representing the files in the store. This would only work for servers using the FileResourceStore, so we will have to take into account that other solutions will be needed once other stores get added.

Another point is that all tests should work on both Windows and Unix systems. Travis should also be updated to test on both as mentioned in #61 .

Set up logging

We need a logging mechanism that all components have easy access to, so they can send debug and info messages.

Have some sort of enum-like for Representation dataType

The dataType field of a Representation is simply a string. This could cause errors in the code due to typos, so it would be interesting to have some field that can be called to make sure the same value is used every time. Changing the type from string to a union of fixed values is not a solution though imo, since that would make extensions more difficult.

Have heartbeat on locks to prevent infinite waiting

To copy my comment from #60:

The one thing I'm wondering if this is the best place to place everything related to the activity monitoring. There are going to be other classes that require the locker, so it might be nice to have some way to make clear to all of them that locks (should) time out. I was thinking of having something like a heartbeat function on the locks (next to the release function they have now) which has to be called to reset the unlock timer, and having the unlock timer in the lock itself. Having such a function would indicate to everyone using the interface that they have to take into account that that exists, and for everyone implementing it (so other locking systems) it would be an indication that the locking mechanism should support this.

Which authentication steps have to be taken

Specifically what does the CredentialsExtractor have to do? This is a combination of me not knowing the full WebID protocol and which responsibilities the extractor has. Is this simply a header that needs to be read or what else has to be done here?

What is stored in RepresentationMetadata

Some other issues already mention this as well. One of the mandatory fields of a RepresentationMetadata is the raw array of Quads. Does that mean that a BodyParser has to convert all the corresponding headers to triples for an incoming representation? And I guess this gets stored in the corresponding .metadata entry then? Is it necessary to store these as well for incoming triples (which would make things like contentType a bit weird)?

Error handling

This issue is to keep track of the discussion about to handle errors internally. Main discussions were had in a pull request here and here.

Currently the errors thrown store a status code internally so the ResponseWriter knows which status code to return in the case of an error. The advantage is that this allows for easily adding new errors in the future. The disadvantage is that this makes some of the classes less independent and mixes some reponsabilities. For example, this makes it more difficult to have CompositeAsyncHandler as a separate package. Not that this is necessary but it might be an indication that there is an issue with this.

How does PATCH work

Will this always be a SPARQL update query or do we support multiple input formats? I would guess the second since the architecture specifies multiple types of patches, but what exactly are a LineBasedPatch, BinaryPatch, ImageFilter, etc. then?

And who is supposed to execute those patches then? The architecture defines a PatchOperationHandler, and modifyResource, a function of a ResourceStore which takes a Patch as input. Does that mean the handler just passes the Patch along to the resource store?

If there are all these different kinds of Patches, how would the necessary permissions get extracted then? A PermissionsExtractor for every type of Patch that can only handle that specific type?

Don't lose status codes in CompositeAsyncHandler

Current best example is the body parsers. These will throw an error with status code 415 if the media type is not supported, but the composite handler will merge these into an error with status code 400. Perhaps the composite handler should check if all errors thrown are of the same type and then use that one (not sure how best to do that yet).

Handle HttpRequests with no body

Currently the SimpleBodyParser handles both Requests with triple date in the bodies as Requests with no bodies. Later on when the BodyParsers get fleshed out we should probably have a separate one for empty bodies. Question still sort of remains of how best to determine that there is no body. Since we're currently using the IncomingMessage interface of node the only way to see what is actually in the body is reading the stream. If we don't want to do that we could also simply check if there is a content-type header present (and assume if there is none there is also no body, instead of interpreting it as a binary stream), or the content-length header.

Create several Util files to group related functions

Streamutil, ContentTypeUtil, etc.

When and where should data be parsed?

The last comment in #12 made me realize that it is not quite clear yet when something has to happen to the data (and I don't want to derail that issue too much).

My current assumption was that the BodyParser always parsed incoming RDF data and internally those quads would then be converted to whatever type the ResourceStore required. But if the body parser should not be invoked for a PUT, this is not the case. Several things should probably be set in stone on this.

For the BodyParser, in what cases should it parse the incoming body? If it's not necessary for a PUT (this should be done by a RepresentationConverter then later on for stores that require it), for what cases would it be necessary then? The main cases with a body are PUT, POST (which is similar to PUT), and PATCH. Even for PATCH the parsing could also be done later on by the PatchingStore. Unless the BodyParser is required to parse the body for validation but should still keep the original input in the Representation.

If the BodyParser should only parse input for certain methods, who would determine that? Should the BodyParser itself be aware of for which methods it should parse or not? Should the RequestParser know when to call the BodyParser based on the method?

This also made me think, is there any reason for the file based ResourceStore of #46 to actually care about a file being text/turtle or not? In the end that store just reads/writes from/to files without ever caring about the contents. The only time it would care is in the case of a PATCH, but that is handled by the PatchingHandler, or with acl files, but that would probably be handled by an AclHandler.

POST requests should return a 201 with Location header

Currently, the user has no idea where the POSTed resource has been created.
As such, the server should return an HTTP 201 response with a Location header containing a reference (URL) to the created resource.

Delete acl when deleting resource

When deleting a resource, it is necessary to also delete the corresponding acl resource (if there is one). This should not be the responsibility of the store since those are acl-unaware, so probably somewhere closer to the DeleteOperationHandler.

Question is also when and how to do this in the call stack. If it is deleted before the resource, there is a possibility of the second delete failing (because a problem with the metadata?), and then you just removed the acl file of a resource. On the other hand, deleting it after deleting the resource is also an issue since you can't delete containers with stuff still in them, and in that case the acl resource could potentially still be in it.

Validate created resources

We should perform (optional?) validation when creating resources, because we can for example create invalid Turtle files, which can cause problems later on, when for example doing an UPDATE.

CORS should probably be handled by a separate class

Currently the cors library is used for Express (with some specific parameters). In case a new HttpServer gets added it would be useful to have a single class that handles all things cors related.