support for non-image media uploads

[neutronx]

support for non-image media uploads

[ginabythebay]

OK, I understand that you have a lot of things to balance when considering changes like this. To get my work done, I have gone ahead and hacked something into my own fork, here: ginabythebay@2bb797f

I would really like to see this functionality, in some form, make it into the your repository...if there is anything I can do that would make this approach palatable for your repository (e.g. restrict this functionality to just pdfs or put it behind some kind of UNSAFE configuration flag, please do let me know!

[xenatisch]

Thank you for the suggestion, @ginabythebay .

It is, in principle an interesting idea; however, there are a few aspects to consider:

• We cannot allow just any file to be uploaded; it will pose a security risk to the server. There will have to be a set of defined extensions (e.g. .pdf, .docx, etc) whose integrity can be verified. Even for .svg, we are adding security measures to ensure the integrity (see this in the developments branch), and Pillow does the work for images.
• Security measures aside, adding new extensions isn't going to be difficult at all. We have designed MarkdownX (both JavaScript and Python) with extensibility in mind. This allows us to make such changes with minimal efforts. So the second point is, what files shall we support, and how can we verify their integrity.
• We can impose limitations on such uploads (e.g. only by staff - who naturally have access to admins). Shall we do this as an imposition? Shall we let the use decide through the settings?
• Default links are not meant for download. The link generated for a download looks like <a href="#" download>something</a> whilst normal links are like <a href="#">something</a>. This is not supported by any Markdown parser (as far as I know). Should be just leave such links without the download attribute which would mean that upon click, the browser would try to navigate (and maybe warn the user in the docs)?

So all in all, we have discussed the suggestions, and it does sound appealing; especially now that we're developing version 2. Let's develop a coherent plan here and then we can go ahead and implement it.

Also, don't worry about the implementation if you don't feel comfortable doing it, we can take care of it. The main thing is to have a discussion here and come up with a well-developed concept.

[ginabythebay]

Hi @xenatisch!

I'm a little confused by your statement that allowing any file to be uploaded will pose a security risk to the server. First, I was expecting that we would leave MARKDOWNX_UPLOAD_CONTENT_TYPES in place, which allows the admin to specify what content types they will allow. Second, I while I can certainly imagine a malicious file sitting in the MEDIA_ROOT could pose a risk to viewers, I am not sure how that would also pose a risk to the server. I was under the impression that django treats things in the MEDIA_ROOT as suspect and would not, for example, execute them.

That said, I would appreciate some way to scan user-uploaded-files for malware. I feel like expecting markdownx to build that is out of scope though. Perhaps the right answer is to provide some kind of hooks so that users can plug in external solutions (which will presumably have dedicated teams keeping them up to date). One such thing I found just now is this: http://www.clamav.net/

For pdfs anyway, I am fine with normal links. My experience is that browsers seem to do something useful in that case. But my perspective is quite limited here.

[xenatisch]

Actually both server and the users. You can have dynamic code evaluation, and Python interpreter is written in C, so there are a few things to consider given that the files are opened and processed within the application. Whilst it may be isolated from the server, the content of the media directory are not necessarily immune. But that's not something we're really concerned about. It's more about having a general integrity check to ensure that the file is what it says on the label - otherwise, there are countless ways to exploit these things.

Regrading the ability for the admins to define extensions, that's an option we can explore. In fact me and @adi- discussed it after we saw your suggestion. It's just that the current system is rather confusing; as in image/png or xml/svg and so on. So the first half of the format is what makes it rather confusing for the users - and it mustn't be confusing. So let's see if we can find a way around it, or whether we can generalise things using, say a dictionary or something.

As for the scope, we are trying - and I mean really really trying - to provide a comprehensive, yet easy-to-use package that is extensible, too. Whilst the security is not our responsibility, I think we should at least provide the infrastructure for it so as to assist those who are less familiar, or at least warn them / make recommendation. Also, That clamav thing looks nice... I like the concept.

With respect to the links, browsers vary. In fact, our base JavaScript which currently relies on JQuery, has been completely rewritten in pure ECMA 2016, but we had to go to so much trouble to provide compatibility for IE 9+ within our code and then trans-compile it back into ECMA 5; why? Well, because Windows 7 still uses the IE and dear old Microsoft has decided that they want to support it until 2020. So we need to be aware of compatibility issues. Now I'm not sure if this is going to be a problem, but we need to look into it (mainly on IE, and I haven't seen one for 5 years).

Oh, and please don't be put off. We really welcome requests and contributions. It's just that every feature we add or deprecate is the result of various discussions, research, and trial implementations. There are quite a few projects relying on this plugin, and we're trying to provide a decent, hassle-free tool for them.

Let's do some experimentations and see how can we best implement this in practice.