Interactive tutorial for third-party libraries?

[dlang-tour]

Interactive tutorial for third-party libraries?

[PetarKirov]

Sooo... if I understand correctly, the following needs to be done in order to support single-file dub packages:

Backend:

• [Optional] Migrate dlang-tour-rdmd to the install.sh script (curl i.dlang.io | bash)
• Implement dlang-tour-dub docker container using the install script
• [Optional] Refactor the installation into common docker task (Is it possible?)
• Implement some sort of persistent (across container runs) ~/.dub folder so that we can manage to keep the build times < 20 sec (actually under 5 sec if we want good UX).

Frontend:

• Add built type selection

• Teach the Run button to take built type selection into account - code
• Add DUB IExecProvider. I would also like to have the option to select compiler vendor and version, but I'm not sure what's the best way to achieve this. I can think of to options:
  • Write a new IExecProvider for each compiler vendor/version combination. Maybe it would require less changes but would scale quite badly.
  • Extend the IExecProvider interface like so:

interface IExecProvider
{
    /* may be extended in the future */
    struct ExecResult
    {
        string output;
        int returnCode;
        @property bool success() { return returnCode == 0; }
        @property void success(bool ok) { returnCode = int(!ok); }
    }

    /* same as before */
    ExecResult compileAndExecute(string source);

    ExecResult compileAndExecuteWithSpecificTool(string source, string toolName);

    unittest
    {
        IExecProvider dubExec = getDubExecProvider();

        assert (dubExec.isToolSupported("dub-1.0.0+dmd-2.0.71") is true);

        auto execResult = dubExec.compileAndExecuteWithSpecificTool(
            q{
                 /+ dub.sdl:
                     name "hello_dub_world"
                 +/
                 int main()
                 {
                     return 42;
                 }
            },
            "dub-1.0.0+dmd-2.0.71"
        );

        assert (execResult.returnCode == 42);
    }

    bool isToolSupported(string toolName);

    unittest
    {
        import exec.docker : Docker;

        IExecProvider docker = new Docker(
            5 /* max time */,
            128 /* max output size */,
            10 /* max queue size */,
            10 /* MB max memory */
        );

        // tools not present in the whitelist are not supported
        assert (docker.isToolSupported("asdasdad") is false);

        // At minimum RDMD with unspecified version *should* be supported.
        assert (docker.isToolSupported("rdmd") is true);

        import exec.off : Off;
        auto off = new Off();

        // IExecProvider-s should not lie about their capabilities
        assert (off.isToolSupported("rdmd") is false);
    }
}

Content

• Explain the built types combobox in somewhere around the welcome chapter
• Write a chapter about DUB and explain how to use it on the dlang-tour site

@stonemaster @wilzbach what do you think? Is the above list of tasks complete? Do you think it should be done differently? Is the feature as whole a good idea? This seems like fun project and I'd like to implement it myself, but I first want to know if @stonemaster approves the overall development plan, so I can start working on concrete pull requests where we can discuss the details ;)

[stonemaster]

First off, I really like the idea and what the direction this might take to provide a platform for trying D with various third party libraries.

There are just some technical difficulties I see which make this more of mid-term goal.

Currently a Docker container is used which is maintained by me. The container includes dmd and all needed dependencies and runs rdmd with source code file. Each transaction creates a new container which is thrown away when the source has been compiled and run. Currently each compilation/run step has a maximum runtime of 20 seconds and if the step doesn't finish within this period the container is killed. This is needed to prevent evil source to block the whole server.
Of course it would be possible to integrate dub into this container but I see a problem in giving users the permission to defined dependencies. The dependencies would have to be fetched and compiled before compiling the user's source which blocks for some time. So a technical solution could be to fetch dependencies at startup and store them in some persistent manner so that the rdmd container can access and use them. This would also mean that it shouldn't possible to define arbitrary dependencies, but just a controlled set which are defined to be fetched before any compilation step. And it means that the dub.sdl shouldn't be editable to users directly but be provided by the third-party tutorial creator to make sure we don't fetch dependencies we don't want to have.

Again, I really like your idea to have dlang-tour more extensible. This is a great idea to make sure more libraries and tutorials are included without each library writer having to think about custom solutions. I am definitely willing to think about this and implement it according enhancements. I will soon create some concrete GitHub issues dealing to iteratively provide this functionality.

But my current focus is to get the first release of the dlang-tour out to public. So polishing and bugfixing will be me on my agenda before I start to do something on this topic.

[PetarKirov]

Thanks for your detailed answer.

I understand what you're saying about the technical difficulties - your current solution with rdmd is simple and robust. Adding DUB support will require considerable amount of work to implement securely. I am ok with postponing DUB after the official realease.

I briefly looked at your container and I have a simple suggestion - instead of manually downloading and installing DMD, you can use the Install script which is also used for popular services like Travic CI. The script is portable and doesn't require any special priveleges. It just unpacks the specified compiler version and the latest stable DUB version in ~/dlang/dub/ and e.g. ~/dlang/dmd-2.071.0-beta1/. Then a simple command is needed to make DMD and DUB available in $PATH:

$ source ~/dlang/dmd-2.071.0-beta1/activate

And if you want the latest DMD nightly:

$ ~/dlang/install.sh dmd-nightly
# Fetching DMD 2016-04-03..
$ ~/dlang/dmd-201-04-03/activate

Also, by default DUB puts all packages in ~/.dub/packages/, however if you don't want to store several gigabytes in the home directory, you can just make a symlink from this directory to e.g. another partition.

I would recommend having a single "repository" of DUB packages shared by all containers. DUB caches the fetch and build step, so the extra slowdown should only happen on first request to use a particular DUB package. There is also an offline option that tells DUB to only use what is already present. IMO, allowing users to manually modify dub.sdl is very important, because otherwise we won't the be able to teach them to effectively use DUB in their own projects. For example I would like to show them how to make some of their project dependencies truly optional. This wouldn't be possible without editing dub.sdl.
Maybe a good compromise is to allow users to manually edit the file, but restrict the available packages. E.g. run DUB only if dub.sdl contains only whitelisted dependecies.

[wilzbach]

Btw this should be a lot easier once dub gets the single-file support

[wilzbach]

Ping as dub now has a very nice way to provide dependencies as part of the file, e.g.

/+ dub.sdl:
name "check_phobos"
dependency "libdparse" version="~>0.7.0-alpha9"
+/

(e.g. have a look at this file)

[wilzbach]

Sooo... if I understand correctly, the following needs to be done in order to support single-file dub packages:

First of all: thanks a lot for taking the initiative 👍
I hope that my comment are at least a bit helpful.

[Optional] Migrate dlang-tour-rdmd to the install.sh script (curl i.dlang.io | bash)

I love the short url curl i.dlang.io | bash, but as a disclaimer I have to say the following three things:

1. in the short variant it doesn't use SSL (curl https://i.dlang.io | bash is supported though)
2. it uses unsupervised downloads from the official github repo to update itself (however I watch the tools repo, so I could easily disable this if someone sees it as potential security issue that the github DNS name will be hijacked - imho if Github is compromised we be screwed anyhow)
3. the script is currently hosted on my server (it's the most stable one, but I still think that it should on a server where multiple people can access it, e.g. one of D foundation's server)

Implement dlang-tour-dub docker container using the install script

See the comment below about switching to Linux sandboxes, that might save you/us quite some trouble ;-)

Implement some sort of persistent (across container runs) ~/.dub folder so that we can manage to keep the build times < 20 sec (actually under 5 sec if we want good UX).

Afaik the initial idea of @stonemaster was to cache the common packages ahead during deployment, but there are at least two problems

1. IIRC Dub always tries to use the latest version (and common packages are updated regularly)
2. We would have a hard job in keeping this list updated

-> so having a shared .dub cache is probably better, but

• we might run into locking issues if two concurrent build happen
• I am not even sure whether DUB supports multiple .cache directories

Maybe you can avoid this by updating the common cache after the build was run (and the result are sent to the client)?

As a warning: with single packages DUB sometimes requires one to build all packages ahead of time (dlang/dub#893), the easiest workaround that I found was to run dub twice.

Add built type selection

I am not sure whether this is a good idea. Can't we use the new, single-file format for dub packages? Then people can also run examples on their machines and we don't need to show a button nor guess about which type to use:

#!/usr/bin/env dub
/+ dub.sdl:
name "fancy_ndslice_example"
dependency "mir" version="0.15.3"
+/

A simple way to distinguish between rdmd and dub would then be to grep for dub[.](sdl|json).
I do understand the interest in having dmd-nightly available, but as the D tour is specifically aimed at newbies to D, it would make the impression that D is still very unstable. Pinging @nazriel here as afaik he is working on a new version of DPaste (with dub support nazriel/dpaste-frontend#5).

Add DUB IExecProvider. I would also like to have the option to select compiler vendor and version, but I'm not sure what's the best way to achieve this. I can think of to options:

As said before I am not sure whether it's good to display the available version in the UI, but it's definitely a good idea to support them via the API, s.t. we could have a full-page "IDE" page where a user can select the compiler version. However keep in mind that this brings another issue of keeping the dmd nightly compiler in sync with the docker images and afaik dmd is currently packaged into the docker image. @MartinNowak suggested to use SeLinux sandboxes for the individual runs, s.t. might be an easier way to keep the compilers up-to-date?

• Write a new IExecProvider for each compiler vendor/version combination. Maybe it would require less changes but would scale quite badly.
• Extend the IExecProvider interface like so:

Definitely the second, but why not instead of "dub-1.0.0+dmd-2.0.71" pass the information a bit more digestible?

struct CompilerTool
{
    bool useDub; // or check whether dubVersion is empty
    string dubVersion = "1.0.0";
    string compiler = "dmd"; /
    string compilerVersion = "2.0.71"
}

Btw afaik there's no way yet to tell the installer which dub version it should use, but probably with the first official release this is coming soon & if it's using a string like "dub-1.0.0+dmd-2.0.71", then I would agree with passing the entire version string.

This seems like fun project and I'd like to implement it myself, but I first want to know if @stonemaster approves the overall development plan,

Just a wow - looking forward to those PRs from myside ;-)

Explain the built types combobox in somewhere around the welcome chapter

Side remark: My latest information is that we wanted to cut down the welcome chapter and move the four installation pages after the end of the basic tour, see

https://trello.com/c/g8ahgOdf/202-first-5-minutes

Write a chapter about DUB and explain how to use it on the dlang-tour site

👍

[stonemaster]

Thanks a lot for this thorough guide and the ideas you are writing on. I
really like the overall direction and appreciate it very much. 👍

[Optional] Migrate dlang-tour-rdmd to the install.sh script (curl i.dlang.io
| bash)

That actually doesn't matter much. It would be nice to have but it is only
done once during building the rdmd docker image. It is actually a matter of
taste and I am totally fine with it. As to @wilzbach's concerns, this is
just done with every release of amew comtainer. On the downside, the
current version has thr current versione hardcoded which simplifies version
tracking,

Implement some sort of persistent (across container runs) ~/.dub folder
so that we >can manage to keep the build times < 20 sec (actually under 5
sec if we want good UX).

What is very important here is that we don't allow users specify arbitrary
dependencies. Otherwise the server's disk might run full because of bad
dependencies. So my idea would be fetch all dependencies during startup of
the tour with a special parameter to the dub container. Them when the users
wants its code executed we start dub with the --offline paramter to
disallow dependecy fetching.

Add built type selection

I am not sure whether this is a good idea. Can't we use the new,
single-file format for dub packages? Then people can also run examples on
their machines and we don't need to show a button nor guess about which
type to use:

#!/usr/bin/env dub/+ dub.sdl:name "fancy_ndslice_example"dependency
"mir" version="0.15.3"+/

I fully support @wilzbach's point. Maybe we should completely switch to
just a dub comtainer? Maybe if no dub.sdl comment is found a generic one is
generated by the tour itself. Otherwise we just all source code examples to
the new format. But that we can decid at a later moment.
Generally I think we should keep easier and introduce to much new other
"dimensions". So we should just dub and one releaed compiler version, I am
also against compiler version for now because it might create confusion for
new users. For different toolchains we have dpaste after all. And we would
need to check just a toolchain for sanity checks during CI.

This seems like fun project and I'd like to implement it myself, but I
first want to know if @stonemaster https://github.com/stonemaster
approves the overall development plan,

I really like the development plan and the overally direction is the right
one 👍

EDIT: Fixed broken citations.

[stonemaster]

@wilzbach could comment on this:

See the comment below about switching to Linux sandboxes, that might save you/us quite some trouble ;-)

I didn't see a comment on that and I would like here your opinion here :-) @MartinNowak once questioned the use of Docker in the tour and I am still very sure that it is the right technology for us here:

• It makes deployment very easy (see the DEPLOYMENT guide)
• Like SELinux it guarantees to have sandboxes with memory constraints configurable
• The setup is easily reproduible on every developer box that runs a more or less recent Linux
• We can re-use Docker's infrastructure for deployment and distribution of the compile sandboxes
• Looking at the Docker execution driver as well as the dlang-tour-rdmd shows how easy it is to get this model working. I wouldn't assume that SELinux provides the same level of low-maintenance..

[wilzbach]

I didn't see a comment on that and I would like here your opinion here :-) @MartinNowak once questioned the use of Docker in the tour and I am still very sure that it is the right technology for us here:

Oh I am quite happy with our current setup as we didn't experience any issue (yet), so I am definitely not going to complain. Moreover I agree with you that Docker is a good technology (my knowledge is too limited to say it's the right). Anyway my point was merely that in case @ZombineDev runs into problems with Docker (e.g. with providing nightlies or the dub cache), there are also other interesting. technologies. Sorry that the message was so unclear :/

OT: there's a fun video about docker:
https://www.youtube.com/watch?v=PivpCKEiQOQ

[wilzbach]

I fully support @wilzbach's point. Maybe we should completely switch to
just a dub comtainer? Maybe if no dub.sdl comment is found a generic one is
generated by the tour itself. Otherwise we just all source code examples to
the new format. But that we can decid at a later moment.

I just made an interesting discovery - if we use DUB instead of rdmd our builds might get faster :)

> python -m timeit -n 10 -s 'import os' 'os.system("rdmd --force foo.d")
10 loops, best of 3: 730 msec per loop
> python -m timeit -n 10 -s 'import os' 'os.system("dub run --single --force foo.d")'
10 loops, best of 3: 578 msec per loop

foo.d is the template meta programming gem,

Associative arrays

rdmd 5 loops, best of 3: 637 msec per loop
dub: 5 loops, best of 3: 584 msec per loop

Loops

rdmd: 5 loops, best of 3: 488 msec per loop
dub: 5 loops, best of 3: 373 msec per loop

The downside is that when a build error occurs (e.g. the range example), DUB is a bit slower)

rdmd: 5 loops, best of 3: 170 msec per loop
dub: 5 loops, best of 3: 212 msec per loop

So in summary: using a DUB as build container makes our builds more flexible and even faster 🎉

[PetarKirov]

Thanks for assigning this to me. I want to push this forward, though realistically I'll be able to make progress on this front after 10-15 Sep, because I'll be quite busy before that. I hope you don't mind the delay.

[wilzbach]

Thanks for assigning this to me.

Sorry I just went through the issues and saw this.

I'll be able to make progress on this front after 10-15 Sep, because I'll be quite busy before that. I hope you don't mind the delay.

No worries - at some point I also have to go back to my other life at university.

[wilzbach]

I want to push this forward, though realistically I'll be able to make progress on this front after 10-15 Sep, because I'll be quite busy before that. I hope you don't mind the delay.

@ZombineDev I might have some time in the next weeks to help you on this. What is your current status? Anything concrete that you are blocked on or just the "usual, unfortunate lack of time"?

[wilzbach]

FYI with dlang-tour/core-exec#5, we have a basic DUB support working.
It's currently based on a hard-coded list of libraries that are fetched into the Docker image and prebuilt, but it's working very nicely (so far) and should be a good start.

Example: https://is.gd/QiFKTf