atomist-attic/artifact-source

ArtifactSource abstraction for source code

artifact-source's Issues

java.nio.file.InvalidPathException: Illegal char <<> at index 0: <input>

Moved from atomist-attic/rug-cli#214

Please refer to it for more details. Thanks :)

Model access permissions in artifacts (file/dir)

Original issue from @jessitron:

atomist-attic/rug#13

findFile doesn't work with leading /

Cannot delete then add a file

A typical pattern for adding a file from a Rug archive is

        const settings = ".settings.xml";
        if (project.fileExists(settings)) {
            project.deleteFile(settings);
        }
        project.copyEditorBackingFileOrFail(settings);

This no longer works. It seems marking the file as deleted at any point results in it being deleted in the resulting change set. So the resulting commit/PR deletes the settings file rather than have it changed to the contents of the settings file from the editor's archive.

Consider throwing exceptions and allowing consumer to handle as required

https://github.com/atomist/artifact-source/blob/master/src/main/scala/com/atomist/source/git/GitRepositoryCloner.scala#L39

This munging of the exception makes it hard to debug and otherwise handle errors appropriately. I feel like this should be handled by the user of the library.

Failing to resetting to sha

While cloning one of my repo, artifact source chocked because it tries to fetch with --unshallow on a full repository (https://github.com/democritus-team/jeanpaul):

Loading rugs of atomist:spring-team-handlers (0.4.0·local) completed
  Cloning democritus-team/jeanpaul#60217cf1d9e683f26564da09a36c763b27455647
Invoking handle method on event-handler SpringBootMetricsOnCommit of atomist:spring-team-handlers failed
com.atomist.source.ArtifactSourceCreationException: Failed to find commit with sha 60217cf1d9e683f26564da09a36c763b27455647
        at com.atomist.source.git.GitRepositoryCloner.com$atomist$source$git$GitRepositoryCloner$$resetToSha(GitRepositoryCloner.scala:82)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$clone$1.apply(GitRepositoryCloner.scala:45)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$clone$1.apply(GitRepositoryCloner.scala:45)
        at scala.Option.map(Option.scala:146)
        at com.atomist.source.git.GitRepositoryCloner.clone(GitRepositoryCloner.scala:45)
        at com.atomist.rug.cli.command.handlers.HandlerRepoResolver.resolveSha(HandlerRepoResolver.scala:22)
        at com.atomist.rug.kind.core.ProjectType.navigate(ProjectType.scala:48)
        at com.atomist.tree.pathexpression.NodesWithTag.com$atomist$tree$pathexpression$NodesWithTag$$findMeUnder(NodesWithTag.scala:65)
        at com.atomist.tree.pathexpression.NodesWithTag.follow(NodesWithTag.scala:90)
        at com.atomist.tree.pathexpression.LocationStep.follow(LocationStep.scala:24)
        at com.atomist.tree.pathexpression.PathExpressionEngine$$anonfun$evaluateAndReport$1.apply(PathExpressionEngine.scala:36)
        at com.atomist.tree.pathexpression.PathExpressionEngine$$anonfun$evaluateAndReport$1.apply(PathExpressionEngine.scala:31)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at com.atomist.tree.pathexpression.PathExpressionEngine.evaluateAndReport(PathExpressionEngine.scala:31)
        at com.atomist.tree.pathexpression.PathExpressionEngine.evaluate(PathExpressionEngine.scala:18)
        at com.atomist.rug.runtime.js.JavaScriptEventHandler.handle(JavaScriptEventHandler.scala:71)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand$$anon$1.run(EventHandlerCommand.scala:51)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand$$anon$1.run(EventHandlerCommand.scala:49)
        at com.atomist.rug.cli.output.ProgressReportingOperationRunner.run(ProgressReportingOperationRunner.java:22)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand.handle(EventHandlerCommand.scala:49)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand.command(EventHandlerCommand.scala:36)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atomist.rug.cli.command.AbstractAnnotationBasedCommand.invokeMethod(AbstractAnnotationBasedCommand.java:47)
        at com.atomist.rug.cli.command.AbstractAnnotationBasedCommand.run(AbstractAnnotationBasedCommand.java:217)
        at com.atomist.rug.cli.command.AbstractCompilingAndOperationLoadingCommand.run(AbstractCompilingAndOperationLoadingCommand.java:162)
        at com.atomist.rug.cli.command.AbstractCommand.run(AbstractCommand.java:36)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atomist.rug.cli.command.ReflectiveCommandRunMethodRunner.invokeCommand(ReflectiveCommandRunMethodRunner.java:17)
        at com.atomist.rug.cli.command.ReflectiveCommandRunner.invokeReflectiveCommand(ReflectiveCommandRunner.java:93)
        at com.atomist.rug.cli.command.ReflectiveCommandRunner.invokeCommand(ReflectiveCommandRunner.java:262)
        at com.atomist.rug.cli.command.ReflectiveCommandRunner.runCommand(ReflectiveCommandRunner.java:60)
        at com.atomist.rug.cli.Runner.runCommand(Runner.java:93)
        at com.atomist.rug.cli.Runner.run(Runner.java:40)
        at com.atomist.rug.cli.Main.invokeRunner(Main.java:34)
        at com.atomist.rug.cli.Main.main(Main.java:25)
Caused by: java.lang.RuntimeException: Nonzero exit value: 128
        at scala.sys.package$.error(package.scala:27)
        at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:132)
        at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:103)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$5.apply(GitRepositoryCloner.scala:80)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$5.apply(GitRepositoryCloner.scala:80)
        at scala.util.Try$.apply(Try.scala:192)
        at com.atomist.source.git.GitRepositoryCloner.com$atomist$source$git$GitRepositoryCloner$$resetToSha(GitRepositoryCloner.scala:77)
        ... 40 more

Running the commands on a frech clone did show that:

git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense
✘-128 /tmp/whatever/jeanpaul [master|✔ ]
11:43 $ echo $?
128

Use .atomistignore over .gitignore (if present)

We need to be able to ignore some files for git checkin, but allow them for adding to an archive. For example, node_modules.

Override mechanics should probably look like .npmignore:

https://docs.npmjs.com/misc/developers#keeping-files-out-of-your-package

deleteBranch doesn't work

07/28 10:56:16.953 [command-handler-pool-3] ERROR [5f6e6d2f-b863-452c-9ddd-6b721e701ff3::atomist::lifecycle-rugs::T095SFFBK::C5XTP0893] c.a.r.f.g.p.DeleteBranchFunction - Failed to delete branch `delete-github-branch`
com.atomist.source.git.DoNotRetryException: HTTP/1.1 422 Unprocessable Entity
    at com.atomist.source.git.github.GitHubServices$$anonfun$com$atomist$source$git$github$GitHubServices$$httpRequest$1.apply(GitHubServices.scala:558)
    at com.atomist.source.git.github.GitHubServices$$anonfun$com$atomist$source$git$github$GitHubServices$$httpRequest$1.apply(GitHubServices.scala:553)
    at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:388)
    at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:380)
    at scala.Option.getOrElse(Option.scala:121)
    at scalaj.http.HttpRequest.toResponse(Http.scala:380)
    at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:360)
    at scalaj.http.HttpRequest.exec(Http.scala:335)
    at com.atomist.source.git.github.GitHubServices.com$atomist$source$git$github$GitHubServices$$httpRequest(GitHubServices.scala:553)
    at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply$mcV$sp(GitHubServices.scala:126)
    at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply(GitHubServices.scala:126)
    at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply(GitHubServices.scala:126)
    at scala.util.Try$.apply(Try.scala:192)
    at com.atomist.source.git.Retry$.retry(Retry.scala:30)
    at com.atomist.source.git.github.GitHubServices.deleteBranch(GitHubServices.scala:125)
    at com.atomist.rug.function.github.pullrequest.DeleteBranchFunction.invoke(DeleteBranchFunction.scala:30)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.atomist.rug.spi.AnnotatedRugFunction$class.run(AnnotatedRugFunction.scala:66)
    at com.atomist.rug.function.github.pullrequest.DeleteBranchFunction.run(DeleteBranchFunction.scala:13)
    at com.atomist.rug.runtime.plans.LocalInstructionRunner$$anonfun$2.apply(LocalInstructionRunner.scala:51)

GitHub Artifactor Source issue with editing new file

• we're testing a terraform Editor and having a problem with what appears to be:

1. create new file
2. edit this newly created file
3. commit

Windows cannot handle certain characters in file path

The following test fails on Windows:

should handle filtering .git and target from artifact-source *** FAILED ***
  java.nio.file.InvalidPathException: Illegal char <*> at index 43: C:\Users\sylvain\Documents\artifact-source\*.class
  at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
  at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
  at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
  at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
  at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
  at java.nio.file.Paths.get(Paths.java:84)
  at com.atomist.source.filter.AbstractPatternFileFilter$$anonfun$getPatterns$3.apply(AbstractPatternFileFilter.scala:65)
  at com.atomist.source.filter.AbstractPatternFileFilter$$anonfun$getPatterns$3.apply(AbstractPatternFileFilter.scala:65)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  ...

the reason is that * is not an allowed character on Windows and thus Paths.get(...) fails in getPatterns.

Add Squash and Merge for PR

Git submodules are not supported

We received a support request reporting that running an editor against a project resulted in a commit that made the desired changed but also deleted all the git submodules from the project. Please investigate.

Ensure we are getting & setting file permissions correctly when dealing with the filesystem

Likely the root cause of atomist-attic/rug#13

Empty directory isn't part of file system artifact-source

Then creating a FileSystemArtifactSource, empty directories appear to be missing. Is that a design decision?

Update Scala to 2.12

The first maintenance release of Scala 2.12, 2.12.1, has been released and this would be a good opportunity to update and take advantage of new features and performance improvements.

Null InputStream on 200/201 response causing Jackson deserialisation failure

An if guard in the case statement for 200 and 201 responses needs to be added to check for a null InputStream, as it appears the GitHub API is now not returning a 204 when a collaborator is added to a repository

Filtering artifact sources with large numbers of files is slow

I was running tests in the rug-rugs project after adding a large number of dependencies under the .atomist/node_modules directory and noticed the tests were running much more slowly. I found the tests that were significantly slower were those that were using the Given the archive root well-known Gherkin step. The TypeScript for this well-known step can be found here.

Given("the archive root", p => {
    p.copyEditorBackingFilesPreservingPath("");
});

Doing some profiling I found that it was the call to ArtifactSource.filter in ProjectMutableView.copyEditorBackingFilesPreservingPath that was the culprit. Specifically the calculation of the cachedDeltas here.

        val deletedArtifacts: Seq[Artifact] = ArtifactSource.this.allArtifacts
          .filter(a => !allArtifacts.exists(_.path.equals(a.path)))

The N×N loop when calculating the filtered cachedDeltas in 300M executions of the innermost comparison.

I have the profiling snapshots if they would help.

The content of zip files should be lazily loaded when converted to an ArtifactSource

Currently, zip files are read with the file content materialised and returned in the artifact source, causing memory issues. This will be changed to write the zip out to a temporary directory and return a FileSystemArtifactSource with lazy loading

Merge artifact-source and github-artifact-source

artifact-source will be the only repository for all ArtifactSource implementations going forward. The work done for this issue will be to copy over all code artefacts from github-artefact-source into this repo. github-artifact-source will be moved to the atomist-recycling organisation.

.atomist/ignore not properly rooting paths

The expected behavior of the .atomist/ignore file is to root absolute paths at the project root. This does not appear to be happening per Michael Duergner in atomist-community.

https://atomist-community.slack.com/archives/C29DX2W07/p1501592971771131

If the following line appears in .atomist/ignore,

/classes/

We expect that a directory named classes at the root of the project, i.e., the same level as the .git and .atomist directories, would be excluded from the archive.

pathElementsFromFile damages file name if it contains the root path

I was puzzled by the following error:

.atomist/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:16 Error: Module not found: ./code-gen

After step debugging I realized the issue comes from the fact this code is checked out under /code by our CI server:

As you can see above, pathElementsFromFile has removed /code from the file name, thus changing ../compiler/code-gen.js into ../compiler-gen.js, which is nonsensical and fails the rug evaluation.

Create GitArtifactSource class and move Github repository cloner to github-artifact-source

A new variant of FileSystemArtifactSource, called GitArtifactSource, will be created to return an ArtifactSource from a local file system containing a Git repository.

Also, GitHub artefacts such as the repository cloner should not be in artifact-source and will be moved to the github-artifact-source repo.

Do not push branch before adding commits

Currently when creating commits on a branch, it appears the branch is created and pushed before the commits are added:

https://github.com/atomist/artifact-source/blob/81034cb4c59ff36e816a4a2139bd9e324b75c94c/src/main/scala/com/atomist/source/git/GitHubServices.scala#L84

and then the changes are added to the branch via a commit and then the branch is pushed again.

This causes unnecessary churn in CI and causes issues in downstream handlers for branch builds, e.g., CascadeBuilds in spring-team-handlers.

Can artifact-source be changed to do all the work it needs to do locally and then just make one push at the end?

Source + Source fails to created directories properly

I think this should pass

    val f = StringFileArtifact(".atomist/editors/Editor.sj", "{}")
    val f2 = StringFileArtifact(".atomist/editors/Editor2.sj", "{}")
    val as = SimpleFileBasedArtifactSource(f) +  SimpleFileBasedArtifactSource(f2)
    as.allFiles.size should be(2)
    as.underPath(".atomist").allFiles.size should be(2)

Externalise ArtifactSource filtering

Problem:

TS projects will always have a .gitignore containing node_modules. However, this currently means that rug-cli will ignore this too, and not make it available to the compiler and not publish it. The current solution is to add !node_modules to .atomist/ignore. Although this works, this cause huge confusion because the expectation is always that the typescript compiler (TSC) will have nothing to do with .gitignore and will decide how/where to load modules from.

Additionally, this only currently works for FileSystemArtifactSource's, which will make it operating on Github repos tricky.

It also feels like it shouldn't be the responsibility of the ArtifactSource to decide what is filter and when.

Proposal:

Rather than complicating the model in this project further, @cdupuis and I discussed the following solution so that users of this library can have finer grained control over filtering.

• Do not apply any filters by default when created ArtifactSources (e.g. for .gitignore or .atomist/ignore)
• Provide constructors (for all ArtifactSource implementations) that accept implementations of some filter interface which filters the ArtifactSource during filesystem recursion (for performance and so that the ArtifactSource doesn't ever know about excluded files)
• Create implementations of said interface for the current filters used in FileArtifactSource (.gitignore etc)
• Create a filter method on all ArtifactSource instances that accept implementations of the filter interface and returns a new ArtifactSource without the excluded files (i.e. non-mutating).

This is a breaking change to the behaviour of this library...

Add function for listing issue events

This is background work for supporting listing commits associated with an issue - see atomist-attic/github-rugs#51

Edits to existing files get lost

@cdupuis is currently working on an issue that leads to edits to existing files not ending up in final artifact source.

Example of the issue is described atomisthq/spring-team-handlers#169

methods using httpRequestOption are not retried

Methods like GitHubServices.getBranch are using retry but are not retried because they are using httpRequestOption which catches exceptions and returns None.

Similarly, the GitHubServices.createRepository method uses the construct httpRequestOption().getOrElse(httpRequest()), so the API call in the httpRequestOption does not get retried.

The retry logic needs to be smarter. Perhaps create httpRequestOptionWithRetry and httpRequestWithRetry and modify code accordingly, e.g., do not retry around that.