Giter Site home page Giter Site logo

artifact-source's Introduction

Atomist 'artifact-source'

Build Status

Defines an ArtifactSource abstraction for source code.

About

A Scala library exposing the core ArtifactSource abstraction, representing a set of artifacts (probably source code) that may be read from or written to a range of sources, including the file system, a zip file, a local Git repository, or GitHub.

This library is used by most Atomist Java and Scala projects.

Design notes

An ArtifactSource is typically constructed in one of two ways:

  • To front a directory-oriented abstraction such as a file system. In this case, directories will front their counterparts in the other representation.
  • Through adding files containing path information. In this case, directories will be inferred.

Path conventions

Files and directories can be given directory paths. Paths begin without any special character, and use / as a delimiter, e.g., src/main/java.

Using

Most users will not need to use this project directly, but will use tools, e.g., rug-cli, that build on this project.

If you wish to develop tools using this project, you will need to add this project as a dependency and the maven repository where it is published to your build tool's configuration. For example, if you use maven, add the dependency to the <dependencies> section and the repository to the <repositories> section of your pom.xml:

<?xml version='1.0' encoding='UTF-8'?>
<project
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0">
	<modelVersion>4.0.0</modelVersion>
    ...
    <dependencies>
        ...
        <dependency>
            <groupId>com.atomist</groupId>
            <artifactId>artifact-source</artifactId>
            <version>0.5.0</version>
        </dependency>
        ...
	</dependencies>
    ...
	<repositories>
		<repository>
			<id>public-atomist-release</id>
			<name>Atomist Release</name>
			<url>https://atomist.jfrog.io/atomist/libs-release</url>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>
	</repositories>
    ...
</project>

Be sure to change the <version> to the one you want to use.

Support

General support questions should be discussed in the #support channel on our community slack team at atomist-community.slack.com.

If you find a problem, please create an issue.

Development

You can build, test, and install the project locally with maven.

$ mvn install

To create a new release of the project, simply push a tag of the form M.N.P where M, N, and P are integers that form the next appropriate semantic version for release. For example:

$ git tag -a 1.2.3

The Travis CI build (see badge at the top of this page) will automatically create a GitHub release using the tag name for the release and the comment provided on the annotated tag as the contents of the release notes. It will also automatically upload the needed artifacts.

artifact-source's People

Contributors

alankstewart avatar cdupuis avatar ddgenome avatar johnsonr avatar kipz avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

artifact-source's Issues

Failing to resetting to sha

While cloning one of my repo, artifact source chocked because it tries to fetch with --unshallow on a full repository (https://github.com/democritus-team/jeanpaul):

Loading rugs of atomist:spring-team-handlers (0.4.0·local) completed
  Cloning democritus-team/jeanpaul#60217cf1d9e683f26564da09a36c763b27455647
Invoking handle method on event-handler SpringBootMetricsOnCommit of atomist:spring-team-handlers failed
com.atomist.source.ArtifactSourceCreationException: Failed to find commit with sha 60217cf1d9e683f26564da09a36c763b27455647
        at com.atomist.source.git.GitRepositoryCloner.com$atomist$source$git$GitRepositoryCloner$$resetToSha(GitRepositoryCloner.scala:82)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$clone$1.apply(GitRepositoryCloner.scala:45)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$clone$1.apply(GitRepositoryCloner.scala:45)
        at scala.Option.map(Option.scala:146)
        at com.atomist.source.git.GitRepositoryCloner.clone(GitRepositoryCloner.scala:45)
        at com.atomist.rug.cli.command.handlers.HandlerRepoResolver.resolveSha(HandlerRepoResolver.scala:22)
        at com.atomist.rug.kind.core.ProjectType.navigate(ProjectType.scala:48)
        at com.atomist.tree.pathexpression.NodesWithTag.com$atomist$tree$pathexpression$NodesWithTag$$findMeUnder(NodesWithTag.scala:65)
        at com.atomist.tree.pathexpression.NodesWithTag.follow(NodesWithTag.scala:90)
        at com.atomist.tree.pathexpression.LocationStep.follow(LocationStep.scala:24)
        at com.atomist.tree.pathexpression.PathExpressionEngine$$anonfun$evaluateAndReport$1.apply(PathExpressionEngine.scala:36)
        at com.atomist.tree.pathexpression.PathExpressionEngine$$anonfun$evaluateAndReport$1.apply(PathExpressionEngine.scala:31)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at com.atomist.tree.pathexpression.PathExpressionEngine.evaluateAndReport(PathExpressionEngine.scala:31)
        at com.atomist.tree.pathexpression.PathExpressionEngine.evaluate(PathExpressionEngine.scala:18)
        at com.atomist.rug.runtime.js.JavaScriptEventHandler.handle(JavaScriptEventHandler.scala:71)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand$$anon$1.run(EventHandlerCommand.scala:51)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand$$anon$1.run(EventHandlerCommand.scala:49)
        at com.atomist.rug.cli.output.ProgressReportingOperationRunner.run(ProgressReportingOperationRunner.java:22)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand.handle(EventHandlerCommand.scala:49)
        at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand.command(EventHandlerCommand.scala:36)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atomist.rug.cli.command.AbstractAnnotationBasedCommand.invokeMethod(AbstractAnnotationBasedCommand.java:47)
        at com.atomist.rug.cli.command.AbstractAnnotationBasedCommand.run(AbstractAnnotationBasedCommand.java:217)
        at com.atomist.rug.cli.command.AbstractCompilingAndOperationLoadingCommand.run(AbstractCompilingAndOperationLoadingCommand.java:162)
        at com.atomist.rug.cli.command.AbstractCommand.run(AbstractCommand.java:36)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atomist.rug.cli.command.ReflectiveCommandRunMethodRunner.invokeCommand(ReflectiveCommandRunMethodRunner.java:17)
        at com.atomist.rug.cli.command.ReflectiveCommandRunner.invokeReflectiveCommand(ReflectiveCommandRunner.java:93)
        at com.atomist.rug.cli.command.ReflectiveCommandRunner.invokeCommand(ReflectiveCommandRunner.java:262)
        at com.atomist.rug.cli.command.ReflectiveCommandRunner.runCommand(ReflectiveCommandRunner.java:60)
        at com.atomist.rug.cli.Runner.runCommand(Runner.java:93)
        at com.atomist.rug.cli.Runner.run(Runner.java:40)
        at com.atomist.rug.cli.Main.invokeRunner(Main.java:34)
        at com.atomist.rug.cli.Main.main(Main.java:25)
Caused by: java.lang.RuntimeException: Nonzero exit value: 128
        at scala.sys.package$.error(package.scala:27)
        at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:132)
        at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:103)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$5.apply(GitRepositoryCloner.scala:80)
        at com.atomist.source.git.GitRepositoryCloner$$anonfun$5.apply(GitRepositoryCloner.scala:80)
        at scala.util.Try$.apply(Try.scala:192)
        at com.atomist.source.git.GitRepositoryCloner.com$atomist$source$git$GitRepositoryCloner$$resetToSha(GitRepositoryCloner.scala:77)
        ... 40 more

Running the commands on a frech clone did show that:

git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense
✘-128 /tmp/whatever/jeanpaul [master|✔ ]
11:43 $ echo $?
128

Do not push branch before adding commits

Currently when creating commits on a branch, it appears the branch is created and pushed before the commits are added:

https://github.com/atomist/artifact-source/blob/81034cb4c59ff36e816a4a2139bd9e324b75c94c/src/main/scala/com/atomist/source/git/GitHubServices.scala#L84

and then the changes are added to the branch via a commit and then the branch is pushed again.

This causes unnecessary churn in CI and causes issues in downstream handlers for branch builds, e.g., CascadeBuilds in spring-team-handlers.

Can artifact-source be changed to do all the work it needs to do locally and then just make one push at the end?

Windows cannot handle certain characters in file path

The following test fails on Windows:

should handle filtering .git and target from artifact-source *** FAILED ***
  java.nio.file.InvalidPathException: Illegal char <*> at index 43: C:\Users\sylvain\Documents\artifact-source\*.class
  at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
  at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
  at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
  at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
  at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
  at java.nio.file.Paths.get(Paths.java:84)
  at com.atomist.source.filter.AbstractPatternFileFilter$$anonfun$getPatterns$3.apply(AbstractPatternFileFilter.scala:65)
  at com.atomist.source.filter.AbstractPatternFileFilter$$anonfun$getPatterns$3.apply(AbstractPatternFileFilter.scala:65)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  ...

the reason is that * is not an allowed character on Windows and thus Paths.get(...) fails in getPatterns.

Edits to existing files get lost

@cdupuis is currently working on an issue that leads to edits to existing files not ending up in final artifact source.

Example of the issue is described atomisthq/spring-team-handlers#169

pathElementsFromFile damages file name if it contains the root path

I was puzzled by the following error:

.atomist/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:16 Error: Module not found: ./code-gen

After step debugging I realized the issue comes from the fact this code is checked out under /code by our CI server:

screen shot 2017-06-05 at 17 02 09

As you can see above, pathElementsFromFile has removed /code from the file name, thus changing ../compiler/code-gen.js into ../compiler-gen.js, which is nonsensical and fails the rug evaluation.

Update Scala to 2.12

The first maintenance release of Scala 2.12, 2.12.1, has been released and this would be a good opportunity to update and take advantage of new features and performance improvements.

Cannot delete then add a file

A typical pattern for adding a file from a Rug archive is

        const settings = ".settings.xml";
        if (project.fileExists(settings)) {
            project.deleteFile(settings);
        }
        project.copyEditorBackingFileOrFail(settings);

This no longer works. It seems marking the file as deleted at any point results in it being deleted in the resulting change set. So the resulting commit/PR deletes the settings file rather than have it changed to the contents of the settings file from the editor's archive.

Externalise ArtifactSource filtering

Problem:

TS projects will always have a .gitignore containing node_modules. However, this currently means that rug-cli will ignore this too, and not make it available to the compiler and not publish it. The current solution is to add !node_modules to .atomist/ignore. Although this works, this cause huge confusion because the expectation is always that the typescript compiler (TSC) will have nothing to do with .gitignore and will decide how/where to load modules from.

Additionally, this only currently works for FileSystemArtifactSource's, which will make it operating on Github repos tricky.

It also feels like it shouldn't be the responsibility of the ArtifactSource to decide what is filter and when.

Proposal:

Rather than complicating the model in this project further, @cdupuis and I discussed the following solution so that users of this library can have finer grained control over filtering.

  • Do not apply any filters by default when created ArtifactSources (e.g. for .gitignore or .atomist/ignore)
  • Provide constructors (for all ArtifactSource implementations) that accept implementations of some filter interface which filters the ArtifactSource during filesystem recursion (for performance and so that the ArtifactSource doesn't ever know about excluded files)
  • Create implementations of said interface for the current filters used in FileArtifactSource (.gitignore etc)
  • Create a filter method on all ArtifactSource instances that accept implementations of the filter interface and returns a new ArtifactSource without the excluded files (i.e. non-mutating).

This is a breaking change to the behaviour of this library...

Git submodules are not supported

We received a support request reporting that running an editor against a project resulted in a commit that made the desired changed but also deleted all the git submodules from the project. Please investigate.

deleteBranch doesn't work

07/28 10:56:16.953 [command-handler-pool-3] ERROR [5f6e6d2f-b863-452c-9ddd-6b721e701ff3::atomist::lifecycle-rugs::T095SFFBK::C5XTP0893] c.a.r.f.g.p.DeleteBranchFunction - Failed to delete branch `delete-github-branch`
com.atomist.source.git.DoNotRetryException: HTTP/1.1 422 Unprocessable Entity
    at com.atomist.source.git.github.GitHubServices$$anonfun$com$atomist$source$git$github$GitHubServices$$httpRequest$1.apply(GitHubServices.scala:558)
    at com.atomist.source.git.github.GitHubServices$$anonfun$com$atomist$source$git$github$GitHubServices$$httpRequest$1.apply(GitHubServices.scala:553)
    at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:388)
    at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:380)
    at scala.Option.getOrElse(Option.scala:121)
    at scalaj.http.HttpRequest.toResponse(Http.scala:380)
    at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:360)
    at scalaj.http.HttpRequest.exec(Http.scala:335)
    at com.atomist.source.git.github.GitHubServices.com$atomist$source$git$github$GitHubServices$$httpRequest(GitHubServices.scala:553)
    at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply$mcV$sp(GitHubServices.scala:126)
    at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply(GitHubServices.scala:126)
    at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply(GitHubServices.scala:126)
    at scala.util.Try$.apply(Try.scala:192)
    at com.atomist.source.git.Retry$.retry(Retry.scala:30)
    at com.atomist.source.git.github.GitHubServices.deleteBranch(GitHubServices.scala:125)
    at com.atomist.rug.function.github.pullrequest.DeleteBranchFunction.invoke(DeleteBranchFunction.scala:30)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.atomist.rug.spi.AnnotatedRugFunction$class.run(AnnotatedRugFunction.scala:66)
    at com.atomist.rug.function.github.pullrequest.DeleteBranchFunction.run(DeleteBranchFunction.scala:13)
    at com.atomist.rug.runtime.plans.LocalInstructionRunner$$anonfun$2.apply(LocalInstructionRunner.scala:51)

Filtering artifact sources with large numbers of files is slow

I was running tests in the rug-rugs project after adding a large number of dependencies under the .atomist/node_modules directory and noticed the tests were running much more slowly. I found the tests that were significantly slower were those that were using the Given the archive root well-known Gherkin step. The TypeScript for this well-known step can be found here.

Given("the archive root", p => {
    p.copyEditorBackingFilesPreservingPath("");
});

Doing some profiling I found that it was the call to ArtifactSource.filter in ProjectMutableView.copyEditorBackingFilesPreservingPath that was the culprit. Specifically the calculation of the cachedDeltas here.

        val deletedArtifacts: Seq[Artifact] = ArtifactSource.this.allArtifacts
          .filter(a => !allArtifacts.exists(_.path.equals(a.path)))

The N×N loop when calculating the filtered cachedDeltas in 300M executions of the innermost comparison.

I have the profiling snapshots if they would help.

methods using httpRequestOption are not retried

Methods like GitHubServices.getBranch are using retry but are not retried because they are using httpRequestOption which catches exceptions and returns None.

Similarly, the GitHubServices.createRepository method uses the construct httpRequestOption().getOrElse(httpRequest()), so the API call in the httpRequestOption does not get retried.

The retry logic needs to be smarter. Perhaps create httpRequestOptionWithRetry and httpRequestWithRetry and modify code accordingly, e.g., do not retry around that.

Source + Source fails to created directories properly

I think this should pass

    val f = StringFileArtifact(".atomist/editors/Editor.sj", "{}")
    val f2 = StringFileArtifact(".atomist/editors/Editor2.sj", "{}")
    val as = SimpleFileBasedArtifactSource(f) +  SimpleFileBasedArtifactSource(f2)
    as.allFiles.size should be(2)
    as.underPath(".atomist").allFiles.size should be(2)

.atomist/ignore not properly rooting paths

The expected behavior of the .atomist/ignore file is to root absolute paths at the project root. This does not appear to be happening per Michael Duergner in atomist-community.

https://atomist-community.slack.com/archives/C29DX2W07/p1501592971771131

If the following line appears in .atomist/ignore,

/classes/

We expect that a directory named classes at the root of the project, i.e., the same level as the .git and .atomist directories, would be excluded from the archive.

Merge artifact-source and github-artifact-source

artifact-source will be the only repository for all ArtifactSource implementations going forward. The work done for this issue will be to copy over all code artefacts from github-artefact-source into this repo. github-artifact-source will be moved to the atomist-recycling organisation.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.