atomist-attic / artifact-source Goto Github PK
View Code? Open in Web Editor NEWArtifactSource abstraction for source code
License: GNU General Public License v3.0
ArtifactSource abstraction for source code
License: GNU General Public License v3.0
A new variant of FileSystemArtifactSource, called GitArtifactSource, will be created to return an ArtifactSource from a local file system containing a Git repository.
Also, GitHub artefacts such as the repository cloner should not be in artifact-source and will be moved to the github-artifact-source repo.
While cloning one of my repo, artifact source chocked because it tries to fetch with --unshallow on a full repository (https://github.com/democritus-team/jeanpaul):
Loading rugs of atomist:spring-team-handlers (0.4.0·local) completed
Cloning democritus-team/jeanpaul#60217cf1d9e683f26564da09a36c763b27455647
Invoking handle method on event-handler SpringBootMetricsOnCommit of atomist:spring-team-handlers failed
com.atomist.source.ArtifactSourceCreationException: Failed to find commit with sha 60217cf1d9e683f26564da09a36c763b27455647
at com.atomist.source.git.GitRepositoryCloner.com$atomist$source$git$GitRepositoryCloner$$resetToSha(GitRepositoryCloner.scala:82)
at com.atomist.source.git.GitRepositoryCloner$$anonfun$clone$1.apply(GitRepositoryCloner.scala:45)
at com.atomist.source.git.GitRepositoryCloner$$anonfun$clone$1.apply(GitRepositoryCloner.scala:45)
at scala.Option.map(Option.scala:146)
at com.atomist.source.git.GitRepositoryCloner.clone(GitRepositoryCloner.scala:45)
at com.atomist.rug.cli.command.handlers.HandlerRepoResolver.resolveSha(HandlerRepoResolver.scala:22)
at com.atomist.rug.kind.core.ProjectType.navigate(ProjectType.scala:48)
at com.atomist.tree.pathexpression.NodesWithTag.com$atomist$tree$pathexpression$NodesWithTag$$findMeUnder(NodesWithTag.scala:65)
at com.atomist.tree.pathexpression.NodesWithTag.follow(NodesWithTag.scala:90)
at com.atomist.tree.pathexpression.LocationStep.follow(LocationStep.scala:24)
at com.atomist.tree.pathexpression.PathExpressionEngine$$anonfun$evaluateAndReport$1.apply(PathExpressionEngine.scala:36)
at com.atomist.tree.pathexpression.PathExpressionEngine$$anonfun$evaluateAndReport$1.apply(PathExpressionEngine.scala:31)
at scala.collection.immutable.List.foreach(List.scala:381)
at com.atomist.tree.pathexpression.PathExpressionEngine.evaluateAndReport(PathExpressionEngine.scala:31)
at com.atomist.tree.pathexpression.PathExpressionEngine.evaluate(PathExpressionEngine.scala:18)
at com.atomist.rug.runtime.js.JavaScriptEventHandler.handle(JavaScriptEventHandler.scala:71)
at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand$$anon$1.run(EventHandlerCommand.scala:51)
at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand$$anon$1.run(EventHandlerCommand.scala:49)
at com.atomist.rug.cli.output.ProgressReportingOperationRunner.run(ProgressReportingOperationRunner.java:22)
at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand.handle(EventHandlerCommand.scala:49)
at com.atomist.rug.cli.command.handlers.event.EventHandlerCommand.command(EventHandlerCommand.scala:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atomist.rug.cli.command.AbstractAnnotationBasedCommand.invokeMethod(AbstractAnnotationBasedCommand.java:47)
at com.atomist.rug.cli.command.AbstractAnnotationBasedCommand.run(AbstractAnnotationBasedCommand.java:217)
at com.atomist.rug.cli.command.AbstractCompilingAndOperationLoadingCommand.run(AbstractCompilingAndOperationLoadingCommand.java:162)
at com.atomist.rug.cli.command.AbstractCommand.run(AbstractCommand.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atomist.rug.cli.command.ReflectiveCommandRunMethodRunner.invokeCommand(ReflectiveCommandRunMethodRunner.java:17)
at com.atomist.rug.cli.command.ReflectiveCommandRunner.invokeReflectiveCommand(ReflectiveCommandRunner.java:93)
at com.atomist.rug.cli.command.ReflectiveCommandRunner.invokeCommand(ReflectiveCommandRunner.java:262)
at com.atomist.rug.cli.command.ReflectiveCommandRunner.runCommand(ReflectiveCommandRunner.java:60)
at com.atomist.rug.cli.Runner.runCommand(Runner.java:93)
at com.atomist.rug.cli.Runner.run(Runner.java:40)
at com.atomist.rug.cli.Main.invokeRunner(Main.java:34)
at com.atomist.rug.cli.Main.main(Main.java:25)
Caused by: java.lang.RuntimeException: Nonzero exit value: 128
at scala.sys.package$.error(package.scala:27)
at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:132)
at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:103)
at com.atomist.source.git.GitRepositoryCloner$$anonfun$5.apply(GitRepositoryCloner.scala:80)
at com.atomist.source.git.GitRepositoryCloner$$anonfun$5.apply(GitRepositoryCloner.scala:80)
at scala.util.Try$.apply(Try.scala:192)
at com.atomist.source.git.GitRepositoryCloner.com$atomist$source$git$GitRepositoryCloner$$resetToSha(GitRepositoryCloner.scala:77)
... 40 more
Running the commands on a frech clone did show that:
git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense
✘-128 /tmp/whatever/jeanpaul [master|✔ ]
11:43 $ echo $?
128
@cdupuis is currently working on an issue that leads to edits to existing files not ending up in final artifact source.
Example of the issue is described atomisthq/spring-team-handlers#169
A typical pattern for adding a file from a Rug archive is
const settings = ".settings.xml";
if (project.fileExists(settings)) {
project.deleteFile(settings);
}
project.copyEditorBackingFileOrFail(settings);
This no longer works. It seems marking the file as deleted at any point results in it being deleted in the resulting change set. So the resulting commit/PR deletes the settings file rather than have it changed to the contents of the settings file from the editor's archive.
An if guard in the case statement for 200 and 201 responses needs to be added to check for a null InputStream, as it appears the GitHub API is now not returning a 204 when a collaborator is added to a repository
Moved from atomist-attic/rug-cli#214
Please refer to it for more details. Thanks :)
We received a support request reporting that running an editor against a project resulted in a commit that made the desired changed but also deleted all the git submodules from the project. Please investigate.
The following test fails on Windows:
should handle filtering .git and target from artifact-source *** FAILED ***
java.nio.file.InvalidPathException: Illegal char <*> at index 43: C:\Users\sylvain\Documents\artifact-source\*.class
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at java.nio.file.Paths.get(Paths.java:84)
at com.atomist.source.filter.AbstractPatternFileFilter$$anonfun$getPatterns$3.apply(AbstractPatternFileFilter.scala:65)
at com.atomist.source.filter.AbstractPatternFileFilter$$anonfun$getPatterns$3.apply(AbstractPatternFileFilter.scala:65)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
...
the reason is that *
is not an allowed character on Windows and thus Paths.get(...)
fails in getPatterns
.
Original issue from @jessitron:
This is background work for supporting listing commits associated with an issue - see atomist-attic/github-rugs#51
We need to be able to ignore some files for git checkin, but allow them for adding to an archive. For example, node_modules.
Override mechanics should probably look like .npmignore:
https://docs.npmjs.com/misc/developers#keeping-files-out-of-your-package
Problem:
TS projects will always have a .gitignore containing node_modules. However, this currently means that rug-cli will ignore this too, and not make it available to the compiler and not publish it. The current solution is to add !node_modules to .atomist/ignore. Although this works, this cause huge confusion because the expectation is always that the typescript compiler (TSC) will have nothing to do with .gitignore and will decide how/where to load modules from.
Additionally, this only currently works for FileSystemArtifactSource's, which will make it operating on Github repos tricky.
It also feels like it shouldn't be the responsibility of the ArtifactSource to decide what is filter and when.
Proposal:
Rather than complicating the model in this project further, @cdupuis and I discussed the following solution so that users of this library can have finer grained control over filtering.
This is a breaking change to the behaviour of this library...
Then creating a FileSystemArtifactSource, empty directories appear to be missing. Is that a design decision?
Likely the root cause of atomist-attic/rug#13
Methods like GitHubServices.getBranch
are using retry
but are not retried because they are using httpRequestOption
which catches exceptions and returns None
.
Similarly, the GitHubServices.createRepository method uses the construct httpRequestOption().getOrElse(httpRequest())
, so the API call in the httpRequestOption
does not get retried.
The retry logic needs to be smarter. Perhaps create httpRequestOptionWithRetry
and httpRequestWithRetry
and modify code accordingly, e.g., do not retry around that.
Currently, zip files are read with the file content materialised and returned in the artifact source, causing memory issues. This will be changed to write the zip out to a temporary directory and return a FileSystemArtifactSource with lazy loading
The expected behavior of the .atomist/ignore
file is to root absolute paths at the project root. This does not appear to be happening per Michael Duergner in atomist-community.
https://atomist-community.slack.com/archives/C29DX2W07/p1501592971771131
If the following line appears in .atomist/ignore
,
/classes/
We expect that a directory named classes
at the root of the project, i.e., the same level as the .git
and .atomist
directories, would be excluded from the archive.
This munging of the exception makes it hard to debug and otherwise handle errors appropriately. I feel like this should be handled by the user of the library.
07/28 10:56:16.953 [command-handler-pool-3] ERROR [5f6e6d2f-b863-452c-9ddd-6b721e701ff3::atomist::lifecycle-rugs::T095SFFBK::C5XTP0893] c.a.r.f.g.p.DeleteBranchFunction - Failed to delete branch `delete-github-branch`
com.atomist.source.git.DoNotRetryException: HTTP/1.1 422 Unprocessable Entity
at com.atomist.source.git.github.GitHubServices$$anonfun$com$atomist$source$git$github$GitHubServices$$httpRequest$1.apply(GitHubServices.scala:558)
at com.atomist.source.git.github.GitHubServices$$anonfun$com$atomist$source$git$github$GitHubServices$$httpRequest$1.apply(GitHubServices.scala:553)
at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:388)
at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:380)
at scala.Option.getOrElse(Option.scala:121)
at scalaj.http.HttpRequest.toResponse(Http.scala:380)
at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:360)
at scalaj.http.HttpRequest.exec(Http.scala:335)
at com.atomist.source.git.github.GitHubServices.com$atomist$source$git$github$GitHubServices$$httpRequest(GitHubServices.scala:553)
at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply$mcV$sp(GitHubServices.scala:126)
at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply(GitHubServices.scala:126)
at com.atomist.source.git.github.GitHubServices$$anonfun$deleteBranch$1.apply(GitHubServices.scala:126)
at scala.util.Try$.apply(Try.scala:192)
at com.atomist.source.git.Retry$.retry(Retry.scala:30)
at com.atomist.source.git.github.GitHubServices.deleteBranch(GitHubServices.scala:125)
at com.atomist.rug.function.github.pullrequest.DeleteBranchFunction.invoke(DeleteBranchFunction.scala:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atomist.rug.spi.AnnotatedRugFunction$class.run(AnnotatedRugFunction.scala:66)
at com.atomist.rug.function.github.pullrequest.DeleteBranchFunction.run(DeleteBranchFunction.scala:13)
at com.atomist.rug.runtime.plans.LocalInstructionRunner$$anonfun$2.apply(LocalInstructionRunner.scala:51)
I was puzzled by the following error:
.atomist/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:16 Error: Module not found: ./code-gen
After step debugging I realized the issue comes from the fact this code is checked out under /code
by our CI server:
As you can see above, pathElementsFromFile
has removed /code
from the file name, thus changing ../compiler/code-gen.js
into ../compiler-gen.js
, which is nonsensical and fails the rug
evaluation.
The first maintenance release of Scala 2.12, 2.12.1, has been released and this would be a good opportunity to update and take advantage of new features and performance improvements.
I think this should pass
val f = StringFileArtifact(".atomist/editors/Editor.sj", "{}")
val f2 = StringFileArtifact(".atomist/editors/Editor2.sj", "{}")
val as = SimpleFileBasedArtifactSource(f) + SimpleFileBasedArtifactSource(f2)
as.allFiles.size should be(2)
as.underPath(".atomist").allFiles.size should be(2)
artifact-source will be the only repository for all ArtifactSource implementations going forward. The work done for this issue will be to copy over all code artefacts from github-artefact-source into this repo. github-artifact-source will be moved to the atomist-recycling organisation.
I was running tests in the rug-rugs project after adding a large number of dependencies under the .atomist/node_modules directory and noticed the tests were running much more slowly. I found the tests that were significantly slower were those that were using the Given the archive root
well-known Gherkin step. The TypeScript for this well-known step can be found here.
Given("the archive root", p => {
p.copyEditorBackingFilesPreservingPath("");
});
Doing some profiling I found that it was the call to ArtifactSource.filter in ProjectMutableView.copyEditorBackingFilesPreservingPath that was the culprit. Specifically the calculation of the cachedDeltas here.
val deletedArtifacts: Seq[Artifact] = ArtifactSource.this.allArtifacts
.filter(a => !allArtifacts.exists(_.path.equals(a.path)))
The N×N loop when calculating the filtered cachedDeltas in 300M executions of the innermost comparison.
I have the profiling snapshots if they would help.
Currently when creating commits on a branch, it appears the branch is created and pushed before the commits are added:
and then the changes are added to the branch via a commit and then the branch is pushed again.
This causes unnecessary churn in CI and causes issues in downstream handlers for branch builds, e.g., CascadeBuilds in spring-team-handlers.
Can artifact-source be changed to do all the work it needs to do locally and then just make one push at the end?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.