tcplugins/tcdebrepository

tcDebRepository provides a Debian Package server from within TeamCity.

tcdebrepository's Issues

Support for more Databases

Running internal H2 with > 200k packages means that TeamCity spends a lot of cpu time in garbage collection.

Support should be added for external DBs to shift the load out of TeamCity.
The following would be considered as potential candidates:

• H2 inside teamcity (already implemented)
• H2 in an external process
• MS SQLServer
• MySQL
• Postgres

Unpack deb file in an emphemeral dir to avoid conflicts

Create a temp dir for each package unpack, or - better yet - stream the archive without writing to disk.

Platform-dependent failures in PluginDataResolverImplTest (Windows)

debrepo.teamcity.entity.helper.PluginDataResolverImplTest exhibits two test failures when run on Windows
(Surefire report: debrepo.teamcity.entity.helper.PluginDataResolverImplTest.txt).

Investigate 304 support - etags etc.

Check if APT supports if modified headers, cache settings, etags modified times.

If they are requested / supported then we should implement relevant headers

Get tcDebRepository working for recent Ubuntu versions (eg, xenial and after)

I presume it is the missing Release file which is causing the problem. So in theory, #13 should resolve it, but I am raising this to track the two separately.

==> default: W
==> default: : 
==> default: The repository 'http://teamcity.docker:8111/app/debrepo/TestRepoName01 xenial Release' does not have a Release file.
==> default: Reading package lists...
==> default: Building dependency tree...
==> default: Reading state information...
==> default: E
==> default: : 
==> default: Unable to locate package tcDummyDeb

Use a DB rather than map backed cache for package repository

Teamcity starts to complain about old gen and heap space after ~40k items in the store(s).

TeamCity server memory usage for PS Old Gen pool exceeded 100% of 341 MB maximum available. 391 MB used of 501 MB total heap available. See the TeamCity documentation for possible solutions.

Add support for "dist" and "component" to be populated from %teamcity% variables

from %teamcity% variables or from tokens in the artifact name
eg

• Regex: build/(\S)/package_name_(\S).+\.deb
• Dist: regex:$1
• Component: regex:$2

Architecture-independent (*_all.deb) packages get indexed yet are invisible for apt

I attempted to build and publish a couple of architecture-independent packages (autotools-dev, debhelper).

I can see the package meta-information properly indexed at http://localhost/app/debrepo/debian/dists/jessie/main/binary-all/Packages. Also, noarch binary packages are available under http://localhost/app/debrepo/debian/pool/main/.

At the same time, apt and and its front-ends (aptitude) don't see any of the architecture-independent packages, while architecture-specific ones (i386, amd64, x32) are perfectly visible. I don't experience anything similar with ftp.debian.org or any of the standard mirrors. I believe additional push-ups are required for architecture-independent packages.

I'm running 3 boxes under Debian Jessie.

Implement Release file support

https://wiki.debian.org/RepositoryFormat?highlight=(Release)#A.22Release.22_files

Restricted access

To be able to use this sweet plugin for proprietary software, restricted access is needed. See this for example: https://debian-administration.org/article/513/Restrict_Access_To_Your_Private_Debian_Repository

Are the actual .deb files protected by normal TeamCity access? In that case I should just be able to use e.g. http://user:[email protected]/app/debrepo/

Please advice

Add support for signed Releases

add InRelease and Release.gpg files "clear signed" with gpg

TeamCity 2017.2: Unexpected Error when editing repos

On TeamCity 2017.2, when editing existing repositories I receive the following unexpected error:

Unexpected Error

This was not supposed to happen. Please provide the error details to your TeamCity server maintainer.
If you maintain this TeamCity installation and the request seems legit please report this error to JetBrains.

Error: org.apache.jasper.JasperException: /plugins/tcDebRepository/debRepository/editDebianRepository.jsp (line: [171], column: [4]) Unable to find taglib [myt] for URI: [/plugins/tcDebRepository/debRepository/tag-lib/mytaglib.tld]

Server time: 2017-12-10 12:20:17
TeamCity: 2017.2 (build 50574)
Servlet container: Apache Tomcat/8.5.23

Trace: org.apache.jasper.JasperException: /plugins/tcDebRepository/debRepository/editDebianRepository.jsp (line: [171], column: [4]) Unable to find taglib [myt] for URI: [/plugins/tcDebRepository/debRepository/tag-lib/mytaglib.tld]
	at org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:42)
	at org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:291)
	at org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:97)
	at org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:348)
	at org.apache.jasper.compiler.Parser.parseIncludeDirective(Parser.java:381)
	at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:484)
	at org.apache.jasper.compiler.Parser.parseElementsScriptless(Parser.java:1497)
	at org.apache.jasper.compiler.Parser.parseBody(Parser.java:1689)
	at org.apache.jasper.compiler.Parser.parseNamedAttributes(Parser.java:1724)
	at org.apache.jasper.compiler.Parser.parseJspAttributeAndBody(Parser.java:1039)
	at org.apache.jasper.compiler.Parser.parseOptionalBody(Parser.java:1017)
	at org.apache.jasper.compiler.Parser.parseCustomTag(Parser.java:1294)
	at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1473)
	at org.apache.jasper.compiler.Parser.parse(Parser.java:145)
	at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
	at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
	at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:203)
	at org.apache.jasper.compiler.Compiler.compile(Compiler.java:374)
	at org.apache.jasper.compiler.Compiler.compile(Compiler.java:351)
	at org.apache.jasper.compiler.Compiler.compile(Compiler.java:335)
	at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:601)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:368)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
	at jetbrains.spring.web.TeamCityJSPServletImpl.access$001(TeamCityJSPServletImpl.java:29)
	at jetbrains.spring.web.TeamCityJSPServletImpl$2.apply(TeamCityJSPServletImpl.java:2)
	at jetbrains.buildServer.util.Util.doUnderContextClassLoader(Util.java:78)
	at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:28)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at jetbrains.spring.web.JspController.doHandle(JspController.java:1)
	at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:101)
	at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174)
	at jetbrains.buildServer.controllers.BaseController.handleRequest(BaseController.java:80)
	at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:34)
	at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:25)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:392)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
	at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:168)
	at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1286)
	at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1041)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:984)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:34)
	at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:25)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:127)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:6)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
	at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:38)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:13)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at jetbrains.buildServer.web.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:9)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at jetbrains.buildServer.web.CSRFFilter.doFilter(CSRFFilter.java:100)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
	at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:37)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:8)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

This did not happen with TeamCity 2017.1.2 from where we upgraded. The used release is 1.0.1. Everything else, e.g. connecting with apt-get and uploading new deb packages works fine.

Minor UX improvements

1. Make project and build names on Edit Repository page into links.
2. Add ability to copy a rule within a repo.

Unexpected Error "... Unable to find taglib ..." when editing repo settings

From the "Debian Package Repositories" page, clicking on the "edit" action for a debian repo results in the following error page:

Unexpected Error

This was not supposed to happen. Please provide the error details to your TeamCity server maintainer.
If you maintain this TeamCity installation and the request seems legit please report this error to JetBrains.

Error: org.apache.jasper.JasperException: /plugins/tcDebRepository/debRepository/editDebianRepository.jsp (line: [171], column: [4]) Unable to find taglib [myt] for URI: [/plugins/tcDebRepository/debRepository/tag-lib/mytaglib.tld]

Server time: 2017-12-28 09:27:57
TeamCity: 2017.2.1 (build 50732)
Servlet container: Apache Tomcat/8.5.23

Show stacktrace

UnsupportedClassVersionError: jetbrains/buildServer/serverSide/ConfigActionFactory : Unsupported major.minor version 52.0

When compiling on Java7.
Java8 in Java7 mode (-source, -target) works fine.

Look at building from TC 9.1 libs if possible.