indeedeng/proctor

Proctor is a Java-based A/B testing framework developed by, and used heavily within, Indeed.

Proctor

Proctor is a A/B testing framework written in Java that enables data-driven product design at Indeed.

Proctor consists of data-model, client specification, client loader, matrix builder, java code generator.

Build Status

Features:

• consistent tests across multiple applications
• group assignment adjustments without code deploys
• rule-based group assignment: e.g. US users experience 50% A, 50% B and non-US users are 25% A, 25% B, 25% C, 25% D
• human-readable test format
• forcing of test groups for internal testing
• java code generation for A/B tests groups

Installation

See Quick Start guide

Example

See proctor demo

Documentation

http://indeedeng.github.io/proctor

Building documentation locally

$ git checkout gh-pages
$ gem install bundler
$ bundle install
$ rake clean serve
  => open http://localhost:4000/ in browser

Building

Need to use Java 11 SDK for using gradle:

List installed java JVMs on MacOS:

$ /usr/libexec/java_home -V

Example output:

Matching Java Virtual Machines (9):
    17.0.6 (arm64) "Azul Systems, Inc." - "Zulu 17.40.19" /Library/Java/JavaVirtualMachines/jdk17.0.6.jdk/Contents/Home
    11.0.18 (arm64) "Azul Systems, Inc." - "Zulu 11.62.17" /Library/Java/JavaVirtualMachines/jdk11.0.18.jdk/Contents/Home
    11.0.17 (arm64) "Azul Systems, Inc." - "Zulu 11.60.19" /Library/Java/JavaVirtualMachines/jdk11.0.17.jdk/Contents/Home
    11.0.16 (arm64) "Azul Systems, Inc." - "Zulu 11.58.15" /Library/Java/JavaVirtualMachines/jdk11.0.16_8.jdk/Contents/Home
    11.0.15 (arm64) "Azul Systems, Inc." - "Zulu 11.56.19" /Library/Java/JavaVirtualMachines/jdk11.0.15_10.jdk/Contents/Home
    1.8.0_362 (arm64) "Azul Systems, Inc." - "Zulu 8.68.0.21" /Library/Java/JavaVirtualMachines/jdk1.8.0_362.jdk/Contents/Home
    1.8.0_345 (arm64) "Azul Systems, Inc." - "Zulu 8.64.0.19" /Library/Java/JavaVirtualMachines/jdk1.8.0_345.jdk/Contents/Home
    1.8.0_332 (arm64) "Azul Systems, Inc." - "Zulu 8.62.0.19" /Library/Java/JavaVirtualMachines/jdk1.8.0_332.jdk/Contents/Home
    1.8.0_322 (arm64) "Azul Systems, Inc." - "Zulu 8.60.0.21" /Library/Java/JavaVirtualMachines/jdk1.8.0_322.jdk/Contents/Home

Set JVM version in MacOS:

$ export JAVA_HOME=`/usr/libexec/java_home -v 11.0.18`

How to Build:

$ ./gradlew build

Local Install

Run the following gradle command and note the local version from the output:

$ ./gradlew clean check publish --refresh-dependencies 

Example output:

Calculating version to use for publish ...
Now using version: 0.local.20230711170543

> Task :proctor-common:compileJava

Example above would create local installation of proctor with version equal to 0.local.20230711170543

Discussion

Use the indeedeng-proctor-users Q&A forum to ask and answer questions about the use of Proctor.

Deploying

Contributing

See Also (other A/B test frameworks)

• seatgeek/sixpack
• bitlove/rollout
• maccman/abba
• assaf/vanity

Code of Conduct

This project is governed by the Contributor Covenant v 1.4.1

License

Apache License Version 2.0

proctor's Issues

Tracking issue for deprecation of dynamic filters

This issue is for explaining deprecation of dynamic filters and for guide of migration to alternative.
(for indeed employee; See https://go.indeed.com/GLYMYNXNA for internal version of same document)

Summary

(test name based) dynamic filters are now deprecated and tag based sharing is the recommended way.

Steps to use tag based sharing

1. Check what tests are currently shared by your dynamic filter
2. Decide tag name for the tag based sharing.
3. Put your tag name into metaTags field of all the tests you want to keep sharing. You can edit it in the usual test edit page. Please do not forget to promote them.
4. Make a code change in dynamicfilters.json to use new filter.
5. For new tests, please do not forget to add your tag name if you want to share it.

Why dynamic filters are deprecated?

The main reason is because it’s not possible to exclude tests once they are matched by the filter as it's not possible to change test name.

• Too general pattern may match more tests than what they want. Such pattern may cause performance issues in the application, or may confuse the owners of mismatching tests.
• Proctor Webapp cannot provide validation of promotion for dynamic clients because a user has no explicit control of which application to share the test.

What’s benefits of tag based sharing?

• You can explicitly choose which application to share at any moment. You don’t have to follow naming pattern when creating a new test.
• You can easily check what tests are shared by the specific filter. E.g. Go to proctor webapp and choose “MetaTags” in the search type and enter tag name in the search box.

Documentation on setting proctor-webapp to use git is incomplete

revision.control=git
# TODO

log4j.properties in resources makes maven verbose

proctor-maven-plugin comes with a log4j.properties that logs in debug, and interferes with projects using logback that do not have a log4j configuration.

I don't think the log configuration should be part of the build . Adding PR: #16

Application yielding HTTP 500 due to missing javax/servlet/jsp/jstl/core/Config class

After fighting through the parameter naming issue, I tried to open the application in the browser and the process is returning an HTTP 500 error. For context, I followed the process outlined in the README for getting things up and running and have it numerous issues along the way.

Here's the repository where I was playing around with getting this up and running: https://github.com/mjpitz/indeedeng-proctor-helm/tree/master/docker/proctor-webapp

Compatibility with guava 21

Proctor can't be run with guava 21.

java.lang.NoSuchMethodError: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
	at com.indeed.proctor.common.model.TestType.register(TestType.java:34) ~[proctor-common-1.1.27.jar:na]
	at com.indeed.proctor.common.model.TestType.<clinit>(TestType.java:96) ~[proctor-common-1.1.27.jar:na]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_102]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_102]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_102]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_102]
	at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call1(AnnotatedConstructor.java:129) ~[jackson-databind-2.8.5.jar:2.8.5]
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromString(StdValueInstantiator.java:318) ~[jackson-databind-2.8.5.jar:2.8.5]

In guava 21 method com.google.common.base.Objects.firstNonNull has been moved to com.google.common.base.MoreObjects.

Following provided directions results in bean initialization error

SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'scheduledTasks' defined in URL [jar:file:/target/tomcat.8080/webapps/expanded/WEB-INF/lib/proctor-webapp-library-9999-SNAPSHOT.jar!/com/indeed/proctor/webapp/util/spring/ScheduledTasks.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'proctorSpecificationSource' defined in class path resource [com/indeed/proctor/webapp/spring/applicationContext.xml]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'int' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4887)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5381)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'proctorSpecificationSource' defined in class path resource [com/indeed/proctor/webapp/spring/applicationContext.xml]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'int' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
	... 24 more
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'int' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1493)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1104)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
	... 38 more

Fairly certain this is due to an incorrect variable name in the spring applicationContext.xml here: https://github.com/indeedeng/proctor/blob/master/proctor-webapp/src/main/resources/com/indeed/proctor/webapp/spring/applicationContext.xml#L50

Should be httpTimeoutMillis not httpTimeout

Unable to create test-definitions using Proctor Webapp

after deploying Proctor Webapp, there are a few console error when opening the Webapp on browser. The test-definitions are not created after filling the details, and on clicking save button, the job-ids would fail.
attaching the console errors:

attaching the consoleOutput for the first error:
consoleOutput.txt
and the console output for the 2nd error

Adding reference for the same query raised on google groups
https://groups.google.com/forum/#!category-topic/indeedeng-proctor-users/all-questions/ceGOE34njAw

Tags are unresolvable

Tried loading a specific version ref and noticed that all the tags in the repository are unresolvable.

https://github.com/indeedeng/proctor/tree/proctor-parent-1.1.52/proctor-webapp

what is the usage of LENGTH

Hi,I really can not understand how to use the "lenght" in range block,in my opinions,the " length" should be the probability of getting the bucketvalue,just like the following example,if I run it 100 times,under ideal conditions I will get "A" 20 times and "B" 80 times,is it right?

"allocations": [
{
"ranges": [
{
"length": 0.2,
"bucketValue": A
},
{
"length": 0.8,
"bucketValue": B
}
]
}
]

TestLibraryFunctionMapper should be in proctor-common, not proctor-consumer

It looks like this test is in the wrong sub-module.

Support prforceGroups names like mts345V

There is currently an issue with our test names in CGP parsing of prforceGroups.

Our test names look like this: mtsXXXV. X are numbers and V is the variant. This correspond to a jira ticket number.

The only limitation of this approach is that there can only be 10 variants. However, this is not an issue for us, as practically, it is very rare to have 10 variants.

Affected piece of code:

https://github.com/indeedeng/proctor/blob/master/proctor-consumer/src/main/java/com/indeed/proctor/consumer/ProctorConsumerUtils.java#L91-L127

fake

2 castle

How to address sample size in the test matrix?

Hey,
Great work! Proctor looks like a great framework.
2 questions here:

1. is the project still in maintenance? active development?
2. in the presentation (https://engineering.indeedblog.com/talks/managing-experiments-behavior-dynamically-proctor/, slide 49),
   sample size is addressed, but there is no explanation of how to use it in the test matrix.

what I mean is that usually before creating an experiment, the analyst is calculating what is the minimum sample size in order to rull which group won.
sample size calculated for i.e.:[https://www.surveysystem.com/sscalc.htm]

In case the desired sample size is reached we want to stop allocating users to that specific experiment.

Is this subject taken care of in the Proctor framework?
Thank you!

generate goal failed: String index out of range: 1

Output log:

mvn proctor:generate -X
Apache Maven 3.0.4 (r1232337; 2012-01-17 10:44:56+0200)
Maven home: C:\Apache\apache-maven-3.0.4
Java version: 1.7.0_45, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.7.0_45\jre
Default locale: ru_RU, platform encoding: Cp1251
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
[INFO] Error stacktraces are turned on.
[DEBUG] Reading global settings from C:\Apache\apache-maven-3.0.4\conf\settings.xml
[DEBUG] Reading user settings from .m2\settings.xml
[DEBUG] Using local repository at .m2\repository
[DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10 for .m2\repository
[INFO] Scanning for projects...
[DEBUG] Extension realms for project com.indeed:proctor-demo:war:0.0.1-SNAPSHOT: (none)
[DEBUG] Looking up lifecyle mappings for packaging war from ClassRealm[plexus.core, parent: null]
[DEBUG] Resolving plugin prefix proctor from [org.apache.maven.plugins, org.codehaus.mojo]
[DEBUG] Resolved plugin prefix proctor to com.indeed:proctor-maven-plugin from POM com.indeed:proctor-demo:war:0.0.1-SNAPSHOT
[DEBUG] === REACTOR BUILD PLAN ================================================
[DEBUG] Project: com.indeed:proctor-demo:war:0.0.1-SNAPSHOT
[DEBUG] Tasks:   [proctor:generate]
[DEBUG] Style:   Regular
[DEBUG] =======================================================================
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building proctor-demo 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[DEBUG] Resolving plugin prefix proctor from [org.apache.maven.plugins, org.codehaus.mojo]
[DEBUG] Resolved plugin prefix proctor to com.indeed:proctor-maven-plugin from POM com.indeed:proctor-demo:war:0.0.1-SNAPSHOT
[DEBUG] Lifecycle default -> [validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes,
generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy]
[DEBUG] Lifecycle clean -> [pre-clean, clean, post-clean]
[DEBUG] Lifecycle site -> [pre-site, site, post-site, site-deploy]
[DEBUG] === PROJECT BUILD PLAN ================================================
[DEBUG] Project:       com.indeed:proctor-demo:0.0.1-SNAPSHOT
[DEBUG] Dependencies (collect): []
[DEBUG] Dependencies (resolve): []
[DEBUG] Repositories (dependencies): [ossrh-releases (https://oss.sonatype.org/content/repositories/releases/, releases+snapshots), central (http://repo.maven.apache.org/maven2, releases)]
[DEBUG] Repositories (plugins)     : [ossrh-releases (https://oss.sonatype.org/content/repositories/releases/, releases+snapshots), central (http://repo.maven.apache.org/maven2, releases)]
[DEBUG] -----------------------------------------------------------------------
[DEBUG] Goal:          com.indeed:proctor-maven-plugin:1.0.0:generate (default-cli)
[DEBUG] Style:         Regular
[DEBUG] Configuration: <?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <outputDirectory default-value="${project.build.directory}/generated-sources/proctor">${outputDirectory}</outputDirectory>
  <project default-value="${project}">${project}</project>
  <topDirectory default-value="${basedir}/src/main/proctor">${topDirectory}</topDirectory>
</configuration>
[DEBUG] =======================================================================
[INFO]
[INFO] --- proctor-maven-plugin:1.0.0:generate (default-cli) @ proctor-demo ---
[DEBUG] Created new class realm maven.api
[DEBUG] Importing foreign packages into class realm maven.api
[DEBUG]   Imported: org.apache.maven.wagon.events < plexus.core
[DEBUG]   Imported: org.sonatype.aether.transfer < plexus.core
[DEBUG]   Imported: org.apache.maven.exception < plexus.core
[DEBUG]   Imported: org.sonatype.aether.metadata < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.util.xml.Xpp3Dom < plexus.core
[DEBUG]   Imported: org.sonatype.aether.collection < plexus.core
[DEBUG]   Imported: org.sonatype.aether.version < plexus.core
[DEBUG]   Imported: org.apache.maven.monitor < plexus.core
[DEBUG]   Imported: org.apache.maven.wagon.repository < plexus.core
[DEBUG]   Imported: org.apache.maven.repository < plexus.core
[DEBUG]   Imported: org.apache.maven.wagon.resource < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.logging < plexus.core
[DEBUG]   Imported: org.apache.maven.profiles < plexus.core
[DEBUG]   Imported: org.sonatype.aether.repository < plexus.core
[DEBUG]   Imported: org.apache.maven.classrealm < plexus.core
[DEBUG]   Imported: org.apache.maven.execution < plexus.core
[DEBUG]   Imported: org.sonatype.aether.artifact < plexus.core
[DEBUG]   Imported: org.sonatype.aether.spi < plexus.core
[DEBUG]   Imported: org.apache.maven.reporting < plexus.core
[DEBUG]   Imported: org.apache.maven.usability < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.container < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.component < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.util.xml.pull.XmlSerializer < plexus.core
[DEBUG]   Imported: org.apache.maven.wagon.authentication < plexus.core
[DEBUG]   Imported: org.apache.maven.lifecycle < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.classworlds < plexus.core
[DEBUG]   Imported: org.sonatype.aether.graph < plexus.core
[DEBUG]   Imported: org.sonatype.aether.* < plexus.core
[DEBUG]   Imported: org.apache.maven.settings < plexus.core
[DEBUG]   Imported: org.codehaus.classworlds < plexus.core
[DEBUG]   Imported: org.sonatype.aether.impl < plexus.core
[DEBUG]   Imported: org.apache.maven.wagon.* < plexus.core
[DEBUG]   Imported: org.apache.maven.toolchain < plexus.core
[DEBUG]   Imported: org.sonatype.aether.deployment < plexus.core
[DEBUG]   Imported: org.apache.maven.wagon.observers < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.util.xml.pull.XmlPullParserException < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.util.xml.pull.XmlPullParser < plexus.core
[DEBUG]   Imported: org.apache.maven.configuration < plexus.core
[DEBUG]   Imported: org.apache.maven.cli < plexus.core
[DEBUG]   Imported: org.sonatype.aether.installation < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.context < plexus.core
[DEBUG]   Imported: org.apache.maven.wagon.authorization < plexus.core
[DEBUG]   Imported: org.apache.maven.project < plexus.core
[DEBUG]   Imported: org.apache.maven.rtinfo < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.lifecycle < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.configuration < plexus.core
[DEBUG]   Imported: org.apache.maven.artifact < plexus.core
[DEBUG]   Imported: org.apache.maven.model < plexus.core
[DEBUG]   Imported: org.apache.maven.* < plexus.core
[DEBUG]   Imported: org.apache.maven.wagon.proxy < plexus.core
[DEBUG]   Imported: org.sonatype.aether.resolution < plexus.core
[DEBUG]   Imported: org.apache.maven.plugin < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.* < plexus.core
[DEBUG]   Imported: org.codehaus.plexus.personality < plexus.core
[DEBUG] Populating class realm maven.api
[DEBUG] com.indeed:proctor-maven-plugin:jar:1.0.0:
[DEBUG]    com.indeed:proctor-codegen:jar:1.0.0:compile
[DEBUG]       com.indeed:proctor-common:jar:1.0.0:compile
[DEBUG]          com.indeed:proctor-tomcat-deps:pom:1.0.0:compile
[DEBUG]             org.apache.tomcat:tomcat-el-api:jar:7.0.8:compile
[DEBUG]             org.apache.tomcat:tomcat-jasper-el:jar:7.0.8:compile
[DEBUG]             org.apache.tomcat:tomcat-jsp-api:jar:7.0.8:compile
[DEBUG]             org.apache.tomcat:tomcat-servlet-api:jar:7.0.8:compile
[DEBUG]          com.indeed:util-core:jar:1.0.0:compile
[DEBUG]          com.indeed:util-varexport:jar:1.0.0:compile
[DEBUG]          jstl:jstl:jar:1.2:compile
[DEBUG]          log4j:log4j:jar:1.2.14:compile
[DEBUG]          org.codehaus.jackson:jackson-core-asl:jar:1.8.0:compile
[DEBUG]          org.codehaus.jackson:jackson-mapper-asl:jar:1.8.0:compile
[DEBUG]          com.google.code.findbugs:jsr305:jar:1.3.9:compile
[DEBUG]          com.google.guava:guava:jar:15.0:compile
[DEBUG]          org.springframework:spring-core:jar:3.0.5.RELEASE:compile
[DEBUG]             org.springframework:spring-asm:jar:3.0.5.RELEASE:compile
[DEBUG]             commons-logging:commons-logging:jar:1.1.1:compile
[DEBUG]          org.springframework:spring-beans:jar:3.0.5.RELEASE:compile
[DEBUG]       com.indeed:proctor-consumer:jar:1.0.0:compile
[DEBUG]          commons-lang:commons-lang:jar:2.4:compile
[DEBUG]          org.springframework:spring-web:jar:3.0.5.RELEASE:compile
[DEBUG]             aopalliance:aopalliance:jar:1.0:compile
[DEBUG]             org.springframework:spring-context:jar:3.0.5.RELEASE:compile
[DEBUG]                org.springframework:spring-aop:jar:3.0.5.RELEASE:compile
[DEBUG]                org.springframework:spring-expression:jar:3.0.5.RELEASE:compile
[DEBUG]       org.freemarker:freemarker:jar:2.3.16:compile
[DEBUG]    org.apache.maven.plugin-tools:maven-plugin-annotations:jar:3.2:compile
[DEBUG]       org.apache.maven:maven-artifact:jar:3.0:compile
[DEBUG]    org.apache.maven:maven-plugin-api:jar:3.1.0:compile
[DEBUG]       org.apache.maven:maven-model:jar:3.1.0:compile
[DEBUG]       org.eclipse.sisu:org.eclipse.sisu.plexus:jar:0.0.0.M2a:compile
[DEBUG]          javax.enterprise:cdi-api:jar:1.0:compile
[DEBUG]             javax.annotation:jsr250-api:jar:1.0:compile
[DEBUG]             javax.inject:javax.inject:jar:1:compile
[DEBUG]          org.sonatype.sisu:sisu-guice:jar:no_aop:3.1.0:compile
[DEBUG]          org.eclipse.sisu:org.eclipse.sisu.inject:jar:0.0.0.M2a:compile
[DEBUG]             asm:asm:jar:3.3.1:compile
[DEBUG]    org.apache.maven:maven-core:jar:3.1.0:compile
[DEBUG]       org.apache.maven:maven-settings:jar:3.1.0:compile
[DEBUG]       org.apache.maven:maven-settings-builder:jar:3.1.0:compile
[DEBUG]       org.apache.maven:maven-repository-metadata:jar:3.1.0:compile
[DEBUG]       org.apache.maven:maven-model-builder:jar:3.1.0:compile
[DEBUG]       org.apache.maven:maven-aether-provider:jar:3.1.0:compile
[DEBUG]          org.eclipse.aether:aether-spi:jar:0.9.0.M2:compile
[DEBUG]       org.eclipse.aether:aether-impl:jar:0.9.0.M2:compile
[DEBUG]       org.eclipse.aether:aether-api:jar:0.9.0.M2:compile
[DEBUG]       org.eclipse.aether:aether-util:jar:0.9.0.M2:compile
[DEBUG]       org.codehaus.plexus:plexus-interpolation:jar:1.16:compile
[DEBUG]       org.codehaus.plexus:plexus-utils:jar:3.0.10:compile
[DEBUG]       org.codehaus.plexus:plexus-classworlds:jar:2.4.2:compile
[DEBUG]       org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:compile
[DEBUG]       org.sonatype.plexus:plexus-sec-dispatcher:jar:1.3:compile
[DEBUG]          org.sonatype.plexus:plexus-cipher:jar:1.4:compile
[DEBUG] Created new class realm plugin>com.indeed:proctor-maven-plugin:1.0.0
[DEBUG] Importing foreign packages into class realm plugin>com.indeed:proctor-maven-plugin:1.0.0
[DEBUG]   Imported:  < maven.api
[DEBUG] Populating class realm plugin>com.indeed:proctor-maven-plugin:1.0.0
[DEBUG]   Included: com.indeed:proctor-maven-plugin:jar:1.0.0
[DEBUG]   Included: com.indeed:proctor-codegen:jar:1.0.0
[DEBUG]   Included: com.indeed:proctor-common:jar:1.0.0
[DEBUG]   Included: com.indeed:proctor-tomcat-deps:pom:1.0.0
[DEBUG]   Included: org.apache.tomcat:tomcat-el-api:jar:7.0.8
[DEBUG]   Included: org.apache.tomcat:tomcat-jasper-el:jar:7.0.8
[DEBUG]   Included: org.apache.tomcat:tomcat-jsp-api:jar:7.0.8
[DEBUG]   Included: org.apache.tomcat:tomcat-servlet-api:jar:7.0.8
[DEBUG]   Included: com.indeed:util-core:jar:1.0.0
[DEBUG]   Included: com.indeed:util-varexport:jar:1.0.0
[DEBUG]   Included: jstl:jstl:jar:1.2
[DEBUG]   Included: log4j:log4j:jar:1.2.14
[DEBUG]   Included: org.codehaus.jackson:jackson-core-asl:jar:1.8.0
[DEBUG]   Included: org.codehaus.jackson:jackson-mapper-asl:jar:1.8.0
[DEBUG]   Included: com.google.code.findbugs:jsr305:jar:1.3.9
[DEBUG]   Included: com.google.guava:guava:jar:15.0
[DEBUG]   Included: org.springframework:spring-core:jar:3.0.5.RELEASE
[DEBUG]   Included: org.springframework:spring-asm:jar:3.0.5.RELEASE
[DEBUG]   Included: commons-logging:commons-logging:jar:1.1.1
[DEBUG]   Included: org.springframework:spring-beans:jar:3.0.5.RELEASE
[DEBUG]   Included: com.indeed:proctor-consumer:jar:1.0.0
[DEBUG]   Included: commons-lang:commons-lang:jar:2.4
[DEBUG]   Included: org.springframework:spring-web:jar:3.0.5.RELEASE
[DEBUG]   Included: aopalliance:aopalliance:jar:1.0
[DEBUG]   Included: org.springframework:spring-context:jar:3.0.5.RELEASE
[DEBUG]   Included: org.springframework:spring-aop:jar:3.0.5.RELEASE
[DEBUG]   Included: org.springframework:spring-expression:jar:3.0.5.RELEASE
[DEBUG]   Included: org.freemarker:freemarker:jar:2.3.16
[DEBUG]   Included: org.apache.maven.plugin-tools:maven-plugin-annotations:jar:3.2
[DEBUG]   Included: org.eclipse.sisu:org.eclipse.sisu.plexus:jar:0.0.0.M2a
[DEBUG]   Included: javax.enterprise:cdi-api:jar:1.0
[DEBUG]   Included: javax.annotation:jsr250-api:jar:1.0
[DEBUG]   Included: javax.inject:javax.inject:jar:1
[DEBUG]   Included: org.sonatype.sisu:sisu-guice:jar:no_aop:3.1.0
[DEBUG]   Included: org.eclipse.sisu:org.eclipse.sisu.inject:jar:0.0.0.M2a
[DEBUG]   Included: asm:asm:jar:3.3.1
[DEBUG]   Included: org.eclipse.aether:aether-spi:jar:0.9.0.M2
[DEBUG]   Included: org.eclipse.aether:aether-impl:jar:0.9.0.M2
[DEBUG]   Included: org.eclipse.aether:aether-api:jar:0.9.0.M2
[DEBUG]   Included: org.eclipse.aether:aether-util:jar:0.9.0.M2
[DEBUG]   Included: org.codehaus.plexus:plexus-interpolation:jar:1.16
[DEBUG]   Included: org.codehaus.plexus:plexus-utils:jar:3.0.10
[DEBUG]   Included: org.codehaus.plexus:plexus-component-annotations:jar:1.5.5
[DEBUG]   Included: org.sonatype.plexus:plexus-sec-dispatcher:jar:1.3
[DEBUG]   Included: org.sonatype.plexus:plexus-cipher:jar:1.4
[DEBUG]   Excluded: org.apache.maven:maven-artifact:jar:3.0
[DEBUG]   Excluded: org.apache.maven:maven-plugin-api:jar:3.1.0
[DEBUG]   Excluded: org.apache.maven:maven-model:jar:3.1.0
[DEBUG]   Excluded: org.apache.maven:maven-core:jar:3.1.0
[DEBUG]   Excluded: org.apache.maven:maven-settings:jar:3.1.0
[DEBUG]   Excluded: org.apache.maven:maven-settings-builder:jar:3.1.0
[DEBUG]   Excluded: org.apache.maven:maven-repository-metadata:jar:3.1.0
[DEBUG]   Excluded: org.apache.maven:maven-model-builder:jar:3.1.0
[DEBUG]   Excluded: org.apache.maven:maven-aether-provider:jar:3.1.0
[DEBUG]   Excluded: org.codehaus.plexus:plexus-classworlds:jar:2.4.2
[DEBUG] Configuring mojo com.indeed:proctor-maven-plugin:1.0.0:generate from plugin realm ClassRealm[plugin>com.indeed:proctor-maven-plugin:1.0.0, parent: sun.misc.Launcher$AppClassLoader@749cd006]
[DEBUG] Configuring mojo 'com.indeed:proctor-maven-plugin:1.0.0:generate' with basic configurator -->
[DEBUG]   (f) outputDirectory = proctor-demo-master\target\generated-sources\proctor
[DEBUG]   (f) project = MavenProject: com.indeed:proctor-demo:0.0.1-SNAPSHOT @ proctor-demo-master\pom.xml
[DEBUG]   (f) topDirectory = proctor-demo-master\src\main\proctor
[DEBUG] -- end configuration --
[INFO] Building resources for com.indeed.demo
log4j:WARN No appenders could be found for logger (freemarker.cache).
log4j:WARN Please initialize the log4j system properly.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.570s
[INFO] Finished at: Sun Jun 08 14:55:43 EEST 2014
[INFO] Final Memory: 9M/154M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.indeed:proctor-maven-plugin:1.0.0:generate (default-cli) on project proctor-demo: Execution default-cli of goal com.indeed:proctor-maven-plugin:1.0.0:generate failed: String index out of range: 1 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.indeed:proctor-maven-plugin:1.0.0:generate (default-cli) on project proctor-demo: Execution default-cli of goal com.indeed:proctor-maven-plugin:1.0.0:generate failed: String index out of range: 1
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-cli of goal com.indeed:proctor-maven-plugin:1.0.0:generate failed: String index out of range: 1
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
        ... 19 more
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 1
        at java.lang.String.charAt(String.java:658)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:762)
        at java.util.regex.Matcher.replaceAll(Matcher.java:906)
        at java.lang.String.replaceAll(String.java:2162)
        at com.indeed.proctor.consumer.gen.FreeMarkerCodeGenerator.packageToPath(FreeMarkerCodeGenerator.java:72)
        at com.indeed.proctor.consumer.gen.FreeMarkerCodeGenerator.generate(FreeMarkerCodeGenerator.java:86)
        at com.indeed.proctor.consumer.gen.TestGroupsGenerator.generate(TestGroupsGenerator.java:34)
        at com.indeed.proctor.consumer.gen.maven.AbstractProctorMojo.processFile(AbstractProctorMojo.java:29)
        at com.indeed.proctor.consumer.gen.maven.AbstractProctorMojo.searchDirectory(AbstractProctorMojo.java:46)
        at com.indeed.proctor.consumer.gen.maven.AbstractProctorMojo.searchDirectory(AbstractProctorMojo.java:43)
        at com.indeed.proctor.consumer.gen.maven.AbstractProctorMojo.searchDirectory(AbstractProctorMojo.java:43)
        at com.indeed.proctor.consumer.gen.maven.AbstractProctorMojo.searchDirectory(AbstractProctorMojo.java:43)
        at com.indeed.proctor.consumer.gen.maven.AbstractProctorMojo.execute(AbstractProctorMojo.java:70)
        at com.indeed.proctor.consumer.gen.maven.ProctorMojo.execute(ProctorMojo.java:35)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
        ... 20 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Tried to change version of plugin to 1.0.7 but had the same error.
Thanks.

Renaming the default branch from "master" to "main"

The default branch name for repositories created on GitHub is now main. See github guidance

Promoting a previously deleted test with the same name

We've identified an issue where it's not possible to promote a test once it's been deleted. We believe the issue is here https://github.com/indeedeng/proctor-webapp-library/blob/master/src/main/java/com/indeed/proctor/common/ProctorPromoter.java#L118 where it's finding the last commit for that test even though it was a "Deleting A/B test" message in the git log. We could try to contribute a fix but it seems that this repo has now been integrated with the main proctor repo and there does not appear to be any new releases of that project since 2018. Any recommendations?

log4j 1.2.14 is vulnerable deserialization of untrusted data (CVE-2019-17571)

Log4j needs to be updated (or replaced) to fix this vulnerability

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17571

Broken link in README

The README links to https://engineering.indeedblog.com/blog/2013/05/indeedeng-data-driven-product-design-slides-video/ which is no longer working.

Don't log exception call stack when advising that error can be ignored

When the proctor loader is verifying rules, if any identifiers required by rules are not instantiated in the test context, it logs an exception at DEBUG level with the message:

Rule ${RULE} contains uninstantiated identifier(s) in [IDENTIFIERS], ignore the failure

The exception call stack is pretty noisy and distracting. I propose to log the message (still at DEBUG), without the exception, i.e. changing the word "ignore" to "ignoring":

Rule ${RULE} contains uninstantiated identifier(s) in [IDENTIFIERS], ignoring the failure

proctor/proctor-common/src/main/java/com/indeed/proctor/common/RuleVerifyUtils.java

Line 69 in ed5afc1

LOGGER.debug(String.format("Rule %s contains uninstantiated identifier(s) in %s, ignore the failure", testRule, absentIdentifiers), e);

LICENSE file does not list copyright holder or copyright year

I noticed the LICENSE file has boilerplate not filled in:

Copyright [yyyy] [name of copyright owner]

This should say Copyright 20xx Indeed, correct?