Variable substitution in URL only works for finished builds

[jelco]

Variable substitution in URL only works for finished builds

[netwolfuk]

Hi @jelco , which template and format are you using?

[jelco]

Tailored JSON without a template. Here, I'll just include the full specification of this particular webhook (with some anonymization):

<webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
  <states>
    <state type="buildFailed" enabled="false" />
    <state type="beforeBuildFinish" enabled="false" />
    <state type="buildFinished" enabled="false" />
    <state type="buildStarted" enabled="true" />
    <state type="buildFixed" enabled="false" />
    <state type="buildBroken" enabled="false" />
    <state type="buildSuccessful" enabled="false" />
    <state type="buildInterrupted" enabled="false" />
    <state type="responsibilityChanged" enabled="false" />
  </states>
  <build-types enabled-for-all="true" enabled-for-subprojects="true" />
  <parameters>
    <param name="body" value="{&quot;state&quot;: &quot;INPROGRESS&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
  </parameters>
  <auth enabled="true" type="userpass" preemptive="true">
    <auth-parameters>
      <param name="password" value="password" />
      <param name="username" value="username" />
    </auth-parameters>
  </auth>
</webhook>

I'm running the custom-templates branch (the latest version you supplied with the pre-emptive authentication).

[netwolfuk]

I'm having trouble getting your example to work for any build state. Can you confirm that it works for you for the finished states. (I note that the finished states are set to false above).

[jelco]

Ah, I understand the confusion. I have three separate webhooks (since the content is different per hook). The full configuration is as follows:

<webhooks enabled="true">
  <webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
    <states>
      <state type="buildFailed" enabled="false" />
      <state type="beforeBuildFinish" enabled="false" />
      <state type="buildFinished" enabled="false" />
      <state type="buildStarted" enabled="true" />
      <state type="buildFixed" enabled="false" />
      <state type="buildBroken" enabled="false" />
      <state type="buildSuccessful" enabled="false" />
      <state type="buildInterrupted" enabled="false" />
      <state type="responsibilityChanged" enabled="false" />
    </states>
    <build-types enabled-for-all="true" enabled-for-subprojects="true" />
    <parameters>
      <param name="body" value="{&quot;state&quot;: &quot;INPROGRESS&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
    </parameters>
    <auth enabled="true" type="userpass" preemptive="true">
      <auth-parameters>
        <param name="password" value="password" />
        <param name="username" value="username" />
      </auth-parameters>
    </auth>
  </webhook>
  <webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
    <states>
      <state type="buildFailed" enabled="false" />
      <state type="beforeBuildFinish" enabled="false" />
      <state type="buildFinished" enabled="true" />
      <state type="buildStarted" enabled="false" />
      <state type="buildFixed" enabled="false" />
      <state type="buildBroken" enabled="false" />
      <state type="buildSuccessful" enabled="true" />
      <state type="buildInterrupted" enabled="false" />
      <state type="responsibilityChanged" enabled="false" />
    </states>
    <build-types enabled-for-all="true" enabled-for-subprojects="true" />
    <parameters>
      <param name="body" value="{&quot;state&quot;: &quot;SUCCESSFUL&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
    </parameters>
    <auth enabled="true" type="userpass" preemptive="true">
      <auth-parameters>
        <param name="password" value="password" />
        <param name="username" value="username" />
      </auth-parameters>
    </auth>
  </webhook>
  <webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
    <states>
      <state type="buildFailed" enabled="true" />
      <state type="beforeBuildFinish" enabled="false" />
      <state type="buildFinished" enabled="true" />
      <state type="buildStarted" enabled="false" />
      <state type="buildFixed" enabled="false" />
      <state type="buildBroken" enabled="false" />
      <state type="buildSuccessful" enabled="false" />
      <state type="buildInterrupted" enabled="false" />
      <state type="responsibilityChanged" enabled="false" />
    </states>
    <build-types enabled-for-all="true" enabled-for-subprojects="true" />
    <parameters>
      <param name="body" value="{&quot;state&quot;: &quot;FAILED&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
    </parameters>
    <auth enabled="true" type="userpass" preemptive="true">
      <auth-parameters>
        <param name="password" value="password" />
        <param name="username" value="username" />
      </auth-parameters>
    </auth>
  </webhook>
</webhooks>

[netwolfuk]

I have been able to reproduce the problem. Using the following URL:
http://localhost:8111/2.0/repositories/organization/${project}/commit/${revision}/statuses/build

And the properties on the build as so:
webhook.project %system.teamcity.projectName%
webhook.revision %system.build.vcs.number%

Before the build starts, TeamCity has not done any change gathering, (and I assume, resolved any parameters).

I suspect that is probably caused by the same as this issue in which Pavel states:

Before changes are collected in build, we actually do not know what will be the display name of the branch. This is why on buildStarted event you get , but on changesLoaded event if changes were collected without errors, you'll see master. So far this is by design.

So the correct answer is probably for me to add "changesLoaded" build event to the listener.

[netwolfuk]

Created #28, and will close this ticket.

[netwolfuk]

ChangesLoaded support added see commit ca4e0cf
It has just built and is ready to download

See if that gives you what you want.

[jelco]

I disabled the buildStarted state line and added this line to the configuration:

<state type="changesLoaded" enabled="true" />

Unfortunately, the result doesn't change a whole lot. I can definitely notice that the hook is triggered at a later stage so it looks like it's picking up the other event just fine, but the error isn't any different:

Error in event handler: java.lang.IllegalArgumentException: Invalid uri 'https://api.bitbucket.org/2.0/repositories/organization/%system.teamcity.projectName%/commit/%system.build.vcs.number%/statuses/build': Invalid URL encoding (enable debug to see stacktrace)

[netwolfuk]

That's a real shame. The first one should just be available as ${projectName} as that's a tcwebhooks PayloadContent bean property. I can't remember if I added the vcs stuff to the bean yet though or how you'd access it. I'll have a dig into it and see.

[netwolfuk]

https://youtrack.jetbrains.com/issue/TW-45285

[netwolfuk]

Hi @Lurenlzm @jelco
I have added experimental support for forcing TeamCity to resolve variables before it would normally do so. This means that (in my limited testing) certain variables can be resolved before the build started event fires.

To test it, please download the plugin and rest plugin from the issue_26-force_resolve_variables branch. Just login as a guest to download the artifacts.

This branch adds support for a new settings on WebHook Project Parameters called "Forced Resolve".

To use this feature:

1. Install a 1.2.0-alpha8 or later tcWebHooks plugin and tcWebHooks REST plugin.
2. Create a WebHooks Parameter for your project. The page to create them is on the bottom of the webhook edit page.
3. Set the name you want to use in your webhook template or URL. eg, my_var_name
4. Set the value to refer to a TeamCity variable using Teamcity Syntax eg, %my_parameter%. Note the use of % not ${}.
5. Set the Force Reload option to Attempt to resolve variable contents early via TeamCity.
6. Refer to ${my_var_name} in your webhook configuration (eg, template, URL or other webhook setting).

tcWebHooks will ask TeamCity to early resolve that parameter before the webhook is executed.

[jan0sch]

Hi, at first thanks for your work. ❤️
I just tried this out but still get the malformed url error message.

Configuration of a custom parameter as described in your screenshot:

Used in the URL via ${CommitHash} but getting the error message:

999 :: Unexpected exception. Please log a bug on GitHub tcplugins/tcWebHooks. Exception was: Malformed escape pair at index 66: https://XXX/api/v1/repos/XXX/XXX/statuses/%build.vcs.number%

For me it looks like the variable is not substituted but the reference to the TeamCity variable (%vcs.build.number% in my case) is just taken as is.

[netwolfuk]

Hi @jan0sch thanks so much for taking the time to test this.

I presume this is for build started. Is it the same for changes loaded?

I'm not at my computer so I can't remember the exact text, but there should be some debug logging in the teamcity-server.log for something like WebHookExtraParameters indicating that it's trying to force resolve that variable. Can you please check is that's running?

Thanks again for your time on this.

[NEdwards-Sharkmob]

Hi @Lurenlzm @jelco I have added experimental support for forcing TeamCity to resolve variables before it would normally do so. This means that (in my limited testing) certain variables can be resolved before the build started event fires.

To test it, please download the plugin and rest plugin from the issue_26-force_resolve_variables branch. Just login as a guest to download the artifacts.

This branch adds support for a new settings on WebHook Project Parameters called "Forced Resolve".

To use this feature:

1. Install a 1.2.0-alpha8 or later tcWebHooks plugin and tcWebHooks REST plugin.
2. Create a WebHooks Parameter for your project. The page to create them is on the bottom of the webhook edit page.
3. Set the name you want to use in your webhook template or URL. eg, my_var_name
4. Set the value to refer to a TeamCity variable using Teamcity Syntax eg, %my_parameter%. Note the use of % not ${}.
5. Set the Force Reload option to Attempt to resolve variable contents early via TeamCity.
6. Refer to ${my_var_name} in your webhook configuration (eg, template, URL or other webhook setting).

tcWebHooks will ask TeamCity to early resolve that parameter before the webhook is executed.

I installed this version as I've been experiencing the same issue.

I found that using the force early resolve fixed the issue I was having where my %build.vcs.number% didn't resolve in Changes Loaded, but it didn't resolve for Build Succeeded or Build Failed events. I was able to verify this with the test webhooks feature.

However, when I tried using ${build.vcs.number} as the value in my project webhook parameter, this worked for all 3 (changes loaded, build succeeded, build failed) - verified via the test feature and in use. Not sure if this would solve the issue being described here, but figured it would be worth mentioning. It would be handy to know if issues could result from doing this too.

[netwolfuk]

Hi @NEdwards-Sharkmob

Thanks for taking the time to test this.

I made an assumption that we'd only want to force resolve for early build stages. From memory, I don't think the force resolve tries to resolve the values in the build finished event. However for consistency, I should add that in.

Please also bear in mind that testing these sorts of variables in the UI based on a finished build often gives misleading results because variables have been resolved by TeamCity during the build and will appear resolved even if the event being tested is an early one. I see you mentioned that you did also test with a real build, but I just wanted to point that out for other people on this thread.

I guess when we're talking about change related metadata it makes sense that those values wont resolve until changes are loaded. I don't know enough about the inner workings of TeamCity to answer that authoritatively.

I'll check the force resolving on later build events and add if required.

[netwolfuk]

Hi all. A new build is available. This enables the resolver to run on the following:
state.equals(BuildStateEnum.BUILD_FINISHED) || state.equals(BuildStateEnum.BUILD_SUCCESSFUL) || state.equals(BuildStateEnum.BUILD_FAILED) || state.equals(BuildStateEnum.BUILD_FIXED) || state.equals(BuildStateEnum.BUILD_BROKEN)

Available from the issue_26-force_resolve_variables link.

[jan0sch]

Variable resolution seems to work now. Thank you very much.

[netwolfuk]

1.2.0 has been released. Closing this issue