controlm/automation-api-community-solutions

Control-M Automation API community solutions

Control-M Automation API community solutions

This repository contains code samples and how-to for Control-M Automation API.

• Download Workbench for Docker - the latest development Control-M environment.
• Download Automation API CLI (ctm-cli.tgz).
• Installation instructions.

Online Documentation

You can find the latest Control-M Automation API documentation, including a programming guide, on the project web page.

Contribution guide

To contribute, please follow these guidelines.

Files, folders and naming conventions

1. Every sample and its associated files must be contained in its own folder. Name this folder something that describes what your sample does. Usually this naming pattern looks like level-sample-purpose (e.g. 201-automate-corrective-flow). Numbering should start at 101. 100 is reserved for things that need to be at the top.

   For consistent categorization, please comply to the following folder structure:

   • 1-general-examples
   • 2-cicd-tooling-integration
   • 3-infrastructure-as-code-examples
   • 4-ai-job-type-examples
   • 5-bots-and-dashboard-examples
   • 6-ide-integrations
   • 7-api-gateway-integrations
   • 8-jobs-as-code
   • 9-aws-integrations

2. For consistent ordering create all folders in lowercase.

3. Include a README.md file that explains the sample. A good description helps other community members to understand your sample. The README.md uses Github Flavored Markdown for formatting text. If you want to add images to your README.md file, store the images in the images folder. Reference the images in the README.md with a relative path (e.g. ![alt text](images/sampleImage.png "Sample Image Text")). This ensures the link will reference the target repository if the source repository is forked.

Questions/Feedback

Please use issues on GitHub for questions or feedback about the examples included in this repository.

automation-api-community-solutions's Issues

lcps_to_ccps.py doesn't handle AI passwords correctly

The lcps_to_ccps.py script converts local connection profiles to centralized connection profiles which includes changing the value of the Password field in the exported JSON to a secret. I found that when trying to convert Application Integrator profiles the script does not account for the fact that the password field name for AI jobs is named AI-Password instead.

To illustrate this below is a profile I exported using CTM CLI:

  "LOCKBOX": {
    "Type": "ConnectionProfile:ApplicationIntegrator:ZCONNECTWF",
    "AI-Timeout (minutes)": "60",
    "AI-Status Interval (seconds)": "600",
    "AI-Password": "*****",
    "AI-Zuora Workflow ID": "82290",
    "AI-Host": "workflow.apps.zuora.com",
    "TargetAgent": "agentname",
    "TargetCTM": "servername"
  }

When I use the script to generate the temporary JSON file for that same profile you can see that it has added the Password element:

    "LOCKBOX": {
        "Type": "ConnectionProfile:ApplicationIntegrator:ZCONNECTWF",
        "AI-Timeout (minutes)": "60",
        "AI-Status Interval (seconds)": "600",
        "AI-Password": "*****",
        "AI-Zuora Workflow ID": "82290",
        "AI-Host": "workflow.apps.zuora.com",
        "Centralized": true,
        "Password": {
            "Secret": "lockbox_secret"
        }

For the moment I am going to use the workaround of modifying the JSON file before telling the script to deploy it but I am sure the script could be refactored to handle AI passwords properly. When I have time I will likely create my own fork and work on that.

Can i generate job configs from exsiting jobs in my Controlm ? So I can use them to automate going forward.

Java 11.0.12 minimum version required for Agent 9.0.21.000+

The Control M Agent 9.0.21.000+ require a minimum Java version of 11.0.12+.

automation-api-community-solutions/3-infrastructure-as-code-examples/tutorial-docker-image-with-embedded-controlm-agent/Dockerfile

Line 27 in 7ce4242

RUN yum -y install java-1.8.0-openjdk \

I have tried to replace openjdk package with Amazon Corretto Headed/Headless package and getting issues with using the automation api to provision the Agent 9.0.21.000 image.
RUN yum install -y java-11-amazon-corretto-headless
changing the package java-1.8.0-openjdk to java-11-amazon-corretto-headless or java-11-amazon-corretto results in the error below.

Cannot find Java. The Java major version must be 11. The minor version must be 11.0.12 or higher. Set the BMC_INST_JAVA_HOME environment variable to the Java home path and then rerun the installation. Refer to Control-M Documentation for more information.

�[32minfo�[39m:    Failed to deploy new image
�[91m�[31merror�[39m:   org.apache.commons.exec.ExecuteException: return error for product DRFZ4.9.0.21.000_Linux-x86_64.tar.z-DEPLOY: using command /home/tomcat/.ctmTemp/DRFZ4.9.0.21.000_Linux-x86_64.tar/setup.sh -silent /home/tomcat/.ctmTemp/agent.linux.xml. rc=1 (Exit value: 1)

�[0m�[32minfo�[39m:    exiting provision process with exit code (0-ok):6
�[91m�[31merror�[39m:   provision image failed
�[0mThe command '/bin/sh -c cd     && ctm provision image $AGENT_DEPLOY_FILE' returned a non-zero code: 1

Do you have a solution for this, or a workaround?

How to set Java heap space for control m agent

Hi,

I am following the example kubernetes-statefulset-agent-using-pvc to run ctm on kuberntes. I am able to build the image but when I execute it I get java.lang.OutOfMemoryError. I believe I fix it increasing the heap size. Does someone know where can I set it up?

Here are the logs:

│ info:    Making SSL trust all certificates and all hostnames                                                                          │
│ info:    setting server to agent port: 7006                                                                                           │
│ info:    setting agent to server port: 7005                                                                                           │
│ info:    setting agent name (alias): statefulset-agent-0                                                                              │
│ info:    setting primary Control-M Server: 10.228.169.20                                                                              │
│ info:    setting authorized Control-M Server host                                                                                     │
│ info:    setting agent communication type to persistent                                                                               │
│ error:   Exception in thread "Exec Stream Pumper" java.lang.OutOfMemoryError: Java heap space                                         │
│     at java.base/java.util.Arrays.copyOf(Arrays.java:3745)                                                                            │
│     at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)                                                   │
│     at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)                                          │
│     at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)                                                  │
│ error:                                                                                                                                │
│     at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:108)                                                                │
│     at java.base/java.lang.Thread.run(Thread.java:834)                                                                                │

Docker image wont build when using workbench VM

I tried to follow the tutorial here using the workbench appliance running in VirtualBox and the docker build step fails with a connection refused being returned by the workbench server.

--2019-10-07 18:48:22--  https://localhost:8443/automation-api/ctm-cli.tgz
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:8443... failed: Connection refused.
Connecting to localhost (localhost)|::1|:8443... failed: Cannot assign requested address.

MacOS: 10.14.6
Docker: 19.03.2
VirtualBox: 6.0.12
Workbench Image: 9.0.18.300-20190218.133426-1

Kubernetes Control M Agent example doesn't work with EM 9.20.0

Hi,

I am trying to build the control m agent image using the instructions provided on https://github.com/controlm/automation-api-community-solutions/tree/master/3-infrastructure-as-code-examples/kubernetes-statefulset-agent-using-pvc. Our control M EM version is 9.20.0

I'm getting the following error while building the image

 => ERROR [builder 10/14] RUN  ctm provision image Agent.Linux && echo installation ended successfully                                                                                          17.1s
------                                                                                                                                                                                                
 > [builder 10/14] RUN  ctm provision image Agent.Linux && echo installation ended successfully:                                                                                                      
#14 0.567 debug:   Locating java command                                                                                                                                                              
#14 0.671 info:    Located java at:/bin/java                                                                                                                                                          
#14 0.671 info:    downloading https://10.228.158.10:8446/automation-api/utils/control-m.services.provision-9.20.0.jar into /home/controlm/.ctm/control-m.services.provision-9.20.0.jar               
info:    4MB/4MB precent: 100%
#14 5.888 debug:   starting command: /bin/java -jar /home/controlm/.ctm/control-m.services.provision-9.20.0.jar -image "Agent.Linux" -server https://10.228.158.10:8446/automation-api -action image -environment myenv -ctms "" -name "" -port "" -cert 0 -file ""
#14 6.148 info:    Making SSL trust all certificates and all hostnames
#14 8.382 info:    image descriptor Agent.Linux was successfully downloaded and parsed
#14 8.383 info:    handling product DRKAI.9.0.00_Linux-x86_64.tar.Z
#14 8.559 info:    downloading https://10.228.158.10:8446/automation-api/silent/agent.linux.xml to /home/controlm/.ctmTemp/agent.linux.xml
#14 10.03 info:    downloading https://10.228.158.10:8446/automation-api/descriptors/ProvisionConfig.json to /home/controlm/.ctmTemp/ProvisionConfig.json
#14 10.06 info:    downloading https://10.228.158.10:8446/automation-api/DRKAI.9.0.00_Linux-x86_64.tar.Z to /home/controlm/.ctmTemp/DRKAI.9.0.00_Linux-x86_64.tar.Z
#14 10.08 info:    file https://10.228.158.10:8446/automation-api/DRKAI.9.0.00_Linux-x86_64.tar.Z not found, may be valid in some cases...
#14 10.08 info:    100% (0MB/0MB) - downloading ended
#14 17.06 info:    Failed to deploy new image
#14 17.06 error:   java.io.IOException: Product 'DRKAI.9.0.00_Linux-x86_64.tar.Z' listed in image 'Agent.Linux' not found
#14 17.06 
#14 17.06 info:    exiting provision process with exit code (0-ok):6
#14 17.09 debug:   image failed: exit code: 6 for '/bin/java -jar /home/controlm/.ctm/control-m.services.provision-9.20.0.jar -image "Agent.Linux" -server https://10.228.158.10:8446/automation-api -action image -environment myenv -ctms "" -name "" -port "" -cert 0 -file ""'

I tried doing curl on https://10.228.158.10:8446/automation-api/DRKAI.9.0.00_Linux-x86_64.tar.Z but got the following error response

{ "errors" : [
    {"message":"Not Found", "code" : "404", "uri" : "/DRKAI.9.0.00_Linux-x86_64.tar.Z"} ]
  }

I also tried for the agent image Agent_18.Linux but got the same response.

Could you please help me to resolve this issue?

Thanks
Anshul

kubernetes-statefulset-agent-using-pvc: ctmping fails from the server

Hi,

I am following the tutorial kubernetes-statefulset-agent-using-pvc to run control m agents on Kubernetes and have followed all the steps as mentioned in the read me.

I am successfully able to run the command kubectl exec -it statefulset-agent-0 -- tcsh -c ag_ping but when I try ctmping -HOSTID statefulset-agent-0 on the server it fails with the error unknown host as the statefulset-agent- doesn't exist
Is there same sort of configuration (networking) that is required?

As we are using the ConnectionInitiation=AgentToSever does it mean that everytime server and agent communication needs to happen agent will make a connection and hence server doesn't need to know about the the location of agent as long as agent is able to communicate with server?

Review of this pattern

Hi,

I'm working on the installation of control-m agent on a docker container and using the pattern where control-m agent has kubernetes service object attached to it which provides a static IP to the agent for incoming connections from the server.

Please note, for the installation of control-m agent on the docker image, I am executing the steps that we will perform on any Linux VM i.e. downloading the DRKAI package, unzipping and running setup.sh command. I didn't use ctm cli as it requires the EM endpoint and credentials at the build time which means that if you have got multiple environments each having its own EM then you need to build different image for each of them and that I believe is an anti-pattern in container world.

Also, I am running the setup.sh at the build time with a placeholder for server name in silent config file. Later during start up time, I replace that placeholder in ctm/data/CONFIG.dat with the actual server name. (I still need to handle authorised servers in case of HA).

Following is the snippet of my docker file

FROM centos:7
ARG CTM_HOME=/home/controlm
ARG AGENT_BINARY_URL=
ENV CTM_DIR=$CTM_HOME
RUN yum -y update \
	&& yum -y install wget \
	&& yum -y install sudo \
	&& yum -y install net-tools \
	&& yum -y install bc

# Add controlm user and root to soduers list
RUN useradd -d $CTM_DIR -m controlm \  
	&& echo 'root ALL=(ALL) ALL' >> /etc/sudoers \
	&& echo 'controlm ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers

USER controlm
WORKDIR $CTM_DIR

RUN  wget -q $AGENT_BINARY_URL

RUN tar -xvf $ARTIFACT_NAME
RUN rm -rf $ARTIFACT_NAME \
    && chmod 755 setup.sh \
    && chmod -R 755 Setup_files

COPY install-controlm-silent-config.xml install-controlm-silent-config.xml
RUN ./setup.sh -silent install-controlm-silent-config.xml
RUN rm -rf Setup_files

ADD run_ctm.sh run_ctm.sh
RUN sudo chmod 755 run_ctm.sh
RUN echo "source $CTM_DIR/.bash_profile" >> $CTM_DIR/.bashrc

EXPOSE 7000-8000

CMD $CTM_DIR/run_ctm.sh

run.sh (start up script)

function run_ctrlm {
  if [ ! $CTM_SERVER_HOST ] ; then
    echo "Environment variable CTM_SERVER_HOST is missing."
    exit 1
  fi
  source ~/.bash_profile
  sed -i -e "s/CTM_SERVER_HOST/$CTM_SERVER_HOST/g" $CTM_DIR/ctm/data/CONFIG.dat
  sed -i -e 's|/bin/su - $agent_owner -c "||g' -e 's|ALL"|ALL|g' $CTM_DIR/ctm/scripts/rc.agent_user
  $CTM_DIR/ctm/scripts/rc.agent_user start
}

run_ctrlm

while /bin/true; do
  sleep 60
done

install-controlm-silent-config.xml

<AutomatedInstallation langpack="eng">
    <target.product>Control-M/Agent 9.0.20.000</target.product>
    <agent.parameters>
        <entry key="field.Authorized.Controlm.Server.Host" value="CTM_SERVER_HOST"/>
        <entry key="ctm_agent.Tracker.Event.Port" value="7035"/>
        <entry key="field.Agent.To.Server.Port.Number" value="7005"/>
        <entry key="ctm_agent.Tcp_ip.Timeout" value="60"/>
        <entry key="field.Server.To.Agent.Port.Number" value="7006"/>
        <entry key="Ignore.Disabling.Agent.Failure" value="false"/>
        <entry key="ctm_agent.Force.Upgrade" value="false"/>
        <entry key="field.Primary.Controlm.Server.Host" value="CTM_SERVER_HOST"/>
    </agent.parameters>
</AutomatedInstallation>

Can you please provide the feedback on this approach?

Also, I want to use persistent volume claim to persist the logs and output for troubleshooting if the pod goes down. So far I understand that I should definitely persist proclog and output directory, but are there some other folders we should persist?

using the jenkins CICD integration for a project

I'm setting up something similar to what you've got in your jenkins CICD integration ( automation-api-community-solutions/2-cicd-tooling-integration/integrating-Control-M-into-DevOps-lifecycle/) but am struggling to understand how this approach will work for a project with multiple folders of jobs... would it be normal to need to create a new jenkins pipeline for each folder as I would think there could be hundreds of these in each project or would the script need to be adapted to cat all the jobs into one master file just for validating build