jfilak/sapcli

Command Line Interface for interaction with SAP tools

sapcli's Issues

potential issues in gcts activities

use is None intead of "not"
RepoActivitiesQueryParams:

use is None instead of not
activities

mass activate

It executes GET /sap/bc/adt/activation/inactiveobjects and then POST /sap/bc/adt/activation?method=activate&preauditRequested=false containing URIs for children only.

'gcts repo property get' should use list instead of table

Current output:

Name      | RID       | Branch           | Commit                                   | Status | vSID | ROLE   | URL                                             
---------------------------------------------------------------------------------------------------------------------------------------------------------------
demo | demo | integra_pipeline | ffbc028c349d093d32353ce92e24bc449779b604 | READY  | 06B  | TARGET | https://github.com/sample/demo

More userfriendly output:

Name: demo
RID: demo
Branch: integra_pipeline
Commit: ffbc028c349d093d32353ce92e24bc449779b604
Status: READY
vSID: 06B
ROLE: TARGET
URL: https://github.com/sample/demo

rfc user must initialize some parameters

FIRSTNAME, LASTNAME, E_MAIL, BAPIPWD, ALIAS

Refactor /cli/checkout

"Now it is clear that file formats and file system should be in a different module. "

Unite commands return codes (exception handling).

add recursive syntax check for packages

aunit run for packages does not work on 740

Do you mind if I open issues where the cause might be a 740 SP20 backend?

Request using ADT:

<?xml version="1.0" encoding="UTF-8"?>
<aunit:runConfiguration xmlns:aunit="http://www.sap.com/adt/aunit">
  <external>
    <coverage active="false"/>
  </external>
  <adtcore:objectSets xmlns:adtcore="http://www.sap.com/adt/core">
    <objectSet kind="inclusive">
      <adtcore:objectReferences>
        <adtcore:objectReference adtcore:uri="/sap/bc/adt/vit/wb/object_type/devck/object_name/%2fABC%2fPACKAGE"/>
      </adtcore:objectReferences>
    </objectSet>
  </adtcore:objectSets>
</aunit:runConfiguration>

Request using sapcli ./sapcli aunit run package "/ABC/PACKAGE":

<?xml version="1.0" encoding="UTF-8"?>
<aunit:runConfiguration xmlns:aunit="http://www.sap.com/adt/aunit">
  <external>
    <coverage active="false"/>
  </external>
  <options>
    <uriType value="semantic"/>
    <testDeterminationStrategy appendAssignedTestsPreview="true" assignedTests="false" sameProgram="true"/>
    <testRiskLevels critical="true" dangerous="true" harmless="true"/>
    <testDurations long="true" medium="true" short="true"/>
  </options>
  <adtcore:objectSets xmlns:adtcore="http://www.sap.com/adt/core">
    <objectSet kind="inclusive">
      <adtcore:objectReferences>
        <adtcore:objectReference adtcore:uri="/sap/bc/adt/packages/%2Fabc%2Fpackage"/>
      </adtcore:objectReferences>
    </objectSet>
  </adtcore:objectSets>
</aunit:runConfiguration>

Result:

<?xml version="1.0" encoding="UTF-8"?>
<aunit:runResult xmlns:aunit="http://www.sap.com/adt/aunit">
  <alerts>
    <alert kind="noTestClasses" severity="tolerable">
      <title>Aufgabendefinition bezieht sich auf keinen Test</title>
      <stack/>
    </alert>
  </alerts>
</aunit:runResult>

It seems like my backend version requires the /sap/bc/adt/vit/wb/object_type/{type}/object_name/{name}-syntax to be used for packages?

Directly testing classes works fine ./sapcli aunit run class "/ABC/MY_CLASS".

sequential activate/mass activate/mass write + activate must create log with objects

We must have the list of objects that are supposed to be activated and was not yet activated

aunit syntax error causes tolerable abortions

Test results must not be reported as SKIP.

add abapgit repo settings Folder Logic

It must be possible to change FOLDER_LOGIC of linked repositories (abapGit settings docu -> https://docs.abapgit.org/ref-dot-abapgit.html)

It would be nice if "abapgit link" accepted some minimal basic settings like Folder Logic or Sub-packages or Starting Folder.

generate class includes

sap.adt.errors.ADTError: ExceptionResourceSaveFailure: ZCL_ABAPGIT_XML===============CCAU does not have any inactive version

<?xml version="1.0" encoding="utf-8"?>
<exc:exception xmlns:exc="http://www.sap.com/abapxml/types/communicationframework">
  <namespace id="com.sap.adt"/>
  <type id="ExceptionResourceSaveFailure"/>
  <message lang="EN">ZCL_ABAPGIT_SERVICES_REPO=====CCAU does not have any inactive version</message>
  <localizedMessage lang="EN">ZCL_ABAPGIT_SERVICES_REPO=====CCAU does not have any inactive version</localizedMessage>
  <properties>
    <entry key="T100KEY-ID">ED</entry>
    <entry key="T100KEY-NO">170</entry>
    <entry key="T100KEY-V1">ZCL_ABAPGIT_SERVICES_REPO=====CCAU</entry>
  </properties>
</exc:exception>

POST /sap/bc/adt/oo/classes/zcl_abapgit_xml/includes?lockHandle=FB8E3E52B4947663F4822A4EA71BFD1E31C73605 HTTP/1.1

Content-Type: application/vnd.sap.adt.oo.classincludes+xml

<?xml version="1.0" encoding="UTF-8"?><class:abapClassInclude xmlns:class="http://www.sap.com/adt/oo/classes" xmlns:adtcore="http://www.sap.com/adt/core" adtcore:name="dummy" class:includeType="testclasses"/>

HTTP/1.1 201 Created

Location  : /sap/bc/adt/oo/classes/zcl_abapgit_xml/includes/testclasses

CI doesn't work

It seems travis stopped working for your project. The build status in README shows "all green", but this is very old status (8months). And moreover, I see the number of pylint and flake8 findings are currently not zero. It would be great to setup CI again.

Tip: Let's you at least github actions, which are directly available from github. I'm offering my services to help you if you are interested. This task, of course, cannot be done via PR only, it requires some setup in repository settings.

requests module with alpine image

Hi Jakub,

I am currently trying to set up sapcli on GitLab to run unit tests on a test system every time a master build is triggered. This is my gitlab-ci.yml:

abapunit:
  image: python:3-alpine
  stage: unittest
  variables:
    SAP_ASHOST: '****'
    SAP_CLIENT: '***'
    SAP_PORT: '***'
  before_script:
    - apk add --no-cache git py3-requests py3-openssl
    - git clone https://github.com/jfilak/sapcli.git
  script:
    - ./sapcli/sapcli
        --ashost $SAP_ASHOST
        --client $SAP_CLIENT
        --no-ssl
        --port $SAP_PORT
        --user $SAP_USER
        --password $SAP_PASSWORD
        aunit run package "/abc/def"
        --output junit4 > report.xml && cat report.xml
  artifacts:
    reports:
      junit: report.xml

Output:

$ apk add --no-cache git py3-requests py3-openssl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/16) Installing nghttp2-libs (1.38.0-r0)
(2/16) Installing libcurl (7.65.1-r0)
(3/16) Installing pcre2 (10.33-r0)
(4/16) Installing git (2.22.0-r0)
(5/16) Installing python3 (3.7.3-r0)
(6/16) Installing py3-cparser (2.19-r2)
(7/16) Installing py3-cffi (1.11.5-r4)
(8/16) Installing py3-idna (2.8-r1)
(9/16) Installing py3-asn1crypto (0.24.0-r1)
(10/16) Installing py3-six (1.12.0-r1)
(11/16) Installing py3-cryptography (2.6.1-r1)
(12/16) Installing py3-openssl (19.0.0-r0)
(13/16) Installing py3-chardet (3.0.4-r1)
(14/16) Installing py3-certifi (2019.6.16-r0)
(15/16) Installing py3-urllib3 (1.25.3-r0)
(16/16) Installing py3-requests (2.21.0-r4)
Executing busybox-1.30.1-r2.trigger
OK: 97 MiB in 51 packages
$ git clone https://github.com/jfilak/sapcli.git
Cloning into 'sapcli'...
$ ./sapcli/sapcli --ashost $SAP_ASHOST --client $SAP_CLIENT --no-ssl --port $SAP_PORT --user $SAP_USER --password $SAP_PASSWORD aunit run package "*****" --output junit4 > report.xml && cat report.xml
Traceback (most recent call last):
  File "./sapcli/sapcli", line 20, in <module>
    spec.loader.exec_module(sapcli)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/builds/*****/sapcli/bin/sapcli", line 14, in <module>
    import sap.adt
  File "/builds/****/sapcli/sap/adt/__init__.py", line 3, in <module>
    from sap.adt.core import Connection  # noqa: F401
  File "/builds/****/sapcli/sap/adt/core.py", line 4, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
Uploading artifacts...
report.xml: found 1 matching files                 
Uploading artifacts to coordinator... ok            id=189 responseStatus=201 Created token=fFMjy5-c
ERROR: Job failed: exit code 1

It seems like the py3-requests package is not the one I need or is not included correctly. Any idea?

I tried Ubuntu at first which didn't show any syntax error but took minutes to run so I wanted to try the alpine image instead.

ATC Priorities: SAP GUI vs SAPCLI

Given I have a class with a syntax error:

When I run the ATC via SAP GUI, it reports two entries:

However, when I run it via sapcli:
./sapcli atc run class z_borinlu_sapcli

It reports the same two ATCs, but the first finding has a distinct priority:
* 2 :: Test Environment (SLIN_UMFLD) :: Program contains syntax errors (0000)
* 2 :: ABAP Unit: Tests of Runtime Category "Short" (BC_AUNIT) :: Critical Error reported from test class (CRIT)

You can simulate it in the [CENSORED] (SAP internal).

Ps: It uses the default/standard checkman variant, and the user parameter is empty (the one that forces code inspector).

Two stage class activation

It is not enough to activate the public class alone but every object that belongs to the class must be activate in one shot.

aunit junit4 does not catch syntax errors in classes

<?xml version="1.0" encoding="UTF-8"?>
<aunit:runResult xmlns:aunit="http://www.sap.com/adt/aunit">
  <program xmlns:adtcore="http://www.sap.com/adt/core" adtcore:uri="/sap/bc/adt/oo/classes/cl_foo" adtcore:type="CLAS/OC" adtcore:name="CL_FOO" uriType="semantic">
    <alerts>
      <alert aunit:hasSyntaxErrors="true" kind="warning" severity="critical">
        <title>CL_FOO has syntax errors and cannot be analyzed for existence of unit tests</title>
        <details>
          <detail text="&quot;ME-&gt;MEMBER&quot; is not type-compatible with formal parameter &quot;BAR&quot;."/>
        </details>
        <stack>
          <stackEntry adtcore:uri="/sap/bc/adt/oo/classes/cl_foo/includes/testclasses#start=428" adtcore:description="CL_FOO======CCAU:428"/>
        </stack>
      </alert>
    </alerts>
  </program>
</aunit:runResult>

ATC - ERROR_LEVEL is not working

Given the statement:
./sapcli --ashost WWWW --client 001 --skip-ssl-validation --port 44300 --user XXXX --password YYYY atc run class CL_CPE_CNV_DATA_ACCESS_ADAPTER -e 1

where -e 1 should means: priorities higher than one are not considered

it returns the Prio 2 without respecting the -e 1 rule:

* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - The Used Object is Not Visible (NVIS)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - The Used Object is Not Visible (NVIS)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - The Used Object is Not Visible (NVIS)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - The Used Object is Not Visible (NVIS)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - The Used Object is Not Visible (NVIS)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - The Used Object is Not Visible (NVIS)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - The Used Object is Not Visible (NVIS)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - Missing Use Access (USEM)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - Missing Use Access (USEM)
* 2 :: Package Check  (SLIN_PAC) :: Package Violation (Error) - Missing Use Access (USEM)

aunit run transport does not resolve LIMU objects

Hi Jakub,

I am trying things out again and noticed the ABAP Unit run of transports does not work as I expect when there is only part of a class locked in a transport. Which should be the normal case after it was first created, even with ADT.

For example I have a transport with multiple LIMU METH and LIMU CINC objects but not the R3TR CLAS one. sapcli -v aunit run transport "..." --as4user "..." --output junit4 does not list the class and does not execute the tests. I suggest the partial objects to be resolved to its parent in this context. If I add the R3TR CLAS object to the tranport task manually the tests are executed.

sapcli/sap/cli/aunit.py

Lines 199 to 220 in 538b18d

	types = {'PROG': sap.adt.Program, 'CLAS': sap.adt.Class, 'FUGR': sap.adt.FunctionGroup}
	mod_log().info('Fetching the transport or task %s', self._number)
	workbench = sap.adt.cts.Workbench(self._connection)
	transport = workbench.fetch_transport_request(self._number, user=user)
	if transport is None:
	raise SAPCliError(f'The transport was not found: {self._number}')
	result = []
	for task in transport.tasks:
	for abap_object in task.objects:
	mod_log().debug('? %s %s', abap_object.type, abap_object.name)
	try:
	# TODO: get rid of the need to create the instances!
	result.append(types[abap_object.type](self._connection, abap_object.name))
	mod_log().info('+ %s %s', abap_object.type, abap_object.name)
	except KeyError:
	pass
	return result

ATC checks and aunit does not work on the include program.

While executing SAPCLI command for ATC and aunit from Azure DevOps checks works fine when executed on the main program. But it doesnt work when used on the include program. Whereas, in S/4HANA system ATC checks and Unit tests are done when executed on include programs.

Commands used in Azure DevOps are -
/sapcli atc run program programname -e 1 -m 100 -o human
./sapcli aunit run program programname --output human --result all --coverage-output human

add execute action for classes

To make use of IF_OO_ADT_CLASSRUN

SyntaxError in sapcli line 14

I just checked out from git and tried to run sapli but go the following error:

/mnt/c/SAPDevelop/sapcli$ ./sapcli --help
  File "./sapcli", line 14
    os.environ['PYTHONPATH'] = f'{base_dir}:{pythonpath}'
                                                        ^
SyntaxError: invalid syntax

I am using Debian on Linux subsystem for Windows.
Python 3.5.3 (and 2.7.13) are installed.
Pointing $PYTHONPATH to /usr/bin/python3 does not help.
Neither does removing Python 2.7.13.
Any ideas?

Proper encoding of skipped AUNIT tests into junit results xml

The current implementation encodes skipped tests as:

<testcase classname="net.cars.engine.ValveTest" name="moveDown" status="SKIP">
  <system-err>Test execution skipped due to missing prerequisites in car engine</system-err>
  <error type="abortion" message="Missing Prerequisites - Skipped due to potential obsolete</error>
</testcase>

It would be great to align with aunit schema/standard:

<testcase classname="net.cars.engine.ValveTest" name="moveDown">
    <skipped />
</testcase>

or with message:

<testcase classname="net.cars.engine.ValveTest" name="moveDown">
    <skipped message="Test execution skipped due to missing prerequisites in car engine"/>
</testcase>

More details regarding junit format:
https://llg.cubic.org/docs/junit/

The relevant piece of the sapcli code seems to be here:
https://github.com/jfilak/sapcli/blob/master/sap/cli/aunit.py#L275

migrate to public ADT ATC API

https://help.sap.com/docs/BTP/65de2977205c403bbc107264b8eccf4b/d8cec788fc104ff9ad9c3757b4dd13d4.html?locale=en-US

Add the possibility to activate abap include with master program

determine master program of includes somehow

To get rid of the following problem:

sap.adt.errors.ADTError: ExceptionActivationServiceFailure: REPS ZABAPGIT_USER_EXIT is used in multiple master programs

add gCTS polling on timeouts

GET /repository//getHistory
{ result: [ { rid: , checkoutTime: , fromCommit: , toCommit: , caller: , request: , type: (PULL | CLONE), state: DONE, rc: } ] }

No testable objects found should not result in an error return code

sapcli/sap/cli/aunit.py

Lines 248 to 252 in 2bcd23d

	testable = obj.get_testable_objects(args.as4user)
	if not testable:
	sap.cli.core.printerr('No testable objects found')
	return 1

Hi,

I often have transports that do not have any objects with unit tests in them. I would like my build on the merge request to not fail if that is the case. Because sapcli returns 1 I cannot distinguish it from 1 error message caused by a unit test. I would have to parse stdout for No testable objects found.

In my opinion a return code of 0 would make sense but I would also be happy with a fixed one that isn't used for normal error reporting (-1).

package create - transport request

There is no transport request parameter for 'package create' command so new request is created for every new package created. ADT api allows to specify transport request, it is passed in request url like POST /sap/bc/adt/packages?corrNr=<request no>

RFE: add support for exporting / importing an ABAP package

Start with Classes.

https://abapgit.slack.com/archives/C4K5GM9D0/p1551968395040800

Unit test runs - Include stack trace for failures

aunit code coverage

store it in an easily comparable format

user action newreleasejobs is not supported

When I try to release a task I get the following error. We're on 7.50 sp 11.

mattias@PC2749:~/sapcli$ ./sapcli cts release task TCGK900054
Traceback (most recent call last):
File "./sapcli", line 23, in
sys.exit(sapcli.main(sys.argv))
File "/home/mattias/sapcli/bin/sapcli", line 157, in main
return args.execute(connection, args)
File "/home/mattias/sapcli/sap/cli/cts.py", line 38, in release
request.release()
File "/home/mattias/sapcli/sap/adt/cts.py", line 99, in release
headers={'Accept': 'application/vnd.sap.adt.transportorganizer.v1+xml'}
File "/home/mattias/sapcli/sap/adt/core.py", line 232, in execute
resp = self._execute_with_session(session, method, url, params=params, headers=headers, body=body)
File "/home/mattias/sapcli/sap/adt/core.py", line 168, in _execute_with_session
Connection._handle_http_error(req, res)
File "/home/mattias/sapcli/sap/adt/core.py", line 141, in _handle_http_error
raise error`
sap.adt.errors.ADTError: ADT_TM_COMMON_EXCEPTION: user action newreleasejobs is not supported

checkin does not remove testclasses (includes)

sapcli checkin does not remove testclasses and most probably all other includes of a class which are in the target system but in the local filesystem directory.

Steps to reproduce:

1. create a class in ADT and add tests
2. checkout the correpsonding package
3. remove the test classes on filesystem
4. run checkin

Actual results:
Testclasses are untouched

Expected results:
Testclasses are removed in the target system too

Additional information:
Lets try to remove the checkedin classes before and create it again.

parse server errors reported in HTML

Show:

403 Forbidden
The request has been blocked by UCON.

Instead of:

<!DOCTYPE html>
<html><head>
<title>Application Server Error</title>
<style>
body { background: #ffffff; text-align: center; width:100%; height:100%; overflow:hidden; }
.content { display: table; position:absolute; width:100%; height:80%; }
.valigned { display: table-cell; vertical-align: middle; }
.lowerCenter { display: table-cell; vertical-align: bottom; }
.footer { position: absolute; bottom: 0; left: 0; width: 100%; z-index: -1; }
.footerLeft { float: left; margin-left: 20px; }
.footerRight { float: right; margin-right: 20px; position: absolute; bottom: 0px; right: 0px; }
.centerText { font-style: normal; font-family: Arial; font-size: 16px; color: #444444; z-index: 1; }
.errorTextHeader { font-style: normal; font-family: Arial; font-size: 40px; color: #444444; margin-top:0px; margin-bottom:12px; }
.detailText { font-style: normal; font-family: Arial; font-size: 16px; color: #444444; margin-top:0px; margin-bottom:0px; }
.bottomText { align: center; font-style: normal; font-family: Arial; font-size: 14px; color: #444444; }
.detailTable { align: bottom; vertical-align: middle; margin-left:auto; margin-right:auto; font-style: normal; font-family: Arial; font-size: 16px; color: #444444; }
</style></head>
<body>
<div class="content">
<div class="valigned">
<div class="centerText">
<p class="errorTextHeader"> <span >403 Forbidden</span> </p>

<p class="detailText"> <span id="msgText">The request has been blocked by UCON.</span></p>
<p class="detailText"> <span id="msgText">Server time:
<script>
var d = "20220725";
var t = "073947";
document.write(d.slice(0,4)+"-"+d.slice(4,6)+"-"+d.slice(6,8)+" "+t.slice(0,2)+":"+t.slice(2,4)+":"+t.slice(4,6)); </script>
</script>
</span> </p>
</div>
<table class="detailTable" border="0">

It means: get errorTextHeader and detailText elements but leave nout detailText node with "Server time:".

The parser must be available in:

• https://github.com/jfilak/sapcli/blob/master/sap/rest

and must be used also in:

• https://github.com/jfilak/sapcli/blob/master/sap/adt

Now I know ADT is a part of ABAP REST but at the time I started sapcli, I had no idea. Unfortunately I have no capacity to merge the module sap.adt with sap.rest - but we can slowly migrate sap.adt to sap.rest

encode bytes to base64 in startrfc

Update FORMATTERS at: https://github.com/jfilak/sapcli/blob/master/sap/cli/startrfc.py#L14

class BytesBase64Encoder(json.JSONEncoder):

    def default(self, obj):
        if isinstance(obj, bytes):
            return base64.b64encode(obj).decode('ascii')

        return json.JSONEncoder.default(self, obj)

def json_format(obj):
    return json.dumps(obj, cls=BytesBase64Encoder)

FORMATTERS = {
    'human': pprint.PrettyPrinter(indent=2).pformat,
    'json': json_format
}

gCTS configuration does not always contain "value"

Provide aunit results in sonar generic execution format

To integrate the test results into Sonar, it would be great if you could provide an additional output format for sonar:

https://docs.sonarqube.org/latest/analysis/generic-test/

I had a go at it, but found the main roadblock is that this requires the full file path for each test suite. I've hacked this in using an os.walk for each test suite, assuming there are only OO testclasses. But this is obviously not a nice solution...

ModuleNotFoundError: No module named 'sap'

When trying to run per the README-instructions I get the following error. Do I need to "build" it somehow ( I don't know python ), or how is it supposed to find the import?

Mattias@Mattiass-MacBook-Pro ~/D/P/sapcli> bin/sapcli --help
Traceback (most recent call last):
  File "bin/sapcli", line 12, in <module>
    import sap
ModuleNotFoundError: No module named 'sap'

parse activation results and support for force activation

sap.errors.SAPCliError: Could not activate the object zabapgit:
<?xml version="1.0" encoding="utf-8"?>
<chkl:messages xmlns:chkl="http://www.sap.com/abapxml/checklist">
  <msg objDescr="Program ZABAPGIT" type="W" line="1" href="/sap/bc/adt/programs/programs/zabapgit/source/main#start=22790,10" forceSupported="true">
    <shortText>
      <txt>Bitte anstelle von "CLIENT SPECIFIED" den Zusatz "USING ALL CLIENTS" verwenden.</txt>
    </shortText>
  </msg>
</chkl:messages>