reportportal/reportportal

Main Repository. ReportPortal starts here - see readme below.

ReportPortal.io

Report Portal organized into multiple repositories.

Repositories structure

ReportPortal server side consists of the following services:

• service-authorization Authorization Service. In charge of access tokens distribution
• service-api API Service. Application Backend
• service-ui UI Service. Application Frontend
• service-index Index Service. Info and health checks per service.
• service-analyzer Analyzer Service. Finds most relevant test fail problem.
• gateway Traefik Gateway Service. Main entry point to application. Port used by gateway should be opened and accessible from outside network.
• rabbitmq Load balancer for client requests. Bus for messages between servers.
• minio Attachments storage.

Available plugins developed by ReportPortal team:

• plugin-bts-jira JIRA Plugin. Interaction with JIRA. Link to download
• plugin-bts-rally Rally Plugin. Interaction with Rally. Link to download
• plugin-saucelabs Sauce Labs Plugin. Interaction with Sauce Labs. Link to download

Client side adapters related repositories:

• client-* - API integrations. Http clients, which process HTTP request sending.
• agent-* - Frameworks integration. Custom reporters/listeners, which monitor test events and trigger event sending via client-*
• logger-* - Logging integration. Logger appenders, which help to collect logs, bind it with test-case item via agent-* and send to server via client-*

Other repositories stored according to next rules

• service-* - micro-services which are a part of Application
• commons-* - common libraries, models, etc., used by micro-services

Installation steps

Simple setup with Docker

It's the best way for demo purposes and small teams. The database is already in the docker-compose.

1. Install Docker (Engine, Compose)

2. Download Example of docker-compose descriptor to any folder

3. Deploy ReportPortal using docker compose plugin within the same folder

$ docker compose -p reportportal up

To start ReportPortal in daemon mode, add '-d' argument:

$ docker compose -p reportportal up -d

4. Open in your browser IP address of deployed environment at port 8080

http://IP_ADDRESS:8080

5. Use next login\pass for access:

• default\1q2w3e
• superadmin\erebus.

Please change admin password for security.

Mentioned compose file deploy all available Bug Tracking System integrations, which not always needed, but use resources.

Production-ready set and Custom deployment with Docker

For production usage, we recommend:

• choose only required Bug Tracking System integration service. Exclude the rest

To customize deployment and make it production-ready please follow customization steps and details

Integration. How to get log data in

You should add Client Side code inside your test automation. It consists of:

• client-* - API integrations. Http clients, which process HTTP request sending. E.g. for Java (client-java-*)
• agent-* - Frameworks integration. Custom reporters/listeners, which monitor test events and trigger event sending via client-*
• logger-* - Logging integration. Logger appenders, which helps to collect logs, bind it with test-case via agent-* and send to server via client-*

Integration steps and documentation

Contribution

There are many different ways to contribute to Report Portal's development, just find the one that best fits with your skills. Examples of contributions we would love to receive include:

• Code patches
• Documentation improvements
• Translations
• Bug reports
• Patch reviews
• UI enhancements

Big features are also welcome but if you want to see your contributions included in Report Portal codebase we strongly recommend you start by initiating a chat through our Team in Slack.

Contribution details

Documentation

• User Manual
• Wiki and Guides

Community / Support

• Slack chat
• Security Advisories
• GitHub Issues
• Stackoverflow Questions
• Twitter
• YouTube Channel

License

Report Portal is Apache 2.0.

reportportal's Issues

Unexpected logouts sometime occurs - increase session life (TTL) for user

https://your.site.com/api/v1/mg2/launch?page.page=1&page.size=50&page.sort=start_time%2CDESC&tab.id=allCases
Status code 401

{error: "invalid_token", error_description: "1f7w7ldc-9cc1-49aa-8557-asdd1h9ac24f"}

Please leave a comment and clarify what exact logs should i attach in case you need them.

Another unexpected logout:
scripts.d34776b3.js:1 Uncaught TypeError: Cannot read property 'clear' of null at i.setupRequestParamsForLevel (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2067429) at i.updateNavigationState (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2066846) at i.update (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2041047) at u (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2139663) at Object.openRouted (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2140049) at i.<anonymous> (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2251137) at Object.<anonymous> (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2248286) at l (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:33984) at Object.add [as done] (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:34298) at i.openLaunches (https://your.conpany.com/ui/compiled/js/scripts.d34776b3.js:1:2248268)

Widget legend - pins color

After adding new defect type I got inappropriate color of pins on my widgets.

Bad attachments breaks all steps of batch

RP have default value for batch: rp.batch.size.logs == 10, When we have wrong logging event (try send big file) - all steps from this batch is lost in RP.
perhaps it will be fixed by: #16

    @Test
    public void bigFileTest() throws Exception {
        //rp.batch.size.logs = 10 (default value for batch)
        URL resource = Resources.getResource("big.file");
        String path = Paths.get(resource.toURI()).toAbsolutePath().toString();
        logger.info("1# event before sending big file");
        logger.info("2# event before sending big file");
        logger.info("3# event before sending big file");
        logger.info("4# event before sending big file");
        logger.info("5# event before sending big file");
        logger.info("RP_MESSAGE#FILE#{}#{}", path, "7# event (very big file)");
        logger.info("8# event after sending big file");
        logger.info("9# event after sending big file");
        logger.info("10# event after sending big file"); //will triggered batch call to RP
    }

Email from [email protected] - make configurable default sender address

Where could I change the default mail from domain?

Divide merged launches

Add the functionality to divide merged tests executions
by anatolii_klymenko

Add feature to stop 'Auto-analisys'

Unable to search by launch number

Some times we need switch to specific execution/launch but currently no any ability to perform search by execution number

[agent-ruby + cucumber] if tags have gap - error returned

rp_tags=["Assemble issues build"] - ERROR
rp_tags=["Assemble_issues_build"] - ok
rp_tags=["issue***", "Assemble_issues_build"] - ERROR
rp_tags=["issue***","Assemble_issues_build"] - ok

shell command

cucumber --tags @build_test HOMEPAGE_URL=http://??? DATABASE=???
rp_uuid=14808c29-63fb-4eb8-90be-a4b6b83d6d34 rp_endpoint=http://???/api/v1 rp_tags=["Assemble issues build"] rp_project=urspectr rp_launch=issues/??? -f ReportPortal::Cucumber::Formatter

error log

Starting virtual X frame buffer: Xvfb.
(): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 1 (Psych::SyntaxError)
/usr/local/lib/ruby/2.2.0/psych.rb:370:in 'parse'
/usr/local/lib/ruby/2.2.0/psych.rb:370:in 'parse_stream'
/usr/local/lib/ruby/2.2.0/psych.rb:318:in 'parse'
/usr/local/lib/ruby/2.2.0/psych.rb:245:in 'load'
/usr/local/bundle/bundler/gems/agent-ruby-4416bda50ea5/lib/reportportal/settings.rb:72:in 'setting'
/usr/local/bundle/bundler/gems/agent-ruby-4416bda50ea5/lib/reportportal/settings.rb:51:in 'block (2 levels) in initialize'
/usr/local/bundle/bundler/gems/agent-ruby-4416bda50ea5/lib/reportportal.rb:66:in 'start_launch'
/usr/local/bundle/bundler/gems/agent-ruby-4416bda50ea5/lib/reportportal/cucumber/report.rb:57:in 'start_launch'
/usr/local/bundle/bundler/gems/agent-ruby-4416bda50ea5/lib/reportportal/cucumber/report.rb:42:in 'initialize'
/usr/local/bundle/bundler/gems/agent-ruby-4416bda50ea5/lib/report_portal/cucumber/formatter.rb:32:in 'new'
/usr/local/bundle/bundler/gems/agent-ruby-4416bda50ea5/lib/report_portal/cucumber/formatter.rb:32:in 'block in initialize'
Stopping virtual X frame buffer: Xvfb.

Add custom defect types to "To Investigate" group

Needed for mark not analysed failures as currently investigating by team member to avoid duplicating of investigation or extra communication in team.

Default Admin credentials

When I tried to log in, admin\erebus didn't work.
Error: user name or password is incorrect

Copyright statements at the start of source code files have not working links

See e.g. https://github.com/reportportal/commons-dao/blob/master/src/main/java/com/epam/ta/reportportal/database/ActivityDocumentHandler.java

The link in a copyright statement is "https://github.com/epam/ReportPortal".
This link seems to be invalid.

Failed to update password : Incorrect Request

Front End:
Failed to load resource
/api/v1/user/password/reset 400

Fetching scripts with an invalid type/language attributes is deprecated and will be removed in M56, around January 2017. See https://www.chromestatus.com/features/5760718284521472 for more details. scripts.d34776b3.js:1 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/. EQCSS.load @ scripts.d34776b3.js:1 https://your.company.com/api/v1/user Failed to load resource: the server responded with a status of 401 () https://your.company.com/epam/externalServices.js Failed to load resource: the server responded with a status of 404 () https://your.company.com/api/v1/user/password/reset Failed to load resource: the server responded with a status of 400 () scripts.d34776b3.js:1 POST https://your.company.com/api/v1/user/password/reset 400 () send @ scripts.d34776b3.js:1 ajax @ scripts.d34776b3.js:1 o @ scripts.d34776b3.js:1 ct @ scripts.d34776b3.js:1 submitChangePass @ scripts.d34776b3.js:1 n @ scripts.d34776b3.js:1 dispatch @ scripts.d34776b3.js:1 y.handle @ scripts.d34776b3.js:1

Backend:
2016-12-12 18:14:02.263 ERROR 1 --- [-8080-exec-2900] .t.r.c.e.r.ReportPortalExceptionResolver : ReportPortalExceptionResolver > Validation failed for argument at index 0 in method: public com.epam.ta.reportportal.ws.model.OperationCompletionRS com.epam.ta.reportportal.ws.controller.impl.UserController.resetPassword(com.epam.ta.reportportal.ws.model.user.ResetPasswordRQ), with 1 error(s): [Field error in object 'resetPasswordRQ' on field 'uuid': rejected value [null]; codes [NotNull.resetPasswordRQ.uuid,NotNull.uuid,NotNull.java.lang.String,NotNull]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [resetPasswordRQ.uuid,uuid]; arguments []; default message [uuid]]; default message [may not be null]]

Add feature to update test results in scope of one launch [re-run functionality]

There is a need to have a chance to update test's result during rerun or other functionality.

Does ReportPortal send something to your servers?

Not sure if you noticed my additional question in closed issue so I repost it as separat issue here.

@DzmitryHumianiuk
I have taken a closer look at documentation and also watched your presantations on youtube. Quite on question remains: When I set it up - will it run fully (100%) on my system (my infrastructure) or will it send results to your servers for analysys?

I need to find Test Case/Suite by name/tag

How can I find test case/suite by name/tag?
Currently even filter by tag works for top view only so I don't have way to find specific test except for via traversing test launch/test suites hierarchy down to test

[Auto-analysis] Special IGNORE flag for Analysis to skip item next time

Add ability to mark reviewed test item (defect object) as NOT FOR ANALYSIS, in order to pay attention to this item once again with next execution

[Swagger] Add more complete documentation for REST API

Current documentation (/ui/swagger-ui.html) lacks quiet a lot of information, e.g.:

• a way to send a log with an attached file (clients use multipart)
• allowed logging levels
• allowed issue types
• GET /{projectName}/item?{queryParams} and query parameters usable by it

It would be better to have more complete documentation for API.

Expand button on Launches page doesn't work

Cannot make use of this button, looks like it does nothing.

[Feature request] a way to share dashboard publicly AKA Provide "guest" user type

Hi,

Please provide "guest" account which can view data without logging in (like in Jenkins, Upsource, etc.).
For example such account can view only shared filters and dashboards.

Log messages not displayed if unable attach file to step

I try send big file to report portal, it my test (testNg):

@Test
    public void bigFileTestWithMinimumBatchSize() throws Exception {
        //rp.batch.size.logs = 1 (set via reportportal.properties)
        URL resource = Resources.getResource("big.file");
        String path = Paths.get(resource.toURI()).toAbsolutePath().toString();
        logger.info("1# event before sending big file");
        logger.info("RP_MESSAGE#FILE#{}#{}", path, "2# event (very big file)");
        logger.info("3# event after sending big file");
    }

Current results:
"2# event (very big file)" event not logged in RP
I think we should log it and root cause (exception)

[Widgets] incorrect time calculation in "Project activity"

[MAILS] Make invitation emails more friendly

The invitation email looks suspicious and 3 members from my team (they are in different locations so they even didn't discuss it) just ignored it and considered like a spam!

Customers can do the same.
The information about the project and sender should help in this case,
e.g. "John Doe invites you to join the XXX project on the Report Portal".

Print logs using monospace font, reduce interval between lines, remove checkerboard for lines

Compare RP logs:

with Robot Framework logs:

And less:

Implenent any way to display error message in test items lists

It is a good idea to see test error message in list of executed test items. Now the only way is put error message to description. Possible implementations: new log item type (displayed message) or possibility to show the first log item in list.

Error 500 on login of EPAM user with accepted invitation sent to work email

I sent an invitaion from RP to some EPAM user to his corporate email.
The user tries to login (the both EPAM and email/password logins shows the same) and see the following error message:

Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.
Thu Dec 01 08:11:30 GMT 2016
There was an unexpected error (type=Internal Server Error, status=500).
Incorrect authentication type: User with your domain email already exists. You cannot create duplicate

Add feature to merge the launches without deleting them

Actual: When we merge two launches they disappear and we can't gather statistic by each of them. We can gather only from merged one.

Expected: After merging launches they stay alive in the 'Lunches' tab.

Why it is important: It is quite convenient to build test suites hierarchy based on Teams/Components
in that turn it is valuable to gather statistic by Teams or Components.

Add "Tests History" tab - aka Tree View

Currently, RP provide view launches/suites etc. for testNg
{EXECUTION NAME}/{SUITE NAME}/{TEST NAME}/{METHOD}
In our projects, we don't depend on suites. Because we consider that the parameters Execution Name, Suite Name, Test Name is not constant. Same class or method can successfully be run in different sets. We just have structure in java packages. All tests from package(or class) cover one feature/functional area.

Main use case that we need to cover: In which execution package/class/method will start failures
Similar feature available on Test Results Analyzer Plugin

RP not provide view under packages/classes/method. Hence other related problems:

1. Unable to create filter with following criterias (package/class/method)
2. For getting status or history for a particular method. We must find out execution name, suite name, test name.

My suggestions:

1. Adding new "Tests History" tab (like in screenshot)
2. Adding new execution model {EXECUTION NAME}/{PACKAGE}/{CLASS}/{METHOD} --> and set it by default for all supported frameworks
3. Add ability to create filters by package/class/method

I need filter to find out all triaged (investigated) test results

Filter should include the same fields as for launches: time, tags, etc.

As user I want to track performance results in RP

I want to save number for test result and specify measurement units (MB/s, IOPS/s, etc.)
I want to be able to get X-Y chart with saved data for particular test.
Optional: I want to setup baseline/threshold so test fails if performance is lower than required.

Refresh button disabled in merge/compare mode

When I am in merge/compare mode at launches page, Refresh button has disbled status:

But, when i'm clicked to refresh, it's work properly.

I propose to leave the Refresh button is enabled, regardless of the current mode (merge/compare).

Remove limit for cont of 20 widgets per dashboard

It's unable to add more then 20 widgets on 1 dashboard.

Clients might like to have more than this limit

Add features to RP from "Cuanto" project

For analyze our tests we use a project: https://github.com/ttop/cuanto/wiki
This project is very outated, but it have advantages, and that it would be good to add in RP:

1. "Grouped Output" tab:
   
   Enhancement connected to: Group tests failed by one reason
   Ability stack-trace grouping in each launch with assigning this failures group to user - provide planning to fix/analyze them

2. "Summary" tab:
   enh. connect to #25 - on my screenshot we see high level metrics for features(packages).
   
   This tab allows to quickly determine the launch status after investigations. It is very useful for managers. Please do not confuse with PR reports.

Allow Admin to Edit, unShare(maybe), Delete shared filters

Let Admin to make changes for shared filters

[WIDGETS] Add widget to represent flaky tests in launches

Add widget, the same as for Most failed test cases, but for flaky test.
Flaky means which changes state from passed to failed each execution.

I need aggregated statistics for launch groups

I have several Jenkins job groups that report results into separate launches. Currently I don't see way to have aggregated statistics
I have 3 types of jobs:

• Several nightly CentOS6 jobs
• Several nightly CentOS7 jobs
• Several weekly long-running/performance jobs scheduled for different week days

I need accumulated statistics for job groups above.
So I need to:

• Have timeline mode for all Widgets (currently only few have such mode)
• Ability to adjust time window and window start for timeline mode (for weekly jobs)
• Ability to group launches into expandable hierarchical view on Launches page basing on following criteria:
  • Time: group launches for specific time period. I want to set period length and when period starts to group my daily and weekly jobs (time is not aligned with Report Portal timezone)
  • Tag: group launches by specific tags
  • Name (with regexp support): group launches by name pattern

Filtering logs with attachments

In my case i have two messages:

1. INFO log with attachment
2. Assertion error
   Also filters not selected

When select filters ERROR + ATTACHMENT - i see "No results found"message. But it is not correct.

1. INFO log with attachment - should be displayed
2. Assertion error - should be displayed

Smart Analysis passes by 'To Investigate' items once modified by user

The case is when somebody adds comment to some test item that is initially in "To Investigate" status, but doesn't change resolution to some specific one ("Product Bug", "System Issue", etc.). One more case is when somebody changes resolution from some particular issue type down to "To Investigate".
In that cases after next launch execution, when Smart Analysis processes launch result, it skips updating issue type for such items, but it does extend the comment using previous resolutions history.
"To Investigate" resolution ALWAYS stands for the fact that decision hasn't been made yet.
If Smart Analysis allocates some resolution for History it should ALWAYS be allowed to use it for "To Investigate" items. For deferred decisions users can use custom issue types, etc.

Add feature to hide 'Match issues in Launch' link for specified user roles

'Match issues in Launch' link is pretty close to 'resolution button'.

Hence we have some cases when someone clicks on the link by mistake and it blocks the whole regression analysis. Since we can't modify the test resolution while auto analysis in progress.

One of possible solutions is to disable this link for certain roles.

Windows support - install ReportPortal with Docker on Windows?

Is this one already tested on Windows?

I am aware of theory that with Docker it should be platform agnostic but did anybody really see it run on a Windows host? ;-)

Wrong tests count

1. Open suite, on suite lvl my tests counter == 1999 (it is correct)
2. Package lvl is opened, on this lvl my tests counter == 1499. (it is wrong)

Test code (testNg):

@Listeners({ReportPortalTestNGListener.class})
public class Main {

    private Logger logger = LoggerFactory.getLogger(this.getClass());

    @DataProvider(name = "loop", parallel = true)
    public static Iterator<Object[]> getDataWithStream() {
        return IntStream.range(1, 2000).mapToObj(i -> new Object[]{UUID.randomUUID().toString()}).iterator();
    }

    @Test(dataProvider = "loop")
    public void workingWithDeque(String testUid) {
        String msg = "Error: " + testUid;
        logger.error(msg);
        throw new AssertionError(msg);
    }
}

Default e-amil host is set with *. epam.com

Hi,

Default e-mail host is set with *.epam.com. It seems that it should be empty.

Provide ability to reload .properties on runtime

I would like to update listener with new properties values before tests are executed.

Basically i need to create reportportal.properties according to starting suite and then update ReportPortalTestNGListener (in my case) with new information.
Currently this is unable because listener will initialize itself on TestNG framework start.

I know that i can pass environment variables thru command line, but it's not the case for me.
My idea is to keep all configuration inside my framework and require only necessary command line parameters. For example -Dclient=SomeClientID is a keyword to setup all the required data for suite.

Serialization exceptions not handled.

We use corporate proxy. And our proxy not allow send any request with RP. I will check it in our network side. But any way report portal not handle this issue:
For understanding:
When suite start RP call reportPortalService.startRootTestItem(rq);
Our proxy return internal html page with status: 503.

And RP catched it:

But method handleException(...) is silent. And i not see any problems in my console.

Because com.google.common.base.Throwables#throwIfUnchecked - not handle "Serialize" exceptions type by default.

Cannot comment on result using monospace, code tag doesn't work also

Before submit:

After submit:

Expected result:
Newlines are preserved, I can also set monospace font for text

Populate TestNG group to PR launch tags

TestNg provide ability to set execution groups, will be nice if PR will add group(s) to launch tags + tags from reportportal.properties file.

By @ADMiNZ

Add support async/non-blocking sending data to RP instead of batch

Root cause of this. sending events with batch - provide increasing tests execution time.
Currently data sending performed after collecting 10 events (rp.batch.size.logs). When sending data to RP the tests are blocked. This is appreciable at poor internet speed. My proposal to use non-blocking client.

1. Using event queue/msg bus + async http client
2. Add more logs when events send started/finished
3. Add warning if data transfering takes long time (configurable via properties)

Robot Framework integration

You might want to add the greate and mighty test automation framework named Robot Framework[1] to your documentation[2]. Checkout this repo: https://github.com/reportportal/agent-Python-RobotFramework

[1] http://robotframework.org/
[2] http://reportportal.io/#documentation/Test-framework-integration

Email client is not compatible with Amazon SES

I use Amazon SES service. I configured Report Portal instance to use SMTP server, but RP cannot send emails.

Logs of API service:

javax.mail.MessagingException: Could not connect to SMTP host: email-smtp.us-west-2.amazonaws.com, port: 465, response: -1
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2042) ~[javax.mail-1.5.4.jar!/:1.5.4]
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:697) ~[javax.mail-1.5.4.jar!/:1.5.4]

It seems current implementation of email client doesn't support SMTPS.

Below is SMTP server details:
Server Name: email-smtp.us-west-2.amazonaws.com
Port: 25, 465 or 587
Use Transport Layer Security (TLS): Yes

LDAP authorization

Good day!

Could you add feature as a user authorization by AD domain by using ldap?

[Log View] make configurable log representation

As a user I want to have ability to change log representation:

• font size
• margins between lines
• coloring schemes
• ....

make configuration per User. And per project.

Extra details
#35 (comment)