wso2/ballerina-integrator

A powerful, simple-to-learn, code-driven approach to programming integrations

Ballerina Integrator

A powerful, simple-to-learn, code-driven approach to programming integrations - based on the Ballerina programming language. It includes first-class cloud-native functions as well as a graphical sequence diagram tool.

The Ballerina Integrator appeals to those using decentralized integration architectures, microservices, and cloud-native apps.

Developers can now efficiently write code for their integration scenarios, merging the worlds of development and integration.

Ballerina

Ballerina is a new general-purpose programming language that makes it easy to write cloud-native applications.

Please visit ballerina.io to read more about Ballerina.

Key Highlights

Develop fast and resource-efficient integrations using code-based technology

Ballerina incorporates fundamental concepts of distributed system integration into the language and offers a type-safe, concurrent environment to implement microservices with distributed transactions, reliable messaging, etc.

Based on the interactions of sequence diagrams, Ballerina has built-in support for common integration patterns and connectors, including distributed transactions, compensation, and circuit breakers.

With first-class support for JSON and XML, Ballerina makes it simple and effective to build robust integration across network endpoints.

Cloud-native, microservices ready

The container-friendly super lightweight Ballerina Integrator runtime seamlessly works with Docker, Kubernetes, Prometheus, and other cloud technologies.

Enables a true agile approach to API-centric integration

With the use of Ballerina language, the Ballerina Integrator marks a revolutionary new phase of API-centric integration. Using this programming language, integration developers can apply agile methodologies, bringing a new degree of speed and agility to integration projects.

Easy to learn; complete with out-of-the-box templates

Easy adoption by any developer familiar with C-based languages. Tooling includes most popular integration templates.

Composition

Mainly consists of below-stated components.

• Ballerina language runtime
• Connectors that enable integrations with popular enterprise systems
• Protocol level connectors to integrate with well known standard systems
• Development tools (Visual Studio Code plugin)
• Integration templates to quickly implement an integration

Key features

• Protocol Connectors: HTTP, File, gRPC, AMQP, NATS, Kafka, JMS
• Connectors: Salesforce, Amazon S3, Amazon SQS, SAP
• Integration Templates: projects with all key integration scenarios using pre-built templates (EIPs, Samples, scenarios..).
• Sequence diagram visualization of your integration scenario.
• Connected design, development, test and deployment experience.

Getting started

You can use the following options to try out Ballerina Integrator and get started.

• Download Ballerina Integrator
• Quick Start Guide
• Learn Documentation
• VS Code Extension

Contributing to Ballerina

As an open source project, the Ballerina Integrator team welcomes contributions from the community. To start contributing, read these contribution guidelines for information on how you should go about contributing to our project.

Reference to the list of Ballerina Integrator connectors can be found here.

Check the issue tracker for open issues that interest you. We look forward to receiving your contributions.

License

Ballerina Integrator code is distributed under Apache License 2.0.

ballerina-integrator's Issues

Add sample for one to many asynchronous invocation

Description:

This sample should demonstrate adding a message to a topic and 2 services listening to the topic individually process the message.

Implement dataservices based examples

Description:

This epic covers the Ballerina integration example use cases exposing databases as services.

Improve and verify functionality of SAP Connector

Description:

Need to verify the functionality of the SAP connector and improve the functionality as needed.

Implement a guide for IBM MQ

Description:

A detailed guide of using IBM MQ as a brokering backend. This should cover details on

• what are the configurations needs to be done
• receiving and sending messages
• transaction support. If it is same as JMS point to relevant doc
• relevant connectors to be used.
• data transformations if required.

Implement gmail connector example

Description:

Write an example with Gmail connector in Ballerina based on doc:
https://docs.wso2.com/display/EI650/Using+the+Gmail+Connector

Add a util class to clean the data in the Backend

Description:
Backend: #43

Implement store forward example

Description:

Write a store forward example in Ballerina based on doc
https://docs.wso2.com/display/EI650/Storing+and+Forwarding+Messages

Implement and Verify Enterprise Messaging Guides

Description:

This epic covers all the enterprise messaging related sub epics and issues. This effort should cover creating and verifying existing guides on enterprise messaging space.

Topics covered

• Sending and receiving messages.
• Guaranteed delivery
• Store forward pattern
• Transactions
• Different protocols (JMS, AMQP, Kafka, NATS etc)

Create a Guide for Amazon SQS Connector

Description:

Guide for using Amazon SQS Connector in integration use cases

depends on #69

Verify Active MQ Connector

• Verify Active MQ connector for EI use cases.

Implement Amazon S3 Connector

Description:

Design and implement an Amazon S3 connector.

Implement business workflow example

Description:

Write a business workflow example with Ballerina based on doc
https://docs.wso2.com/display/EI650/Managing+a+Business+Process

Provide XML validation with XSD

Description:

Provide support for XML schema validation using XSD.

Implement and verify File Integration use cases

• File protocols (FTP, SFTP, local)
• File consumers (consuming based on a delimiter, streaming files)
• File producer/writers

Generate a SOAP request on a given WSDL

Description:
SOAP Ballerina client should be able to automatically generate a SOAP request and receive a SOAP response properly on a given WSDL.

Implement VS code plugin support to create integration projects using templates

Integration developer should be able to create a new Ballerina integration project using VS code plugin.
This should create the necessary directory structure and files.

User can opt to create the project using an existing integration template as well.

Implement Introductory Integration Tutorials example

Description:

Integration Tutorials example is an introductory example showcasing the widely used synapse integration use cases. We need to implement the same with Ballerina.

Synapse doc link: https://docs.wso2.com/display/EI650/Integration+Tutorials

Verify web services/SOAP use case

• Exposing SOAP services
• Invoking SOAP services /SOAP Connector

Create static web pages using the examples in MD format

Description:

Need to create a guide like web pages using the markdown files in the repo.

SOAP support in Enterprise Integrator Ballerina Profile

Description:
Existing legacy SOAP services should be supported in Ballerina profile in EI. This epic would implement the major components of the SOAP support.

Implement Sending Simple Message to a Service example

Description:

Write the sample in Ballerina
doc: https://docs.wso2.com/display/EI650/Sending+a+Simple+Message+to+a+Service

Verify Kafka Connector for EI use cases

• Verify common use cases of Kafka and EI.
• Verify scalability and load testing.

Implement integrating with database example

Description:

Write the sample in Ballerina

doc: https://docs.wso2.com/display/EI611/Sending+a+Simple+Message+to+a+Datasource

Add guide for the data driven testing-hello service

Description:
Write a simple hello service and write tests based on data driven approach.

Verify Active MQ Artemis

• Verify all use cases that we have done for EI Artemis integration
• Load and stability testing.

Implement ActiveMQ Connector examples

Description:

Need to verify use cases for connectivity with ActiveMQ broker and write examples as needed for the use cases.
This use cases will cover scenarios such as

• Receiving messages from queue/topic
• Sending messages to queue/topic
• Shared subscription scenario
• JMS based management operations such as creating temporary topics

Implement content transformation example

Description:

Write content trasnformation example in Ballerina based on doc
https://docs.wso2.com/display/EI650/Transforming+Message+Content

Issues in Appointment Reservation - Hospital Service

Description:
Following issues are found in hospital-service. (ClemencyHospitalService)

Issue 1

When it is added a different hospital(a hospital which doctor is not available) under doctor while appointment reservation, still the appointment is made. Once the appointment details are received through getAppointment, it displays the data of the doctor's original hospital. This is for the already defined doctors in clemency-hospital-service.bal.

Steps to reproduce

1. Get doctors under category surgery for the already available doctors.
   http://localhost:9090/healthcare/surgery

Following payload is received.

 {
        "name": "anne clement",
        "hospital": "clemency medical center",
        "category": "surgery",
        "availability": "8.00 a.m - 10.00 a.m",
        "fee": 12000
    },
    {
        "name": "thomas collins",
        "hospital": "grand oak community hospital",
        "category": "surgery",
        "availability": "9.00 a.m - 11.00 a.m",
        "fee": 7000
    },
    {
        "name": "seth mears",
        "hospital": "pine valley community hospital",
        "category": "surgery",
        "availability": "3.00 p.m - 5.00 p.m",
        "fee": 8000
    }

2.Reserve the appointment as below. For the hospital provide any bogus value.

{  
	"patient":
	{  
        "name":"K Winslet",
        "dob":"1970-03-19",
        "ssn":"234-987-175",
        "address":"Canada",
        "phone":"32456789765",
        "email":"[email protected]"
    },
	"doctor":"anne clement",
	"hospital":"sdfgh",
	"appointmentDate":"2020-01-22"
}

3. Check the response payload.

{
    "appointmentNumber": 18,
    "doctor": {
        "name": "anne clement",
        "hospital": "clemency medical center",
        "category": "surgery",
        "availability": "8.00 a.m - 10.00 a.m",
        "fee": 12000
    },
    "patient": {
        "name": "K Winslet",
        "dob": "1970-03-19",
        "ssn": "234-987-175",
        "address": "Canada",
        "phone": "32456789765",
        "email": "[email protected]"
    },
    "fee": 12000,
    "confirmed": false,
    "appointmentDate": "2020-01-22"
}

This issue occurs for the already available doctors only. Once we add a new doctor and retry the same steps following response is received.
'Doctor K Amugoda is not available in clemency medical center'. This should be the expected behavior for already available doctors.

Issue 2

Appointments can be reserved for a past date.

Issue 3

Following fields are not validated.
email address, phone, dob, date

Issue 4

Adding a new doctor is repeated in hospital service which is in healthcare service.

VSCode based Ballerina Integrations plugin

Need to create a Ballerina integration specific VS Code plugin to provide assistance to an integration developer who is developing his integration use cases using Ballerina.

This includes functionality,

• Creating integration projects using templates
• Browsing connectors in Ballerina Central
• Code snippets templates support.
• Context sensitive integration use case specific intelli sense.

Implement inital static web page generation using the markdown files

Description:

The implementation can be done using either Jekyll or MkDocs.

Implement Connectors for well-known Integration Backends

Description:

Need to implement or verify existing connectors for well-known integration use cases with the following services

• Salesforce Connector for SOAP and REST (Need to improve the existing connector)
• SAP Connector ( Verify and improve the connector as needed)
• Amazon S3 Connector
• Amazon SQS Connector
• File Connector ( Use built in Ballerina I/O. Verify and improve as needed)
• Kafka Connector ( Verify and improve the connector as needed )

Add support for event notification in Salesforce connector

Description:
Consume Salesforce Streaming API to support near real-time streaming of events.

Add support for batch data processing in Salesforce connector

Description:
Consume Salesforce Bulk API and introduce jobs and batch data processing.

Implement a guide for exposing a SOAP service as a REST API

Description:
A Ballerina client should be implemented to consume a SOAP service as a REST API

Implement and observability example

Description:

Write an observability example in Ballerina based on doc
https://docs.wso2.com/display/EI650/Using+the+Analytics+Dashboard

MTOM Protocol Support in SOAP Ballerina Client

Description:
MTOM Protocol Should be Supported in SOAP Ballerina Client

Move Integration specific Ballerina By Guides to Ballerina Integrator repository

Verify NATS Connector

• Verify NATS server and for EI use cases
• NATS server use cases
• NATS streaming use cases

Implement Amazon SQS Connector

Description:

Implement Amazon SQS Ballerina connector using the REST API.

We might need to consider following when designing the API's

• Handling requests for producing and receiving messages
• Handling requests in batches.
• Improving producer and consumer throughput using horizontal scaling

Implement Hospital Service Backend in Ballerina

Description:

Implement the hospital service backend using Ballerina

Java code: https://github.com/VijithaEkanayake/Hospital-Service

Improve Salesforce Connector

Description:

Need to verify current Salesforce connector and do necessary improvements. Find out relevant integration use cases and fix any feature gaps.

Implement content based routing example

Description:

Write the sample in Ballerina. Implement the example mentioned in doc:
https://docs.wso2.com/display/EI650/Routing+Requests+Based+on+Message+Content

Implement File Connector Guide(s)

Description:

This guide(s) should cover the following use cases

• Retrieving files from a location and doing some message mediation logic
• Sending messages to a file location
• Reading and writing from/to an FTP file source
• Samba support
• FTPS / SFTP support

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Add a sample for asynchronous service invocation

Description:

A sample for asynchronous service invocation using ActiveMQ broker in the middle.
This will cover

1. listen for HTTP message (Json)
2. Covert message to JMS and send to ActiveMQ queue
3. Listen to queue and receive JMS messages
4. convert back to Json and invoke the service.

There is no need to consider on delivery reliability on this implementation.

Issues in Settle Payment payload - Healthcare Service

Description:
Settle Payment function has the following issues in healthcare-service.bal.

Issue 1 description

Payments can be settled for invalid appointment numbers, patient details, doctor category, doctor hospital.

Prerequisites

1. Reserve an appointment as below.
   http://localhost:9090/clemency/categories/surgery/reserve

{  
	"patient":
	{  
        "name":"K Winslet",
        "dob":"1970-03-19",
        "ssn":"234-987-175",
        "address":"Canada",
        "phone":"32456789765",
        "email":"[email protected]"
    },
	"doctor":"anne clement",
	"hospital":"clemency medical center",
	"appointmentDate":"2020-01-22"
}

Steps to reproduce:

1. Settle payment as below with an invalid appointment number.

{
                        "appointmentNumber":289,
                        "doctor":
                        {
                            "category": "surgery",
                            "name": "anne clement",
                            "hospital": "clemency medical center",
                            "availability": "10am - 6pm",
                            "fee": 10000.00
                        },
                        "patient":
                        {  
                            "name":"Kate Winslet",
                            "dob":"1970-03-19",
                            "ssn":"234-987-175",
                            "address":"Canada",
                            "phone":"32456789765",
                            "email":"[email protected]"
                        },
                        "fee":1800.0,
                        "confirmed":true,
                        "cardNumber":"3456812345"
                    }

Expected: There should be an error.
Actual: Successful response with a payment ID.

Settled payment successfully with payment ID: ebcc0452-65f1-4a79-a1e1-d4dfcc688544

2. Retry the same scenario by providing invalid patient details, doctor category, doctor hospital.

Issue 2 description

Once the payment is settled and confirmed through the payload, the appointment does not get confirmed when it is retrieved through getAppointment.

{
                        "appointmentNumber":14,
                        "doctor":
                        {
                            "category": "surgery",
                            "name": "anne clement",
                            "hospital": "clemency medical center",
                            "availability": "10am - 6pm",
                            "fee": 10000.00
                        },
                        "patient":
                        {  
                            "name":"Kate Winslet",
                            "dob":"1970-03-19",
                            "ssn":"234-987-175",
                            "address":"Canada",
                            "phone":"32456789765",
                            "email":"[email protected]"
                        },
                        "fee":1800.0,
                        "confirmed":true,
                        "cardNumber":"3456812345"
                    }

Once it is retrieved though the getAppointment() the result is as below.
http://localhost:9090/clemency/categories/appointments/14

{
    "appointmentNumber": 14,
    "doctor": {
        "name": "anne clement",
        "hospital": "clemency medical center",
        "category": "surgery",
        "availability": "8.00 a.m - 10.00 a.m",
        "fee": 12000
    },
    "patient": {
        "name": "Kate Winslet",
        "dob": "1970-03-19",
        "ssn": "234-987-175",
        "address": "Canada",
        "phone": "32456789765",
        "email": "[email protected]"
    },
    "fee": 12000,
    "confirmed": false,
    "appointmentDate": "2020-01-22"
}

The confirmed field is still false in the response.

Issue 3 description

In appointment confirmation, fee cannot be less than doctor fee. But it allowed to make the payment settlement with a payload as below.

{
                        "appointmentNumber":14,
                        "doctor":
                        {
                            "category": "surgery",
                            "name": "anne clement",
                            "hospital": "clemency medical center",
                            "availability": "10am - 6pm",
                            "fee": 10000.00
                        },
                        "patient":
                        {  
                            "name":"Kate Winslet",
                            "dob":"1970-03-19",
                            "ssn":"234-987-175",
                            "address":"Canada",
                            "phone":"32456789765",
                            "email":"[email protected]"
                        },
                        "fee":180.0,
                        "confirmed":true,
                        "cardNumber":"3456812345"
                    }

Add Tests for Hospital service sample

Description:
Need to create test cases for the Hospital service sample

Related Issues:
Resolves #4

Implement service orchestration example

Description:

Write an example of service orchestration based on doc
https://docs.wso2.com/display/EI650/Exposing+Several+Services+as+a+Single+Service

X509 Certificate Support for SOAP Ballerina Client

Description:
X509 Certificates should be supported in SOAP Ballerina Client.

Verify schema validation use case - JSON Schema and XSD

• Verify message validation use cases for JSON and XML formats.

Add Tests for Content Based Routing sample

Description:
Need to create a test case for the ticket Content Based Routing

Related Issues:
Content Based Routing: #7