Unable to capture Tests for Application running with HTTPS protocol

[avnibansal1995]

Unable to capture Tests for Application running with HTTPS protocol

[slayerjain]

@avnibansal1995 we do have a solution for this, however, will be able to prioritise this once we have more upvotes/use-case from Keploy cloud users.

[github-actions]

Thank you and congratulations 🎉 for opening your very first issue in keploy

[slayerjain]

Hi @avnibansal1995 , Thanks for creating the issue. Currently keploy supports outgoing (egress) HTTPS connections.

For incoming requests (ingress) TLS termination at the load balancer is more common, and so application level termination is not supported yet. Please feel free to discuss your use case with me on our slack channel or just remove TLS while testing with keploy for now :)

[avnibansal1995]

@slayerjain . Thank you for your quick response.
Are we planning to implement this in future releases to support Keploy with incoming HTTPS connections ?

[avnibansal1995]

@slayerjain - As suggested - To address the issue, I've set up nginx locally to function as a load balancer, and I'm running my application internally using HTTP. This configuration enables me to capture test cases using Keploy. During recording, the application starts first, followed by the nginx image, as the application server must be running for nginx to operate.

However, the Keploy test encounters a failure in this scenario. During recording, it captures the URL specified in the nginx conf file, which is 'http://backend/v1/ping'. When I execute the Keploy test, it initiates the application before nginx is fully operational. Consequently, the API tests fail at this point.

[slayerjain]

@avnibansal1995 you can use add the --delay 30 flag to ask keploy to wait for certain period of time (in this case 30 second).

eg: keploy test -c "python app.py" --delay 30

[slayerjain]

also it might easier for you to just disable TLS at the application itself, and not really run the nginx at all :)

[avnibansal1995]

@slayerjain - I included the '--delay' flag with a 60-second delay in the Keploy test command, and I started nginx simultaneously. During this time window, the API works correctly when accessed via Postman. However, despite these adjustments, the test set still fails with the error message: 'http://192.168.0.4/v1/ping': dial tcp 192.168.0.4:80: connect: connection refused.

In addition - about not using nginx and just disable tls , our use case actually necessitates receiving client requests exclusively via HTTPS

[slayerjain]

@avnibansal1995 is the ip 192.168.0.4 correct

[avnibansal1995]

@slayerjain - Is there a way to convert the hostname while recording the tests ?
for example : when using nginx(loadbalancer) the recorded test file shows the url as : http://backend/v1/ping based on the config provided inside nginx.conf file. We want to be able to record the test set with url : http://0.0.0.0:8080/ .

[slayerjain]

@avnibansal1995 this is done on purpose.

Since keploy records the request right before it comes to the application we strip out the actual recorded host because while replaying the host will likely change.

Eg: client maybe hitting curl example.com but there maybe hops from CDN (eg: cloudflare), then loadbalancer, then maybe nginx and then finally the application.

While testing we try to find the application IP (either localhost or container ip) and then send request to that.

[slayerjain]

can you please post more details about your setup? (Eg: keploy record and test commands, docker-compose if relevant..) please feel free to strip out confidential information.

[avnibansal1995]

Below is the configuration in my docker compose file :

version: '3.8'

volumes:
  node_modules:
  db:

services:
  postgres:
    image: postgres:13.8
    restart: always
    container_name: postgres
    environment:
      POSTGRES_USER: ${DATABASE_USERNAME}
      POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
      POSTGRES_DB: ${DATABASE_NAME}
    ports:
      - "${DATABASE_PORT}:${DATABASE_PORT}"
    volumes:
      - db:/var/lib/postgresql/data
    networks:
      - keploy-network

  myapp:
    build:
      context: pkg
      dockerfile: Dockerfile
    restart: on-failure
    container_name: myapp
    environment:
     PROVISION_URL: ${PROVISION_URL}
     ACCESS_KEY: ${ACCESS_KEY}
    ports:
      - "8000:8000"
    links:
      - postgres
    depends_on:
      - postgres
    networks:
      - keploy-network
  
  nginx:
    build:
      context: nginx
      dockerfile: Dockerfile
    container_name: nginx
    ports:
      - "80:80"
      - "443:443"
    networks:
      - keploy-network

networks:
  keploy-network:
    external: true
    driver: bridge

Using MacOS - Installed keploy using steps provided here : https://keploy.io/docs/server/macos/installation/
Before running keploy commands I am bringing up my postgres database. Then running keploy record. The request is coming to keploy record from nginx url example and is recorded in the below manner :- https://0.0.0.0:443/v1/ping
KEploy commands I am running :
Record : keploy record -c "docker run -p 8000:8000 --name myApp --network keploy-network --rm myapp" --containerName keploy --buildDelay 10s --debug
Test: keploy test -c "docker run -p 8000:8000 --name myApp --network keploy-network --rm myapp" --containerName keploy --buildDelay 20s --delay 30

Test.yaml file :

version: api.keploy.io/v1beta1
kind: Http
name: test-1
spec:
    metadata: {}
    req:
        method: GET
        proto_major: 1
        proto_minor: 0
        url: http://backend/v1/ping
        header:
            Accept: '*/*'
            Accept-Encoding: gzip, deflate, br
            Connection: close
            Host: backend
            Postman-Token: test
            User-Agent: PostmanRuntime/7.36.1
        body: ""
        body_type: ""
        timestamp: 2024-02-06T11:54:48.501881623Z
        host: ""
    resp:
        status_code: 200
        header:
            Content-Length: "121"
            Content-Type: application/json; charset=utf-8
            Date: Tue, 06 Feb 2024 11:54:48 GMT
            Vary: Origin
        body: '{"serviceName":"myapp","healthStatus":"HEALTHY","serviceStatus":"online","currentTime":"2024-02-06T11:54:48.509241843Z"}'
        body_type: ""
        status_message: ""
        proto_major: 0
        proto_minor: 0
        timestamp: 2024-02-06T11:54:50.549608857Z
    objects: []
    assertions:
        noise:
            body.currentTime: []
            header.Date: []
    created: 1707220490
curl: |
    curl --request GET \
      --url http://backend/v1/ping \
      --header 'Postman-Token: test' \
      --header 'Accept-Encoding: gzip, deflate, br' \
      --header 'Host: backend' \
      --header 'Connection: close' \
      --header 'User-Agent: PostmanRuntime/7.36.1' \
      --header 'Accept: */*' \

but while running the test i get the below error:

[avnibansal1995]

Nginx config file :

upstream backend {
    server backend:8000;

}

# HTTPS server configuration
server {
    listen 443 ssl;
    server_name myapp;

    # SSL/TLS certificate and key
    ssl_certificate /etc/nginx/certs/app-cert.crt;
    ssl_certificate_key /etc/nginx/certs/app-private.key;

    # SSL configuration (e.g., protocols, ciphers)

    # Proxy pass requests to the backend servers
    location / {
        proxy_pass http://backend/;
        # Add more proxy settings as needed
    }
}

[slayerjain]

@avnibansal1995 Thanks for sharing the details. Are you able to see a mocks.yaml created which would contain postgres mocks?

In the record/test command while using docker --containerName flag should point to the container that you can to record or test, keploy value seems wrong. please do docker container ls, and find the right container name.

Also, if you try to put the myapp's container name, it might not work because keploy will try to read the incoming calls but not understand encrypted HTTP calls. The mocks.yaml with the postgres mocks should be created.

If you put the nginx's container name, then the incoming requests should be captured as well as replayed, but postgres mocking will not be done because the postgres calls are happening from the application, and not the nginx container.

To ensure both the tests and mocks are generated, keploy doens't support incoming TLS (HTTPS) into the application.