Giter Site home page Giter Site logo

globocom / globonetworkapi Goto Github PK

View Code? Open in Web Editor NEW
83.0 56.0 67.0 8.02 MB

API to automate IP Networking management, resource allocation and provisioning.

Home Page: http://globonetworkapi.readthedocs.org/

License: Apache License 2.0

Makefile 0.10% Python 98.82% Shell 0.39% HTML 0.10% PLpgSQL 0.55% Dockerfile 0.03%
network-programming cloud provisioning automation datacenter network-topology ipam dcim

globonetworkapi's Introduction

GloboNetworkAPI

Build Status Documentation Status License

Globo NetworkAPI is a REST API that manages IP networking resources. It is supposed to be not just an IPAM, but a centralized point of network control, allowing documentation from physical and logical network and starting configuration requests to equipments.

Globo NetworkAPI is made to support a Web User Interface features, exposing its functionality to be used with any other client.

This web tool helps network administrator manage and automate networking resources (routers, switches and load balancers) and document logical and physical networking.

They were created to be vendor agnostic and to support different orchestrators and environments without loosing the centralized view of all network resources allocated.

It was not created to be and inventory database, so it does not have CMDB functionalities.

Features

  • LDAP authentication
  • Supports cabling documentation (including patch-panels/DIO’s)
  • Separated Layer 2 and Layer 3 documentation (vlan/network)
  • IPv4 and IPv6 support
  • Automatic allocation of Vlans, Networks and IP’s
  • ACL (access control list) automation (documentation/versioning/applying)
  • Load-Balancer support
  • Automated deploy of allocated resources on switches, routers and load balancers
  • Load balancers management
  • Expandable plugins for automating configuration

Documentation

Documentation

Run Unit Tests

To run the unit tests just run make build_img && make start && make test_ci this instruction will run all the unit tests specified in the networkapi/tests/__init__.py file.

How to contribute

Check this out at Contributing file.

Authors

Authors

globonetworkapi's People

Contributors

alejandraklachquin avatar arumeida avatar brunodevel avatar edersonbrilhante avatar fabiancabau avatar gustavosoares avatar henriquebonadio avatar henriquebonadio-zz avatar jonathanalcantara avatar leopoldomauricio avatar malinoski avatar marcelometal avatar pantuza avatar rafaelcassau avatar renan-lopes-rodrigues avatar renanlopes97s avatar s2it-globo avatar suptel avatar treviza153 avatar victor-mendes-eduardo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

globonetworkapi's Issues

wrong behavior of validated flag and overwrite param in add_block and apply_l7 funcions

Expected behavior:

  • When add_block() is called, the API should insert the requested block in the rule present in l7_filter_current field, and write the resulting rule int the l7_filter_to_apply field.
  • The add_block() should set the validated flag of the l7 to 1 if the block was sucessfully inserted, once it does not make sense for an automated script to call the add_block followed by the validate_l7 function.
  • The add_block() should check if the validated flag is already set to 1 (which means an l7 script is already validated and waiting for apply_l7 function). If it is the case, the overwrite parameter should be checked. If it is present, the requested block is inserted in the rule present in l7_filter_current field, and write the resulting rule int the l7_filter_to_apply field. If overwrite is not set, an error should be raised
  • The apply_l7() function should set the validated flag back to 0, once the l7 has been apllyed to the load balancer. Also, the rule present in l7_filter_current should be moved to the l7_filter_rollback field, and the rule present in l7_filter_to_apply should be moved to the l7_filter_current field, resulting in an empty l7_filter_to_apply field, with validated flag equal to 0.

Create and manage a pool without any VIP request association

Today, when we allocate a VIP requisition, it creates one server pool for each group of servers that attend the same VIP port.
We need to be able to allocate these server pools "request" apart from VIP request. We need similar form, but to be able to allocate a server pool by itself, not connected to any VIP request. We also need the server pool to have this healthcheck configuration by itself, today the healthcheck configuration is a feature of the VIP request (one per VIP).
We need the same kind of functionalities that we have in the VIP request:

  • We need to be able to deploy(create)/delete this server pool in the LB. If the server pool is associated with any VIP request, the API should deny the deletion.
  • We need to be able to add, delete, enable and disable a member (server:port) in the server pool
  • We need to be able to check the status of a member (server:port) in a server pool. This status should be checked in the load balancer by the external plugin (scripts).

Wrong path for DB scripts in docs

Install docs are pointing to invalid location at:

mysql -u -p -h < <$NETWORKAPI_FOLDER/docs/scripts/database_configuration.sql

mysql -u -p -h < <$NETWORKAPI_FOLDER/docs/scripts/load_example_environment.sql

the correct location is $NETWORKAPI_FOLDER/dev/

Adding a new pool/vip port into a already created vip

When adding port to a VIP that is already created and linking it with pool that are not deployed, it tries to use the pool in the LB without creating it, raising error:

[ERROR] 24/Aug/2016:14:59:22 +0000 - U:OKAMA , P:/api/v3/vip-request/deploy/7169/, N:networkapi.api_vip_request.views.views_v3:152 , T:ZHU12KWAR5IUBBZUSOCDJW, MSG:Error: Error applying command on equipment. Equipment returned error status. <<Error: Error applying command on equipment. Equipment returned error status. <<Server raised fault: 'Exception caught in System::urn:iControl:System/Session::submit_transaction()

Allocating vlans with same number in different environments

API is not checking if equipments are shared between environment when allocating vlan in different environments when two requests at the same time. This results in same vlan number being allocated in different environments, but they will have to be created in same equipments. That goes wrong.

Error when deleting pool request

Fix delete method of pool:
api is trying to delete the members from the pool in the equipment, but as the pool does not exist in the equipment (it is a condition in order to delete the request!), we have an script error.
We have to remove this part:
command = settings.POOL_REAL_REMOVE % (id_pool, id_ip, port_ip)
code, _, _ = exec_script(command)
if code != 0:
raise exceptions.ScriptDeletePoolException()

in def delete(request) from api_pools/views.

Insert a description field in "Ambiente VIP" (Environment VIP)

I would like to have some kind of description field in "Ambiente VIP" to be used as a help in the WebUI, to tell the user what are the caracteristics of their choices when creating a VIP.
When there are several choices, user starts to get confused. Need a text to explain them.

Deadlock when allocating vlan on /vlan/no-network/

When allocating vlans in switches with multiple environments that has vlan number range in common, a deadlock can occur when locking the envinronments id.

Ex:
request 1: [<Ambiente: 120>, <Ambiente: 121>, <Ambiente: 158>, <Ambiente: 88>, <Ambiente: 159>, <Ambiente: 173>]
request 2: [<Ambiente: 88>, <Ambiente: 120>, <Ambiente: 158>, <Ambiente: 159>, <Ambiente: 173>]

request 1 can locks at 88 and request 2 locks at 120 if started at the same time.

Shared IP that is part of a VIP causing problems listing VIP and deleting equipment association

OBS: Dois equipamentos podem ter o mesmo IP cadastrado. Isso é o esperado para IP<>equipamentos.
It is expected an IP to be registered to one or more equipments.
The problem occurs when we add this shared IP to a VIP (Virtual Server - LB). When listing this VIP, we get the error "Falha ao acessar a fonte de dados. (Database failure).". As the VIP pool member is related to an IP, the issue is possibly because the NetworkAPI tries to find a unique equipment to this VIP but it gets more than one.
If the IP is registered to more than one equipment, the VIP has to show them all.

Another problem: I tryed to delete the association of one of the servers but it does not let me do that. When I tryed to do that, it gave the me error:

"Não foi possível excluir o ip de id 42002 pois ele está sendo usado em uma requisição VIP."

I am not trying to delete the IP itself. It should still be registered in the DB for the other equipment and this error should not happen.

Returning error when changing healthcheck parameter in "PUT request-vip/id"

Although the client is sending correct data, with no values set in healthcheck and healthcheck expect, NetworkAPI is raising error:

2015-03-31 11:06:01,203 DEBUG Requisição do usuário gcloud_citta concluída com falha. Conteúdo: [0133Valor do tipo do healthcheck inválido ou inconsistente em relação ao valor do healthcheck_expect].

Change server status in a pool

Need to change the status of all the servers in a pool (enable/disable) in a single request, instead of checking servers one by one.

/equipamento/get_real_related/<id_equip>/ crashes when IP is a poolmember in a pool that is in use by two or more vip_requests

[DEBUG] 09/Jul/2015:15:49:43 +0000 - U:MARCUS.VINICIUS, P:/equipamento/get_real_related/21191/, T:0YOZULJPSXIOQPOLTRP3VA, MSG:INICIO da requisição GET. Data: [].
[DEBUG] 09/Jul/2015:15:49:43 +0000 - U:gcloud, P:/ip/checkvipip/, T:7NCSGOAQTCQVIMZVVT4JEA, MSG:INICIO da requisição POST. Data: [<ip_map><id_evip>38</id_evip>192.168.255.130</ip_map>].
[DEBUG] 09/Jul/2015:15:49:43 +0000 - U:MARCUS.VINICIUS, P:/equipamento/get_real_related/21191/, T:0YOZULJPSXIOQPOLTRP3VA, MSG:Requisição concluída com falha. Conteúdo: [0001Falha ao acessar a fonte de dados].
[DEBUG] 09/Jul/2015:15:49:43 +0000 - U:MARCUS.VINICIUS, P:/equipamento/get_real_related/21191/, T:0YOZULJPSXIOQPOLTRP3VA, MSG:FIM da requisição.

API is allocating same range twice with concurrent requests

Requests below allocated same network for vlan:

[DEBUG] 27/Aug/2015:12:14:15 +0000 - U:XXXXXX, P:/network/ipv4/add/, T:MMB4XS3TRF-OIEVLNOQGMQ, MSG:INICIO da requisição POST. Data: [<id_vlan>25965</id_vlan><id_tipo_rede>6</id_tipo_rede>].

[DEBUG] 27/Aug/2015:12:14:15 +0000 - U:XXXXXX, P:/network/ipv4/add/, T:MPOEFZC0RQM5VWFD_GHMIQ, MSG:INICIO da requisição POST. Data: [<id_vlan>25966</id_vlan><id_tipo_rede>6</id_tipo_rede>].

VIP request return inconsistences

POST request is returning id with type different from PUT request:

POST is returning number, PUT is returning string:

POST https://networkapi/api/vip/request/save/

Request:
{"ip":101873,"finalidade":"Homologacao","cliente":"Usuario Interno","ambiente":"Homologacao BE-TESTE API","cache":"(nenhum)","timeout":"5","persistencia":"(nenhum)","host":"vip-daniel-com-pool.globoi.com","areanegocio":"time-evolucao","nome_servico":"vip-daniel-com-pool.globoi.com","vip_ports_to_pools":[{"port_vip":80,"server_pool":12494}]}

Response:
HTTP/1.1 201 CREATED
{"id": 11034, "ip": 101873, "ipv6": null, "l7_filter": null, "filter_applied": null, "filter_rollback": null, "filter_valid": true, "applied_l7_datetime": null, "healthcheck_expect": null, "rule": null, "rule_applied": null, "rule_rollback": null, "areanegocio": "time-evolucao", "nome_servico": "vip-daniel-com-pool.globoi.com", "host": "vip-daniel-com-pool.globoi.com", "vip_ports_to_pools": [{"id": 12495, "requisicao_vip": 11034, "server_pool": 12494, "port_vip": 80}], "finalidade": "Homologacao", "cliente": "Usuario Interno", "ambiente": "Homologacao BE-TESTE API"}

PUT https://networkapi/api/vip/request/save/11034

Request:
{"ip":101873,"finalidade":"Homologacao","cliente":"Usuario Interno","ambiente":"Homologacao BE-TESTE API","cache":"(nenhum)","timeout":"5","persistencia":"source-ip","host":"teste-vip-daniel.lb.globoi.com","areanegocio":"time-evolucao","nome_servico":"teste-vip-daniel.lb.globoi.com","vip_ports_to_pools":[{"port_vip":80,"requisicao_vip":11034,"server_pool":12494}]}

Response:
{"id": "11034", "ip": 101873, "ipv6": null, "l7_filter": null, "filter_applied": null, "filter_rollback": null, "filter_valid": true, "applied_l7_datetime": null, "healthcheck_expect": null, "rule": null, "rule_applied": null, "rule_rollback": null, "areanegocio": "time-evolucao", "nome_servico": "teste-vip-daniel.lb.globoi.com", "host": "teste-vip-daniel.lb.globoi.com", "vip_ports_to_pools": [{"id": null, "requisicao_vip": null, "server_pool": 12494, "port_vip": 80}], "finalidade": "Homologacao", "cliente": "Usuario Interno", "ambiente": "Homologacao BE-TESTE API"}

wrong behavior of rollback_l7 function

Expected behavior:

  • The rollback_l7() should use the rule present in the l7_filter_rollback and apply this in the load balancer (as if it was calling apply_l7()). Once the rule was applied to the load balancer, the l7_filter_current should be moved to the l7_filter_rollback, and the l7_filter_rollback should be moved to the l7_filter_current (they will switch places).
  • The l7_filter_to_apply field and the validated flag should not be touched. It is only a switching between the current rule applied to the last ruled applied.

Create a VIP request using pools that already exist

When creating a VIP request, we need to have the option to use existing pools instead of specifing the servers:ports and create new pool entries in the DB/API.
Today, deleting the VIP request deletes every server pool associated with it. API calls should have the option to remove the VIP request AND the server pool or to remove ONLY the VIP request, without removing the server pool from the LB and from the API database. The default option should be to delete everything, as this is the most used functionality.

Possibility to delete a created VIP requisition

How to reproduce:

  1. Create a VIP req, validate and create it (deploy).
  2. Select the VIP req and click on "remover"
  3. The req is deleted from DB but the VIPs is kept deployed on LB.

Expected behavior:

  1. NetworkAPI should return an error message when the VIP Req is in created state (criado=1).
  2. NetworkAPI WebUI should show the error to user.

Logs:
2014-07-31 11:14:14,323 RequisicaoVipDeleteResource DEBUG INICIO da requisição DELETE para URL /vip/delete/6630/. XML: [].
2014-07-31 11:14:14,325 RequisicaoVipDeleteResource DEBUG Usuário da requisição: HENRIQUE.
2014-07-31 11:14:14,387 MemcachedLock WARNING Added=True
2014-07-31 11:14:15,841 RequisicaoVipDeleteResource DEBUG Requisição do usuário HENRIQUE concluída com sucesso.
2014-07-31 11:14:15,842 RequisicaoVipDeleteResource DEBUG FIM da requisição do usuário HENRIQUE.
2014-07-31 11:14:16,247 RequestVipGetIdIpResource DEBUG INICIO da requisição POST para URL /requestvip/get_by_ip_id/. XML: [<start_record>0</start_record>False<custom_search></custom_search><end_record>25</end_record><id_vip/>].
2014-07-31 11:14:16,248 RequestVipGetIdIpResource DEBUG Usuário da requisição: HENRIQUE.
2014-07-31 11:14:16,259 RequestVipGetIdIpResource INFO Find all Vip Requests
2014-07-31 11:14:16,641 RequestVipGetIdIpResource DEBUG Requisição do usuário HENRIQUE concluída com sucesso.
2014-07-31 11:14:16,643 RequestVipGetIdIpResource DEBUG FIM da requisição do usuário HENRIQUE.

#Change priority-group of real members.

When you have a pool where all members have the same priority (e.g priority 1) and you add new members, all of them with this same priority, the priority is changed to 0 for all members on database, although on equipment only new members will be set as priority 0 and old members will stay the same (1 e.g).

Deleting IPv4 during VIP creation process

It's possible to delete an IPv4 during request VIP creation process and it does not raises any error.
Consequently deletes the vip request while configuring LB's leaving trashed configuration in LB's.

[DEBUG] 03/Jun/2016:12:47:26 +0000 - U:User, P:/vip/create/, N:networkapi.extra_logging.middleware:74 , T:KADXHOGUT7QAXDSVMRAQ2Q, MSG:INICIO da requisição POST. Data: [<id_healthcheck_expect /><portas_servicos></portas_servicos><reals_prioritys><reals_priority /></reals_prioritys><reals_weights><reals_weight /></reals_weights><id_vip>12153</id_vip>].
[DEBUG] 03/Jun/2016:12:47:26 +0000 - U:User, P:/vip/create/, N:Usuario:197 , T:KADXHOGUT7QAXDSVMRAQ2Q, MSG:Using local authentication for user 'User'
[WARNING] 03/Jun/2016:12:47:26 +0000 - U:User, P:/vip/create/, N:MemcachedLock:47 , T:KADXHOGUT7QAXDSVMRAQ2Q, MSG:Added=True

[DEBUG] 03/Jun/2016:12:47:28 +0000 - U:User, P:/ip4/delete/112588/, N:networkapi.extra_logging.middleware:74 , T:PQZR0OXZSKSSQNDTO7EMBW, MSG:INICIO da requisição GET. Data: [].
[DEBUG] 03/Jun/2016:12:47:28 +0000 - U:User, P:/ip4/delete/112588/, N:Usuario:197 , T:PQZR0OXZSKSSQNDTO7EMBW, MSG:Using local authentication for user 'User'
[INFO] 03/Jun/2016:12:47:28 +0000 - U:User, P:/ip4/delete/112588/, N:IPv4DeleteResource:44 , T:PQZR0OXZSKSSQNDTO7EMBW, MSG:Delete an IP4
[WARNING] 03/Jun/2016:12:47:28 +0000 - U:User, P:/ip4/delete/112588/, N:MemcachedLock:47 , T:PQZR0OXZSKSSQNDTO7EMBW, MSG:Added=True
[DEBUG] 03/Jun/2016:12:47:29 +0000 - U:User, P:/ip4/delete/112588/, N:networkapi.extra_logging.middleware:82 , T:PQZR0OXZSKSSQNDTO7EMBW, MSG:Requisição concluída com sucesso.
[DEBUG] 03/Jun/2016:12:47:29 +0000 - U:User, P:/ip4/delete/112588/, N:networkapi.extra_logging.middleware:84 , T:PQZR0OXZSKSSQNDTO7EMBW, MSG:FIM da requisição.

[DEBUG] 03/Jun/2016:12:47:31 +0000 - U:User, P:/vip/create/, N:networkapi.extra_logging.middleware:82 , T:KADXHOGUT7QAXDSVMRAQ2Q, MSG:Requisição concluída com sucesso.
[DEBUG] 03/Jun/2016:12:47:31 +0000 - U:User, P:/vip/create/, N:networkapi.extra_logging.middleware:84 , T:KADXHOGUT7QAXDSVMRAQ2Q, MSG:FIM da requisição.

NetworkAPI is marking created flag=1 when external plugin returns error

How to reproduce:

  1. Create a VIP requisition with some wrong parameter (in this case, and invalid L7 filter).
  2. Validate it
  3. Create it (deploy)
  4. The plugin returns an error and the VIP is not created. NetworkAPI logs the error, but the VIP is marked as created in the DB (criado=1).

Expected behavior:

  1. The req should not have been marked criado=1 in cases the plugin finishes with errors.

Logs:
-- networkapi.log --
2014-07-31 11:41:36,139 CreateVipResource DEBUG INICIO da requisição POST para URL /vip/create/. XML: [<id_vip>6630</id_vip>].
2014-07-31 11:41:36,141 CreateVipResource DEBUG Usuário da requisição: HENRIQUE.
2014-07-31 11:41:36,167 MemcachedLock WARNING Added=True
2014-07-31 11:41:58,459 CreateVipResource DEBUG Requisição do usuário HENRIQUE concluída com falha. Conteúdo: [0002Falha ao executar o script. Causa: ......
2014-07-31 11:41:58,462 CreateVipResource DEBUG FIM da requisição do usuário HENRIQUE.
2014-07-31 11:41:58,476 RequestVipGetByIdResource DEBUG INICIO da requisição GET para URL /requestvip/getbyid/6630/. XML: [].
2014-07-31 11:41:58,478 RequestVipGetByIdResource DEBUG Usuário da requisição: HENRIQUE.
2014-07-31 11:41:58,487 RequestVipGetByIdResource INFO Find Vip Request by id
2014-07-31 11:41:58,543 RequestVipGetByIdResource DEBUG Requisição do usuário HENRIQUE concluída com sucesso.
2014-07-31 11:41:58,544 RequestVipGetByIdResource DEBUG FIM da requisição do usuário HENRIQUE.

-- PLUGIN --
[networkapi@test_server ~]$ ./plugin_call_for_testing
(...)
Finalizando execucao com 3 ERRO(S) ENCONTRADO(S). Verificar a configuração dos equipamentos.(...)
[networkapi@test_server ~]$ echo $?
1
[networkapi@test_server ~]$

POST /network/add/ error

Request to this method is always getting error when network is not the same of any other.
When checking subnets, it checks all vlans from all environments, including the environment itself but it is raising error if the vlan being tested is from same environment.
It should only raise error when the network is in the same environment BUT is a sub/super network of the desired one.

Search pools by member error when the server is member of a pool that has no relation with vip

When we have a server that is member of a pool that hasn't a entry in "VipPortToPool" (so the pool is not correlated to any vip - ie, with any instance from table RequisicaoVips), the API is unable to display anything when searching for pools that a server is member from (note that the API knows it is a member of at least one pool, but is unable to list it/them). Error shown: "unable to access data source/Falha ao acessar a fonte de dados".

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.