vinyldns/vinyldns-python

Python client library for VinylDNS

VinylDNS

VinylDNS is a vendor-agnostic front-end for enabling self-service DNS and streamlining DNS operations. VinylDNS manages millions of DNS records supporting thousands of engineers in production at Comcast. The platform provides fine-grained access controls, auditing of all changes, a self-service user interface, secure RESTful API, and integration with infrastructure automation tools like Ansible and Terraform. It is designed to integrate with your existing DNS infrastructure, and provides extensibility to fit your installation.

VinylDNS helps secure DNS management via:

• AWS Sig4 signing of all messages to ensure that the message that was sent was not altered in transit
• Throttling of DNS updates to rate limit concurrent updates against your DNS systems
• Encrypting user secrets and TSIG keys at rest and in-transit
• Recording every change made to DNS records and zones

Integration is simple with first-class language support including:

• Java
• Python
• Go
• JavaScript

Table of Contents

• Quickstart
  • Quickstart Optimization
• Things to Try in the Portal
  • Verifying Your Changes
  • Other things to note
• Code of Conduct
• Developer Guide
• Contributing
• Maintainers and Contributors
• Credits

Quickstart

Docker images for VinylDNS live on Docker Hub at https://hub.docker.com/u/vinyldns/. To start up a local instance of VinylDNS on your machine with docker:

1. Ensure that you have docker and docker-compose
2. Clone the repo: git clone https://github.com/vinyldns/vinyldns.git
3. Navigate to repo: cd vinyldns
4. Run ./quickstart/quickstart-vinyldns.sh. This will start up the api at localhost:9000 and the portal at localhost:9001
5. See Things to Try in the Portal for getting familiar with the Portal
6. To stop the local setup, run ./utils/clean-vinyldns-containers.sh.

There exist several clients at https://github.com/vinyldns that can be used to make API requests, using the endpoint http://localhost:9000.

Quickstart Optimization

If you are experimenting with Quickstart, you may encounter a delay each time you run it. This is because the API and Portal are rebuilt every time you launch Quickstart. If you'd like to cache the builds of the API and Portal, you may want to first run:

Script	Description
build/assemble_api.sh	This will create the API jar file which will then be used by Quickstart
build/assemble_portal.sh	This will create the Portal zip file which will then be used by Quickstart

Once these scripts are run, the artifacts are placed into the artifacts/ directory and will be reused for each Quickstart launch. If you'd like to regenerate the artifacts, simply delete them and rerun the scripts above.

Things to Try in the Portal

1. View the portal at http://localhost:9001 in a web browser
2. Login with the credentials professor and professor
3. Navigate to the groups tab: http://localhost:9001/groups
4. Click on the New Group button and create a new group, the group id is the uuid in the url after you view the group
5. Connect a zone by going to the zones tab: http://localhost:9001/zones.
   1. Click the -> Connect button
   2. For Zone Name enter ok with an email of [email protected]
   3. For Admin Group, choose a group you created from the previous step
   4. Leave everything else as-is and click the Connect button at the bottom of the form
6. A new zone ok should appear in your My Zones tab (you may need to refresh your browser)
7. You will see that some records are preloaded in the zone already, this is because these records are preloaded in the local docker DNS server and VinylDNS automatically syncs records with the backend DNS server upon zone connection
8. From here, you can create DNS record sets in the Manage Records tab, and manage zone settings and ACL rules in the Manage Zone tab
9. To try creating a DNS record, click on the Create Record Set button under Records, Record Type = A, Record Name = my-test-a, TTL = 300, IP Addressess = 1.1.1.1
10. Click on the Refresh button under Records, you should see your new record created

Verifying Your Changes

VinylDNS will synchronize with the DNS backend. For the Quickstart this should be running on port 19001 on localhost .

To verify your changes, you can use a DNS resolution utility like dig

$ dig @127.0.0.1 -p 19001 +short my-test-a.ok
1.1.1.1

This tells dig to use 127.0.0.1 as the resolver on port 19001. The +short just makes the output a bit less verbose. Finally, the record we're looking up is my-test-a.ok. You can see the returned output of 1.1.1.1 matches the record data we entered.

Other things to note

1. Upon connecting to a zone for the first time, a zone sync is executed to provide VinylDNS a copy of the records in the zone
2. Changes made via VinylDNS are made against the DNS backend, you do not need to sync the zone further to push those changes out
3. If changes to the zone are made outside of VinylDNS, then the zone will have to be re-synced to give VinylDNS a copy of those records
4. If you wish to modify the url used in the creation process from http://localhost:9000, to say http://vinyldns.yourdomain.com:9000, you can modify the quickstart/.env file before execution.
5. Further configuration can be ac https://www.vinyldns.io/operator/config-portal & https://www.vinyldns.io/operator/config-api

Code of Conduct

This project, and everyone participating in it, are governed by the VinylDNS Code Of Conduct. By participating, you agree to this Code.

Developer Guide

See DEVELOPER_GUIDE.md for instructions on setting up VinylDNS locally.

Contributing

See the Contributing Guide.

Maintainers and Contributors

The current maintainers (people who can merge pull requests) are:

• Ryan Emerle (@remerle)
• Sriram Ramakrishnan (@sramakr)
• Jim Wakemen (@jwakemen)

See AUTHORS.md for the full list of contributors to VinylDNS.

See MAINTAINERS.md for documentation specific to maintainers

Credits

VinylDNS would not be possible without the help of many other pieces of open source software. Thank you open source world!

Given the Apache 2.0 license of VinylDNS, we specifically want to call out the following libraries and their corresponding licenses shown below.

• logback-classic
  • Eclipse Public License 1.0
• logback-core
  • Eclipse Public License 1.0
• h2 database
  • Mozilla Public License, version 2.0
• pureconfig
  • Mozilla Public License, version 2.0
• pureconfig-macros
  • Mozilla Public License, version 2.0
• junit
  • Eclipse Public License 1.0

vinyldns-python's Issues

Build out README

Can definitely leverage https://github.com/vinyldns/vinyldns-ruby for help on this. A few things to include:

• this will be installed with pip install vinyldns-python
• example usage:

from vinyldns_python import *
vinyl_client = client.VinylDNSClient("http://localhost:9000", "testUserAccessKey", "testUserSecretKey")
vinyl_client.list_zones()

• users need to know their api url, access key, and secret key. those can be accessed in the portal (link to https://www.vinyldns.io/portal/credentials)
• the python client supports everything in https://www.vinyldns.io/api/; can look at each public method in https://github.com/vinyldns/vinyldns-python/blob/master/vinyldns_python/client.py for the syntax. include details for each

Add flake8

See #13 - same idea. Run flake8 and address any issues. We will want this tied into travis (or a commit hook) so we remain well-formatted.

Flake8: http://flake8.pycqa.org/en/latest/

Add RecordSets search support

Description

The VinylDNS API has been updated to include a new global recordset search endpoint at /recordsets. This client should be updated to support this new endpoint.

Fix flake8 style guide failures

We are using the lowercase letter L as a variable name with can be confused with I or 1.

tests/test_membership.py:34:50: E741 ambiguous variable name 'l'
tests/test_membership.py:35:50: E741 ambiguous variable name 'l'
tests/test_membership.py:80:9: E741 ambiguous variable name 'l'
tests/test_membership.py:103:9: E741 ambiguous variable name 'l'
tests/test_membership.py:120:9: E741 ambiguous variable name 'l'
tests/test_membership.py:139:9: E741 ambiguous variable name 'l'
tests/test_membership.py:161:9: E741 ambiguous variable name 'l'
tests/test_batch_change.py:47:13: E741 ambiguous variable name 'l'
tests/test_batch_change.py:66:9: E741 ambiguous variable name 'l'
tests/test_batch_change.py:246:9: E741 ambiguous variable name 'l'
tests/test_records.py:32:50: E741 ambiguous variable name 'l'
tests/test_records.py:113:9: E741 ambiguous variable name 'l'
tests/test_records.py:139:9: E741 ambiguous variable name 'l'
tests/test_records.py:179:9: E741 ambiguous variable name 'l'
tests/test_zones.py:48:46: E741 ambiguous variable name 'l'
tests/test_zones.py:110:9: E741 ambiguous variable name 'l'
tests/test_zones.py:144:9: E741 ambiguous variable name 'l'

Date fields throw BadRequest

Ran the following commands:

client = VinylDNSClient("http://localhost:9000", "xxx", "xxx")
existing_txt = [rs for rs in existing_records.record_sets if rs.type == "TXT"][0]
data_to_add = TXTData("some-test-txt-to-add")
existing_txt.records.append(data_to_add)
existing_txt.records
[<vinyldns.record.TXTData object at 0x102e970f0>, <vinyldns.record.TXTData object at 0x102e53e48>]
updated = client.update_record_set(existing_txt)
Traceback (most recent call last):
File "", line 1, in
File "/Users/rstar201/Desktop/Vinyl/python/vinyldns-python/src/vinyldns/client.py", line 519, in update_record_set
to_json_string(record_set), **kwargs)
File "/Users/rstar201/Desktop/Vinyl/python/vinyldns-python/src/vinyldns/client.py", line 157, in __make_request
return self.__check_response(response)
File "/Users/rstar201/Desktop/Vinyl/python/vinyldns-python/src/vinyldns/client.py", line 164, in __check_response
raise BadRequestError(response.text)
vinyldns.client.BadRequestError: {"errors":["Invalid date format 2019-06-25T16:37:09+00:00"]}

the api doesnt use created/update on an update_recordset call anyway, so i just cleared those fields to avoid it, but we should fix to make sure people dont run into this same issue

Add ownerGroupId to BatchRecordChanges create method

In support of the new shared zones feature coming to VinylDNS we need to add the optional ownerGroupId field to the BatchRecordChanges create method. Refer to the VinylDNS documentation for more information.

Example Request:

{
    "comments": "this is optional",
    "ownerGroupId": "f42385e4-5675-38c0-b42f-64105e743bfe"
    "changes": [
        {
            "inputName": "example.com.",
            "changeType": "Add",
            "type": "A",  
            "ttl": 3600, 
            "record": {
                "address": "1.1.1.1"
            } 
        }, 
        {
            "inputName": "192.0.2.195",
            "changeType": "Add",
            "type": "PTR", 
            "ttl": 3600,
            "record": {
                "ptrdname": "ptrdata.data."
            }
        }, 
        {
            "inputName": "cname.example.com.",
            "changeType": "DeleteRecordSet",
            "type": "CNAME"
        }, 
        {
            "inputName": "update.another.example.com.",
            "changeType": "DeleteRecordSet",
            "type": "AAAA"
        }, 
        {
            "inputName": "update.another.example.com.",
            "changeType": "Add",
            "type": "AAAA", 
            "ttl": 4000,
            "record": {
                "address": "1:2:3:4:5:6:7:8"
            }
        }
    ]
}

Additional Information

• If no ownerGroupId is provided the attribute should not be included in the request.

Add headers check

In this repo, there are few enough files (and likely wont be many more) that I think a pre-commit hook should work.

If there is some built-in python way to do this, do that. If not, a script is fine. See https://github.com/vinyldns/vinyldns/blob/master/bin/add-license-headers.sh for an example of how we scripted this for the non-scala files in the api

Add support for isTest flag to zone model

The API supports an isTest flag in the zone connection request (model: https://github.com/vinyldns/vinyldns/blob/1a2d12a7e983a7229aec573a618b275edf1b3343/modules/core/src/main/scala/vinyldns/core/domain/zone/Zone.scala#L114).

The zone model needs to include isTest so that String serialization prints out this info.

Python 3 support

See title. Need to have a python 3 version of this.

Unsure what amount of code changes are required for this, need to investigate

Add Parameter Type Information to Docstrings

Description

Docstrings are missing type info for parameters, which has led to confusion amongst some users. We should also enhance the docstrings with more detailed information where necessary.

Add pep257 linter

https://pep257.readthedocs.io/en/latest/ seem to work.

Need to run the linter and address any issues it finds. Will also want to tie this into our travis checks (see #10)

Add maintainers guide detailing release instructions

Don't see info on how to run a release

Add support for get zone by name API endpoint

API endpoint documentation: https://www.vinyldns.io/api/get-zone-by-name.html

Add environment variable support

Right now, to use this client, you have to create a new client:

client = VinylDNSClient(url, access_key, secret_key)
zones = client.getZones()

It would be nice to allow people to set the environment variables and call getZones (or all the other api functions) directly. This is similar to how the ruby client works (see https://github.com/vinyldns/vinyldns-ruby#requirements)

Setup travis

This is probably a bit blocked by #9 and maybe #7.

Once we have testing at least, we would want travis to run the tests and pass (or fail) prs accordingly. Failed PRs should not be mergable.

Add ownerGroupId to RecordSet create method

In support of the new shared zones feature coming to VinylDNS we need to add the optional ownerGroupId field to the RecordSet create method.

Example Request:

{
  "name": "foo",
  "type": "A",
  "ttl": 300,
  "records": [
    {
      "address": "10.10.10.10"
    }
  ],
  "zoneId": "2467dc05-68eb-4498-a9d5-78d24bb0893c",
  "ownerGroupId": "f42385e4-5675-38c0-b42f-64105e743bfe"
}

Additional Information

• If no ownerGroupId is provided the attribute should not be included in the request.

Need to release new updates?

Saw that couple of commits were added, especially the global "search_record_sets" in client.py. For some reason, even a new pip install does not reflect that change in packages. Would you be able to help me with this one? Thanks!

Add testing framework

Needs a little research (or expertise) here. We need some kind of testing framework.
Unclear on how to unit test this, since it is just an API wrapper for vinyldns api. There seem to be a lot of unit testing API tools out there, would like whoever starts this up to laydown some opinions on the matter. If not, this looked interesting and it works with pytest which we use for the vinyldns api - https://github.com/taverntesting/tavern

whatever testing is added here should be tied into travis

AC:

• We have one function tested (pick any, say getRecordSet)
• That test is running travis

If the tests arent just copy/paste, we will make followup cards to test the other functions

Add travis build badge to Readme

We use Travis CI to build and test the VinylDNS Python client. Adding the Travis badge will quickly indicate that to users and let them know if the master branch is building and passing.

Should be: [![Travis build](https://api.travis-ci.org/vinyldns/vinyldns-python.svg?branch=master)](https://travis-ci.org/vinyldns/vinyldns-python)

see the main VinylDNS Readme for reference

Create gh-pages Documentation Site for the Python Client

Description

The terraform-provider-vinyldns docs site uses docsify. It would be beneficial to users if the python client used a similar docs-as-code solution.

Start vinyldns-python client

Background
We have most of the makings of a good vinyldns python client in the functional tests at https://github.com/vinyldns/vinyldns/blob/master/modules/api/functional_test/vinyldns_python.py

With a little cleanup, this can be moved into the vinyldns-python repository and setup as a first class project.

Scope of work

• Port the vinyldns_python.py library to the vinyldns-python repository
• Clean up anything that is not-standard for python
• Add unit tests? Unclear on how to unit test this, since it is just an API wrapper for vinyldns api. There seem to be a lot of unit testing API tools out there, would like whoever starts this up to laydown some opinions on the matter. If not, this looked interesting and it works with pytest which we use for vinyldns - https://github.com/taverntesting/tavern
• Setup a release process (setup.py) to be able to build and release the project to pypi
• Add LICENSE and license headers
• Add a README on usage
• Setup travis

add support for cancel_batch_change API endpoint

API documentation: https://www.vinyldns.io/api/cancel-batchchange.html

Will be similar to the reject_batch_change method: https://github.com/vinyldns/vinyldns-python/blob/master/src/vinyldns/client.py#L666-L675

Sanitize the python client

Background
Presently, the methods on the python client do not properly handle error conditions. The in line docs for the methods also do not indicate how each method could fail. This will make it difficult for anyone to use.

Solution

• Clean up the return types for all of the methods.
• Return None on a 404
• raise an error on other error codes coming back like Forbidden, Conflict, BadRequest
• Remove unnecessary methods like color and ping and especially post_status
• Rename create_zone to connect_zone
• Rename delete_zone to abandon_zone
• Make functions that are not to be consumed private, for example build_request

get_record_set method returning key error

Looks like a parsing issue.

— Steps to Reproduce
Example:
local_client.get_record_set(zone_id,record_set_id)
get_record_set
line 198, in from_dict
zone_id=d['zoneId'],
KeyError: 'zoneId'

— Possibly reasons
Method: get_record_set
line 525 : Client.py
return RecordSet.from_dict(data) if data is not None else None
needs to be changed to
return RecordSet.from_dict(data['recordSet']) if data is not None else None

Add ownerGroupId to RecordSet update method

In support of the new shared zones feature coming to VinylDNS we need to add the optional ownerGroupId field to the RecordSet update method.

Example Request

{
  "id": "dd9c1120-0594-4e61-982e-8ddcbc8b2d21",
  "name": "already-exists",
  "type": "A",
  "ttl": 38400,
  "records": [
    {
      "address": "6.5.4.3"
    }
  ],
  "zoneId": "2467dc05-68eb-4498-a9d5-78d24bb0893c",
  "ownerGroupId": "f42385e4-5675-38c0-b42f-64105e743bfe"
}

Additional Information

• Get recordSet first then update
• If ownerGroupId is removed or set to an empty string, the attribute should not be included in the update request.
• If ownerGroupId is not modified from the retrieved recordSet it should be sent in the update.
• If ownerGroupId is modified, send the modification