Support for HDFS files

[mrunesson]

Support for HDFS files

[mrunesson]

Plan to handle tag files is to add a command line option --hdfs to tags_to_atlas command. This means program will for each table defined in table_tags.csv look up its storage location and tag that storage location.

Second step is extend command rules_to_ranger to be able to convert hive resources in ranger_policy.json to hdfs resources with similar access rules. This will only be done on commands in the ranger_policy.json file with block:

"options": {
        "expandHiveResourceToHdfs": true,
        "hdfsService": "hadoop"
}

For tag based rules, intention is to require user to extend the policy block to also cover hdfs.

Note: Row filtering and masking will still not work on jobs reading from HDFS.

[mrunesson]

Released in v1.1.0