identitypython/pyff

SAML metadata aggregator

python SAML metadata aggregator

This is a SAML metadata aggregator written in python. It is based on the model for metadata exchange by Ian Young: http://iay.org.uk/blog/2008/10/metadata_interc.html

Features

• Fully customizable processing pipelines in yaml.
• Easy to retrieve, analyze, transform, sign and publish SAML metadata.
• Operate in batch or online mode using embedded HTTP server.
• Provide a full MDX implementation.
• Make use of PKCS#11 tokens and HSMs for key protection.
• Fully compatible with thiss.io discovery service.
• Fully compatible with mdq-browser frontend app.

Dependencies

• pyXMLSecurity
• PyKCS11 (optional)
• pygments
• gunicorn (for the standalone pyffd server)
• ... cf requirements.txt

More information

Project homepage: https://pyff.io/

pyff's Issues

content negotiation

I don't think content neg. is working correctly. The content_type argument to request forces the response to have that Content-Type. Instead this argument should have the effect of preferring the chosen content type.

fix or remove buildout

make DS js into a (set of) jquery plugins

ERROR:root:Error fetching metadata: XML schema validation failed

Instead of a traceback the metadata source and position of the offending xml element should be reported

Feature request: get all the entityID stored in pyff's repository

It would be usefull to be able to get all the entityID stored in pyff's repository in json format. Like /entities gives the signed xml contains all entites, /entities.json would give a simple json array:
{ "https://entity.id.1.com/idp", "https://entity.id.2.com/idp", "etc..." }

bin/buildout gives error on master branch

I am unable to 'build' the latest release of pyFF. After git clone of master:

python bootstrap.py

Creating directory '/root/git/pyFF/bin'.
Creating directory '/root/git/pyFF/parts'.
Creating directory '/root/git/pyFF/eggs'.
Creating directory '/root/git/pyFF/develop-eggs'.
Generated script '/root/git/pyFF/bin/buildout'.

bin/buildout

While:
Installing.
Checking for upgrades.
Getting distribution for 'zc.buildout>=2.3.1'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/buildout.py", line 1946, in main
getattr(buildout, command)(args)
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/buildout.py", line 475, in install
self._maybe_upgrade()
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/buildout.py", line 910, in _maybe_upgrade
allow_hosts = self._allow_hosts
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/easy_install.py", line 848, in install
return installer.install(specs, working_set)
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/easy_install.py", line 635, in install
for_buildout_run=for_buildout_run):
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/easy_install.py", line 477, in _get_dist
dist, avail = self._satisfied(requirement)
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/easy_install.py", line 253, in _satisfied
if _final_version(dist.parsed_version)
File "/root/git/pyFF/eggs/zc.buildout-2.3.1-py2.7.egg/zc/buildout/easy_install.py", line 1408, in _final_version
return not parsed_version.is_prerelease
AttributeError: 'tuple' object has no attribute 'is_prerelease'

Clarify req argument for signing

Aloha!

pyFF has a lot of good comments in the code that explains what the function does. However, for the sign pipe (in builtins), the comment is somewhat confusing:

"def sign(req,*opts):
"""
:param req: The request
:param opts: Options (unused)
:return: returns the signed working document

Sign the working document. The 'key' argument references either a PKCS#11 uri or the filename containing a PEM-encoded non-password protected private RSA key. The 'cert' argument may be empty in which case the cert is looked up using the PKCS#11 token, or may point to a file containing a PEM-encoded X.509 certificate."

What you might want to add is that req is a dictionary that is expected to contain key:data pairs for key and cert. Just to make it easier to understand.

support external pipes using pkg_resources entry points

empty load directory throws error

if the configuration contains an empty directory to load files, this message is raised:
ERROR:root:'NoneType' object has no attribute 'xpath'

i18n

in DS:

• pick images and other elements based on browser language prefs
• i18n for static text elements

include sp logo in DS

Make it display nice on the mobile platform too!

disco: get html OK, json empty

I can see IDPs using the browser directly, but using the disco service the list is empty.

http://pastebin.com/tycNe5f1

md.merge does not work

Merging trees does not have the intended effect after the storage layer was implemented. The correct way to do it may be to split merge into a function that operates on the current tree/document and one that operates on the store. This gets even more complicated with the new tracking feature.

ERROR:root:addinfourl instance has no attribute 'iterlines'

Hello,

We are running pyff 0.9.3dev

We have a pipeline definition that looks like this:

Test to trigger the "addinfourl instance has no attribute 'iterlines'" error

• load:
  • http://mds.edugain.org edugain-signer.crt

    Filter test entities

• fork merge remove:
  • select:
    • http://cnb.rediris.es/simplesaml/saml2/idp/metadata.php
• publish: /tmp/all-edugain-metadata.xml

When running pyff it fails:

/opt/pyff/bin/pyff --loglevel=DEBUG test.fd

...
DEBUG:root:importing builtins from pyff.pipes to find select
DEBUG:root:string lookup http://cnb.rediris.es/simplesaml/saml2/idp/metadata.php
DEBUG:root:basic lookup http://cnb.rediris.es/simplesaml/saml2/idp/metadata.php

DEBUG:root:recursively fetching members from 'http://cnb.rediris.es/simplesaml/saml2/idp/metadata.php'

Traceback (most recent call last):
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/init.py", line 52, in main
plumbing(p).process(md, state={'batch': True, 'stats': {}})
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/pipes/init.py", line 165, in process
self._process(req)
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/pipes/init.py", line 184, in _process
ot = pipe(req, *opts)
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/pipes/builtins.py", line 135, in fork
ip._process(ireq)
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/pipes/init.py", line 184, in _process
ot = pipe(req, *opts)
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/pipes/builtins.py", line 474, in select
ot = req.md.entity_set(args, name)
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/mdrepo.py", line 667, in entity_set
for ent in self.lookup(member):
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/mdrepo.py", line 646, in lookup
l = self._lookup(member, xp)
File "/opt/pyff/lib/python2.6/site-packages/pyFF-0.9.3dev-py2.6.egg/pyff/mdrepo.py", line 607, in _lookup
return [self._lookup(line, xp) for line in urllib.urlopen(member).iterlines()]

AttributeError: addinfourl instance has no attribute 'iterlines'

ERROR:root:addinfourl instance has no attribute 'iterlines'

I guess the error happens because the edugain metadata no more includes entity http://cnb.rediris.es/simplesaml/saml2/idp/metadata.php. Should I adapt the pipeline definition to prevent pyff to die ?

CherryPy and Eventlet

Aloha!

Noticed that the builtins pipes requires CherryPy and eventlet. You might want to add them to the list of dependencies in the README.

refactor discojson.xslt as a pipe

This will probably be a significant performance boost as we will be able to avoid doing a lot of xslt in discovery. It also allows us to be smarter about generating title: and avoid having to support XML in a DS-only deployment of pyFF. We can also do proper error handling and data verification to avoid some of the non-XML-related parsing issues that have arisen in production (eg faulty image data or bad geolocation coordinates)

pyffd needs application/xml to serve a DS

it should only require a application/json support

version switch

pyffd has a version switch.
It would be nice to have the same for pyff command.

version javascript URLs

avoid caching missfires

Invalid input metadata format doesn't interrupt pipeline processing

I am running the latest pyff code (head from github).
We had an issue while running a pyff pipeline that merges two metadata file, one from eduGAIN and one from our federation registry. Our input metadata file was syntactically incorrect. Pyff detected that but however generated the output metadata with only eduGAIN SAML entity in it.

I would expect that pyff would interrupt the pipeline if one of the input source is missing, thus preventing the output metadata from being updated with incomplete data.

This issue is somehow related to #47

I had a look at the code but I am not a python developer and I'm not familiar with pyff internals. Shouldn't that MetadataException be handled in pipes::_process() subroutine?

Here is a pipeline that can be used to reproduce the problem:

Running this pipeline with pyff should fail because one of the input metadata file has an incorrect format

• load:
  • /tmp/saml-metadata-format-issue.xml
  • /var/www/html/edugain/sps-edugain-metadata.xml
• select
• finalize:
  Name: https://metadata.renater.fr/
  cacheDuration: PT1H
  validUntil: P6D
• publish: /tmp/output-metadata.xml
• stats

Here is the pyff output while running this pipeline:
DEBUG:root:Processing

• load: [/tmp/saml-metadata-format-issue.xml, /var/www/html/edugain/sps-edugain-metadata.xml]
• select
• finalize: {Name: 'https://metadata.renater.fr/', cacheDuration: PT1H, validUntil: P6D}
• {publish: /tmp/output-metadata.xml}
• stats

DEBUG:root:load parsing '/tmp/saml-metadata-format-issue.xml'
DEBUG:root:file /tmp/saml-metadata-format-issue.xml verify None as None via None
DEBUG:root:load parsing '/var/www/html/edugain/sps-edugain-metadata.xml'
DEBUG:root:file /var/www/html/edugain/sps-edugain-metadata.xml verify None as None via None
DEBUG:root:fetching (caching: True) 'file:///tmp/saml-metadata-format-issue.xml'
ERROR:root:Extra content at the end of the document, line 87, column 1
DEBUG:root:fetching (caching: True) 'file:///var/www/html/edugain/sps-edugain-metadata.xml'
ERROR:root:'file:///tmp/saml-metadata-format-issue.xml' generated an exception: no valid metadata found at 'file:///tmp/saml-metadata-format-issue.xml'
DEBUG:iso8601.iso8601:Parsed 2015-05-04T10:30:31Z into {'tz_sign': None, 'second_fraction': None, 'hour': '10', 'daydash': '04', 'tz_hour': None, 'month': None, 'timezone': 'Z', 'second': '31', 'tz_minute': None, 'year': '2015', 'separator': 'T', 'monthdash': '05', 'day': None, 'minute': '30'} with default timezone <iso8601.iso8601.Utc object at 0x7f614dd449d0>
DEBUG:iso8601.iso8601:Got '2015' for 'year' with default None
DEBUG:iso8601.iso8601:Got '05' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 5 for 'month' with default 5
DEBUG:iso8601.iso8601:Got '04' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'day' with default 4
DEBUG:iso8601.iso8601:Got '10' for 'hour' with default None
DEBUG:iso8601.iso8601:Got '30' for 'minute' with default None
DEBUG:iso8601.iso8601:Got '31' for 'second' with default None
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/ws-federation.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' oasis-200401-wss-wssecurity-secext-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' oasis-200401-wss-wssecurity-utility-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xml.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xmldsig-core-schema.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-addr.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' MetadataExchange.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' saml-schema-metadata-2.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xenc-schema.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' sstc-saml-schema-assertion-2.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-securitypolicy-1.2.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-authorization.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/cs-sstc-schema-assertion-1.1.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-saml-metadata-ui-v1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/privacy.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/shibboleth-metadata-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/shibboleth-trust-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/shibboleth.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-metadata-attr.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-saml-metadata-algsupport.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-saml-idp-discovery.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/atom.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/xrd-1.0-os.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/saml-metadata-rpi-v1.0.xsd' for 'None'
DEBUG:root:returning 916 valid entities
DEBUG:root:{'Status': 'success', 'Time Spent': '0.0 seconds', 'Last-Modified': '2015-04-28 12:31:27.419980', 'URL': None, 'Bytes': '6350212', 'Expiration Time': '2015-05-04 12:30:31.343367', 'Cached': 'False', 'Tries': '1', 'Size': '916', 'Date': '2015-04-28 12:50:01.060455', 'Description': 'Remote metadata', 'Cache TTL': '517230.0', 'Validation Errors': {}}
DEBUG:root:retrying []
DEBUG:root:selecting using args: None
DEBUG:root:calling store lookup https://federation.renater.fr/edugain/
DEBUG:root:selecting 916 entities before validation
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/ws-federation.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' oasis-200401-wss-wssecurity-secext-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' oasis-200401-wss-wssecurity-utility-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xml.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xmldsig-core-schema.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-addr.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' MetadataExchange.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' saml-schema-metadata-2.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xenc-schema.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' sstc-saml-schema-assertion-2.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-securitypolicy-1.2.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-authorization.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/cs-sstc-schema-assertion-1.1.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-saml-metadata-ui-v1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/privacy.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/shibboleth-metadata-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/shibboleth-trust-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/shibboleth.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-metadata-attr.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-saml-metadata-algsupport.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/sstc-saml-idp-discovery.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/atom.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/xrd-1.0-os.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff-latest/lib/python2.7/site-packages/pyFF-0.10.0.dev0-py2.7.egg/pyff/schema/saml-metadata-rpi-v1.0.xsd' for 'None'
DEBUG:root:publish /tmp/output-metadata.xml

DEBUG:root:output_file=/tmp/output-metadata.xml, resource_name=/tmp/output-metadata.xml

total size: 916
selected: 916
idps: 4

sps: 916

bootstrap 3.0

Unaccessible MDUI logo results in infinit fetch-loop of /role/1x1t.png

If MDUI of an SP includes a Logo tag pointing to a non-existing image, PyFF will fall back to fetching /role/1x1t.png, which doesn't exist, resulting in an infinit number of requests to /role/1x1t.png.

PyFF should probably fetch /static/img/1x1t.png or something similar instead and should abort if no valid Logo was found.

load as doesn't default to the URL

when doing load foo as bar, foo shows up as bar in the repository but when doing load foo, foo shows up as 'null'.

Allow configuration of url-path

To add pyFF as a service 'on top' of a existing website we like to use apache mod-proxy. Using mod-proxy we need to rewrite the html-code since there is already content in the docroot of the webserver.

When pyFF can serve webpages when accessed via http:////* (instead of http:///* no rewrite is necessary.

We know there are other options, but those are more complex and require more configuration.

selecting entities

select-ing an entity in the way the docs suggest in fork, "Merging":

- fork merge:
    - select: http://sp.example.com/shibboleth-sp

does not seem to work, they seem to fall though _lookup.
Using them within XPaths works, though, but is a bit verbose:

- fork merge:
    - select: "!//md:EntityDescriptor[@entityID='http://sp.example.com/shibboleth-sp']"

Finalize command fails with error 'datetime.timedelta' object has no attribute 'total_seconds'

Hi Leif,

I am evaluating pyFF as a metadata tool for our federation in the eduGAIN context.
I successfully install it and ran some simple exemples. However pyff fails if I add the finalize command to the pipeline.

Do you have a clue of what goes wrong?

I hope that's the right place to report my issue; I could find a pyFF users mailing list.

Here is my pipeline definition:

• load:
  • http://mds.edugain.org edugain-signer.crt
• select: "http://mds.edugain.org!//md:EntityDescriptor[md:IDPSSODescriptor]"
  #- xslt:

  stylesheet: tidy.xsl

• finalize:
  cacheDuration: PT5H
  validUntil: P10D
• sign:
  key: pkcs11:///usr/lib64/libeToken.so/59B496386E4DBBEF?pin=MYPIN
• publish: /tmp/edugain-idp.xml
• stats

Here is the output of pyff:
%/opt/pyff/bin/pyff --loglevel=DEBUG example2.fd
DEBUG:root:Processing

• load: ['http://mds.edugain.org edugain-signer.crt']
• {select: 'http://mds.edugain.org!//md:EntityDescriptor[md:IDPSSODescriptor]'}
• finalize: {cacheDuration: PT5H, validUntil: P10D}
• sign: {key: 'pkcs11:///usr/lib64/libeToken.so/59B496386E4DBBEF?pin=MYPIN'}
• {publish: /tmp/edugain-idp.xml}
• stats

DEBUG:root:importing builtins from pyff.pipes to find load
DEBUG:root:load http://mds.edugain.org edugain-signer.crt
DEBUG:root:remote http://mds.edugain.org edugain-signer.crt None
DEBUG:root:Starting fetcher for http://mds.edugain.org
DEBUG:root:Waiting for next thread to finish...
DEBUG:root:fetching http://mds.edugain.org without using cache
DEBUG:root:got 1314739 bytes from http://mds.edugain.org
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/ws-federation.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' oasis-200401-wss-wssecurity-secext-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' oasis-200401-wss-wssecurity-utility-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xml.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xmldsig-core-schema.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-addr.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' MetadataExchange.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' saml-schema-metadata-2.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' xenc-schema.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' sstc-saml-schema-assertion-2.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-securitypolicy-1.2.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' ws-authorization.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/cs-sstc-schema-assertion-1.1.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/sstc-saml-metadata-ui-v1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/privacy.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/shibboleth-metadata-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/shibboleth-trust-1.0.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/shibboleth.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/sstc-metadata-attr.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/sstc-saml-metadata-algsupport.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/sstc-saml-idp-discovery.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/atom.xsd' for 'None'
DEBUG:root:resolve SYSTEM URL' /opt/pyff/lib/python2.6/site-packages/pyff/schema/xrd-1.0-os.xsd' for 'None'
DEBUG:root:verifying signature using edugain-signer.crt
DEBUG:root:key size: 2048 bits
DEBUG:root:Looking for #eduGAIN using id attribute 'ID'
DEBUG:root:transform: http://www.w3.org/2000/09/xmldsig#enveloped-signature
DEBUG:root:transform: http://www.w3.org/2001/10/xml-exc-c14n
DEBUG:root:using hash algorithm sha1
DEBUG:root:digest for #eduGAIN: 9zmDg91vvpunMVQ6nX49As+ttlg=
DEBUG:root:<ds:DigestValue xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">9zmDg91vvpunMVQ6nX49As+ttlg=/ds:DigestValue

DEBUG:root:SignedInfo C14N: <ds:SignedInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">/ds:CanonicalizationMethod
<ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1">/ds:SignatureMethod
<ds:Reference URI="#eduGAIN">
ds:Transforms
<ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature">/ds:Transform
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"><ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="ds eduidmd idpdisc init md mdattr mdrpi mdui ns9 saml saml1md samla shibmd #default xsi">/ec:InclusiveNamespaces/ds:Transform
/ds:Transforms
<ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1">/ds:DigestMethod
ds:DigestValue9zmDg91vvpunMVQ6nX49As+ttlg=/ds:DigestValue
/ds:Reference
/ds:SignedInfo
DEBUG:root:SignedInfo digest: PmSazR9Zu4WCeqAEN83aw3tXwaM=
DEBUG:root:got fresh metadata (date: 2013-04-08 09:14:19)
DEBUG:root:importing builtins from pyff.pipes to find select
DEBUG:root:string lookup http://mds.edugain.org!//md:EntityDescriptor[md:IDPSSODescriptor]
DEBUG:root:selecting http://mds.edugain.org filtered by //md:EntityDescriptor[md:IDPSSODescriptor]
DEBUG:root:string lookup http://mds.edugain.org
DEBUG:root:basic lookup http://mds.edugain.org
DEBUG:root:xpath filter //md:EntityDescriptor[md:IDPSSODescriptor] <- <lxml.etree._ElementTree object at 0x16414d0>
DEBUG:root:selecting 112 entities from 1 entity set(s) before validation
DEBUG:root:importing builtins from pyff.pipes to find finalize
Traceback (most recent call last):
File "/opt/pyff/lib/python2.6/site-packages/pyff/init.py", line 44, in main
plumbing(p).process(md,state={'batch': True, 'stats': {}})
File "/opt/pyff/lib/python2.6/site-packages/pyff/pipes/init.py", line 154, in process
self._process(req)
File "/opt/pyff/lib/python2.6/site-packages/pyff/pipes/init.py", line 172, in _process
ot = pipe(req,*opts)
File "/opt/pyff/lib/python2.6/site-packages/pyff/pipes/builtins.py", line 840, in finalize
req.state['cache'] = int(offset.total_seconds() / 50)
AttributeError: 'datetime.timedelta' object has no attribute 'total_seconds'
ERROR:root:'datetime.timedelta' object has no attribute 'total_seconds'
<module 'logging' from '/usr/lib64/python2.6/logging/init.pyc'>

[('http://mds.edugain.org', 'edugain-signer.crt', None, 0)]

---

Thanks.

Certificate from SoftHSM missing from signed metadata

Possibly more of a pyXMLSecurity issue, but I'll start here: When a certificate object accessed via PKCS11 is not publicly available (i.e., pkcs11-tool -O does not show it without logging in at the same time) pyFF does not include the certificate in the signed XML. While this is technically valid this makes it hard(er) to verify such signatures since the key used to sign is unkown.
In the case of SoftHSM this is possibly a configuration error (or environmental, due to use of old software, I've only tried Debian 7 or CentOS 6 so far) but I've failed to change the "private" attribute on the cert object and would therefore appreciate a workaround on the PKCS11 client side, e.g. getting the certificate object while we still have a session via PKCS11 during signing.

The problem can be experienced with the (immensly helpful!) files p11setup.sh and p11.fd from pyFF's examples directory, but to make this easier I've created a minimal self-contained testcase over at https://github.com/peter-/pyff-issue63-testcase

MXD query path

What is the path to query an EntityDescriptor?

MXD query path

querying for an entityId results in an HTTP 404 or 500 status.

E.g. calling up http://mdx.test.portalverbund.gv.at:8081/entities; extracting the entityID https://v-portal.tirol.gv.at/IdPWeb/shibboleth from it; append it urlencoded to the path results in HTTP 500.

The log at http://mdx.test.portalverbund.gv.at:8081 says:

GET /entities/https:%2F%2Fv-portal.tirol.gv.at%2FIdPWeb%2Fshibboleth 

37.221.197.149 - - [15/Jan/2014:21:17:13] "GET /entities/https:%2F%2Fv-portal.tirol.gv.at%2FIdPWeb%2Fshibboleth HTTP/1.1" 500 4053 "" "curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
[15/Jan/2014:21:17:56]  request path: https://v-portal.tirol.gv, ext: at/IdPWeb/shibboleth, headers: {'Remote-Addr': '37.221.197.149', 'Host': 'mdx.test.portalverbund.gv.at:8081', 'Accept': 'application/samlmetadata+xml', 'User-Agent': 'curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2'}
[15/Jan/2014:21:17:56]  Processing
. . .
[15/Jan/2014:21:17:56]  string lookup https://v-portal.tirol.gv
[15/Jan/2014:21:17:56]  basic lookup https://v-portal.tirol.gv
[15/Jan/2014:21:17:56]  recursively fetching members from 'https://v-portal.tirol.gv'
[15/Jan/2014:21:17:56] HTTP Traceback (most recent call last):
  File "/var/virtualenv/pyff/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
. . . 
  File "/var/virtualenv/pyff/lib/python2.7/site-packages/pyFF-0.10.0dev-py2.7.egg/pyff/mdrepo.py", line 699, in _lookup
    return [self._lookup(line, xp) for line in urllib.urlopen(member).iterlines()]
AttributeError: addinfourl instance has no attribute 'iterlines'

37.221.197.149 - - [15/Jan/2014:21:17:56] "GET /entities/https:%2F%2Fv-portal.tirol.gv.at%2FIdPWeb%2Fshibboleth HTTP/1.1" 500 4053 "" "curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"

It appears to me that the transformation from PATHINFO to the entityID is broken, because it takes everything after the last dot as an extension, in this case at/IdPWeb/shibboleth. So it is looking for https://v-portal.tirol.gv, which is not the full entityID.

Add a browsing option for finding IdPs in the DS

logging problems

When calling pyff with --log=mylog.txt pyff throws "Invalid log level".

pyff --loglevel=INFO --log=mylog.txt pyff throws "Invalid log level".

it should be possible to have pyff stop a pipline unless all load statements succeed

This is related to #61

entitydescriptor tag cleanup

If pyff is used to build an aggregate file out of multiple small files containing single entityDescriptor elements, it seems new entityDescriptor open tags appear on the same line as the preceding entityDescriptor close tag, e.g.:

</md:EntityDescriptor><md:EntityDescriptor entityID="https://example.edu/shibboleth">

If pyff includes files with multiple entityDescriptor elements, it seems to put open tags on new lines, e.g.

</md:EntityDescriptor>
<md:EntityDescriptor entityID="https://example.edu/shibboleth">

I would like to see all entityDescriptor open tags appear on a new line. Perhaps a change to tidy.xml?

[Errno 1] Operation not permitted

This error seems to correlate with the number of entity descriptors loaded.

Debug log:
http://pastebin.com/VSfb24bc

IDP disco ?return vs. <idpdisc:DiscoveryResponse>

When the SP announces the response URI in metadata, and does not send the return parameter, mdx.py does not return the reponse.

ERROR:root:Error fetching metadata: Cached metadata expired

This error is not self explaining, and also the severity might be too high.

pyff should stop processing if one input MD is missing

We are using pyFF to agrgegate our federation metadata with eduGAIN metadata. We had an issue that made us publish the aggregate metadata with eduGAIN metadata only.

After some investigation I found out that pyff does not raise an exception if an input MD cannot be loaded. I suggest this not only adds an error log but raise an exception to prevent an uncomplete output file from being generated.

Here is a sample pipeline where one input metadata file is missing:

• load:
  • /var/www/html/edugain/sps-edugain-metadata.xml
  • /tmp/unknown-metadata.xml
  • select
  • finalize:
    Name: https://federation.renater.fr/
    cacheDuration: PT1H
    validUntil: P6D
  • publish: /tmp/test-metadata.xml
  • stats

Here is pyff output:
None
None
ERROR:root:Don't know how to load '/tmp/unknown-metadata.xml' as None verify None via None
[('file:///var/www/html/edugain/sps-edugain-metadata.xml', None, None, None)]
[('file:///var/www/html/edugain/sps-edugain-metadata.xml', None, None, 0, None)]

ERROR:root:[Errno 1] Operation not permitted

total size: 2
selected: 123
idps: 0
sps: 123

It seems that this part of builtins.py should be changed:

    else:
        log.error("Don't know how to load '%s' as %s verify %s via %s" %
                  (url, params['as'], params['verify'], params['via']))

ERROR:root:5959715988794703976924350164109075903676969800262701808439137620588727237252180621228689184824-octet short

When I try to sign I receive the message cited in the title.

Details:

$ pyff --loglevel=INFO pvAt.fd
[('http://metadata.portalverbund.at//testfed-metadata-unsigned.xml', None, None, 0)]
ERROR:root:5959715988794703976924350164109075903676969800262701808439137620588727237252180621228689184824-octet short

pvAt.fd

• load:
  • http://metadata.portalverbund.at//testfed-metadata-unsigned.xml
• select
• store:
  directory: /tmp/
• sign:
  key: /Users/admin/devl/pycharm/pyFF.local/keys/metadataPortalverbundAt-key.pem
  cert: /Users/admin/devl/pycharm/pyFF.local/keys/metadataPortalverbundAt-cer.pem
• publish: /Users/admin/devl/pycharm/pyFF.local/out/testPvAt-signed.xml

parse_metadata should be able to filter invalid EntityDescriptors

instead of killing the full aggregate on error

optionally turn the search box into a dropdown list on mobile

Tilde shortcut for home dir not expanded

When a fd file contains a path like ~/out pyFF claims that it cannot find the path.

Either the tilde should be expanded or a correct message be thrown

the DS IdP list always displays the full IdP list

generated metadata validity timestamp misses "Z" timezone suffix

I am executing

TZ=UTC ./pyff --loglevel=INFO ../etc/edugain-sp.fd

and get a resulting file with

validUntil="2015-10-18T05:58:27.063864"

but the software consuming this dataset complains that there needs to be a trailing "Z" to indicate UTC time zone.

Other users of pyFF seem to get the timezone info. Maybe it's just something about my python environment (in which case it would be worth documenting it in pyFF's installation instructions). I have python 2.7.8 on openSUSE 13.2.

refactor fetch_metadata using workerpool

certreport problem

described in
https://groups.google.com/forum/#!msg/pyff-users/wU_-am41_94/2IcqOKpai14J
it seems that xmlssec is a culprit

Use servicename+description before defaulting to organization name in discojson.xslt

The display-names for SPs are very confusing today.

MDX response not signed by pyffd

When requesting an entity descriptor from MDX the signature is missing.

I am using a 1:1 copy of the config provided in example 5, except modified file paths. There is nothing in the log file that looks suspicious.

config and log: http://pastebin.com/yC7fCxyM

support for REFEDS hide-from-discovery

Entities carrying the http://macedir.org/entity-category entity attibute with the http://refeds.org/category/hide-from-discovery attribute value should not be shown in pyFF's DS.
(At least not unless some other UI element is added to allow subjects to also see any hidden entities, which would probably add too much noise to the UI.)

Redirects twice when sending discovery response back to the SP

After selecting an IdP, using the "Proceed to Login" or "Login and Use Forever->Use this time only" option results in two redirects with the discovery response back to the SP when using Firefox (43.0.4).

Using "Login and Use Forever->Yes I am Sure - Use Forever" only issues one redirect.

support alternative entrypoint in standalone mode

For instance when using "load via" you have to be able to reference a separate pipeline. This is impossible today because pyff in standalone mode always runs the "root" pipeline. Changing the default would be inappropriate as it would break backwards compatibility but it would be simple to create an option to specify the pipeline to run.