Invalid XML in generated reports

[wolfgangkarall]

Invalid XML in generated reports

[marcbradshaw]

Note: The Database schema (for mysql at least) specifies 'CHARACTER SET ascii', so this will need to be updated to handle the storage of UTF-8 in reports.
rfc7489 specifies that domains must be converted to a-label form, but is ambiguous regarding the remaining data in the report.
A quick fix may be to convert everything to ascii before saving the report, but this is likely to break (or at least not fix, because they are likely already broken) EAI addresses.

[msimerson]

A quick fix may be to convert everything to ascii before saving the report

Sounds like the right choice, based on my read of RFC 8616.

but this is likely to break (or at least not fix, because they are likely already broken) EAI addresses.

True, but will it matter? New reports will be saved with the new converted a-label form, which should fix all future reports, and solve this issue, right?

RFC 8616, Section 6

DMARC and Internationalized Mail

   DMARC RFC7489 defines a policy language that domain owners can
   specify for the domain of the address in an RFC5322.From header
   field.

   Section 6.6.1 of RFC7489 specifies, somewhat imprecisely, how IDNs
   in the RFC5322.From address domain are to be handled.  That section
   is updated to say that all U-labels in the domain are converted to
   A-labels before further processing.  Section 7.1 of RFC7489 is
   similarly updated to say that all U-labels in domains being handled
   are converted to A-labels before further processing.

   DMARC policy records, described in Sections 6.3 and 7.1 of RFC7489,
   can contain email addresses in the "rua" and "ruf" tags.  Since a
   policy record can be used for both internationalized and conventional
   mail, those addresses still have to be conventional addresses, not
   internationalized addresses.  DMARC and Internationalized Mail
   DMARC RFC7489 defines a policy language that domain owners can
   specify for the domain of the address in an RFC5322.From header
   field.

   Section 6.6.1 of RFC7489 specifies, somewhat imprecisely, how IDNs
   in the RFC5322.From address domain are to be handled.  That section
   is updated to say that all U-labels in the domain are converted to
   A-labels before further processing.  Section 7.1 of RFC7489 is
   similarly updated to say that all U-labels in domains being handled
   are converted to A-labels before further processing.

   DMARC policy records, described in Sections 6.3 and 7.1 of RFC7489,
   can contain email addresses in the "rua" and "ruf" tags.  Since a
   policy record can be used for both internationalized and conventional
   mail, those addresses still have to be conventional addresses, not
   internationalized addresses.

[msimerson]

Because the data column that stored domains and author info were explicitly declared as ASCII, I think (limited testing) that mysql would have converted any unicode characters to a ? character. Near as I can tell, that original character data is lost. If I'm wrong and MySQL stored the character code correctly, then the changes below will automatically do The Right Thing.

Now that MySQL 8 is the minimum supported version, changing the schema to enable UTF-8 chars is no longer messy and fraught with pitfalls.

The SQL code shown on the mysql wiki page should do the needful.