ekgf/data-product

Artifacts of the EKGF Data Product Workgroup

data-product's Issues

DPROD Examples

Example of a Data Product with Equity Trades. Contains two datasets, one for trades in LSEG and one in Euronext

"@context": "https://ekgf.github.io/data-product-spec/dprod.jsonld",
"id":  "equity trade-xxx",
"@id": "https://y.com/products/equity-trade-xxx",
"@type": "DataProduct",
"title": "Equity Trade XXX",
"description": "Trade data defining the outcome of equity trades between parties in different stock markets, where the terms are primarily reflected in the tradable product. Additionally, Trade includes attributes such as the trade date, transacting parties, and settlement terms. Some attributes, such as the parties, are already defined in the Party Product and are simply referenced in Trade",
"dataProductOwner": "https://www.schema.xxx/person/AnnTaylor",
"lifecycle" : "Consume",
"outputPort": {
  "@type": "dcat:DataService",
  "id":  "equity-trade-euronext-xxx-tabular-adls-prod",
  "@id": "https://y.com/service/equity-trade-euronext-xxx-adls-prod-1",
  "dcat:endpointURL": "abfss://[email protected]/demo/full/trade-euronext",
  "dcat:endpointDescription": "Details for accessing storage account",
  "isAccessServiceOf": {
    "@id": "https://y.com/service/equity-trade-euronext-xxx-tabular",
    "id":  "equity-trade-euronext-xxx-tabular",
    "@type": "dcat:Distribution",
    "dcterms:format": "application/parquet",
     "dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
    "isDistributionOf": {
      "@type": "dcat:Dataset",
      "@id": "https://y.com/dataset/equity-trade-euronext-paris,
      "datasetOwner": "https://www.schema.xxx/person/JohnBarks",
      "title":  "Equity Trade Euronext Paris XXX",
      "id":  "equity-trade-euronext-paris-xxx",
      "dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
        } 
    }
  }
 "outputPort": {
   "@type": "dcat:DataService",
  "id":  "equity-trade-lseg-xxx-tabular-adls-prod",
  "@id": "https://y.com/service/equity-trade-lseg-xxx-adls-prod-1",
  "dcat:endpointURL": "abfss://[email protected]/demo/full/trade-lseg",
  "dcat:endpointDescription": "Details for accessing storage account",
  "isAccessServiceOf": {
    "@id": "https://y.com/service/equity-trade-lseg-xxx-tabular",
    "id":  "equity-trade-lseg-xxx-tabular",
    "@type": "dcat:Distribution",
    "dcterms:format": "application/parquet",
     "dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
    "isDistributionOf": {
      "@type": "dcat:Dataset",
      "@id": "https://y.com/dataset/equity-trade-lseg-xxx,
      "title":  "Equity Trade LSEG XXX",
      "id":  "equity-trade-lseg-xxx",
      "dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
        } 
    }
  }
}

Example of a Data Product providing sba pool rates either through an api or a database connection

  "@context": "https://ekgf.github.io/data-product-spec/dprod.jsonld",
  "id":  "sba-pool-rates",
  "@id": "https://y.com/products/sba-pool-rates",
  "@type": "DataProduct",
  "title": "SBA Pool Rates",
  "description": "Rates for SBE Pool that are MortgageBackedSecurities ",
  "dataProductOwner": "https://www.schema.xxx/person/johnSmith",
  "lifecycle" : "Consume",
  "outputPort": {
    "@type": "dcat:DataService",
    "id":  "sba-pool-tabular-prod1",
    "@id": "https://y.com/service/sba-pool-rates-tabular-prod1",
    "dcat:endpointURL": "jdbc:oracle:[email protected]:43534/PGPERG.WORLD",
    "isAccessServiceOf": {
      "@id": "https://y.com/distribution/sba-pool-tabular",
      "id":  "sba-pool-tabular",
      "@type": "dcat:Distribution",
      "dcterms:format": "https://www.iana.org/assignments/media-types/application/sql",
      "table": [
          "http://server/xxxx/database/yyyy/schema/zzzz/table/sba-pool",
          "http://server/xxxx/database/yyyy/schema/zzzz/table/sba-pool-identfier",  
          ]
      "isDistributionOf": {
        "@type": "dcat:Dataset",
        "@id": "https://y.com/dataset/sba-pool,
        "id":  "sba-pool",
        "dcat:conformsTo": "https://spec.edmcouncil.org/fibo/ontology/SEC/Debt/MortgageBackedSecurities/SBA-Pool"
      }
    }
   }
   "outputPort": {
     "@type": "dcat:DataService",
     "id":  "sba-pool-rates-json-prod1",
     "@id": "https://y.com/service/sba-pool-rates-json-prod1",
     "dcat:endpointURL": "http://y.com/api/sbl-rates-us",
     "isAccessServiceOf": {
       "@id": "https://y.com/distribution/sba-pool-rates-json",
       "id":  "sba-pool-rates-json",
       "@type": "dcat:Distribution",
       "dcterms:format": "https://www.iana.org/assignments/media-types/application/json",
       "conformsTo": [
             "http://confluent-registry-y/rates-json-schema.json,  
           ]
       "isDistributionOf": {
         "@type": "dcat:Dataset",
         "@id": "https://y.com/dataset/sba-pool-rates",
         "id":  "sba-pool-rates",
         "geographicalCoverage" :"https://y.com/country/US"
         "dcat:conformsTo": "https://spec.edmcouncil.org/fibo/ontology/SEC/Debt/MortgageBackedSecurities/SBA-Pool"
       }
     }
    }
}

** Note **
Representation of the different physical models in the distributions is not defined yet from the DPROD model. This work will be coordinated according to the odrl market licencing requirements

define dprod: prefix in namespace section

Maybe I overlooked it but would have expected that the dprod: itself is defined in the namspaces section. like: https://www.w3.org/TR/vocab-dcat-3/#namespaces

Comments about DPROD 11/24/2023

Please don't take offense. My Black Friday search wasn't successful yet and so I need to vent a little bit.

Writing and commenting as I read. Especially, this may mean that I comment on something that you define much further down without a forward reference. That is somewhat intentional.

DCAT is a catalog. DPROD is for products listed in the catalog. Why would one be a profile/extension of the other?

You use the term “Semantic Data…” What is the semantic of “semantic” in this context?

As pointed out elsewhere: Why emphasize or limit this for “Mesh.” It is a cool word and very popular. But what exactly makes products only suitable in a Mesh? Imagine a data-product that lists the geo-coordinates of fire hydrants around the country. Couldn’t I just download the file and move on? What would be meshy or semantic about that?

I disagree with your notion of shift left/right. The data-owner (maker) should also own the meta-data. Granted, all publishers cannot and should not handle data-integration. But they should handle explaining what it is they are offering. Likewise, I really don’t want a central team to manage my ontology. They are just going to make a mess of it. It is my ontology.

“...and ontological classes…” what does this mean?

“... input and output ports…” ports is undefined

If memory serves, in DCAT a data service is a means to access the dataset described in the catalog. Are you suggesting that data services are extended to describe meta-data?

“...semantic meaning…” This basically says: “meaning meaning.” You sure that is what you mean? ;-)

“...This allows for…” Of course it allows. Its absence also would not prohibit it. It just might make it less convenient.

“...semantics ensure that all stakeholders have a common understanding…” You ought to be really careful with such statements. I have seen ontologies and that didn’t mean that they were useful or supported understanding, especially not by humans. Imagine I wrote you an ontology and obfuscated all names and IRIs, replaced them with UUIDs and eliminated all regular comments and explanations.The formal structure and hence implied formal semantics would remain the same. Good luck emailing one this ontology, say as an RDF/Turtle file and expecting anybody to find that it creates a “common understanding.”

“...fundamental idea behind a Data Mesh…” Colloquialism. Intuitive but neither semantic nor appropriate. If you want to explain what a data mesh is supposed to be, put a reference there assuming that something even half normative exists.

“...can be programmatically understood…” will there be LLMs involved? Or did you mean “...can be verified against something and interpreted programmatically…”?

“...ensure that these products can interact…” are products “active” in the sense that they do something? Like “interact?”

“DPROD maps to the Data Mesh notion of a port to the DCAT notion of a DataService, so we can declare a DataProduct by and specify and input and output ports and these ports are Dataservice.” Could not parse.

DCAT states that a data service is defined as: “A collection of operations that provides access to one or more datasets or data processing functions.” You might make it very clear that you are changing that definition. Is it really just that: One inputPort and one OutputPort? Nothing else?

“A data product is a rational, managed, and governed collection of data, with purpose, value and ownership, meeting consumer needs over a planned life-cycle.”

How do you define and measure “rational?” How do you assign and measure “value?” How is value expressed? In US$? I suppose I could write more here. I’ll leave it as: You don’t define what a data product is. You list a bunch of cool, marketable attributes.

I’ll stop here because the rest seems unfinished.Maybe a parting thought: Reading all of this, I still don’t get at all how I would understand (you know, the semantic bit) what a specific dataproduct (like listOfFireHydrants) would be described and how I would use that. I get that somehow a port is involved. Now what?

Footnote: You mention odrl but don’t seem to use/reference it elsewhere.

Include a terminal state for the Data Product

In the 16th January version, the lifecycle status of the Data Product is defined to be: Ideation, Design, Build, Deploy, Consume.

The CDMC Designing and Managing Data Products document includes a status for an obsolete Data Product - 'Retire'.

CDMC+ Data Products Comments