Data Lineage endpoint and features

[miha42-github]

Data Lineage endpoint and features

[miha42-github]

This has been implemented in a simpler fashion than was documented above. Data lineage is now a part of every endpoint's operation. This enables the users of the service to understand for that endpoint where the data has come from and ideally which key python module is used to report on the data.

Here is an example with the keys for data and dependencies being the important points below:

{
  "code": 200,
  "message": "Wikipedia data and EDGAR has been detected and merged for the company [International Business Machines Corporation].",
  "module": "Query-> merge_data",
  "data": {
    "name": "INTERNATIONAL BUSINESS MACHINES CORP",
    "cik": "51143",
    "sic": "3570",
    "sicDescription": "Computer & office Equipment",
    "tickers": [
      "NYSE",
      "IBM"
    ],
    "exchanges": [
      "NYSE"
    ],
    "ein": "130871985",
    "description": "The International Business Machines Corporation ( IBM ), nicknamed Big Blue , is an American multinational technology corporation headquartered in Armonk, New York and present in over 175 countries. It specializes in computer hardware, middleware, and software, and provides hosting and consulting services in areas ranging from mainframe computers to nanotechnology. IBM is the largest industrial research organization in the world, with 19 research facilities across a dozen countries, and has held the record for most annual U.S. patents generated by a business for 29 consecutive years from 1993 to 2021.  IBM was founded in 1911 as the Computing-Tabulating-Recording Company (CTR), a holding company of manufacturers of record-keeping and measuring systems. It was renamed "International Business Machines" in 1924 and soon became the leading manufacturer of punch-card tabulating systems. For the next several decades, IBM would become an industry leader in several emerging technologies, including electric typewriters, electromechanical calculators, and personal computers. During the 1960s and 1970s, the IBM mainframe, exemplified by the System/360, was the dominant computing platform, and the company produced 80 percent of computers in the U.S. and 70 percent of computers worldwide.  After pioneering the multipurpose microcomputer in the 1980s, which set the standard for personal computers, IBM began losing its market dominance to emerging competitors. Beginning in the 1990s, the company began downsizing its operations and divesting from commodity production, most notably selling its personal computer division to the Lenovo Group in 2005. IBM has since concentrated on computer services, software, supercomputers, and scientific research. Since 2000, its supercomputers have consistently ranked among the most powerful in the world, and in 2001 it became the first company to generate more than 3,000 patents in one year, beating this record in 2008 with over 4,000 patents. As of 2022, the company held 150,000 patents.  As one of the world's oldest and largest technology companies, IBM has been responsible for several technological innovations, including the automated teller machine (ATM), dynamic random-access memory (DRAM), the floppy disk, the hard disk drive, the magnetic stripe card, the relational database, the SQL programming language, and the UPC barcode.The company has made inroads in advanced computer chips, quantum computing, artificial intelligence, and data infrastructure. IBM employees or alumni have won various recognitions for their scientific research and inventions, including six Nobel Prizes and six Turing Awards.  IBM is a publicly traded company and one of 30 companies in the Dow Jones Industrial Average. It is among the world's largest employers, with over 297,900 employees worldwide in 2022. Despite its relative decline within the technology sector, IBM is the seventh largest technology company by revenue, and 49th largest overall, according to _Fortune._ It is also consistently ranked among the world's most recognizable, valuable, and admired brands, with devoted following among many tech enthusiasts and consumers.",
    "website": [
      "https://www.ibm.com/",
      "https://www.ibm.com/uk-en",
      "https://www.ibm.com/us-en/",
      "https://www.ibm.com/de-de/"
    ],
    "category": "Large accelerated filer",
    "fiscalYearEnd": "1231",
    "stateOfIncorporation": "NY",
    "phone": "9144991900",
    "entityType": "operating",
    "companyFactsURL": "https://data.sec.gov/api/xbrl/companyfacts/CIK0000051143.json",
    "firmographicsURL": "https://data.sec.gov/submissions/CIK0000051143.json",
    "filingsURL": "https://www.sec.gov/cgi-bin/browse-edgar?CIK=51143&action=getcompany",
    "transactionsByIssuer": "https://www.sec.gov/cgi-bin/own-disp?action=getissuer&CIK=51143",
    "transactionsByOwner": "https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=51143",
    "city": "ARMONK",
    "stateProvince": "NY",
    "zipPostal": "10504",
    "address": "1 NEW ORCHARD RD",
    "industryGroup": "357",
    "industryGroupDescription": "Computer And Office Equipment",
    "majorGroup": "35",
    "majorGroupDescription": "Industrial And Commercial Machinery And Computer Equipment",
    "division": "D",
    "divisionDescription": "Manufacturing",
    "forms": {
      "2018-2-27-000104746918001117": {
        "filingIndex": "https://www.sec.gov/Archives/edgar/data/51143/000104746918001117/0001047469-18-001117-index.html",
        "formType": "10-K"
      }
    },
    "wikipediaURL": "https://en.wikipedia.org/wiki/IBM",
    "type": "Public company",
    "industry": [
      "software industry",
      "computer hardware",
      "IT service management",
      "information technology consulting",
      "information technology"
    ],
    "country": "United States of America",
    "isin": "US4592001014",
    "longitude": -73.7203574803287,
    "latitude": 41.113410466869084,
    "googleMaps": "https://www.google.com/maps/place/1%20New%20Orchard%20Rd%2C%20Armonk%2C%20New%20York%2C%2010504",
    "googleNews": "https://news.google.com/search?q=INTERNATIONAL%20BUSINESS%20MACHINES%20CORP",
    "googlePatents": "https://patents.google.com/?assignee=INTERNATIONAL%20BUSINESS%20MACHINES%20CORP",
    "googleFinance": "https://www.google.com/finance/quote/IBM:NYSE"
  },
  "dependencies": {
    "modules": {
      "edgar": "https://github.com/miha42-github/company_dns",
      "wikipedia": "https://github.com/miha42-github/company_dns",
      "wptools": "https://pypi.org/project/wptools/",
      "geopy": "https://pypi.org/project/geopy/"
    },
    "data": {
      "sicData": "https://github.com/miha42-github/sic4-list",
      "oshaSICQuery": "https://www.osha.gov/data/sic-search",
      "wikiData": "https://www.wikidata.org/wiki/Wikidata:Data_access"
    }
  }
}