universexyz/datascraper

datascraper's Issues

Universe DataScraper Architecture

Universe DataScraper Architecture

Goals of the system

The goal of the system includes scraping the ethereum blockchain for past and present NFT transfers and NFT metadata, to establish a searchable database of NFTs and their onchain activity.

Software Flows:

• Monitor New Transfer Events (~every 30 seconds)
  • Relay to services via GRPC:
    • Service thats writes transfers to pg
    • Service thats indexes elastisearch
    • Service that decodes and stores NFT content in db if not already processed (tokenURI -> fetch -> metadata)
    • Service that checks and cancels orders if any transfers have caused listings in Unvierse OrderBook to go stale
• Backfill
  • Scrape chunks of blocks starting at runtime going backwards filling transfer events until ~Jan 2017 (ERC721 EIP finalized)
    • Service thats writes transfers to pg
    • Service thats indexes elastisearch
    • Service that decodes and stores NFT content in db if not already processed (tokenURI -> fetch -> metadata)

Interfaces/API Endpoints

• GET NFT (by collection address & tokenID)
• GET NFTs (by collection address)
• GET NFTs (by user address)
• GET NFT Transfers (N most recent)
• GET NFT Transfers (by collection address & tokenID)
• GET NFT Transfers (by collection address)
• Get NFT Transfers (by user address)

Database Tables:

• Transfers:
  • NFT Contract Address
  • TokenID
  • From
  • To
  • Amount
  • Type (ERC721 or ERC1155)
  • Transaction Hash
  • logIndex
  • (Transaction Hash, NFT Contract Address, TokenID) should be unique in db
• NFTs
  • NFT Contract Address
  • TokenID
  • Owner
  • NFT Name
  • Optimized_url (image)
  • Thumbnail_url
  • Attributes JSON
• NFT Collections
  • Collection Name
  • Collection Symbol
  • Number of NFTs