The Social Marketplace Automotive Risk Engine (SMARE) Project. This project aims to detect supicious listings and potential instances of insurance fraud posted on the most popular social marketplace sites, such as Craigslist and Facebook Marketplace. In partnership between Statefarm and UTA CSE Senior Design. View the live deployment at smare.lryanle.com.
- frontend
- Next: A React framework for building web applications with server-side rendering.
- Typescript: Typed superset of JavaScript that compiles to plain JavaScript.
- shadcn/ui: A UI library for React, built using Tailwind CSS.
- Tailwind: Utility-first CSS framework for rapidly building custom designs.
- Prisma: Next-generation ORM for Node.js and TypeScript.
- Nextauth: Authentication for Next.js.
- Framer Motion: A React library to power animations.
- Lucide: Open-source icon library.
- Rechart: A composable charting library built on React components.
- Remark: A Markdown processor powered by plugins part of the unified collective.
- SWR: React Hooks library for data fetching.
- zod: TypeScript-first schema validation with static type inference.
- backend
- Selenium: A suite of tools for automating web browsers.
- BS4 (Beautiful Soup): A Python library for pulling data out of HTML and XML files.
- Pymongo: Python driver for MongoDB.
- Pandas: A fast, powerful, flexible, and easy-to-use open-source data analysis and manipulation tool.
- Imblearn: Python library to tackle the problem of imbalanced datasets.
- Loguru: A Python logging library that aims to make life easier for developers.
- OpenAI API: A generative AI API.
- frontend: Contains the frontend application with various components and settings.
- frontend/app: The main page for the client-/customer-facing portion of our SaaS. Main feature is a dashboard to present information retreived from our pipeline.
- frontend/app/api: Our middleware to retrieve, process, and present information from our DB to the web application.
- backend: Houses backend logic including cleaners, models, scrapers, and utilities.
- backend/src/cleaners: Contains all functionality related to cleaning data in the backend pipeline.
- backend/src/models: Contains our 6 models to score and help flag social marketplace listings that are suspicious.
- backend/src/cleaners: Retrieves data from various social media marketplaces and places them in our data pipeline flow.
- backend/src/utilities: Utilities for all backend-related processes such as logging and our custom DB adapter.
- documentation: Contains project documentation including sprint reports and charters.
Make a copy of the .env.example file and make the following changes.
-
remove
.examplefrom the extension -
Paste the username and password provided in MongoDB Atlas (if you should have access but do not, please contact @waseem-polus)
-
Paste the connection URL provided provided in MongoDB Atlas. Include the password and username fields using
${VARIABLE}syntax to embed the value of the variable
Prerequisites
- python3
- pipenv
Installing dependencies
Navigate to scrapers/ and open the virtual environment using
pipenv shellThen install dependencies using
pipenv installScraper Usage
To create build a production-ready Docker Image use
pipenv run buildTo create build a development Docker Image use
pipenv run devIf there is an existing smarecontainer, run the following:
pipenv run stopTo run a docker container "smarecontainer" use (Note: delete any containers with the same name before running)
pipenv run contthen
# Scrape Craigsist homepage
pipenv run craigslist
# Scrape Facebook Marketplace homepage
pipenv run facebook
Waseem Polus |
Ryan Lahlou |
Temitayo Aderounmu |
Athiya Manoj |
Yeabsra Gebremeskel |
This project is licensed under the MIT License - see the MIT License file for details.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent