timsankara/arxivpaperembeddings

Flask Application for ArXiv and OpenAI API Integration

This professional-grade Flask application provides RESTful endpoints to interact with arXiv papers, generate embeddings using OpenAI's API, and store these embeddings in AWS DynamoDB. It is designed for ease of use, extensibility, and to serve as a foundational example for further development.

Table of Contents

• Prerequisites
• Installation
• Configuration
• Running the Application
• API Endpoints
  • Test Endpoint
  • Get ArXiv Papers
  • Create Embedding from Body
• ArxivAPI Class
  • Initialization
  • Fetching Data
  • Parsing Data
  • Displaying Papers
  • Running the Process
• Notes
• Contributing
• License

Prerequisites

• Python 3.7+
• Flask
• boto3
• openai
• requests
• python-dotenv

Installation

1. Clone the repository:

   git clone <repository-url>
   cd <repository-directory>

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows use `venv\Scripts\activate`

3. Install the dependencies:

   pip install -r requirements.txt

Configuration

1. Create a .env file in the root directory of the project and add the following environment variables:

   OPENAI_API_KEY=your_openai_api_key
   AWS_ACCESS_KEY_ID=your_aws_access_key_id
   AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key

2. Ensure that the TABLE_NAME variable in the app.py file is set to the name of your DynamoDB table:

   TABLE_NAME = 'EmbeddingsTable'  # Change to your DynamoDB table name

Running the Application

To run the Flask application, use the following command:

python index.py