emcf/thepipe

Feed PDFs, URLs, Slides, YouTube, and more into Vision-Language models with one line of code⚡

thepipe's Issues

Audio transcript extraction

Looking to support mp3, wav

Audio is not standard in commercial multimodal models today in 2024. Because of this, I am also looking to transcribe audio to text, probably via Whisper.

Make docker image

Apologies there is no docker image yet! 😅
I am on the case.

file type scanning

Thoughts on a scan feature that prints file types of the directory/file selected for Piping without extracting any data? It would be clearer what file types are causing failure if something isn't supported.

`ai_extraction=True` not working locally

Hi! Not sure if this is a bug or a feature, but I'd love to use the ai_extraction option to improve the handling of PDF documents. However, enabling this option overwrites the local=True option.

MWE:

from thepipe.thepipe_api import thepipe 
source = 'example.pdf'
messages = thepipe.extract(source, local=True, verbose=True, ai_extraction=True)

Throws the error:
Failed to extract from example.pdf: No valid API key given. Visit https://thepi.pe/docs to learn more.

It works without enabling ai_extraction, but I don't like that it adds every page as an image to the messages because this massively increases the token count for longer PDFs.
As a workaround, I adapted the extract_pdf function only to extract PDF pages as images if the page contains an image. It would be great to have this as an option. (I know this approach is not optimal as it misses tables and some images containing only SVG objects; maybe a better option is possible only based on the fitz library, but I am no expert in this package).

def extract_pdf(file_path: str, ai_extraction: bool = False, text_only: bool = False, verbose: bool = False, limit: int = None) -> List[Chunk]:
    chunks = []
    if ai_extraction:
        with open(file_path, "rb") as f:
            response = requests.post(
                url=API_URL,
                files={'file': (file_path, f)},
                data={'api_key': THEPIPE_API_KEY, 'ai_extraction': ai_extraction, 'text_only': text_only, 'limit': limit}
            )
        try:
            response_json = response.json()
        except json.JSONDecodeError:
            raise ValueError(f"Our backend likely couldn't handle this request. This can happen with large content such as videos, streams, or very large files/websites. Re")
        if 'error' in response_json:
            raise ValueError(f"{response_json['error']}")
        messages = response_json['messages']
        chunks = create_chunks_from_messages(messages)
    else:
        import fitz
        # extract text and images of each page from the PDF
        with open(file_path, 'rb') as file:
            doc = fitz.open(file_path)
            for page in doc:
                text = page.get_text()
                image_list = page.get_images(full=True)
                if text_only:
                    chunks.append(Chunk(path=file_path, text=text, image=None, source_type=SourceTypes.PDF))
                elif image_list:
                    pix = page.get_pixmap()
                    img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
                    chunks.append(Chunk(path=file_path, text=text, image=img, source_type=SourceTypes.PDF))

                else: chunks.append(Chunk(path=file_path, text=text, image=None, source_type=SourceTypes.PDF))

            doc.close()
    return chunks

Error when trying to Pipe Linkedin profile

thepipe https://www.linkedin.com/in/spencer-reitsma-8a3938151/
Extracting from website...
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Python312\Scripts\thepipe.exe_main.py", line 7, in
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\thepipe.py", line 60, in main
chunks = extractor.extract_from_source(source=args.source, match=args.match, ignore=args.ignore, limit=args.limit, verbose=args.verbose, ai_extraction=args.ai_extraction, text_only=args.text_only, local=args.local)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 57, in extract_from_source
return extract_url(url=source, text_only=text_only, local=local)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 292, in extract_url
raise ValueError(f"{response['error']}")
ValueError: Page.evaluate: Execution context was destroyed, most likely because of a navigation
PS D:\Downloads\Project Templates for reference only>

Add .ino functionality for GitHub repos related to arduino

Add .ino functionality for GitHub repos related to arduino

Feature requests 🔨

Accepting requests features in this thread, please feel free to suggest!
The roadmap so far includes:

• Cloud storage extraction (Google Drive, OneDrive)
• E-Commerce platform extraction (Amazon)
• Markdown formatted extraction (PDF to Markdown, URL to Markdown, etc)

Some videos (without audio) fail to extract

error 'NoneType' object has no attribute 'write_audiofile' occuring on line video.subclip(start_time, end_time).audio.write_audiofile(audio_path, codec='pcm_s16le')

Could probably fix with a simple none check

audio = video.subclip(start_time, end_time).audio
if audio is None:
  transcription = None
else:
  audio.write_audiofile(audio_path, codec='pcm_s16le')
  ...

add syntax to match multiple patterns with match/ignore functionality.

currently ignore only accepts one file type to be ignored

No longer working after addition of THEPIPE_API_KEY

I have added the env var THEPIPE_API_KEY to my .env, .bashrc and at the commandline. It is not getting accepted.

{response['error']}")
ValueError: Valid environment variable THEPIPE_API_KEY not found. You may need to restart if you have set your API key. Visit https://thepi.pe/docs to learn more.

Running "Locally"

Multiple Questions:
What are the resources recommend/required for local extraction?

When running locally can you provide us the option to expose a port and receive POST requests? That way we can have an on prem machine that can work interchangeably with your API for client machines.

Swap Whisper Version

I was looking at your pipeline and thought you might be better served by using https://github.com/Vaibhavs10/insanely-fast-whisper or allow a bit of wiggle room in your framework to allow an optional parameter for feeding in a seperate processor for video transcription problems. This is over an order of magnitude improvement on vanilla whisper and has cpu/gpu modes. You may want to just allow a whole pipeline to be fed to futureproof this particular endpoint to new tooling

Pytesseract error when text_only is True within GitHub Action

If Tesseract OCR is not installed correctly, image extraction with text_only=True will yield tesseract is not installed or it's not in your PATH. See README file for more information.. This occurs with improper Tesseract installation, such as the case with the current GitHub actions CI setup for this repo.

Further discussion here: https://stackoverflow.com/questions/50951955/pytesseract-tesseractnotfound-error-tesseract-is-not-installed-or-its-not-i

Video frame + transcript extraction

Looking to support extraction of mp4, mov, webm, avi files as well as youtube for a Vision-Language model (not a video model)

Video and audio is not standard in commercial multimodal models today. Because of this, I am looking to transcribe audio.