meltanolabs/map-gpt-embeddings

A Singer tap to build vectorized datasets from any knowledge base,using natural language processing (NLP).

map-gpt-embeddings's Issues

bug: batches <50 dont get processed because the batch size is never reached

Keep track of records split index

Today the mapper will split the record text into sub records https://github.com/MeltanoLabs/map-gpt-embeddings/blob/2155fec353868ef7b6334a8c88a3895130440272/map_gpt_embeddings/mappers.py#L129C13-L129C25 if its too big to be embedded on its own. This means a record that enters the mapper with ID = 1 could exit the mapper as 2 records both with ID = 1. I suspect this has downstream implications if the target is updating based on ID, the second record would win and the first would be clobbered.

Potential Solution

We could keep the split record index and append it to the key_properties so that the output is a composite PK. For example:

• input of record ID = 1
• mapper splits it into 2 smaller chunks
• record ID = 1 with embedding_index = 1, along with record ID = 1 with embedding_index = 2
• when the target receives these 2 distinct records it will know to concatenate the key properties into 1:1 and 1:2 so both records persist

Confirm naming of the mapper

Initially was map-openai-doc-splitter and map-openai-embeddings. The second of these two can perform both functions, so it probably makes sense to just combine them and have a config option to enable/disable the embeddings operations.

Technically, the langchain module is used for doc splitting, not openai.

While the embeddings do go through OpenAI library directly, the langchain libraries also have methods to wrap this process.

Options considered:

• map-langchain
• map-langchain-transforms
• map-langchain-embeddings
• map-chatgpt-transforms
• map-chatgpt-prep
• map-chatgpt-embeddings
• map-llm-transforms
• map-llm-embeddings
• map-llm-document-prep