How to overcome memory issues when predicting large batches of data?

[unitaryai]

How to overcome memory issues when predicting large batches of data?

[laurahanu]

Hello, you should initialise the model only once and assign it to a variable so this doesn't happen at each iteration:

model = Detoxify("original", device=device)
comments_list = comments["text"].to_list()
 df = pd.DataFrame()

 for i in range(0, len(comments_list), 32):
     comms = comments_list[i : i + 32]
     results = model.predict(comms)
     results = pd.DataFrame(results)
     df = df.append(results, ignore_index=True)

Now you should be able to use a bigger batch size as well.
Hope this helps!

[SaadAhmed433]

Thanks for the suggestion and pointing it out. The change worked pretty well infact insanely well.

Previously it was averaging around 7 mins to get all the predictions, now everything is done in about 8 seconds.

[laurahanu]

Great, glad it helped!