High memory usage when scraping lot of queries

[admbyz]

High memory usage when scraping lot of queries

[admbyz]

@lexciobotariu
i am running 30 queries per instance with 1 core (-c 1) this should solve your problem if speed is not important.
This is the script that i'm using which search "keywords.txt" and generates keywords01.txt to keywordsn.txt for every 30 queries and finally create run.sh for running scraper synchronously.

make a build.sh and copy the code below and make the file executable (chmod +x build.sh) and run it (./build.sh or bash build.sh) then run the created run.sh file.

#!/bin/bash

create_keywords_file() {
    local file_prefix="$1"
    local start_line=$2
    local end_line=$3

    sed -n "${start_line},${end_line}p" keywords.txt > "${file_prefix}.txt"
    echo "./google-maps-scraper -input \"${file_prefix}.txt\" -results \"${file_prefix}.csv\" -exit-on-inactivity 3m -c 1 -depth 14 &" >> run.sh
    echo "${file_prefix}=\$!" >> run.sh
    echo "wait \$${file_prefix}" >> run.sh
}

main() {
    if [ ! -f keywords.txt ]; then
        echo "Error: keywords.txt not found!"
        exit 1
    fi

    total_lines=$(wc -l < keywords.txt)
    lines_per_file=30
    num_files=$((total_lines / lines_per_file))
    remainder=$((total_lines % lines_per_file))

    for ((i=1; i<=num_files; i++)); do
        start_line=$(( (i - 1) * lines_per_file + 1 ))
        end_line=$(( i * lines_per_file ))
        file_prefix="keywords$(printf "%02d" $i)"
        create_keywords_file "$file_prefix" "$start_line" "$end_line"
        echo "sleep 30" >> run.sh
    done

    if [ "$remainder" -gt 0 ]; then
        start_line=$((num_files * lines_per_file + 1))
        end_line=$((total_lines))
        file_prefix="keywords$(printf "%02d" $((num_files + 1)))"
        create_keywords_file "$file_prefix" "$start_line" "$end_line"
        echo "sleep 30" >> run.sh
    fi

    chmod +x run.sh
    echo "run.sh file created successfully!"
}

main

@gosom
i am using latest golang on both platform 1.22.1

Didn't used docker and i am off for 3 days after that i will test and comment here again.

[gosom]

The latest release (v1.2.1) has perfomance enhancements and memory usage looks to be stable.

@admbyz can you try this one ?

PS. I have only tested in fedora linux but looks good

[admbyz]

Sure i will try without splitting my keywords and let you know.
Thanks for the bump.

PS :
@gosom I tried with -c 1 for a while and it seems problem is gone because now it ends running instances correctly didnt saw any instance more than 250mb so i ended that session and now running with -c 8 with 973 keywords. I'll update post when its done.

Problem is gone tested on xubuntu 22.04 with latest updates and compiled scraper with go 1.22.2.
973 keywords with -c 8 depth 14. Scraper finished its job successfully. No memory leaks and collected 38mb worth of data.

goodjob @gosom thank you for the effort!

[lexciobotariu]

Hi @admbyz, I have the same problem with 10 cores and 32GB RAM, I was doing 10 queries at the time and run it manually again and again with new queries.
Would you mind sharing the bash file?
Thanks.

[gosom]

@admbyz which go version you use?

[gosom]

Do you have the same issue when you run using a docker container?

[OSZII]

@gosom any updates, my 32 GB ram aren't enough?

I'm using go 1.22.1 on Ubuntu 22

this should be enough to go way beyond 32GB:

Friseur München, deutschland
Restaurant München, deutschland
Dolmetscher München, deutschland
Tischler München, deutschland
Maler München, deutschland
Sanitär Installateur München, deutschland
Heizungsbauer München, deutschland
Schlosser München, deutschland
Elektriker München, deutschland
Fliesenleger München, deutschland
Zimmermann München, deutschland
Glaser München, deutschland
Dachdecker München, deutschland
Maurer München, deutschland
Metallbauer München, deutschland
Steinmetz München, deutschland
Schreiner München, deutschland
Installateur für Heizung, Lüftung und Sanitär München, deutschland
Bodenleger München, deutschland
Stuckateur München, deutschland
Kaminbauer München, deutschland
Ofenbauer München, deutschland
Parkettleger München, deutschland
Raumausstatter München, deutschland
Bautischler München, deutschland
Restaurator München, deutschland
Bootsbauer München, deutschland
Uhrmacher München, deutschland
Goldschmied München, deutschland
Silberschmied München, deutschland
Graveur München, deutschland
Uhrmacher München, deutschland
Modellbauer München, deutschland
Drechsler München, deutschland
Holzbildhauer München, deutschland
Kunstschmied München, deutschland
Sattler München, deutschland
Tapezierer München, deutschland
Polsterer München, deutschland
Schuhmacher München, deutschland
Immobilienmakler München, deutschland
Reisebüro München, deutschland
Blumenladen München, deutschland
Buchhandlung München, deutschland
Autowerkstatt München, deutschland
Elektronikgeschäft München, deutschland
Schuhgeschäft München, deutschland
Optiker München, deutschland
Fahrradladen München, deutschland
Goldschmied München, deutschland
Juwelier München, deutschland
Tattoostudio München, deutschland
Fotostudio München, deutschland
Hochzeitsphotograph München, deutschland
Rechtsanwaltskanzlei München, deutschland
Steuerberater München, deutschland
Architekturbüro München, deutschland
Innenarchitekt München, deutschland
Restaurant München, deutschland
Café München, deutschland

[arceushui]

#7
I have attached a memory graph in this issue, can help to check it? It seems like playwright didn't close correctly

[admbyz]

@gosom
I tried docker from win10 pc with -c 1 -depth 14 but memory usage was much more because of wsl and got error much faster because docker + wsl started with like 8gb ram usage.

[OSZII]

@gosom Works fine for me too! Thanks Mate!