Giter Site home page Giter Site logo

monkekode / store-crawler-google-places Goto Github PK

View Code? Open in Web Editor NEW

This project forked from davidjohnbarton/crawler-google-places

0.0 0.0 0.0 5.68 MB

Scrape information about places from Google Maps. Gives you extra information that you can't get using the Google Places API.

Home Page: https://www.apify.com/drobnikj/crawler-google-places

TypeScript 98.90% Dockerfile 1.10%

store-crawler-google-places's Introduction

What is Google Maps Scraper what does it do?

This Google Maps scraper lets you get more and faster data from Google Places than the official Google Places API. Our unofficial Google Maps API enables you to extract all of the following data from Google Maps:

🔗 Title, subtitle, category, place ID, and URL

📍 Address, location, plus code and exact coordinates

☎️ Phone and website, if available

🏷 Menu and price, if available

🔒 Temporarily or permanently closed status

⌚️ Popular times - histogram & live occupancy

⭐️ Average rating (totalScore), review count, and review distribution

🍔 List of images (optional)

➕ List of detailed characteristics (additionalInfo, optional)

🧑‍🍳 Opening hours (optional)

🔍 People also search (optional)

🏨 Hotel booking URL and price + nearby hotels

The scraper also supports the scraping of all detailed information about reviews:

✅ Review text

✅ Published date

✅ Stars

✅ Review ID & URL

✅ Response from owner - text and published date

✅ List of review images

Personal data extraction about reviewers has to be explicitly enabled in input (see Personal data section):

  • Reviewer name
  • Reviewer ID, URL & photo
  • Reviewer number of reviews
  • Is Local Guide

Google Maps Scraper also provides other handy features:

🗺 Define search area - allows you to define the geographical area to scrape to a country, state, county, city, or postal code, thus speeding up the search (integration with Nomatim Maps API)

🔍 Automatic zooming - ensures maximum results

🌏 Language & translation settings

🗂 Reviews sorting and filtering

🛡 Proxy configuration

💻 Browser & scraping configuration

🙋 Updates from customers & Questions and answers

How does Google Maps Scraper work?

It works exactly as if you were searching through Google Maps and copying information from each page you find. It opens Google Maps website, goes to a specified location, then writes your search query into the search bar. Then it presses the Next page button until it reaches the final page or maxCrawledPlaces. It enqueues all the places as separate pages and then copypastes all visible data into an organized document. To understand the process fully, just try it out in your browser - the scraper does exactly the same thing, only much faster.

How much will scraping Google Maps cost?

Apify provides you with $5 free usage credits to use every month on the Apify Free plan, and you could get up to 2,000 reviews from this Google Maps Scraper for those credits. But the number of platform credits you use depends on the complexity of your search.

If you need to get more data regularly, you should grab an Apify subscription. We recommend our $49/month Personal plan - you can get up to 20,000 Google Maps results every month with the free $49 in monthly usage credits from that plan! For more details about usage credits, see this video guide on ▷ how to choose the right subscription plan.

For more details about Google Maps scraper usage, see the Cost of usage tab.

What are the advantages over the Google Maps API?

With the Google Maps API, you get $200 worth of credit usage every month free of charge. That means 28,500 maploads per month. However, the Google Maps API caps your search results to 60, regardless of the radius you specify. So, if you want to scrape data for bars in New York, for example, you'll get results for only 60 of the thousands of bars in the area.

Google Maps Scraper imposes no rate limits or quotas and provides more cost-effective, comprehensive results, and also scrapes histograms for popular times, which aren't available in the official API.

Is it legal to scrape Google Maps?

Web scraping is legal if you are extracting publicly available data which is most data on Google Maps. However, you should respect boundaries such as personal data and intellectual property regulations. You should only scrape personal data if you have a legitimate reason to do so, and you should also factor in Google's Terms of Use.

How do I use Google Maps Scraper?

To understand how to configure and run the scraper, follow our step-by-step guide on how to scrape Google Maps or watch a short video tutorial ▷ on YouTube.

Apify - G Maps

You can also follow along our extended guide on how to use the Google Maps Scraper geolocation features 🔗 .

What can I use the extracted data from Google Maps for?

You can use the extracted data to:

👉🏽 create a potential customer base and prospection files

👉🏽 find new clients

👉🏽 generate leads

👉🏽 search and analyze businesses similar to yours

👉🏽 monitor brand sentiment and service quality, and identify fake reviews

👉🏽 find where to buy products

👉🏽 analyze geo-spatial data for scientific or engineering work

👉🏽 develop a working market strategy

For more ideas on how to use the extracted data, check out our industries pages for concrete ways web scraping results are already being used across many projects and businesses of all types and sizes - in travel and logistics, for instance.

Want more options?

Google Maps Reviews Scraper ⭐️

If you only want to scrape reviews, this targeted data scraper is a great option. Google Maps Reviews Scraper extracts all reviews for a single place on Google Maps.

All you need to do is enter the URL of the location you want to scrape, and you'll get a dataset of all reviews, which you can download for business analysis and market research.

Easy Google Maps Scraper 🌎

If you want an easier but more limited Google Maps scraping tool, this is a very handy scraper if you're only looking to extract, say, 40 results rather than 400. Google Maps Reviews Scraper is set up and configured to make your scraping tasks super quick and easy.

Google Places API Radar Search📍

To use this tool, you need to acquire your own Google API key, as you would use the official Google API. So, why use Google Places API Radar Search instead of the API?

Nearby search with the official API returns only 60 results. Google Places API Radar Search overrides this limit and gets all places in a specified location.

Gas Prices Scraper ⛽️

Use Gas Prices Scraper to find the lowest gas prices and timestamps of price updates from gas stations in your area.

Input

Input example

The Google Maps Scraper has the following input options:

Apify  -  G Maps  Scraper  input

Or here's its equivalent in JSON:

{
  "searchStringsArray": [
    "pet shelter in Prague"
  ],
  "maxCrawledPlacesPerSearch": 10,
  "language": "en",
  "maxImages": 10,
  "exportPlaceUrls": false,
  "maxReviews": 0,
  "scrapeReviewerName": true,
  "scrapeReviewerId": true,
  "scrapeReviewerUrl": true,
  "scrapeReviewId": true,
  "scrapeReviewUrl": true,
  "scrapeResponseFromOwnerText": true,
  "proxyConfig": {
    "useApifyProxy": true
  },
  "oneReviewPerRow": false,
  "reviewsSort": "newest",
  "reviewsTranslation": "originalAndTranslated",
  "allPlacesNoSearchAction": ""
}

Output

The output from Google Maps Scraper is stored in a dataset. After the run is finished, you can download the dataset in various data formats (JSON, CSV, XML, RSS, HTML Table).

Output example

Simplified tabulated example: Apify  -  G Maps  Scraper  input

Or here's its equivalent in JSON for a single place:

{
  "title": "Sdružení na ochranu zvířat v krajní nouzi",
  "description": "Naše zraněné přírůstky po výstavě – poškozené ouško u 6letého Pouziho, vykloubená nožka 5měsíčního bílomouratého Míši, 8měsíční černý dlouhosrstý Mikulka se zraněnou nožkou, holýma zadníma nožkama a holým ocáskem, Cyril sražený autem, neovládá zadní nožky.",
  "subTitle": null,
  "price": null,
  "menu": null,
  "categoryName": "Pet adoption service",
  "address": "Na Pláni 2006, 150 00 Praha 5, Czechia",
  "locatedIn": null,
  "neighborhood": "Na Pláni 2006",
  "street": "Na Pláni 2006",
  "city": "Prague 5",
  "postalCode": "150 00",
  "state": null,
  "countryCode": "CZ",
  "plusCode": "397W+49 Prague 5, Czechia",
  "website": "https://www.kocici-utulek.cz/",
  "phone": "+420 603 225 948",
  "temporarilyClosed": false,
  "claimThisBusiness": false,
  "location": {
    "lat": 50.062872,
    "lng": 14.3958755
  },
  "permanentlyClosed": false,
  "totalScore": 4.3,
  "isAdvertisement": false,
  "rank": 9,
  "placeId": "ChIJuxlGAU6UC0cRI_jkTCUIboA",
  "categories": [
    "Pet adoption service"
  ],
  "cid": "9254343240589834275",
  "url": "https://www.google.com/maps/place/Sdru%C5%BEen%C3%AD+na+ochranu+zv%C3%AD%C5%99at+v+krajn%C3%AD+nouzi/@50.062872,14.3958755,17z/data=!3m1!4b1!4m5!3m4!1s0x470b944e014619bb:0x806e08254ce4f823!8m2!3d50.0628787!4d14.3958708?hl=en",
  "searchPageUrl": "https://www.google.com/maps/search/pet+shelter+in+Prague/@37.6,-95.665,4z?hl=en",
  "searchPageLoadedUrl": "https://www.google.com/maps/search/pet+shelter+in+Prague/@37.6,-95.665,4z?hl=en",
  "searchString": "pet shelter in Prague",
  "scrapedAt": "2022-07-28T12:42:10.969Z",
  "reviewsCount": 40,
  "reviewsDistribution": {
    "oneStar": 6,
    "twoStar": 0,
    "threeStar": 1,
    "fourStar": 2,
    "fiveStar": 31
  },
  "imageUrls": [
    "https://lh5.googleusercontent.com/p/AF1QipOeEWgXD8Jjmj3DpIa7U9VeJ3E83xaRpefxbYZh=w1920-h1080-k-no",
    "https://lh5.googleusercontent.com/p/AF1QipN2XNEQQrjtpMIHLe0WlJHYWd4nhniifUiy9BYq=w1920-h1080-k-no",
    "https://lh5.googleusercontent.com/p/AF1QipOShH8UZgA-gtJlc83n2uBLhgkd5HRacPIOx_V6=w1920-h1080-k-no",
    "https://lh5.googleusercontent.com/p/AF1QipPrcwuF0i7y32PUwX-ff-jdkRovD7XQ6fmBWHmr=w1920-h1080-k-no"
  ],
  "reviews": [],
  "reviewsTags": [
    {
      "title": "facebook",
      "count": 10
    },
  ],
  "placesTags": [
    {
      "title": "Parking",
      "count": 142
    }
  ],
  "orderBy": [],
  "peopleAlsoSearch": [
    {
        "category": "Czech restaurants",
        "title": "Restaurant Mlýnec",
        "reviewsCount": 2561,
        "totalScore": 4.7
    }
  ]
}

Additional fields are also extracted for specific cases:

Reviews

"reviews": [
    {
        "name": "What’s in your pantry?",
        "text": "One of the best pizza place in Pardubice. Great selection of pizzas for good value of money.",
        "publishAt": "4 years ago",
        "publishedAtDate": "2018-08-26T15:21:22.259Z",
        "likesCount": 0,
        "reviewId": "ChdDSUhNMG9nS0VJQ0FnSUN3aU5mS3Z3RRAB",
        "reviewUrl": "https://www.google.com/maps/reviews/data=!4m8!14m7!1m6!2m5!1sChdDSUhNMG9nS0VJQ0FnSUN3aU5mS3Z3RRAB!2m1!1s0x0:0x79ba84b5e3261fea!3m1!1s2@1:CIHM0ogKEICAgICwiNfKvwE%7CCgsI8oqL3AUQ-KHBew%7C?hl=en-US",
        "reviewerId": "117322307728195096581",
        "reviewerUrl": "https://www.google.com/maps/contrib/117322307728195096581?hl=en-US",
        "reviewerPhotoUrl": "https://lh3.googleusercontent.com/a-/AD5-WCmhximZjAUEJaQ3T-z36al5yqCsyk_VhpBZ5zv3pqs=s120-c-c0x00000000-cc-rp-mo-ba3-br100",
        "reviewerNumberOfReviews": 45,
        "isLocalGuide": true,
        "stars": 4,
        "rating": null,
        "responseFromOwnerDate": null,
        "responseFromOwnerText": null,
        "reviewImageUrls": [
            "https://lh5.googleusercontent.com/p/AF1QipMn71E_8-sqjiLKYJbbI5gqrYhVn3n3ueHpGknt=w150-h150-k-no-p",
            "https://lh5.googleusercontent.com/p/AF1QipOHWWSXkDgNn_d8kbpPxMmmb7PaHQKMULJggIMe=w150-h150-k-no-p",
            "https://lh5.googleusercontent.com/p/AF1QipMpm926guA2_yc6hamEVQpp0GWkePyv0e6OXwgU=w150-h150-k-no-p"
        ]
    }
]

Hotels

"moreHotelsOptions": [
  {
    "url": "https://www.booking.com/hotel/us/pittsburgh-3454-forbes.html?&checkin=2023-03-10&checkout=2023-03-11&group_adults=2&req_adults=2&show_room=33940104_356536769_2_2_0&lang=en&selected_currency=USD&exrt=1.00000000&ext_price_total=345.21&ext_price_tax=42.39&xfc=USD&group_children=0&req_children=0&&exfcam=_2&ts=1673860104&no_rooms=1&utm_source=metagha&utm_medium=mapresults&utm_campaign=US&utm_term=hotel-339401&utm_content=dev-desktop_los-1_bw-53_dow-Friday_defdate-1_room-0_gstadt-2_rateid-public_aud-0_gacid-_mcid-10_ppa-0_clrid-0_ad-0_gstkid-0_checkin-20230310_&aid=2127489&label=metagha-link-MRUS-hotel-339401_dev-desktop_los-1_bw-53_dow-Friday_defdate-1_room-0_gstadt-2_rateid-public_aud-0_gacid-_mcid-10_ppa-0_clrid-0_ad-0_gstkid-0_checkin-20230310_",
    "title": "Booking.com",
    "price": "$303"
  },
  {
  "url": "https://www.hotels.com/Hotel-Search?selected=118998&startDate=2023-03-10&endDate=2023-03-11&",
  "title": "Hotels.com",
  "price": "$303"
  },
],
"similarHotelsNearby": [
  {
    "name": "Residence U Malvaz",
    "rating": 4.3,
    "reviews": 406,
    "description": "Cozy hotel offering a restaurant",
    "price": "$70"
  },
  {
    "name": "Hotel U Zlatého Stromu",
    "rating": 3.6,
    "reviews": 656,
    "description": "Refined rooms, plus breakfast & dining",
    "price": "$64"
  },
]

Updates from customers

"updatesFromCustomers": {
    "text": "Disneyland California Adventure small area with large park all inclusive celebrations. This is a glimpse into Los Reyes parade.  I'm a true fan. Thanks",
    "language": "en",
    "postDate": "a week ago",
    "postedBy": {
        "name": "Kayla Arredondo",
        "url": "https://www.google.com/maps/contrib/102968882116587973980?hl=en-US",
        "title": "Local Guide",
        "totalReviews": 225
    },
    "media": [
        {
            "link": "https://lh3.googleusercontent.com/ggms/AF1QipNNaoT0NSbcWOPSduvZNqJ0kSqUs-dod32FeBtr=m18",
            "postTime": "a week ago"
        }
    ]
}

Questions and answers

"questionsAndAnswers": {
    "question": "Which is the best easier way to drop off a family to Disneyland Park",
    "answer": "best way for drop off family is at down town Disney. Drop them off then you can take a short walk to the park. ",
    "askDate": "5 years ago",
    "askedBy": {
        "name": "Cecilia Salcedo",
        "url": "https://www.google.com/maps/contrib/109041536347893604294"
    },
    "answerDate": "5 years ago",
    "answeredBy": {
        "name": "Gabby Lujan",
        "url": "https://www.google.com/maps/contrib/105966144333216697667"
    }
}

Tips and tricks: define search area

This step-by-step guide🔗 will show you the ways you can customize geolocation features and overcome the 120 results limit.

Country, state, county, city, and postal code vs. latitude and longitude

You can speed up the search and consume fewer platform credits by reducing the radius of the geographical area to scrape. You can use any combination of the geolocation parameters: country, state, county, city & postalCode. Keep in mind that the first five fields and the coordinate options are mutually exclusive.

Automatic zooming

The scraper automatically zooms the map to ensure maximum results are extracted. Higher zoom ensures more (less known) places are scraped. Logically, the smaller the area is, the higher zoom should be used. Currently, the default zoom values are:

  • no geolocation -> 12
  • country or state -> 12
  • county -> 14
  • city -> 15
  • postalCode -> 16 If you need even more results or a faster run, you can override these values with the zoom input parameter. zoom can be any number between 1 (whole globe) and 21 (few houses).

Custom search area

If your location can’t be found or you want to customize it, you can use the custom search area function for the creation of start URLs. As an example, see the geojson field in Nominatim Api (see here for the example of Cambridge in Great Britain). There are several types of search area geometry that you can use. All follow official Geo Json RFC.

  • Keep note that the order of longitude and latitude is reversed in GeoJson compared to Google Maps website. The first field must be longitude, the second field must be latitude.
  • Adjust zoom to the size of the polygon. The smaller the polygon, the higher zoom should be used. For country-wise polygon, we recommend zoom 12-14. For city-wise polygon, we recommend zoom 15-16. And for postal code-wise polygon, we recommend zoom 17-20.

https://geojson.io is an excelent tool to create customGeolocation of any type with the correct format.

Polygon

The most common type is polygon, which is a set of points that define the location. The first and last coordinate must be equal (to close the polygon). customGeolocation should have this format:

This example covers most of the city of London, UK.

{
    "type": "Polygon",
    "coordinates": [
        [
            [
                // Must be the same as last one
                -0.322813, // Longitude
                51.597165 // Latitude
            ],
            [
                -0.314990,
                51.388023
            ],
            [
                0.060493,
                51.389199
            ], 
            [
                0.051936,
                51.600360
            ],
            [
                // Must be the same as first one
                -0.322813, 
                51.597165
            ]
        // ...
        ]
    ]
}

MultiPolygon

MultiPolygon can combine more polygons that are not continuous together.

{
    "type": "MultiPolygon",
    "coordinates": [
        [ // first polygon
            [
                [
                    12.0905752, // Longitude
                    50.2524063  // Latitude
                ],
                [
                    12.1269337,
                    50.2324336
                ],
                // ...
            ]
        ],
        [
            // second polygon
            // ...
        ]
    ]
}

Circle

For a circle, we can use the Point type with our custom parameter radiusKm. Don't forget to change the radius to fit your needs. This example covers the city of Basel in Switzerland.

{
    "type": "Point",
    "coordinates": ["7.5503", "47.5590"],
    "radiusKm": 8
}

Advanced configuration

One review per row

Normally, each result item contains data about a single place. Each item is displayed as one row in tabulated formats. There is a lot of data about each place, so the tabulated formats get very messy and hard to analyze. Fortunately, there is a solution.

You can tick on the oneReviewPerRow input toggle to get one review per row as output. If you already have a dataset and need to adjust its format, read further.

For example, if you need to analyze reviews, you can configure the download to only contain the data you need and adjust the row/column format. Here's how to get a list of reviews with a place title one review per row: copy the download link in the format you need, paste it to a different tab, and add &unwind=reviews&fields=reviews,title to the end of the link URL, and then press Enter to download it. unwind=reviews means that each review will be on its own row. fields=reviews,title means that only reviews and title will be downloaded, skipping the other data. Otherwise, the output would be very big, but it's also no problem if you don't use fields at all.

The whole download link for, e.g. CSV would look like this (with dataset ID):https://api.apify.com/v2/datasets/DATASET_ID/items?clean=true&format=csv&attachment=true&unwind=reviews&fields=reviews,title

Changelog

This scraper is under active development. We are always implementing new features and fixing bugs. If you would like to see a new feature, please submit an issue on GitHub. Check CHANGELOG.md for a list of recent updates.

Resources on how to scrape Google Maps

Integrations and Google Maps Scraper

Last but not least, Google Maps Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. You can integrate with Make, Zapier, Slack, Airbyte, GitHub, Google Sheets, Google Drive, and more. Or you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever Google Maps Scraper successfully finishes a run.

Using Google Maps Scraper with the Apify API

The Apify API gives you programmatic access to the Apify platform. The API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run Apify actors. The API also lets you access any datasets, monitor actor performance, fetch results, create and update versions, and more.

To access the API using Node.js, use the apify-client NPM package. To access the API using Python, use the apify-client PyPI package.

Check out the Apify API reference docs for full details or click on the API tab for code examples.

store-crawler-google-places's People

Contributors

andreybykov avatar apify-alexey avatar beshoo avatar cybairfly avatar davidjohnbarton avatar dependabot[bot] avatar drobnikj avatar el15ka avatar gahabeen avatar hamzaalwan avatar jochenfromm avatar levent91 avatar lhotanok avatar lucie20 avatar mattiashtd avatar metalwarrior665 avatar mnmkng avatar natashalekh avatar olehveselov92 avatar pocesar avatar sauermar avatar sergemarkin avatar zpelechova avatar zzbazza avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.