reyemb / aisexplorer Goto Github PK

This tool allows the identification and tracking of marine vessels by geolocation, MMSI, or via MarineTraffic URLs. It ensures user privacy via proxy support, and includes fallback strategies for optimal performance.

Python 100.00%

vessel vessel-detection ship-tracking vessel-trackimg ais-data ais proxy marine-data

aisexplorer's Introduction

AISExplorer

AISExplorer is a tool for locating vessels or scraping vessel data in a specific area. However, due to recent updates, the use of free proxies, previously scraped from sources like sslproxies, has been discontinued due to errors such as 403 Forbidden responses.

Next Steps

Explore additional sources for proxy lists.
Implement a method to customize the number of retries.
Introduce functionality to reset filters.

Changelog

2023-11-10

Due to Captcha implementation, login functionality is broken.
Sending requests through proxies now leads to a 403 error; proxy support has been removed.

2023-1-21

Integrated login functions for additional features.

2021-12-10

Implemented fallback if a proxy fails.
Introduced the ability to retrieve data directly via URL.
Added checks for Cloudflare's filtering mechanisms.

2021-12-5

Early stages of filter implementation.
Retry options were added for resilience.
New exceptions were introduced for better error handling.

2021-11-27

Proxy support was added (discontinued as of 2023-11-10).

Installation

pip install aisexplorer

Usage

Find vessel by MMIS

Retrieve the current location of a vessel using its MMSI identifier.

from aisexplorer.AIS import AIS

AIS().get_location(211281610)

Find vessels in Area

Retrieve data for up to 500 vessels within a designated area.

maximum 500 vessels

from aisexplorer.AIS import AIS

AIS(return_df= True).get_area_data("EMED")

The output is limited to 500 rows. Area codes can be referenced from the MarineTraffic help section. Areas can be found here

Get Table via URL

Directly access table data using a MarineTraffic URL.

from aisexplorer.AIS import AIS

AIS(return_df = True).get_data_by_url("https://www.marinetraffic.com/en/data/?asset_type=vessels&columns=time_of_latest_position:desc,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,ship_type,show_on_live_map,area,lat_of_latest_position,lon_of_latest_position,speed,length,width&area_in|in|West%20Mediterranean,East%20Mediterranean|area_in=WMED,EMED&time_of_latest_position_between|gte|time_of_latest_position_between=60,NaN")

Use Proxies

Previously, AISExplorer allowed fetching data using proxies for anonymization. This feature is no longer supported due to compatibility issues with the data source.

Get Data for user created fleets

No longer available as it required user login, which is now deprecated due to captcha implementation.

aisexplorer's People

Contributors

Stargazers

Watchers

Forkers

esadakcam 0000vt abethecreator youssef02 abioyeayo pychang-ai arunsaxena200 end18

aisexplorer's Issues

Scraper not working properly

Hi.
I think that something is broken.
I'm receiving bad answers for all the queries I tried.

Using first code in example:

from aisexplorer.AIS import AIS
AIS().get_location(211281610)

get this:

Response code: 403 - headers: {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Vessel-Image': '005bf958a6548a79c6d3a42eba493e339624'} <!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
    

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=7c3b2a8ab8e60e83')"></div>
        <form id="challenge-form" action="/en/reports?asset_type=vessels&amp;columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&amp;mmsi%7Ceq%7Cmmsi=211281610&amp;__cf_chl_f_tk=5lumYa8u0Q0vGXEyl7aDdqDED60HcE1cQby3PRkjsu4-1683480367-0-gaNycGzND3s" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="T0A9Q_oWWTNJ90qfXxdKt7Kwd5GWi2HMmEERwiWf.YE-1683480367-0-AcanNC-UP989ezPBKwc5slRCauGpYvNedJ-BUPQzWF062vpPdVQsTT1-tJJ8Osqp8__2cxZPliy1Ikb-_r84MeU4foBC4a3kX6aEddMtr1g-4bmzsJcRgknGOFa4QkwSNi5D5xov-Emlh7b5lRRPrOAUue3thMEyHvEbJhMG2fSif1xQ3U24QM2uJR94kshPObC7tPE_GYY0wO4F4OjwT8Ej2-dRmxtyGweEo9TvqiQFUB7ocJOqE_GL1_wCXJ3aBrLurBkpgAOXl5368HK_twvHbRK0G3fUbQqfVWDQ8Yy3SuZmUV2T00wgK1nocpJMsSNiO2FbeORCtyNXQY4Lkr0V13PhjlFdKJjD0kfLXawIcNwLpBxF6HqBp2d5RP73AtCZ7pAVwzAGIQL7Dxvc5SJ_wxkCcmsH3hIT_wilOAZ234lntIZNxQGmlTSgs-D6OJf-OUibuONhBtE15byTA1c_WS4xbmQom8Tz0HFzNb14WmJPF9q8ZqUPTNhPVxb_7p6yDSVAxaUdZhnW58PvJQQvE09QYnOCBoaJh8uYKBfaCuJO9ckW8PRPrBfyOnULTxuGb2y234fGfIQH3HORYoeC4u8rIQ5FUuW9pJilmgJBIt-7gtoKS4DC72x456nAaqAgDKN7HKvzmQNVPBwhI0QghVelsRO_JH1q79lrGTLITGPKNOo2S7Yr4Zl19XN_R8WE5fX1hGMmtJhFTytSHyNcAu9apqfQ7VSxxZl2LVH_Uoc3Sb3Mi2vamUozqiJ30KO5dz-zR1UqyirSxDwXNZCXVsB1mLQftQKoa-XSln8z3lGLd6Yi5hGDjDuFFU9Q-VDF1JZq8a3rKDZse8ch2kqct7hxk27UTmR2178V5yHjpcd9Mpoz_DVGu5LD2tq0AiMGe-fsEpQOLR2FSNLKiAiOxMsSLDO5C2btgbjNfVojohliYMHlTR2SLE-vGOFFKqU2xtm8S61IZgYcsiyAyagVAetUg4X5-pCiNLN9BwQF8ocCnfwUg9xlNFPnpb-uYMZXZM58QinR5oQLMEwTQD31MylSCVasWfKlpHKls0mG3g1nBPDOiAzmaTBWXTCrDuBVSAcYv0LTL5HI4-Dk0exkJjGsDFVnzXAF3XuVdfT6-Nn4lANkT6UqjbjezIJ62Gwsh14VphvD5Ii2QVcbqeRNpJuokr4x0GK_HSPUg6Q1C1OLnf3LRbVh4JJUe55H0EH5zFBTzyUfSFOGM23OisSvUWH1wiYVgYH0qVLVKrXLe_6srHU5qCYxc6vILriYSo8EWfxIpvjS6C_9Zcvha2icj5_5rFfEdV4oT8jdlIEkdTIbZ1udYi6ikclh3pOQPnuTaejmiH37Fk7XZq48fpFqzNunT2p4emw30h_Je-TKJ37JGtmG31WlIagNI3Agyos9e-a2okUFn5KHEomxhwGX-iuMs0Pf-d-80pmtlZSVox__5odxbPkPraIlv3jQH7ESTVJUAc2dtSPR59SAEpHgG12H0_whplmpFX6y0YLAAE_iKipswwMG-5ld4tCxK1dL1urDTujt4ohvRQscgVJnFvU8CkYVHaT3-y681B0ZA50u3nLfe5HHzYsXnbyQlRnslUViA7f7e3-AoMoiHdn_Kqs0tu6O1gK94KmqDG1nlikLle7zUgOUyEG7FiU5K_HJ_2jYGjOnbq4_VIomU6xpxiejiwwbbMG48HhK4U_G2Apz7_Fyhi2xzbTiTSpr4kHoS_1gAILBs637Fd4Ym1WBfjMKv7dW1fN8iuOYmreo07v54qvvHwJmzbaVm9DNSJM3Ef0TmJCUpFAqONt2KgHm1iPXxoiK0I6ZSgLA_C-iwRpU_q6txXG_vvBc9csAEgIvlH2AvDLDhmLxkJXXEYuuUzB5jwpcDcRimXuQfedNSWVBpAEE1eExRP7mPk7jknU9A24sX1ptbv_ztoti1Lztxgaj8Q9S9aq0oONoAjttkwrEF3gUznLlTURlUc2TovAe5WgwLQfl3nCkXkyYjpmKKaqJwLQos3W95nj0drk_QE664jRvtOvqgGbtNTkm1Ng9YMUeVrJpMgNqR4MZE7iFzBW1tR4SRa0CatM4FVUoKDp5qyU3s0hH7xenjFveyk3IP9Y1xoay1_yLY3q5NF4mmkUET-_ujvwZyXbvB4ssWoPGSlkDjsK4uWO0JYkSMOLQTj_x6UJvvEp6s7TqSmU7OEMfnNxnqDAQ1Xy5vnxM0hnbNSUaxa8LaD96beFFcSqxCMhY-4Nee_oLp17sUwy7UJMP-OBGaIzH3fgGvhovrSIECKcPRooaScHTuvFYIexxYcv2_dK6UQrzbiGTUmjGf2jqhpcZ7oL2LN3AG4lrlIFr8t18lCikLuHipVLZhcTY4T-5bBSxIq0LLPjFfYJAIly-GZCQAmiuI8AU16FiRYTZIV3dGtXZGHG_xvIIV4QY-LnBKC2zK3UqoJEwVotDjYjtahQ0U16KURQXfAAvCQm4Hztw4emF94yJp4TgumCN_V2PjK9tbo3N-a6IVghUogTI4saqAPIakxt6FCP26EZNnt2islUEGdyjnLl7DF3mV23q_wTBu8zSPdRTpP_88VB2z1RhRsy6RBr8riygz3C4kI1E2W0E3K4leFpGLIBPP_1PimAdZ0ZRpQVSAoAB2lgXe-zK2hLuGUlWwslmeBioVIFsZdO3ougya14PX94YcypCJX0yKsqA2AjxdAUNzJe1oA2NbjVLkH7v2lAThkPp_-u5nKuB3R4VYkHPZT4aVYIq628Nno8thHEmsZhuZ9mOX7WoYJ82V1HubS2dBmJX6bSm8k6pU4eFs2djiefMDEHdydxxXNN9SnrL3Nsg-w5577lQxYY4Q1GLTa1pyY1aeDrqznR8lOMl0jaKjY_3mDkGSBUVNHBIDJBFnZ_RfaWBvjew-QkfBAbFv03rmXkeeAm5zziBGT8EZat0wMKLXiIpIXCFqi7WHguNMjMpNm7DAGA59PKkfg_Me5wHt5dNLfO-fto45YEAeNKxqiA8T6EhMUbO7MHEsWv9boRVpEmsebe4JHo1FnymUs9siIuHm851V55POXZzLcwZLGnzlfLmqcqZvOruPEpYPOjVa5w8atzzZRnIeGDcRNn9_RDy5sPFo6art-1_FrN46xrXdZSLW9KOguNeV4DAJmjcDqS3rqeLGnXbVpzzbIroRTH6v_mhkWeznPjU0EkImXJSSluMXgRwxQsLxtTcvkW0pfmhkMxTWsmqqPv7tl-sAfaNb7Vgb0-i96NUIrZ3NJLZ4uM3s4iH8vVRZ5VSh1U8X-vt18LPPPh4hB9NkoPM1bAGR1xQdSX8gemRn6H0peRv3R0lC8iBf7DmTjJUF23pKd22mtwW32YMbp7Im9lSVIhgUzv-v32-oDOtdhaTEUOnbNb4mhfYn2YyavmbhNc2o8GQ9XrAgAmO2nFdpEzSmyDOBaEexJc2JWGMqt_c1QZfudZTV7KtVZ6hBXwAx8nLXMQw52mBi5ynlcFl8FkAzNPqt_Y-ZGBg6FDR9GYJMZsk5aXkJvwE8jNxP8jcG98KsJtNMr7Y8p_7kxgtvolyXGbH7jrNg3j5mdXP--ToNgXfO1Uxhr3Me58HgR7miXw-HzbIAH_QOsI8KDAnQ4gYFHW5uj5narzO9vHqyVIegBe763BSQlx-5fFN9BkaNm5JzRVl-Rp1iTeMe6cpAe1NQJvdDMFKwTO_TZ3BeGq8PDZhMZxfRxhpjc8kEmDO-6K0xHzA5olzcYgGmH55r67ToKqcd7apj42Oyd4f0_M36ipP3-nP2-3kdP3vtJi-8PXmkGSUcPHvkvjKvyjse4WtZvVxASm2jOeyfAhrnv0K_XLb8ksL7u5tu9aCHTX3QW2M-i85pGt97cq7wOO3OnmFZ5BTgB9h4dtM0IpsC52p0A">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cZone: 'www.marinetraffic.com',
            cType: 'managed',
            cNounce: '80480',
            cRay: '7c3b2a8ab8e60e83',
            cHash: 'fbedeeeda5c255e',
            cUPMDTk: "\/en\/reports?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi%7Ceq%7Cmmsi=211281610&__cf_chl_tk=5lumYa8u0Q0vGXEyl7aDdqDED60HcE1cQby3PRkjsu4-1683480367-0-gaNycGzND3s",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cMTimeMs: '0',
            cTplV: 5,
            cTplB: 'cf',
            cK: "",
            cRq: {
                ru: 'aHR0cHM6Ly93d3cubWFyaW5ldHJhZmZpYy5jb20vZW4vcmVwb3J0cz9hc3NldF90eXBlPXZlc3NlbHMmY29sdW1ucz10aW1lX29mX2xhdGVzdF9wb3NpdGlvbixmbGFnLHNoaXBuYW1lLHBob3RvLHJlY29nbml6ZWRfbmV4dF9wb3J0LHJlcG9ydGVkX2V0YSxyZXBvcnRlZF9kZXN0aW5hdGlvbixjdXJyZW50X3BvcnQsaW1vLG1tc2ksc2hpcF90eXBlLHNob3dfb25fbGl2ZV9tYXAsYXJlYSxhcmVhX2xvY2FsLGxhdF9vZl9sYXRlc3RfcG9zaXRpb24sbG9uX29mX2xhdGVzdF9wb3NpdGlvbixmbGVldCxzdGF0dXMsZW5pLHNwZWVkLGNvdXJzZSxkcmF1Z2h0LG5hdmlnYXRpb25hbF9zdGF0dXMseWVhcl9vZl9idWlsZCxsZW5ndGgsd2lkdGgsZHd0LGN1cnJlbnRfcG9ydF91bmxvY29kZSxjdXJyZW50X3BvcnRfY291bnRyeSxjYWxsc2lnbiZtbXNpJTdDZXElN0NtbXNpPTIxMTI4MTYxMA==',
                ra: 'TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwOC4wLjAuMCBTYWZhcmkvNTM3LjM2',
                rm: 'R0VU',
                d: '9UmkfU54/yHlLK2blqdZT0IBQgM/eh1L6qHHbLgbN6GLx8JhKbjmQtJDI7wKYHzJSn6VakJMB4vZ5DpcdSF7mUztNmomx0LAL7vgHzjx+vtHVbzfulmOctt8QAYLgBVhF1Iot7+Qkst/9bRSMtQqDFuNNjjJvy7mVx34K92OiWq83ANVOgXyQq0yGhKPX4GjBG4JS1aTiU0O1PZJr8A9DTsuYEHRhfBK6kwypmrQoNIFFBDBhL1O9vDtMUiJ3BxAuG4mLgzxWvkcELdi2asnN0c1TT6GNRSh+tfgp3fGTTO75NL7ERKW3tEoA7wqeaG6LqV+zoTs4qTAAfNE9mY6l0v/B780P8gdoxm4OSNv31sAZMiwS9iCh2u9aS0QmpGX90qlLO5zXD1Bd/DYS0nwilHUuuZjCAcPisM6NveXA3dPEEBqbaa5L6kLwQLfI7kgSqv/F9X98KWR4aBmvEoLza0zC33b3MK4bipJ0fgbEjzGPEcsyxK2q7jG/zmutz5CUoelwxxvdBuX3ebd9IkaWkJ5zzbn5hdCtEoK8niZO9rLD4svIy3Ndi3ekDf7gTiSwDEReVVL3s/jpxC1E4NlaKozoBVNn7ekF/cKizTZ145H6VDsW3xHqq3IYrRLv7nR',
                t: 'MTY4MzQ4MDM2Ny44MDkwMDA=',
                m: 'jW3sPvu77ZpogKHxA0t0C7d8rM2XLHjjwSXEkSbWy0o=',
                i1: 'uQt3aWHaMbHxXSVN54HIPg==',
                i2: 'pDEbQ9i8KJhOeILugKX5Uw==',
                zh: '9ODJ9wUnoPjm/IAh4FoxBVEPcuhbtkHqgDVY7Mg1mFw=',
                uh: 'JkAkY486s8b7quQvdaIQRGNNNpmu+ejtUZNcz6o7wFM=',
                hh: 'qi1pc4iF5r4NNN+eu2J/M9aMOtWqOev4stD1qgOIFxo=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=7c3b2a8ab8e60e83');
        trkjs.setAttribute('alt', '');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=7c3b2a8ab8e60e83';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/en\/reports?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi%7Ceq%7Cmmsi=211281610&__cf_chl_rt_tk=5lumYa8u0Q0vGXEyl7aDdqDED60HcE1cQby3PRkjsu4-1683480367-0-gaNycGzND3s" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


</body>
</html>
 - referer_url: https://www.marinetraffic.com/en/data/?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi|eq|mmsi=211281610 - request_url: https://www.marinetraffic.com/en/reports?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi|eq|mmsi=211281610

What's wrong?
Thanks

Request: login functionality

Could you add a login function, so we can use more function of marinetraffic?

From what I found, the following payload is needed to POST to https://www.marinetraffic.com/en/users/ajax_login.

payload = {
'action': 'login',
'email': 'emailaddress',
'password': 'password',
'is_ajax': 'true'
}
A cookie is then generated and saved in the session for further use.

Still working?

Hi there
I just found this repo and now I'm unsure if it's still working or not due to the captcha obstacle?
Or just a part? I want to create an layer for the Grafana Leaflet plugin https://github.com/pR0Ps/grafana-trackmap-panel to show AIS Vessel in the map together with the Seamarks from the OpenSeaMap.
Can you clarify a bit please?

No response with script

With an easy debug script, nothing happens. It just hangs and waits and no response is retrieved.

`from aisexplorer.AIS import AIS

print(AIS().get_location(255805840))`

After a while:

aisexplorer.Exceptions.NoResultsError: After 10 attempts still no results are given. If you think this is an error in the module raise an issue at https://github.com/reyemb/AISExplorer

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.