Giter Site home page Giter Site logo

Comments (14)

cecobask avatar cecobask commented on May 28, 2024 2

I don’t think the success rate would change if you were to decrease the sync workflow schedule frequency. When I was debugging the issue locally, I ran the sync 15-20 times in sequence. The success rate was around 40%. My understanding is that IMDb load balances traffic in a specific way. Unfortunately, I couldn’t figure out the pattern that they follow for routing. If someone knowledgeable in this area comes across this issue, I would appreciate any guidance! 👀

from imdb-trakt-sync.

Donostio avatar Donostio commented on May 28, 2024 1

Mine seem to be ok. Just one fail seven hours ago - exit code 2.

from imdb-trakt-sync.

cecobask avatar cecobask commented on May 28, 2024 1

Hi folks, this issue is intermittent and I first spotted it about a day or two ago.
I debugged the application locally and got the following output:

HTTP/2.0 403 Forbidden
Connection: close
Alt-Svc: h3=":443"; ma=86400
Content-Type: text/html
Date: Tue, 07 May 2024 15:09:39 GMT
Server: Server
Strict-Transport-Security: max-age=47474747; includeSubDomains; preload
Vary: Content-Type,Accept-Encoding,User-Agent
Via: 1.1 8c71fe23914182493dae4cb15c841346.cloudfront.net (CloudFront)
X-Amz-Cf-Id: 4AWuFRwkMn70jCd1R8IE5qXn2hjzmeo_bYIlki6f7je2wXwOlPFTcw==
X-Amz-Cf-Pop: VIE50-P1
X-Amz-Rid: MKPPP8ERFEQHADPJ88BC
X-Cache: Error from cloudfront

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>

The line X-Cache: Error from cloudfront is of particular interest here.
I assume there was a change on the IMDb side, related to their configuration of AWS CloudFront.

If anyone knows how to resolve this issue, all contributions are welcome!

from imdb-trakt-sync.

Donostio avatar Donostio commented on May 28, 2024 1

Mine seem to be ok. Just one fail seven hours ago - exit code 2.

Mine is failing regularly now (has been for a week now) on an intermittent basis.

from imdb-trakt-sync.

cecobask avatar cecobask commented on May 28, 2024 1

Thanks for reporting it! I will take a look today

from imdb-trakt-sync.

cecobask avatar cecobask commented on May 28, 2024 1

Hi @rafae667, you were right that the sync is no longer working if you don't specify the list identifiers.
After investigating, I discovered that IMDb changed the list page's HTML content and the scraper could no longer identify the list IDs.
I've updated the scraper selectors accordingly and it should be working now!

from imdb-trakt-sync.

rafae667 avatar rafae667 commented on May 28, 2024 1

The IMDB went through some UI changes. I can confirm it's working without the IMDB_LISTS.
Thank you so much for the quick fix!

from imdb-trakt-sync.

Valkryst avatar Valkryst commented on May 28, 2024

Based on my successful/failed syncs, I don't see an obvious pattern to the issue. I just updated my sync job to run once every 12h and I'll see if that helps 🤞

from imdb-trakt-sync.

Donostio avatar Donostio commented on May 28, 2024

It appears to be constantly failing now.

from imdb-trakt-sync.

cecobask avatar cecobask commented on May 28, 2024

Sorry for the delay in providing a workaround for this.
I identified the root cause and pushed a fix, please sync your forks to avail of it.
The solution was to include a User-Agent header in HTTP requests to IMDb.
Additionally, it looks like the HTML content of the watchlist page has changed, so I updated the selector accordingly 👍

from imdb-trakt-sync.

rafae667 avatar rafae667 commented on May 28, 2024

Thanks very much for the fix! Great work.

I noticed that my two custom lists aren't being synced. Before the fix, it was syncing.

from imdb-trakt-sync.

cecobask avatar cecobask commented on May 28, 2024

Thanks very much for the fix! Great work.

You're very welcome, @rafae667!

I noticed that my two custom lists aren't being synced. Before the fix, it was syncing.

There were changes relating to the configuration recently - secrets naming.
Can you please verify if you have a secret called IMDB_LISTS? Its value should be comma-separated values (e.g. ls000000000,ls111111111)

from imdb-trakt-sync.

cecobask avatar cecobask commented on May 28, 2024

The above statement assumes you want to sync only the specified IMDb lists.
If you would like to sync all your IMDb lists to Trakt then you don't need to define the IMDB_LISTS secret.
There are also some limitations to the number of Trakt lists you can have without a premium account: trakt/api-help#350

from imdb-trakt-sync.

rafae667 avatar rafae667 commented on May 28, 2024

I don't have that secret set, so all lists should sync.
I only have 3 lists in total, the watchlist and 2 custom ones.

I tried with the IMDB_LISTS set and it synced the lists I wanted. Without it, the lists aren't synced.

from imdb-trakt-sync.

Related Issues (17)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.