Use undocumented API to get around token requirement

[ovity]

Use undocumented API to get around token requirement

[buunguyen]

This is a great find! From my testing, it looks like this API doesn't return two things:

• Submodules
• Folders

(Folders can be built from paths, except empty folders.)

This is very unfortunate as it looks very promising. Basically it requires us to trade between the robustness of Octotree (i.e. present everything and anything) and usability (no longer require access tokens for private repos or users who surf 60+ repos an hour). Really tough choice.

(Now, the fact that this is not a documented API is another issue. But that is solvable by coding to fallback to the current documented API.)

Love to hear everybody's thought.

[sullivanmatt]

One option, though a bit complex, would be to use the documented API, but fall back to this tree-list endpoint if the documented API returns an auth error. Then perhaps also leave the token option in place for more "advanced" users who want empty folders and submodules to work.

[strayer]

"(Folders can be built from paths, except empty folders.)"

I thought folders can't be empty in Git?

[buunguyen]

You're right. That was a brain fart. So it reduces to handling submodules & dealing with undocumented API.

[buunguyen]

The more I think about it, the more I hesitate about using this API due to its being undocumented. When it's changed and assume we fall back to the current API, current matters (private repos, rate limit) come into the picture again. But if we don't fallback, on a beautiful day all, users will see some error message instead of the code tree.

Reversing the API invocation order as suggested by @sullivanmatt doesn't appear to be a good option either. Assume a user runs out of rate limit, the code switch to tree-list; switch till when? (Obviously it should not make double requests every time.) Besides, if it's a private repo, it will do double request every time because it doesn't know before hand if a repo is private. Even then, if the tree-list API is changed, we need tokens. Back to square one.

Therefore, while I really like the idea of not having to require token, I think unless this API will become part of the official GitHub API, we should not adopt it in Octotree.

[sullivanmatt]

If the user were to run out of API quota and the code has to "switch over", I think double requests is a good trade-off, actually. The likelihood of a user expiring their public quota is pretty low, and the double-requests is a small price to pay for a seamless transition (in my opinion). You (rightly) mentioned that this would still cause a double-request for private repos - but in my testing that additional request would only add < 300-500ms of latency, which most users would barely notice.

The undocumented API isn't ideal, but again, if it ever moves or changes substantially, you can easily fall back to the documented API.

You mentioned the possible case where you could end up "back to square one", which you are right about, but back to square one is a huge edge case (while using private repo, undocumented API changes / moves, falls back to documented API, requires OAuth token). And sure, that could be annoying - but right now that annoyance of having to token is already happening. The net effect is almost certainly overwhelmingly positive, and at absolute worst, the user is no better or worse off if it all fails.

I understand your hesitation about not wanting to use an undocumented API, and I respect whatever decision you end up making. Thanks for taking the time to look into it!

[buunguyen]

Hi, thanks for the input.

The additional 300-500ms latency isn't good enough for me as a trade-off. Plus, it's just the average case scenario, the software might feel sluggish in worse case. Also, the "documented API -> undocumented API -> documented API + token" approach doesn't appeal to me at all, there should be a better way. Finally, falling back is easy, but detecting when/if API changes at runtime can be an ordeal.

I want robustness and high performance (not worse than currently) while not introducing much complexity to the code (~ future bugs).

Anyway, I think there are a few things I can try to utilize this API. I will keep everyone posted.