Update the Paradox GitHub package information

[blaenk]

Evil has migrated to GitHub and its repo information is reflected in the melpa recipes but one still can't view the commit-list in Paradox because it claims it's not a GitHub package, because the paradox data source constructed from the melpa recipes hasn't been updated for some time.

[blaenk]

Perhaps it'd be useful to come up with a way to do this on the user-end, to avoid running into stale data for lack of timely regeneration. Perhaps it can be done asynchronously every so often, not sure if that's possible.

[Malabarba]

This specific part could actually be done on the client (it's a single json file provided by Melpa). The original reason why I cached Paradox information on a server is that Paradox displays the number of starts for a repository, and there's no way to get that unless we do one query for each package (which is way too much for the client).

[blaenk]

I think that'd be a good idea. For example, I personally don't use the stars functionality, so it's pretty unfortunate that a very useful, basic function is limited by a more relatively frivolous feature (star count). If the star counts are a bit outdated it's not that big of an issue, but not being able to see a new package's commit list is pretty annoying.

[podiki]

Agreed, this has been coming up more and more recently with packages and Paradox saying "not a Gihub repo." Displaying commit logs is one of my favorite (and I would argue most useful) features of Paradox, so it would be great if it was given priority.

[rprimus]

Mon Jul 30 19:55:24 BST 2018

Was about to open an issue, but found this (after not finding an instance of puthash related to paradox--package-repo-list (in MELPA package)).

All of the sly-* packages are not listed as GitHub repos - and hence no commit logs.

This package (paradox) is listed as having 362 stars, but actually has 407 stars, so not been
updated in some time - [10 months ago] (84722e5).

Questions:

1. How often is this data source supposed to be updated?
2. Can the updating process be either:

   Server side:

   a. automated (run at least once a week) AND
   b. monitored for failure (not running correctly or data not uploaded to github successfully) OR

Update:

Mon Jul 30 21:12:41 BST 2018
##### Client side:
c. provided as client-side code - allowing the user to periodically run and have locally?
[Running helpers/run.sh now ...]

[Malabarba]

Yes, the process can and should be automated (I think I still have the script for that). It's supposed to be run on a server a few times a day. I used to run it on my pc at the university, but now that I work at a regular company I don't want to use my company computer for that.

Obviously the correct thing to do is run it on something like heroku or AWS, so it's a little more permanent, but I never got around to it.

[Malabarba]

For context, the reason why we need to run this script is to compile a single file with star count of all packages (which takes thousands of rest api requests, and shouldn't be done on the client's computer).

Other alternatives are to:

1. Use the graphql github api. I've never looked into it, but it's possible that it allows querying all the star counts in a single request.
2. Drop star counts support from paradox. The other information (repository names and urls) can be obtained Live from melpas packages.json file, so we don't need a separate server just to compile the star counts.

[rprimus]

Wed Aug 1 12:55:58 BST 2018

helpers/run.sh appears to be the script. Maybe run it now and a reminder to run once a week/month would be good initially - until it can be automated.

The script is responsible for creating all 4 hash tables - including paradox--package-repo-list
and paradox--wiki-packages- which adds to the used features of paradox.

[rprimus]

Wed Aug 1 18:20:05 BST 2018

When using recipes.json as a base, there are currently 67 recipes that produce incorrect repo info. [See this issue for details.]

An extra step to resolve these would be to make an http HEAD request to the repo and use the new location (in the hash table) if a 301 is encountered.

[rprimus]

Tue Aug 7 18:12:14 BST 2018

Think I've figured it out.

The github api also correctly handles redirects for relocated repos. (There is now no need to add -H 'Accept: application/vnd.github.quicksilver-preview+json' to the request.)

Curl output for `capitaomorte/sly`

: ; curl --location --include https://api.github.com/repos/capitaomorte/sly
HTTP/1.1 301 Moved Permanently
Server: GitHub.com
Date: Tue, 07 Aug 2018 16:24:40 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 168
Status: 301 Moved Permanently
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 51
X-RateLimit-Reset: 1533660162
Location: https://api.github.com/repositories/17603517
X-GitHub-Media-Type: github.v3; format=json
Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-Runtime-rack: 0.029221
X-GitHub-Request-Id: D479:1ACB:1FD706B:447A725:5B69C7C8

HTTP/1.1 200 OK
Server: GitHub.com
Date: Tue, 07 Aug 2018 16:24:40 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 5072
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 50
X-RateLimit-Reset: 1533660162
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: "38591387d18e70457812c6fe5253cf7a"
Last-Modified: Tue, 07 Aug 2018 13:46:07 GMT
X-GitHub-Media-Type: github.v3; format=json
Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-Runtime-rack: 0.034807
X-GitHub-Request-Id: D479:1ACB:1FD707A:447A73C:5B69C7C8

{
  "id": 17603517,
  "node_id": "MDEwOlJlcG9zaXRvcnkxNzYwMzUxNw==",
  "name": "sly",
  "full_name": "joaotavora/sly",
  "owner": {
    "login": "joaotavora",
    "id": 387011,
    "node_id": "MDQ6VXNlcjM4NzAxMQ==",
    "avatar_url": "https://avatars0.githubusercontent.com/u/387011?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/joaotavora",
    "html_url": "https://github.com/joaotavora",
    "followers_url": "https://api.github.com/users/joaotavora/followers",
    "following_url": "https://api.github.com/users/joaotavora/following{/other_user}",
    "gists_url": "https://api.github.com/users/joaotavora/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/joaotavora/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/joaotavora/subscriptions",
    "organizations_url": "https://api.github.com/users/joaotavora/orgs",
    "repos_url": "https://api.github.com/users/joaotavora/repos",
    "events_url": "https://api.github.com/users/joaotavora/events{/privacy}",
    "received_events_url": "https://api.github.com/users/joaotavora/received_events",
    "type": "User",
    "site_admin": false
  },
  "private": false,
  "html_url": "https://github.com/joaotavora/sly",
  "description": "Sylvester the Cat's Common Lisp IDE",
  "fork": false,
  "url": "https://api.github.com/repos/joaotavora/sly",
  "forks_url": "https://api.github.com/repos/joaotavora/sly/forks",
  "keys_url": "https://api.github.com/repos/joaotavora/sly/keys{/key_id}",
  "collaborators_url": "https://api.github.com/repos/joaotavora/sly/collaborators{/collaborator}",
  "teams_url": "https://api.github.com/repos/joaotavora/sly/teams",
  "hooks_url": "https://api.github.com/repos/joaotavora/sly/hooks",
  "issue_events_url": "https://api.github.com/repos/joaotavora/sly/issues/events{/number}",
  "events_url": "https://api.github.com/repos/joaotavora/sly/events",
  "assignees_url": "https://api.github.com/repos/joaotavora/sly/assignees{/user}",
  "branches_url": "https://api.github.com/repos/joaotavora/sly/branches{/branch}",
  "tags_url": "https://api.github.com/repos/joaotavora/sly/tags",
  "blobs_url": "https://api.github.com/repos/joaotavora/sly/git/blobs{/sha}",
  "git_tags_url": "https://api.github.com/repos/joaotavora/sly/git/tags{/sha}",
  "git_refs_url": "https://api.github.com/repos/joaotavora/sly/git/refs{/sha}",
  "trees_url": "https://api.github.com/repos/joaotavora/sly/git/trees{/sha}",
  "statuses_url": "https://api.github.com/repos/joaotavora/sly/statuses/{sha}",
  "languages_url": "https://api.github.com/repos/joaotavora/sly/languages",
  "stargazers_url": "https://api.github.com/repos/joaotavora/sly/stargazers",
  "contributors_url": "https://api.github.com/repos/joaotavora/sly/contributors",
  "subscribers_url": "https://api.github.com/repos/joaotavora/sly/subscribers",
  "subscription_url": "https://api.github.com/repos/joaotavora/sly/subscription",
  "commits_url": "https://api.github.com/repos/joaotavora/sly/commits{/sha}",
  "git_commits_url": "https://api.github.com/repos/joaotavora/sly/git/commits{/sha}",
  "comments_url": "https://api.github.com/repos/joaotavora/sly/comments{/number}",
  "issue_comment_url": "https://api.github.com/repos/joaotavora/sly/issues/comments{/number}",
  "contents_url": "https://api.github.com/repos/joaotavora/sly/contents/{+path}",
  "compare_url": "https://api.github.com/repos/joaotavora/sly/compare/{base}...{head}",
  "merges_url": "https://api.github.com/repos/joaotavora/sly/merges",
  "archive_url": "https://api.github.com/repos/joaotavora/sly/{archive_format}{/ref}",
  "downloads_url": "https://api.github.com/repos/joaotavora/sly/downloads",
  "issues_url": "https://api.github.com/repos/joaotavora/sly/issues{/number}",
  "pulls_url": "https://api.github.com/repos/joaotavora/sly/pulls{/number}",
  "milestones_url": "https://api.github.com/repos/joaotavora/sly/milestones{/number}",
  "notifications_url": "https://api.github.com/repos/joaotavora/sly/notifications{?since,all,participating}",
  "labels_url": "https://api.github.com/repos/joaotavora/sly/labels{/name}",
  "releases_url": "https://api.github.com/repos/joaotavora/sly/releases{/id}",
  "deployments_url": "https://api.github.com/repos/joaotavora/sly/deployments",
  "created_at": "2014-03-10T18:24:08Z",
  "updated_at": "2018-08-07T13:46:07Z",
  "pushed_at": "2018-08-06T20:26:21Z",
  "git_url": "git://github.com/joaotavora/sly.git",
  "ssh_url": "[email protected]:joaotavora/sly.git",
  "clone_url": "https://github.com/joaotavora/sly.git",
  "svn_url": "https://github.com/joaotavora/sly",
  "homepage": "",
  "size": 23974,
  "stargazers_count": 338,
  "watchers_count": 338,
  "language": "Common Lisp",
  "has_issues": true,
  "has_projects": true,
  "has_downloads": true,
  "has_wiki": true,
  "has_pages": true,
  "forks_count": 27,
  "mirror_url": null,
  "archived": false,
  "open_issues_count": 30,
  "license": null,
  "forks": 27,
  "open_issues": 30,
  "watchers": 338,
  "default_branch": "master",
  "network_count": 27,
  "subscribers_count": 35
}

---

Also, just noticed this:
Received a redirect reply, please file a bug report (M-x ‘paradox-bug-report’)

I've added -L to the curl flags to get the full response. paradox--github-parse-response-code needs to have a separate case for a 301 code that moves point to the section following \n\n - where the 200 response starts (before returning t).

It'll prolly take you 2 mins to write that.

[tarsius]

Use the graphql github api. I've never looked into it, but it's possible that it allows querying all the star counts in a single request.

Not in a single request but close enough. I have just implemented this for emir and that might be of use to you too.

See emacscollective/emir@021df24 (here on Github - I have added some comments).

[tarsius]

By the way, I have also updated the recipes for @joaotavora's packages on Melpa.

[rprimus]

Tue Aug 7 23:35:10 BST 2018

By the way, I have also updated the recipes for @joaotavora's packages on Melpa.

@tarsius - I see this was done in melpa/melpa@4150455 - 7 repos. Thank you!

[Malabarba]

Not in a single request but close enough. I have just implemented this for emir and that might be of use to you too.

Nice to know. Hopefully I can spend some time on that this year. :-D

[Malabarba]

Use the graphql github api. I've never looked into it, but it's possible that it allows querying all the star counts in a single request.

Not in a single request but close enough. I have just implemented this for emir and that might be of use to you too.

See emacscollective/emir@021df24 (here on Github - I have added some comments).

Thanks for the help @tarsius, I've finally managed to look into it. :-)

Unfortunately, it seems that pagination is limited to 100 repositories, so we'd still have to make dozens of queries. While that's a lot better than thousands, I'm not sure it would be a good user experience to try to do that while refreshing the package list.

[tarsius]

Actually I have updated this to use GraphQL and to be done asynchronously.

The difficult bit is implemented in emir-gh-foreach-query. You would have to copy that and replace the call to epkgs and adjust the above lambda.

emir-update-github-packages uses emir-gh-foreach-query to collect the stargazers and some information that you don't need. Once all information has been fetched, then the below callback is called, which you should replace with one that stores the stargazers in a variable or something like that.

[Malabarba]

Yes, I went through the code and I can see that being done. The problem is that all of this has to happen before the package buffer is rendered, and I can't make the user wait for 42 consecutive requests.

I could run the requests async after the package buffer is rendered, but populating the buffer with the star-counts after it has already been rendered is going to be a lot of work. :-P

[tarsius]

The problem is that all of this has to happen before the package buffer is rendered

There's a middle ground. Instead of periodically fetching the data on some server and then distributing the information you could just do the same locally. I don't see a need to suddenly make sure this information is completely up-to-date. So it should be enough to provide a separate "update melpa statistics" command and/or to automatically update periodically and in the background.