Giter Site home page Giter Site logo

cljdoc-analyzer's Introduction

cljdoc

An effort to create a central documentation hub for the Clojure & ClojureScript ecosystem.

👋 Need help getting started? Say hi on Telegram, Twitter or Clojurians Slack in #cljdoc.

CircleCI Contributors Slack

Rationale

📹 I (Martin) gave a talk at ClojuTRE about cljdoc which is probably a good intro if you want to understand what cljdoc is and why it exists. If you prefer text, read on for the Rationale.

Publishing Clojure library documentation is an often manual and error prone process. Library authors who want to provide documentation need to set up tooling to create such documentation, host it and keep it updated. In combination all these steps introduce a significant amount of friction that often leads to there not being any HTML documentation at all. If there is documentation it’s often only a matter of time until it’s out of date with the latest release.

In short: Publishing documentation is hard. Harder than it has to be.

By fully automating the process of publishing documentation we can take a great burden from the shoulders of library maintainers and let them focus on shipping great libraries with great documentation.

A central place and consistent UI for all Clojure/Script library documentation will also make it easier for developers to find and work with documentation.

By centralizing this publishing process we can also build up a global understanding of the Clojure/Script ecosystem enabling many more interesting use-cases down the road.

Goals

  • Provide an easy way to host library documentation for Clojure/Script library authors

  • Deal with all the boring stuff: hosting, updating, keeping old versions around

  • Build an ecosystem-encompassing database (+ API) of artifacts, namespaces and their contents.

  • Support API documentation, articles and examples.

  • Encourage the writing of more and better documentation.

Contributing

  1. Take look at our Contributing doc

  2. Get up and running by following the steps in Running cljdoc locally

  3. Understand why things are the way they are by reading our Architecture Decision Records

Contributors

martinklepsch borkdude avichalp dawranliou residentsummer tonsky danielcompton samihda saskali jorinvo IamDrowsy angusiguess greg-kargin rakyi nikolap mhuebert elarous mk bbatsov SevereOverfl0w roman01la mfikes anthonygalea devurandom jsimpson-ovo Tavistock alex-dixon timothypratley kkinnear karls ikitommi pesterhazy eerohele nberger ajoberstar polymeris ryrobes julienba crimeminister seancorfield holyjak lread nha mjhanninen jacobobryant lins05 jimmyhmiller awkay tobias frozar moonbrv plexus davidjameshumphreys KingMob green-coder kolharsam miikka renatoalencar kloimhardt superstructor corasaurus-hex rvalentini rollacaster vemv piotr-yuxuan RickMoynihan FieryCod cloojure IGJoshua clyfe sritchie metasoarous BrunoBonacci anthony-khong aaronEberhart mpenet cldwalker dAnjou wilkerlucio jpmonettas pragyanatvade kxygk JulienRouse nblumoe cjohansen marcomorain f-f ghost smahood witek thheller zcaudate rubygeek Biserkov bhb aviflax gnl jmlsf gacelita l3nz bfontaine kumarshantanu glts Shashai daveyarwood erasmas esuomi j-cr lilactown noprompt oskarkv patrickaroo puredanger serioga somecho viebel vlaaad AdamFrey Raynes anmonteiro atroche ccfontes davidsantiago floybix hugoduncan ieure metametadata moea moxaj mtnygard odyssomay rm-hull rwilson sbauer322 tsulej weavejester ptaoussanis tengstrand

Did we leave someone out? That certainly was not our intention, please do let us know!

License

EPL-2.0 see LICENSE

cljdoc-analyzer's People

Contributors

adamfrey avatar anmonteiro avatar atroche avatar ccfontes avatar cldwalker avatar davidsantiago avatar floybix avatar frozar avatar holyjak avatar hugoduncan avatar ieure avatar igjoshua avatar kingmob avatar lread avatar martinklepsch avatar metametadata avatar moea avatar moxaj avatar mtnygard avatar odyssomay avatar raynes avatar rm-hull avatar rwilson avatar sbauer322 avatar severeoverfl0w avatar sritchie avatar tobias avatar tsulej avatar viebel avatar weavejester avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

cljdoc-analyzer's Issues

JavaScript requires are only faked for lib sources and not its deps

Currently

Metagetta takes a look at jar source namespaces and fakes out any JavaScript requires.
A JavaScript require is detected by the namespace being a string.

Example from emmy:

  (:require #?(:cljs ["complex.js$default" :as Complex])
            #?(:cljs [goog.object :as obj])
            [emmy.generic :as g]
            [emmy.util :as u]
            [emmy.value :as v])

"complex.js$default" is a JavaScript require.

But...

Cljdoc cljs API analysis does not fake out any JavaScript requires for a lib's dependencies.

This means that if lib a namespace x loads lib b namespace y and namespace y includes some JavaScript require, the analysis will fail.

Example the wild

This issue was reported by @sritchie on Slack for emmy-viewers v0.1.0.

Relevant portion of failure:

Analyzing for cljs
Calling `mentat.clerk-utils.build/halt!` on shutdown...

2023-06-27 00:14:07,203 ERROR cljdoc-analyzer.runner - STDERR
 WARNING: abs already refers to: #'clojure.core/abs in namespace: clojure.math.numeric-tower, being replaced by: #'clojure.math.numeric-tower/abs
{:clojure.main/message
 "Execution error (ExceptionInfo) at cljs.analyzer/error (analyzer.cljc:780).\nNo such namespace: fraction.js/bigfraction.js, could not locate fraction/js_SLASH_bigfraction/js.cljs, fraction/js_SLASH_bigfraction/js.cljc, or JavaScript source providing \"fraction.js/bigfraction.js\" (Please check that namespaces with dashes use underscores in the ClojureScript file name) in file file:/root/.m2/repository/org/mentat/emmy/0.31.0/emmy-0.31.0.jar!/emmy/value.cljc\n",
 :clojure.main/triage
 {:clojure.error/class clojure.lang.ExceptionInfo,
  :clojure.error/line 780,
  :clojure.error/cause
  "No such namespace: fraction.js/bigfraction.js, could not locate fraction/js_SLASH_bigfraction/js.cljs, fraction/js_SLASH_bigfraction/js.cljc, or JavaScript source providing \"fraction.js/bigfraction.js\" (Please check that namespaces with dashes use underscores in the ClojureScript file name) in file file:/root/.m2/repository/org/mentat/emmy/0.31.0/emmy-0.31.0.jar!/emmy/value.cljc",
  :clojure.error/symbol cljs.analyzer/error,
  :clojure.error/source "analyzer.cljc",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.ExceptionInfo,
    :message
    "Could not generate ClojureScript documentation for demo/mathbox.cljs",
    :data {},
    :at
    [cljdoc_analyzer.metagetta.utils$default_exception_handler
     invokeStatic
     "utils.clj"
     118]}
   {:type clojure.lang.ExceptionInfo,
    :data
    {:clojure.error/source nil,
     :clojure.error/line nil,
     :clojure.error/column nil,
     :clojure.error/phase :compilation},
    :at [cljs.analyzer$analyze invokeStatic "analyzer.cljc" 4366]}
   {:type clojure.lang.ExceptionInfo,
    :message
    "No such namespace: fraction.js/bigfraction.js, could not locate fraction/js_SLASH_bigfraction/js.cljs, fraction/js_SLASH_bigfraction/js.cljc, or JavaScript source providing \"fraction.js/bigfraction.js\" (Please check that namespaces with dashes use underscores in the ClojureScript file name) in file file:/root/.m2/repository/org/mentat/emmy/0.31.0/emmy-0.31.0.jar!/emmy/value.cljc",
    :data {:tag :cljs/analysis-error},
    :at [cljs.analyzer$error invokeStatic "analyzer.cljc" 780]}],
  :trace
  [[cljs.analyzer$error invokeStatic "analyzer.cljc" 780]
   [cljs.analyzer$error invoke "analyzer.cljc" 776]
   [cljs.analyzer$error invokeStatic "analyzer.cljc" 778]
   [cljs.analyzer$error invoke "analyzer.cljc" 776]
   [cljs.analyzer$analyze_deps invokeStatic "analyzer.cljc" 2736]
   [cljs.analyzer$analyze_deps invoke "analyzer.cljc" 2703]
   [cljs.analyzer$ns_side_effects invokeStatic "analyzer.cljc" 4228]
   [cljs.analyzer$ns_side_effects invoke "analyzer.cljc" 4223]
   [cljs.analyzer$analyze_STAR_$fn__3858 invoke "analyzer.cljc" 4350]
   [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
   [clojure.core$reduce invokeStatic "core.clj" 6885]
   [clojure.core$reduce invoke "core.clj" 6868]
   [cljs.analyzer$analyze_STAR_ invokeStatic "analyzer.cljc" 4350]
   [cljs.analyzer$analyze_STAR_ invoke "analyzer.cljc" 4341]
   [cljs.analyzer$analyze invokeStatic "analyzer.cljc" 4369]
   [cljs.analyzer$analyze invoke "analyzer.cljc" 4352]
   [cljs.analyzer$analyze_file$fn__3987 invoke "analyzer.cljc" 4885]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4880]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4852]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_deps invokeStatic "analyzer.cljc" 2734]
   [cljs.analyzer$analyze_deps invoke "analyzer.cljc" 2703]
   [cljs.analyzer$ns_side_effects invokeStatic "analyzer.cljc" 4228]
   [cljs.analyzer$ns_side_effects invoke "analyzer.cljc" 4223]
   [cljs.analyzer$analyze_STAR_$fn__3858 invoke "analyzer.cljc" 4350]
   [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
   [clojure.core$reduce invokeStatic "core.clj" 6885]
   [clojure.core$reduce invoke "core.clj" 6868]
   [cljs.analyzer$analyze_STAR_ invokeStatic "analyzer.cljc" 4350]
   [cljs.analyzer$analyze_STAR_ invoke "analyzer.cljc" 4341]
   [cljs.analyzer$analyze invokeStatic "analyzer.cljc" 4369]
   [cljs.analyzer$analyze invoke "analyzer.cljc" 4352]
   [cljs.analyzer$analyze_file$fn__3987 invoke "analyzer.cljc" 4885]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4880]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4852]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_deps invokeStatic "analyzer.cljc" 2734]
   [cljs.analyzer$analyze_deps invoke "analyzer.cljc" 2703]
   [cljs.analyzer$ns_side_effects invokeStatic "analyzer.cljc" 4228]
   [cljs.analyzer$ns_side_effects invoke "analyzer.cljc" 4223]
   [cljs.analyzer$analyze_STAR_$fn__3858 invoke "analyzer.cljc" 4350]
   [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
   [clojure.core$reduce invokeStatic "core.clj" 6885]
   [clojure.core$reduce invoke "core.clj" 6868]
   [cljs.analyzer$analyze_STAR_ invokeStatic "analyzer.cljc" 4350]
   [cljs.analyzer$analyze_STAR_ invoke "analyzer.cljc" 4341]
   [cljs.analyzer$analyze invokeStatic "analyzer.cljc" 4369]
   [cljs.analyzer$analyze invoke "analyzer.cljc" 4352]
   [cljs.analyzer$analyze_file$fn__3987 invoke "analyzer.cljc" 4885]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4880]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4852]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_deps invokeStatic "analyzer.cljc" 2734]
   [cljs.analyzer$analyze_deps invoke "analyzer.cljc" 2703]
   [cljs.analyzer$ns_side_effects invokeStatic "analyzer.cljc" 4228]
   [cljs.analyzer$ns_side_effects invoke "analyzer.cljc" 4223]
   [cljs.analyzer$analyze_STAR_$fn__3858 invoke "analyzer.cljc" 4350]
   [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
   [clojure.core$reduce invokeStatic "core.clj" 6885]
   [clojure.core$reduce invoke "core.clj" 6868]
   [cljs.analyzer$analyze_STAR_ invokeStatic "analyzer.cljc" 4350]
   [cljs.analyzer$analyze_STAR_ invoke "analyzer.cljc" 4341]
   [cljs.analyzer$analyze invokeStatic "analyzer.cljc" 4369]
   [cljs.analyzer$analyze invoke "analyzer.cljc" 4352]
   [cljs.analyzer$analyze_file$fn__3987 invoke "analyzer.cljc" 4885]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4880]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4852]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_deps invokeStatic "analyzer.cljc" 2734]
   [cljs.analyzer$analyze_deps invoke "analyzer.cljc" 2703]
   [cljs.analyzer$ns_side_effects invokeStatic "analyzer.cljc" 4228]
   [cljs.analyzer$ns_side_effects invoke "analyzer.cljc" 4223]
   [cljs.analyzer$analyze_STAR_$fn__3858 invoke "analyzer.cljc" 4350]
   [clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
   [clojure.core$reduce invokeStatic "core.clj" 6885]
   [clojure.core$reduce invoke "core.clj" 6868]
   [cljs.analyzer$analyze_STAR_ invokeStatic "analyzer.cljc" 4350]
   [cljs.analyzer$analyze_STAR_ invoke "analyzer.cljc" 4341]
   [cljs.analyzer$analyze invokeStatic "analyzer.cljc" 4369]
   [cljs.analyzer$analyze invoke "analyzer.cljc" 4352]
   [cljs.analyzer$analyze_file$fn__3987 invoke "analyzer.cljc" 4885]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4880]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer$analyze_file invokeStatic "analyzer.cljc" 4852]
   [cljs.analyzer$analyze_file invoke "analyzer.cljc" 4838]
   [cljs.analyzer.api$analyze_file invokeStatic "api.cljc" 186]
   [cljs.analyzer.api$analyze_file invoke "api.cljc" 171]
   [cljdoc_analyzer.metagetta.clojurescript$analyze_file$fn__1540$fn__1541$fn__1542
    invoke
    "clojurescript.clj"
    142]
   [cljs.compiler$with_core_cljs invokeStatic "compiler.cljc" 1477]
   [cljs.compiler$with_core_cljs invoke "compiler.cljc" 1466]
   [cljs.compiler.api$with_core_cljs invokeStatic "api.clj" 47]
   [cljs.compiler.api$with_core_cljs invoke "api.clj" 35]
   [cljdoc_analyzer.metagetta.clojurescript$analyze_file$fn__1540$fn__1541
    invoke
    "clojurescript.clj"
    142]
   [clojure.lang.AFn applyToHelper "AFn.java" 152]
   [clojure.lang.AFn applyTo "AFn.java" 144]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
   [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
   [clojure.lang.RestFn invoke "RestFn.java" 425]
   [cljdoc_analyzer.metagetta.clojurescript$analyze_file$fn__1540
    invoke
    "clojurescript.clj"
    138]
   [cljdoc_analyzer.metagetta.clojurescript$analyze_file
    invokeStatic
    "clojurescript.clj"
    137]
   [cljdoc_analyzer.metagetta.clojurescript$analyze_file
    invoke
    "clojurescript.clj"
    136]
   [cljdoc_analyzer.metagetta.clojurescript$load_source
    invokeStatic
    "clojurescript.clj"
    150]
   [cljdoc_analyzer.metagetta.clojurescript$load_source
    invoke
    "clojurescript.clj"
    145]
   [cljdoc_analyzer.metagetta.clojurescript$read_namespaces$source_loader__1561
    invoke
    "clojurescript.clj"
    219]
   [clojure.core$map$fn__5935 invoke "core.clj" 2772]
   [clojure.lang.LazySeq sval "LazySeq.java" 42]
   [clojure.lang.LazySeq seq "LazySeq.java" 51]
   [clojure.lang.Cons next "Cons.java" 39]
   [clojure.lang.RT boundedLength "RT.java" 1790]
   [clojure.lang.RestFn applyTo "RestFn.java" 130]
   [clojure.core$apply invokeStatic "core.clj" 669]
   [clojure.core$apply invoke "core.clj" 662]
   [cljdoc_analyzer.metagetta.clojurescript$read_namespaces
    invokeStatic
    "clojurescript.clj"
    224]
   [cljdoc_analyzer.metagetta.clojurescript$read_namespaces
    invoke
    "clojurescript.clj"
    181]
   [cljdoc_analyzer.metagetta.main$read_namespaces
    invokeStatic
    "main.clj"
    116]
   [cljdoc_analyzer.metagetta.main$read_namespaces
    invoke
    "main.clj"
    113]
   [cljdoc_analyzer.metagetta.main$get_metadata$fn__1652
    invoke
    "main.clj"
    147]
   [clojure.core$mapv$fn__8535 invoke "core.clj" 6979]
   [clojure.lang.ArraySeq reduce "ArraySeq.java" 119]
   [clojure.core$reduce invokeStatic "core.clj" 6885]
   [clojure.core$mapv invokeStatic "core.clj" 6970]
   [clojure.core$mapv invoke "core.clj" 6970]
   [cljdoc_analyzer.metagetta.main$get_metadata
    invokeStatic
    "main.clj"
    144]
   [cljdoc_analyzer.metagetta.main$get_metadata invoke "main.clj" 130]
   [cljdoc_analyzer.metagetta.main$_main invokeStatic "main.clj" 178]
   [cljdoc_analyzer.metagetta.main$_main invoke "main.clj" 153]
   [clojure.lang.AFn applyToHelper "AFn.java" 154]
   [clojure.lang.AFn applyTo "AFn.java" 144]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.main$main_opt invokeStatic "main.clj" 514]
   [clojure.main$main_opt invoke "main.clj" 510]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause
  "No such namespace: fraction.js/bigfraction.js, could not locate fraction/js_SLASH_bigfraction/js.cljs, fraction/js_SLASH_bigfraction/js.cljc, or JavaScript source providing \"fraction.js/bigfraction.js\" (Please check that namespaces with dashes use underscores in the ClojureScript file name) in file file:/root/.m2/repository/org/mentat/emmy/0.31.0/emmy-0.31.0.jar!/emmy/value.cljc",
  :data {:tag :cljs/analysis-error}}}

Execution error (ExceptionInfo) at cljs.analyzer/error (analyzer.cljc:780).
No such namespace: fraction.js/bigfraction.js, could not locate fraction/js_SLASH_bigfraction/js.cljs, fraction/js_SLASH_bigfraction/js.cljc, or JavaScript source providing "fraction.js/bigfraction.js" (Please check that namespaces with dashes use underscores in the ClojureScript file name) in file file:/root/.m2/repository/org/mentat/emmy/0.31.0/emmy-0.31.0.jar!/emmy/value.cljc

The failure is triggered by "fraction.js/bigfraction.js" not being faked out in emmy-viewer's emmy dependency.

Why does emmy analysis pass, but not emmy-viewers?

Note that emmy itself passes cljs analysis presumably because, unlike emmy-viewers, it does not load any namespaces that have JavaScript requires.

Observation

Just because a library passes cljs API analysis does not indicate it will work with vanilla ClojureScript. Library authors should do their own testing to verify that their libs work under vanilla ClojureScript. (Many library authors do all their dev work in shadow-cljs these days)

Next Up

So... can I have cljs analysis scan the entire classpath for namespaces with JavaScript requires and fake them all out (instead of just the jar's own source files)? I'll take a stab at it.

Running Tests is Taking Longer

While @holyjak and I were investigating an issue we noticed that test times have been creeping significantly upward over the last 10 months.

As of 9-Feb-2021 (builds are randomly picked and do not at all imply cause):

My guess is that we've simply added more integration tests and integration tests take a while to run.
I've created this issue as a reminder to check if that is true.

Analysis of instaparse fails to load into cljdoc database

Symptom

@KingMob reported on Slack:

Does anyone know why the instaparse cljdocs aren't building? I tried triggering a rebuild of the latest 1.4.12, and the Circle build is all green, but the cljdoc build says there was an error. Looks like the last successful cljdoc was for v1.4.9.

Repro - cljdoc

I fired up cljdoc locally and tried to rebuild instaparse 1.4.12. I see in cljdoc log:

Failed to insert namespace {:name "instaparse.abnf", :doc "This is the context free grammar that recognizes ABNF notation.\n", :platform "clj"}
...
[SQLITE_CONSTRAINT_UNIQUE] A UNIQUE constraint failed (UNIQUE constraint failed: namespaces.version_id, namespaces.platform, namespaces.name)

Explore - cljdoc analyzer

Run cljdoc-analyzer locally:

clojure -M -m cljdoc-analyzer.main analyze --project instaparse/instaparse --version 1.4.12  --output-filename out.edn

And I see a duplicate instaparse.abnf namespace under the clj analysis (among others).
This would cause the cljdoc failure.

Look at jar

Running

unzip -l ~/.m2/repository/instaparse/instaparse/1.4.12/instaparse-1.4.12.jar

Outputs:

  Length      Date    Time    Name
---------  ---------- -----   ----
      234  2022-05-05 00:55   META-INF/MANIFEST.MF
     3314  2022-05-05 00:55   META-INF/maven/instaparse/instaparse/pom.xml
     3240  2022-05-05 00:55   META-INF/leiningen/instaparse/instaparse/project.clj
    67972  2022-05-05 00:55   META-INF/leiningen/instaparse/instaparse/README.md
    11432  2022-05-05 00:55   META-INF/leiningen/instaparse/instaparse/LICENSE
        0  2022-05-05 00:55   META-INF/
        0  2022-05-05 00:55   META-INF/maven/
        0  2022-05-05 00:55   META-INF/maven/instaparse/
        0  2022-05-05 00:55   META-INF/maven/instaparse/instaparse/
      153  2022-05-05 00:55   META-INF/maven/instaparse/instaparse/pom.properties
        0  2022-05-05 00:55   instaparse/
    10452  2022-05-05 00:55   instaparse/abnf.clj
    16383  2022-05-05 00:55   instaparse/auto_flatten_seq.clj
    13274  2022-05-05 00:55   instaparse/cfg.clj
     1065  2022-05-05 00:55   instaparse/combinators.clj
     6904  2022-05-05 00:55   instaparse/combinators_source.clj
    15939  2022-05-05 00:55   instaparse/core.clj
     3052  2022-05-05 00:55   instaparse/failure.clj
    42727  2022-05-05 00:55   instaparse/gll.clj
     4621  2022-05-05 00:55   instaparse/line_col.clj
     3736  2022-05-05 00:55   instaparse/print.clj
     2104  2022-05-05 00:55   instaparse/reduction.clj
    10024  2022-05-05 00:55   instaparse/repeat.clj
     2951  2022-05-05 00:55   instaparse/transform.clj
      673  2022-05-05 00:55   instaparse/util.clj
    10382  2018-12-18 00:11   instaparse/abnf.cljc
    16301  2016-12-23 16:08   instaparse/auto_flatten_seq.cljc
    13205  2018-12-18 00:31   instaparse/cfg.cljc
      988  2016-12-23 16:08   instaparse/combinators.cljc
     6820  2018-04-08 19:29   instaparse/combinators_source.cljc
    15869  2022-05-05 00:46   instaparse/core.cljc
     2979  2022-05-05 00:46   instaparse/failure.cljc
    42658  2018-04-08 19:29   instaparse/gll.cljc
     4547  2016-12-23 16:08   instaparse/line_col.cljc
      703  2016-12-23 16:08   instaparse/macros.clj
     3665  2016-12-23 16:08   instaparse/print.cljc
     2029  2016-12-23 16:08   instaparse/reduction.cljc
     9952  2016-12-23 16:08   instaparse/repeat.cljc
     2876  2016-12-23 16:08   instaparse/transform.cljc
      603  2018-12-18 00:11   instaparse/util.cljc
     4671  2018-04-08 20:16   instaparse/viz.clj
      476  2016-12-23 16:08   instaparse/viz.cljs
---------                     -------
   358974                     42 files

After a bit of study, I see most of the .clj files are generated for Clojure <= 1.6 compatibility: Engelberg/instaparse#149

I expect this has something to do with the duplicate namespaces in the analysis output.
I'll follow up by studying how cljdoc-analyzer loads instaparse namespaces.

regresssion: reitit main project is failing to analyze

From Slack...

The reitit main artifact is failing to analyze.
Reitit is special in that it uses modules.
The main artifact has no Clojure source.

Local reproduction

clojure -M -m cljdoc-analyzer.main analyze --project fi.metosin/reitit --version 0.7.0-alpha6  --output-filename out.edn

Results in a no Clojure/Clojurescript sources found exception

{:clojure.main/message
  "Execution error (ExceptionInfo) at cljdoc-analyzer.metagetta.utils/infer-platforms-from-src-dir (utils.clj:118).\nno Clojure/Clojurescript sources found\n",
  :clojure.main/triage
  {:clojure.error/class clojure.lang.ExceptionInfo,
   :clojure.error/line 118,
   :clojure.error/cause "no Clojure/Clojurescript sources found",
   :clojure.error/symbol
   cljdoc-analyzer.metagetta.utils/infer-platforms-from-src-dir,
   :clojure.error/source "utils.clj",
   :clojure.error/phase :execution},
  :clojure.main/trace
  {:via
   [{:type clojure.lang.ExceptionInfo,
     :message "no Clojure/Clojurescript sources found",
     :data {},
     :at
     [cljdoc_analyzer.metagetta.utils$infer_platforms_from_src_dir
      invokeStatic
      "utils.clj"
      118]}],
   :trace
   [[cljdoc_analyzer.metagetta.utils$infer_platforms_from_src_dir
     invokeStatic
     "utils.clj"
     118]
    [cljdoc_analyzer.metagetta.utils$infer_platforms_from_src_dir
     invoke
     "utils.clj"
     102]
    [cljdoc_analyzer.metagetta.main$determine_languages
     invokeStatic
     "main.clj"
     117]
    [cljdoc_analyzer.metagetta.main$determine_languages
     invoke
     "main.clj"
     115]
    [cljdoc_analyzer.metagetta.main$get_metadata
     invokeStatic
     "main.clj"
     135]
    [cljdoc_analyzer.metagetta.main$get_metadata invoke "main.clj" 123]
    [cljdoc_analyzer.metagetta.main$_main invokeStatic "main.clj" 172]
    [cljdoc_analyzer.metagetta.main$_main invoke "main.clj" 149]
    [clojure.lang.AFn applyToHelper "AFn.java" 154]
    [clojure.lang.AFn applyTo "AFn.java" 144]
    [clojure.lang.Var applyTo "Var.java" 705]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.main$main_opt invokeStatic "main.clj" 514]
    [clojure.main$main_opt invoke "main.clj" 510]
    [clojure.main$main invokeStatic "main.clj" 664]
    [clojure.main$main doInvoke "main.clj" 616]
    [clojure.lang.RestFn applyTo "RestFn.java" 137]
    [clojure.lang.Var applyTo "Var.java" 705]
    [clojure.main main "main.java" 40]],
   :cause "no Clojure/Clojurescript sources found",
   :data {}}}

Is this due to artifact group name change?

Maybe we are special-casing this artifact?
Let's try the old group id:

clojure -M -m cljdoc-analyzer.main analyze --project metosin/reitit --version 0.7.0-alpha5  --output-filename out.edn

Nope, same error.

Slack guess

On Slack it was guessed this change might be the cause #83

I thought our intent was to fail on artifacts with no sources, and I did make that change, so yes, I think this guess is right!

The code used to fail if the jar had no files at all, a check I think I'll still skip after fixing.

I'll restore old behaviour of it being ok for an artifact to have no Clojure Sources.

Metosin malli 0.10.0 is failing analysis

Full build log

clojure -Sdeps "${CLJDOC_ANALYZER_DEP:-$DEP_FALLBACK}" -M -m cljdoc-analyzer.cljdoc-main "${CLJDOC_ANALYZER_ARGS:-$FALLBACK}"

Cloning: https://github.com/cljdoc/cljdoc-analyzer.git
Checking out: https://github.com/cljdoc/cljdoc-analyzer.git at 5489e7de9542038a57a09abe583fc187d6a360e1
Cloning: https://github.com/cljdoc/cljdoc-shared.git
Checking out: https://github.com/cljdoc/cljdoc-shared.git at 53c113e375df94d50cf96c6f2c1b3842dc0e9b35
2023-01-12 17:27:37,200 INFO  cljdoc-analyzer.cljdoc-main - args:
{:project "metosin/malli",
 :version "0.10.0",
 :jarpath
 "https://repo.clojars.org/metosin/malli/0.10.0/malli-0.10.0.jar",
 :pompath
 "https://repo.clojars.org/metosin/malli/0.10.0/malli-0.10.0.pom",
 :languages nil,
 :repos
 {"clojars" {:url "https://repo.clojars.org/"},
  "central" {:url "https://repo.maven.apache.org/maven2/"}}}

2023-01-12 17:27:37,204 INFO  cljdoc-analyzer.runner - cljdoc-analyzer-version 1.0.705
2023-01-12 17:27:37,241 INFO  cljdoc-analyzer.runner - args:
{:pompath
 "https://repo.clojars.org/metosin/malli/0.10.0/malli-0.10.0.pom",
 :namespaces :all,
 :output-filename
 "/tmp/cljdoc/analysis-out/cljdoc-analysis-edn/metosin/malli/0.10.0/cljdoc-analysis.edn",
 :exclude-with [:no-doc :skip-wiki :mranderson/inlined],
 :extra-repos nil,
 :project "metosin/malli",
 :version "0.10.0",
 :languages nil,
 :jarpath
 "https://repo.clojars.org/metosin/malli/0.10.0/malli-0.10.0.jar"}

2023-01-12 17:27:37,244 INFO  cljdoc-analyzer.runner - Downloading https://repo.clojars.org/metosin/malli/0.10.0/malli-0.10.0.jar
Downloading: org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Downloading: org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Downloading: mvxcvi/arrangement/2.0.0/arrangement-2.0.0.pom from clojars
Downloading: fipp/fipp/0.6.26/fipp-0.6.26.pom from clojars
Downloading: borkdude/edamame/1.0.0/edamame-1.0.0.pom from clojars
Downloading: borkdude/dynaload/0.3.5/dynaload-0.3.5.pom from clojars
Downloading: org/clojure/test.check/1.1.1/test.check-1.1.1.pom from central
Downloading: org/clojure/core.rrb-vector/0.1.2/core.rrb-vector-0.1.2.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Downloading: org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
Downloading: org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Downloading: fipp/fipp/0.6.26/fipp-0.6.26.jar from clojars
Downloading: mvxcvi/arrangement/2.0.0/arrangement-2.0.0.jar from clojars
Downloading: borkdude/dynaload/0.3.5/dynaload-0.3.5.jar from clojars
Downloading: borkdude/edamame/1.0.0/edamame-1.0.0.jar from clojars
Downloading: org/clojure/core.rrb-vector/0.1.2/core.rrb-vector-0.1.2.jar from central
Downloading: org/clojure/test.check/1.1.1/test.check-1.1.1.jar from central
2023-01-12 17:27:38,472 INFO  cljdoc-analyzer.runner - dependencies for analysis:
org.clojure/clojure 1.10.1
  org.clojure/core.specs.alpha 0.2.44
  org.clojure/spec.alpha 0.2.176
org.clojure/clojurescript 1.10.773
  org.clojure/data.json 0.2.6
  org.clojure/google-closure-library 0.0-20191016-6ae1f72f
    org.clojure/google-closure-library-third-party 0.0-20191016-6ae1f72f
  com.cognitect/transit-clj 0.8.309
    com.cognitect/transit-java 0.8.332
      commons-codec/commons-codec 1.10
      com.fasterxml.jackson.core/jackson-core 2.8.7
      org.msgpack/msgpack 0.6.12
        com.googlecode.json-simple/json-simple 1.1.1
        org.javassist/javassist 3.18.1-GA
  com.google.javascript/closure-compiler-unshaded v20200315
    com.google.elemental2/elemental2-core 1.0.0-RC1
    com.google.errorprone/error_prone_annotations 2.3.1
    com.google.jsinterop/jsinterop-annotations 1.0.2
    com.google.javascript/closure-compiler-externs v20200315
    com.google.guava/guava 25.1-jre
      org.codehaus.mojo/animal-sniffer-annotations 1.14
      com.google.j2objc/j2objc-annotations 1.1
      org.checkerframework/checker-qual 2.0.0
      com.google.code.findbugs/jsr305 3.0.2
    args4j/args4j 2.0.26
    com.google.jsinterop/base 1.0.0
    com.google.re2j/re2j 1.3
    com.google.protobuf/protobuf-java 3.11.1
    com.google.code.gson/gson 2.7
cljdoc-analyzer/metagetta /root/.gitlibs/libs/cljdoc/cljdoc-analyzer/5489e7de9542038a57a09abe583fc187d6a360e1/modules/metagetta
  org.clojure/tools.namespace 1.3.0
    org.clojure/java.classpath 1.0.0
    org.clojure/tools.reader 1.3.6
metosin/malli /tmp/cljdoc-metosin-malli-0.10.018037313147676282944
  fipp/fipp 0.6.26
    org.clojure/core.rrb-vector 0.1.2
  mvxcvi/arrangement 2.0.0
  borkdude/dynaload 0.3.5
  borkdude/edamame 1.0.0
  org.clojure/test.check 1.1.1
javax.servlet/javax.servlet-api 4.0.1

2023-01-12 17:27:38,473 INFO  cljdoc-analyzer.runner - launching metagetta for: metosin/malli languages: :auto-detect
2023-01-12 17:27:43,659 INFO  cljdoc-analyzer.runner - metagetta results:
exit-code 1
stdout:
 Args: {:namespaces :all,
        :root-path
        "/tmp/cljdoc-metosin-malli-0.10.018037313147676282944/src/main/clojure",
        :languages :auto-detect,
        :output-filename "/tmp/metosin-malli12338667028703218248.edn",
        :exclude-with [:no-doc :skip-wiki :mranderson/inlined]}
 Java version 17.0.5
 Clojure version 1.10.1
 ClojureScript version 1.10.773
 Analyzing for clj
stderr:
 {:clojure.main/message
  "Execution error at malli.instrument/eval8557$loading (instrument.clj:1).\nInvalid token: ::mg/keys\n",
  :clojure.main/triage
  {:clojure.error/class java.lang.RuntimeException,
   :clojure.error/line 1,
   :clojure.error/cause "Invalid token: ::mg/keys",
   :clojure.error/symbol malli.instrument/eval8557$loading,
   :clojure.error/source "instrument.clj",
   :clojure.error/phase :execution},
  :clojure.main/trace
  {:via
   [{:type clojure.lang.ExceptionInfo,
     :message "Could not generate Clojure documentation for malli.dev",
     :data {},
     :at
     [cljdoc_analyzer.metagetta.utils$default_exception_handler
      invokeStatic
      "utils.clj"
      118]}
    {:type clojure.lang.Compiler$CompilerException,
     :message
     "Syntax error reading source at (malli/generator.cljc:55:14).",
     :data
     {:clojure.error/phase :read-source,
      :clojure.error/line 55,
      :clojure.error/column 14,
      :clojure.error/source "malli/generator.cljc"},
     :at [clojure.lang.Compiler load "Compiler.java" 7643]}
    {:type java.lang.RuntimeException,
     :message "Invalid token: ::mg/keys",
     :at [clojure.lang.Util runtimeException "Util.java" 221]}],
   :trace
   [[clojure.lang.Util runtimeException "Util.java" 221]
    [clojure.lang.LispReader interpretToken "LispReader.java" 412]
    [clojure.lang.LispReader read "LispReader.java" 305]
    [clojure.lang.LispReader readDelimitedList "LispReader.java" 1398]
    [clojure.lang.LispReader$MapReader invoke "LispReader.java" 1355]
    [clojure.lang.LispReader read "LispReader.java" 285]
    [clojure.lang.LispReader readDelimitedList "LispReader.java" 1398]
    [clojure.lang.LispReader$VectorReader invoke "LispReader.java" 1347]
    [clojure.lang.LispReader read "LispReader.java" 285]
    [clojure.lang.LispReader readDelimitedList "LispReader.java" 1398]
    [clojure.lang.LispReader$ListReader invoke "LispReader.java" 1243]
    [clojure.lang.LispReader read "LispReader.java" 285]
    [clojure.lang.LispReader read "LispReader.java" 216]
    [clojure.lang.Compiler load "Compiler.java" 7631]
    [clojure.lang.RT loadResourceScript "RT.java" 381]
    [clojure.lang.RT loadResourceScript "RT.java" 372]
    [clojure.lang.RT load "RT.java" 459]
    [clojure.lang.RT load "RT.java" 424]
    [clojure.core$load$fn__6839 invoke "core.clj" 6126]
    [clojure.core$load invokeStatic "core.clj" 6125]
    [clojure.core$load doInvoke "core.clj" 6109]
    [clojure.lang.RestFn invoke "RestFn.java" 408]
    [clojure.core$load_one invokeStatic "core.clj" 5908]
    [clojure.core$load_one invoke "core.clj" 5903]
    [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
    [clojure.core$load_lib invokeStatic "core.clj" 5947]
    [clojure.core$load_lib doInvoke "core.clj" 5928]
    [clojure.lang.RestFn applyTo "RestFn.java" 142]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$load_libs invokeStatic "core.clj" 5985]
    [clojure.core$load_libs doInvoke "core.clj" 5969]
    [clojure.lang.RestFn applyTo "RestFn.java" 137]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$require invokeStatic "core.clj" 6007]
    [clojure.core$require doInvoke "core.clj" 6007]
    [clojure.lang.RestFn invoke "RestFn.java" 436]
    [malli.instrument$eval8557$loading__6721__auto____8558
     invoke
     "instrument.clj"
     1]
    [malli.instrument$eval8557 invokeStatic "instrument.clj" 1]
    [malli.instrument$eval8557 invoke "instrument.clj" 1]
    [clojure.lang.Compiler eval "Compiler.java" 7177]
    [clojure.lang.Compiler eval "Compiler.java" 7166]
    [clojure.lang.Compiler load "Compiler.java" 7636]
    [clojure.lang.RT loadResourceScript "RT.java" 381]
    [clojure.lang.RT loadResourceScript "RT.java" 372]
    [clojure.lang.RT load "RT.java" 459]
    [clojure.lang.RT load "RT.java" 424]
    [clojure.core$load$fn__6839 invoke "core.clj" 6126]
    [clojure.core$load invokeStatic "core.clj" 6125]
    [clojure.core$load doInvoke "core.clj" 6109]
    [clojure.lang.RestFn invoke "RestFn.java" 408]
    [clojure.core$load_one invokeStatic "core.clj" 5908]
    [clojure.core$load_one invoke "core.clj" 5903]
    [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
    [clojure.core$load_lib invokeStatic "core.clj" 5947]
    [clojure.core$load_lib doInvoke "core.clj" 5928]
    [clojure.lang.RestFn applyTo "RestFn.java" 142]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$load_libs invokeStatic "core.clj" 5985]
    [clojure.core$load_libs doInvoke "core.clj" 5969]
    [clojure.lang.RestFn applyTo "RestFn.java" 137]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$require invokeStatic "core.clj" 6007]
    [clojure.core$require doInvoke "core.clj" 6007]
    [clojure.lang.RestFn invoke "RestFn.java" 457]
    [malli.dev$eval7794$loading__6721__auto____7795 invoke "dev.clj" 1]
    [malli.dev$eval7794 invokeStatic "dev.clj" 1]
    [malli.dev$eval7794 invoke "dev.clj" 1]
    [clojure.lang.Compiler eval "Compiler.java" 7177]
    [clojure.lang.Compiler eval "Compiler.java" 7166]
    [clojure.lang.Compiler load "Compiler.java" 7636]
    [clojure.lang.RT loadResourceScript "RT.java" 381]
    [clojure.lang.RT loadResourceScript "RT.java" 372]
    [clojure.lang.RT load "RT.java" 459]
    [clojure.lang.RT load "RT.java" 424]
    [clojure.core$load$fn__6839 invoke "core.clj" 6126]
    [clojure.core$load invokeStatic "core.clj" 6125]
    [clojure.core$load doInvoke "core.clj" 6109]
    [clojure.lang.RestFn invoke "RestFn.java" 408]
    [clojure.core$load_one invokeStatic "core.clj" 5908]
    [clojure.core$load_one invoke "core.clj" 5903]
    [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
    [clojure.core$load_lib invokeStatic "core.clj" 5947]
    [clojure.core$load_lib doInvoke "core.clj" 5928]
    [clojure.lang.RestFn applyTo "RestFn.java" 142]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$load_libs invokeStatic "core.clj" 5985]
    [clojure.core$load_libs doInvoke "core.clj" 5969]
    [clojure.lang.RestFn applyTo "RestFn.java" 137]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$require invokeStatic "core.clj" 6007]
    [clojure.core$require doInvoke "core.clj" 6007]
    [clojure.lang.RestFn invoke "RestFn.java" 408]
    [cljdoc_analyzer.metagetta.clojure$read_ns$fn__1614
     invoke
     "clojure.clj"
     92]
    [cljdoc_analyzer.metagetta.clojure$read_ns
     invokeStatic
     "clojure.clj"
     91]
    [cljdoc_analyzer.metagetta.clojure$read_ns invoke "clojure.clj" 86]
    [cljdoc_analyzer.metagetta.clojure$read_namespaces$fn__1627
     invoke
     "clojure.clj"
     160]
    [clojure.core$map$fn__5866 invoke "core.clj" 2755]
    [clojure.lang.LazySeq sval "LazySeq.java" 42]
    [clojure.lang.LazySeq seq "LazySeq.java" 51]
    [clojure.lang.Cons next "Cons.java" 39]
    [clojure.lang.RT boundedLength "RT.java" 1792]
    [clojure.lang.RestFn applyTo "RestFn.java" 130]
    [clojure.core$apply invokeStatic "core.clj" 665]
    [clojure.core$mapcat invokeStatic "core.clj" 2783]
    [clojure.core$mapcat doInvoke "core.clj" 2783]
    [clojure.lang.RestFn invoke "RestFn.java" 423]
    [cljdoc_analyzer.metagetta.clojure$read_namespaces
     invokeStatic
     "clojure.clj"
     160]
    [cljdoc_analyzer.metagetta.clojure$read_namespaces
     invoke
     "clojure.clj"
     122]
    [cljdoc_analyzer.metagetta.main$read_namespaces
     invokeStatic
     "main.clj"
     116]
    [cljdoc_analyzer.metagetta.main$read_namespaces
     invoke
     "main.clj"
     113]
    [cljdoc_analyzer.metagetta.main$get_metadata$fn__1814
     invoke
     "main.clj"
     147]
    [clojure.core$mapv$fn__8445 invoke "core.clj" 6912]
    [clojure.lang.ArraySeq reduce "ArraySeq.java" 111]
    [clojure.core$reduce invokeStatic "core.clj" 6827]
    [clojure.core$mapv invokeStatic "core.clj" 6903]
    [clojure.core$mapv invoke "core.clj" 6903]
    [cljdoc_analyzer.metagetta.main$get_metadata
     invokeStatic
     "main.clj"
     144]
    [cljdoc_analyzer.metagetta.main$get_metadata invoke "main.clj" 130]
    [cljdoc_analyzer.metagetta.main$_main invokeStatic "main.clj" 178]
    [cljdoc_analyzer.metagetta.main$_main invoke "main.clj" 153]
    [clojure.lang.AFn applyToHelper "AFn.java" 154]
    [clojure.lang.AFn applyTo "AFn.java" 144]
    [clojure.lang.Var applyTo "Var.java" 705]
    [clojure.core$apply invokeStatic "core.clj" 665]
    [clojure.main$main_opt invokeStatic "main.clj" 514]
    [clojure.main$main_opt invoke "main.clj" 510]
    [clojure.main$main invokeStatic "main.clj" 664]
    [clojure.main$main doInvoke "main.clj" 616]
    [clojure.lang.RestFn applyTo "RestFn.java" 137]
    [clojure.lang.Var applyTo "Var.java" 705]
    [clojure.main main "main.java" 40]],
   :cause "Invalid token: ::mg/keys"}}
 
 Execution error at malli.instrument/eval8557$loading (instrument.clj:1).
 Invalid token: ::mg/keys
2023-01-12 17:27:43,664 ERROR cljdoc-analyzer.runner - Analysis failed with code 1
2023-01-12 17:27:43,664 ERROR cljdoc-analyzer.runner - STDOUT
 Args: {:namespaces :all,
       :root-path
       "/tmp/cljdoc-metosin-malli-0.10.018037313147676282944/src/main/clojure",
       :languages :auto-detect,
       :output-filename "/tmp/metosin-malli12338667028703218248.edn",
       :exclude-with [:no-doc :skip-wiki :mranderson/inlined]}
Java version 17.0.5
Clojure version 1.10.1
ClojureScript version 1.10.773
Analyzing for clj

2023-01-12 17:27:43,664 ERROR cljdoc-analyzer.runner - STDERR
 {:clojure.main/message
 "Execution error at malli.instrument/eval8557$loading (instrument.clj:1).\nInvalid token: ::mg/keys\n",
 :clojure.main/triage
 {:clojure.error/class java.lang.RuntimeException,
  :clojure.error/line 1,
  :clojure.error/cause "Invalid token: ::mg/keys",
  :clojure.error/symbol malli.instrument/eval8557$loading,
  :clojure.error/source "instrument.clj",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.ExceptionInfo,
    :message "Could not generate Clojure documentation for malli.dev",
    :data {},
    :at
    [cljdoc_analyzer.metagetta.utils$default_exception_handler
     invokeStatic
     "utils.clj"
     118]}
   {:type clojure.lang.Compiler$CompilerException,
    :message
    "Syntax error reading source at (malli/generator.cljc:55:14).",
    :data
    {:clojure.error/phase :read-source,
     :clojure.error/line 55,
     :clojure.error/column 14,
     :clojure.error/source "malli/generator.cljc"},
    :at [clojure.lang.Compiler load "Compiler.java" 7643]}
   {:type java.lang.RuntimeException,
    :message "Invalid token: ::mg/keys",
    :at [clojure.lang.Util runtimeException "Util.java" 221]}],
  :trace
  [[clojure.lang.Util runtimeException "Util.java" 221]
   [clojure.lang.LispReader interpretToken "LispReader.java" 412]
   [clojure.lang.LispReader read "LispReader.java" 305]
   [clojure.lang.LispReader readDelimitedList "LispReader.java" 1398]
   [clojure.lang.LispReader$MapReader invoke "LispReader.java" 1355]
   [clojure.lang.LispReader read "LispReader.java" 285]
   [clojure.lang.LispReader readDelimitedList "LispReader.java" 1398]
   [clojure.lang.LispReader$VectorReader invoke "LispReader.java" 1347]
   [clojure.lang.LispReader read "LispReader.java" 285]
   [clojure.lang.LispReader readDelimitedList "LispReader.java" 1398]
   [clojure.lang.LispReader$ListReader invoke "LispReader.java" 1243]
   [clojure.lang.LispReader read "LispReader.java" 285]
   [clojure.lang.LispReader read "LispReader.java" 216]
   [clojure.lang.Compiler load "Compiler.java" 7631]
   [clojure.lang.RT loadResourceScript "RT.java" 381]
   [clojure.lang.RT loadResourceScript "RT.java" 372]
   [clojure.lang.RT load "RT.java" 459]
   [clojure.lang.RT load "RT.java" 424]
   [clojure.core$load$fn__6839 invoke "core.clj" 6126]
   [clojure.core$load invokeStatic "core.clj" 6125]
   [clojure.core$load doInvoke "core.clj" 6109]
   [clojure.lang.RestFn invoke "RestFn.java" 408]
   [clojure.core$load_one invokeStatic "core.clj" 5908]
   [clojure.core$load_one invoke "core.clj" 5903]
   [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
   [clojure.core$load_lib invokeStatic "core.clj" 5947]
   [clojure.core$load_lib doInvoke "core.clj" 5928]
   [clojure.lang.RestFn applyTo "RestFn.java" 142]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$load_libs invokeStatic "core.clj" 5985]
   [clojure.core$load_libs doInvoke "core.clj" 5969]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$require invokeStatic "core.clj" 6007]
   [clojure.core$require doInvoke "core.clj" 6007]
   [clojure.lang.RestFn invoke "RestFn.java" 436]
   [malli.instrument$eval8557$loading__6721__auto____8558
    invoke
    "instrument.clj"
    1]
   [malli.instrument$eval8557 invokeStatic "instrument.clj" 1]
   [malli.instrument$eval8557 invoke "instrument.clj" 1]
   [clojure.lang.Compiler eval "Compiler.java" 7177]
   [clojure.lang.Compiler eval "Compiler.java" 7166]
   [clojure.lang.Compiler load "Compiler.java" 7636]
   [clojure.lang.RT loadResourceScript "RT.java" 381]
   [clojure.lang.RT loadResourceScript "RT.java" 372]
   [clojure.lang.RT load "RT.java" 459]
   [clojure.lang.RT load "RT.java" 424]
   [clojure.core$load$fn__6839 invoke "core.clj" 6126]
   [clojure.core$load invokeStatic "core.clj" 6125]
   [clojure.core$load doInvoke "core.clj" 6109]
   [clojure.lang.RestFn invoke "RestFn.java" 408]
   [clojure.core$load_one invokeStatic "core.clj" 5908]
   [clojure.core$load_one invoke "core.clj" 5903]
   [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
   [clojure.core$load_lib invokeStatic "core.clj" 5947]
   [clojure.core$load_lib doInvoke "core.clj" 5928]
   [clojure.lang.RestFn applyTo "RestFn.java" 142]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$load_libs invokeStatic "core.clj" 5985]
   [clojure.core$load_libs doInvoke "core.clj" 5969]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$require invokeStatic "core.clj" 6007]
   [clojure.core$require doInvoke "core.clj" 6007]
   [clojure.lang.RestFn invoke "RestFn.java" 457]
   [malli.dev$eval7794$loading__6721__auto____7795 invoke "dev.clj" 1]
   [malli.dev$eval7794 invokeStatic "dev.clj" 1]
   [malli.dev$eval7794 invoke "dev.clj" 1]
   [clojure.lang.Compiler eval "Compiler.java" 7177]
   [clojure.lang.Compiler eval "Compiler.java" 7166]
   [clojure.lang.Compiler load "Compiler.java" 7636]
   [clojure.lang.RT loadResourceScript "RT.java" 381]
   [clojure.lang.RT loadResourceScript "RT.java" 372]
   [clojure.lang.RT load "RT.java" 459]
   [clojure.lang.RT load "RT.java" 424]
   [clojure.core$load$fn__6839 invoke "core.clj" 6126]
   [clojure.core$load invokeStatic "core.clj" 6125]
   [clojure.core$load doInvoke "core.clj" 6109]
   [clojure.lang.RestFn invoke "RestFn.java" 408]
   [clojure.core$load_one invokeStatic "core.clj" 5908]
   [clojure.core$load_one invoke "core.clj" 5903]
   [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
   [clojure.core$load_lib invokeStatic "core.clj" 5947]
   [clojure.core$load_lib doInvoke "core.clj" 5928]
   [clojure.lang.RestFn applyTo "RestFn.java" 142]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$load_libs invokeStatic "core.clj" 5985]
   [clojure.core$load_libs doInvoke "core.clj" 5969]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$require invokeStatic "core.clj" 6007]
   [clojure.core$require doInvoke "core.clj" 6007]
   [clojure.lang.RestFn invoke "RestFn.java" 408]
   [cljdoc_analyzer.metagetta.clojure$read_ns$fn__1614
    invoke
    "clojure.clj"
    92]
   [cljdoc_analyzer.metagetta.clojure$read_ns
    invokeStatic
    "clojure.clj"
    91]
   [cljdoc_analyzer.metagetta.clojure$read_ns invoke "clojure.clj" 86]
   [cljdoc_analyzer.metagetta.clojure$read_namespaces$fn__1627
    invoke
    "clojure.clj"
    160]
   [clojure.core$map$fn__5866 invoke "core.clj" 2755]
   [clojure.lang.LazySeq sval "LazySeq.java" 42]
   [clojure.lang.LazySeq seq "LazySeq.java" 51]
   [clojure.lang.Cons next "Cons.java" 39]
   [clojure.lang.RT boundedLength "RT.java" 1792]
   [clojure.lang.RestFn applyTo "RestFn.java" 130]
   [clojure.core$apply invokeStatic "core.clj" 665]
   [clojure.core$mapcat invokeStatic "core.clj" 2783]
   [clojure.core$mapcat doInvoke "core.clj" 2783]
   [clojure.lang.RestFn invoke "RestFn.java" 423]
   [cljdoc_analyzer.metagetta.clojure$read_namespaces
    invokeStatic
    "clojure.clj"
    160]
   [cljdoc_analyzer.metagetta.clojure$read_namespaces
    invoke
    "clojure.clj"
    122]
   [cljdoc_analyzer.metagetta.main$read_namespaces
    invokeStatic
    "main.clj"
    116]
   [cljdoc_analyzer.metagetta.main$read_namespaces
    invoke
    "main.clj"
    113]
   [cljdoc_analyzer.metagetta.main$get_metadata$fn__1814
    invoke
    "main.clj"
    147]
   [clojure.core$mapv$fn__8445 invoke "core.clj" 6912]
   [clojure.lang.ArraySeq reduce "ArraySeq.java" 111]
   [clojure.core$reduce invokeStatic "core.clj" 6827]
   [clojure.core$mapv invokeStatic "core.clj" 6903]
   [clojure.core$mapv invoke "core.clj" 6903]
   [cljdoc_analyzer.metagetta.main$get_metadata
    invokeStatic
    "main.clj"
    144]
   [cljdoc_analyzer.metagetta.main$get_metadata invoke "main.clj" 130]
   [cljdoc_analyzer.metagetta.main$_main invokeStatic "main.clj" 178]
   [cljdoc_analyzer.metagetta.main$_main invoke "main.clj" 153]
   [clojure.lang.AFn applyToHelper "AFn.java" 154]
   [clojure.lang.AFn applyTo "AFn.java" 144]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.core$apply invokeStatic "core.clj" 665]
   [clojure.main$main_opt invokeStatic "main.clj" 514]
   [clojure.main$main_opt invoke "main.clj" 510]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause "Invalid token: ::mg/keys"}}

Execution error at malli.instrument/eval8557$loading (instrument.clj:1).
Invalid token: ::mg/keys



Exited with code exit status 1

Exploration & Diagnosis

The crux seems to be:

Syntax error reading source at (malli/generator.cljc:55:14).

If we look at the source we see :as-alias is being used https://github.com/metosin/malli/blob/500bece4d1e3254b0ba7ac6108c8514ccbcb8623/src/malli/generator.cljc#L5-L15

When a lib does not specify a default version of Clojure/ClojureScript, cljdoc analyzer provides one.
Cljdoc analyzer is currently defaulting to 1.10.x versions, but :as-alias is a 1.11 feature.

Next Steps

I'll update cljdoc analyzer to default to current versions of Clojure and ClojureScript.

Consider returning args/options in resulting edn

Would be nice to know what :namespaces and :languages and any other options that were requested when looking a generated edn.

But remember.. if we include changing state we are no longer idempotent.

dev: add eastwood linting

Whenever I do interop, I like to add eastwood linting.
I'm about to do some more interop as part of #66.

Eastwood reports on deprecated usages in Java libs and reflection issues.

I won't bother with metagetta eastwood linting at this time.

Our dependency parsing is sometimes too naive

Currently

Cljdoc-analyzer naively uses jsoup (an HTML/XML parser) to parse the pom to learn about an artifact's dependencies.

Scenarios this can fail include:

  1. pom properties (ex. version is <version>${clojure.version}</version>
  2. managed dependencies (ex. version is specified under <dependencyManagement> but not under <dependencies>) (actually we do naively try to handle this).
  3. parent poms are not considered

Example problem: cljdoc/cljdoc#742

Analysis

We already use clojure tools deps to resolve dependencies.
It could resolve all of the above.

Except for one case.
Clojure tools deps, by design, filters out everything but runtime dependencies.

Cljdoc currently includes provided scope deps as a way for a library to specify optional dependencies. These are, by design, excluded by clojure tools deps.

Specific usages

The cljdoc analyzer currently uses naively parsed <dependencies> to:

  1. add pom provided deps to the classpath (it also, btw, includes system and test scope dependencies and deps marked as optional, I'll refer to all of these as provided)
  2. remove problematic deps
    1. those with an artifact name starting with boot- (not sure the history of this one, but whatevs for now)
    2. org.clojure/tools.reader to avoid conflicting with cljs tools.reader
  3. ensure key deps use recent versions, if not, automatically bump them
  4. ensure key deps are present, ex. javax.servlet/javax.servlet-api is always included because some enough libs assumed it was provided.

In all the above adjustments, we currently only look at deps specified in the analyzed lib pom.
We don't look at transitive deps.

Thoughts

We could simply take advantage of clojure tools deps resolving support.
Except that it does not include provided scope dependencies.

Scope Provided deps

Option 1: Hack tools deps to return provided deps

There might be just a couple of spots to alter-var-root.

Con: brittle, we'd be hacking tools deps internals and those are subject to change
Pro: might be easy in the near term

Option 2: Use pomegranate

Con: Deps resolution differs from tools deps, but we could use it solely to discover provided deps.
Former con: lib was stale with CVEs, but I recently did a maintenance pass to resolve that.
Pro: We write less code
Neutral: Depends on maven resolver libs, but so does clojure tools deps

Option 3: Use maven resolver directly

Pro: no extra deps
Con: java interop
Con: might be just duplicating something quite small that works already (pomegranate)

Current choice

I think I'll explore option 2 first and see where that leads.

Transitive deps?

Should we be looking at these too? (see "specific usages" above).

For provided deps we probably just want to check non-transitives.
For other deps fixups we could fixup transitive runtime deps, but we'll start with current behaviour of only looking at non-transitive deps.

Future-features

I'm not going to worry right now about our plan to someday build docs from git-sources and how decisions for this issue might impact that feature.

Upstream usage

Does the cljdoc web site want/need any of this information?
The work could be saved for cljdoc if it needs it.
We do save the raw pom and I do see naively parsed dependencies loaded by cljdoc... but I don't think this data is currently used.

org.clojure/clojure is failing to analyze

Raised cljdoc/cljdoc#595, will track org.clojure/clojure here.

Testing with Clojure 1.11.1

Issue 1 - class cleanup

We try to cleanup classes from jars that we analyze.
The attempt is to delete any __init.classes that have .clj or .cljc sources.

Our technique does not pickup classes print generated by definterface and proxy.
In the clojure jar, these are generated to:

  • /clojure/pprint/PrettyFlush.class
  • /clojure/pprint/proxy$java/io/Writer*.class

To check: did our analyzer handle this case once, or did Clojure change?

Options:

  • Include the above classes in deletion cleanup. Can we reliably determine that they should be deleted? Or will this be custom handling for clojure analysis?
    • Clojure does set .class source file for generated interfaces, we could check that with the asm library, this would handle cases like PrettyFlush.class
    • For proxies, clojure generates under a proxy$... dir, we could assume that a class under a dir that has a proxy$ prefix is generated by Clojure.

Issue 2 - jsr166y.jar

There is some deprecated Clojure code that requires jsr166y.jar.

When the analyzer tries to load this namespace, it fails because jsr166y.jar is not on the classpath.

To check: this code has been around forever, how did this ever work?

Aha. If I look at the Clojure 1.9.0 pom I see:

    <dependency>
      <groupId>org.codehaus.jsr166-mirror</groupId>
      <artifactId>jsr166y</artifactId>
      <version>1.7.0</version>
      <scope>provided</scope>
    </dependency>

This is missing starting from Clojure 1.10 onwards as part of CLJ-2363.

Options:

  1. include jsr166y.jar on the classpath when analyzing clojure.
    • I'm not finding a compatible jar on the web, I get an error for a missing jsr166y.forkjoin.ParallelArrayWithBounds
    • If I dig deeper, I find that jsr166y restructured their classes around here but the org.codehaus.jsr166-mirror release is way after that.
    • If I clone git clone [email protected]:codehaus/jsr166-mirror.git, then git reset 17a5e808a900c0c8976a5cf4a1201067a0834894 --hard then ant jsr166yjar (with JDK8) then include build/jsr166ylib/jsr166y.jar on the classpath when building clojure then all is good.
  2. exclude deprecated namespace from analysis (I think this is what Clojure team does? I don't see it documented in their API docs).

User facing command line will fail if artifact references additional maven repos

Problem

If an artifact references additional maven repositories, the convenience cmd line syntax we created for users will fail.

Repro

miikka/clj-branca references the center maven repo.

❮ clojure -M -m cljdoc-analyzer.main analyze --project miikka/clj-branca --version 0.1.0 --output-filename foo.edn     
** ERROR: **
Exception: #error {
 :cause Could not find artifact com.goterl.lazycode:lazysodium-java:jar:4.3.0 in central (https://repo1.maven.org/maven2/)
 :data {:lib com.goterl.lazycode/lazysodium-java, :coord {:mvn/version 4.3.0, :deps/manifest :mvn, :dependents [miikka/clj-branca], :parents #{[miikka/clj-branca]}}}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message Could not find artifact com.goterl.lazycode:lazysodium-java:jar:4.3.0 in central (https://repo1.maven.org/maven2/)
   :data {:lib com.goterl.lazycode/lazysodium-java, :coord {:mvn/version 4.3.0, :deps/manifest :mvn, :dependents [miikka/clj-branca], :parents #{[miikka/clj-branca]}}}
   :at [clojure.tools.deps.extensions.maven$get_artifact invokeStatic maven.clj 167]}]
 :trace
 [[clojure.tools.deps.extensions.maven$get_artifact invokeStatic maven.clj 167]
  [clojure.tools.deps.extensions.maven$get_artifact invoke maven.clj 155]
  [clojure.tools.deps.extensions.maven$eval3790$fn__3793 invoke maven.clj 178]
  [clojure.lang.MultiFn invoke MultiFn.java 244]
  [clojure.tools.deps$download_libs$fn__3368$fn__3369 invoke deps.clj 464]
  [clojure.tools.deps.util.concurrent$submit_task$task__2978 invoke concurrent.clj 35]
  [clojure.lang.AFn call AFn.java 18]
  [java.util.concurrent.FutureTask run FutureTask.java 264]
  [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1136]
  [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 635]
  [java.lang.Thread run Thread.java 833]]}

Work-around

Specify the extra-repro:

❯ clojure -M -m cljdoc-analyzer.main analyze --project miikka/clj-branca --version 0.1.0 --output-filename foo.edn \      
>   --extra-repo "jcenter https://jcenter.bintray.com"
2023-06-06 18:27:50,746 INFO  cljdoc-analyzer.runner - cljdoc-analyzer-version 1.0.731
2023-06-06 18:27:50,784 INFO  cljdoc-analyzer.runner - args:
{:exclude-with [:no-doc :skip-wiki :mranderson/inlined],
 :project "miikka/clj-branca",
 :version "0.1.0",
 :output-filename "foo.edn",
 :jarpath
 "/home/lee/.m2/repository/miikka/clj-branca/0.1.0/clj-branca-0.1.0.jar",
 :pompath
 "/home/lee/.m2/repository/miikka/clj-branca/0.1.0/clj-branca-0.1.0.pom",
 :extra-repos {"jcenter" {:url "https://jcenter.bintray.com"}},
 :languages nil}

2023-06-06 18:27:50,868 INFO  cljdoc-analyzer.runner - dependencies for analysis:
org.clojure/clojure 1.10.1
  org.clojure/core.specs.alpha 0.2.44
  org.clojure/spec.alpha 0.2.176
org.clojure/clojurescript 1.11.60
  com.cognitect/transit-java 1.0.362
    com.fasterxml.jackson.core/jackson-core 2.8.7
    org.msgpack/msgpack 0.6.12
      com.googlecode.json-simple/json-simple 1.1.1
      org.javassist/javassist 3.18.1-GA
    javax.xml.bind/jaxb-api 2.3.0
  org.clojure/google-closure-library 0.0-20211011-0726fdeb
    org.clojure/google-closure-library-third-party 0.0-20211011-0726fdeb
  org.clojure/tools.reader 1.3.6
  com.google.javascript/closure-compiler-unshaded v20220502
cljdoc-analyzer/metagetta /home/lee/proj/oss/cljdoc/cljdoc-analyzer/modules/metagetta
  org.clojure/tools.namespace 1.4.0
    org.clojure/java.classpath 1.0.0
javax.servlet/javax.servlet-api 4.0.1
miikka/clj-branca /tmp/cljdoc-miikka-clj-branca-0.1.0442218218102351566
  miikka/clj-base62 0.1.0
  byte-streams/byte-streams 0.2.4
    manifold/manifold 0.1.8
      org.clojure/tools.logging 0.3.1
      io.aleph/dirigiste 0.1.5
      riddley/riddley 0.1.14
    primitive-math/primitive-math 0.1.6
    clj-tuple/clj-tuple 0.2.2
  com.goterl.lazycode/lazysodium-java 4.3.0
    co.libly/resource-loader 1.3.7
    net.java.dev.jna/jna 5.5.0
    org.slf4j/slf4j-api 2.0.0-alpha1

2023-06-06 18:27:50,869 INFO  cljdoc-analyzer.runner - launching metagetta for: miikka/clj-branca languages: :auto-detect
2023-06-06 18:27:54,595 INFO  cljdoc-analyzer.runner - metagetta results:
exit-code 0
stdout:
 Args: {:namespaces :all,
        :root-path
        "/tmp/cljdoc-miikka-clj-branca-0.1.0442218218102351566/src/main/clojure",
        :languages :auto-detect,
        :output-filename "/tmp/miikka-clj-branca6578671240962030381.edn",
        :exclude-with [:no-doc :skip-wiki :mranderson/inlined]}
 Java version 17.0.7
 Clojure version 1.10.1
 ClojureScript version 1.11.60
 Analyzing for clj
 ⏱  Analysis for clj took: 1.3s
 Done
2023-06-06 18:27:54,604 INFO  cljdoc-analyzer.runner - results file: /home/lee/proj/oss/cljdoc/cljdoc-analyzer/foo.edn
2023-06-06 18:27:54,604 INFO  cljdoc-analyzer.runner - Analysis succeeded.

Should we bother fixing?

Probably. It is confusing. And...

The clojure tools invocation of cljdoc-analyzer is using the same technique to resolve the project dep and will fail in the same way.

This is more important because this invocation is used by (the now somewhat neglected) (https://github.com/cljdoc/cljdoc-check-action)[cljdoc-check-action].

Ideas

We'd like to resolve our jar and pom without resolving its deps.
Perhaps I can twist org.clojure/tools.deps's arm to do that?

Don't fail analysis due to unknown tagged literals

Currently, if the analyzed source code contains an unknown tagged literal, an exception is thrown and the analysis fails. However, most of the time, we don't need to have a perfect representation of the code, we only care about top-level functions and such so, 99.9% we shouldn't really care about those tagged literals and their values.

Possible solutions

1. Detect and apply data readers from the project

We know where data readers are declared, in data_readers.clj[c] at the top of the classpath, so we could check for that and apply those data readers, f.ex. by rebinding *data-readers* or by supplying the map of data readers to the reader that reads the code (if supported; if we use tools.reader, there is *data-readers*).

Pros: We would see the code that the runtime sees.

Cons: More work on our part.

2. Ignore unknown tagged literals without failing

Alternatively, we could define a fallback data reader that simply replaces everything unknown with nil, that would be I assume good enough for analyzing the code and simple.

From the docs:

If no data reader is found for a tag, the function bound in default-data-reader-fn will be invoked with the tag and value to produce a value. If default-data-reader-fn is nil (the default), a RuntimeException will be thrown.

Analysis of POM-only artifacts fails

See this build:
https://app.circleci.com/pipelines/github/cljdoc/builder/46360/workflows/bf4aca79-e747-4b37-9877-2f70ad3f4c7b/jobs/62735

2023-10-04 09:12:09,207 ERROR cljdoc-analyzer.runner - Could not find artifact com.kohlschutter.junixsocket:junixsocket-core:jar:2.8.0 in central (https://repo1.maven.org/maven2/)
2023-10-04 09:12:09,207 ERROR cljdoc-analyzer.runner - STDOUT
 nil
2023-10-04 09:12:09,207 ERROR cljdoc-analyzer.runner - STDERR
 nil

junixsocket-core is a POM-only dependency of unixsocket-http (whose analysis fails in the above build). Thus, there is no JAR in the Maven repository.

This is how the dependency is declared in project.clj:

[com.kohlschutter.junixsocket/junixsocket-core "2.8.0" :extension "pom"]

Verify that cljdoc-analyzer works from jar

I have verified that cljdoc-analyzer works when referenced via git url whi sha in deps.edn.

I have yet to verify that it works when bundled as a jar.

The somewhat unusual thing about cljdoc-analyzer is that it has the internal metagetta subproject.

Decide what to do with shared-utils

Currently some namespaces from cljdoc's shared-utils module are copied here. This means some code is duplicated and could get out of sync (not too likely or regularly but still possible).

Could we include shared-utils via a git dependency?

fails to analyze re-frame 0.12.0

clj -m cljdoc-analyzer.cljdoc-main '{:project "re-frame/re-frame", :version "0.12.0", :jarpath "https://repo.clojars.org/re-frame/re-frame/0.12.0/re-frame-0.12.0.jar", :pompath "https://repo.clojars.org/re-frame/re-frame/0.12.0/re-frame-0.12.0.pom", :repos {"clojars" {:url "https://repo.clojars.org/"}, "central" {:url "https://repo.maven.apache.org/maven2/"}}}'

Problem seems to be that it depends on Reagent and Reagent requires "react" but because we're only looking at re-frame's code we can't properly stub out react for analysis.

Handle string requires / foreign libs in CLJS

Via cljdoc/cljdoc#289

Some libraries like https://github.com/Lokeh/hx require "react" to be provided via foreign-libs / npm and analysis for them doesn't work because "react" can't be found. I think if we would provide a deps.cljs file that claims there is a foreign lib called "react" we should be able to get this to work. This assumes that for the analysis the contents of the underlying JS file are irrelevant, which I think is true.

This issue is also somewhat related: cljdoc/cljdoc#297

General handling of npm-deps / foreign-libs

@thheller outlined an approach here #5 (comment):

Some time ago I already created an example to "fake" string requires. Your code likely changed over time but the basic gist is that you call ana/parse-ns first which will tell you which strings were used. You than fake those into the compiler state with a dummy name and then proceed with the actual analyze call.

Language autodetect is detecting clj for some cljs only projects

Currently

Language analysis auto-detection is based on extensions on files in the analyzed jar file.

But...

Sometimes a cljs-only project will contain a jar that contains project.clj.
This file is typically under META-INF.

So...

I'll skip all files under META-INF when inferring languages.

cljdoc-analyzer is failing to analyze versions of zprint that it previously analyzed successfully

I am totally confused by a cljdoc problem I have encountered.

Presenting problem:

All of my cljdoc local trial runs, as well as cljdoc-analyzer runs, yield:

"Cannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null"

It appears to successfully analyze the clj code, but fail on doing the cljs analysis (entire output available later):

 Analyzing for clj
 Clojure version 1.11.1
 ⏱  Analysis for clj took: 1.803s
 Analyzing for cljs
 ClojureScript version 1.11.60
stderr:
 {:clojure.main/message
  "Execution error (NullPointerException) at sci.impl.copy-vars/var-meta (copy_vars.cljc:65).\nCannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null\n",

Background:

I routinely run cljdoc locally to ensure that my docs will work with cljdoc when I push them to GitHub. This generally works fine, and for instance I successfully ran cljdoc locally on zprint 1.2.7 prior to releasing it. It worked locally. And, indeed, when I pushed 1.2.7, cljdoc successfully analyzed my code and docs and you can see the docs for zprint 1.2.7 work fine online with cljdoc today.

I have been doing the same thing while working on zprint 1.2.8, and it was working fine. I last had it work on Sept 1. On Sept 9, I was unable to get the analysis of the code to work (received the above error), no matter what I tried. I tried a newer version of cljdoc locally, no luck. I had done very little to the code in the intervening period, so I was quite surprised. I tried a lot to things, all to no avail.

I then tried to work with older jar files, thinking I had messed something up. Still, the same problem.

I cloned the 1.2.7 version off of GitHub into a new directory, and tried that. Same problem.

I then looked at cljdoc issues. Nothing obvious. I stumbled into cljdoc-analyzer. Maybe this would help, or give better errors, or something?

For what it is worth, I'm running on macOS on MacBook Air M1, using:

clone/zprint % sdk use java 17.0.5-oracle

Using java version 17.0.5-oracle in this shell.

I installed it locally:

√ clone/zprint % clojure -Ttools install io.github.cljdoc/cljdoc-analyzer '{:git/tag "RELEASE"}' :as cljdoc
...
cljdoc: Installed io.github.cljdoc/cljdoc-analyzer RELEASE

Then I built the jar in the cloned version of zprint, 1.2.7, with

lein clean
lein install

and ran the analyzer on it:

√ clone/zprint % clojure -Tcljdoc analyze-local
2023-09-10 13:16:12,329 INFO  cljdoc-analyzer.deps-tool - args:
{:project "/Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint",
 :version "snapshot",
 :pompath "pom.xml",
 :jarpath "target/zprint-1.2.7.jar"}

2023-09-10 13:16:12,331 INFO  cljdoc-analyzer.runner - cljdoc-analyzer-version 1.0.771
2023-09-10 13:16:12,340 INFO  cljdoc-analyzer.runner - args:
{:project "/Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint",
 :version "snapshot",
 :jarpath "target/zprint-1.2.7.jar",
 :pompath "pom.xml",
 :extra-repos nil,
 :namespaces :all,
 :exclude-with [:no-doc :skip-wiki :mranderson/inlined],
 :output-filename
 "output-/Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint-snapshot.edn"}

2023-09-10 13:16:12,613 INFO  cljdoc-analyzer.runner - dependencies for analysis:
org.clojure/clojure 1.11.1
  org.clojure/core.specs.alpha 0.2.62
  org.clojure/spec.alpha 0.3.218
zprint /private/var/folders/f6/hc_hlh0n5n1gvpvnjnnrr6200000gn/T/cljdoc--Users-kkinnear-homeroot-kkinnear-clojure-clone-zprint-snapshot4711732175783906790
  rewrite-clj/rewrite-clj 1.1.47
  org.babashka/sci 0.7.39
    borkdude/edamame 1.1.17
    org.babashka/sci.impl.types 0.0.2
    borkdude/sci.impl.reflector 0.0.1
org.clojure/clojurescript 1.11.60
  com.cognitect/transit-java 1.0.362
    com.fasterxml.jackson.core/jackson-core 2.8.7
    org.msgpack/msgpack 0.6.12
      com.googlecode.json-simple/json-simple 1.1.1
      org.javassist/javassist 3.18.1-GA
    javax.xml.bind/jaxb-api 2.3.0
  org.clojure/google-closure-library 0.0-20211011-0726fdeb
    org.clojure/google-closure-library-third-party 0.0-20211011-0726fdeb
  org.clojure/tools.reader 1.3.6
  com.google.javascript/closure-compiler-unshaded v20220502
cljdoc-analyzer/metagetta /Users/kkinnear/.gitlibs/libs/io.github.cljdoc/cljdoc-analyzer/f1b29aff3d81f01f420f414983c50d87da907616/modules/metagetta
hiccup/hiccup 1.0.5
javax.servlet/javax.servlet-api 4.0.1

2023-09-10 13:16:12,613 INFO  cljdoc-analyzer.runner - launching metagetta for: /Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint languages: :auto-detect
2023-09-10 13:16:19,929 INFO  cljdoc-analyzer.runner - metagetta results:
exit-code 1
stdout:
 Args: {:namespaces :all,
        :root-path
        "/var/folders/f6/hc_hlh0n5n1gvpvnjnnrr6200000gn/T/cljdoc--Users-kkinnear-homeroot-kkinnear-clojure-clone-zprint-snapshot4711732175783906790/src/main/clojure",
        :languages :auto-detect,
        :output-filename
        "/var/folders/f6/hc_hlh0n5n1gvpvnjnnrr6200000gn/T/-Users-kkinnear-homeroot-kkinnear-clojure-clone-zprint7695835249934503793.edn",
        :exclude-with [:no-doc :skip-wiki :mranderson/inlined]}
 Java version 17.0.5
 Analyzing for clj
 Clojure version 1.11.1
 ⏱  Analysis for clj took: 1.803s
 Analyzing for cljs
 ClojureScript version 1.11.60
stderr:
 {:clojure.main/message
  "Execution error (NullPointerException) at sci.impl.copy-vars/var-meta (copy_vars.cljc:65).\nCannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null\n",
  :clojure.main/triage
  {:clojure.error/class java.lang.NullPointerException,
   :clojure.error/line 65,
   :clojure.error/cause
   "Cannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null",
   :clojure.error/symbol sci.impl.copy-vars/var-meta,
   :clojure.error/source "copy_vars.cljc",
   :clojure.error/phase :execution},
  :clojure.main/trace
  {:via
   [{:type clojure.lang.ExceptionInfo,
     :message
     "Could not generate ClojureScript documentation for zprint/core.cljc",
     :data {},
     :at
     [cljdoc_analyzer.metagetta.utils$default_exception_handler
      invokeStatic
      "utils.clj"
      124]}
    {:type clojure.lang.ExceptionInfo,
     :data
     {:clojure.error/source
      "file:/Users/kkinnear/.m2/repository/org/babashka/sci/0.7.39/sci-0.7.39.jar!/sci/impl/io.cljc",
      :clojure.error/line 55,
      :clojure.error/column 17,
      :clojure.error/phase :macroexpansion,
      :clojure.error/symbol sci.impl.copy-vars/copy-var},
     :at
     [cljs.analyzer$macroexpand_1_STAR_$fn__3704
      invoke
      "analyzer.cljc"
      4029]}
    {:type java.lang.NullPointerException,
     :message
     "Cannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null",
     :at
     [sci.impl.copy_vars$var_meta invokeStatic "copy_vars.cljc" 65]}],
...

and I get the same problem. When I look at my local cljdoc output for zprint 1.2.7, I see:

[API import failed in build #47](http://localhost:8000/builds/47)

but when I look at the "real" cljdoc online, for zprint 1.2.7, I see that it works fine.

I've attached the file log.txt which contains the entirety of the output from the above command: log.txt.

I am totally confused by this problem, and have no idea what could be wrong. Given that I successfully analyzed zprint 1.2.7 with my local copy of cljdoc running (where I run the sever locally), and now cannot successfully analyze that same code, I'm at a loss as to what could be the problem. I had upgraded babashka along the way, but tried downgrading it to a previous version -- no change.

I've probably done something that is causing this, but I have absolutely no idea what it could be. Any help would be appreciated.

Thanks!

Macros that are not loaded should be included in cljs analysis

Currently

While looking at cljdoc/cljdoc#543 I was reminded that
ClojureScript analysis only includes macros in .clj files if they happen to be loaded by other namespaces in a library.

But...

Since all Clojure macros are available to ClojureScript, all macros should be included under ClojureScript analysis.

But tell me Lee...

What if a library author wanted to exclude a macro from their API?

Well, the macro or macro namespace could be marked with ^:no-doc.

But if a library author wants to include a macro for ClojureScript but not Clojure?
Well, I think one way would be to use reader conditionals in a .cljc file like so:

(ns testing123.macros)

(defmacro cljs-doc-only #?(:clj {:no-doc true}) [x]
  `(println "ho" ~x))

(defmacro clj-doc-only #?(:cljs {:no-doc true}) [x]
  `(println "hey" ~x))

#?(:clj (defmacro clj-code-only [x]
          `(println "foo" ~x))
   :cljs (defmacro cljs-code-only [x]
           `(println "bar" ~x)))

So...

Does this mean we have to always do Clojure analysis to pick up macros in .clj files, even if the user has requested ClojureScript-only analysis?

Maybe. Not sure yet.

Older artifacts that refernce `http` maven repos will fail to build

In the old days

Folks referenced maven repos via http.

These days

For security reasons, this is no longer allowed, maven repos must be referenced via https.

Problem

We have older artifacts that reference maven repos via http.
These built fine back when this was legal, but re-analyzing these libraries will now fail.

Example: https://repo.clojars.org/babbage/babbage/0.1.0/babbage-0.1.0.pom references

<repository>
 <id>central</id>
 <url>http://repo1.maven.org/maven2/</url>
 <snapshots>
  <enabled>false</enabled>
 </snapshots>
 <releases>
  <enabled>true</enabled>
  </releases>
</repository>

So...

Could we not automatically upgrade any http specified maven repository urls to https?

Is using an older version of clojure important?

As I review the changes I have made I notice that the codox fork/branch specified clojure 1.9 as a dep.

Metagetta specifies latest clojure and clojurescript deps which makes me question deps.clj's work to ensure clojure/clojurescript are included and of recentish versions.

@martinklepsch do you know/remember of a technical need to ever analyze with older versions of clojure/clojurescript?

I'm guessing the idea was to try to stick with the version of clojure specified by the project being analyzed unless it was just too old?

Clojurescript reader behaves differently on macos and linux

Logged while migrating cljdoc analysis runner to cljdoc-analyzer for historical interest.

Background
The extended tests for cljdoc analysis runner include verification of orchestra v2018.11.07-1. When bringing over these tests to cljdoc-analyzer, I decided to add in verification of generated edn results. I setup expected edn by performing initial runs on macOS.

Expected behavior
Actual metadata edn matches expected edn for all extended tests.

Actual behavior
When on circleci, which we have setup to use linux, actual results do not match expected results for orchestra. The circleci results include extra publics for orchestra-cljs.spec.test.

Initial analysis
The circleci actual result looks correct, the macOS result looks incorrect.

What's going on?
The clojurescript reader processes one file at a time. It turns out that linux is processing files in a different order than macOS. It seems when two files return the same namespace the last namespace processed wins.

On macOS orchestra cljs source was processed in this order:

  1. orchestra/core.cljc
  2. orchestra/detail.cljc
  3. orchestra_cljs/spec/test.cljs
  4. orchestra_cljs/spec/test.cljc

And in my tests on a circleci docker image, this order

  1. orchestra/core.cljc
  2. orchestra/detail.cljc
  3. orchestra_cljs/spec/test.cljc
  4. orchestra_cljs/spec/test.cljs

Because test.cljs and test.cljc both specify namespace orchestra-cljs.spec.test and test.cljs includes macros from test.cljc, on linux we get both macros and fns, but on macOS we only get the macros.

We are not likely to see a similar problem for the clojure reader because it uses the clojure.tools.namespace which operates on a directory (or jar) rather than one file at a time.

Initial ideas
It seems to me that publics for duplicate namespaces should me merged when processing cljs.

Review clojure tools use and general use invokations

Context

I'm working on cljdoc/cljdoc-check-action#4.

Currently

We have 3 invocation paths for cljdoc-analyzer

  1. invoked by cljdoc itself via cljdoc-analyzer.cljdoc-main - never resolves lib
  2. invoked for general use via cljdoc-analyzer.main - always resolves lib
  3. invoked for general use as Clojure tool via cljdoc-analyzer.deps-tools
    1. analyze command - when requested, resolves lib
    2. analyze-local command - never resolves lib

The GitHub cljdoc check action uses the 3rd variant.

The differences between all of the above are confusing and hard to remember/maintain/explain.

Cljdoc usage...

Specifies all of: project version pompath and jarpath.

Both project and version are treated as trusted descriptors and not used to resolve the lib.

Why are project and version specified when they could be read from pom?
Maybe because cljdoc knows project and version and just passes them on?
Maybe for certain local test uses (cljdoc ingest maybe?), it needs to override these values?

Clojure Tools Invocation

When invoked as a Clojure tool, we have two commands: analyze and analyze-local.
But maybe we don't need the distinction?

  1. If project and version are specified (and jarpath and pompath are not) resolve the dep and automatically populate: jarpath, pompath.
  2. Otherwise
    1. If jarpath is not specified, search under ./target for a single matching candidate.
    2. If pompath is not specified, try to find a suitable pom in jar
      1. if project is specified use it to disambiguate find pom in jar META-INF/maven/$project /pom.xml.
      2. Otherwise match single pom.xml in jar and discover from pom: project, version
  3. extra-repos and output-filename supported regardless.

Ah. I see analyze command has a :download arg. It will only resolve the lib if this arg is specified. But if jarpath is not specified wouldn't you always need to resolve?

Note that jarpath and pompath can be local files or URLs.
Pompath is slurped and jarpath is downloaded if necessary.

I'll have to sleep on this.

Consolidation of convenience usages

When invoked for convenience, we have two separate entry point namespaces.

  1. cljdoc-analyzer.main
  2. cljdoc-analyzer.deps-tools

The first entry point takes some care to validate arguments via cli-matic.
The second does not validate args.

The first analyze command seems to sorta replicate analyze for 2nd, but the 2nd hardcodes exclude-with.
The first does not replicate analyze-local for the 2nd.

Consolidating both usages would help to ensure they stay in sync and that args are validated in the same way.
For example, I see languages did not get added deps tools usage.

Docs

I'll validate/update docs for as part of resolving this issue.

Cljdoc vs convenience usages

So... folks will typically use convenience invocations of cljdoc to diagnose issues with cljdoc analysis. The Clojure tools invocation is used by the GitHub check action. If convenience usages behaviour drifts too far from cljdoc usage, we are likely doing folks a disservice.

Listen to repo config delivered from cljdoc

As part of migration, I decided to ignore the repositories configured in cljdoc and rely on internal cljdoc-analyzer config only because the cljdoc repositories config is redundantly expressed in the internal cljdoc-analyzer config.

But...

If anyone is running cljdoc privately, they might want to override the default repositories configuration in cljdoc and would naturally expect cljdoc-analyzer to pick this up.

As I don't know if this use case is active/interesting, I really should probably just restore previous pre-migration behavior.

Exclude MrAnderson namespaces from analyzed projects

Exclude MrAnderson namespaces in analyzed projects as described cljdoc/cljdoc#287:

Some projects like cider/cider-nrepl include namespaces from other dependencies as copied files, to avoid dependency conflicts. These projects often use MrAnderson to copy those files into their source tree. Since v0.5.1 MrAnderson now adds some additional metadata that will allow us to identify files that have been included by it: benedekfazekas/mranderson@aec6b20

Because those files are usually not relevant to the user of a library we should not show them.

This will require some minor tweaks to our Codox fork here:

1. https://github.com/cljdoc/codox/blob/cljs-proper/codox/src/codox/reader/clojure.clj#L154

2. https://github.com/cljdoc/codox/blob/cljs-proper/codox/src/codox/reader/clojurescript.clj#L138

Note that the branch in this repo is cljs-proper. Docstrings of the affected functions should also be updated.

Related to #195 but a bit more specific.

Capture classpath isolation / runner approach in rationale

A big part of the cljdoc analyzer project is setting up the classpath for the project under analysis. Doing this the cljdoc analyzer:

  • comes without any dependencies that might affect dependency resolution and
  • handles common issues around project dependencies in the deps namespace

I think these things are worth capturing in the rationale as they are responsible for a much of the code beyond the metagetta module that actually extracts API information.

Consider a release workflow

Currently...

Cljdoc analyzer is referenced as a git dep via git long sha by cljdoc.
When a cljdoc developer wants a new version of cljdoc-analyzer to go live, they edit cljdoc to reference the appropriate long sha.

But now...

Cljdoc analyzer can be run as a tool.
Which will likely be referenced from a GitHub Action.

Observations

When using cljdoc-analyzer as a tool, a user will want to be using the same release as cljdoc uses in production.

We'll likely want to implement a release workflow.

Current work-around

@holyjak is tagging the cljoc-analyzer git repo with clojure-cli-experiment to support his GitHub Action work.

Ideas

@holyjak and I chatted and thought of:

  • Releasing cljdoc-analyzer to clojars. Folks wanting to use the current production release would use RELEASE as their :mvn/version.
    • This would probably be the way we'd go if it weren't for our odd little internal metagetta module which I think would end up maybe being a subjar? (I experimented with jarring cljdoc-analyzer ages ago, but don't entirely remember the result).
  • Git tagging the current production release with latest. This is a docker-inspired strategy that might work for us.

For both the above, we thought that having cljdoc-analyzer spit out an immutable version on launch would help with repeatability in any trouble-shooting.

Track test times on CircleCI

It would be nice to track how long individual tests take over time.

CircleCI allows test info to uploaded in the form of Junit test reports.

Kaocha has a Junit plugin that can write these reports.

I've done this is the past at one point on another project.
I'll do it for this project as well (unless someone else beats me to it!).

Metagetta tests fail after upgrading to ClojureScript v1.10.866

Step 1
Verify tests pass before making changes:

cd modules/metagetta
clojure -M:test

Step 2
Update ClojureScript version to 1.10.866 in modules/metagetta/deps.edn.

Rerun tests from modules/metagetta dir:

❯ clojure -M:test
unit:   100% [==================================================] 9/9

Randomized with --seed 1363283077
ERROR in cljdoc-analyzer.metagetta.specials-test/analyze-unknown-tagged-literal-test (errors.clj:34)
Uncaught exception, not in assertion.
Exception: clojure.lang.ExceptionInfo: /Users/lee/proj/oss/cljdoc/cljdoc-analyzer-master/modules/metagetta/test-sources-special/metagetta_test_special/unknown_tagged_literal.cljc [line 4, col 38] No reader function for tag uknown-tagged-literal.
{:type :reader-exception, :ex-kind :reader-error, :file "/Users/lee/proj/oss/cljdoc/cljdoc-analyzer-master/modules/metagetta/test-sources-special/metagetta_test_special/unknown_tagged_literal.cljc", :line 4, :col 38}
 at clojure.tools.reader.impl.errors$throw_ex.invokeStatic (errors.clj:34)
    clojure.tools.reader.impl.errors$throw_ex.doInvoke (errors.clj:24)
    ...
    clojure.tools.reader.impl.errors$reader_error.invokeStatic (errors.clj:40)
    clojure.tools.reader.impl.errors$reader_error.doInvoke (errors.clj:36)
    ...
    clojure.tools.reader.impl.errors$throw_unknown_reader_tag.invokeStatic (errors.clj:206)
    clojure.tools.reader.impl.errors$throw_unknown_reader_tag.invoke (errors.clj:205)
    clojure.tools.reader$read_tagged.invokeStatic (reader.clj:861)
    clojure.tools.reader$read_tagged.invoke (reader.clj:848)
    clojure.tools.reader$read_dispatch.invokeStatic (reader.clj:73)
    clojure.tools.reader$read_dispatch.invoke (reader.clj:68)
    clojure.tools.reader$read_STAR_.invokeStatic (reader.clj:925)
    clojure.tools.reader$read_STAR_.invoke (reader.clj:907)
    clojure.tools.reader$read_delimited.invokeStatic (reader.clj:198)
    clojure.tools.reader$read_delimited.invoke (reader.clj:191)
    clojure.tools.reader$read_list.invokeStatic (reader.clj:209)
    clojure.tools.reader$read_list.invoke (reader.clj:205)
    clojure.tools.reader$read_STAR_.invokeStatic (reader.clj:925)
    clojure.tools.reader$read_STAR_.invoke (reader.clj:907)
    clojure.tools.reader$read.invokeStatic (reader.clj:978)
    clojure.tools.reader$read.invoke (reader.clj:951)
    cljs.analyzer$forms_seq_STAR_$forms_seq___2869$fn__2870$fn__2871.invoke (analyzer.cljc:4290)
    cljs.analyzer$forms_seq_STAR_$forms_seq___2869$fn__2870.invoke (analyzer.cljc:4283)
    ...
    cljs.analyzer$parse_ns$fn__2891.invoke (analyzer.cljc:4409)
    cljs.analyzer$parse_ns.invokeStatic (analyzer.cljc:4372)
    cljs.analyzer$parse_ns.invoke (analyzer.cljc:4351)
    cljs.analyzer.api$parse_ns.invokeStatic (api.cljc:168)
    cljs.analyzer.api$parse_ns.invoke (api.cljc:151)
    cljs.analyzer.api$parse_ns.invokeStatic (api.cljc:164)
    cljs.analyzer.api$parse_ns.invoke (api.cljc:151)
    cljs.analyzer.api$parse_ns.invokeStatic (api.cljc:161)
    cljs.analyzer.api$parse_ns.invoke (api.cljc:151)
    cljdoc_analyzer.metagetta.clojurescript$get_string_dependencies$fn__4773.invoke (clojurescript.clj:166)
    ...
    cljdoc_analyzer.metagetta.clojurescript$get_string_dependencies.invokeStatic (clojurescript.clj:168)
    cljdoc_analyzer.metagetta.clojurescript$get_string_dependencies.invoke (clojurescript.clj:155)
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces.invokeStatic (clojurescript.clj:201)
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces.invoke (clojurescript.clj:172)
    cljdoc_analyzer.metagetta.main$read_namespaces.invokeStatic (main.clj:82)
    cljdoc_analyzer.metagetta.main$read_namespaces.invoke (main.clj:78)
    cljdoc_analyzer.metagetta.main$get_metadata$fn__4856.invoke (main.clj:111)
    ...
    cljdoc_analyzer.metagetta.main$get_metadata.invokeStatic (main.clj:116)
    cljdoc_analyzer.metagetta.main$get_metadata.invoke (main.clj:95)
    cljdoc_analyzer.metagetta.specials_test$analyze_special_source.invokeStatic (specials_test.clj:10)
    cljdoc_analyzer.metagetta.specials_test$analyze_special_source.invoke (specials_test.clj:6)
    cljdoc_analyzer.metagetta.specials_test$fn__4942.invokeStatic (specials_test.clj:15)
    cljdoc_analyzer.metagetta.specials_test/fn (specials_test.clj:13)
    kaocha.plugin.capture_output$capture_output_pre_test_hook$fn__2626$fn__2627.invoke (capture_output.clj:92)
    kaocha.type.var$eval4965$fn__4967$fn__4972.invoke (var.clj:30)
    kaocha.type.var$eval4965$fn__4967.invoke (var.clj:27)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2618$fn__2619.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2618.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:128)
    kaocha.testable$run.invoke (testable.clj:119)
    kaocha.testable$run_testable.invokeStatic (testable.clj:212)
    kaocha.testable$run_testable.invoke (testable.clj:157)
    kaocha.testable$run_testables.invokeStatic (testable.clj:225)
    kaocha.testable$run_testables.invoke (testable.clj:215)
    ...
    kaocha.type.ns$run_tests$fn__3529.invoke (ns.clj:21)
    ...
    kaocha.type.ns$run_tests.invokeStatic (ns.clj:21)
    kaocha.type.ns$run_tests.invoke (ns.clj:17)
    kaocha.type.ns$eval3547$fn__3548.invoke (ns.clj:64)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2618$fn__2619.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2618.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:128)
    kaocha.testable$run.invoke (testable.clj:119)
    kaocha.testable$run_testable.invokeStatic (testable.clj:212)
    kaocha.testable$run_testable.invoke (testable.clj:157)
    kaocha.testable$run_testables.invokeStatic (testable.clj:225)
    kaocha.testable$run_testables.invoke (testable.clj:215)
    kaocha.test_suite$run.invokeStatic (test_suite.clj:7)
    kaocha.test_suite$run.invoke (test_suite.clj:5)
    kaocha.type.clojure.test$eval4121$fn__4122.invoke (test.clj:21)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2618$fn__2619.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2618.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:128)
    kaocha.testable$run.invoke (testable.clj:119)
    kaocha.testable$run_testable.invokeStatic (testable.clj:212)
    kaocha.testable$run_testable.invoke (testable.clj:157)
    kaocha.testable$run_testables.invokeStatic (testable.clj:225)
    kaocha.testable$run_testables.invoke (testable.clj:215)
    kaocha.api$run$fn__3124$fn__3125$fn__3126.invoke (api.clj:137)
    ...
    kaocha.api$run$fn__3124$fn__3125.invoke (api.clj:111)
    ...
    kaocha.api$run$fn__3124.invoke (api.clj:110)
    ...
    kaocha.api$run.invokeStatic (api.clj:96)
    kaocha.api$run.invoke (api.clj:83)
    kaocha.runner$run.invokeStatic (runner.clj:133)
    kaocha.runner$run.invoke (runner.clj:74)
    kaocha.runner$_main_STAR_.invokeStatic (runner.clj:172)
    kaocha.runner$_main_STAR_.doInvoke (runner.clj:145)
    ...
    kaocha.runner$_main.invokeStatic (runner.clj:183)
    kaocha.runner$_main.doInvoke (runner.clj:181)
    ...
╭───── Test output ───────────────────────────────────────────────────────
│ Analyzing for cljs
╰─────────────────────────────────────────────────────────────────────────
9 tests, 9 assertions, 1 errors, 0 failures.

Notes

  1. An astute fellow on Slack noticed uknown-tagged-literal in the exception message looked like a spelling mistake.
    He was right, but it is inconsequential. The test is verifying that source can analyzed even when it has unknown tagged literals, it could have just as easily been named asdf-xx-wonka-woo to satisfy the test.
  2. I encountered an issue in rewrite-clj when running its tests with figwheel-main with ClojureScript v1.10.866, this turned out to be due to a regression in ClojureScript. So I am now, perhaps entirely inappropriately, biased in thinking this issue might be related or due to another issue in ClojureScript.
  3. Have reported issue in #cljs-dev channel in Clojurian Slack.

Metagetta tests fail after upgrading to ClojureScript 1.11.60

Step 1
Verify tests pass before making changes:

bb test-metagetta

All green.

Step 2
Update ClojureScript version to 1.11.60 in modules/metagetta/deps.edn.

Rerun tests:

❯ bb test-metagetta
[ TASK test-metagetta  ]---------------------------------------------------------
TASK test-regular 
>> Testing variant: test-sources
--- unit (clojure.test) ---------------------------
cljdoc-analyzer.metagetta.specials-test
  analyze-unknown-tagged-literal-test ERROR

cljdoc-analyzer.metagetta.main-test
  analyze-clj-code-test
  analyze-clj-and-cljs-code-test
  analyze-select-namespace-no-matches-test
  analyze-specify-namespaces-subset-test
  analyze-specify-namespaces-wildcard-test
  analyze-cljs-code-test
  analyze-clj-and-cljs-via-auto-detect-code-test

Randomized with --seed 1284455068

ERROR in cljdoc-analyzer.metagetta.specials-test/analyze-unknown-tagged-literal-test (utils.clj:118)
Uncaught exception, not in assertion.
Exception: clojure.lang.ExceptionInfo: Could not generate ClojureScript documentation for metagetta_test_special/unknown_tagged_literal.cljc
{}
 at cljdoc_analyzer.metagetta.utils$default_exception_handler.invokeStatic (utils.clj:118)
    cljdoc_analyzer.metagetta.utils$default_exception_handler.invoke (utils.clj:117)
    ...
    cljdoc_analyzer.metagetta.clojurescript$load_source.invokeStatic (clojurescript.clj:153)
    cljdoc_analyzer.metagetta.clojurescript$load_source.invoke (clojurescript.clj:135)
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces$source_loader__5781.invoke (clojurescript.clj:207)
    ...
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces.invokeStatic (clojurescript.clj:212)
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces.invoke (clojurescript.clj:171)
    cljdoc_analyzer.metagetta.main$read_namespaces.invokeStatic (main.clj:72)
    cljdoc_analyzer.metagetta.main$read_namespaces.invoke (main.clj:68)
    cljdoc_analyzer.metagetta.main$get_metadata$fn__5849.invoke (main.clj:103)
    ...
    cljdoc_analyzer.metagetta.main$get_metadata.invokeStatic (main.clj:100)
    cljdoc_analyzer.metagetta.main$get_metadata.invoke (main.clj:86)
    cljdoc_analyzer.metagetta.specials_test$analyze_special_source.invokeStatic (specials_test.clj:10)
    cljdoc_analyzer.metagetta.specials_test$analyze_special_source.invoke (specials_test.clj:6)
    cljdoc_analyzer.metagetta.specials_test$fn__5935.invokeStatic (specials_test.clj:15)
    cljdoc_analyzer.metagetta.specials_test/fn (specials_test.clj:13)
    kaocha.plugin.capture_output$capture_output_pre_test_hook$fn__2670$fn__2671.invoke (capture_output.clj:92)
    kaocha.type.var$eval5958$fn__5960$fn__5965.invoke (var.clj:30)
    kaocha.type.var$eval5958$fn__5960.invoke (var.clj:27)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662$fn__2663.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:129)
    kaocha.testable$run.invoke (testable.clj:120)
    kaocha.testable$run_testable.invokeStatic (testable.clj:211)
    kaocha.testable$run_testable.invoke (testable.clj:158)
    kaocha.testable$run_testables.invokeStatic (testable.clj:224)
    kaocha.testable$run_testables.invoke (testable.clj:214)
    ...
    kaocha.type.ns$run_tests$fn__3674.invoke (ns.clj:21)
    ...
    kaocha.type.ns$run_tests.invokeStatic (ns.clj:21)
    kaocha.type.ns$run_tests.invoke (ns.clj:17)
    kaocha.type.ns$eval3692$fn__3693.invoke (ns.clj:60)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662$fn__2663.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:129)
    kaocha.testable$run.invoke (testable.clj:120)
    kaocha.testable$run_testable.invokeStatic (testable.clj:211)
    kaocha.testable$run_testable.invoke (testable.clj:158)
    kaocha.testable$run_testables.invokeStatic (testable.clj:224)
    kaocha.testable$run_testables.invoke (testable.clj:214)
    kaocha.test_suite$run.invokeStatic (test_suite.clj:7)
    kaocha.test_suite$run.invoke (test_suite.clj:5)
    kaocha.type.clojure.test$eval5103$fn__5104.invoke (test.clj:21)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662$fn__2663.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:129)
    kaocha.testable$run.invoke (testable.clj:120)
    kaocha.testable$run_testable.invokeStatic (testable.clj:211)
    kaocha.testable$run_testable.invoke (testable.clj:158)
    kaocha.testable$run_testables.invokeStatic (testable.clj:224)
    kaocha.testable$run_testables.invoke (testable.clj:214)
    kaocha.api$run$fn__3169$fn__3173$fn__3174.invoke (api.clj:141)
    ...
    kaocha.api$run$fn__3169$fn__3173.invoke (api.clj:115)
    ...
    kaocha.api$run$fn__3169.invoke (api.clj:114)
    ...
    kaocha.api$run.invokeStatic (api.clj:99)
    kaocha.api$run.invoke (api.clj:86)
    kaocha.runner$run$fn__3232.invoke (runner.clj:132)
    ...
    kaocha.runner$run.invokeStatic (runner.clj:130)
    kaocha.runner$run.invoke (runner.clj:73)
    kaocha.runner$_main_STAR_.invokeStatic (runner.clj:176)
    kaocha.runner$_main_STAR_.doInvoke (runner.clj:144)
    ...
    kaocha.runner$_main.invokeStatic (runner.clj:187)
    kaocha.runner$_main.doInvoke (runner.clj:185)
    ...
Caused by: clojure.lang.ExceptionInfo: /Users/lee/proj/oss/cljdoc/cljdoc-analyzer-master/modules/metagetta/test-sources-special/metagetta_test_special/unknown_tagged_literal.cljc [line 4, col 38] No reader function for tag uknown-tagged-literal.
{:type :reader-exception, :ex-kind :reader-error, :file "/Users/lee/proj/oss/cljdoc/cljdoc-analyzer-master/modules/metagetta/test-sources-special/metagetta_test_special/unknown_tagged_literal.cljc", :line 4, :col 38}
 at cljs.vendor.clojure.tools.reader.impl.errors$throw_ex.invokeStatic (errors.clj:34)
    cljs.vendor.clojure.tools.reader.impl.errors$throw_ex.doInvoke (errors.clj:24)
    ...
    cljs.vendor.clojure.tools.reader.impl.errors$reader_error.invokeStatic (errors.clj:40)
    cljs.vendor.clojure.tools.reader.impl.errors$reader_error.doInvoke (errors.clj:36)
    ...
    cljs.vendor.clojure.tools.reader.impl.errors$throw_unknown_reader_tag.invokeStatic (errors.clj:205)
    cljs.vendor.clojure.tools.reader.impl.errors$throw_unknown_reader_tag.invoke (errors.clj:204)
    cljs.vendor.clojure.tools.reader$read_tagged.invokeStatic (reader.clj:871)
    cljs.vendor.clojure.tools.reader$read_tagged.invoke (reader.clj:858)
    cljs.vendor.clojure.tools.reader$read_dispatch.invokeStatic (reader.clj:73)
    cljs.vendor.clojure.tools.reader$read_dispatch.invoke (reader.clj:68)
    cljs.vendor.clojure.tools.reader$read_STAR_.invokeStatic (reader.clj:935)
    cljs.vendor.clojure.tools.reader$read_STAR_.invoke (reader.clj:917)
    cljs.vendor.clojure.tools.reader$read_delimited.invokeStatic (reader.clj:198)
    cljs.vendor.clojure.tools.reader$read_delimited.invoke (reader.clj:191)
    cljs.vendor.clojure.tools.reader$read_list.invokeStatic (reader.clj:209)
    cljs.vendor.clojure.tools.reader$read_list.invoke (reader.clj:205)
    cljs.vendor.clojure.tools.reader$read_STAR_.invokeStatic (reader.clj:935)
    cljs.vendor.clojure.tools.reader$read_STAR_.invoke (reader.clj:917)
    cljs.vendor.clojure.tools.reader$read.invokeStatic (reader.clj:988)
    cljs.vendor.clojure.tools.reader$read.invoke (reader.clj:961)
    cljs.analyzer$forms_seq_STAR_$forms_seq___3874$fn__3875$fn__3876.invoke (analyzer.cljc:4441)
    cljs.analyzer$forms_seq_STAR_$forms_seq___3874$fn__3875.invoke (analyzer.cljc:4437)
    ...
    cljs.analyzer$analyze_file$fn__3987.invoke (analyzer.cljc:4888)
    cljs.analyzer$analyze_file.invokeStatic (analyzer.cljc:4880)
    cljs.analyzer$analyze_file.invoke (analyzer.cljc:4838)
    cljs.analyzer$analyze_file.invokeStatic (analyzer.cljc:4852)
    cljs.analyzer$analyze_file.invoke (analyzer.cljc:4838)
    cljs.analyzer.api$analyze_file.invokeStatic (api.cljc:186)
    cljs.analyzer.api$analyze_file.invoke (api.cljc:171)
    cljdoc_analyzer.metagetta.clojurescript$analyze_file$fn__5762$fn__5763.invoke (clojurescript.clj:132)
    cljs.compiler$with_core_cljs.invokeStatic (compiler.cljc:1477)
    cljs.compiler$with_core_cljs.invoke (compiler.cljc:1466)
    cljs.compiler.api$with_core_cljs.invokeStatic (api.clj:47)
    cljs.compiler.api$with_core_cljs.invoke (api.clj:35)
    cljdoc_analyzer.metagetta.clojurescript$analyze_file$fn__5762.invoke (clojurescript.clj:132)
    cljdoc_analyzer.metagetta.clojurescript$analyze_file.invokeStatic (clojurescript.clj:127)
    cljdoc_analyzer.metagetta.clojurescript$analyze_file.invoke (clojurescript.clj:126)
    cljdoc_analyzer.metagetta.clojurescript$load_source.invokeStatic (clojurescript.clj:140)
    cljdoc_analyzer.metagetta.clojurescript$load_source.invoke (clojurescript.clj:135)
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces$source_loader__5781.invoke (clojurescript.clj:207)
    ...
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces.invokeStatic (clojurescript.clj:212)
    cljdoc_analyzer.metagetta.clojurescript$read_namespaces.invoke (clojurescript.clj:171)
    cljdoc_analyzer.metagetta.main$read_namespaces.invokeStatic (main.clj:72)
    cljdoc_analyzer.metagetta.main$read_namespaces.invoke (main.clj:68)
    cljdoc_analyzer.metagetta.main$get_metadata$fn__5849.invoke (main.clj:103)
    ...
    cljdoc_analyzer.metagetta.main$get_metadata.invokeStatic (main.clj:100)
    cljdoc_analyzer.metagetta.main$get_metadata.invoke (main.clj:86)
    cljdoc_analyzer.metagetta.specials_test$analyze_special_source.invokeStatic (specials_test.clj:10)
    cljdoc_analyzer.metagetta.specials_test$analyze_special_source.invoke (specials_test.clj:6)
    cljdoc_analyzer.metagetta.specials_test$fn__5935.invokeStatic (specials_test.clj:15)
    cljdoc_analyzer.metagetta.specials_test/fn (specials_test.clj:13)
    kaocha.plugin.capture_output$capture_output_pre_test_hook$fn__2670$fn__2671.invoke (capture_output.clj:92)
    kaocha.type.var$eval5958$fn__5960$fn__5965.invoke (var.clj:30)
    kaocha.type.var$eval5958$fn__5960.invoke (var.clj:27)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662$fn__2663.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:129)
    kaocha.testable$run.invoke (testable.clj:120)
    kaocha.testable$run_testable.invokeStatic (testable.clj:211)
    kaocha.testable$run_testable.invoke (testable.clj:158)
    kaocha.testable$run_testables.invokeStatic (testable.clj:224)
    kaocha.testable$run_testables.invoke (testable.clj:214)
    ...
    kaocha.type.ns$run_tests$fn__3674.invoke (ns.clj:21)
    ...
    kaocha.type.ns$run_tests.invokeStatic (ns.clj:21)
    kaocha.type.ns$run_tests.invoke (ns.clj:17)
    kaocha.type.ns$eval3692$fn__3693.invoke (ns.clj:60)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662$fn__2663.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:129)
    kaocha.testable$run.invoke (testable.clj:120)
    kaocha.testable$run_testable.invokeStatic (testable.clj:211)
    kaocha.testable$run_testable.invoke (testable.clj:158)
    kaocha.testable$run_testables.invokeStatic (testable.clj:224)
    kaocha.testable$run_testables.invoke (testable.clj:214)
    kaocha.test_suite$run.invokeStatic (test_suite.clj:7)
    kaocha.test_suite$run.invoke (test_suite.clj:5)
    kaocha.type.clojure.test$eval5103$fn__5104.invoke (test.clj:21)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662$fn__2663.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2662.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:129)
    kaocha.testable$run.invoke (testable.clj:120)
    kaocha.testable$run_testable.invokeStatic (testable.clj:211)
    kaocha.testable$run_testable.invoke (testable.clj:158)
    kaocha.testable$run_testables.invokeStatic (testable.clj:224)
    kaocha.testable$run_testables.invoke (testable.clj:214)
    kaocha.api$run$fn__3169$fn__3173$fn__3174.invoke (api.clj:141)
    ...
    kaocha.api$run$fn__3169$fn__3173.invoke (api.clj:115)
    ...
    kaocha.api$run$fn__3169.invoke (api.clj:114)
    ...
    kaocha.api$run.invokeStatic (api.clj:99)
    kaocha.api$run.invoke (api.clj:86)
    kaocha.runner$run$fn__3232.invoke (runner.clj:132)
    ...
    kaocha.runner$run.invokeStatic (runner.clj:130)
    kaocha.runner$run.invoke (runner.clj:73)
    kaocha.runner$_main_STAR_.invokeStatic (runner.clj:176)
    kaocha.runner$_main_STAR_.doInvoke (runner.clj:144)
    ...
    kaocha.runner$_main.invokeStatic (runner.clj:187)
    kaocha.runner$_main.doInvoke (runner.clj:185)
    ...
╭───── Test output ───────────────────────────────────────────────────────
│ Analyzing for clj
│ INFO [metagetta.utils] Beware: ns/file  metagetta-test-special.unknown-tagged-literal  includes the unknown tagged literal ` uknown-tagged-literal `, ignoring it and replacing the value with data. This should not influence the analysis unless the value is a top-level public var.
│ ⏱  Analysis for clj took: 0.015s
│ Analyzing for cljs
╰─────────────────────────────────────────────────────────────────────────
8 tests, 8 assertions, 1 errors, 0 failures.

Top 1 slowest kaocha.type/clojure.test (17.20694 seconds, 100.0% of total time)
  unit
    8.60347 seconds average (17.20694 seconds / 2 tests)

Top 2 slowest kaocha.type/ns (17.20062 seconds, 100.0% of total time)
  cljdoc-analyzer.metagetta.main-test
    1.53317 seconds average (10.73217 seconds / 7 tests)
  cljdoc-analyzer.metagetta.specials-test
    6.46845 seconds average (6.46845 seconds / 1 tests)

Top 3 slowest kaocha.type/var (10.76797 seconds, 62.6% of total time)
  cljdoc-analyzer.metagetta.specials-test/analyze-unknown-tagged-literal-test
    6.46617 seconds cljdoc_analyzer/metagetta/specials_test.clj:13
  cljdoc-analyzer.metagetta.main-test/analyze-clj-and-cljs-code-test
    2.44396 seconds cljdoc_analyzer/metagetta/main_test.clj:199
  cljdoc-analyzer.metagetta.main-test/analyze-select-namespace-no-matches-test
    1.85784 seconds cljdoc_analyzer/metagetta/main_test.clj:222
Error while executing task: test-regular
Error while executing task: test-metagetta

Notes

  • Symptom looks the same as #41

  • It has been a while since we updated ClojureScript let's retry with more releases:

    Version Result
    1.10.914 (what we are currently using) pass
    1.11.4 pass
    1.11.51 fail
    1.11.54 fail
    1.11.57 fail
    1.11.60 fail

Feedback / Comments

Hey @lread — thanks for your interest in working on this, I think we're pretty much on the same page. I tried to add some notes on the various parts you requested feedback on.

I'm also watching this repository now so feel free to use the issues for anything else that comes up.

Hope this all makes sense ✌️


Questions / Notes

  • I’m ok dropping the codox namespace prefix if we also remove any traces from the existing cljdoc codebase. I’ve grown to dislike that some stuff is just called codox-edn and the like. We could have more descriptive names that are less linked to an implementation detail (i.e. the fact that we’re using codox).
  • Generally integrating stuff back into Codox would be nice but at the same time codox is kind of working as intended and maybe also “done” so this idea of feeding things back into it might not really be that important.
  • The analyser should not skip no-doc stuff as you already pointed out. I think we should just return that as is and remove that at either of these two steps:
    • importing data into the database or
    • reading / rendering it
  • “just curious, is protocol :members → :type always :var?”
    • I don’t know, is it? 😝 I found the type stuff for :members a little confusing at first, I think :var might just be the general fallback as well?
  • Parallel Support: Not sure what you mean here, clojure.parallel failed to load which caused org.clojure/clojure to break analysis. Is that what you mean by parallel support?

Inputs

  • :language:auto-detect and/or array specification sounds great to me.
    • As an aside: we run stuff once for each language while generally this could also be done in a single run. This is an optimisation that I’ve meant to implement for a long time.
  • :root-paths & :source-paths — if the problem we’re solving with the analyzer is analysing source files I think we should only provide paths based on the jar root. Everything else can be done by looking at pom.xml and I think that’s probably not terribly important.
  • :namespace at some point the regex felt necessary but we’ve been fine without it now so I guess fine to skip it.
  • :exception-handler — I’d love to see cljdoc handle single namespaces that could not be analyzed more gracefully. Right now a single failed namespace will break the entire API analysis. I would suggest keeping this but not entirely sure if you were suggesting to remove it.
  • :writer OK to remove, if people want to call a function on the returned data they can just do that themselves :)
  • :exclude-vars I’d keep it around for now, not sure what for but maintenance cost seems low?

Outputs

Need to stay backwards compatible as you also noted. I’d be open to have more things in explicit columns but at the same time I’m not seeing a lot of gains.

Future

Proposal: I like option #3. It makes cljdoc-analyzer potentially also interesting as an ad hoc tool.

Agree. We could also integrate the various hardcoded fixes for some jars (missing deps etc) and thus have a shared place where the community can document what is needed to analyze a specific jar.

Generally cljdoc was also always meant to serve this raw analysis data through an API. This isn’t really dependent on the analyser and more on coming up with a nice API around it though.

Consider omitting blank docstrings

Currently

Dynamic analysis can generate :doc "\n" for fns without docstrings.

Some example code that does this:
https://github.com/juxt/bidi/blob/d1bfcc9c4fe247b8ada19cd7ee25acd81dd93f26/src/bidi/vhosts.clj#L13-L16

I think

A blank docstring adds no value to API docs.

And

It also adds noise to diffs when looking at static (see cljdoc/cljdoc#543) vs dynamic analysis.

So...

I think I'll have cljdoc-analyzer omit any blank docstrings from API analysis.
I'll test for blankness clojure.string/blank?.

Support excluding namespaces before analysis (internally for now)

Currently...

We carried over codox support for excluding namespaces via :namespaces option.

But...

This support excludes namespaces AFTER analysis. This does not help if you want to exclude a namespace because it will fail to analyze.

And...

In support of analyzing clojure itself, I need to exclude the clojure.parallel namespace.
See #53.

So...

Like I did with :no-doc namespace metadata, I'll exclude speicified :namespaces after namespaces are found, but before they are analyzed.

Note...

Including/excluding by :namespaces will remain internal for now.
I find the single option to both include and exclude namespaces requires too much expertise with regular expressions. When/if we expose this feature to cljdoc users, we'll review usability.

Continue analysing other namespaces after a failure

I noticed shadow-cljs imports were failing. Looking at a recent build, I could see the error was:

Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:476).\norg.graalvm.polyglot.PolyglotException$StackFrame\n",
  :clojure.main/triage
  {:clojure.error/class java.lang.ClassNotFoundException,
   :clojure.error/line 476,
   :clojure.error/cause
   "org.graalvm.polyglot.PolyglotException$StackFrame",
   :clojure.error/symbol java.net.URLClassLoader/findClass,
   :clojure.error/source "URLClassLoader.java",
   :clojure.error/phase :execution},

Even if we can't analyse that one namespace, there are probably other namespaces that could be analysed. It seems like it would be good to have an option to keep analysing other namespaces. At the end of the analysis, cljdoc could still show that there were errors, and in the namespace sidebar show the failed namespace in red.

This would let the rest of the library continue to be analysed, even if one part fails.

I'm not well versed in how this all works, so maybe this isn't possible/a good idea.

Consider MrAnderson for isolation

We currently achieve dependency isolation by launching metagetta as a subprocess.

Another option to consider is using MrAnderson.

Cljdoc-analyzer would still be launched as a subprocess by cljdoc, but metagetta could simply be an inprocess call.

Metagetta analysis duration reporting can throw

While helping out getting a sensible cljdoc preview workflow up and running for polylith, an error in metegetta time reporting was uncovered.

The symptom exhibited like so:

2023-07-25 03:45:33,145 ERROR cljdoc-analyzer.runner - STDERR
 {:clojure.main/message
 "Execution error (IndexOutOfBoundsException) at java.util.regex.Matcher/start (Matcher.java:482).\nNo group 1\n",
 :clojure.main/triage
 {:clojure.error/class java.lang.IndexOutOfBoundsException,
  :clojure.error/line 482,
  :clojure.error/cause "No group 1",
  :clojure.error/symbol java.util.regex.Matcher/start,
  :clojure.error/source "Matcher.java",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type java.lang.IndexOutOfBoundsException,
    :message "No group 1",
    :at [java.util.regex.Matcher start "Matcher.java" 482]}],
  :trace
  [[java.util.regex.Matcher start "Matcher.java" 482]
   [java.util.regex.Matcher
    appendExpandedReplacement
    "Matcher.java"
    1090]
   [java.util.regex.Matcher appendReplacement "Matcher.java" 997]
   [java.util.regex.Matcher replaceAll "Matcher.java" 1181]
   [clojure.string$replace invokeStatic "string.clj" 106]
   [clojure.string$replace invoke "string.clj" 75]
   [cljdoc_analyzer.metagetta.main$get_metadata$fn__1673
    invoke
    "main.clj"
    146]
   [clojure.core$mapv$fn__8535 invoke "core.clj" 6979]
   [clojure.lang.ArraySeq reduce "ArraySeq.java" 114]
   [clojure.core$reduce invokeStatic "core.clj" 6885]
   [clojure.core$mapv invokeStatic "core.clj" 6970]
   [clojure.core$mapv invoke "core.clj" 6970]
   [cljdoc_analyzer.metagetta.main$get_metadata
    invokeStatic
    "main.clj"
    144]
   [cljdoc_analyzer.metagetta.main$get_metadata invoke "main.clj" 130]
   [cljdoc_analyzer.metagetta.main$_main invokeStatic "main.clj" 178]
   [cljdoc_analyzer.metagetta.main$_main invoke "main.clj" 153]
   [clojure.lang.AFn applyToHelper "AFn.java" 154]
   [clojure.lang.AFn applyTo "AFn.java" 144]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.main$main_opt invokeStatic "main.clj" 514]
   [clojure.main$main_opt invoke "main.clj" 510]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause "No group 1"}}
Execution error (IndexOutOfBoundsException) at java.util.regex.Matcher/start (Matcher.java:482).
No group 1

I diagnosed this to be a faulty regex in converting analysis duration.
I think it would fail for any duration with multiple units, ex. PT1M24S
(Which means this bug was likely obscuring the real cause of failure for polylith because analysis taking over 60 seconds is rather a long time).

Will follow up with PR.

Clojure reader not sorting by namespace

Logged while migrating cljdoc analysis runner to cljdoc-analyzer for historical interest.

Actual behavior
Clojure reader is not sorting results by namespace.

Expected behavior
Clojure reader should sort results by namespace to:

  1. be consistent with the ClojureScript reader
  2. provide the same output for the the same inputs to support testing

Cljdoc-analyzer can fail mysteriously

This issue originated from a diagnosis session on Slack with @mccraigmccraig.

The ingestion of alia on cljdoc was failing with less than helpful error messaging.

Here's what was showing on circleci:

WARNING: When invoking clojure.main, use -M
2020-12-22 18:10:17,140 INFO  cljdoc-analyzer.cljdoc-main - args:
{:project "cc.qbits/alia",
 :version "5.0.0-alpha1",
 :jarpath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar",
 :pompath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.pom",
 :repos
 {"clojars" {:url "https://repo.clojars.org/"},
  "central" {:url "https://repo.maven.apache.org/maven2/"}}}

2020-12-22 18:10:17,155 INFO  cljdoc-analyzer.runner - args:
{:project "cc.qbits/alia",
 :version "5.0.0-alpha1",
 :jarpath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar",
 :pompath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.pom",
 :extra-repos nil,
 :namespaces :all,
 :exclude-with [:no-doc :skip-wiki],
 :output-filename
 "/tmp/cljdoc/analysis-out/cljdoc-edn/cc.qbits/alia/5.0.0-alpha1/cljdoc.edn"}

2020-12-22 18:10:17,160 INFO  cljdoc-analyzer.runner - Downloading https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - nil
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - STDOUT
 nil
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - STDERR
 nil

We managed to reproduce this failure locally and after some trial and error we noticed that alia's lein project was generating a pom with dependencyManagement that included org.clojure/clojure. The dependencies section later included, as you might expect, org.clojure/clojure without any specific version.

Here's a minimal project.clj to illustrate. I've included an additional arbritray dependency to demonstrate a point.

(defproject lread/repro-odd-cljdoc-failure "1.0.0"
  :description "Trying to reproduce cljdoc failure"

  ;; it seems to be caused by include clojure as a managed dependency
  :managed-dependencies [[org.clojure/clojure "1.10.1"]
                         [version-clj/version-clj "0.1.2"]]

  ;; and referencing it without a version
  :dependencies [[org.clojure/clojure]
                 [version-clj/version-clj]])

We'll include a dummy namespace under src/my_stuff/api.clj just so we have something to analyze:

(ns my-stuff.api
  (:require [version-clj.core :refer [version-compare]]))

(defn oh-hi []
  ;; just referencing a lib for the sake of referncing a lib
  (version-compare "1.0.0" "1.0.0"))

Steps to reproduce:

1. install locally
lein install

2. run cljdoc-analyzer
I ran from source:

clojure -m cljdoc-analyzer.main analyze \
  --project  lread/repro-odd-cljdoc-failure --version 1.0.0 \
  --output-filename "/tmp/blarp.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki 

result

WARNING: When invoking clojure.main, use -M
2020-12-23 15:03:56,682 INFO  cljdoc-analyzer.runner - args:
{:project "lread/repro-odd-cljdoc-failure",
 :version "1.0.0",
 :exclude-with [:no-doc :skip-wiki],
 :output-filename "/tmp/erp.edn",
 :jarpath
 "/Users/lee/.m2/repository/lread/repro-odd-cljdoc-failure/1.0.0/repro-odd-cljdoc-failure-1.0.0.jar",
 :pompath
 "/Users/lee/.m2/repository/lread/repro-odd-cljdoc-failure/1.0.0/repro-odd-cljdoc-failure-1.0.0.pom",
 :extra-repos {}}

2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - nil
2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - STDOUT
 nil
2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - STDERR
 nil

work-around
If we put explicitly specified a version for org.clojure/clojure under dependencies things analyze just fine.

  :dependencies [[org.clojure/clojure "1.10.1"]
                 [version-clj/version-clj]])

And rerun lein install
Followed by cljdoc-analyzer (see step 2 above).
Then the analysis succeeds.
Notice that version-clj/version-clj could remain maven version free.

thoughts
We don't know why the ingest failed, but we do know:

  • dependencyManagent followed by versionless dependencies do work for deps other than org.clojure/clojure
  • the error message that cljdoc-analyzer/cljdoc produced was less than spectacular.

We can at least report the symptom, which I suspect might be "no dependencies found".

Fully qualify deps in deps.edn

The newer versions of Clojure CLI keep nagging us to do this, so might as well do this.

Will follow up with a commit.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.