Try to switch to Solr 9.x instead of 8.x

[treygrainger]

Try to switch to Solr 9.x instead of 8.x

[epugh]

I am glad that you wrote the examples using the JSON DSL, this is going to help the community migrate to this syntax... I'm contemplating rewriting the entire ref guide to feature the JSON syntax instead of the older one ;-).....

[treygrainger]

Will plan to upgrade to latest 8.x version prior to final book release. There were a LOT of changes in Solr 9 that could significantly change the way some things work. Will probably also update to Solr 9, but likely after the book is out.

In general, most version-specific code is encapsulated in helper functions in python (it's mostly around setup / schema / collection management - the actual API usage for searching is typically very stable across Solr versions), so likely there won't be much that's actually in the printed book that changes with the Solr 9 upgrade, hence that could be done in the code repo post-release if needed.

I'll take a shot at the Solr 9 upgrade prior to the final release, but I'm anticipating potentially a lot of changes being needed due to the way some things were modularized / removed in Solr 9. TBD.

[alexott]

Let try to upgrade to the latest 8.x, we can try 9.x after the release

[epugh]

Interested bystander here ;-). If there are any "rough spots" that you see in Solr that you've identified through the book writing effort, I'd be very interested in hearing about them and getting them sorted. Help me help you make the migration ;-).

[treygrainger]

Thank, Eric. Will do!

[treygrainger]

@epugh I attempted an upgrade to 8.11.2, and all initially seem to have gone smoothly.

I attempted to upgrade to 9.4.0, however, and queries are busted. Looks like something whacky happened with the JSON query DSL no longer supporting the defType parameter and so many of the queries throughout the book (which all post JSON queries and use defType=edismax) are returning zero results.

I added some debugging lines, and get this:

query = "ipad"

collection = "products"
request = {
    "query": query,
    "fields": ["upc", "name", "manufacturer", "score"],
    "limit": 5,
    "params": {
      "qf": "name manufacturer longDescription",
      "defType": "edismax",
      "indent": "true",
      "sort": "score desc, upc asc",
      "echoParams": "all"
    }
}

print(requests.post(f"{SOLR_URL}/{collection}/select", json=request).json())

response:

{
   "responseHeader":{
      "zkConnected":true,
      "status":0,
      "QTime":2,
      "params":{
         "q":"ipad",
         "defType":"lucene",
         "df":"_text_",
         "qf":"name manufacturer longDescription",
         "indent":"true",
         "echoParams":"all",
         "fl":[
            "upc",
            "name",
            "manufacturer",
            "score"
         ],
         "json":"{\"query\": \"ipad\", \"fields\": [\"upc\", \"name\", \"manufacturer\", \"score\"], \"limit\": 5, \"params\": {\"qf\": \"name manufacturer longDescription\", \"defType\": \"edismax\", \"indent\": \"true\", \"sort\": \"score desc, upc asc\", \"echoParams\": \"all\", \"useParams\": \"defType\"}}",
         "sort":"score desc, upc asc",
         "rows":"5",
         "rid":"aips-solr-2612"
      }
   },
   "response":{
      "numFound":0,
      "start":0,
      "maxScore":0.0,
      "numFoundExact":true,
      "docs":[
         
      ]
   }
}

Seems that the defType is now forced to use lucene and so the json query DSL is busted. Looks like it's documented here: https://issues.apache.org/jira/browse/SOLR-16916

I'll try downgrading to Solr 9.1 or so to see if I can get it working, but short of maybe rewriting all the JSON DSL queries throughout the book (which is what most of the book uses to express query syntax) to use a different syntax that overwrites with localparams, it doesn't looks to me like the latest version of Solr 9x is going to work well.

[treygrainger]

Reproduction steps:

1. Pull branch with Solr 9.4.0 upgrade:
   git checkout "solr-9x-upgrade"

2. Build and start docker container (will take a while):
   cd docker
docker-compose up --build

3. Index the dataset:
   --Index data: http://localhost:8888/notebooks/ch4/1.ch4-setting-up-the-retrotech-dataset.ipynb (run whole notebook)

4. On the first listing (4.5), search fails and returns zero results:
   -- http://localhost:8888/notebooks/ch4/2.ch4-signals-boosting.ipynb

You can replace the content of the listing with my python code from the last comment and see the echoParams output demonstrating the problem.

[treygrainger]

Looks like this issue has a pending fix here that just needs to be merged and released in a Solr bug fix version: apache/solr#2045

[treygrainger]

I was just able to successfully run with both Solr 9.1 and 9.3, so looks like this defType issue was newly introduced into 9.4.0. I'll focus my upgrade efforts on 9.3.0 for now to test out other functionality and plan to upgrade to a newer version once the issue is fixed.