Wrong numbers assigned when breaking bibliography into segments with multiple citation orders

[plk]

Wrong numbers assigned when breaking bibliography into segments with multiple citation orders

[aboruvka]

Just to add on to this: defernumbers doesn't reassign labelnumber when one or more bibliographies use the sorting option. Related options that need defernumbers enabled also don't work.

\documentclass{article}
\usepackage[defernumbers,sortcites,sorting=none]{biblatex}
\addbibresource{biblatex-examples.bib}
\begin{document}
\cite{bertram,glashow,knuth:ct:b,knuth:ct:a}
%\cite{bertram,weinberg,glashow,knuth:ct:b,knuth:ct:a}
\printbibliography[title={Articles},type=article,sorting=ynt]
\printbibliography[title={Books},type=book,sorting=ydnt]
%\printbibliography[title={Books},type=book,sorting=ydnt,omitnumbers]
%\printbibliography[title={Books},type=book,sorting=ydnt,resetnumbers]
%\printbibliography[title={Books},type=book,sorting=ydnt,prefixnumbers={B}]
\end{document}

@plk: The local labelnumber written to the aux file is correct. The labelnumber printed in the bibliography is the one retrieved with getdata. This always seems to follow the assignment based on the last sort order appearing in the bbl file. Here it would be ynt.

[plk]

@aboruvka So the problem is only when using defernumbers?

[aboruvka]

@plk Sorry - a correction. Without defernumbers enabled you get the "Setting 'defernumbers=true' recommended" warning for all cases in the above example. For the first case (no use of omitnumbers, resetnumbers, prefixnumbers) the labelnumbers follow the assignment in the order of the bibliography, not cite order. So I guess the case without defernumbers isn't handled correctly either.

[plk]

It's not clear to me what the example above is supposed to look like without defernumbers? I changed the sorting of the books to make it more obvious in the example and ran without defernumbers. The output looks ok to me? The most obvious problem is, with defernumbers, nothing changes ...

[aboruvka]

Without defernumbers enabled, labelnumber assignment should follow the load-time sorting scheme. I've changed it to cite order in the example. Regardless of the defernumbers setting the labelnumber assignment always follows the last scheme appearing in the bbl file. In the example it's ynt.

[plk]

I think I have an idea why. I hadn't realised how tricky list-local sorting makes this. It's because the entries are read multiple times with the last .bbl sorting list section overriding everything else.

[plk]

I have committed a patch to try to address this - can someone have a test? It needs updated biber 1.3 from SF too which ensures that the last sortlist in the .bbl is the global sortlist (this is an easy fix for the case without defernumbers). biblatex was resetting all data for an entry at \entry which only makes sense if you know that each entry is listed once. In the multi-sorting output, each entry is output multiple times in each sorted list and so \entry is run multiple times but the .aux is only read once, at the begining. So, the localnumber value was erased on the second read of the same entry. This was the bug. It's not clear to me that we need to empty the data for an entry at \entry as it will always be the same anyway and we need to preserve things that come only once from the .aux.

[aboruvka]

Seems to have a persistent rerun warning. The labelnumbers switch back and forth between the incorrect and correct ones. I'm not sure if I'm testing with all the right files, though - I'm using biblatex built from the script available over here and the dev biber binary from SF (dated Sept. 11).

[plk]

I just noticed this too. Looking into it.

[plk]

Try it again. We also need to force biblatex to always write the localnumbers to the .aux - just pull 2.3 dev again.

[aboruvka]

Everything looks good on a clean run (no aux file). After making changes that should alter the existing labelnumbers (e.g. new citations, change in option settngs), the labelnumbers in the aux file persist. I updated the example so you can try this out.

[plk]

Can you pull 2.3 dev again and test? After a lot of thrashing around, it seems to work now by removing the checksum creation for localnumber changes which otherwise makes everything just thrash around internally.

[aboruvka]

Running with this document body first:

\cite{bertram,glashow,knuth:ct:b,knuth:ct:a}
\printbibliography[title={Articles},type=article,sorting=ynt]
\printbibliography[title={Books},type=book,sorting=ydnt]

And then adding an entry:

\cite{bertram,weinberg,glashow,knuth:ct:b,knuth:ct:a}
\printbibliography[title={Articles},type=article,sorting=ynt]
\printbibliography[title={Books},type=book,sorting=ydnt]

I get persistent rerun messages. The labelnumbers in the bibliography switch back and forth between 1, 5, 2, 3, 4 and 3, 1, 2, 4, 5.

[plk]

Hmm, this is a really tough problem. Now I know what that particular checksum is about though ...

[aboruvka]

Yes, it is tough. Perhaps Joseph could help with this one? I can't get my head around syncing the local and label numbers with the current bbl format.

[plk]

The problem is not so much the cycling .aux numbers ... as long as the "re-run" messages stop when the numbers are correct. It's the interaction between the logic of when to write the .aux localnumbers and when to recalculate them when they exist that is tricky.

[josephwright]

@aboruvka I'll take a look at the weekend

[aboruvka]

@josephwright Any luck?

[plk]

I had another quick look this week and managed to get it almost working but it's very tricky and depends on the code which decides when to issue the the rerun messages. I'll try to look at it again next week.

[aboruvka]

@plk OK. Sorry I can't pitch in on fixing this one. I had a look awhile back, but only learned that a fix would be beyond my ability.

[plk]

@aboruvka - I have committed another attempt. Now it seems ok with adding a citation. It still needs another unprompted run or .aux file deletion when removing a citation ... but isn't that normal anyway? I could probably make it prompt to re-run biber on citation changes which reduce the citations if necessary? I don't think it ever did that though.

[aboruvka]

@plk I just tested all of the defernumbers-related options and everything looks fine, which is great as this one seemed really tough to fix. About entry deletion - that is indeed normal/expected behaviour. It takes another biber run to remove the uncited entry from the bbl file. As long as it's in there it'll get printed in the bibliography. At and prior to 1.7 the user isn't prompted to run latex or biber/bibtex. So this seems OK to me.

[plk]

Ok good - I may just see if I can get it to prompt to re-run biber on citation deletion - I have an idea and it would be a nice feature ...

[aboruvka]

That would be awesome. With really long bibliographies it is tough for the user to catch extraneous entries.

[plk]

@aboruvka Try pulling again now - I think I have citation deletion re-run messages working. It needs .aux file support - there is no way to do this otherwise.

[aboruvka]

It works, but I can't be sure about unintended side effects. So I tried running the test script. Example 10 (references per section) has a persistent rerun warning. All the other examples look OK.

[plk]

Ah, ok, thanks. I haven't run the test script yet - I'll check that.

[plk]

@aboruvka - Can you try again and test a little bit? All tests pass now and I think it's still fixed ...

[aboruvka]

Results with your run-of-the-mill edits look great as far as I can see. With less obvious changes I encountered some minor issues - all of them can be resolved by re-running biber and LaTeX. Going from the body:

\cite{bertram,weinberg,glashow,knuth:ct:b,knuth:ct:a}
\printbibliography[title={Articles},type=article,sorting=ynt]

to:

\cite{bertram,weinberg,glashow,knuth:ct:b,knuth:ct:a}
\printbibliography[title={Articles},type=article,sorting=ynt]
\printbibliography[title={Books},type=book,sorting=ydnt]

gives a second bibliography that is persistently empty.

Introducing a new sorting scheme in an existing bibliography also generates a persistenly empty bibliography. Going from:

\cite{bertram,weinberg,glashow,knuth:ct:b,knuth:ct:a}
\printbibliography[title={Articles},type=article,sorting=ynt]
\printbibliography[title={Books},type=book,sorting=ydnt]

to:

\cite{bertram,weinberg,glashow,knuth:ct:b,knuth:ct:a}
\printbibliography[title={Articles},type=article,sorting=ynt]
\printbibliography[title={Books},type=book,sorting=nty]

is one example.

In 18-numeric-hybrid.tex use an additional sorting scheme:

\printbibliography[heading=subbibliography,title={Online References},
  env=online,type=online,sorting=none]

After compiling this document, try swapping the order of the online citations:

\citeonline{markey},
\citeonline{ctan}

The old labelnumbers persist.

[plk]

Ah, ok, this is right, I need to look at this - the sorting schemes used to generate lists are passed in the .bcf so you do need to re-run biber if changing the scheme or keeping the "none" scheme and changing the order of cites. I'm glad I didn't realise all this before I started looking at this, it would have been overwhelming ...

[plk]

@aboruvka - I think I've fixed these two things now - can you check? It's nice to catch these - they are all oversights from the move to the 2.0 code and how the multiple sort lists functionality interacts with defernumbers and re-run messages in general. In general, if the user add or removed any sorting specifications anywhere, biber should be re-run so I've put in a check for this. sorting=none is a special case since a re-run is required if cites change order so there is a special check for this too now.

[aboruvka]

@plk Looks good!

[plk]

Good, I'll close this finally and hope nothing else crops up ...

[nokyotsu]

Great! Just wanted to send a quick thanks to all for your hard work on getting this issue fixed. How long would it take the update to propagate to, say, TeX Live? I would like to update the answer on the original question at TeX.SE which originated this issue.

[plk]

I am planning a new biblatex+biber release for by the end of the month and this means it will be in TL a few days after the release. I just want to make sure there are no more issues reported. Users can already update to biblatex 2.3 and biber 1.3 from the development folders on SourceForge to get these fixes now.