The update-xapian-index job is noisy

[mysociety]

The update-xapian-index job is noisy

[frabcus]

I didn't do this originally, because I couldn't distinguish errors we
didn't care about from errors we did. And sometimes it does reveal
errors we do care about. Trade off up to you, of course!

Also, worth upgrading to the latest PDF conversion library, I think
it will reduce the number of errors. (My tests last time I did it,
about a year ago, showed it also improved quality of text
conversion)

On Mon, Jun 20, 2011 at 02:29:38AM -0700, robinhouston wrote:

The update-xapian-index job spits out a variety of error messages that seem to be related to the processing of attachments, specifically PDF files. There is nothing we can do about these, since we don’t control the PDF files, so there should at least be an option to suppress such messages when the thing is run from cron.

Reply to this email directly or view it on GitHub:
https://github.com/sebbacon/alaveteli/issues/52

[robinhouston]

Hi Francis,

Good to hear from you! I trust scraperwiki is leaping from height to greater height like a nimble stag.

I don't know if you've looked at the change I committed last night, but it attempts to distinguish errors (which are shown) from warnings (which are suppressed) by looking at the exit code of the process. If the exit code is 0, then everything the process printed to stderr is discarded; if it it's non-zero they are printed to our own stderr.

Or do you mean that information we care about is sometimes printed to stderr even if the process returns successfully? If that is what you mean, can you remember any examples?

Thanks for the tip about upgrading the PDF conversion library.

Cheers,
Robin

On 21 Jun 2011, at 06:36, [email protected] wrote:

I didn't do this originally, because I couldn't distinguish errors we
didn't care about from errors we did. And sometimes it does reveal
errors we do care about. Trade off up to you, of course!

Also, worth upgrading to the latest PDF conversion library, I think
it will reduce the number of errors. (My tests last time I did it,
about a year ago, showed it also improved quality of text
conversion)

On Mon, Jun 20, 2011 at 02:29:38AM -0700, robinhouston wrote:

The update-xapian-index job spits out a variety of error messages that seem to be related to the processing of attachments, specifically PDF files. There is nothing we can do about these, since we don’t control the PDF files, so there should at least be an option to suppress such messages when the thing is run from cron.

Reply to this email directly or view it on GitHub:
https://github.com/sebbacon/alaveteli/issues/52

Reply to this email directly or view it on GitHub:
https://github.com/sebbacon/alaveteli/issues/52#issuecomment-1408120

[frabcus]

Oh that's a good idea!

I don't think I ever tried doing that, so you're way ahead of me now.

Francis

On Tue, Jun 21, 2011 at 02:04:49AM -0700, robinhouston wrote:

Hi Francis,

Good to hear from you! I trust scraperwiki is leaping from height to greater height like a nimble stag.

I don't know if you've looked at the change I committed last night, but it attempts to distinguish errors (which are shown) from warnings (which are suppressed) by looking at the exit code of the process. If the exit code is 0, then everything the process printed to stderr is discarded; if it it's non-zero they are printed to our own stderr.

Or do you mean that information we care about is sometimes printed to stderr even if the process returns successfully? If that is what you mean, can you remember any examples?

Thanks for the tip about upgrading the PDF conversion library.

Cheers,
Robin

On 21 Jun 2011, at 06:36, [email protected] wrote:

I didn't do this originally, because I couldn't distinguish errors we
didn't care about from errors we did. And sometimes it does reveal
errors we do care about. Trade off up to you, of course!

Also, worth upgrading to the latest PDF conversion library, I think
it will reduce the number of errors. (My tests last time I did it,
about a year ago, showed it also improved quality of text
conversion)

On Mon, Jun 20, 2011 at 02:29:38AM -0700, robinhouston wrote:

The update-xapian-index job spits out a variety of error messages that seem to be related to the processing of attachments, specifically PDF files. There is nothing we can do about these, since we don’t control the PDF files, so there should at least be an option to suppress such messages when the thing is run from cron.

Reply to this email directly or view it on GitHub:
https://github.com/sebbacon/alaveteli/issues/52

Reply to this email directly or view it on GitHub:
https://github.com/sebbacon/alaveteli/issues/52#issuecomment-1408120

Reply to this email directly or view it on GitHub:
https://github.com/sebbacon/alaveteli/issues/52#issuecomment-1408920