Comments (7)
Hi Daryl,
This looks indeed like a race condition. Your scour.conf has overlapping directory entries (lines 3 and 4 in this case). The scour program launches a thread for each line. Therefore, by the time one thread reads a file under one directory that file may have already been seen in the other thread and deleted. Hence, leaving the first thread wondering and displaying the ERROR above. Note that the age for each directory entry is 2 and 8. Therefore, the missing file under /data/gempak/nexrad must have been age 8 or older. One way of preventing this rare case from happening is to lock the resource.
Best regards,
--Mustapha
from ldm.
Greetings, thanks for the response. The age of the file is less than 8, you can see that by the filename timestamp. So yeah, the lstat
would perhaps be attempting to lookup a file that was deleted by the other thread..
from ldm.
I have seen this occur in the "old" way of scouring as well in LDM 6.13.10 and earlier, but it's now a moot point.
from ldm.
Perhaps a command line switch could be offered to disable threaded scouring? Or maybe this particular error could be sent to a lower priority log level?
from ldm.
The new scour program spawns as many threads as there are directory entries (in scour.conf.) Therefore, to make it mono-threaded (without code change) it suffices to provide one directory entry at a time (to ensure non-concurrency.) It is also possible to enforce sequentiality with minor code change and a switch if warranted. Setting this error to a lower priority log level is also possible and only requires minimum code change.
from ldm.
@akrherz Or one could modify their scour(1) configuration-file to avoid overlapping entries.
from ldm.
@akrherz Or one could modify their scour(1) configuration-file to avoid overlapping entries.
Agreed, but that is brittle as I may add a new folder and forget to add a custom entry for it and very annoying as I have to add one entry for each sub-folder. Additionally, overlapping entries make total sense in my mind.
I have a blanket policy for anything in /data/gempak
being at most 10 days old and then anything in /data/gempak/nexrad
being at most 2 days old.
from ldm.
Related Issues (20)
- Too many ignoring too-old product WARN messages in ldmd.log in version 6.13.15 HOT 4
- LDM 6.13.16 not using full queue size as 6.13.13 did HOT 2
- registry.xml ntp server issue HOT 2
- Expand duplicate-product hashtable
- LDM 6.13.16 pqact will stop ingesting with a small product queue HOT 7
- ldmadmin newlog does not rotate log files when LDM is used as a NOAAport ingester HOT 1
- LDM 6.14.5 printmetrics change with port count reversed? HOT 2
- Verify that "ldmadmin pqactHUP" signals *all* pqact(1) processes
- "ldmadmin vetqueuesize" doesn't modify the size and slots parameter values in the registry
- MDM project is still open source now?I want got it download url or other solution。 HOT 3
- Allow pqinsert to read from STDIN HOT 1
- Add examples of non-integral day specifications to scour(1) man-page
- When ingesting large volume feeds, product queue becomes way too large HOT 10
- netstat has been replaced by ss HOT 1
- LDM does not check pqact files loaded from included configuration files
- Periodic missing satellite sectors HOT 6
- PQACT question and minor documentation error
- Using the LDM to ingest from more than one satellite receiver on a single server HOT 1
- pqact option of -gunzip HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ldm.