mdfind takes multiple minutes if a NTFS bootcamp volume is mounted

[munki]

mdfind takes multiple minutes if a NTFS bootcamp volume is mounted

[natewalck]

From [email protected] on November 26, 2010 23:31:22

Labels: -Type-Defect -Priority-Medium Type-Enhancement Priority-Low

[natewalck]

From [email protected] on November 26, 2010 23:34:10

I am happy to make this change after receiving input from others on the idea.

This probably won't help any speed problems people using read-only NFS automounts might be having (though it could if these were all in a top-directory location easy to exclude before running mdfind).

[natewalck]

From [email protected] on November 27, 2010 00:41:48

Test code below (replaces munkicommon.getSpotlightInstalledApplications).

It provides a massive speedup on my test computer with bootcamp - from over 2 minutes to under 5 seconds for the entire getAppData routine.

On another test computer without the bootcamp problem I am seeing this modified function slow things down a little more than expected (from sub 1 second to about 4 seconds to execute; the mdfind specifically on the /Applications directory is slower than the same mdfind on the entire computer).

ie: it does have some performance hit for computers not experiencing the problem; however that performance hit is only a few seconds.

def mdfindAppsInDir(directory,applist):
"""Use spotlight to search for items of type application in directory
and append to applist (input parameter is modified).
"""
argv = ['/usr/bin/mdfind', '-0', '-onlyin', directory,
'kMDItemKind = 'Application'']
p = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = p.communicate()
rc = p.wait()

if rc == 0:
    for app_path in stdout.split('\0'):
        if (not app_path.startswith('/Volumes/') and not
        isExcludedFilesystem(app_path)):
            applist.append(app_path)

def getSpotlightInstalledApplications():
"""Get paths of currently installed applications per Spotlight.
Return value is list of paths.
Ignores apps installed on other volumes
"""
skipDirs = ['Volumes','tmp','.vol','.Trashes',
'.Spotlight-V100','.fseventsd','Network','net','cores','dev']
applist = []

for f in os.listdir('/'):
    if not f in skipDirs:
        p = os.path.join('/',f)
        if os.path.isdir(p) and not os.path.islink(p) and not os.path.ismount(p):
            if f.endswith('.app'):
                applist.append(p)
            else:
                mdfindAppsInDir(p,applist)

return applist

[natewalck]

From [email protected] on November 27, 2010 00:43:05

Status: Started

[natewalck]

From [email protected] on December 06, 2010 13:48:43

overall I'm semi OK with this. I had hoped to eventually widen the removal of /Volumes from mdfind output, and only remove if they are network paths. Local disks should still be under consideration.

be careful to use munkicommon.listdir() instead of os.listdir(), and while we're in here we should decode utf-8 output from mdfind into unicode objects.

[natewalck]

From [email protected] on December 06, 2010 13:58:01

There's no one perfect answer for this problem; it may be specific to each deployment.

[natewalck]

From [email protected] on December 06, 2010 14:05:57

munkicommon.isExcludedFilesystem suits me well enough for the filesystems to be excluded. NFS or read-only get excluded (not sure what it does to AFP / SMB mounts). I would be happy to walk into Volumes as well, and only run the mdfind function on directories within that return false for isExcludedFilesystem.

For my site walking into Volumes is unnecessary (but doesn't hurt either). Most computers have only one drive, some have more but in no standard way / not named the same thing. Managed applications will always be installed on the boot drive here.

[natewalck]

From [email protected] on December 08, 2010 01:00:42

Any thoughts on making the spotlight call from within python itself? I presume this starts up a thread that we don't really want to start up ...
Making a NSMetadataQuery call directly allows setting of the search scope to an array of folders (mdfind allows only one to be specified at each command line call).

This code:

• includes /Users/Shared but excludes the rest of /Users (current code seems to include /Users)
• excludes network filesystems
• includes local volumes, but not if they are read-only (eg: boot-camp volume)

Oddly I'm noticing that on one of my systems with bootcamp I get the multi-minute delay, while on another spotlight appears to entirely ignore the read-only NTFS volume with bootcamp on it (and thus returns quickly). Perhaps slightly different builds of 10.6.5 (one is a MBAir).

from Foundation import NSMetadataQuery, NSPredicate, NSRunLoop, NSDate

def findAppsInDirs(dirlist,applist):
query = NSMetadataQuery.alloc().init()
query.setPredicate_(NSPredicate.predicateWithFormat_("(kMDItemKind = "Application")"))
#query.setSearchScopes_(['/Applications/Utilities','/Library','/Volumes/Untitled','/Users/Shared'])
query.setSearchScopes_(dirlist)
query.startQuery()
while query.isGathering():
NSRunLoop.currentRunLoop().runUntilDate_(NSDate.dateWithTimeIntervalSinceNow_(1))
query.stopQuery()

for item in query.results():
    p = item.valueForAttribute_("kMDItemPath")
    if not isExcludedFilesystem(p):
        applist.append(p)

def getSpotlightInstalledApplications():
"""Get paths of currently installed applications per Spotlight.
Return value is list of paths.
Ignores apps installed on other volumes
"""
skipDirs = ['Volumes','tmp','.vol','.Trashes','Users',
'.Spotlight-V100','.fseventsd','Network','net','cores','dev']
dirlist = []
applist = []

for f in listdir('/'):
    if not f in skipDirs:
        p = os.path.join('/',f)
        if os.path.isdir(p) and not os.path.islink(p) and not isExcludedFilesystem(p):
            if f.endswith('.app'):
                applist.append(p)
            else:
                dirlist.append(p)

for f in listdir('/Volumes'):
        p = os.path.join('/Volumes',f)
        if os.path.isdir(p) and not os.path.islink(p) and not isExcludedFilesystem(p):
            dirlist.append(p)

dirlist.append('/Users/Shared')

#print dirlist

findAppsInDirs(dirlist,applist)

return applist

[natewalck]

From [email protected] on December 08, 2010 10:51:29

I think calling Spotlight via the API sounds great. I have two concerns:

1. Is NSRunLoop available at all times to a running Python script? I thought that required an application object...

2. I'm not clear what you mean when you said "Oddly I'm noticing that on one of my systems with bootcamp I get the multi-minute delay" -- do you mean you get the delay even if you call Spotlight specifically excluding the NTFS filesystem?

[natewalck]

From [email protected] on December 09, 2010 03:41:59

"1) Is NSRunLoop available at all times to a running Python script? I thought that required an application object..."
It is working fine. I don't understand it well enough, but I understand it to be a thread control mechanism - one which is also used by all the GUI stuff. But it is Apple's generic wrap concept that deals with threads in command line stuff too.

"2) I'm not clear what you mean when you said "Oddly I'm noticing that on one of my systems with bootcamp I get the multi-minute delay" -- do you mean you get the delay even if you call Spotlight specifically excluding the NTFS filesystem?"
No - this code works without delay on all machines. I simply had second thoughts about whether it was worth fixing as munki-head doesn't display this fault consistently on all machines. However it seems it is more likely to show the fault on the first few runs after boot. Spotlight doesn't seem to be perfectly consistent in its behaviour if not told exactly what to do (but this change tells it exactly which paths it is allowed to look in).

I'm happy with the code below now, probably can commit the changes. Note: not tested on 10.5 yet (have few 10.5 computers left, and am upgrading them to 10.6 before installing munki).

This change does change behaviour in terms of what is included / excluded.

The unchanged LaunchServices code appears to find applications anywhere on the boot drive - including in user homes, these are later excluded in updatecheck.

The changed Spotlight code:

• finds applications on the boot drive and other local r/w volumes
• excludes /Users (but allows /Users/Shared) at point of search.

from Foundation import NSDate, NSMetadataQuery, NSPredicate, NSRunLoop

def findAppsInDirs(dirlist):
"""Do spotlight search for type applications within the
list of directories provided. Returns a list of paths to applications
these appear to always be some form of unicode string.
"""
applist = []
query = NSMetadataQuery.alloc().init()
query.setPredicate_(NSPredicate.predicateWithFormat_(
"(kMDItemKind = "Application")"))
query.setSearchScopes_(dirlist)
query.startQuery()
#Spotlight isGathering phase - this is the initial search. After the
# isGathering phase Spotlight keeps running returning live results from
# filesystem changes, we are not interested in that phase.
#Run for 0.3 seconds then check if isGathering has completed.
while query.isGathering():
NSRunLoop.currentRunLoop().runUntilDate_(
NSDate.dateWithTimeIntervalSinceNow_(0.3))
query.stopQuery()

for item in query.results():
    p = item.valueForAttribute_("kMDItemPath")
    if not isExcludedFilesystem(p):
        applist.append(p)
return applist

def getSpotlightInstalledApplications():
"""Get paths of currently installed applications per Spotlight.
Return value is list of paths.
Ignores apps installed on non-local or read-only volumes,
ignores apps within user accounts.
"""
skipdirs = ['Volumes', 'tmp', '.vol', '.Trashes', 'Users',
'.Spotlight-V100', '.fseventsd', 'Network', 'net',
'home', 'cores', 'dev']
dirlist = []
applist = []

for f in listdir(u'/'):
    if not f in skipdirs:
        p = os.path.join(u'/', f)
        if os.path.isdir(p) and not os.path.islink(p) \
                            and not isExcludedFilesystem(p):
            if f.endswith('.app'):
                applist.append(p)
            else:
                dirlist.append(p)

for f in listdir(u'/Volumes'):
    p = os.path.join(u'/Volumes', f)
    if os.path.isdir(p) and not os.path.islink(p) \
                        and not isExcludedFilesystem(p):
        dirlist.append(p)

dirlist.append(u'/Users/Shared')

applist.extend(findAppsInDirs(dirlist))
return applist

[natewalck]

From [email protected] on January 30, 2011 15:46:51

Committed in r1003 . Closing issue.

Further discussions still to be had on why we care if we find an app in a non-default location, and what we should do about it.

Status: Verified