heads up: building texmath when ghc is configured to use Clang rather than GCC can cause HIGH memory usage

[jgm]

heads up: building texmath when ghc is configured to use Clang rather than GCC can cause HIGH memory usage

[cartazio]

as much as I might be loath to suggest it, migrating some of those lookup structures to C might result in more predictable/faster build times for users. though I and a few other folks will try to track down what issues may lie with GHC

[cartazio]

the blow up seems to come from the metadata for profiling builds making clang freak out https://gist.github.com/cartazio/5595d81cae0474f3f1f6

[cartazio]

tl;dr, i think ghc is putting cost center info on every slot in the texmath lookup tables, and thats creating TOO big a cost center table for the profiled way when built with clang

[jgm]

So this only affects builds with profiling support turned on?

+++ Carter Tazio Schonwald [Dec 24 14 09:35 ]:

tl;dr, i think ghc is putting cost center info on every slot in the texmath lookup tables, and thats creating TOO big a cost center table for the profiled way when built with clang

---

Reply to this email directly or view it on GitHub:
#70 (comment)

[cartazio]

yup. And will probably only hit mac users (because thats the only main GHC distro that uses clang ). If they wait long enough the build WILL finish though.

[mpickering]

@cartazio I've wanted to move these tables out of source, do you just mean having the table in a C file and using FFI?

[cartazio]

yeah, you could just FFI them as a a Pointer, then use Storables on the haskell side, happy to help

[cartazio]

remind me and I'll try to dig up some example haskell codes that do this already sometime next weekend

[jgm]

That would be very helpful.

+++ Carter Tazio Schonwald [Jan 13 15 21:31 ]:

remind me and I'll try to dig up some example haskell codes that do this already

---

Reply to this email directly or view it on GitHub:
#70 (comment)

[wrengr]

I'm also running into this problem when using gcc on OSX: the mmap when compiling Text.TeXMath.Unicode.ToTeX fails. Would disabling profiling help with that too?

[wrengr]

Looks like yes: with profiling disabled it finishes quickly with gcc too

[jgm]

@cartazio, if you could supply an example that would point the way forward here, that would be great.
(My laptop is heating up now as texmath compiles...)

[cartazio]

@wrengr gcc on mac IS clang

ok, sorry for the MIA

theres actually a "simpler" way you could hack it. (may not be simpler per se, )

1. the compilation time blowup is presumably because of a superlinearity in the compiler complexity
2. if you split the offending list into 4-8 disjoint ranges, each in their own private internal module, and do a teeny bit of wrapper logic

anyways, writing up the model implementation right now, and i'll email @jgm the code plus link a (truncated!) gist of the approach here

[cartazio]

oooo, it works,

-- {-#  LANGUAGE ForeignFunctionInterface #-}

--module Text.TeXMath.Unicode.ToASCII (getASCII) where

module ToASCII where
import qualified Data.IntMap as M
import Data.Char (ord)
import Data.Maybe (fromMaybe)
import Foreign.C
import Foreign.Ptr
import System.IO.Unsafe
import Foreign.Storable
--import Data.Int
-- | Approximates a single unicode character as an ASCII string
--  (each character is between 0x00 and 0x7F).
getASCII :: Char -> String
getASCII u = fromMaybe "" (M.lookup (ord u) table)

--table :: M.IntMap String

table :: M.IntMap String
table = M.fromList $ zip realKeyKey realValVal

foreign import ccall unsafe "&keylookup" keyKey :: Ptr CInt
foreign import ccall unsafe "&toASCIILut" valVal :: Ptr (Ptr CChar)

{-# NOINLINE realKeyKey #-}
realKeyKey :: [Int]
realKeyKey = unsafePerformIO $ mapM (\off ->  do  daInt <- peekElemOff keyKey off ; return $ fromIntegral daInt ) [ 0 .. 8922]

{-# NOINLINE realValVal #-}
realValVal :: [String]
realValVal = unsafePerformIO $ mapM (\off -> do daPtrStr <- peekElemOff valVal off ; peekCString daPtrStr ) [0 .. 8922]

then in ghci i do eg

*ToASCII> getASCII 'µ'
"u"

yay!

ill turn this into a pr now

[cartazio]

note that I make this lookup table code available under a BSD/MIT license, so when other folks wanna adapt the idea to their code, they are welcome to use it as is (though this project itself is gpl now for some reason :) )

[cartazio]

Text.TeXMath.Unicode.ToTeX is still egregiously slow, but i leave adapting the same approach to that module to someone else :)

so i'm gooona propose we also remove -auto-all -caf-all from the default build profile options for now

[cartazio]

pr #71 does the first bits

[jgm]

Thanks! ToTeX will be harder because of the more complex data structure. But this is a big improvement already; my laptop no longer sounds like a vacuum cleaner while compiling texmath.

[cartazio]

Half the change Is in my PR I disabled prof auto all and friends. :)
though I suspect the blow up still happenes on totex even then.

Once you migrate all the lookup structures to c code, it reenabling those
profiling flags will no longer blow up compile time.
On Mar 15, 2015 1:50 PM, "John MacFarlane" [email protected] wrote:

Thanks! ToTeX will be harder because of the more complex data structure.
But this is a big improvement already; my laptop no longer sounds like a
vacuum cleaner while compiling texmath.

—
Reply to this email directly or view it on GitHub
#70 (comment).

[cartazio]

I'll take a look at the totex structure this evening. Now that I've got one part working, should be super easy to adapt to the rest.

[jgm]

Great!

[cartazio]

nearly done, stay tuned :)

[cartazio]

ok, the PR for the ToTex stuff SHOULD pass this next round of travis CI

all thats left is

1. the entity maps code
2. doing a subsequent refactor so that @mpickering's code gen approach can be migrated to this style so that its a bit easier to upgrade / mess with in the future. (i probably should have done that from the outset in retrospect, but at least you have some worked out proof of concept implemntations that demonstrate the idea :) )

[cartazio]

(if one of you wanna take over on those fronts, please do :) )

[cartazio]

another option would have been to add those unicode files as "extra data files", but that adds to relocatability headaches so i didn't consider that route

[jgm]

@cartazio I wonder whether the changes you made are related to this report I just got?
jgm/pandoc@693bc5f#commitcomment-10723486

[cartazio]

The version bound stuff? Did he try doing a cabal update? I'm afk, but I'll write out build plan debugging directions later

[jgm]

+++ Carter Tazio Schonwald [Apr 14 15 14:46 ]:

The version bound stuff? Did he try doing a cabal update? I'm afk, but I'll write out build plan debugging directions later

I should have given more background. I recently raised the lower bound on texmath, which forced installation of a version with the changes you pushed (the C data files). I thought this might be related to the problem he was having, though it seems that it cleared up after a cabal clean.

[jgm]

See #96 for another approach.

[jgm]

I've closed #71 after merging it into the morecbits branch in this repository.
It builds fine, we just need to change it so that the cbits are automatically generated.

[jgm]

I've added a build script for @cartazio's cbits and committed to master.
Result is a decent speedup in profiling build (from 14+ minutes to a bit under 9 minutes on the machine I'm using now, macos with clang).
The TeX reader is still the biggest bottleneck, though.