Comments (17)
rfindler
commented on July 18, 2024
Can you share something that illustates this slowness? While it may make sense to push this kind of thing to the client, it may also be that there is a missing opportunity to use one of the many caches in there or somehow otherwise speed things up.
from drracket.
6cdh
commented on July 18, 2024
Yes. I find the slowest function is get-index-entry-info in document-variable:
Each document-variable call takes several milliseconds, and it will be called thousand times in a big file.
I have a customized check-syntax library in my op_setup.rkt repo which only made the function document-variable empty. And I use this benchmark script gist.
The lib.rkt in that repo has 900 loc and 10k lines of expanded code.
This is the result with check-syntax in Racket 8.9 CS in that repo:
⬢ [archlinux:latest] ❯ racket bm.rkt lib.rkt
expand
cpu time: 468 real time: 474 gc time: 96
traversal
cpu time: 3858 real time: 3904 gc time: 114After replace drracket/check-syntax with "check-syntax.rkt", this is the result with the customized check-syntax:
⬢ [archlinux:latest] ❯ racket bm.rkt lib.rkt
expand
cpu time: 475 real time: 481 gc time: 108
traversal
cpu time: 193 real time: 195 gc time: 14from drracket.
6cdh
commented on July 18, 2024
It turns out the most time of get-index-entry-info spent on xref-binding->definition-tag calls which is probably cacheable.
from drracket.
rfindler
commented on July 18, 2024
I've pushed something that adds a cache there and see about a 4.5x speedup for the traversal time of the "racket bm.rkt lib.rkt" benchmark on my machine. Not as dramatic as your machine but our machines are probably not directly comparable so I am curious to know what you see if you include this commit.
from drracket.
6cdh
commented on July 18, 2024
nice! It reduced from ~3600 ms to ~800 ms for me and 4.5x faster.
My previous benchmark result was with fully disabled document-variable, it makes sense that it's much faster.
from drracket.
6cdh
commented on July 18, 2024
thanks. Another bottleneck sometimes I found is current-module-name-resolver:
Added a time around the expression ((current-module-name-resolver) datum #f #f #t), I got this result with the latest cached check-syntax:
⬢ [archlinux:latest] ❯ racket bm.rkt copier.rkt
expand
cpu time: 1086 real time: 1100 gc time: 245
traversal
cpu time: 0 real time: 0 gc time: 0
cpu time: 932 real time: 944 gc time: 195
cpu time: 1 real time: 1 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 1162 real time: 1176 gc time: 213
and
⬢ [archlinux:latest] ❯ racket bm.rkt check-syntax.rkt
expand
cpu time: 1035 real time: 1050 gc time: 227
traversal
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 853 real time: 866 gc time: 179
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 1077 real time: 1093 gc time: 200
It's heavy single call, and looks like there is another expand inside it. I don't know if it can be improved.
Those are the only two bottlenecks I've noticed.
from drracket.
rfindler
commented on July 18, 2024
This one is because it has to compile some dependencies. In the case of copier.rkt, it is compiling check-syntax.rkt. If you do "raco make check-syntax.rkt" you should see a speed up there.
from drracket.
6cdh
commented on July 18, 2024
It's a good suggestion. The expand time and traversal time is improved a lot, from 2s to 800ms in total.
from drracket.
rfindler
commented on July 18, 2024
I made a change (just locally on my machine) to eliminate the extra indirection via a thread (not safe in general but fine for the benchmark) and it reveals that about 43% of the traversal time is being spent in database queries. That doesn't seem optimal so it still seems reasonable to do more work here.
I also made another change to actually do the profile and it runs the traversal step a bunch of times to try to make the results more accurate; these are the results I see.
Total cpu time observed: 2704ms (out of 2716ms)
Number of samples taken: 55 (once every 49ms)
===========================================================================================
Caller
Idx Total Self Name+src Local%
ms(pct) ms(pct) Callee
===========================================================================================
[1] 2704(100.0%) 0(0.0%) body of "/Users/robby/git/6cdh/op_setup.rkt/bm.rkt" ...
profile-thunk [2] 100.0%
-------------------------------------------------------------------------------------------
body of "/Users/robby/git/6cdh/op_setup.rkt/bm.rkt" [1]100.0%
[2] 2704(100.0%) 0(0.0%) profile-thunk ...are/pkgs/profile-lib/main.rkt:9:0
??? [3] 100.0%
-------------------------------------------------------------------------------------------
profile-thunk [2] 100.0%
[3] 2704(100.0%) 0(0.0%) ??? ...acket/share/pkgs/profile-lib/main.rkt:40:10
??? [4] 100.0%
-------------------------------------------------------------------------------------------
??? [3] 100.0%
[4] 2704(100.0%) 0(0.0%) ??? ...acket/share/pkgs/profile-lib/main.rkt:59:31
traversal [5] 100.0%
-------------------------------------------------------------------------------------------
??? [4] 100.0%
[5] 2704(100.0%) 0(0.0%) traversal ...bby/git/6cdh/op_setup.rkt/bm.rkt:25:0
...row-higher-order.rkt:375:33 [6] 100.0%
-------------------------------------------------------------------------------------------
traversal [5] 100.0%
[6] 2704(100.0%) 0(0.0%) ...row-higher-order.rkt:375:33 ...order.rkt:375:33
expanded-expression [7] 100.0%
-------------------------------------------------------------------------------------------
...row-higher-order.rkt:375:33 [6] 100.0%
[7] 2704(100.0%) 0(0.0%) expanded-expression ...ncheck/traversals.rkt:48:10
annotate-variables [8] 83.7%
try-next [9] 14.5%
annotate-contracts [10] 1.8%
-------------------------------------------------------------------------------------------
expanded-expression [7] 100.0%
[8] 2263(83.7%) 0(0.0%) annotate-variables ...yncheck/traversals.rkt:743:0
document-variable [11] 82.6%
connect-identifier [13] 15.3%
initialize-binder-connections [15] 2.2%
-------------------------------------------------------------------------------------------
try-next [9] 11.5%
body-loop [21] 14.6%
level+tail+mod-loop [12] 14.6%
expanded-expression [7] 59.4%
[9] 392(14.5%) 0(0.0%) try-next .../private/syncheck/traversals.rkt:325:6
level+tail+mod-loop [12] 78.1%
try-next [9] 11.5%
body-loop [21] 10.4%
-------------------------------------------------------------------------------------------
expanded-expression [7] 100.0%
[10] 50(1.8%) 0(0.0%) annotate-contracts .../contract-traversal.rkt:11:0
loop [14] 100.0%
-------------------------------------------------------------------------------------------
annotate-variables [8] 100.0%
[11] 1868(69.1%) 0(0.0%) document-variable ...yncheck/traversals.rkt:1488:0
hash-ref! [16] 94.8%
build-docs-label [19] 5.2%
-------------------------------------------------------------------------------------------
body-loop [21] 1.2%
level+tail+mod-loop [12] 16.0%
try-next [9] 82.7%
[12] 392(14.5%) 196(7.2%) level+tail+mod-loop ...ncheck/traversals.rkt:219:2
try-next [9] 16.7%
level+tail+mod-loop [12] 16.0%
collect-general-info [20] 15.6%
add-binders [22] 12.5%
stx-pair? [18] 12.5%
body-loop [21] 2.5%
-------------------------------------------------------------------------------------------
annotate-variables [8] 100.0%
[13] 346(12.8%) 49(1.8%) connect-identifier ...yncheck/traversals.rkt:917:0
connect-identifier-to-a-require [17] 85.8%
-------------------------------------------------------------------------------------------
annotate-contracts [10] 5.3%
loop [14] 94.7%
[14] 50(1.8%) 0(0.0%) loop ...ivate/syncheck/contract-traversal.rkt:22:2
loop [14] 94.7%
stx-pair? [18] 5.3%
-------------------------------------------------------------------------------------------
annotate-variables [8] 100.0%
[15] 49(1.8%) 49(1.8%) initialize-binder-connections ...ersals.rkt:1181:0
-------------------------------------------------------------------------------------------
document-variable [11] 100.0%
[16] 1770(65.5%) 48(1.8%) hash-ref! .../racket/private/more-scheme.rkt:377:2
??? [23] 97.3%
-------------------------------------------------------------------------------------------
connect-identifier [13] 100.0%
[17] 296(11.0%) 100(3.7%) connect-identifier-to-a-require ...rsals.rkt:959:0
connect-syntaxes [24] 66.4%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 49.7%
loop [14] 50.3%
[18] 98(3.6%) 98(3.6%) stx-pair? .../collects/racket/private/stx.rkt:40:4
-------------------------------------------------------------------------------------------
document-variable [11] 100.0%
[19] 98(3.6%) 0(0.0%) build-docs-label ...syncheck/traversals.rkt:1515:0
loop [25] 100.0%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 100.0%
[20] 98(3.6%) 0(0.0%) collect-general-info ...heck/traversals.rkt:266:12
loop [27] 50.0%
add-disappeared-uses [26] 50.0%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 33.3%
try-next [9] 66.7%
[21] 98(3.6%) 0(0.0%) body-loop ...rivate/syncheck/traversals.rkt:293:10
try-next [9] 83.3%
level+tail+mod-loop [12] 16.7%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 100.0%
[22] 49(1.8%) 0(0.0%) add-binders ...vate/syncheck/traversals.rkt:1410:0
add-id [30] 100.0%
-------------------------------------------------------------------------------------------
hash-ref! [16] 100.0%
[23] 1722(63.7%) 0(0.0%) ??? ...lib/drracket/private/syncheck/xref.rkt:47:4
xref-binding-tag [28] 74.2%
xref-tag->path+anchor [29] 25.8%
-------------------------------------------------------------------------------------------
connect-identifier-to-a-require [17] 100.0%
[24] 197(7.3%) 197(7.3%) connect-syntaxes ...syncheck/traversals.rkt:1198:0
-------------------------------------------------------------------------------------------
loop [25] 25.3%
build-docs-label [19] 74.7%
[25] 98(3.6%) 98(3.6%) loop .../private/map.rkt:40:19
loop [25] 25.3%
-------------------------------------------------------------------------------------------
collect-general-info [20] 100.0%
[26] 49(1.8%) 49(1.8%) add-disappeared-uses ...check/traversals.rkt:723:0
-------------------------------------------------------------------------------------------
loop [27] 50.0%
collect-general-info [20] 50.0%
[27] 49(1.8%) 0(0.0%) loop ...ket/private/syncheck/traversals.rkt:1372:2
loop [27] 50.0%
add-id [30] 50.0%
-------------------------------------------------------------------------------------------
??? [23] 100.0%
[28] 1278(47.3%) 0(0.0%) xref-binding-tag ...le-lib/scribble/xref.rkt:122:2
loop [31] 100.0%
-------------------------------------------------------------------------------------------
??? [23] 100.0%
[29] 444(16.4%) 0(0.0%) xref-tag->path+anchor ...b/scribble/xref.rkt:160:0
compose-class [32] 100.0%
-------------------------------------------------------------------------------------------
loop [27] 50.0%
add-binders [22] 50.0%
[30] 98(3.6%) 49(1.8%) add-id ...t/private/syncheck/traversals.rkt:1565:0
do-ref [33] 50.0%
-------------------------------------------------------------------------------------------
loop [31] 39.7%
xref-binding-tag [28] 60.3%
[31] 1278(47.3%) 0(0.0%) loop ...bble/scribble-lib/scribble/search.rkt:75:5
resolve-search [34] 53.7%
loop [31] 39.7%
resolve-get/tentative [36] 5.4%
??? [38] 1.3%
-------------------------------------------------------------------------------------------
xref-tag->path+anchor [29] 100.0%
[32] 444(16.4%) 99(3.7%) compose-class ...private/class-internal.rkt:2142:0
gen-for-each [35] 55.5%
make-field-info [37] 11.2%
??? [39] 11.0%
-------------------------------------------------------------------------------------------
add-id [30] 100.0%
[33] 49(1.8%) 49(1.8%) do-ref ...ollects/syntax/private/id-table.rkt:77:2
-------------------------------------------------------------------------------------------
loop [31] 100.0%
[34] 1084(40.1%) 0(0.0%) resolve-search ...ibble-lib/scribble/core.rkt:73:0
resolve-get/ext-id* [40] 100.0%
-------------------------------------------------------------------------------------------
compose-class [32] 100.0%
[35] 246(9.1%) 0(0.0%) gen-for-each ...lects/racket/private/map.rkt:267:2
??? [41] 100.0%
-------------------------------------------------------------------------------------------
loop [31] 100.0%
[36] 145(5.4%) 0(0.0%) resolve-get/tentative ...ib/scribble/core.rkt:69:0
resolve-get/where [43] 100.0%
-------------------------------------------------------------------------------------------
compose-class [32] 100.0%
[37] 50(1.8%) 50(1.8%) make-field-info ...rivate/class-internal.rkt:295:0
-------------------------------------------------------------------------------------------
loop [31] 100.0%
[38] 50(1.8%) 0(0.0%) ??? ...contract/private/arrow-val-first.rkt:486:18
module-path-index->taglet [42] 100.0%
-------------------------------------------------------------------------------------------
compose-class [32] 100.0%
[39] 49(1.8%) 49(1.8%) ??? ...scribble-lib/scribble/html-render.rkt:263:2
-------------------------------------------------------------------------------------------
resolve-search [34] 100.0%
[40] 1084(40.1%) 0(0.0%) resolve-get/ext-id* ...-lib/scribble/core.rkt:55:0
resolve-get/where [43] 100.0%
-------------------------------------------------------------------------------------------
gen-for-each [35] 100.0%
[41] 246(9.1%) 0(0.0%) ??? ...s/racket/private/class-internal.rkt:2664:32
loop [44] 100.0%
-------------------------------------------------------------------------------------------
??? [38] 100.0%
[42] 50(1.8%) 0(0.0%) module-path-index->taglet ...scribble/tag.rkt:98:0
do-module-path-index->taglet [45] 100.0%
-------------------------------------------------------------------------------------------
resolve-get/tentative [36] 11.8%
resolve-get/ext-id* [40] 88.2%
[43] 1229(45.4%) 50(1.8%) resolve-get/where ...le-lib/scribble/core.rkt:24:0
??? [46] 96.0%
-------------------------------------------------------------------------------------------
??? [41] 100.0%
[44] 246(9.1%) 246(9.1%) loop .../racket/private/class-internal.rkt:2678:49
-------------------------------------------------------------------------------------------
module-path-index->taglet [42] 100.0%
[45] 50(1.8%) 0(0.0%) do-module-path-index->taglet ...ibble/tag.rkt:64:0
combine-relative-elements [47] 100.0%
-------------------------------------------------------------------------------------------
resolve-get/where [43] 100.0%
[46] 1180(43.6%) 0(0.0%) ??? ...ribble-lib/scribble/base-render.rkt:1218:28
send-arg69 [48] 100.0%
-------------------------------------------------------------------------------------------
do-module-path-index->taglet [45] 100.0%
[47] 50(1.8%) 50(1.8%) combine-relative-elements ...llapse-noctc.rkt:75:2
-------------------------------------------------------------------------------------------
??? [46] 100.0%
[48] 1180(43.6%) 0(0.0%) send-arg69 ...scribble-lib/scribble/xref.rkt:68:19
??? [49] 100.0%
-------------------------------------------------------------------------------------------
send-arg69 [48] 100.0%
[49] 1180(43.6%) 0(0.0%) ??? .../plt/pkgs/racket-index/setup/xref.rkt:122:2
try [50] 100.0%
-------------------------------------------------------------------------------------------
??? [49] 100.0%
[50] 1180(43.6%) 0(0.0%) try .../plt/pkgs/racket-index/setup/xref.rkt:126:6
call-with-database [51] 100.0%
-------------------------------------------------------------------------------------------
try [50] 100.0%
[51] 1180(43.6%) 0(0.0%) call-with-database ...llects/setup/doc-db.rkt:71:0
loop [52] 100.0%
-------------------------------------------------------------------------------------------
call-with-database [51] 100.0%
[52] 1180(43.6%) 50(1.8%) loop ...plt/racket/collects/setup/doc-db.rkt:507:2
??? [53] 66.5%
start-transaction method in transactions% [54] 29.3%
-------------------------------------------------------------------------------------------
loop [52] 100.0%
[53] 784(29.0%) 0(0.0%) ??? ...plt/racket/collects/setup/doc-db.rkt:523:11
call-with-lock* method in locking% [58] 62.3%
??? [55] 37.7%
-------------------------------------------------------------------------------------------
loop [52] 100.0%
[54] 346(12.8%) 0(0.0%) start-transaction method in transactions% ...338:4
call-with-lock* method in locking% [58] 100.0%
-------------------------------------------------------------------------------------------
??? [53] 100.0%
[55] 296(10.9%) 49(1.8%) ??? ...t/plt/racket/collects/setup/doc-db.rkt:94:3
~s [56] 50.0%
query-rows0 [57] 33.4%
-------------------------------------------------------------------------------------------
??? [55] 100.0%
[56] 148(5.5%) 148(5.5%) ~s ...plt/racket/collects/racket/format.rkt:150:16
-------------------------------------------------------------------------------------------
??? [55] 100.0%
[57] 99(3.7%) 0(0.0%) query-rows0 ...private/generic/functions.rkt:119:0
compose-statement [59] 50.0%
call-with-lock* method in locking% [58] 50.0%
-------------------------------------------------------------------------------------------
query-rows0 [57] 5.6%
start-transaction method in transactions% [54] 39.1%
??? [53] 55.3%
[58] 884(32.7%) 50(1.8%) call-with-lock* method in locking% ...on.rkt:209:4
??? [60] 88.8%
call-with-lock [61] 5.6%
-------------------------------------------------------------------------------------------
query-rows0 [57] 100.0%
[59] 50(1.8%) 0(0.0%) compose-statement ...te/generic/functions.rkt:90:0
find-method/who [70] 100.0%
-------------------------------------------------------------------------------------------
call-with-lock* method in locking% [58] 100.0%
[60] 784(29.0%) 50(1.8%) ??? ...ollects/db/private/generic/common.rkt:180:8
??? [62] 93.7%
-------------------------------------------------------------------------------------------
call-with-lock* method in locking% [58] 100.0%
[61] 50(1.8%) 0(0.0%) call-with-lock ...private/generic/common.rkt:172:2
lock-acquire/start-atomic [63] 100.0%
-------------------------------------------------------------------------------------------
??? [60] 100.0%
[62] 735(27.2%) 0(0.0%) ??? ...llects/db/private/generic/common.rkt:215:18
??? [64] 59.7%
??? [65] 40.3%
-------------------------------------------------------------------------------------------
call-with-lock [61] 100.0%
[63] 50(1.8%) 50(1.8%) lock-acquire/start-atomic ...eric/common.rkt:128:2
-------------------------------------------------------------------------------------------
??? [62] 100.0%
[64] 439(16.2%) 0(0.0%) ??? ...ollects/db/private/generic/common.rkt:360:8
end-transaction* method in connection% [66] 100.0%
-------------------------------------------------------------------------------------------
??? [62] 100.0%
[65] 296(10.9%) 0(0.0%) ??? ...ollects/db/private/generic/common.rkt:340:8
start-transaction* method in connection% [67] 100.0%
-------------------------------------------------------------------------------------------
??? [64] 100.0%
[66] 439(16.2%) 0(0.0%) end-transaction* method in connection% ...kt:384:4
query1 method in connection% [68] 100.0%
-------------------------------------------------------------------------------------------
??? [65] 100.0%
[67] 296(10.9%) 0(0.0%) start-transaction* method in connection% ...:364:4
query1 method in connection% [68] 100.0%
-------------------------------------------------------------------------------------------
start-transaction* method in connection% [67] 40.3%
end-transaction* method in connection% [66] 59.7%
[68] 735(27.2%) 47(1.7%) query1 method in connection% ...onnection.rkt:89:4
prepare1 method in statement-cache% [69] 46.7%
prepare1* method in connection% [73] 26.8%
finalize method in prepared-statement% [71] 6.7%
find-method/who [70] 6.7%
??? [72] 6.7%
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 100.0%
[69] 343(12.7%) 0(0.0%) prepare1 method in statement-cache% ...n.rkt:560:4
make-sql-classifier [74] 43.3%
prepare1* method in connection% [73] 42.3%
use-cache? method in statement-cache% [75] 14.4%
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 50.0%
compose-statement [59] 50.0%
[70] 99(3.7%) 99(3.7%) find-method/who ...ivate/class-internal.rkt:3969:0
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 100.0%
[71] 50(1.8%) 50(1.8%) finalize method in prepared-statement% ...rkt:73:4
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 100.0%
[72] 49(1.8%) 49(1.8%) ??? ...s/racket/private/class-internal.rkt:2619:34
-------------------------------------------------------------------------------------------
prepare1 method in statement-cache% [69] 42.4%
query1 method in connection% [68] 57.6%
[73] 342(12.6%) 0(0.0%) prepare1* method in connection% ...ction.rkt:273:4
??? [76] 56.6%
do-make-object/real-class [77] 43.4%
-------------------------------------------------------------------------------------------
prepare1 method in statement-cache% [69] 100.0%
[74] 148(5.5%) 50(1.8%) make-sql-classifier ...te/generic/common.rkt:609:2
sql-skip-comments [78] 66.7%
-------------------------------------------------------------------------------------------
prepare1 method in statement-cache% [69] 100.0%
[75] 50(1.8%) 50(1.8%) use-cache? method in statement-cache% ...rkt:477:4
-------------------------------------------------------------------------------------------
prepare1* method in connection% [73] 100.0%
[76] 194(7.2%) 194(7.2%) ??? ...ts/db/private/sqlite3/connection.rkt:470:22
-------------------------------------------------------------------------------------------
prepare1* method in connection% [73] 100.0%
[77] 148(5.5%) 50(1.8%) do-make-object/real-class ...s-internal.rkt:3629:0
continue-make-object [79] 66.7%
-------------------------------------------------------------------------------------------
make-sql-classifier [74] 100.0%
[78] 99(3.7%) 99(3.7%) sql-skip-comments ...vate/generic/common.rkt:616:0
-------------------------------------------------------------------------------------------
do-make-object/real-class [77] 100.0%
[79] 99(3.7%) 0(0.0%) continue-make-object .../class-internal.rkt:3655:0
??? [80] 100.0%
-------------------------------------------------------------------------------------------
continue-make-object [79] 100.0%
[80] 99(3.7%) 0(0.0%) ??? ...llects/db/private/generic/prepared.rkt:14:2
extract-arg [81] 100.0%
-------------------------------------------------------------------------------------------
??? [80] 100.0%
[81] 99(3.7%) 0(0.0%) extract-arg ...t/private/class-internal.rkt:3767:0
assq [82] 100.0%
-------------------------------------------------------------------------------------------
extract-arg [81] 100.0%
[82] 99(3.7%) 99(3.7%) assq ...et/collects/racket/private/list.rkt:187:13
-------------------------------------------------------------------------------------------
from drracket.
6cdh
commented on July 18, 2024
I saw the usage of the database in [50] pkgs/racket-index/setup/xref.rkt is query only. Is it possible to preprocess the database, eliminate the need for it, and use a hashtable instead?
from drracket.
rfindler
commented on July 18, 2024
Just doing a quick check, it looks like we get to that line 1970 times and in 1516 of them, the p or the key arguments are different.
from drracket.
6cdh
commented on July 18, 2024
I mean the hash table is faster than database query. This is a test:
I exported data from /usr/share/doc/racket/docindex.sqlite
$ cp /usr/share/doc/racket/docindex.sqlite a.sqlite
$ echo 'select * from documented' | sqlite3 a.sqlite > data.txtthen use this script
#lang racket
(define ht (make-hash))
(define stags '())
(for ([line (file->lines "data.txt")])
(match (string-split line "|")
[(list stag pathid)
(hash-set! ht stag (string->number pathid))
(set! stags (cons stag stags))]
[_ (error "unknown item:" line)]))
(define (hash-query keys)
(for/list ([key keys])
(hash-ref ht key)))
(require db)
(define (db-query conn keys)
(for/list ([key keys])
(define rows (query-rows conn
(virtual-statement "SELECT pathid FROM documented WHERE stag=$1")
key))
(define pathid (and (pair? rows) (vector-ref (car rows) 0)))
pathid))
(define keys (take (shuffle stags) 10000))
(length keys)
(time (void (hash-query keys)))
(define conn (sqlite3-connect #:database "a.sqlite"
#:mode 'read-only
#:busy-retry-limit 0))
(time (void (db-query conn keys)))
(equal? (hash-query keys) (db-query conn keys))result:
10000
cpu time: 8 real time: 8 gc time: 0
cpu time: 897 real time: 906 gc time: 73
#tfrom drracket.
6cdh
commented on July 18, 2024
The function call-with-retry/transaction is slow, but we will not need it if we preprocess the database and use hash table.
from drracket.
rfindler
commented on July 18, 2024
I think we don't want to stop using a database for the documentation.
Unless I misunderstand your meaning?
Robby
…On Wed, Aug 16, 2023 at 11:12 PM 6cdh ***@***.***> wrote:
The function call-with-retry/transaction is slow, but we will not need it
if we preprocess the database and use hash table.
—
Reply to this email directly, view it on GitHub
<#633 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADBNMA7FV4DS7WZTU6O4ZTXVWK3TANCNFSM6AAAAAA3MLSIF4>
.
You are receiving this because you commented.Message ID:
***@***.***>
from drracket.
6cdh
commented on July 18, 2024
No. I didn't mean stop using a database.
I mean only stop using the databse at the bottleneck part.
My idea is that only change the code at [49] pkgs/racket-index/setup/xref.rkt:122 from
(lambda (key use-id)
;; connect database
;; call-with-retry/transaction
;; database query
;; disconnect database
)to
(let ([hash-table (preprocess-database main-db user-db)])
(lambda (key use-id)
;; hash table access
))from drracket.
rfindler
commented on July 18, 2024
I am not following how this can be achieved. How do you know what to put
into the hash table?
…On Thu, Aug 17, 2023 at 6:39 AM 6cdh ***@***.***> wrote:
No. I didn't mean stop using a database.
I mean only stop using the databse at the bottleneck part.
My idea is that only change the code at [49]
pkgs/racket-index/setup/xref.rkt:122 from
(lambda (key use-id)
;; connect database
;; call-with-retry/transaction
;; database query
;; disconnect database
)
to
(let ([hash-table (preprocess-database main-db user-db)])
(lambda (key use-id)
;; hash table access
))
—
Reply to this email directly, view it on GitHub
<#633 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADBNMGTVYLG2D3Y6IHUCATXVX7FTANCNFSM6AAAAAA3MLSIF4>
.
You are receiving this because you commented.Message ID:
***@***.***>
from drracket.
6cdh
commented on July 18, 2024
All rows in the table documented of the database as the function doc-db-key->path only cares that table.
The schema is
sqlite> .schema documented
CREATE TABLE documented (stag VARCHAR(256), pathid SMALLINT, UNIQUE (stag, pathid));
CREATE INDEX documentedStags on documented (stag);
Currently the number of rows is
sqlite> select count(*) from documented;
83935
It's not too large.
from drracket.
Related Issues (20)
- DrRacket top bar is unreadable when the MacOs change theme with setting appearance auto HOT 2
- Unstable (crashing) on macOS 14.0 Public Beta (23A5301h) HOT 1
- Proposal: New info key `drracket:comments` HOT 34
- Delete unsaved file, keep backup => DrRacket crashes on startup
- Pressing the X for a failed test run makes DrRacket go into fullscreen when it shouldn't HOT 5
- Can't press up to move the caret when there's a word wrapping
- Update "DrRacket support for #lang-based Languages" documentation? HOT 2
- Wrong background color for recycling symbol. HOT 5
- Gtk issue on close-tab HOT 4
- Memory leaks to 32gb within 10 seconds HOT 5
- `ffi` in drracket won't use new shared library after modification & recompiling HOT 4
- Error when launching DrRacket HOT 9
- The indentation guide covers the #lang line HOT 3
- Theme selection buggy HOT 6
- Unequal preview box width
- Indent guide should not be shown on word wrapped lines
- Automatic Theme Sync in DrRacket HOT 2
- DrRacket window resizing behaviour
- DrRacket Getting Stuck Auto-saving Files HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drracket.