Comments (18)
I install SQLite. It solved the problem and now it's working. Thanks for the help!
from dbt5.
Sry to bother u, what's your result after the cmd "dbt5-build"?
Are the tables in database dbt5 empty or having data?
from dbt5.
I have data in tables. As an exampple, some data from broker table:
dbt5=# select * from broker;
b_id | b_st_id | b_name | b_num_trades | b_comm_total
------------+---------+------------------------+--------------+--------------
4300000011 | ACTV | Michael X. Gramlich | 1706834 | 50529291.58
4300000012 | ACTV | Oscar Z. Wolfer | 1717247 | 50634252.98
4300000013 | ACTV | Sam V. Peffer | 1736180 | 50967066.08
4300000014 | ACTV | Timothy W. Yerbic | 1842448 | 54202883.19
4300000015 | ACTV | James Q. Knoll | 1644365 | 48127252.42
Becouse of file system i change folder for .txt files with data.
So as a result i see different tables with some data. And after that i start "dbt5 run".
from dbt5.
The err info in my bh.out is "Error: Unrecognized option: u", i have no idea about this.
from dbt5.
main...NovDm:dbt5:main
changes that i did to my fork.
i add -q option to dbt5-run to specify dbuser for BrokerageHouseMain work. becouse i work under the user which doesn't match any dbuser.
so -q arg is parsed and user as -u option for start of BrokerageHouseMain
eval "${EGENHOME}/bin/BrokerageHouseMain ${DB_HOSTNAME_ARG} -d ${DB_NAME} \ ${DB_PORT_ARG} ${DB_USER_ARG} -o ${BH_OUTPUT_DIR} > ${BH_OUTPUT_DIR}/bh.out 2>&1"
in my case ${DB_USER_ARG} is "-u postgres"
from dbt5.
right, i followed u added this -q option
so should i add the 'u' in the BrokerageHouseMain.cpp parse_command_line?
from dbt5.
========================================== ==================================
Reported Throughput: 0.00 trtps Configured Customers: 5000
========================================== ==================================
================== ========== ========== ========== ==========
Response Times (s) Minimum Average 90th %tile Maximum
================== ========== ========== ========== ==========
Broker Volume 0.00 0.00 0.00 0.00
Customer Position 0.00 0.00 0.00 0.00
Market Feed 0.00 0.00 0.00 0.00
Market Watch 0.00 0.00 0.00 0.00
Security Detail 0.00 0.00 0.00 0.00
Trade Lookup 0.00 0.00 0.00 0.00
Trade Order 0.00 0.00 0.00 0.00
Trade Result 0.00 0.00 0.00 0.00
Trade Status 0.00 0.00 0.00 0.00
Trade Update 0.00 0.00 0.00 0.00
Data Maintenance 0.00 0.00 N/A 0.00
================== ========== ========== ========== ==========
================== ========== ========== ========== ========== ==========
Transaction Mix Txn Count Mix %tile Rollbacks Warnings Invalid
================== ========== ========== ========== ========== ==========
Broker Volume 4814 5.501 4814 0 0
Customer Position 12792 14.617 12792 0 0
Market Feed 0 0.000 0 0 0
Market Watch 17702 20.227 17702 0 0
Security Detail 13774 15.739 13774 0 0
Trade Lookup 7862 8.984 7862 0 0
Trade Order 9931 11.348 9931 0 0
Trade Result 0 0.000 0 0 0
Trade Status 18668 21.331 18668 0 0
Trade Update 1968 2.249 1968 0 0
Data Maintenance 1 N/A 1 0 0
================== ========== ========== ========== ========== ==========
================================================================== ==========
Test Duration and Timings
================================================================== ==========
Ramp-up Time (minutes) 0.0
Measurement Interval (minutes) 1.0
Total Number of Transactions Completed in Measurement Interval 87516
================================================================== ==========
OMG, why the reult is like this?
from dbt5.
yes there should be some corrections in BrokerageHouse.cpp and BrokerageHouseMain.cpp for -u option.
We need this to be added here
// new database connection
pDBConnection = new CDBConnection(pThrParam->pBrokerageHouse->m_szHost,
pThrParam->pBrokerageHouse->m_szDBName,
pThrParam->pBrokerageHouse->m_szDBPort);
Also some changes in CDBConnection .cpp and .h...
from dbt5.
you see better picture than i see.
In my case Total Number of Transactions Completed in Measurement Interval is zero...
from dbt5.
Good day! Try to build and start benchmark on debian12. I have made some changes and create a fork. Due to file system and os users add some new options for files. Also solve the problem with libpq-dev by changing LDFLAGS with LIBS in dbt5-build-egen.in file.
What's broken here is not obvious to me, but I believe using LIBS instead of LDFLAGS is just making these flags show up in egen's Makefile in different places, because it blindly pulls in LDFLAGS and LIBS from the environment. I don't believe using LIBS is correct, so I wondering what's really happening...
I use flat generation and remove DCOMPILE_CUSTOM_LOAD becouse i cant finde files for custom load and have some error with it.
Using the -l FLAT
command line argument with dbt5-build should have worked, unless something else was broken.
The custom load files for postgres are in src/include/custom. It'll save disk space and time to be able to use the direct loading, but if the flat files are working for the time being, then we can worry about that later.
Commands for generate:
make -f Makefile.cmake debug
dbt5-build-egen --include-dir=/opt/dbt5 --source-dir=/opt/dbt5/src --patch-dir=/opt/dbt5/patches /tmp/egen
dbt5-build -c 5000 -t 5000 --db-host=localhost --db-port=5432 --db-user=postgres -o /u02 --tpcetools=/tmp/egen -m FLAT pgsql
than i tried to start:dbt5 run -c 1000 -t 1000 -n dbt5 -q postgres -h localhost -d 120 --tpcetools=/tmp/egen -p 5432 pgsql /tmp/results
and i get this: Reported Throughput: 0.00 trtps. Also there is some errors in brokerageHouse work:-->dbt5 - Brokerage House -->Listening on port: 30000 --> -->Using the following database settings: --> Database hostname: localhost --> Database port: 5432 --> Database name: dbt5 -->Using the following Market Exchange Emulator settings: --> Hostname: localhost --> Port: 30010 -->Brokerage House opened for business, waiting for traders... -->*** array length (0) does not match expections (20): ../src/transactions/pgsql/DBConnection.cpp:2368 -->*** array length (0) does not match expections (20): ../src/transactions/pgsql/DBConnection.cpp:2389 -->*** array length (0) does not match expections (20): ../src/transactions/pgsql/DBConnection.cpp:2162 -->*** array length (0) does not match expections (20): ../src/transactions/pgsql/DBConnection.cpp:2183 -->*** array length (0) does not match expections (20): ../src/transactions/pgsql/DBConnection.cpp:2162 -->*** array length (0) does not match expections (20): ../src/transactions/pgsql/DBConnection.cpp:2183 -->*** array length (2) does not match expections (1): ../src/transactions/pgsql/DBConnection.cpp:591 -->*** array length (3) does not match expections (2): ../src/transactions/pgsql/DBConnection.cpp:591 -->*** array length (4) does not match expections (2): ../src/transactions/pgsql/DBConnection.cpp:591 -->*** array length (5) does not match expections (3): ../src/transactions/pgsql/DBConnection.cpp:591
and so on. Full file is here bh.txt Can you help me to fix this?
These "array length" messages are more for debugging than an indication that the code is fatal (i.e. the c++ code should not be crashing and it will attempt to continue more transactions). They are an indication that the transaction logic may be bad (i.e. the SQL statements might no be right) and supposedly these are possible with correct statements even at the smaller data base scale factors, according to the benchmark specification.
Maybe we can understand more from the database logs if you can share them somewhere.
from dbt5.
OMG, why the reult is like this?
Are you able to share the postgresql logs? The BrokerageHouse_Error.log may also have some clues.
from dbt5.
I've since cut a v0.10.0 release. Using the client side app logic might be easier to debug database related issues, but there aren't any script or usage changes otherwise.
from dbt5.
Good day! Try to build and start benchmark on debian12. I have made some changes and create a fork. Due to file system and os users add some new options for files. Also solve the problem with libpq-dev by changing LDFLAGS with LIBS in dbt5-build-egen.in file.
What's broken here is not obvious to me, but I believe using LIBS instead of LDFLAGS is just making these flags show up in egen's Makefile in different places, because it blindly pulls in LDFLAGS and LIBS from the environment. I don't believe using LIBS is correct, so I wondering what's really happening...
And it just occurred to me that some linkers (or is it because of the libraries, I don't remember) are sensitive to the location of the -l flags. You are right that that -l flags should be in LIBS with the way the EGen Makefile is written. I have a patch baking to split that but slightly differently than what you did.
diff --git a/scripts/dbt5-build-egen.in b/scripts/dbt5-build-egen.in
index 9427e3b4..3ef2d65d 100755
--- a/scripts/dbt5-build-egen.in
+++ b/scripts/dbt5-build-egen.in
@@ -151,6 +151,7 @@ fi
(cd "${EGENDIR}/prj" && \
make -f Makefile clean && \
CCFLAGS="-Wall -D__unix -g -std=c++98 -ansi -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -D_LARGEFILE64_SOURCE -DCOMPILE_CUSTOM_LOAD -DPGSQL -I${EGENDIR}/inc ${INCLUDEDIR} -I${EGENDIR}/Utilities/inc ${EXTRAINCLUDE}" \
- LDFLAGS="-lpthread -lpq ${EXTRALDDIR}" \
+ LDFLAGS="${EXTRALDDIR}" \
+ LIBS="-lpthread -lpq" \
make -j$(nproc) -f Makefile) \
|| (echo "ERROR: cannot build egen" && exit 1)
from dbt5.
I will try this patch. As for the logs, I'll upload them in a couple of days.
from dbt5.
FYI, I flushed out a bunch of issues I ran into with the pl/c code and the C++ code. I cut a v0.10.2 release. Some may have been contributing to the experience here.
from dbt5.
I took last version, and try to run it. Add cp -pr ${SOURCEDIR}/include/* ${EGENDIR}/inc/
to scripts/dbt5-build-egen.in. Realy dont know why, but it didn't copy automaticaly. Use your variant of LDFLAGS and LIBS. Try to start custom or flat load but in both cases, i got 0 trps.
My changes for now: https://github.com/osdldbt/dbt5/compare/main...NovDm:dbt5:main
Attach work log, results folder and operation log from postgres. I set log_min_duration_statement = 250ms.
postgresql-Thu.log
work_log.txt
And archive with files from result folder.
results.zip
from dbt5.
I see some work in database, but i dont know why it has 0 trps without any errors
from dbt5.
Attach work log, results folder and operation log from postgres.
A quick look at the brokerage house, market exchange, and driver output files do look like the workload ran correctly.
The output in post-process.log suggests it's just sqlite3 is missing. sqlite3 is used to calculate the metrics. When I process your files by hand, i get this:
% find /tmp/results -name 'mix*log' | xargs dbt5 post-process
========================================== ==================================
Reported Throughput: 70.00 trtps Configured Customers: Unspecified
========================================== ==================================
================== ========== ========== ========== ==========
Response Times (s) Minimum Average 90th %tile Maximum
================== ========== ========== ========== ==========
Broker Volume 0.00 0.02 0.06 0.11
Customer Position 0.00 0.48 1.31 3.03
Market Feed 0.02 0.05 0.09 0.17
Market Watch 0.02 1.57 3.06 5.02
Security Detail 0.00 0.05 0.12 0.21
Trade Lookup 0.05 35.30 147.71 159.65
Trade Order 0.00 0.04 0.09 0.28
Trade Result 0.00 0.02 0.04 0.13
Trade Status 0.09 1.03 1.82 3.67
Trade Update 0.17 51.84 155.52 165.41
Data Maintenance 0.01 0.01 N/A 0.01
================== ========== ========== ========== ==========
================== ========== ========== ========== ========== ==========
Transaction Mix Txn Count Mix %tile Rollbacks Warnings Invalid
================== ========== ========== ========== ========== ==========
Broker Volume 82 4.751 0 0 0
Customer Position 192 11.124 0 0 0
Market Feed 14 0.811 0 0 0
Market Watch 276 15.991 0 0 0
Security Detail 211 12.225 0 0 0
Trade Lookup 104 6.025 0 0 0
Trade Order 158 9.154 3 0 0
Trade Result 140 8.111 0 0 0
Trade Status 316 18.308 0 0 0
Trade Update 29 1.680 0 0 0
Data Maintenance 2 N/A 0 0 0
================== ========== ========== ========== ========== ==========
================================================================== ==========
Test Duration and Timings
================================================================== ==========
Ramp-up Time (minutes) 1.7
Measurement Interval (minutes) 2.0
Total Number of Transactions Completed in Measurement Interval 1726
================================================================== ==========
Hopefully installing SQLite takes care of that.
from dbt5.
Related Issues (18)
- Sub-transaction for trade_cleanup.sql Frame 1 HOT 2
- Sub-transaction for trade_update.sql Frame 1 HOT 2
- Sub-transaction for market_feed.sql Frame 1 HOT 4
- Sub-transaction for market_feed.sql Frame 1
- Sub-transaction for trade_update.sql Frame 3 HOT 1
- Sub-transaction for trade_order.sql Frame 5 HOT 1
- Sub-transaction for trade_order.sql Frame 6 HOT 1
- Sub-transaction for trade_result.sql Frame 6 HOT 1
- Sub-transaction for data_maintenance.sql Frame 1 HOT 1
- Build issues on Debian HOT 8
- custom postgresql loader is not generating the correct number of rows HOT 2
- Error during egen build on IBM power ppc64le HOT 5
- Possible type mismatch for t_comm parameter in C version of TRF5_1? HOT 4
- update pgsql client-side code to use parameterized statements HOT 1
- update postgres stored function calls to use parameterized sql HOT 1
- add controls to build script to specify which stored functions/procedures to use
- brokeragehousemain, marketexchangemain, drivermain not properly handling uppercase arguments HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt5.