Comments (8)
from univocity-trader.
done in the latest commit
from univocity-trader.
from univocity-trader.
Thanks Jbax, I was counting on buying your optimizer jar and using it as well, I don't think both things are mutually exclusive. I'll send you an email about that.
Having support for an embedded database isn't as much for performance (althought we'll get to that later) as convenience. Use cases:
- Unit tests and examples would benefit from using an embedded database, preferably an in-memory database. That way they don't depend on an external installation of a database and newcomers to the project (like me) can get up and running faster. Yeah, you can always use the FileCandleRepository, but arguably using an embedded database is more versatile.
- For development you don't really need to use a MySQL database (installed or using docker like with mykolakaraman PR), I think it would be convenient for dev to use an embedded database (you can always use MySQL/MariaDB if you like, it's about choice).
- Wen deploying that to a server you can install MySQL (either on the OS or using Docker). But when using a high powered server in AWS that you may run for 1 hour or less and then destroy it would be cumbersome to have to install MySQL every time and then import the database dump, so using an embedded database which is literally copying a file which contains the database can't get easier and more convenient, specially if you're going to destroy the server one hour later.
About performance, that is difficult to say, I'd go with it depends. In some cases an embedded database can be much faster than a server/client database like MySQL, specially if the embedded database is using in-memory mode and MySQL uses TCP. I'd go as far as to say that even when using file mode it can be faster as well. I haven't done benchmarks myself but what I have found online seem to point in that direction...
http://www.h2database.com/html/performance.html
I know that link is from h2 itself, I know I've seen benchmarks in other places but cant' find them now.
In summary, H2 looks like it would be faster than MySQL, specially H2 in-memory, but I'm not sure if it could have issues with very large datasets (hundreds of millions of rows of candlestick historical data for multiple tickers). But since it's way more convenient to deploy that to a short-lived server, I think it's worth giving it a try!
from univocity-trader.
I was trying that. For H2, you can try these values on application.yml
#
# H2 Database
#
database.jdbc.driver=org.h2.Driver
database.jdbc.url=jdbc:h2:~/data/univocity.db;MODE=MySQL;DATABASE_TO_LOWER=TRUE
database.user=sa
database.password=
For tests I was looking at something like this
private static final String H2_DRIVER = "org.h2.Driver";
private static final String JDBC_URL = "jdbc:h2:mem:univocity;DB_CLOSE_DELAY=-1;MODE=MySQL;DATABASE_TO_LOWER=TRUE;INIT=runscript from 'classpath:/db/h2/candle.sql'\\;runscript from 'classpath:/db/h2/gap.sql'";
private static final String DB_USER = "sa";
private static final String DB_PASSWORD = "dummy";
Strategy.Simulator simulator = Strategy.simulator();
simulator.configure().database()
.jdbcDriver(H2_DRIVER)
.jdbcUrl(JDBC_URL)
.user(DB_USER)
.password(DB_PASSWORD);
On pom.xml
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.200</version>
</dependency>
And finally for the creation scripts, on the resources
folder I added a db.h2
with h2 compatible versions of the scripts
CREATE TABLE IF NOT EXISTS candle
(
symbol VARCHAR(32) NOT NULL,
open_time BIGINT NOT NULL,
close_time BIGINT NOT NULL,
open DECIMAL(20, 10) NOT NULL,
high DECIMAL(20, 10) NOT NULL,
low DECIMAL(20, 10) NOT NULL,
close DECIMAL(20, 10) NOT NULL,
volume DECIMAL(20, 10) NOT NULL,
ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
;
ALTER TABLE candle ADD CONSTRAINT candle_symbol_time_uq UNIQUE (symbol, open_time, close_time);
CREATE INDEX candle_symbol_idx ON candle (symbol) USING HASH;
CREATE INDEX candle_symbol_open_idx ON candle (symbol, open_time) USING BTREE;
CREATE INDEX candle_symbol_close_idx ON candle (symbol, close_time) USING BTREE;
CREATE INDEX candle_open_close_idx ON candle (symbol, open_time, close_time) USING BTREE;
-- Once table has data, you can run this to keep all rows sorted in the database.
ALTER TABLE candle ORDER BY symbol, open_time, close_time;
and
-- Used to store known gaps in history that can't be retrieved from the exchange.
-- Delete rows from table if you want to try filling the gaps again.
CREATE TABLE IF NOT EXISTS gap
(
symbol VARCHAR(32) NOT NULL,
open_time BIGINT NOT NULL,
close_time BIGINT NOT NULL,
CONSTRAINT gap_symbol_time_uq UNIQUE (symbol, open_time, close_time)
);
from univocity-trader.
The creation scripts might need some work as some of the syntax may not be supported by H2
from univocity-trader.
from univocity-trader.
@jbax In fact, I have already changed to H2 database for normal operation, and the memory usage is very good. I will use MySQL database for local test and H2 for online run, MySQL and H2 have their own use scenarios, so you don't need to favor any one
from univocity-trader.
Related Issues (20)
- Setup a discord channel for developers HOT 1
- TD Ameritrade interface HOT 1
- Hope to support cross market Pairs trading HOT 4
- Unresolved dependencies when build with maven after cloning project HOT 1
- Build fails if you don't have Interactive Brokers jar HOT 2
- Build fails because of maven-bundle-plugin
- Add support for BInance Spot TestNet
- Add a new indicator
- Broken build HOT 8
- Take profit
- Previous candles HOT 2
- VWMA
- Last candles, price signal
- Deprecations of Binance APIs HOT 3
- Interactive Brokers resolution error in maven HOT 6
- Attachment orders missing in LiveTrading? HOT 12
- Setter for simulation backfill fails to set value
- Bracket orders for Binance
- Running Chart window HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from univocity-trader.