Giter Site home page Giter Site logo

trackair / 1brc Goto Github PK

View Code? Open in Web Editor NEW

This project forked from gunnarmorling/1brc

0.0 0.0 0.0 2.6 MB

1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java

Home Page: https://www.morling.dev/blog/one-billion-row-challenge/

License: Apache License 2.0

Python 3.20% Java 87.38% Go 8.57% Dockerfile 0.85%

1brc's Introduction

版本 time
baseline 247906ms
TrackMMap.java 51103ms
TrackThread.java 6828ms
GraalVM 3122ms

基线火焰图

avatar

优化1: IO优化

利用mmap减少BufferedReader多次读取文件字符串

优化2: 减少字符串强转

发现paraseDouble占用特别大，我可以利用数据只保留一位小数前提下，将文本中读取到的字节码转为int运行，避免字符串重复拷贝

优化3: 多线程处理

可以利用多线程分割文件，同时去计算每个分块中最小，最大，总和

优化4：GraalVM

本机运用GraalVM可以缩小到3s左右

大佬优化

自定义哈希表

指定大小：不需要考虑容量扩张，因为比赛的气象站大小已经确定 开放地址法：如果hash值对应的槽位（slot）已经存在，那么一直向右移动到第一个空的槽位 计算hash值和判断key是否相等：直接通过读取文件对应字节计算和判断，避免创建String对象

作者提醒的优化方向:

1.使用sun.misc.Unsafe取代MemorySegment，避免边界校验 2.避免重新重新读取同一个字节：复用已经加载的值做哈希计算和分号搜索 3.一次处理8个字节，使用SWAR技术搜索分号 4.通过统计气象站名称长度分布，优化名称比较的分支预测命中率

具体规则和查看官方README2.md

1brc's People

Contributors

gunnarmorling avatar hundredwatt avatar alexanderyastrebov avatar artsiomkorzun avatar ebarlas avatar mtopolnik avatar royvanrijn avatar abeobk avatar thomaswue avatar filiphr avatar zerninv avatar roman-r-m avatar jerrinot avatar ianopolousfast avatar ddimtirov avatar serkan-ozal avatar gonix avatar jgrateron avatar trackair avatar iziamos avatar merykitty avatar yavuztas avatar gamlerhart avatar richardstartin avatar nipafx avatar kuduwa-keshavram avatar ianopolous avatar gnabyl avatar armandino avatar panagiotisdrakatos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.