paulsena / cribl-log-collector-interview Goto Github PK
View Code? Open in Web Editor NEWTake home interview project for Cribl. This is a log file collector web service.
Take home interview project for Cribl. This is a log file collector web service.
when parsing a 1gb+ text file, the service crashes
Trying to read in logs from a file within the node_logs
directory from https://zenodo.org/records/8196385/files/HDFS_v2.zip?download=1 gives the following result:
> curl -u cribl:password "http://localhost:8080/cribl/log/tail?filename=hadoop-hdfs-datanode-mesos-01.log&numEntries=1"
{"type":"about:blank","title":"Bad Request","status":400,"detail":"File name or filter input is invalid. Only alphanumeric characters, numbers, spaces, and periods are allowed.","instance":"/cribl/log/tail"}
Test issue - ignore, just troubleshooting something for a Cribl team-mate who is reviewing this PR
andy@Andy-laptop ~/src/interview/Cribl-Log-Collector-Interview (main) $ ./mvnw test
...
...
...
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.650 s -- in com.cribl.logcollector.SpringBootApplicationTests
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] CriblFileWatcherTest.setUp:37 » NoSuchFile data/test.txt
[INFO]
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.665 s
[INFO] Finished at: 2024-02-13T10:44:28-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project CriblLogCollector:
[ERROR]
[ERROR] Please refer to /home/andy/src/interview/Cribl-Log-Collector-Interview/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
running ./mvnw test
results in a failure with NoSuchFile data/test.txt
Steps to reproduce:
curl -u cribl:password "http://localhost:8080/cribl/log/tail?filename=HDFS.log&numEntries=1"
(update the file name to match your uncompressed file)Expected result: getting just the last entry would be nearly instantaneous - reading only the last line should be fairly performant
Actual result: locally for me, this takes about 7 seconds
I did note that successive queries are faster due to the caching you've implemented... however, this seems to be consuming significantly more memory than I would expect.
Here's a top
capture from before performing the api call to tail the file:
and then again after:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.