paulsena/cribl-log-collector-interview

Take home interview project for Cribl. This is a log file collector web service.

cribl-log-collector-interview's Issues

Unable to access file within /var/log

From the acceptance criteria (point 4.a), there should be the ability to specify a filename with /var/log when selecting the log file to read. However, I'm currently unable to do this:

Out of Memory when trying to parse 1 gb text file

when parsing a 1gb+ text file, the service crashes

Hyphens should be allowed in log names

Trying to read in logs from a file within the node_logs directory from https://zenodo.org/records/8196385/files/HDFS_v2.zip?download=1 gives the following result:

> curl -u cribl:password "http://localhost:8080/cribl/log/tail?filename=hadoop-hdfs-datanode-mesos-01.log&numEntries=1"
{"type":"about:blank","title":"Bad Request","status":400,"detail":"File name or filter input is invalid. Only alphanumeric characters, numbers, spaces, and periods are allowed.","instance":"/cribl/log/tail"}

numEntries parameter is ignored when results are previously cached

Test issue

Test issue - ignore, just troubleshooting something for a Cribl team-mate who is reviewing this PR

test failures (missing data dir)

andy@Andy-laptop ~/src/interview/Cribl-Log-Collector-Interview (main) $ ./mvnw test

...
...
...

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.650 s -- in com.cribl.logcollector.SpringBootApplicationTests
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   CriblFileWatcherTest.setUp:37 » NoSuchFile data/test.txt
[INFO] 
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.665 s
[INFO] Finished at: 2024-02-13T10:44:28-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project CriblLogCollector: 
[ERROR] 
[ERROR] Please refer to /home/andy/src/interview/Cribl-Log-Collector-Interview/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

running ./mvnw test
results in a failure with NoSuchFile data/test.txt

Accessing last line in a file performs poorly

Steps to reproduce:

1. Download and unzip https://github.com/logpai/loghub/blob/master/HDFS#hdfs_v1 into the configured directory for the service
2. Start the service
3. curl -u cribl:password "http://localhost:8080/cribl/log/tail?filename=HDFS.log&numEntries=1" (update the file name to match your uncompressed file)

Expected result: getting just the last entry would be nearly instantaneous - reading only the last line should be fairly performant

Actual result: locally for me, this takes about 7 seconds

I did note that successive queries are faster due to the caching you've implemented... however, this seems to be consuming significantly more memory than I would expect.
Here's a top capture from before performing the api call to tail the file:

and then again after: