niiknow/text-file-diff

line by line diff of two large text files

text-file-diff

line by line diff of two large (sorted) text files

This is especially useful with csv/tsv/psv files:

• compare products file for changes to import
• compare log files
• compare database export of large organization employee/users

NOTE

The benefits of this script is that it allow user full control of their file diffs. It work great on large file because it does not require you to load the entire file into memory or string.

This script expect input of two sorted text files. If the files are not sorted, the unix sort command may be of help: https://en.wikipedia.org/wiki/Sort_(Unix)

ForEach line in File1, compare to line in File2 
   equal: incr both files to next line
   line1 > line2: new line detected, incr File2 to next line
   line1 < line2: deleted line, incr File1 to next line

Since the list will be sorted, the performance of this script is expected to be approximately O(|A| log |A| + |B| log |B|), where A is File1 and B is File2.

This library also has no dependency.

Install

$ npm install text-file-diff

Usage

import TextFileDiff from 'text-file-diff';
const m = new TextFileDiff();
m.on('compared', (line1, line2, compareResult, lineReader1, lineReader2) => {
    // event triggered immediately after line comparison
    // but before +- event
  });

m.on('-', line => {
    // when a line is in file1 but not in file2
  });

m.on('+', line => {
    // when a line is in file2 but not in file1
  });

// run the diff
m.diff('tests/file1.txt', 'tests/file2.txt');

// or run m.diffStream(stream1, stream2)

TextFileDiff constructor also accept optional options object. Valid options are:

1. skipHeader - boolean indicating if first line should be skipped
2. compareFn - line comparer function that take in two line (line1 and line2), and return line1 > line2 ? 1 : (line1 < line2 ? -1 : 0) allowing you to define how you want to perform line compare.

NOTE: use diffStream if you need control over encoding

Example

$ ./bin/text-file-diff tests/file1.txt tests/file2.txt

Point of Interest

• Alternate shell 'diff' command:

$ diff -u f1.txt f2.txt | sed -n '1,2d;/^[-+|]/p' | sed 's/^\(.\{1\}\)/\1|/'

• https://github.com/paulfitz/daff - multi languages diff framework
• https://github.com/kpdecker/jsdiff - popular nodejs string diff

MIT

text-file-diff's Issues

Great package.. would be awesome to also have a types file :)

would be great to have a types file then can be used in typescript too

Be able work with in-memory strings / streams.

It would be great if we could work with in-memory strings / streams.
What if there is no actual file on the fs?

I believe it could be accomplished by abstracting away the source of the strings from the work on them.

How to ignore line sort?

compare files of different length

I really like this package! Thanks for your hard work!

What I wish it could do: when one file is longer (has more lines) than the other, that it doesn't stop but append all the missing lines to the output (either as being new (+) or old (-) )

Infinite loop comparing file with leading lower case character

Some unexpected behaviour when comparing an empty file with some file containing a line of text with a lower case character.

Behaviour of the tools is different, when using e.g. a "some" instead of a "Some" prefix:'

Expected output:

line comparison: "some,csv,data" vs. "null
compare result: -1"
some,csv,data

Observed:

line comparison: "some,csv,data" vs. "null
compare result: 1"
null
line comparison: "some,csv,data" vs. "null
compare result: 1"
null
line comparison: "some,csv,data" vs. "null
compare result: 1"
null
....

const { assert } = require('chai')
const TextFileDiff = require('text-file-diff')
const fs = require('fs')

const collaboratorSourceFile = __dirname + "/test-source.csv"
const collaboratorFileToCompare = __dirname + "/test-backup.csv"

const getFileDiff = function (sourceFile, fileToCompare) {
  let diff = '';
  const textFileDiff = new TextFileDiff();
  textFileDiff.on('compared', (line1, line2, compareResult, lineReader1, lineReader2) => {
    console.log(`line comparison: "${line1}" vs. "${line2}`)
    console.log(`compare result: ${compareResult}"`)
  });

  textFileDiff.on('-', line => {
    console.log(line) // when a line is in file1 but not in file2
    diff = (diff.length === 0 ? '' : diff + '\r\n') + '-' + line
  });

  textFileDiff.on('+', line => {
    console.log(line) // when a line is in file2 but not in file1
    diff = (diff.length === 0 ? '' : diff + '\r\n') + '+' + line
  });

  textFileDiff.diff(sourceFile, fileToCompare);

  return diff;
}


describe('TextFileDiff', () => {

  it('create a correct diff with sourceFile content and empty fileToCompare', () => {
    fs.closeSync(fs.openSync(collaboratorSourceFile, 'w'));
    fs.closeSync(fs.openSync(collaboratorFileToCompare, 'w'));
    fs.writeFileSync(collaboratorSourceFile, 'Some,csv,data')

    let diff = getFileDiff(collaboratorSourceFile,collaboratorFileToCompare)

    assert.equal('-Some,csv,data', diff)
  })

  it('create a correct diff with another sourceFile content and empty fileToCompare', () => {
    fs.closeSync(fs.openSync(collaboratorSourceFile, 'w'));
    fs.closeSync(fs.openSync(collaboratorFileToCompare, 'w'));
    fs.writeFileSync(collaboratorSourceFile, 'some,csv,data')

    let diff = getFileDiff(collaboratorSourceFile,collaboratorFileToCompare)

    assert.equal('-some,csv,data', diff)
  })

})
``´