korolmi/qvdfile

QVD File reader and writer (exploratory version)

qvdfile

QVD File reader and writer (exploratory version)

This is python3 object created to explore QVD file, it can be used

• to read QVD file Metadata
• to read column values from QVD file
• to read rows from QVD file

Sample usage:

import qvdfile 

qvd = qvdfile.QvdFile ("test.qvd")

print("File has {} records".format(qvd.attribs["NoOfRecords"]))

print("File has column '{}' with first value of '{}'".format(
  qvd.fields[0]["FieldName"],qvd.getFieldVal(qvd.fields[0]["FieldName"],0)))
  
print("First row of the file is {}".format(qvd.getRow(0)))

The structure of QVD file is described in Wiki.

Performance

This is explarotary version and it is not suited for production usage. Code simplicity is the first priority. If you need better performance please contact the author: QVD files can be read much much faster by other version of software.

Reading fields

QVD file does not allow indexing - it is impossible to read field #N without reading (N-1) previous fields. This version contains no caching - if you read field #4 and then read field #5 you will read 4 fields first and then read 4 same fields again... This is done specificly for simplicity.

It is faster to read first fields then the last fields. It is nearly impossible to read field #10 000 000 (while it is OK to read field #1000).

Reading rows

The same is true for rows: the smaller the index the faster it is read. The reason is simple - QVD files usually contain first symbols in the first rows. I would not try to read row #100 000 (and will not try #1000 as well).

For performance contact the author.

Dependencies

Code is simple and depends only on bitstring module:

pip install bitstring

Unit tests

Unit tests are provided as a sort of documentation - they describe what to expect from the code (or how it was supposed to be used by the author).

qvdfile's Issues

[issue] very high complexity while reading large qvd file

I tried to use the code to read a file having ~1M records. The performance is very poor as it shows 700 hours to complete.
How can it be improved? What would you suggest as I'm using Linux for this task.
Thank you.

Fails on filenames including more than one '.'

When attempting to handle filenames/filepaths that include more than one '.' character, the library infers the wrong file type since it arbitrarily looks for the "qvd"/"xml" extension after the first period. Better to use a regular expression to make this determination. I fixed a fork of this by revising qvdfile.py by adding:

import re

And changing line 25 to:

self.mode = os.O_RDONLY if re.match('^.*qvd$', name, re.IGNORECASE) else os.O_WRONLY