eagleflo/mpyq

Python library for reading MPQ archives.

mpyq's Issues

Fail gracefully when magic signatures not found

mpyq could have failed more gracefully with the corrupt file in #19.

MPQ Readfile Error

Hi, I've been using your library for sometime now as a dependency to my sc2reader library. One of my users recently reported an issue that stems from your read_file function; in particular:

 File "XXXX/mpyq/mpyq.py", Line 203, in read_file:
     for i in range(len(positions) - (2 if crc else 1)):

  UnboundLocalError: local variable `crc` referenced before assignment

Its a pretty simple issue but have trying to resolve it myself, I don't know the MPQ format well enough to devise the correct fix here. I have a file that generates this error, how should I send it to you?

Unit tests

mpyq definitely needs unit tests. I should actually have started with them, since most of development time was very exploratory. The main benefit I can currently see is protection against regressions -- now that the library is working, I want to refactor some methods inside without worrying about breaking anything. There isn't that much code yet so it's better to do it now than not at all.

Support for Python 3.x

Adding support for Python 3.x would not be a huge ordeal based on a small spike I did.

Invalid hash_table_offset?

This file has a length of 65892 bytes but archive.header has the following offset values:

• 'hash_table_offset' = 16421632
• 'block_table_offset' = 16487168

This makes table data an empty string and causes a struct.error when you try to unpack the table entries.

  File "/home/graylinkim/projects/sc2reader/env/bin/mpyq", line 8, in <module>
    load_entry_point('mpyq==0.2.0', 'console_scripts', 'mpyq')()
  File "/home/graylinkim/projects/mpyq/mpyq.py", line 392, in main
    archive = MPQArchive(args.file)
  File "/home/graylinkim/projects/mpyq/mpyq.py", line 99, in __init__
    self.hash_table = self.read_table('hash')
  File "/home/graylinkim/projects/mpyq/mpyq.py", line 170, in read_table
    return [unpack_entry(i) for i in range(table_entries)]
  File "/home/graylinkim/projects/mpyq/mpyq.py", line 168, in unpack_entry
    struct.unpack(entry_class.struct_format, entry_data))
struct.error: unpack requires a string argument of length 16

I want to turn around and say that the MPQ file is corrupt and there is nothing we can do but the person submitting the files (see GraylinKim/sc2reader#100) is suggesting that it came directly from his SCII client and that it opens just fine. @dsjoerg says that he has received several more files with a similar issues that he can provide.

Is there some way we can prove this one way or another?

Add a changelog

For some reason I omitted creating one earlier. It's not too hard yet to just look at the commit history, but avoiding unnecessary extra work for users is a decent thing to do. Besides, it's available at PyPI as well.

MPQArchive.read_file Error

Using the latest version off Pypi...

graylin@graylin-laptop:/home/sc2reader$ mpyq -x "/media/34A86959A8691A9E/Program Files/StarCraft II/Versions/Base19679/patch.SC2Archive"
Traceback (most recent call last):
  File "/usr/local/bin/mpyq", line 9, in <module>
    load_entry_point('mpyq==0.1.11', 'console_scripts', 'mpyq')()
  File "/usr/local/lib/python2.6/dist-packages/mpyq.py", line 399, in main
    archive.extract_to_disk()
  File "/usr/local/lib/python2.6/dist-packages/mpyq.py", line 249, in extract_to_disk
    for filename, data in self.extract().items():
  File "/usr/local/lib/python2.6/dist-packages/mpyq.py", line 239, in extract
    return dict((f, self.read_file(f)) for f in self.files)
  File "/usr/local/lib/python2.6/dist-packages/mpyq.py", line 239, in <genexpr>
    return dict((f, self.read_file(f)) for f in self.files)
  File "/usr/local/lib/python2.6/dist-packages/mpyq.py", line 218, in read_file
    file_data[:4*(sectors+1)])
struct.error: unpack requires a string argument of length 8

I'm not really sure how to diagnose this. You've got your own copy of the file you can hopefully replicate with. Do you have any ideas what is going on here?

Test for an archive with empty files

This was broken under Python 3 for a long time.

Support file-objects being passed in the constructor

Sometimes you already have a file object and want to use that instead of feeding mpyq a file name. Simple, but appreciated :-)

Support for different localizations

Currently the library does not care about localization in any way. It should be possible to specify exact localization you are interested in when reading files. Might require some work to make extract() compatible with them.

Support for more compression schemes

So far the development of the library has been totally dominated by Starcraft II replay format, and each file inside SC2 replays is compressed by DEFLATE. I need to investigate other kinds of MPQs and support other compression schemes as I encounter them.

Issue with mpyq, StormReply, and Python 3

Hey,

So Blizzard recently released their heroprotocol and I have been doing some work on getting the whole thing into a tidy python package with some tests. One of the things that has come from this is the realization that everything works OK on Python 2.7 but falls over on Python 3.

You can see the results in Travis CI here.

TestHeroProtocolAllVersions.test_version[tests/replays/protocol32524.StormReplay] 
self = <tests.test_heroprotocol_all_versions.TestHeroProtocolAllVersions object at 0x2f65a10>
version = 'tests/replays/protocol32524.StormReplay'
    def test_version(self, version):
        search = '^(tests\/replays\/protocol)(\d*)(\.StormReplay)$'
        results = re.search(search, version)
        hp = HeroProtocol(version)
>       assert hp.version() == int(results.group(2))
tests/test_heroprotocol_all_versions.py:41: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
heroprotocol/heroprotocol.py:96: in version
    return self.decode_header()['m_version']['m_baseBuild']
heroprotocol/heroprotocol.py:91: in decode_header
    decoder = VersionedDecoder(contents, self.protocol.typeinfos)
heroprotocol/heroprotocol.py:72: in protocol
    header = decoder.instance(protocol29406.replay_header_typeid)
heroprotocol/decoders.py:173: in instance
    return getattr(self, typeinfo[0])(*typeinfo[1])
heroprotocol/decoders.py:252: in _struct
    self._expect_skip(5)
heroprotocol/decoders.py:185: in _expect_skip
    if self._buffer.read_bits(8) != expected:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <heroprotocol.decoders.BitPackedBuffer object at 0x2f65690>, bits = 8
    def read_bits(self, bits):
        result = 0
        resultbits = 0
        while resultbits != bits:
            if self._nextbits == 0:
                if self.done():
                    raise TruncatedError(self)
>               self._next = ord(self._data[self._used])
E               TypeError: ord() expected string of length 1, but int found
heroprotocol/decoders.py:62: TypeError

Was wondering if anyone here had a suggestion for tracking this down and possibly a suggestion for how to fix. Thanks in advance for your help! :)

argparse dependency must be installed manually

Python 2.6.4

When trying to use either setup.py install or easy_install I'm getting ImportError for the argparse module.

Fix: run easy_install argparse

Support MPQ format versions 3 and 4

See #19 for more details.

mistake in README.md

This is my first time posting to github so apologies if I'm putting this in the wrong spot. In README.md the example used to show how to read a file from an archive appears to call a non existent function.

this:

archive.read('replay.details')

should be this:

archive.read_file('replay.details').

Cross-Platform Filepaths

Hello,

Thanks for this tool! It still chugs along extracting Blizzard files :)

It appears when run on a Unix platform that files get extracted into single files concatenating subdirectory names with Windows separators e.g. a single file called A\B\file.xml. I'm not sure if that's due to Windows convention inside the particular files I have or if that's a universal property of MPQs.

It seems desirable to have a normal directory tree get extracted from that (and then to support recompressing from that normal directory tree).

Does the problem as I describe make sense? Would PRs for this functionality be of interest?

Thank you!

Tests for existing command line options

Some of then were broken under Python 3 for a long time.

Support for encrypted MPQs

I still haven't encountered any, but according to references encrypted MPQs are possible. Possibly Warcraft III maps?

Add support for MPQ files that don't begin with the MPQ header

According to references there are MPQ files which do not begin with the MPQ header, such as Install.exe from StarCraft and Brood War installation CDs. In these cases, either MPQ header or MPQ user data header begins at some multiple of 512 bytes from the beginning.