beardypig/pymp4

A Python MP4 Parser and toolkit

pymp4's Issues

Extend to EoF when size32 is 0

See the spec: https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap1/qtff1.html#//apple_ref/doc/uid/TP40000939-CH203-BBCGDDDF

0, which is allowed only for a top-level atom, designates the last atom in the file and indicates that the atom extends to the end of the file.

I don't see it's handled.

Update PyPy

Hello, I noticed that the PyPy version of pymp4 is missing the last two commits. Would it be possible to issue a new release?

The two commits contain many default values that are really handy, UNITY_MATRIX and especially the structure for the avc1 atom, which I think is really essential for MP4.

Update to construct 2.9

I'm trying to package a project that use pymp4, but find it challenging as pymp4 is based on construct 2.8, while construct 2.9 is now the default version in repositories..

There are a number of breaking changes, and I'm aware that the project hasn't received commits in a year, but it would be nice to be able to package pymp4 against recent version of its dependencies.

Missing `colr` box support

The parser doesn't appear to support parsing of the colr box (# ISO/IEC 14496-12:2022, Section 12.1.5) which leads to parsing failure when encountered. It is supported by pyisobmf: https://github.com/chemag/pyisobmff/blob/450f8549fc16c763707c1ba7c73f305700489d58/isobmff/iprp.py#L73

Majority of tests fail on Python 3.10 and newer

See the following pytest warning and summary information:

tests/test_box.py::BoxTests::test_ftyp_build
  /home/runner/.cache/pypoetry/virtualenvs/pymp4-_EUa4su2-py3.7/lib/python3.7/site-packages/construct/core.py:1000: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    if not isinstance(obj, collections.Sequence):

FAILED tests/test_box.py::BoxTests::test_ftyp_build - AttributeError: module 'collections' has no attribute 'Sequence'
FAILED tests/test_box.py::BoxTests::test_mdhd_build - AttributeError: module 'collections' has no attribute 'Sequence'
FAILED tests/test_box.py::BoxTests::test_mdhd_parse - ValueError: closing stream but 1 unwritten bytes remain, 8 is encoded unit
FAILED tests/test_box.py::BoxTests::test_moov_build - AttributeError: module 'collections' has no attribute 'Sequence'

All of these except one are failing because Python 3.9+ had Sequence move from collections to collections.abc, i.e.:

-from collections import Sequence
+from collections.abc import Sequence

Sequence is never used within pymp4's code, but it is used heavily by construct. This indicates that construct 2.8.8 does not have true support for 3.9+ and upgrading to a newer version of construct is likely required.

It's not immediately clear how we will proceed from here as I'm not sure when construct resolved this internally, and it is unsure if the update that does fix it needs any more fixes.

I would recommend looking into upgrading construct to 2.10.x. Updating to the newest 2.8.x release (2.8.22) doesn't seem to resolve this issue (after doing other changes, see https://github.com/devine-dl/pymp4/tree/construct-2.8.22-patch

A reminder that there is already a pull request by another kind person, #9, which seems to do a lot of the work for upgrading to 2.10.x

EDIT: Confusingly it seems support for Python 3.9 is there, and the warning states it's dropping support in Python 3.10. As of the current master, tests up as far as Python 3.9 do in fact succeed. Not sure where the difference is here. Either way, tests on Python 3.10+ support are definitely broken.

unable to get all box data

Hello, I apologize in advance if this is user error and not a bug. The mp4 file in question is: here

My code:

boxes = Box.parse_stream(open('init.mp4', 'rb'))
[box for box in boxes]

Gives output:
['offset', 'type', 'major_brand', 'minor_version', 'compatible_brands', 'end']

However when I use mp4dump (not pymp4 cli) I see the full box data. Am I doing something wrong?

more pull requests?

Not really an issue but I wasn't sure how to reach out. I noticed that you recently merged a couple of my pull requests so wondered if I should submit some more? I haven't touched my fork in a while but I had more changes I never worked into pull requests
e.g. basic udta parsing:
finnhughes@67f5a69
or enabling support for extended size fields in mdat:
finnhughes@7f846a7

PEP517

Could you move the buildsystem to PEP517?

Error trying to get box and box type

i am trying to get the box and box type using pymp4 using the following

    boxes = Box.parse_stream(io.BytesIO(data))
    for box in boxes:
        if box.type == b"senc":
            print("found")

but all i get is string dosent have type error and i dont see a lot of documentation on this project and how it can be used apart from a few examples

Definition of tenc box seems invalid or too leniant

I've attached a sample tenc box from https://sho.com, specifically the show Yellowjackets S02E05 on the 2160p track.

It fails to parse with the following error:

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python310\Scripts\mp4dump.exe\__main__.py", line 7, in <module>
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\pymp4\cli.py", line 26, in dump
    box = Box.parse_stream(fd)
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\construct\core.py", line 186, in parse_stream
    return self._parse(stream, context, "parsing")
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\pymp4\parser.py", line 46, in _parse
    obj = self.subcon._parse(stream2, context, path)
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\construct\core.py", line 855, in _parse
    subobj = list(sc._parse(stream, context, path).items())
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\construct\core.py", line 297, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\construct\core.py", line 1544, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\construct\core.py", line 859, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\User\AppData\Roaming\Python\Python310\site-packages\construct\core.py", line 2700, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.ConstError: expected 0 but parsed 25
    parsing -> _reserved1

It's failing at the byte 0xD with the value 0x19 (25 decimal). This offset would be the _reserved1 definition.

tenc_box_from_showtime_exact.zip

Recent version number tags are incorrect

The git tags for 1.3.0, 1.3.1, and 1.3.2 are wrong. You seemed to have some problems when doing the new release, but we still have the change to fix the tags.

Perhaps remove the tag for 1.3.0 and 1.3.1, but keep the tag for 1.3.2, since technically we are now locked into being 1.3.2 when it should have been 1.3.0 but seemingly issues prevented you from doing that.

Ubuntu-18.04 GitHub CI runner has been deprecated and fully removed

I'm not sure if you noticed but ever since we pulled the new GitHub CI/CD workflow setup, it stopped working and all GitHub workflow runs since then just freeze. This is because of unlucky timing. Once I finished that pull request and had it running to the point where tests just need fixing, GitHub decided to effectively remove that runner image entirely (see actions/runner-images#6002). So by the time you then pulled that PR, it was removed and unsupported.

This means as stated in #19 (comment) we would need to drop support for another version, Python 3.6. We could keep support for Python 3.6 by using ubuntu-20.04 instead of ubuntu-latest (22.04) but since Python 3.6 is already EOL, I feel like it makes more sense to just go ahead with ubuntu-latest.

crash in parsing file

generate mp4

MP4Box -crypt drm.xml output.mp4 -out output_enc.mp4
MP4Box -dash 5000 -url-template -bs-switching no -out output.mpd -rap output_enc.mp4

#parse mp4
with BufferedReader(inp) as reader:
while reader.peek(1):
box = Box.parse_stream(reader)

#crash
python3 decrypt.py -k 63cb5f7184dd4b689a5c5ff11ee6a328 -i output_enc_dashinit.mp4 -o output_enc_pydec.mp4 ─╯
Traceback (most recent call last):
File "/Users/userxxx/Downloads/dash_one/decrypt.py", line 134, in
main()
File "/Users/userxxx/Downloads/dash_one/decrypt.py", line 50, in main
decrypt(args.key, args.input, args.output)
File "/Users/userxxx/Downloads/dash_one/decrypt.py", line 85, in decrypt
box = Box.parse_stream(reader)
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 186, in parse_stream
return self._parse(stream, context, "parsing")
File "/usr/local/lib/python3.9/site-packages/pymp4/parser.py", line 46, in _parse
obj = self.subcon._parse(stream2, context, path)
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 855, in _parse
subobj = list(sc._parse(stream, context, path).items())
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 297, in _parse
return self.subcon._parse(stream, context, path)
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 1544, in _parse
obj = self.cases.get(key, self.default)._parse(stream, context, path)
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 859, in _parse
subobj = sc._parse(stream, context, path)
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 2696, in _parse
return self.subcon._parse(stream, context, path)
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 297, in _parse
return self.subcon._parse(stream, context, path)
File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 427, in _parse
return stream.read()
File "/usr/local/lib/python3.9/site-packages/construct/lib/bitstream.py", line 158, in read
raise IOError("could only read %s bytes, requested %s" % (len(data),count))
OSError: could only read 92845772 bytes, requested 4260952056

Add ESDS box to parser.

Can you add ESDS Box for audios to parser, please?
This is an example, if you need more examples or full init file tell me.

      [stsd] size=12+156
        entry-count = 1
        [enca] size=8+144
          data_reference_index = 1
          channel_count = 2
          sample_size = 16
          sample_rate = 48000
          [esds] size=12+24
            [ESDescriptor] size=2+22
              es_id = 2
              stream_priority = 0
              [DecoderConfig] size=2+17
                stream_type = 5
                object_type = 64
                up_stream = 0
                buffer_size = 16777215
                max_bitrate = 96000
                avg_bitrate = 96000
                DecoderSpecificInfo = 11 90

Codecov deactivated?

While master still has one failing test (which I have not found out why the seemingly exact same two objects are failing assertion) the Construct 2.10 upgrade is now from all the pull requests finally fully passing all CI workflow steps including tests and coverage. However, the upload the codecov is failing because you seemingly deactivated your configuration with Codecov on the Codecov dashboard?

[2023-04-17T20:27:27.901Z] ['error'] There was an error running the uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 400 - [ErrorDetail(string='This repository has been deactivated. To resume uploading to it, please activate the repository in the codecov UI: https://app.codecov.io/github/beardypig/pymp4/settings', code='invalid')]

Is it possible to re-activate it? Or will we go ahead and just remove Codecov (though I feel like it should stay).

By the way. I appreciate you giving me contributor access, but it seems like your hands are full with regard to the upkeep of this project. You don't seem to have the time or need to work on the project and I am wondering if you're willing to let someone else continue maintaining the project? I am willing to take it fully off your hands by transferring the entire project to me. I will try to maintain it as much as I can, as pymp4 is currently used by one of my main bigger projects as well as some of my others. So I have a reason or a push to ensure this project thrives. If not, that is understandable and I will still continue contributing where possible.

TENC box version 1 unsupported

I've attached a zip of an mp4 stream where it has a TENC box of version 0x01 rather than 0x00. It otherwise seems to function the same as 0x00. test.zip. I've also come across another TENC box of version 0x76 (??). Quite strange. However, this one does not seem to contain the typical information a box would have. test_0x76.zip

Large size not handled

According to the spec:

size: is an integer that specifies the number of bytes in this box, including all its fields and contained
boxes; if size is 1 then the actual size is in the field largesize; if size is 0, then this box is the last
one in the file, and its contents extend to the end of the file (normally only used for a Media Data Box)

I don't see pymp4 handles the largesize.