maxton/liborbispkg

Library, GUI, CLI for creating, inspecting, and modifying PS4 PKG, SFO, PFS, and related filetypes

LibOrbisPkg

I am developing an open source library for reading and writing PS4 PKG files. This project's goal is to eliminate the need to use proprietary SDK tools. Without a proper open PKG tool, the PS4 homebrew scene cannot flourish.

All code in this repository is licensed under the GNU LGPL version 3, which can be found in LICENSE.txt.

Download

The latest builds are available to download at AppVeyor.

Usage

PkgEditor

PkgEditor is a GUI tool with which you can edit GP4 projects, create and edit SFO files, and build PKG and PFS archives. The tool also supports opening PKGs directly. You can see the header, entries, and if the package is a fake PKG or you enter a passcode, you can browse files as well.

Screenshots

PkgTool

PkgTool is a command line tool for common PKG/PFS/SFO tasks. Integrate it into your build scripts!

Usage: PkgTool.exe <verb> [options ...]

Verbs:
  pfs_buildinner <input_project.gp4> <output_pfs.dat>
    Builds an inner PFS image from the given GP4 project.

  pfs_buildouter [--encrypt] <input_project.gp4> <output_pfs.dat>
    Builds an outer PFS image, optionally encrypted, from the given GP4 project.

  pfs_extract [--verbose] <input.dat> <output_directory>
    Extracts all the files from a PFS image to the given output directory. Use the verbose flag to print filenames as they are extracted.

  pkg_build <input_project.gp4> <output_directory>
    Builds a fake PKG from the given GP4 project in the given output directory.

  pkg_extract [--verbose] [--passcode <...>] <input.pkg> <output_directory>
    Extracts all the files from a PKG to the given output directory. Use the verbose flag to print filenames as they are extracted.

  pkg_extractentry [--passcode <...>] <input.pkg> <entry_id> <output.bin>
    Extracts the selected entry from the given PKG file.

  pkg_extractinnerpfs [--passcode <...>] <input.pkg> <output_pfs.dat>
    Extracts the inner PFS image from a PKG file.

  pkg_extractouterpfs [--encrypted] [--passcode <...>] <input.pkg> <pfs_image.dat>
    Extracts and decrypts the outer PFS image from a PKG file. Use the --encrypted flag to leave the image encrypted.

  pkg_listentries <input.pkg>
    Lists the entries in a PKG file.

  pkg_makegp4 [--passcode <...>] <input.pkg> <output_dir>
    Extracts all content from the PKG and creates a GP4 project in the output directory

  pkg_validate [--verbose] <input.pkg>
    Checks the hashes and signatures of a PKG.

  sfo_deleteentry <param.sfo> <entry_name>
    Deletes the named entry from the SFO file.

  sfo_listentries <param.sfo>
    Lists the entries in an SFO file.

  sfo_new <param.sfo>
    Creates a new empty SFO file at the given path.

  sfo_setentry [--value <...>] [--type <...>] [--maxsize <...>] [--name <...>] <param.sfo> <entry_name>
    Creates or modifies the named entry in the given SFO file.

Thanks

Everyone who helped, either directly or indirectly, but especially the following:

• flatz
• idc

liborbispkg's Issues

Create GUI

GUI needs to have the ability to create and edit GP4 projects, and then build images and pkgs

Research PKG/PFS signatures

PS4 probably won't accept them unless there are signatures. Could also look into modifying fakepkg kernel patches to not need sigs?

Write/generate PFS image seed at 0x370 and unknown digest at 0x380

Some sort of index, then the PFS seed, then a hash, maybe?

• seed
• digest

Better SFO editing

The SFO editor works, but you have to know all the magic numbers to use for things like "ATTRIBUTE". It would be nice to have some better UI for that.

Remove GameArchives dependency

Since LibOrbisPkg can now read and write PKG files on its own, there is no reason to drag along GameArchives. Unfortunately, GameArchives is still a dependency because LibArchiveExplorer is used for the file view GUI in PkgEditor. All that's needed to remove this dependency is a new file view UI that uses the memory-mapped file classes.

This will have the added benefit of also adding parallel extraction to the PkgEditor GUI.

Create command line tool

Should at the very least be able to instruct the tool to build a PFS or PKG given a GP4.

More features TBD

Implement entry encryption

Some entries (image key, license.dat, license.info) are encrypted in normal PKGs. I'm not sure if the PS4 requires them to be encrypted, but it's probably a good idea to implement this anyway.

Main parts to this:

• Generate keys in the ENTRY_KEYS entry
• Determine encryption mode (probably AES-CBC-256?)
• Encrypt the entries that require it

Add other sce_sys files to the PKG entries, and filter from the PFS

There are more files than just param.sfo that should be saved only in the PKG's body and not in the PFS image.

Can I use this in my GM:S -> ps4 tool?

I'd like to use the library :) maybe modify it a bit to work for my needs .

Build PKG without encryption/signatures

We can calculate the signatures and replace/resign/encrypt afterward. Getting PKGs that are the right size and layout is a top priority.

Validate and Re-fake-sign PKGs

There should be a way to validate a fPKG's digests/signatures, and to re-sign in case the PKG was modified. These two systems should work on the same data structure, something like what the PFS builder is using to do inode data block digests

Implement PFSC compression

Even if you don't choose to compress any files, by default the PFS image is stored in a PFSC container where the first few blocks are compressed with zlib since the filesystem metadata is pretty highly compressible. I don't think this is necessary for compatibility but it would allow us to make bit-for-bit identical PKGs to the "official" tools as long as the compression level is matched.

Improve initial PKG view

The first thing you see when opening a PKG in PkgEditor is not much better than what you'd see if you opened it in a hex editor. Let's scrap that and make a better initial tab.

Sort the dirents but not the file data blocks by name

Unknown SFO type: 0000

PkgTool.exe sfo_listentries param.sfo

Unhandled Exception: System.Exception: Unknown SFO type: 0000
at LibOrbisPkg.SFO.ParamSfo.FromStream(Stream s)
at PkgTool.Program.<>c.<.cctor>b__6_11(String[] args)
at PkgTool.Verb.Run(Verb[] verbs, String[] args, String name)
at PkgTool.Program.Main(String[] args)

LibOrbisPkg/PkgTool fails to extract inner PSF/file from fake PKG

Build from git as of 03.01.2019 and testing on FF7 Re Demo pkg.

Here is validate and listentries output:
https://pastebin.com/raw/MWWMeTvP
https://pastebin.com/raw/X5Xm4a7S

An attempt to extract fails:

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'count')
   at System.Linq.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument)
   at System.Linq.Enumerable.Range(Int32 start, Int32 count)
   at LibOrbisPkg.PFS.PfsReader.LoadDir(UInt32 dinode, Dir parent, String name)
   at LibOrbisPkg.PFS.PfsReader..ctor(IMemoryReader r, UInt64 pfs_flags, Byte[] ekpfs)
   at LibOrbisPkg.PFS.PfsReader..ctor(MemoryMappedViewAccessor r, UInt64 pfs_flags, Byte[] ekpfs)
   at PkgTool.Program.<>c.<.cctor>b__6_4(Dictionary`2 flags, Dictionary`2 optionals, String[] args)
   at PkgTool.Verb.Run(Verb[] verbs, String[] args, String name)
   at PkgTool.Program.Main(String[] args)

Let me know if any debug data is needed.

Support multi-core process?

Sony orbis-pub is very faster than LibOrbisPkg, I know it's still work-in-progress project,
But if it's have that it will be a great.
Thanks.

SFO Editor

Not everyone is fortunate enough to have param.sfo files lying around, and they're required to build a PKG. I already wrote a PARAM.SFO viewer, it shouldn't take much to turn it into an editor.

Handle missing param.sfo better

PkgEditor throws an arcane, uncaught exception if you didn't put a param.sfo in sce_sys. Improve this UX.

Don't lock the UI thread while building a PKG

It makes the program seem to hang and is bad UX. The log should update as the PKG is being built.

PFS: Duplicate file

Something funky's going on; there's two copies of keystone in this PKG

Implement Outer PFS Signatures

The outer PFS image uses signed inodes, which means each data block pointer has a 32-byte hash bundled up with it to validate the data in each block. The hash is computed with HMAC-SHA256 using the PFS Sign Key, which is derived from the EKPFS and the PFS Seed.

It's important that we respect the hierarchical structure of these signatures that arises when indirect block pointers are used, as shown in the diagram below.

In this example, we need to hash data blocks 17 and 18 before we hash block 2; likewise we have to hash blocks 1201 and 1202 before we hash block 1001, and block 1001 before block 3.

Support fixed timestamps

Right now we are using the current time as a timestamp for the PKG and for the file times. If support is added for a fixed timestamp specified in the GP4, then we can test with reproducible PKG builds.

Better API for PKG build properties

Right now the only way to build a PKG is using a GP4 project which works but is inflexible. What if we want to use files that aren't on the hard drive but in memory or in another container/package? There should be a new PKG properties class which GP4 is converted to before building and which is accessible from the public API. It will use a list of GameArchives.IFiles instead of a list of paths.

2 errors regarding open package file.

When I tried to open some files using PkgEditor, I got these error.

After some digging, I found what causes it:

First error shows "Access to the path "....." is denied."
Caused when open a ReadOnly pkg file.

Second error shows "Error loading outer PFS: Access to the path is denied"
Caused when the fake pkg file size is missing some bytes at the end of the pkg.
also this one will not list Files, instead it will show "The package passcod is required...."

Pkgtool.exe Unhandled Exception: System.ArgumentException

when using pkgtool.exe extractpkg <input.pkg> "passcode" <output_directory> I get error message:
Unhandled Exception: System.ArgumentException: The path is not of a legal form.
at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
at System.IO.Path.GetFullPathInternal(String path)
at System.IO.DirectoryInfo.Init(String path, Boolean checkHost)
at GameArchives.Local.LocalDirectory..ctor(String path) in C:.......\LibOrbisPkg-master\GameArchives Library\Local\LocalDirectory.cs:line 44
Do you have any idea?

Add other package types

Right now all packages are assumed to be AddCont pkgs. Add a dropdown or toggle or something to allow changing the project type to GP. DP support may have to wait.

Improve build speed with memory mapped file IO and threads

I think it is possible to get a huge increase in build speed by speeding up PFS encryption and signing with memory mapped files. Using memory mapped files would allow the encryption to take place in parallel without worrying about stream concurrency. Currently encryption is the slowest link in the chain.

• Encryption
• Signing

Build PFS from GP4

Right now the PFS code is copied from MakePFS. Need to modify it to load files as specified in a GP4 project.

Error

I Get this error when opening a PKG built by command line, it also error when installing on PS4

Generate image key entry

This one must be important, since it's encrypted by default.

PKG build error: "Length of the data to encrypt is invalid"

Look into this
Converting "Yellow Ledbetter - Pearl Jam" CON in LibForge

Unhandled Exception: System.Security.Cryptography.CryptographicException: Length of the data to encrypt is invalid.
   at System.Security.Cryptography.RijndaelManagedTransform.EncryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast)
   at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
   at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at LibOrbisPkg.Util.Crypto.AesCbcCfb128Encrypt(Byte[] out, Byte[] in, Int32 size, Byte[] key, Byte[] iv)
   at LibOrbisPkg.PKG.PkgWriter.WriteBody(Pkg pkg, String contentId, String passcode)
   at LibOrbisPkg.PKG.PkgBuilder.Write(Stream s, Action`1 Logger)
   at LibForge.Util.PkgCreator.BuildPkg(String proj, String outPath)
   at ForgeTool.Program.Main(String[] args)

Add support to create Additional Content without data (AL)

Mono compatibility

This lib is mono compatible ?

Create discmap.plt

This might be necessary for app pkgs.

List PKG/PFS filesystem contents in PkgTool

Wanting similar functionality like that in PkgEditor when opening a pkg file and viewing the Files tab. Could output similar to GNU Find.

question: passcode from package

i noticed in Keys.cs: // Used to encrypt Passcode in ENTRY_KEYS

is possible to reverse process, to get passcode from package retail/debug?

regards

AES-XTS Encryption

Find a good way to do this in .NET. Might have to implement XTS but AES is already available in System.Security.Cryptography

package supported types

small question, because there is no any info. what types of packages are supported for makepkg, i'm asking because i tried to make theme package, but tool build package as DLC, not theme (<volume_type>pkg_ps4_theme</volume_type>). regards

Out of bounds error in KeysEntry.Write

Found from LibForge, check "Bush - Glycerine"

Unhandled Exception: System.ArgumentException: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.
   at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
   at LibOrbisPkg.PKG.KeysEntry.Write(Stream s)
   at LibOrbisPkg.PKG.PkgWriter.WriteBody(Pkg pkg, String contentId, String passcode)
   at LibOrbisPkg.PKG.PkgBuilder.Write(Stream s, Action`1 Logger)
   at LibForge.Util.PkgCreator.BuildPkg(String proj, String outPath)
   at ForgeTool.Program.Main(String[] args)

Question: extract contents from a debug package

Hi @maxton , is the tool able to extract the contents of debug packages or is a passcode always required for debug and retail packages?

Add extra data to param.sfo

The publishing tools augment the param.sfo with additional information, like the version of publishing tools, etc. We are faking this right now by adding length to the files. Should implement an SFO reader and writer, create SFO data structures, and add missing info to param.sfo when building a PKG.

Implement new EKPFS generation

Newer PKGs appear to use a different method to generate an EKPFS from a passcode. Because of this, we can only decrypt newer PFS images with an EKPFS or XTS key-pair.

Cannot extract large files in the latest commit.

"Not enough memory resources are available to process this command."
When trying to extract large file, tried on CUSA-06426 update pkg.
pkg-size is 1.63GB, after extract it became 3.2GB, contain one large file with size of 3.2GB.

Implement Outer PFS Signing

The Outer PFS image uses signed inodes for data integrity. Signed inodes contain a hash of the data block with each direct/indirect block pointer, using HMAC-SHA256 with the PFS Signing key, which is derived from the EKPFS and the PFS Seed.

We will have to generate these hashes in the proper order so that e.g. blocks of indirect block pointers have all their hashes filled in before they are themselves hashed.

In this example we need to calculate the hashes for data blocks 1201, 1202, etc. before calculating the hash for block 1001, which also must be done before block 3; likewise, the hashes for blocks 17, 18, etc. must be calculated before the hash for block 2.

Generate and encrypt RIF secret, and sign license.dat

The encrypted secret is the part of the license that the PS4 decrypts inside SAMU to get EKPFS etc without needing a passcode. We need to generate these secret keys, then encrypt them and sign the license.dat that contains them.

TODO:

• Generate secret
• Encrypt secret
• Sign license.dat

Generate General Digests entry

It looks like system_digest is left empty on debug/fake pkgs? Either way, we should still do these:

• content digest - relies on calculating the major param digest
• game digest - this is just the same as pfs_image_digest
• header digest - the sha256 hash of the first 64 bytes of the pkg and the 128 bytes at 0x400
• major param digest
• param digest (we're already calculating this; it's just the param.sfo file digest)

The content digest is the sha256 hash of the concatenation of

• The Content ID (48 bytes null-padded)
• The DRM type (4 bytes Big Endian)
• The Content Type (4 bytes Big Endian)
• The PFS Image Digest (32 bytes)
• The major param digest (32 bytes)

extractentry - decrypt encrypted entries

if possible, decrypt encrypted entries like: license.dat, license.info

regards

Write PFSC image

Right now we're writing the uncompressed unsigned unencrypted pfs_image.dat directly into the PKG which is wrong. Really, the outer PFS should be signed and encrypted; the inner pfs_image.dat should be written as a file in that outer PFS in the PFSC format

Add "New GP4" option

Right now we can only edit existing project files, that's not very user friendly.