Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/panzi/rust-u4pak

unpack, pack, list, check and mount Unreal Engine 4 .pak archives
https://github.com/panzi/rust-u4pak

Last synced: 3 months ago
JSON representation

unpack, pack, list, check and mount Unreal Engine 4 .pak archives

Awesome Lists containing this project

README 

        
= Rust U4Pak

Mathias Panzenböck ; L. Sprengel 

v1.4.0

:icons: font

:sectnums:

:toc: left

:toclevels: 3



image:https://img.shields.io/github/actions/workflow/status/panzi/rust-u4pak/rust.yml?branch=main["Build status", link=https://github.com/panzi/rust-u4pak/actions/workflows/rust.yml]

image:https://img.shields.io/github/v/release/panzi/rust-u4pak["Release",link="https://github.com/panzi/rust-u4pak/releases"]

image:https://img.shields.io/github/license/panzi/rust-u4pak?cachebust=1["License",link="https://github.com/panzi/rust-u4pak/blob/main/LICENSE.txt"]



More or less re-implementation of https://github.com/panzi/u4pak[Python U4Pak]

for fun, ease of use (standalone binary), and speed (multi-threading).



This is a tool to pack, unpack, check, and list the contents of Unreal Engine 4

packages. Under Linux it can also be used to read-only FUSE-mount archives. Note

that only a limited number of pak versions are supported.



.Supported Versions

|====

|Pak Version |Unreal Engine Version |Reading |Writing

|1  |< 4.0 |✔️ |✔️

|2  |4.0 - 4.2 |✔️ |✔️

|3  |4.3 - 4.15 |✔️ |✔️

|4  |4.16 - 4.19 |✔️ |❌

|5  |4.20 |✔️ |❌

|6  |- |✔️ |❌

|7  |4.21 |✔️ |❌

|8  |4.22 - 4.24 |✔️ |❌

|9  |4.25 |✔️ |❌

|10 |-    |✔️ |❌

|11 |4.26 - 4.27 |✔️ |❌

|====



.Supported Compression

|====

|Name |Supported

|None |✔️

|https://zlib.net[Zlib] |✔️

|https://www.gnu.org/software/gzip/[Gzip] |❌

|https://lz4.github.io/lz4/[LZ4] |❌

|====



Instead of passing arguments you can also put the arguments in a file with the

extension .u4pak and pass the path to that instead. This is useful for Windows

users that aren't used to a terminal. You can even associate the extension with

u4pak.exe so that it will be automatically opened with it when you double click

it. File paths in a .u4pak file are relative to the directory containing the

file. The syntax of these files is not shell syntax. If you don't have any white

space, double quotes (`"`), or hash marks (`#`) in your file names you don't have

to worry about anything. `#` is used to start a comment line (only if it doesn't

touch any non-white space on it's left) and `"` is used to quote arguments

containing white space, `#`, or `"`. In order to write a `"` in a quoted argument

you simply need to double it, meaning an argument that contains nothing but a

single `"` is written as `""""`. Newlines are ignored like any other white space.

An example .u4pak file whould be:



```sh

# This is packing my project:

pack

--version=4

--mount-point=../../..



"C:\Users\Alice\My Documents\U4Project\NewArchive.pak"



":rename=/:C:\Users\Alice\My Documents\U4Project\Some Files"

":zlib,rename=/:Some Other Files"

```



If u4pak.exe is run by double clicking or by dropping a .u4pak file on it it

won't immediately close the terminal window, but will instead ask you to press

ENTER. It does this so you have a chance to read the output. Since I don't use

Windows (I cross compile on Linux and test with Wine) I could not test this

particular feature. If it doesn't work please report a bug. In order to force

the "Press ENTER to continue..." message you can pass the argument

`--pause-on-exit=always` (Windows-only).



== Usage



```

u4pak [--pause-on-exit=] [SUBCOMMAND]

```



Or:



```

u4pak "C:\Path\to\arguments.u4pak"

```



== Sub-Commands



|====

| Sub-Command |Description

| check       | Check consistency of a package

| help        | Prints general help message or the help of the given subcommand(s)

| info        | Show summarized information of a package

| list        | List content of a package

| mount       | Mount package as read-only filesystem (Linux-only)

| pack        | Create a new package

| unpack      | Unpack content of a package

|====



For help to the various sub-commands run `u4pak help SUBCOMMAND`.



== File Format



Byte order is little endian and the character encoding of file names seems to be

ASCII (or ISO-8859-1/UTF-8 that coincidentally only uses ASCII compatiple

characters).



Offsets and sizes seem to be 64bit or at least unsigned 32bit integers. If

interpreted as 32bit integers all sizes (except the size of file names) and offsets

are followed by another 32bit integer of the value 0, which makes me guess these

are 64bit values. Also some values exceed the range of signed 32bit integers, so

they have to be at least unsigned 32bit integers.



This information was reverse engineered from the Elemental

https://wiki.unrealengine.com/Linux_Demos[Demo] for Linux (which contains a 2.5

GB .pak file), the https://www.techpowerup.com/download/unreal-engine-4-five-tech-demos/[Unreal Engine 4 - Five Tech Demos],

version 7 was reverse engineered from the https://store.steampowered.com/app/813630/Supraland/[Supraland Demo],

and the Conan Exiles variant was reverse egineered from the

https://steamcommunity.com/sharedfiles/filedetails/?id=1765361591[SandstormFix_EXP workshop item].

Reverse engineering was done by poking around the pak files with a hex editor,

no kind of decompilation was done by me.



Versions >= 8 where reversed from an empty unreal engine project.



.Basic layout

* Data Records

* Index

** Index Header

** Index Records

* Footer



In order to parse a file you need to read the footer first. The footer contains

an offset pointer to the start of the index records. The index records then

contain offset pointers to the data records.



[[encoded-record]]

=== Encoded Record



[subs="quotes"]

----

*Offset  Size  Type         Description*

     0     4  uint32_t     bitfield containing record information

                              0-5  : Compression block size

                              6-21 : Compression blocks count

                              22   : Encrypted

                              23-28: Compression method

                              29   : Size 32-bit safe?

                              30   : Uncompressed size 32-bit safe?

                              31   : Offset 32-bit safe?

_if offset 32 bit_

     4     4  uint32_t     offset

_else_

     4     8  uint64_t     offset

_end_

_if uncompressed size 32 bit_

     ?     4  uint32_t     uncompressed size

_else_

     ?     8  uint64_t     uncompressed size

_end_

_if compression method != 0x00_

  _if size 32 bit_

     ?     4  uint32_t     size

  _else_

     ?     4  uint32_t     size

  _end_

_end_

_if compression block count > 0 && (encrypted || compression block count != 1)_

  _for _ in 0..compression block count_

     ?     4  uint32_t     block size

  _end_

_end_

----



=== Record

NOTE: This structure, while still present in version >= 10 is not used anymore by default. See xref:encoded-record[Encoded Record] for the current record information.



[subs="quotes"]

----

*Offset  Size  Type         Description*

     0     8  uint64_t     offset

     8     8  uint64_t     size (N)

    16     8  uint64_t     uncompressed size

    24     4  uint32_t     compression method:

                              0x00 ... none

                              0x01 ... zlib

                              0x10 ... bias memory

                              0x20 ... bias speed

_if version <= 1_

    28     8  uint64_t     timestamp

_end_

     ?    20  uint8_t[20]  data sha1 hash

_if version >= 3_

 _if compression_method != 0x00_

  ?+20     4  uint32_t     block count (M)

  ?+24  M*16  CB[M]        compression blocks

 _end_

     ?     1  uint8_t      is encrypted

   ?+1     4  uint32_t     The uncompressed size of each compression block.

_end_                        The last block can be smaller, of course.

_if variant == "Conan Exiles"_

     ?     4  uint32_t     Unknown field. For Conan Exiles index record only

                           seen it to have the value 0.

_end_

----



=== Compression Block (CB)

[horizontal]

Size:: 16 bytes



[subs="quotes"]

----

*Offset  Size  Type         Description*

     0     8  uint64_t     compressed data block start offset.

                           version <= 4: offset is absolute to the file

                           version 7: offset is relative to the offset

                                      field in the corresponding Record

     8     8  uint64_t     compressed data block end offset.

                           There may or may not be a gap between blocks.

                           version <= 4: offset is absolute to the file

                           version 7: offset is relative to the offset

                                      field in the corresponding Record

----



=== Data Record

[subs="quotes"]

----

*Offset  Size  Type            Description*

     0     ?  Record          file metadata (offset field is 0, N = compressed_size)

_if variant == "Conan Exiles"_

     ?    20  ?               Unknown. Maybe another SHA-1 sum of something?

                              The first 4 bytes have values other than the extra

                              4 bytes in the index record, which is why I didn't

                              put those into the general record structure.

_else if version >= 4 and compression_method != 0x00_

     ?     4  uint32_t        Unknown.

_end_

     ?     N  uint8_t[N]      file data

----



[NOTE]

====

Starting with version 4 there is an additional 4 bytes in the repeated

*data* record copy (the record that precedes the actual file date, not the

record in the index). I don't know what that is. It is not always the same value.

E.g. it is the same for some files, but different for others. The first 2 bytes

seem to be always the same (`0x78 0x9c` in a v4 and a v7 pak that I saw), so maybe

its 2 16 bit fields?



This is why I've deactivated packing for versions > 3.

====



[[fdi]]

=== Full Directory Index

The data structure is a map>.



[subs="normal"]

----

*Offset  Size  Type            Description*

     0     4  uint32_t        directory count (D)

_for i in 0..D_

     ?     4  int32_t         directory name size (DS)

                              For some games a negative value means it's a UTF-16

                              string in 2 * -S bytes.

   ?+4    DS  char[DS]        directory name (includes terminating null byte)

?+4+DS     4  uint32_t        file count (F)

  _for j in 0..F_

     ?     4  int32_t         file name size (FS)

                              For some games a negative value means it's a UTF-16

                              string in 2 * -S bytes.

   ?+4    FS  char[FS]        file name (includes terminating null byte)

?+4+FS     4  uint32_t        offset in xref:directory-info[encoded entry info]

  _end_

_end_

----



=== Index Record

NOTE: This structure, while still present in version >= 10 is not used anymore by default. See xref:fdi[Full Directory Index] for record information.



[subs="quotes"]

----

*Offset  Size  Type            Description*

     0     4  int32_t         file name size (S)

                              For some games a negative value means it's a UTF-16

                              string in 2 * -S bytes.

     4     S  char[S]         file name (includes terminating null byte)

   4+S     ?  Record          file metadata

_if variant == "Conan Exiles"_

     ?     4  ?               Unknown. Only saw all 0 so far.

_end_

----



=== Index

[subs="normal,callouts"]

.Version >= 10

----

*Offset  Size  Type            Description*

     0     4  int32_t         mount point size (S)

                              For some games a negative value means it's a UTF-16

                              string in 2 * -S bytes.

     4     S  char[S]         mount point (includes terminating null byte)

   S+4     4  int32_t         entry count

   S+8     8  uint64_t        path hash seed <1>

  S+16     4  uint32_t        has path hash index

_if has path index != 0_

  S+20     8  int64_t         path hash index offset

  S+28     8  int64_t         path hash index size

  S+36    20  uint8_t[20]     path hash index hash

_end_

     ?     4  uint32_t        has full directory index

_if has full directory index != 0_

   ?+4     8  int64_t         full directory index offset

  ?+12     8                  full directory index size

  ?+20    20  uint8_t[20]     full directory index hash

_end_

     ?     4  int32_t         encoded entry info size (P)

   ?+4     P  uint8_t[P]      encoded entry info [[directory-info]]

   ?+P     4  uint32_t        file count, probably unused / 0 (N)

 ?+P+4     ?  IndexRecord[N]  records

----

<1> Needs clarification



[subs="quotes"]

.Legacy (Version < 10)

----

*Offset  Size  Type            Description*

     0     4  int32_t         mount point size (S)

                              For some games a negative value means it's a UTF-16

                              string in 2 * -S bytes.

     4     S  char[S]         mount point (includes terminating null byte)

   S+4     4  uint32_t        record count (N)

   S+8     ?  IndexRecord[N]  records

----



=== Footer

[horizontal]

.Size and index features

|====

|Versions |Size |Index Encryption |Encryption Key Guid |Compression method name |Frozen Index

|v1 - v3 |44 bytes |❌ |❌ |❌ |❌

|v4 - v6 | 45 bytes |✔️ |❌ |❌ |❌

|v7 | 65 bytes |✔️ |✔️ |❌ |❌

|v8 | 193 bytes |✔️ |✔️ |✔️(Max. 4) |❌

|v9 | 226 bytes |✔️ |✔️ |✔️(Max. 5) |✔️

|v10 - v11 | 225 bytes |✔️ |✔️ |✔️(Max. 5) |❌

|====



[subs="quotes"]

----

*Offset  Size  Type         Description*

_if version >= 7_

     0    20  uint8_t[20]  encryption key Guid

_end_

_if version >= 4_

     ?     1  uint8_t      encrypted index

_end_

     ?     4  uint32_t     magic: 0x5A6F12E1

   ?+4     4  uint32_t     version: 1 - 11

   ?+8     8  uint64_t     index offset

  ?+16     8  uint64_t     index size

  ?+24    20  uint8_t[20]  index sha1 hash

_if version == 9_

  ?+44     1  uint8_t      frozen index

_end_

_if version == 8_

  ?+44   128  uint8_t[128] compression methods (4 * 32 chars)

_else if version > 8_

     ?   160  uint8_t[160] compression methods (5 * 32 chars)

_end_

----



== Related Projects



* https://github.com/panzi/fezpak[fezpak]: pack, unpack, list and mount FEZ .pak archives

* https://github.com/panzi/psypkg[psypkg]: pack, unpack, list and mount Psychonauts .pkg archives

* https://github.com/panzi/bgebf[bgebf]: unpack, list and mount Beyond Good and Evil .bf archives

* https://github.com/panzi/unvpk[unvpk]: extract, list, check and mount Valve .vpk archives (C++)

* https://github.com/panzi/rust-vpk[rust-vpk]: Rust rewrite of the above (Rust)

* https://github.com/panzi/t2fbq[t2fbq]: unpack, list and mount Trine 2 .fbq archives

* https://github.com/panzi/u4pak[u4pak]: old Python version of this program



== MPLv2 License



Rust U4Pak - pack, unpack, check, list and mount Unreal Engine 4 packages



Copyright (C) {localyear} Mathias Panzenböck, L. Sprengel



This Source Code Form is subject to the terms of the Mozilla Public

License, v. 2.0. If a copy of the MPL was not distributed with this

file, You can obtain one at https://mozilla.org/MPL/2.0/.