APFS file system exploration
2023-08-14 ｜ Research ｜ #Words: 2112
This article aims to provide a low-level explanation of APFS, the file system used by Apple devices. Because I noticed that there was a lack of documentation on how APFS works in details, so I do some research and share my findings.
Tip
Don’t confuse partitions in APFS with the conception of partitions in Windows. In APFS, a Volume is shown in the Finder, but it is not a direct partitions of the hard disk. There is an intermediate layer called container, which can contain multiple volumes. This is different from the Windows partition concept, each partition is a separate disk space directly in Windows.
APFS structure
The internal structure of APFS is as follows:
The diagram above shows only one volume, and then describes each part of volume. The “objects” in “Storage for objects and file data” includes the storage disk what we see in Finder, which is also called as volume.
The abbreviation of “GUID partition table” is GPT, which is well-known to many people on Windows (not the chatbot), and GPT partition is part of the EFI technology (EFI aka UEFI, invented by Intel).
Each container of APFS is arranged in order on the hard disk, without any gaps, which is different from Windows’ GPT partition.
This is also why on Windows you just can extend or adjust the size of the last partition, and not the previous (APFS’s free space is shared among the various volumes, so it’s easy to expand, as below). And why it takes a long time to create a new APFS partition on macOS (because it have to move the files from other partitions to the right place).
How does the system know that this container exists on the hard disk? Sequential reading can know the first container, but what about the second and others?
First of all, the system does not find the containers by sequential reading. This is the job of the GUID partition table at the beginning of the hard disk, which contains the start and end addresses of each partition. When needed, it just jumps to them, using a directory-like structure. As shown in the digram below (from the official standard 《GUID Partition Table (GPT) Disk Layout》):
You can see how partitions are divided and jump through the main partition table or the backup partition table.
If you look at the hexadecimal of an APFS formatted hard drive, you can see the following situation (note that the “GPT Header” part starts from line 200):
You can see the beginning of structure looks like other UEFI format device.
Protective MBR
One difference between APFS and some other file systems using GPT partitions is that APFS has a Protective MBR section, regardless of whether they are boot disks or not.
The Protective MBR must be the first logical block on a hard disk that uses GPT partitions. This partition makes the computer think that the hard disk is usable and has been used, so it won’t pop up a window to ask you whether to partition, format, etc. The EFI itself does not use this block.
Protective MBR looks like:
The meaning of each part as follows (the offset is from the beginning of this part and is also in hexadecimal):
| Part | Offset (Byte) | Length (Byte) | content |
|---|---|---|---|
| BootCode | 0 | 440 | EFI will not use this part. The hexadecimal of 440 is 1b8, so you can see that 1b8 and the previous part in the above picture are all 00 |
| UniqueMB RDiskSignature | 440 | 4 | Not used, set to 0 |
| Unknown | 444 | 2 | Not used, set to 0 |
| PartitionRecord | 446 | 16*4 | An array of four MBR partition records. MBR can have up to 4 partitions. |
| Signature | 510 | 2 | is set to 0xAA55, which is the 55 aa at the end of the 1f0 line in the picture |
| Reserved | 512 | Logical block size - 512 | Remaining reserved portion of the logical block, set to 0 |
If you have questions about the left index number, remember it is in hexadecimal. For example, the hexadecimal notation of the last 512 is 200.
GPT Header
GPT Header is the core part of UEFI technology and GPT partition, which stores a lot of information. The various parts of GPT Header are as follows:
The meaning of each part is as follows (the offset is from the beginning of this part and is also in hexadecimal):
| Part | Offset (Byte) | Length (Byte) | content |
|---|---|---|---|
| Signature | 0 | 8 | This part is the string “EFI PART” in ASCII format, encoded with 64 bits, which is 45 46 49 20 50 41 52 54 in the picture above |
| Revision | 8 | 4 | The number of revisions of GPT Header. This version is not equal to the UEFI specified version. 00 00 01 00 above means version 1.0 |
| Header size | 12 | 4 | The size of the GPT Header, in bytes. This part must be greater than or equal to 92 and less than or equal to the logical block size. The above 5C 00 00 00 means 92 bytes, which is the minimum size. The above picture only has 60 bytes because the last few blank lines are omitted |
| Header’s CRC32 | 16 | 4 | CRC32 checksum of the GPT Header structure. First set this value to 0, and then calculate the 32-bit CRC32 checksum of the GPT Header structure. The checksum here is 15 51 0f ff |
| Reserved | 20 | 4 | This part must be all 0 |
| MyLBA | 24 | 8 | This LBA (logical block address) contains the data structure. When verifying the GPT, the LBA in the GUID Partition Table pointed to by the MyLBA entry will be checked. In the above picture, it is 01 00 00 00 00 00 00 00 |
| AlternateLBA | 32 | 8 | The LBA address of the alternate GPT Header, in the above picture it is ff ff bf 46 07 00 00 00 |
| The first usable LBA | 40 | 8 | The first usable logical block of a partition described by GUID Partition Entry, in the above figure is 22 00 00 00 00 00 00 00 |
| The last usable LBA | 48 | 8 | The last used logical block of a partition described by GUID Partition Entry, in the above picture is de ff bf 46 07 00 00 00 |
| Hard disk GUID | 56 | 16 | The GUID that identifies the hard disk, in the above picture is 84 ec 00 4a 8e 8c fd 47 8a 78 25 48 bf a4 90 99 |
| Partition entry LBA | 72 | 8 | Start LBA of GUID partition entry array, in the above picture it is 02 00 00 00 00 00 00 00 |
| Number of partition entries | 80 | 4 | The number of partition entries in the GUID partition entry array, in the above picture is 80 00 00 00, which is 8 |
| The size of of partition entry | 84 | 4 | The size of the GUID partition entry structure in the GUID partition entry array. This part should be set to the size of 128 x 2n. n is an integer greater than or equal to 0. This part Typically 128, 256, etc., but earlier versions allowed any multiple of 8. 80 00 00 00 in the picture above is actually 80 in hexadecimal, which is 128 bytes |
| CRC32 of partition partition entry | 88 | 4 | CRC32 of GUID partition entry array |
| Reserved | 92 | All remaining | Reserved blocks, all 0 |
GPT Partition Entry Array
The GPT Partition Entry Array contains an array that stores GPT partition entries. I still explaining what each part is, here is the first entry in the GPT partition entry array:
| Part | Offset (Byte) | Length (Byte) | content |
|---|---|---|---|
| Partition Type GUID | 0 | 16 | A unique ID that defines the partition target and type. If the partition is not in use, this field is 0. In the picture above: 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b. The meaning behind each bit in this part will be explained in detail later |
| Unique Partition GUID | 16 | 16 | A unique GUID for each partition entry. Each partition is created with such a unique GUID, and this GUID must also be assigned a value when the GPT partition entry is created. The allocation in the above picture is f8 56 1c 99 8d 6f 9e 4f bb fa 01 ca 52 57 cc 05 |
| Starting LBA | 32 | 8 | The starting LBA of the partition defined in this entry |
| Endpoint LBA | 40 | 8 | Partition endpoint LBA defined in this entry |
| Attributes | 48 | 8 | These attribute bits are all reserved by UEFI, so here they are all 00 00 00 00 00 00 00 00 |
| Partition Name | 56 | 72 | An unterminated character containing a human-readable name. Since this is an EFI partition, what is shown is EFI System Partition |
| Reserved | 128 | Previously set size ~ 128 | This part must be 0 |
Depending on operating system and partition, the value of the partition type GUID is also different. For example, the GUID value of the EFI system partition above is 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b. Sometimes you will see C12A7328-F81F-11D2-BA4B-00A0C93EC93B in other documents, where each - separates each part. The number above is little endian, you can calculate them correspondingly, they are same.
The partition type GUID of the APFS partition is ef 57 34 7c 00 00 aa 11 aa 11 00 30 65 43 ec ac, or 7C3457EF-0000-11AA-AA11-00306543ECAC
The meaning of each part in the above “Attributes” is as follows:
| Bits | Name | Meaning |
|---|---|---|
| 0 | Request Partition | If this bit is set, this partition must exist for the platform to run. Deleting or modifying the contents of this partition may result in the loss of platform functionality and the inability to start or run. Therefore, this partition should not be deleted or modified unless the operating system, software, or firmware recognizes the partition. |
| 1 | No block IO transfer protocol | If this bit is set, the firmware cannot generate the EFI_BLOCK_IO_PROTOCOL part for this partition. If this part is not set, no file system mapping will be created for this partition in UEFI |
| 2 | Legacy BIOS Bootable | This is left to systems with legacy PC-AT BIOS firmware to implement notifications of certain restricted, special-purpose software running on a system bootable from a GPT partition |
| 3~47 | Undefined, must be 0, reserved for future extensions to the UEFI specification | |
| 48~63 | Reserved for GUID specific purposes. The usage of these bits is set based on the partition type GUID. If any one of bits 0 to 47 is modified, these bits must be retained |
You can see that in each entry in the GPT partition entry array, the starting and ending points are marked, so jump and obtain through these parts.
References and further reading
If you want to try to view the above content, you can follow this blog of mine: “How to check disk in hexadecimal by bytes in terminal on Mac or Linux”.
For implementation details of the APFS format, please refer to Apple’s document: “Apple File System Reference”(“Apple File System” is the full name of APFS).
The official UEFI document is: “GUID Partition Table (GPT) Disk Layout”.
“data recovery technology depth Secret, VisualFoxPro (Second Edition)” by Liu Wei.
I hope these will help someone in need~