Difference between revisions of "DBPF"

From SC4D Encyclopaedia
Jump to navigation Jump to search
sc4e>Warrior
m
m (34 revisions imported)
 
(26 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{transcluded|1=[http://www.sims2wiki.info/wiki.php?title=DatabasePackedFile Sims2Wiki page]}}
+
'''Database Packed File (DBPF)''' is a proprietary archive format used in many Maxis developed games, including The Sims 2, [[SimCity 4]] and Spore. DBPF files are similar to Zip files, as they hold other files inside them which are usually compressed. All plugins for SimCity 4 (excluding the [[Plugin:SC4 Extra Cheats.dll|SC4 Extra Cheats.dll]]) are DBPF files, the file name's extension is irrelevant. Note that [[savegame]] (.sc4) files are also DBPF files.  DBPF files are sometimes referred to as ''packages'' following the convention used in The Sims 2, where the files sometimes have the .package file extension.
  
Archive format used many Maxis developed games, including The Sims 2, [[SimCity 4]] and Spore.
+
DBPF files can be identified by opening the file in notepad, or preferably a hex editor, as the first 4 characters/bytes will be the letters DBPF. If a file in the plugins folder does not start with these 4 letters the game will not load them, however the [[Reader]] will. This means that if you want to disable a plugin you can open the DBPF file in a hex editor and change the first byte to something other than 44 (in Hexadecimal) (D) and the game will not load the plugin, but you will still be able to edit it in the reader (saving the disabled plugin with the reader will not re enable it, to re enable you must change the first byte back to 44 (hexadecimal)).
  
=Format=
+
Entries may be compressed with RefPack/[[QFS compression]] depending on their entry data.
Most Sims 2 Data is as you probably know stored in various types of archive files that all have the same internal format such as package, cache, etc. This format is proprietary to Maxis and has been used in the [[SimCity 4]], and The Sims Online as well as this one.
 
  
Entries may be compressed with [[RefPack]] compression depending on their entry data.
+
For an examination of the Spore DBPF format, see [[Spore DBPF]].
  
 +
==Format==
  
==Overview==
+
===Overview===
This is the general structure of DatabasePackedFile packages:
+
This is the general structure of Database Packed File packages:
  
Header (96 bytes)
+
Header (96 bytes)
 
  File 1
 
  File 1
 
  File 2
 
  File 2
 
  ...
 
  ...
 
  File n
 
  File n
  Index Entry 1 (20 bytes)
+
  Index Entry 1
 
  Index Entry 2
 
  Index Entry 2
 
  ...
 
  ...
 
  Index Entry n  
 
  Index Entry n  
  
Header first, individual files following with no filenames and a small file header area at the beginning of each followed by an index of all the files in the archive. Note that files may appear after the index table in files that are updated.
+
Header first, individual files following with no file names and a small file header area at the beginning of each, followed by an index of all the files in the archive. Note that files may appear after the index table in files that are updated.
  
==Header==
+
===Header===
 
{| width="100%" border="1"
 
{| width="100%" border="1"
 +
! width="5%" |Offset
 
! width="20%" | Name
 
! width="20%" | Name
 
! width="5%" | Version
 
! width="5%" | Version
 
! width="10%" | Type/Size
 
! width="10%" | Type/Size
! width="65%" | Info
+
! width="60%" | Info
  
 
|-
 
|-
| Magic
+
| 00
 +
| Identifier
 
|  
 
|  
 
| char[4]
 
| char[4]
Line 38: Line 40:
  
 
|-
 
|-
 +
| 04
 
| Major Version
 
| Major Version
 
|  
 
|  
Line 44: Line 47:
 
* 1.0 seen in Sim City 4, The Sims 2
 
* 1.0 seen in Sim City 4, The Sims 2
 
* 1.1 seen in The Sims 2
 
* 1.1 seen in The Sims 2
* 2.0 seen in SPORE
+
* 2.0 seen in Spore, The Sims 3
 +
* 3.0 seen in SimCity
  
 
|-
 
|-
 +
| 08
 
| Minor Version
 
| Minor Version
 
|  
 
|  
Line 52: Line 57:
  
 
|-
 
|-
 +
| 12
 
| unknown
 
| unknown
 
| 2.0
 
| 2.0
 
| u32
 
| u32
|  
+
| rowspan="2" | Unused, possibly reserved.  Can put data here.
  
 
|-
 
|-
 +
| 16
 
| unknown
 
| unknown
 
| 2.0
 
| 2.0
 
| u32
 
| u32
|
 
  
 
|-
 
|-
 +
| 20
 
| unknown
 
| unknown
 
| 2.0
 
| 2.0
Line 70: Line 77:
  
 
|-
 
|-
 +
| 24
 
| Date Created
 
| Date Created
 
| 1.0
 
| 1.0
 
| u32
 
| u32
|  
+
| in Unix time stamp format (the number of seconds since 1/1/1970) (Unused in Version 1.1)
  
 
|-
 
|-
 +
| 28
 
| Date Modified
 
| Date Modified
 
| 1.0
 
| 1.0
 
| u32
 
| u32
|  
+
| in Unix time stamp format (the number of seconds since 1/1/1970) (Unused in Version 1.1)
  
 
|-
 
|-
 +
| 32
 
| Index major version
 
| Index major version
 
| < 2.0
 
| < 2.0
Line 88: Line 98:
  
 
|-
 
|-
 +
| 36
 
| Index entry count
 
| Index entry count
 
|  
 
|  
 
| u32
 
| u32
| Number of entries in the Index table.
+
| Number of entries in the Index Table.
  
 
|-
 
|-
 +
| 40
 
| Offset of first Index entry
 
| Offset of first Index entry
 
| < 2.0
 
| < 2.0
 
| u32
 
| u32
|  
+
| Location of first index entry
  
 
|-
 
|-
 +
| 44
 
| Index size
 
| Index size
 
|  
 
|  
 
| u32
 
| u32
| Size of the Index table in bytes.
+
| Size of the Index table, in bytes.
  
 
|-
 
|-
 +
| 48
 
| Hole entry count
 
| Hole entry count
 
| < 2.0
 
| < 2.0
 
| u32
 
| u32
|  
+
| Number of Hole entries in the Hole Record
  
 
|-
 
|-
 +
| 52
 
| Hole offset
 
| Hole offset
 
| < 2.0
 
| < 2.0
 
| u32
 
| u32
|  
+
| Location of the hole Record
  
 
|-
 
|-
 +
| 56
 
| Hole size
 
| Hole size
 
| < 2.0
 
| < 2.0
 
| u32
 
| u32
|  
+
| Size of the hole Record
  
 
|-
 
|-
 +
| 60
 
| Index minor version
 
| Index minor version
 
| < 2.0?
 
| < 2.0?
 
| u32
 
| u32
| Version 1.1+ in The Sims 2 only. In DBPF 2.0, it has a value of 3, unknown if used.
+
| 01 = 7.0, 02 = 7.1, Version 1.1+ in The Sims 2 only. In DBPF 2.0, it has a value of 03, unknown if used.
  
 
|-
 
|-
 +
| 64
 
| Index offset
 
| Index offset
 
| 2.0
 
| 2.0
Line 136: Line 154:
  
 
|-
 
|-
 +
| 68
 
| unknown
 
| unknown
 
| 2.0
 
| 2.0
Line 142: Line 161:
  
 
|-
 
|-
 +
| 72
 
| reserved
 
| reserved
 
|  
 
|  
 
| char[24]
 
| char[24]
|  
+
| Reserved for use in future versions.
  
 
|}
 
|}
  
==Index Table==
+
===Index Table===
There are 2 known formats of indexes in the game. The format version is found in the file header.
+
There are two known formats of indexes in the game. The format version is found in the file header.  Index Tables list the contents of the DBPF package, giving the information below depending on the Index Table version used.
  
===DBPF 1.x, Index table 7.0===
+
A third format has been discovered, and is used in The Sims 2, but the structure is currently unknown. The Index Minor Version is an actual zero, so this would be called "Index Table 7.-1" by the logic used here. This format has been seen in files that ship with that game, and is believed to have been seen in files generated by that game, but has not yet been seen in user-created files (for obvious reasons). These files were discovered by the simple expedient of having a file-scanning tool choke on them during a scan of the entire game.
(20 bytes, some DBPF 1.1 entries use this format)
 
  
;DWORD
+
The index table is very similar to the directory file ([[DIR]]) within a DPBF package.  The difference being that the Index Table lists ''every'' file in the package, whereas the directory file only lists the compressed files within the package.  [[Reader]] presents a directory file that is a mashup of these two entities, listing every file in the package, as well as indicating whether or not that particular file is compressed.
:Type ID (see [[Category:File_Formats/SC4]])
 
;DWORD
 
:Group ID
 
;DWORD
 
:Instance ID
 
;DWORD
 
:Location of the file in the archive
 
;DOWRD
 
:Size of the file  
 
  
===DBPF 1.x, Index table 7.1===
+
====DBPF 1.x, Index Table 7.0====
(24 bytes)
+
(20 bytes, some DBPF 1.1 entries use this format) A typical entry in this table looks like:
  
;DWORD
+
'''DWORD'''
:Type ID (see [[Category:File_Formats/SC4]])
+
  Type ID
;DWORD
+
'''DWORD'''
:Group ID
+
  Group ID
;DWORD
+
'''DWORD'''
:Instance ID
+
  Instance ID
;DWORD
+
'''DWORD'''
:Second Instance ID
+
  Location of the file in the archive  
;DWORD
+
'''DWORD'''
:Location of the file in the archive  
+
  Size of the file  
;DOWRD
 
:Size of the file
 
  
==Hole Table==
+
====DBPF 1.x, Index Table 7.1====
 +
(24 bytes) A typical entry in this table looks like:
  
===DBPF 1.x===
+
'''DWORD'''
A Hole record contains the location and size of all holes in a DBPF file.
+
  Type ID
 +
'''DWORD'''
 +
  Group ID
 +
'''DWORD'''
 +
  Instance ID
 +
'''DWORD'''
 +
  Second Instance ID (a.k.a. Resource ID)(Not used in SimCity 4, but is used for GUID Based systems in The Sims 2)
 +
'''DWORD'''
 +
  Location of the file in the archive
 +
'''DWORD'''
 +
  Size of the file
  
;repeating
+
===Hole Table===
;DWORD
+
A Hole Table contains the location and size of all holes in a DBPF file. Holes are created when the game deletes something from a DBPF. The holes themselves are simply junk data of the appropriate length to fill the hole.
:Hole Location
 
;DWORD
 
:Hole size
 
  
Holes are created when the game deletes something from a DBPF. The holes themselves are simply junk data of the appropriate length to fill the hole.
+
====DBPF 1.x====
 +
(8 bytes) A typical entry in this table looks like:
  
==File==
+
'''DWORD'''
Bulk of DBPF packages. Each file is either uncompressed or compressed.  To check if a file is compressed you first need to read the [[DIR]] resource, if it exists.  (See the corresponding entry for a description).  If no DIR entry exists, then no files within the package are compressed.
+
  Hole Location
 +
  '''DWORD'''
 +
  Hole size
  
===Compression===
+
==Files Within DBPF Files==
 +
The files included in a DBPF file make up the bulk of the DBPF package.  Each file is either uncompressed or compressed.  To check if a file is compressed you first need to read the [[DIR]] file, if it exists.  If no DIR entry exists, then no files within the package are compressed.
  
When reading a DBPF file, the first thing you should do is check to see if any chunks inside are compressed.  This is done via the [[E86B1EEF|DIR]] resource.
+
===Directory File===
 +
The directory file ([[DIR]]) is included in DBPF files whenever there are compressed files within the package.
  
For example code and a description of the mechanics, see [[DBPF Compression]].
+
===Compression===
 
+
When reading a DBPF file, the first thing you should do is check to see if any chunks inside are compressed.  This is done via the [[DIR]] (E86B1EEF) resource. For example code and a description of the mechanics, see [[DBPF Compression]].
===Directory Files===
 
 
 
These are used for compression lists. See [[E86B1EEF]]
 
  
===Header===
+
===File Header===
Only if the file is compressed
+
(9 bytes) The file header only exists if the file is compressed, and looks like this:
  
;DWORD
+
'''DWORD'''
:Compressed Size of file
+
  Compressed Size of file
;WORD
+
'''WORD'''
:Compression ID (0x10FB) QFS Compression. (See [[packages]] for compression information)
+
  Compression ID (0x10FB) QFS Compression. (See [[DBPF Compression]] for more information)
;3 bytes
+
'''3 bytes'''
:Uncompressed Size of file
+
  Uncompressed size of the file
  
 
===Body===
 
===Body===
Raw data of the files.  See [[InternalFormats]] for a list.
+
Raw data of the files, may be compressed or uncompressed.  See [[List of File Formats]] for a list of the file types that may exist within a DBPF file.
 
 
=Pseudocode=
 
  
To read files inside a .package file you need to perform the following steps, in order
+
==Pseudocode==
 +
To read files inside a DBPF file, you need to perform the following steps, in order:
  
* Read the DBPF header
+
* Read the DBPF header.
* Read the Index tables
+
* Read the Index Tables.
* Check for a DIR record
+
* Check for a DIR record.
* Check for the file you want to extract
+
* Check for the file you want to extract.
* Is this file compressed? If so, uncompress it
+
* Is this file compressed? If so, decompress it.
* Read the file data and process accordingly
+
* Read the file data and process accordingly.
  
For an example PHP class for reading DBPF files, see [[DBPF/Source Code]]
+
For an example PHP class for reading DBPF files, see [[DBPF Source Code]]
  
=Visual Format Guide=
+
==Visual Format Guide==
 +
These visual guides to the DBPF file format were created by [[People:DarkMatter|DarkMatter]]:
  
Having recently started poking around with the DBPF format again for the SimCity 4 community on SimTropolis, I created some visual guides to the DBPF archive format. Hopefully they will be found useful here.
 
 
{|
 
{|
 
| [[Image:DBPF_File_Format_v1.1.png|thumb|left|200x152px|alt=DBPF v1.x Visual Guide|DBPF v1.x]]
 
| [[Image:DBPF_File_Format_v1.1.png|thumb|left|200x152px|alt=DBPF v1.x Visual Guide|DBPF v1.x]]
Line 242: Line 259:
 
|}
 
|}
  
Some information was taken from old DatGen source code, such as the Created/Modified timestamps and possible Flags DWORD. That information may not be accurate, as it was some time ago that I last chatted with Maxoids about the DBPF format.
+
Some information was taken from old DatGen source code, such as the Created/Modified timestamps and possible Flags DWORD. That information may not be accurate, per DarkMatter's own description. Reason given is that it was some time ago that he "last chatted with Maxoids about the DBPF format."
  
- DarkMatter
 
  
 +
{{navbox/FORMAT}}
 +
{{NAVBOX/SimCityFORMAT}}
  
[[category:File_Formats/SC4]]
+
[[Category:File Formats/SC2013]]
 +
[[Category:File Formats/SC4]]
 +
[[Category:MTS2]]

Latest revision as of 19:54, 8 August 2019

Database Packed File (DBPF) is a proprietary archive format used in many Maxis developed games, including The Sims 2, SimCity 4 and Spore. DBPF files are similar to Zip files, as they hold other files inside them which are usually compressed. All plugins for SimCity 4 (excluding the SC4 Extra Cheats.dll) are DBPF files, the file name's extension is irrelevant. Note that savegame (.sc4) files are also DBPF files. DBPF files are sometimes referred to as packages following the convention used in The Sims 2, where the files sometimes have the .package file extension.

DBPF files can be identified by opening the file in notepad, or preferably a hex editor, as the first 4 characters/bytes will be the letters DBPF. If a file in the plugins folder does not start with these 4 letters the game will not load them, however the Reader will. This means that if you want to disable a plugin you can open the DBPF file in a hex editor and change the first byte to something other than 44 (in Hexadecimal) (D) and the game will not load the plugin, but you will still be able to edit it in the reader (saving the disabled plugin with the reader will not re enable it, to re enable you must change the first byte back to 44 (hexadecimal)).

Entries may be compressed with RefPack/QFS compression depending on their entry data.

For an examination of the Spore DBPF format, see Spore DBPF.

Format

Overview

This is the general structure of Database Packed File packages:

Header (96 bytes)
File 1
File 2
...
File n
Index Entry 1
Index Entry 2
...
Index Entry n 

Header first, individual files following with no file names and a small file header area at the beginning of each, followed by an index of all the files in the archive. Note that files may appear after the index table in files that are updated.

Header

Offset Name Version Type/Size Info
00 Identifier char[4] DBPF
04 Major Version u32
  • 1.0 seen in Sim City 4, The Sims 2
  • 1.1 seen in The Sims 2
  • 2.0 seen in Spore, The Sims 3
  • 3.0 seen in SimCity
08 Minor Version u32
12 unknown 2.0 u32 Unused, possibly reserved. Can put data here.
16 unknown 2.0 u32
20 unknown 2.0 u32 Should always be zero in DBPF 2.0.
24 Date Created 1.0 u32 in Unix time stamp format (the number of seconds since 1/1/1970) (Unused in Version 1.1)
28 Date Modified 1.0 u32 in Unix time stamp format (the number of seconds since 1/1/1970) (Unused in Version 1.1)
32 Index major version < 2.0 u32 Always 7 in The Sims 2, Sim City 4. If this is used in 2.0, then it is 0 for SPORE.
36 Index entry count u32 Number of entries in the Index Table.
40 Offset of first Index entry < 2.0 u32 Location of first index entry
44 Index size u32 Size of the Index table, in bytes.
48 Hole entry count < 2.0 u32 Number of Hole entries in the Hole Record
52 Hole offset < 2.0 u32 Location of the hole Record
56 Hole size < 2.0 u32 Size of the hole Record
60 Index minor version < 2.0? u32 01 = 7.0, 02 = 7.1, Version 1.1+ in The Sims 2 only. In DBPF 2.0, it has a value of 03, unknown if used.
64 Index offset 2.0 u32 Offset to Index table in DBPF 2.0.
68 unknown 2.0 u32
72 reserved char[24] Reserved for use in future versions.

Index Table

There are two known formats of indexes in the game. The format version is found in the file header. Index Tables list the contents of the DBPF package, giving the information below depending on the Index Table version used.

A third format has been discovered, and is used in The Sims 2, but the structure is currently unknown. The Index Minor Version is an actual zero, so this would be called "Index Table 7.-1" by the logic used here. This format has been seen in files that ship with that game, and is believed to have been seen in files generated by that game, but has not yet been seen in user-created files (for obvious reasons). These files were discovered by the simple expedient of having a file-scanning tool choke on them during a scan of the entire game.

The index table is very similar to the directory file (DIR) within a DPBF package. The difference being that the Index Table lists every file in the package, whereas the directory file only lists the compressed files within the package. Reader presents a directory file that is a mashup of these two entities, listing every file in the package, as well as indicating whether or not that particular file is compressed.

DBPF 1.x, Index Table 7.0

(20 bytes, some DBPF 1.1 entries use this format) A typical entry in this table looks like:

DWORD
 Type ID
DWORD
 Group ID
DWORD
 Instance ID
DWORD
 Location of the file in the archive 
DWORD
 Size of the file 

DBPF 1.x, Index Table 7.1

(24 bytes) A typical entry in this table looks like:

DWORD
 Type ID
DWORD
 Group ID
DWORD
 Instance ID
DWORD
 Second Instance ID (a.k.a. Resource ID)(Not used in SimCity 4, but is used for GUID Based systems in The Sims 2)
DWORD
 Location of the file in the archive 
DWORD
 Size of the file

Hole Table

A Hole Table contains the location and size of all holes in a DBPF file. Holes are created when the game deletes something from a DBPF. The holes themselves are simply junk data of the appropriate length to fill the hole.

DBPF 1.x

(8 bytes) A typical entry in this table looks like:

DWORD
 Hole Location
DWORD
 Hole size

Files Within DBPF Files

The files included in a DBPF file make up the bulk of the DBPF package. Each file is either uncompressed or compressed. To check if a file is compressed you first need to read the DIR file, if it exists. If no DIR entry exists, then no files within the package are compressed.

Directory File

The directory file (DIR) is included in DBPF files whenever there are compressed files within the package.

Compression

When reading a DBPF file, the first thing you should do is check to see if any chunks inside are compressed. This is done via the DIR (E86B1EEF) resource. For example code and a description of the mechanics, see DBPF Compression.

File Header

(9 bytes) The file header only exists if the file is compressed, and looks like this:

DWORD
 Compressed Size of file
WORD
 Compression ID (0x10FB) QFS Compression. (See DBPF Compression for more information)
3 bytes
 Uncompressed size of the file

Body

Raw data of the files, may be compressed or uncompressed. See List of File Formats for a list of the file types that may exist within a DBPF file.

Pseudocode

To read files inside a DBPF file, you need to perform the following steps, in order:

  • Read the DBPF header.
  • Read the Index Tables.
  • Check for a DIR record.
  • Check for the file you want to extract.
  • Is this file compressed? If so, decompress it.
  • Read the file data and process accordingly.

For an example PHP class for reading DBPF files, see DBPF Source Code

Visual Format Guide

These visual guides to the DBPF file format were created by DarkMatter:

DBPF v1.x Visual Guide
DBPF v1.x
DBPF v2.0 Visual Guide
DBPF v2.0

Some information was taken from old DatGen source code, such as the Created/Modified timestamps and possible Flags DWORD. That information may not be accurate, per DarkMatter's own description. Reason given is that it was some time ago that he "last chatted with Maxoids about the DBPF format."