Guide To Exploring File Formats - GameDevs
THE DEFINITIVE GUIDE TOEXPLORING FILE FORMATSMr.Mouse and WATTO 2004 XeNTaX.com & WATTO.orgVersion 1.0, November 1st 2004
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS1.Table of Contents1. Table of Contents. 22. Introduction . 4What is a GRA?.5What is a GRAF? .63. Tools. 7Hex Editors.7Hex Workshop .84. Terms, Definitions, and Data Structures. 10Files.10Bits .11Bytes .1216-bit (2-byte) numbers .1332-bit (4-byte) numbers .1464-bit (8-byte) numbers .15Strings .16Hexadecimal Numbering .17Signed and Unsigned Numbers.18Big-Endian and Little-Endian .19File Offsets .205. Archive Patterns. 21Directory Archives .22Tree Archives .23Chunked Archives .26Split Chunk Archives .27External Directory Archives .286. Checking Your Results. 29Common Types Of Fields.29Validating Your Fields.30Padding .31Filename Patterns .327. Encryption and Compression . 337.1 The basics .33XOR.34NOT.35SHL .35SHR.357. 2 Encryption .36Painkiller Encryption .37Compression .448. Worked Examples . 47Quake *.PAK .479. Appendix. 53A: BinaryByte Number Table.53B. American Standard Code for Information Interchange (ASCII) Table57C. Format of some common Game Archives.59D: Useful References .63 2004 Mr.Mouse and WATTO2
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSE: Common File Format Tags .6410. Legal Information . 65 2004 Mr.Mouse and WATTO3
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS2.IntroductionComputer games are vast and many, however most computer games havesomething in common – they need a place to store all their important files likeimages, movies, and sounds. To do this, computer game developers typicallystore their data into a big archive file.There are many reasons for storing all your data files in one big archive, somereasons include reducing the number of files on a CD, hiding the data files tostop people hacking the game, and so that all data files can be accessedusing a single data stream.However, the bad news for gamers is that there are almost as many differentarchives as there are different computer games – every game developercreates their own archive formats, and they even change their formatsbetween games or departments in the company.This brings us to the focus of the tutorial – how to explore the archives andgrab the files from within them. This tutorial will attempt to make it easy foranyone to explore a new format, with the aim of promoting gamemodifications and enhancements by the community.In the following pages, we will discuss the terms Game Resource Archives(GRAs) and Game Resource Archive Formats (GRAFs), common data types,and other definitions. From there, we will explain the fundamentals of crackinga file format, including the tools you use, and the patterns to look out for.Thanks for reading our guide; we wish you the best of luck in yourexploration . 2004 Mr.Mouse and WATTO4
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSWhat is a GRA?We should start this off by defining the definition of an archive. An archive is afile that usually stores many small individual data files inside it. The most wellknown archive format that you will be familiar with is *.zip archive, which youshould recognise as a way to package many files together into a single file.A GRA (Game Resource Archive) defines an archive that is used by a specificgame, and contains resource or data files that are used within the game, suchas images, sounds, scripts, and text. 2004 Mr.Mouse and WATTO5
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSWhat is a GRAF?GRAF (Game Resource Archive Format) describes the way a game archive isconstructed, and in particular, how and where the files are found within thearchive. These formats usually differ for each individual game, howeveroccasionally a game developer will stick with a particular format for a fewgames of the same vintage.Programmers usually define their GRAFs according to the needs andstructure of the game itself, i.e. the game engine. A common principle,however, is to use a format that can be quickly and easily be opened by thegame, even though the files in the archive may be different types. Forexample, different levels in a game need different sounds and textures, sooften the sounds and images for each level are packaged into the samearchive, even though clearly images and sounds are handled differently.In addition, the actual resources used in a game change frequently as thegame is being developed. To make it easy for the game to adapt to thechanges, the archive format is structured to be ‘universal’ in their approach ofstoring files. In simpler terms, the archive needs to provide the game enginewith a common and recognisable pattern of resource files, and a list of thefiles that the GRA archive contains. 2004 Mr.Mouse and WATTO6
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS3.ToolsHex EditorsHex editors are the standard class of program that you use if you wish tocrack the formats of a newly encountered game.The generic hex editor displays the contents of any file in a similar way that apiece of text in a word processor would be shown to you: from left to right, lineby line, all the way down to the last character. However a hex editor differsfrom a word processor in that it shows the file as hexadecimal numbers ratherthan letters.There are many hex editors available for free download over the Internet,however the one that we can fully recommend is Hex Workshop fromBreakpoint Software. Hex Workshop will be the editor used in all thescreenshots, and the one used in the examples. Hex Workshop includessome handy little functions, which is partly the reason why we recommend it,such as an easy to use hexadecimal calculator, lists of all data types that areat the current location of your cursor, bookmarking, and colour mapping. 2004 Mr.Mouse and WATTO7
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSHex WorkshopExamine Figure 1 carefully, as it shows the hex editor that will be usedthroughout this guide, Hex Workshop. After installation you can either start itand open any file you want using the File menu, alternatively (and this you willdo most of the time) you can select to open files from the context menu inWindows Explorer. To do that, right click on any file in Explorer and select“Hex edit with Hex Workshop”.Once you have opened a file, you will be presented with a similar view of thefile content as depicted in Figure 1. You can examine the file’s hexadecimalinterpretation (A), or ASCII interpretation of the same bytes (B). The table tothe far left shows the offsets of the lines shown. As you can see, we have justopened one of Doom 3’s PK4 files. We will later see that these are actuallyZIP files. For now, you can see the file starts with the characters “PK”. In thiscase, it is actually an ID tag that is used in all ZIP archives. PK stands forPhillip Katz, the author of the ZIP compression technique, who died at the ageof 37 due to fatal effects of alcoholism in April 2000. The current cursorposition is at offset 18 (see figure) and the Data Interpreter (C) shows therelative value at this file position. You can see it shows different data types(see Chapter 4) ranging from bytes to strings to binary values using the bytesequence starting at this offset. In the Figure, we have colour mapped andbookmarked (D) some areas of our interest. You can select any range ofbytes in the file and bookmark or colour map it. Simply drag the cursor alongyour area of interest holding the left mouse button and right click thehighlighted area to show the context menu from where you select the options.When you bookmark it, you can choose how the bookmark should beinterpreted (value), and give a description. The bookmarks will be shown withtheir offset in the file and the size in bytes (length). This is a handy feature asyou can click on any bookmark to quickly switch to that offset. When youleave Hex Workshop it will ask you if you want to save your colour map orbookmarks. Later, you can load these regardless of the file you have opened.Thus, if you have solved the puzzle of a GRAF, you can apply the bookmarksand colour mapping to other files that you expect to have the same format.Hex Workshop has another handy GoTo method, which you too can accessfrom the context menu. There are a number of options. With the basic GoTooption you can type in an address (offset) in the file you wish to go to. Twomore powerful options are accessed by selecting a range of bytes (preferablyone that you expect to represent an offset value) and then right clicking it. Youcan then either GoTo the value of your selected range or GoTo that theaddress of your value plus the current offset of your variable.For instance, suppose you expect a 4-byte value (i.e. 32-bit) at offset 4 torepresent the offset of a resource in a GRA. You can then opt to select the 4bytes from this position (i.e. byte 4, 5, 6, and 7) and jump to the location in thefile this 32-bit number points to. On the other hand, it might be that your filehas an ID tag (GRAIS) of 4 bytes (i.e. bytes 0-3, see also offsets) and theGRAF does not count the 4 bytes of this tag when referring to offsets ofresources. Thus the number you are labelling as an offset may in fact be arelative offset, meaning that it is relative to some other offset (in this case theoffset where the GRAIS ends). You can easily jump to this relative location 2004 Mr.Mouse and WATTO8
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSwith Hex Workshop, as the other GoTo option you are offered is to jump to thelocation your value depicts the offset of the currently selected area. You cansee that this program can come in very handy indeed if you are looking tounravel the puzzle of new archive formats.Figure 1. Layout of Hex Workshop. A. Hexadecimal representation of filecontent in sequential byte order. B. ASCII interpretation of file content. C.Interpretation of data at current cursor position. D. User window showingbookmarked areas in the file and the description given to these areas by theuser. 2004 Mr.Mouse and WATTO9
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS4.Terms, Definitions, and Data StructuresTo understand the patterns and construction or archives, we must firstintroduce the concept of data structures.FilesFiles are typically dealt with as a series of bytes stored one after the other,and when combined form a representation of a piece of data. If you have a filethat is 12 bytes in size, it indicates that the file contains 12 single bytes thatcan be read in a certain way to represent a value. File sizes start off at thepreliminary byte, followed by kilobytes (KB, 1024 bytes) and megabytes (MB)that are generic terms representing thousands and millions of bytesrespectively. 2004 Mr.Mouse and WATTO10
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSBitsWhen we talk about the basic structure of a file, we typically think in terms ofbytes. However, the actual underlying file structure, at its absolute simplest, isa sequence of bits or binary values, but we usually don’t deal with this level ofrepresentation too often because binary values don’t represent anythingmeaningful in their singular state – they need to be grouped into sets of 8 bitsto become a valid representation of data.A bit, or binary value, is the language of a computer, and thus the underlyingstructure of everything readable by a computer. A bit only has 2 possiblevalues – 1 or 0 – thus it is obvious why they are limited in what they represent.However, when we take groups of bits and join them together, their valuescouple together into larger numbers that hold greater value.The number of grouped bits needed to represent basic data is 8 bits, morecommonly known as a byte. A byte can hold any value between 0 and 255,much more than the 2 possible combinations available to a bit.So how do coupled bits represent a larger numerical value such as that of abyte? This is achieved quite easily by referring to each of the 8 bits as anincreasing power of 2. If we take a look at a single bit, we can think of it ashaving either the value 1x20 or 0x20 – thus giving us the values 1 or 0. If weadd a bit to the left, the power of the new bit is either 1x21 or 0x21 – either 2 or0. By adding the values of these 2 bits together, you should be able to seethat all possible combinations will give us the values 0, 1, 2, or 3. This isshown in the table below.Bit 1 (21)0011Bit 0 (20)0101Value0 (0x21 0x20)1 (0x21 1x20)2 (1x21 0x20)3 (1x21 1x20)If we continue this pattern for the remaining 6 bits (to make a total of 8 bits ie1 byte) then we can provide powers up to 27, and thus if all 8 bits had thevalue 1 and we added them together, we would end up with the number 255.Appendix A shows all possible values of a byte, and the 8 bits used to createthe value. 2004 Mr.Mouse and WATTO11
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSBytesAs described above, a byte is composed of 8 bits, and can thus contain avalue between 0 and 255. When you open a file in a hex editor, each valueyou see represents a single byte value. You will probably not see numbersbetween 0 and 255 representing the bytes in a file; rather you will mostprobably see hexadecimal representations of the values, ranging from 00 toFF. For an explanation of hexadecimal numbering, see the HexadecimalNumbering section below. 2004 Mr.Mouse and WATTO12
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS16-bit (2-byte) numbersFrom this point forward, we need to be careful what we call each of these datatypes. Why? Because each programming language uses different names forthe same values.A 16-bit value is commonly known in older programming languages (C orearlier) as a word or an Integer. Newer programming languages, such as Javaand the .NET languages, call it a Short.A 16-bit number is just as the name suggests, a number created by 16 bits ina row. To determine the value of the 16-bit number, we follow the sameprocess as when we wanted to get the value of a byte.Each of the 16 bits that make up the 16-bit number represents a power of 2 –the leftmost bit represents 215 and the rightmost bit 20. Just as with bytes, wejust go through each bit and calculate the bitvalue x power.An example – lets say we have the following 16 bits 101111000001100Working from left to right, we get the value 1x215 0x214 1x213 1x212 1x211 1x210 0x29 0x21 0x20If you work this out, you should end up with the number 24076.If all 16 bits had the value 1, you would end up with the number 65535 –therefore the value of a 16-bit number ranges between 0 and 65535. 2004 Mr.Mouse and WATTO13
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS32-bit (4-byte) numbersIf you have been following us so far, you should be able to see just how youwould calculate the value of a 32-bit number. A 32-bit number uses 32 bits torepresent it, and therefore has a value between 0 and 4294967295. This is avery large range, and thus is the reason why most values used in archives arestored in groups of 32-bits.A 32-bit number in older programming languages is known as a dword or aLong. In newer languages, this is known as an Integer. 2004 Mr.Mouse and WATTO14
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS64-bit (8-byte) numbersAs with the previous example, this is a number represented by 64 bits, 44073709551615. However, they can also represent 9313486232E308 to -4.94065645841247E-324 for negative valuesand from 4.94065645841247E-324 to 1.79769313486232E308 for positivevalues. These numbers are very rarely used in GRAs, but will be becomingmore popular as things like 64-bit processors take off. Programminglanguages can refer to these as Longs, Doubles and even Floats. 2004 Mr.Mouse and WATTO15
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATSStringsOne of the most common tasks performed on a computer is word processing,so naturally we need some way of representing text in a document. A piece oftext in a document is called a String, which more formally means a sequenceof characters.Although there are many languages in the world, and Spanish being the mostspoken Latin based language, in Informatics the first Latin language used inthe Western world is English. The English script consists of 52 letters (upperand lower case), 10 numbers, and
THE DEFINITIVE GUIDE TO EXPLORING FILE FORMATS Hex Workshop Examine Figure 1 carefully, as it shows the hex editor that will be used throughout this guide, Hex Workshop. After installation you can either start it and open any file you want using the File menu, alternatively (and this you will
DND 04 formats in Ms. Word Engineering (ENG) 03 formats in Ms. Word Marketing (MKT) 05 formats in Ms. Word Operation (OPN) 15 formats in Ms. Word Services (SER) 03 formats in Ms. Word System (SYS) 17 formats in Ms. Word & Ms. Excel Training (TRG) 07 formats in Ms. Word 5. Standard Operating Procedures (SOPs) 06 SOPs in Ms. word 6. Process Flow .
HACCP Manual 05 chapter and 04 annexure in MS Word 2. Procedures 14 procedures in MS Word 3. HACCP docs. 18 files in MS Word 4. Formats 52 formats in MS Word Engineering (ENG) 06 formats in MS Word Purchase (PUR) 05 formats in MS Word Despatch (DES) 03 formats in MS Word Housekeeping (HKC) 09 formats in MS Word .
Also Available from Thomson Delmar Learning Exploring Visual Effects/Woody/Order # 1-4018-7987-X Exploring Sound Design for Interactive Media/Cancellaro/Order #1-4018-8102-5 Exploring Digital Software on the Mac/Rysinger/Order # 1-4018-7791-5 Exploring DVD Authoring/Rysinger/Order # 1-4018-8020-7 exploring DIGITAL VIDEO Second Edition Rysinger
LibreOffice File Formats Native document format is Open Document format (ODT, ODS, ODP, ODG - Text, Spreadsheet, Presentation, Graph) Supports other formats including MS Office Formats - Compatibility good but less than 100% Recent MS Office versions support LO File Formats - Compatibility good but less than 100% iPad/iPhone display LibreOffice ODT & ODP format
ASSURED FILE TRANSFER What was previously referred to as "trusted downloading" is now "assured file transfer." This short provides guidance on the requirements for assured file transfer, and includes an approved file formats job aid of DCSA authorized file types and formats for assured file transfer procedures. DATA SPILLS
work/products (Beading, Candles, Carving, Food Products, Soap, Weaving, etc.) ⃝I understand that if my work contains Indigenous visual representation that it is a reflection of the Indigenous culture of my native region. ⃝To the best of my knowledge, my work/products fall within Craft Council standards and expectations with respect to
The File Formats manual is supplied pursuant to and subject to the terms of the Sony Computer Entertainment PlayStation License and Development Tools Agreements, the Licensed Publisher Agreement and/or the Licensed Developer Agreement. The File Formats manual is intended for distribution to and use by only Sony Computer Entertainment
www.surabooks.com áfu« bjhLnth« ïa‰ãaš. òâa. 11-M« tF ò. Ruh g Ënfõ‹ br‹id. 2018-M« M L¡fhd òâa nj î â l â‹go jahÇ¡f g LŸsJ
