need some helps..
I'm working on a C++ library to handle SC2 replays.
Currently got <almost> all the "replayinfo" reversed, and only messages from "messageevents".
I'll try to provide a public SVN soon so that people can join & participate.
Sounds great. Please let me know when it's ready. Thanks :)
I will be posted up the info on Sourcepeek.com as soon as I get time.
We've got a semi working parser on Nibbits.com, but it appears that the file requires you to read it by bit (rather than byte) so its sort of complex.
Right now, we are able to read (by bytes) most of the file, until right after the "Needed Modules", which are the 40 byte blobs which contain "s2ma", the realm id, and the file hash. There's a chunk in here which requires a bit stream, but then its fine again once you hit the map title. We ended up reverse-seeking as a quick solution on Nibbits.
Newbiz let me know when you get that up, hit me up here, or on irc (dcramer[nibbits] on Rizon).
I have the exact same problem here :/
At the end of the replay.info there are 5 chunks of 40 bytes, and then a varying length chunk until the name of the map.
I can't find any logical way to compute the length of this chunk.
Currently, I'm just doing a trial/error to guess the bytes that may describe the length of the chunk.
Here are some dumps, I stressed the bytes that should store the length, one way or another...:
10 0E 02 06 08 01 64 02 06 15 80 24 2F 3F A6 AF 00 00 00 00 00
20 0C 02 06 08 03 01 64 02 06 15 01 80 24 2C 06 8F B4 00 00 00 00 00
10 0E 02 06 08 03 02 64 02 06 15 01 80 24 95 B6 1B A2 00 00 00 00 00
20 0C 02 06 08 01 64 02 06 15 01 81 24 A6 BF E3 22 00 00 00 00 00
20 2C 00 06 08 01 64 02 06 14 01 80 24 02 05 8F 00 C0 04 82 13 49 00 C0 04 94 B7 9E 32 00 00 00 00
20 2C 00 06 08 03 01 64 02 06 14 03 80 24 02 05 0F C0 04 82 13 09 C0 84 0E FB 69 19 00 00 00 00
^^ ^^ ^^ ^^ ^^ ^ ^^
By the way, it would be interesting to share our parsers. The only source of information that I have is http://code.google.com/p/vgce/source/browse/trunk/docs/Blizzard/Starcraft%20II/replays.txt (http://code.google.com/p/vgce/source/browse/trunk/docs/Blizzard/Starcraft%20II/replays.txt) (which is often false) and nibbits.com to download replays ;)
Here's Nibbits initial parser (incomplete).
It's failing on the List(Boolean) chunk at the moment, but the structure is accurate.
http://github.com/dcramer/nibbits-shared/blob/master/sc2/parsers/replays.py (http://github.com/dcramer/nibbits-shared/blob/master/sc2/parsers/replays.py)
Has the replay file format changed recently due to patches ?
I have clearly different dumps from the first release to the latest :/
Here is a dump of the message events file. These are chat messages.
A message is easy to parse, it's basically composed of a timestamp (from last message) followed by a player ID (starting at 1). Then comes the message string, with its length before.
The timestamp & player ID are ended by a terminal 0 (like cstrings) since timestamp has a varying length.
I can't figure out the global file header for now. It seems to have a fixed 40 bytes headers, and then something variable :/
------------- 2 players: TO ALL ---------------
2C 01 00 02 68 69
h i
68 01 00 05 66 72 6F 6D 3F
f r o m ?
F8 02 00 06 73 77 65 64 65 6E
s w e d e n
50 02 00 04 79 6F 75 3F
y o u ?
01 68 01 00 07 66 69 6E 6C 61 6E 64
f i n l a n d
01 60 01 00 0B 6D 69 6E 64 2E 6B 69 76 76 69 3F
m i n d . k i v v i ?
01 98 02 00 0E 6E 65 76 65 72 20 68 65 61 72 64 20 6F 66
n e v e r h e a r d o f
01 E7 01 00 04 67 6C 68 66
g l h f
E0 02 00 04 73 61 6D 65
s a m e
02 44 BF 01 00 02 67 67
g g
01 54 02 00 02 67 67
g g
^^ message size
^^ end of header
^^ player ID
^^ ^^ timestamp
------------- 4 players: TO ALL ---------------
51 02 00 02 67 67
g g
8C 03 00 02 67 67
g g
C0 04 00 02 67 67
g g
09 BD 03 83 00 32 8E 84 00 97 94 06 28 01 00 1E 68 6F 77 20 64 6F 20 79 6F 75 20 73 74 6F 70 20 6D 61 73 73 20 73 74 61 6C 6B 65 72 73 3F
h o w d o y o u s t o p m a s s s t a l k e r s ?
05 18 03 00 12 6D 61 73 73 20 73 74 61 6C 6B 65 72 73 20 73 75 63 6B
m a s s s t a l k e r s s u c k
C8 04 00 15 69 64 6B 20 49 20 64 6F 6E 27 74 20 70 6C 61 79 20 74 6F 73 73
i d k I d o n ' t p l a y t o s s
09 0D 01 00 18 79 6F 75 20 74 68 69 6E 6B 20 73 74 61 6C 6B 65 72 73 20 73 75 63 6B 3F
y o u t h i n k s t a l k e r s s u c k ?
05 1A 03 00 07 74 68 65 79 20 64 6F
t h e y d o
^^ message size
^^ end of header
^^ player ID
^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ timestamp
= Message Structure (replay.message.events) =
* Unknown Function, 7 bytes for each, always at header
** 0x0, maybe Frame ?
** unknown
** 0x80, maybe OPcode ?
** 0x0
** 0x0
** unknown
** unknown
* Message Function
** Accumulate Frames, 1-N bytes, Big Endian and Frame/64=Second
** Sender, 1 byte
** OPcode, 1 byte, 0x0 to all, 0x2 to alliance
** Message Len, 1 byte
** Message Content, n bytes from Message Len
* Blink Function (alt+g on Map)
** Accumulate Frames, 1-N bytes
** Player, 1 byte
** OPCode, 1 byte, 0x83
** X, 4 bytes
** Y, 4 bytes
= Info Structure (replay.info) =
* Players, 1 byte, always 0x10
** Player Name Length, 1 byte
** Player Name, N bytes from Player Name Length
** Player Info, 5 bytes
* MapInfo
** Unknown, 4 bytes
** unkDLen, 1 byte, always 0x4
** unkDefault, unkDLEn bytes, "Dflt"
** allianceLocked, 1 byte, (allianceLocked & 0x01) if alliances are locked
** Unknown, 1 byte
** Game Speed, 1 byte. 0 ~ 4, 4 is More Faster
** Unknown, 11 bytes
** Map Cache Length, 1 byte, always 0x4B
** Map Cache Path, 0x4B bytes
* Unknown, 686 bytes
* S2MA * 5
** S2MA, 4 bytes, "s2ma"
** Zero, 1 byte
** BattleNet, 3 bytes, "KRB"=KR Battle.Net, "EUB", "USB"
** MD5?, 32 bytes
* Unknown, X bytes
* MapLen1, 1 byte
* MapLen2, 1 byte
* Map Name, MapLen1<<2+MapLen2 bytes
* Zero, 1 byte, always 0x0
* Players, 4 bytes, always 0x10
** Player Name Length, 1 byte
** Player Name, N bytes from Player Name Length
** Player Race Length, 1 byte
** Player Race, N bytes from Player Race Length
** Player Color Length, 1 byte
** Player Color, N bytes from Player Color Length
= Sync Structure (replay.sync.events) =
* 1 bytes, always 4
* 2 bytes, unknown
* 1 bytes, always 0 or 1
Elapse Time (sec) = (filesize of replay.sync.events) / 64;
= Game Structure (replay.game.events) =
Start OPcode
* 00 01 1B, player1 initial
* 00 02 1B, player2 initial
* 00 10 05, game start
Action OPcode
* Accumulate Frames, 1~N bytes, Big Endian
* Player, 1 byte, 0x21 or 0x61 -> player1
* OpCode, 1 byte
OPCodes
* 0x81, Move Camera
** X, 4 bytes
** Y, 4 bytes
** Hor, 4 bytes
** Vet, 4 bytes
** Unknown, 4 bytes
* 0x0B, Unit Action, like building, morph, research, upgrade, order, ability
* 0x3C,
* 0xAC, Select Unit
* 0x0D,
* 0x1D,
Thank you very much ! These are very valuable information ^^
Did you figure it out yourself or are you in a parser project ? If not, you can join my effort @ http://projects.coderbasement.com/projects/show/sc2replay (http://projects.coderbasement.com/projects/show/sc2replay)
Bttw, could you explain a little bit the messageevents header structure ?
* Message Header, 7 bytes for each, pos 0 3 4 = 0x0, pos 2 = 0x80
I don't really understand this line. What do you mean by [/size][/color]pos 0 3 4 = 0x0, pos 2 = 0x80 ?[/color][/size][/font]
Thanks again
Okay thanks a lot.
So I end up with something like :
00 21 80 00 00 1D 00 // Header 0
00 21 80 00 00 20 00 // Header 1
00 21 80 00 00 24 00 // Header 2
00 21 80 00 00 26 01 // Header 3
00 21 80 00 00 2E 00 // Header 4
00 21 80 00 00 32 00 // Header 5
elapsed frames | player id | zero | msg length | msg
05 2C | 01 | 00 | 02 | 68 69
68 | 01 | 00 | 05 | 66 72 6F 6D 3F
F8 | 02 | 00 | 06 | 73 77 65 64 65 6E
50 | 02 | 00 | 04 | 79 6F 75 3F
01 68 | 01 | 00 | 07 | 66 69 6E 6C 61 6E 64
01 60 | 01 | 00 | 0B | 6D 69 6E 64 2E 6B 69 76 76 69 3F
01 98 | 02 | 00 | 0E | 6E 65 76 65 72 20 68 65 61 72 64 20 6F 66
01 E7 | 01 | 00 | 04 | 67 6C 68 66
E0 | 02 | 00 | 04 | 73 61 6D 65
02 44 BF | 01 | 00 | 02 | 67 67
01 54 | 02 | 00 | 02 | 67 67
Any idea how to guess the number of headers at the beginning of the file ?
Quote from: newbiz on March 04, 2010, 02:07:31 AM
Any idea how to guess the number of headers at the beginning of the file ?
A poor idea is:
while(BMessage[HeadPointer]==0 && BMessage[HeadPointer+2]==0x80 && BMessage[HeadPointer+3]==0 && BMessage[HeadPointer+4]==0) {
HeadPointer += sizeof(SC2MessageHead);
}
lol ok ^^
If that's not considered harrassment, would you mind explaining the Accumulate Frames, 1-N bytes, Big Endian and Frame/64=Second line ?[/color][/size][/font]
I got a replay where the first message is a "gg", at the very end of the game, so the accumulated frame field is quite long ( 193 bytes ). How should I interpret it ?
A sum of 64b big endian integers storing elapsed frames ?
elapsed frames | player id | zero | msg length | msg
05 2C | 01 | 00 | 02 | 68 69
68 | 01 | 00 | 05 | 66 72 6F 6D 3F
The second message was chatted at (0x52c+0x68)/64 sec.
I don't have any frame field that is over 5 bytes. Maybe there is something wrong.
I attached the dump (renamed to .txt so that i can upload it here).
From what you've wrote i have the following headers:
00 22 80 00 00 1D 00
00 21 80 00 00 1D 00
00 22 80 00 00 32 00
00 21 80 00 00 32 00
00 23 80 00 00 1C 00
00 23 80 00 00 1E 01
00 23 80 00 00 25 00
00 23 80 00 00 27 01
00 23 80 00 00 2F 00
00 23 80 00 00 32 00
And then a bunch of (unknown?) 193 bytes until the first message.
I have no clue how to interpret it.
00 22 80 00 00 1D 00
00 21 80 00 00 1D 00
00 22 80 00 00 32 00
00 21 80 00 00 32 00
00 23 80 00 00 1C 00
00 23 80 00 00 1E 01
00 23 80 00 00 25 00
00 23 80 00 00 27 01
00 23 80 00 00 2F 00
00 23 80 00 00 32 00
35 B8 03 83 00 F1 F0 81 01 00 20 01
99 E6 03 83 00 84 93 86 00 3C 1A 06
01 EE 03 83 00 B4 D3 81 00 45 E7 06
01 B5 04 83 01 11 9A 85 00 6C 10 06
01 B8 03 83 01 00 69 84 00 FE 7D 07
09 3F 03 83 00 8E 3A 80 01 27 54 03
3D 67 03 83 01 02 05 82 00 F3 0E 05
11 44 03 83 01 00 69 84 01 08 4B 00
10 03 83 01 06 D8 83 00 FE 7D 07
41 3E 03 83 00 ED 1D 80 01 05 06 05
05 E3 03 83 00 C1 B0 86 01 00 20 01
19 51 03 83 00 AE 64 82 00 6D 1C 01
59 4E 03 83 01 28 9E 82 00 EF CA 03
29 47 03 83 00 99 7C 80 01 1F 29 04
15 04 03 83 00 AE 64 82 00 6B 79 07
31 71 03 83 00 69 3C 85 00 28 80 05
0D 51 02 00 02 67 67
8C 03 00 02 67 67
C0 04 00 02 67 67
09 BD 03 83 00 32 8E 84 00 97 94 06
28 01 00 1E 68 6F 77 20 64 6F 20 79 6F 75 20 73 74 6F 70 20 6D 61 73 73 20 73 74 61 6C 6B 65 72 73 3F
05 18 03 00 12 6D 61 73 73 20 73 74 61 6C 6B 65 72 73 20 73 75 63 6B
C8 04 00 15 69 64 6B 20 49 20 64 6F 6E 27 74 20 70 6C 61 79 20 74 6F 73 73
09 0D 01 00 18 79 6F 75 20 74 68 69 6E 6B 20 73 74 61 6C 6B 65 72 73 20 73 75 63 6B 3F
05 1A 03 00 07 74 68 65 79 20 64 6F
01 4B 03 02 14 75 6E 6C 65 73 73 20 75 20 6C 65 74 20 68 69 6D 20 67 65 74
01 7E 03 00 19 75 6E 6C 65 73 73 20 75 20 6C 65 74 20 68 69 6D 20 67 65 74 20 6C 69 6B 65
30 03 00 03 69 64 6B
18 01 00 2A 79 6F 75 27 64 20 70 72 65 66 65 72 20 6D 61 73 73 20 70 72 6F 64 75 63 69 6E 67 20 6D 61 72 69 6E 65 73 20 74 68 65 6E 20 3F
10 03 00 02 36 30
01 8E 03 00 0C 69 74 20 77 6F 72 6B 65 64 20 3B 33
01 5A 01 00 03 6C 6F 6C
E0 01 00 02 3A 29
01 66 03 83 00 44 3F 82 00 54 9B 03
01 CA 01 00 02 67 67
01 4C 03 00 02 67 67
Okay.. So there's a new type of header after the 7-bytes-long ones ?
Will work on it.
It's the function of Alt+g (blink on map).
* Unknown Function, 7 bytes for each, always at header
** 0x0, maybe Frame ?
** unknown
** 0x80, maybe OPcode ?
** 0x0
** 0x0
** unknown
** unknown
* Message Function
** Accumulate Frames, 1-N bytes, Big Endian and Frame/64=Second
** Sender, 1 byte
** OPcode, 1 byte, 0x0 to all, 0x2 to alliance
** Message Len, 1 byte
** Message Content, n bytes from Message Len
* Blink Function (alt+g on Map)
** Accumulate Frames, 1-N bytes
** Player, 1 byte
** OPCode, 1 byte, 0x83
** X, 4 bytes
** Y, 4 bytes
Quote from: newbiz on March 04, 2010, 01:40:02 AM
Thank you very much ! These are very valuable information ^^
Did you figure it out yourself or are you in a parser project ? If not, you can join my effort @ http://projects.coderbasement.com/projects/show/sc2replay (http://projects.coderbasement.com/projects/show/sc2replay)
I'm not in a parser project, just for fun.
I'm also writing these in
Sourcepeek.com.
That's very kind of you :)
Also, do you know if messages & blinks can be interleaved, or blinks are always before ? if so, it would become a real pain to parse.
messages & blinks can be interleaved in any order.
This format is sooo ambiguous to parse...
09 01 02 83 00 32 8E 84 00 97 94 06
How can you know if this is a blink or a message ?
Blink:
byte 0-1 : frames
byte 2 : player
byte 3 : opcode
byte 4-... : X & Y
Message:
byte 0 : frames
byte 1 : player
byte 2 : type (all)
byte 3 : length
byte 4-... : message
Check if second byte smaller than 0x10 (max players) and bigger than 0x1 (min player). And check third byte equal to OPcode(0x0, 0x83).
Dump of the unknown chunk inside replay.info
It seems to be organized in 2 parts, each containing <almost> fixed length structures.
// <- ... map cache name length ...
// <- ... map cache name ...
// Unknown chunk
6E D5 18 ED 00 00 50 F8 DF 07 BB 00 10 00 00 F0
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
D0 FB 3F C3 41 02 00 00 80
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
//
// 0
// 10
// 0 0 0 0 22
// 01 2 01 20 44
// ?? ?? 12 03 83 41 88 00 00 ?? ?? ??
47 00 80 02 81 00 40 00 00 FC 07 C0
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7E 00 10 00 83 00 08 00 00 E0 1F 38
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
F0 21 02 00 03 80 01 00 00 00 FF 07
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80 4F 00 40 00 83 00 20 00 00 F8 07 E0
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7C 10 08 00 03 20 04 00 00 C0 3F 1C
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
E0 43 01 00 03 80 00 80 00 00 FE 07 80
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5F 00 20 00 83 00 10 00 00 F0 0F 70
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
F8 10 04 00 03 40 02 00 00 80 7F 0E
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
C0 47 00 80 02 81 00 40 00 00 FC 07 C0
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7E 00 10 00 83 00 08 00 00 E0 1F 38
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
F0 21 02 00 03 80 01 00 00 00 FF 07
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80 4F 00 40 00 83 00 20 00 00 F8 07 E0
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7C 10 08 00 03 20 04 00 00 C0 3F 1C
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
E0 43 01 00 03 80 00 80 00 00 FE 07 80
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1F 28
// ... "s2ma" structs x5
Anyone still working on this?
yes
Hey guys.
I just entered the beta this week and start to put some interest in your work. I think I might be able to help.
Here's my first contribution:
After staring at this post (http://darkblizz.org/Forum2/starcraft-ii-beta/anyone-working-on-decode-the-replay-file-format/msg5168/#msg5168) for a few minutes, I noticed that every time the "timestamp" field is 2 bytes long, the first byte ends either with 1, 5, 9 or D.
Also, every time this field is 1 byte long, it ends either with 0 or 8.
This is an obvious indication that the field length is actually encoded in the 2 least significant bits of the first byte.
When you read this first byte, you should put a 0b00000011 (0x03) mask on it and you get your "timestamp" length (first byte not included). This should resolve the ambiguity you were talking about, newbiz.
And btw, as you noticed that the "time unit" inside the replay file was 1/64th of a second, it's possible that the 6 remaining bits of the first byte are actually sub-second unit whereas the other byte(s) are full seconds. Because 6 bits is exactly what you need to count from 0 to 63.
Hope this helps (and hope it's clear too cause I'm not a native speaker).
I'll try to see what I can find with the other files. Any progress with them?
chriskang (http://darkblizz.org/Forum2/../../../profile/?u=4375) I wish I'd seen your message before wasting such a shitload of time on figuring timestamps out.
Anyway, if someone is still interested, while I can't claim I figured out the meaning everything in new replay.details file, I at least can parse it. It is very straightforward if you take one thing in consideration: numbers (following 02 bytes, as lengths of strings, and also following 09) are encoded in utf-8 like manner and then shifted one bit right. here's how it goes:Number consists of variable number of bytes. If most significant bit is cleared, then it's the last byte making up a number. Other seven bytes are treated as part of number, most significant bytes coming first. Then the resulting number, made up of groups of 7 bits gets shifted by one bit to the right.
byte sequence resulting number (dec)
=================== ====================
00 0
01 0
02 1
03 1
04 2
7f 63
80 <incorrect by itself>
80 01 64
80 02 128
82 01 65
84 01 66
F2 14 1337 (actually encountered in every replay, once for
each player. Who invited skiddies to blizzard?!)
80 80 8D F1 B0 08 144000000000 (encountered as well, near the end of map.
Number varies but billions are constant)
So, even though it looks like stuff is at different offsets every time, it's only because numbers have different length. Read them properly and everything is very parseable.