-
Notifications
You must be signed in to change notification settings - Fork 30
/
dpkt_notes.html
273 lines (273 loc) · 12.4 KB
/
dpkt_notes.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<title>dpkt notes</title>
</head>
<body>
<h1>Jeff Silverman's dpkt documentation</h1>
This page is a notebook of things I have learned about dpkt.<br>
<br>
<h2>HTTP response bodies</h2>
Status: New
<br>
Owner: ----
<br>
Labels: Type-Defect Priority-Medium
<br>
<br>
New issue 69 by <a class="moz-txt-link-abbreviated"
href="mailto:[email protected]:">[email protected]:</a> is
dpkt able to parse the HTTP response ????
<br>
<a class="moz-txt-link-freetext"
href="http://code.google.com/p/dpkt/issues/detail?id=69">http://code.google.com/p/dpkt/issues/detail?id=69</a>
<br>
<br>
Hi,
<br>
<br>
I am currently using dpkt to parse libpcap format file.
What I want to do is to extract the raw content of the HTTP response.
However, I found my script ONLY works for the HTTP responses that have
small content length (<1500Byte).<br>
Updates:
<br>
Status: WontFix
<br>
<br>
Comment #2 on issue 69 by dugsong: is dpkt able to parse the HTTP
response ????
<br>
<a class="moz-txt-link-freetext"
href="http://code.google.com/p/dpkt/issues/detail?id=69">http://code.google.com/p/dpkt/issues/detail?id=69</a>
<br>
<br>
dpkt doesn't do any TCP stream reassembly. You need to do that
yourself.
<br>
<br>
Here's an example:
<br>
<br>
<a class="moz-txt-link-freetext"
href="http://code.google.com/p/dsniff/source/browse/trunk/dsniff/lib/reasm.py">http://code.google.com/p/dsniff/source/browse/trunk/dsniff/lib/reasm.py</a>
<br>
<br>
If you're doing it live, you need to do your own stream parsing
instead, e.g. dsniff's HTTP stream parser:
<br>
<br>
<a class="moz-txt-link-freetext"
href="http://code.google.com/p/dsniff/source/browse/trunk/dsniff/lib/http.py">http://code.google.com/p/dsniff/source/browse/trunk/dsniff/lib/http.py</a>
<br>
<br>
Good luck!
<br>
<h3>Using dpkt to decode HTTP headers</h3>
Oversimplifying outrageously, you can do something like this:<br>
<br>
<pre> eth = dpkt.ethernet.Ethernet(buf)<br> ip = eth.data<br> tcp = ip.data<br><br># If the destination port is 80 and there is data in the packet, then probably this is an HTTP request. TO be certain, there should<br># be a TCP finite state machine in here.<br> if tcp.dport == 80 and len(tcp.data) > 0:<br> http_req = dpkt.http.Request(tcp.data)<br><br></pre>
<p>The dpkt.http.Request object has the following attributes:<br>
</p>
<pre>['_Request__methods', '_Request__proto', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__hdr_defaults__', '__init__', '__len__', '__metaclass__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'body', 'data', 'headers', 'method', 'pack', 'pack_hdr', 'unpack', 'uri', 'version']<br>(Pdb) print http.method<br>GET<br>(Pdb) print http.body<br><br>(Pdb) print http.headers<br>{'accept-language': 'en-us,en;q=0.5', 'accept-encoding': 'gzip,deflate', 'connection': 'keep-alive', 'keep-alive': '115', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'user-agent': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.16) Gecko/20110323 Ubuntu/10.04 (lucid) Firefox/3.6.16', 'accept-charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'host': 'www.commercialventvac.com'}<br>(Pdb) print http.pack<br><bound method Request.pack of Request(version='1.1')><br>(Pdb) print http.pack()<br>GET / HTTP/1.1<br>accept-language: en-us,en;q=0.5<br>accept-encoding: gzip,deflate<br>connection: keep-alive<br>keep-alive: 115<br>accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<br>user-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.16) Gecko/20110323 Ubuntu/10.04 (lucid) Firefox/3.6.16<br>accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7<br>host: www.commercialventvac.com<br><br><br>(Pdb) print http.pack_hdr()<br>accept-language: en-us,en;q=0.5<br>accept-encoding: gzip,deflate<br>connection: keep-alive<br>keep-alive: 115<br>accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<br>user-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.16) Gecko/20110323 Ubuntu/10.04 (lucid) Firefox/3.6.16<br>accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7<br>host: www.commercialventvac.com<br><br>(Pdb) <br><br></pre>
<h3>dsniff has a tcp decoder, I think.</h3>
Except I can't find it. It is available for downloading from
ubuntu. Find the source code.<br>
<br>
<h2>SCTP Stream Control Transmission Protocol</h2>
Get the source code from <span style="text-decoration: underline;">https://github.com/philpraxis/pysctp.git</span>.
In
order
to
build the C software, I need the package libsctp-dev, which
I installed using synaptic. The installation script puts the file
<span style="font-family: monospace;">_sctp.</span>h in<span
style="font-family: monospace;"> /usr/local</span>. It should go
in <span style="font-family: monospace;">/usr/local/include</span>.
Fixed
that.
pysctp
comes with two SCTP clients, but no SCTP
servers, so it is difficult to test. I sent a message to the
author of the software, <a href="mailto:[email protected]">Philippe
Langlois <Phil __AT__ p1sec.com></a> .<br>
<span style="text-decoration: underline;"><br>
</span><br>
<h2>An analysis of the file http_ipv4.cap</h2>
<p>
I think the decode_tcp_iterator.py file is not doing the sequence
numbers correctly. The ack numbers that are n/a are because the ACK
flag is clear.
</p>
<table>
<tbody>
<tr>
<th>Packet</th>
<th>direction</th>
<th>absolute seq num</th>
<th>relative seq num</th>
<th>absolute ack num</th>
<th>relative ack num</th>
</tr>
<tr>
<td>1</td>
<td>C->S</td>
<td>988184821</td>
<td>0</td>
<td>n/a</td>
<td>n/a</td>
</tr>
<tr>
<td>2</td>
<td>S->C</td>
<td>2171931972</td>
<td>1</td>
<td>988184822</td>
<td>1</td>
</tr>
<tr>
<td>3</td>
<td>C->S</td>
<td>988184822</td>
<td>1</td>
<td>2171931972</td>
<td>1</td>
</tr>
<tr>
<td>4</td>
<td>C->S</td>
<td>988184822</td>
<td>111</td>
<td>2171931972</td>
<td>111</td>
</tr>
<tr>
<td>5</td>
<td>S->C</td>
<td>2171931973</td>
<td>1</td>
<td>2171931973</td>
<td>111</td>
</tr>
<tr>
<td>6</td>
<td>C->S</td>
<td>988184832</td>
<td>111</td>
<td>2171932883</td>
<td>911</td>
</tr>
<tr>
<td>7</td>
<td>S->C</td>
<td>2171932883</td>
<td>911</td>
<td>988184932</td>
<td>2339</td>
</tr>
<tr>
<td>8</td>
<td>C->S</td>
<td> 988184932<br>
</td>
<td>911</td>
<td>2171934311<br>
</td>
<td>2339</td>
</tr>
<tr>
<td style="vertical-align: top;">9<br>
</td>
<td style="vertical-align: top;">S->C<br>
</td>
<td style="vertical-align: top;">2171934311<br>
</td>
<td style="vertical-align: top;">2339<br>
</td>
<td style="vertical-align: top;">988184932<br>
</td>
<td style="vertical-align: top;">3391<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">10<br>
</td>
<td style="vertical-align: top;">C->S<br>
</td>
<td style="vertical-align: top;">988184932<br>
</td>
<td style="vertical-align: top;">3391<br>
</td>
<td style="vertical-align: top;">2171935363<br>
</td>
<td style="vertical-align: top;">111<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">11<br>
</td>
<td style="vertical-align: top;">S->C (fin)<br>
</td>
<td style="vertical-align: top;">2171935363<br>
</td>
<td style="vertical-align: top;">111<br>
</td>
<td style="vertical-align: top;">988184932<br>
</td>
<td style="vertical-align: top;">3392<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">12<br>
</td>
<td style="vertical-align: top;">C->S (fin)<br>
</td>
<td style="vertical-align: top;">988184932<br>
</td>
<td style="vertical-align: top;">3392<br>
</td>
<td style="vertical-align: top;">2171935364<br>
</td>
<td style="vertical-align: top;">112<br>
</td>
</tr>
<tr>
<td style="vertical-align: top;">13<br>
</td>
<td style="vertical-align: top;">S->C<br>
</td>
<td style="vertical-align: top;">2171935364<br>
</td>
<td style="vertical-align: top;">3392<br>
</td>
<td style="vertical-align: top;">988184933<br>
</td>
<td style="vertical-align: top;">112<br>
</td>
</tr>
</tbody>
</table>
<h2>Troubles decoding IPv6</h2>
I am having troubles getting a capture file that contains IPv6
packets. I think that the files from tcpdump have a disagreement
with libpcap about something:<br>
<br>
<pre>jeffs@heavy:~/python/PythonClass/dpkt_doc$ <kbd>python -m pdb decode_tcp_iterator_2.py http_ipv6.cap</kbd> <br>> /home/jeffs/python/PythonClass/dpkt_doc/decode_tcp_iterator_2.py(5)<module>()<br>-> import dpkt<br>(Pdb) c<br>Traceback (most recent call last):<br> File "/usr/lib/python2.6/pdb.py", line 1285, in main<br> pdb._runscript(mainpyfile)<br> File "/usr/lib/python2.6/pdb.py", line 1204, in _runscript<br> self.run(statement)<br> File "/usr/lib/python2.6/bdb.py", line 368, in run<br> exec cmd in globals, locals<br> File "<string>", line 1, in <module><br> File "decode_tcp_iterator_2.py", line 159, in <module><br> pc = dpkt.pcap.Reader ( open(sys.argv[1] ) ) # file interface<br> File "dpkt/pcap.py", line 105, in __init__<br> self.dloff = dltoff[self.__fh.linktype]<br>KeyError: 101<br>Uncaught exception. Entering post mortem debugging<br>Running 'cont' or 'step' will restart the program<br>> /home/jeffs/python/PythonClass/dpkt_doc/dpkt/pcap.py(105)__init__()<br>-> self.dloff = dltoff[self.__fh.linktype]<br>(Pdb) <kbd>print self.__fh.linktype</kbd><br>*** AttributeError: 'Reader' object has no attribute '__fh'<br>(Pdb) print self.__fh.linktype<br>*** AttributeError: 'Reader' object has no attribute '__fh'<br>(Pdb) <br><br></pre>
If I use a capture file from wireshark, it gets past this point, but
then it has a bad value for the ethertype:<br>
<br>
<pre>jeffs@heavy:~/python/PythonClass/dpkt_doc$ python -m pdb decode_tcp_iterator_2.py http_ipv6_wireshark_2.cap <br>> /home/jeffs/python/PythonClass/dpkt_doc/decode_tcp_iterator_2.py(5)<module>()<br>-> import dpkt<br>(Pdb) list 72<br> 67 eth = dpkt.ethernet.Ethernet(buf)<br> 68 # Also, this changes a little bit with IPv6. To tell the difference between IPv4 and IPv6, you have to look<br> 69 # at the ethertype field, which is given by http://www.iana.org/assignments/ethernet-numbers. IPv4 is 0x800 or 2048<br> 70 # and IPv6 is 0x86DD or 34525<br> 71 # This is simplistic - IPv4 packets can be fragmented. Also, this only works for IPv4. IPv6 has a different Ethertype <br> 72 if eth.type == dpkt.ethernet.ETH_TYPE_IP :<br> 73 ip = eth.data<br> 74 if ip.v != 4 :<br> 75 raise ValueError, "In packet %d, the ether type is IPv4 but the IP version number is %d not 4" % (<br> 76 packet_cntr, ip.v )<br> 77 # Deal with IP fragmentation here<br>(Pdb) break 72<br>Breakpoint 1 at /home/jeffs/python/PythonClass/dpkt_doc/decode_tcp_iterator_2.py:72<br>(Pdb) c<br>counter src prt dst prt flags<br>> /home/jeffs/python/PythonClass/dpkt_doc/decode_tcp_iterator_2.py(72)decode_tcp()<br>-> if eth.type == dpkt.ethernet.ETH_TYPE_IP :<br>(Pdb) print eth.type<br>129<br>(Pdb) print dpkt.ethernet.ETH_TYPE_IP<br>2048<br>(Pdb) print dpkt.ethernet.ETH_TYPE_IP6<br>34525<br>(Pdb) <br><br></pre>
<p>I think this is a bug in wireshark, because in the packet decoding
process, it says that there is no information available for the link
layer. Wireshark recognizes that the packet is IPv6. 129 is
in the range of the IEEE802.3 length field, so the problem may be that
this is an IEEE802.3 packet and not an Ethernet packet. I am also
doing the packet capture from the sixxs device.<br>
<br>
</p>
<p>
</p>
</body>
</html>