Sendrecv overhaul: async sniffing & major cleanup

[gpotter2]

This PR:

• ✔️ REQUIRES Fix FlagsField & dissection improvements #2035
• contains an alternative fix for packet.time_sent can't be set because of object copy during list comprehension #1199, that binds the sent_time in __iter__ rather than requiring extra memory space to store the timestamps. This brings a small performance boost, see below.
• Implements AsyncSniff, removes tons of duplications, and allow us to use Sessions everywhere
  • sniff uses AsyncSniff
  • sndrcv uses AsyncSniff
  • sndrcvflood uses sndrcv.
• use classes to handle sniffing and sndrcv: they allow greater flexibility when sharing properties than the previous method.
• drops Event usage: not required anymore if we use a class.
• removes process from sndrcv: no one uses it, and it's extremly messy. (My bad). We could reimplement it much easily now that we use AsyncSniffer within sndrcv
• fixes heavy memory leaks in scapy/automaton.py
• Add docs about AsyncSniffer
• ❕ Change of behavior: packets that are None are ignored on all platforms (previously only Windows/BPF), instead of closing the socket (and using the very unclear TimeoutElapsed). EOFError will now be the default exception to close a socket (this behavior is only used in *PcapReader)
• fixed BSD bug: conf.use_bpf would always enable itself even if conf.use_pcap was True. (Missing parenthesis)
• migrates scapy.arch.bpf.supersocket sockets to use the recv_raw mechanic

It's a bit harder to implement than it looks, because it must be able to get out of a frozen select(), on sockets that don't get any more packets.

Perfs:

Using the benchmarking of #1259

Note: the following stats should only be read relatively speaking. The test is performed using a short packet (Ether()/IP()) on loopback. It isn't intended to match a real usage by any mean.

	secdev/master	gpotter2/perfsndrcv
Python 2.7	~ 5700/s	~ 5900/s
Python 3.7	~ 6800/s	~ 7300/s

Notes:

async_select_unrequired=False means that the select function is blocking, therefore a control socket (fake) should be passed. It should be set to True otherwise (Windows/OSX/Pcapdnet)... We could potentially rename it is_blocking_socket=False if it's clearer (but I only thought of that later 😄)

Hopes for the future:

Thanks to the use of *sniff within the sndrcv suite, we have less code to maintain + it will allow us to enable:

• Sessions in sndrcv*: match a chunked HTTP response to its answer ?
• multi-thread dissection (I have great hopes for this. I plan to use multiprocessing.dummy and try to paralellize packet dissections in pools). Edit: to anyone still reading this, note AsyncSniffer does NOT improve performances (or barely). Splitting the work load in multiple threads is useless. We could try to split it across processes, but that makes it much harder to implement. Eventually, the best option to optimize your code is to disable dissection of packets you don't use. We should make an util to make that easier someday

Related:

fixes #1523
fixes #1505
fixes #1937

[codecov]

Codecov Report

Merging #1999 into master will increase coverage by 0.27%.
The diff coverage is 86.05%.

@@            Coverage Diff             @@
##           master    #1999      +/-   ##
==========================================
+ Coverage   86.86%   87.14%   +0.27%     
==========================================
  Files         197      197              
  Lines       44505    44512       +7     
==========================================
+ Hits        38661    38791     +130     
+ Misses       5844     5721     -123

Impacted Files	Coverage Δ
scapy/contrib/isotp.py	88.32% <100%> (+0.01%)	⬆️
scapy/contrib/cansocket_native.py	80.55% <100%> (+0.27%)	⬆️
scapy/packet.py	79.67% <100%> (+0.18%)	⬆️
scapy/layers/l2.py	84.23% <100%> (+0.03%)	⬆️
scapy/arch/common.py	89.65% <100%> (-0.57%)	⬇️
scapy/contrib/cansocket_python_can.py	91.93% <100%> (+0.13%)	⬆️
scapy/arch/windows/native.py	79.24% <100%> (+0.39%)	⬆️
scapy/automaton.py	85.44% <100%> (+3.34%)	⬆️
scapy/scapypipes.py	86.61% <100%> (-0.11%)	⬇️
scapy/arch/pcapdnet.py	69.65% <100%> (-1%)	⬇️
... and 22 more

[gpotter2]

PR is currently failing with PCAP/DNET tests. I'm not sure why yet

[gpotter2]

So I've been working on asynchronous sniffing for a while now, PR isn't finished (fails on OSX), but at least it passes on Windows and Linux (I have access to debug machines)

@polybassa I'm pretty sure this could be used in the isotp implementation. However, I'm having troubles getting it to work with ISOTP sockets: it tend to fail a bit randomly https://travis-ci.com/secdev/scapy/jobs/203122929

I can usually restart the tests to make them pass.

If you could have a look it would be amazing, but the PR is quite hard to understand, and I don't want to bother too much, so it's up to you.

[polybassa]

Hi, I’ll check that PR. Unfortunately I don’t have time for it, today. But tomorrow I’ll look into it.

…

On 26. May 2019, at 16:02, Gabriel ***@***.***> wrote: So I've been working on asynchronous sniffing for a while now, PR isn't finished (fails on OSX/pcapdnet), but at least it passes on Windows and Linux (I have access to debug machines) @polybassa I'm pretty sure this could be used in the isotp implementation. However, I'm having troubles getting it to work with ISOTP sockets: https://travis-ci.com/secdev/scapy/jobs/203122929 If you could have a look it would be amazing, but the PR is quite hard to understand, so it's up to you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[polybassa]

Hi, I've looked into your commit and tested it manually. None of my unit tests for ISOTP fails at my local machine. AsyncSniffer works fine with ISOTPNativeSocket and ISTOPSoftSocket if the socket is provided as parameter opened_socket. Do you have any specific place, where ISOTPSocket fail?

[gpotter2]

Thanks for the feedback I've actually updated the StreamSocket a bit since the commit. It might have been related to that.

It seems to have been fixed, sorry for wasting your time. I'll see if it remains consistent

[polybassa]

No problem. I'm happy to help you.

[gpotter2]

I managed to circle the issue that happened only on the OSX builds. It's related to a timeout after sending, relative to BPF. The issue shouldn't be too hard to fix, I setup a Vagrant box running Freebsd for the occasion

Last issue on the table: ISOTP failures. I've disabled the tests for now.
To be honest, there are some parts of isotp.py I don't understand. I have the feeling a lot of the code there related to sockets management could be reworked (Even though it was great work), using either this code or scapy.automaton (probably for the rx_callbacks). It's very ugly in some parts (parts that could probably be reimplemented via AsyncSniffer)

[gpotter2]

@polybassa I finally managed to fix BSD*. The last commit fails because of isotp.uts tests (see the Travis build: https://travis-ci.com/secdev/scapy/builds/114015453) :/

The PR nearly is in its final state, the only thing left is to fix that. It works all right if I disable those.
I'll try to figure it out, but you're much more familiar with this part than I am. I would be very grateful if you could have a (another?) look.

Also, note that this PR will probably allow to rewrite CANReceiverThread and ISOTPSniffer, in a prettier way.

[polybassa]

@gpotter2
The tests that are failing, are tests with a mocked socket. This means, the problem is not related to vcan or to CANSockets. Either the implementation of the MockSocket has a bug or the implementation of ISOTPSoftSocket is the problem. Maybe @epozzobon can help us here.

[gpotter2]

@polybassa You were very right ! Thanks a lot, I had missed the CANMockSocket 😄

I've updated it using the new mechanic in be21a50, and it appears to have solved the issue !

[gpotter2]

@guedou @p-l- This PR is ready to be reviewed.
It's quite big, and it did expand a bit more than I expected. I however managed to split the changes in different commits, so following them individually should do.

It's still a pretty cool feature

[guedou]

I need to take more time to review the changes in sendrecv.py but I like the idea of refactoring the code!

[p-l-]

This is brilliant. Thanks (and congrats) @gpotter2!

[porceCodes]

Document:
https://scapy.readthedocs.io/en/latest/usage.html#asynchronous-sniffing