SNMP walk using getBulk timeouts sometime (randomly)

[dirttech]

I am using below example for pysnmp. Sometimes it works well but sometimes it stops midway and gives timeout error.

from pysnmp.carrier.asyncore.dispatch import AsyncoreDispatcher
from pysnmp.carrier.asyncore.dgram import udp
from pyasn1.codec.ber import encoder, decoder
from pysnmp.proto import api
from time import time

# Protocol version to use
pMod = api.protoModules[api.protoVersion1]
# pMod = api.protoModules[api.protoVersion2c]

# SNMP table header
headVars = [pMod.ObjectIdentifier((1, 3, 6))]

# Build PDU
reqPDU = pMod.GetNextRequestPDU()
pMod.apiPDU.setDefaults(reqPDU)
pMod.apiPDU.setVarBinds(reqPDU, [(x, pMod.null) for x in headVars])

# Build message
reqMsg = pMod.Message()
pMod.apiMessage.setDefaults(reqMsg)
pMod.apiMessage.setCommunity(reqMsg, 'public')
pMod.apiMessage.setPDU(reqMsg, reqPDU)

startedAt = time()


def cbTimerFun(timeNow):
    if timeNow - startedAt > 3:
        raise Exception("Request timed out")


# noinspection PyUnusedLocal
def cbRecvFun(transportDispatcher, transportDomain, transportAddress,
              wholeMsg, reqPDU=reqPDU, headVars=headVars):
    while wholeMsg:
        rspMsg, wholeMsg = decoder.decode(wholeMsg, asn1Spec=pMod.Message())
        rspPDU = pMod.apiMessage.getPDU(rspMsg)
        # Match response to request
        if pMod.apiPDU.getRequestID(reqPDU) == pMod.apiPDU.getRequestID(rspPDU):
            # Check for SNMP errors reported
            errorStatus = pMod.apiPDU.getErrorStatus(rspPDU)
            if errorStatus and errorStatus != 2:
                raise Exception(errorStatus)
            # Format var-binds table
            varBindTable = pMod.apiPDU.getVarBindTable(reqPDU, rspPDU)
            # Report SNMP table
            for tableRow in varBindTable:
                for name, val in tableRow:
                    print('from: %s, %s = %s' % (
                        transportAddress, name.prettyPrint(), val.prettyPrint()
                    )
                          )
            # Stop on EOM
            for oid, val in varBindTable[-1]:
                if not isinstance(val, pMod.Null):
                    break
            else:
                transportDispatcher.jobFinished(1)

            # Generate request for next row
            pMod.apiPDU.setVarBinds(
                reqPDU, [(x, pMod.null) for x, y in varBindTable[-1]]
            )
            pMod.apiPDU.setRequestID(reqPDU, pMod.getNextRequestID())
            transportDispatcher.sendMessage(
                encoder.encode(reqMsg), transportDomain, transportAddress
            )
            global startedAt
            if time() - startedAt > 3:
                raise Exception('Request timed out')
            startedAt = time()
    return wholeMsg


transportDispatcher = AsyncoreDispatcher()

transportDispatcher.registerRecvCbFun(cbRecvFun)
transportDispatcher.registerTimerCbFun(cbTimerFun)

transportDispatcher.registerTransport(
    udp.domainName, udp.UdpSocketTransport().openClientMode()
)
transportDispatcher.sendMessage(
    encoder.encode(reqMsg), udp.domainName, ('10.172.3.10', 161)
)
transportDispatcher.jobStarted(1)

transportDispatcher.runDispatcher()

transportDispatcher.closeDispatcher()

I also debugged the code. I am copying some of it below -

.2018-06-05 15:44:36,948 pysnmp: sendMessage: outgoingMessage queued (67 octets)
00000: 30 41 02 01 01 04 09 4E 61 67 61 72 72 6F 52 4F
00016: A5 31 02 03 4F 9D 29 02 01 00 02 01 19 30 24 30
00032: 22 06 1E 2B 06 01 04 01 F3 67 02 02 01 04 01 02
00048: 01 04 81 00 00 0B 81 2A 5A 81 16 0A 81 2F 81 2E
00064: 05 05 00
2018-06-05 15:44:36,952 pysnmp: handle_write: transportAddress ('0.0.0.0', 58746) -> ('10.172.3.10', 161) outgoingMessage (67 octets)
00000: 30 41 02 01 01 04 09 4E 61 67 61 72 72 6F 52 4F
00016: A5 31 02 03 4F 9D 29 02 01 00 02 01 19 30 24 30
00032: 22 06 1E 2B 06 01 04 01 F3 67 02 02 01 04 01 02
00048: 01 04 81 00 00 0B 81 2A 5A 81 16 0A 81 2F 81 2E
00064: 05 05 00
Took  3.0183279514312744  seconds.
[<ObjectIdentifier value object at 0x513bd90 tagSet <TagSet object at 0x513bad0 tags 0:0:6> payload [1.3.6.1.4.1.1482...150.10.175.174.5]>] <<<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>> handling Exception Request timed out

As you can see there is no response and hence the timeout error. This is very random. Sometimes it comes in first couple of seconds sometimes there is no error. and sometime it comes midway.

[etingof]

Well, it would not surprise me if some SNMP packets get dropped by the network (it's UDP) so that without retrying your SNMP request has no chances to succeed.

The script you pasted does not seem to have the retrying functionality. You could probably add another .sendMessage() call into your cbTimeFun to do some retries before timing out hard. That would probably require you to maintain the state of the pending requests (including the request message) so that you could re-send it on timeout.

Alternatively, you could just use the high-level pysnmp API which has all the retrying mechanisms in place.