Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libsmdev_handle_read_buffer: offset exceeds media size. #3

Open
devgc opened this issue Nov 20, 2018 · 3 comments
Open

libsmdev_handle_read_buffer: offset exceeds media size. #3

devgc opened this issue Nov 20, 2018 · 3 comments
Assignees
@joachimmetz
Labels
needs a closer look

Comments

@devgc
Copy link

devgc commented Nov 20, 2018

I think this is the right repository for this issue. I was having an issue in DFVFS where volume shadows are not being opened for some logical volumes and think I have narrowed it down to this library. Here is the exact pysmdev error as shown in the following example (Python 2.7 and 3.6 both produce it):

pyvshadow_volume_open_file_object: unable to open volume. pyvshadow_file_object_read_buffer: unable to read from file object with error: 'pysmdev_handle_read_buffer: unable to read data. libsmdev_handle_read_buffer: offset exceeds media size.'.
 pyvshadow_file_object_io_handle_read: unable to read from file object.
 libbfio_handle_read_buffer: unable to read from handle.
 libvshadow_io_handle_read_ntfs_volume_header: unable to read NTFS backup volume header data.
 libvshadow_volume_open_read: unable to read NTFS volume header.
 libvshadow_volume_open_file_io_handle: unable to read from file IO handle.

with libsmdev_handle_read_buffer: offset exceeds media size. being the issue.

This is the same error I am seeing when trying the open the VShadow object in dfvfs.

I understand this will be difficult to reproduce because it is specific to my logical volume and I have not found a test image where the resulting error is reproduced. Maybe you can formulate a hypothesis by seeing what I did, and what the output was?

Whats weird is it works on my D: but not C:.
C: is on a SSD with bitlocker turned on.
D: is on a HD with bitlocker turned on.
More fs and drive info can be found at the bottom of this comment.

Below is the code I used to produce the error followed by the results of it being ran on my system:

import pysmdev
import pyvshadow


def main():
    ######################################################
    # using normal file handle
    ######################################################
    print("Test 1: VShadow from C as file handle")
    with open('\\\\.\\C:', "rb") as volume_handle:
        vshadow_volume = pyvshadow.volume()
        try:
            vshadow_volume.open_file_object(
                volume_handle
            )
            print("VSS Found for C: Store count -> {}".format(
                vshadow_volume.get_number_of_stores()
            ))
        except Exception as error:
            error_str = str(error).replace("\r", "\n")
            print("Error Opening VShadow using <file> on [C:]")
            print(error_str)
    print("\n")

    print("Test 2: VShadow from D as file handle")
    # As regular file.
    with open('\\\\.\\D:', "rb") as volume_handle:
        vshadow_volume = pyvshadow.volume()
        try:
            vshadow_volume.open_file_object(
                volume_handle
            )
            print("VSS Found for D: Store count -> {}".format(
                vshadow_volume.get_number_of_stores()
            ))
        except Exception as error:
            error_str = str(error).replace("\r", "\n")
            print("Error Opening VShadow using <file> on [D:]")
            print(error_str)
    print("\n")

    ######################################################
    # using pysmdev handle
    ######################################################
    print("Test 3: VShadow from C as pymsdev.handle")
    smdev_handle = pysmdev.handle()
    smdev_handle.open("\\\\.\\C:")

    vshadow_volume = pyvshadow.volume()
    try:
        vshadow_volume.open_file_object(
            smdev_handle
        )
        print("VSS Found for C: Store count -> {}".format(
            vshadow_volume.get_number_of_stores()
        ))
    except Exception as error:
        error_str = str(error).replace("\r", "\n")
        print("Error Opening VShadow using <pysmdev.handle> on [C:]")
        print(error_str)
    print("\n")

    print("Test 4: VShadow from D as pymsdev.handle")
    smdev_handle = pysmdev.handle()
    smdev_handle.open("\\\\.\\D:")

    vshadow_volume = pyvshadow.volume()
    try:
        vshadow_volume.open_file_object(
            smdev_handle
        )
        print("VSS Found for D: Store count -> {}".format(
            vshadow_volume.get_number_of_stores()
        ))
    except Exception as error:
        error_str = str(error).replace("\r", "\n")
        print("Error Opening VShadow using <pysmdev.handle> on [D:]")
        print(error_str)
    print("\n")


if __name__ == "__main__":
    main()
Test 1: VShadow from C as file handle
VSS Found for C: Store count -> 11


Test 2: VShadow from D as file handle
VSS Found for D: Store count -> 11


Test 3: VShadow from C as pymsdev.handle
Error Opening VShadow using <pysmdev.handle> on [C:]
pyvshadow_volume_open_file_object: unable to open volume. pyvshadow_file_object_read_buffer: unable to read from file object with error: 'pysmdev_handle_read_buffer: unable to read data. libsmdev_handle_read_buffer: offset exceeds media size.'.
 pyvshadow_file_object_io_handle_read: unable to read from file object.
 libbfio_handle_read_buffer: unable to read from handle.
 libvshadow_io_handle_read_ntfs_volume_header: unable to read NTFS backup volume header data.
 libvshadow_volume_open_read: unable to read NTFS volume header.
 libvshadow_volume_open_file_io_handle: unable to read from file IO handle.


Test 4: VShadow from D as pymsdev.handle
VSS Found for D: Store count -> 11

Python 2.6 x64 - Library Versions:

>>> import pysmdev
>>> pysmdev.get_version()
u'20171112'
>>> import pyvshadow
>>> pyvshadow.get_version()
u'20170902'

Python 3.6 x64 - Library Versions:

>>> import pysmdev
>>> pysmdev.get_version()
'20171112'
>>> import pyvshadow
>>> pyvshadow.get_version()
'20170902'

Not sure if the following helps or not. But I thought I would include it just in case.

C: Volume info:

NTFS Version   :                   3.1
LFS Version    :                   2.0
Number Sectors :                   0x0000000037c60f44
Total Clusters :                   0x0000000006f8c1e8
Free Clusters  :                   0x00000000050b7fa9
Total Reserved :                   0x000000000000233f
Bytes Per Sector  :                512
Bytes Per Physical Sector :        512
Bytes Per Cluster :                4096
Bytes Per FileRecord Segment    :  1024
Clusters Per FileRecord Segment :  0
Mft Valid Data Length :            0x000000005b140000
Mft Start Lcn  :                   0x00000000000c0000
Mft2 Start Lcn :                   0x0000000000000002
Mft Zone Start :                   0x0000000003729ce0
Mft Zone End   :                   0x0000000003736500
Max Device Trim Extent Count :     0
Max Device Trim Byte Count :       0x0
Max Volume Trim Extent Count :     62
Max Volume Trim Byte Count :       0x40000000

---DRIVE INFO---
PSComputerName              : DESKTOP
ConfigManagerErrorCode      : 0
LastErrorCode               :
NeedsCleaning               :
Status                      : OK
DeviceID                    : \\.\PHYSICALDRIVE0
StatusInfo                  :
Partitions                  : 3
BytesPerSector              : 512
ConfigManagerUserConfig     : False
DefaultBlockSize            :
Index                       : 0
InstallDate                 :
InterfaceType               : SCSI
MaxBlockSize                :
MaxMediaSize                :
MinBlockSize                :
NumberOfMediaSupported      :
SectorsPerTrack             : 63
Size                        : 480101368320
TotalCylinders              : 58369
TotalHeads                  : 255
TotalSectors                : 937697985
TotalTracks                 : 14884095
TracksPerCylinder           : 255
__GENUS                     : 2
__CLASS                     : Win32_DiskDrive
__SUPERCLASS                : CIM_DiskDrive
__DYNASTY                   : CIM_ManagedSystemElement
__RELPATH                   : Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE0"
__PROPERTY_COUNT            : 51
__DERIVATION                : {CIM_DiskDrive, CIM_MediaAccessDevice, CIM_LogicalDevice, CIM_LogicalElement...}
__SERVER                    : DESKTOP
__NAMESPACE                 : root\cimv2
__PATH                      : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE0"
Availability                :
Capabilities                : {3, 4}
CapabilityDescriptions      : {Random Access, Supports Writing}
Caption                     : ATA INTEL SSDSC2BP48 SCSI Disk Device
CompressionMethod           :
CreationClassName           : Win32_DiskDrive
Description                 : Disk drive
ErrorCleared                :
ErrorDescription            :
ErrorMethodology            :
FirmwareRevision            : 0420
Manufacturer                : (Standard disk drives)
MediaLoaded                 : True
MediaType                   : Fixed hard disk media
Model                       : ATA INTEL SSDSC2BP48 SCSI Disk Device
Name                        : \\.\PHYSICALDRIVE0
PNPDeviceID                 : SCSI\DISK&VEN_ATA&PROD_INTEL_SSDSC2BP48\5&ADA8C50&0&000000
PowerManagementCapabilities :
PowerManagementSupported    :
SCSIBus                     : 0
SCSILogicalUnit             : 0
SCSIPort                    : 0
SCSITargetId                : 0
SerialNumber                : x
Signature                   : 809290414
SystemCreationClassName     : Win32_ComputerSystem
SystemName                  : DESKTOP
Scope                       : System.Management.ManagementScope
Path                        : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE0"
Options                     : System.Management.ObjectGetOptions
ClassPath                   : \\DESKTOP\root\cimv2:Win32_DiskDrive
Properties                  : {Availability, BytesPerSector, Capabilities, CapabilityDescriptions...}
SystemProperties            : {__GENUS, __CLASS, __SUPERCLASS, __DYNASTY...}
Qualifiers                  : {dynamic, Locale, provider, UUID}
Site                        :
Container                   :

D: Volume info:

NTFS Volume Serial Number :        0xaad2d584d2d55563
NTFS Version   :                   3.1
LFS Version    :                   2.0
Number Sectors :                   0x00000000e8e077ff
Total Clusters :                   0x000000001d1c0eff
Free Clusters  :                   0x000000001a528d1d
Total Reserved :                   0x00000000000013ef
Bytes Per Sector  :                512
Bytes Per Physical Sector :        512
Bytes Per Cluster :                4096
Bytes Per FileRecord Segment    :  1024
Clusters Per FileRecord Segment :  0
Mft Valid Data Length :            0x0000000008d80000
Mft Start Lcn  :                   0x00000000000c0000
Mft2 Start Lcn :                   0x0000000000000002
Mft Zone Start :                   0x00000000000c8d80
Mft Zone End   :                   0x00000000000cc840
Max Device Trim Extent Count :     0
Max Device Trim Byte Count :       0x0
Max Volume Trim Extent Count :     62
Max Volume Trim Byte Count :       0x40000000

---DRIVE INFO---
PSComputerName              : DESKTOP
ConfigManagerErrorCode      : 0
LastErrorCode               :
NeedsCleaning               :
Status                      : OK
DeviceID                    : \\.\PHYSICALDRIVE1
StatusInfo                  :
Partitions                  : 1
BytesPerSector              : 512
ConfigManagerUserConfig     : False
DefaultBlockSize            :
Index                       : 1
InstallDate                 :
InterfaceType               : SCSI
MaxBlockSize                :
MaxMediaSize                :
MinBlockSize                :
NumberOfMediaSupported      :
SectorsPerTrack             : 63
Size                        : 2000396321280
TotalCylinders              : 243201
TotalHeads                  : 255
TotalSectors                : 3907024065
TotalTracks                 : 62016255
TracksPerCylinder           : 255
__GENUS                     : 2
__CLASS                     : Win32_DiskDrive
__SUPERCLASS                : CIM_DiskDrive
__DYNASTY                   : CIM_ManagedSystemElement
__RELPATH                   : Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE1"
__PROPERTY_COUNT            : 51
__DERIVATION                : {CIM_DiskDrive, CIM_MediaAccessDevice, CIM_LogicalDevice, CIM_LogicalElement...}
__SERVER                    : DESKTOP
__NAMESPACE                 : root\cimv2
__PATH                      : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE1"
Availability                :
Capabilities                : {3, 4}
CapabilityDescriptions      : {Random Access, Supports Writing}
Caption                     : ATA WDC WD2003FZEX-0 SCSI Disk Device
CompressionMethod           :
CreationClassName           : Win32_DiskDrive
Description                 : Disk drive
ErrorCleared                :
ErrorDescription            :
ErrorMethodology            :
FirmwareRevision            : 1A01
Manufacturer                : (Standard disk drives)
MediaLoaded                 : True
MediaType                   : Fixed hard disk media
Model                       : ATA WDC WD2003FZEX-0 SCSI Disk Device
Name                        : \\.\PHYSICALDRIVE1
PNPDeviceID                 : SCSI\DISK&VEN_ATA&PROD_WDC_WD2003FZEX-0\5&ADA8C50&0&000100
PowerManagementCapabilities :
PowerManagementSupported    :
SCSIBus                     : 0
SCSILogicalUnit             : 0
SCSIPort                    : 0
SCSITargetId                : 1
SerialNumber                :      xxxxxxxxxxxxxxxx
Signature                   : 1619285509
SystemCreationClassName     : Win32_ComputerSystem
SystemName                  : DESKTOP
Scope                       : System.Management.ManagementScope
Path                        : \\DESKTOP\root\cimv2:Win32_DiskDrive.DeviceID="\\\\.\\PHYSICALDRIVE1"
Options                     : System.Management.ObjectGetOptions
ClassPath                   : \\DESKTOP\root\cimv2:Win32_DiskDrive
Properties                  : {Availability, BytesPerSector, Capabilities, CapabilityDescriptions...}
SystemProperties            : {__GENUS, __CLASS, __SUPERCLASS, __DYNASTY...}
Qualifiers                  : {dynamic, Locale, provider, UUID}
Site                        :
Container                   👍
@joachimmetz joachimmetz self-assigned this Nov 20, 2018
@joachimmetz
Copy link
Member

Could this be related to: libyal/libcfile#2

is the offset of the NTFS backup header pointing beyond the volume device size reported by pysmdev?

Does the same happen when reading the full disk with pysmdev?

@joachimmetz
Copy link
Member

I'll have a closer look as soon as time permits.

@devgc
Copy link
Author

devgc commented Nov 20, 2018 •
edited
Loading

I created an image of the logical volume C: and I was able to open the volume shadow object. Maybe this gives more support for this issue having to do with libyal/libcfile#2?

Otherwise, here are some more offsets. The backup header is within the size that the smdev handle sees (smdev_handle.media_size).

Here are the offsets of my C:

Partition Size:          479092181504
smdev.handle.media_size: 479092181504
File System Size:        479092178944
Backup Header location:  479092180992 [479092178944 + 2048]

Here are the offsets of my D:

Partition Size:          2000396746752
smdev.handle.media_size: 2000396746752
File System Size:        2000396742656
Backup Header:           2000396746240 [2000396742656 + 3584]

In either case, the backup volume is the last block in partition.

I tried reading the last block using pysmdev but had no success.

>>> smdev_handle = pysmdev.handle()
>>> smdev_handle.open("\\\\.\\C:")
>>> backup_location = smdev_handle.media_size - 512
>>> print("backup_location: " + str(backup_location))
backup_location: 479092180992
>>> smdev_handle.seek_offset(backup_location)
>>> backup_raw = smdev_handle.read(512)
>>> print(len(backup_raw))
0
>>> smdev_handle = pysmdev.handle()
>>> smdev_handle.open("\\\\.\\D:")
>>> backup_location = smdev_handle.media_size - 512
>>> print("backup_location: " + str(backup_location))
backup_location: 2000396746240
>>> smdev_handle.seek(backup_location)
>>> backup_raw = smdev_handle.read(512)
>>> print(len(backup_raw))
0

I cant seem to read outside the file system size with pysmdev. But, this cant be the issue though because I am still able to open the vshadow object for D: and its backup is outside the file system size too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joachimmetz joachimmetz

Labels
needs a closer look
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joachimmetz @devgc