Skip to content

Commit

Permalink
Added fsfat back-end #580 (#672)
Browse files Browse the repository at this point in the history
  • Loading branch information
joachimmetz authored Aug 23, 2022
1 parent bd8331f commit 6c6ce51
Show file tree
Hide file tree
Showing 35 changed files with 1,538 additions and 40 deletions.
8 changes: 4 additions & 4 deletions .github/workflows/test_docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: test_docker
on: [push]
jobs:
test_fedora:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
strategy:
matrix:
version: ['36']
Expand All @@ -17,7 +17,7 @@ jobs:
- name: Install dependencies
run: |
dnf copr -y enable @gift/dev
dnf install -y @development-tools python3 python3-devel libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi python3-cryptography python3-dfdatetime python3-dtfabric python3-idna python3-mock python3-pbr python3-pytsk3 python3-pyxattr python3-pyyaml python3-setuptools python3-six
dnf install -y @development-tools python3 python3-devel libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfsfat-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi python3-cryptography python3-dfdatetime python3-dtfabric python3-idna python3-mock python3-pbr python3-pytsk3 python3-pyxattr python3-pyyaml python3-setuptools python3-six
- name: Run tests
env:
LANG: C.utf8
Expand All @@ -37,7 +37,7 @@ jobs:
python3 ./setup.py build
python3 ./setup.py install
test_ubuntu:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
strategy:
matrix:
version: ['22.04']
Expand All @@ -57,7 +57,7 @@ jobs:
run: |
add-apt-repository -y ppa:gift/dev
apt-get update -q
apt-get install -y build-essential python3 python3-dev libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi-backend python3-cryptography python3-dfdatetime python3-distutils python3-dtfabric python3-idna python3-mock python3-pbr python3-pytsk3 python3-pyxattr python3-setuptools python3-six python3-yaml
apt-get install -y build-essential python3 python3-dev libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfsfat-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi-backend python3-cryptography python3-dfdatetime python3-distutils python3-dtfabric python3-idna python3-mock python3-pbr python3-pytsk3 python3-pyxattr python3-setuptools python3-six python3-yaml
- name: Run tests
env:
LANG: en_US.UTF-8
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/test_docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ on:
- main
jobs:
build:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
strategy:
matrix:
include:
- python-version: '3.8'
toxenv: 'docs'
container:
image: ubuntu:20.04
image: ubuntu:22.04
steps:
- uses: actions/checkout@v2
- name: Set up container
Expand All @@ -35,7 +35,7 @@ jobs:
add-apt-repository -y ppa:deadsnakes/ppa
add-apt-repository -y ppa:gift/dev
apt-get update -q
apt-get install -y build-essential git libffi-dev python${{ matrix.python-version }} python${{ matrix.python-version }}-dev python${{ matrix.python-version }}-venv libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi-backend python3-cryptography python3-dfdatetime python3-distutils python3-dtfabric python3-idna python3-mock python3-pbr python3-pip python3-pytsk3 python3-pyxattr python3-setuptools python3-six python3-yaml
apt-get install -y build-essential git libffi-dev python${{ matrix.python-version }} python${{ matrix.python-version }}-dev python${{ matrix.python-version }}-venv libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfsfat-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi-backend python3-cryptography python3-dfdatetime python3-distutils python3-dtfabric python3-idna python3-mock python3-pbr python3-pip python3-pytsk3 python3-pyxattr python3-setuptools python3-six python3-yaml
- name: Install tox
run: |
python3 -m pip install tox
Expand Down
8 changes: 3 additions & 5 deletions .github/workflows/test_tox.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,10 @@ on:
- main
jobs:
build:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
strategy:
matrix:
include:
- python-version: '3.6'
toxenv: 'py36'
- python-version: '3.7'
toxenv: 'py37'
- python-version: '3.8'
Expand All @@ -26,7 +24,7 @@ jobs:
- python-version: '3.8'
toxenv: 'lint'
container:
image: ubuntu:20.04
image: ubuntu:22.04
steps:
- uses: actions/checkout@v2
- name: Set up container
Expand All @@ -45,7 +43,7 @@ jobs:
add-apt-repository -y ppa:deadsnakes/ppa
add-apt-repository -y ppa:gift/dev
apt-get update -q
apt-get install -y build-essential git libffi-dev python${{ matrix.python-version }} python${{ matrix.python-version }}-dev python${{ matrix.python-version }}-venv libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi-backend python3-cryptography python3-dfdatetime python3-distutils python3-dtfabric python3-idna python3-mock python3-pbr python3-pip python3-pytsk3 python3-pyxattr python3-setuptools python3-six python3-yaml
apt-get install -y build-essential git libffi-dev python${{ matrix.python-version }} python${{ matrix.python-version }}-dev python${{ matrix.python-version }}-venv libbde-python3 libewf-python3 libfsapfs-python3 libfsext-python3 libfsfat-python3 libfshfs-python3 libfsntfs-python3 libfsxfs-python3 libfvde-python3 libfwnt-python3 libluksde-python3 libmodi-python3 libphdi-python3 libqcow-python3 libsigscan-python3 libsmdev-python3 libsmraw-python3 libvhdi-python3 libvmdk-python3 libvsgpt-python3 libvshadow-python3 libvslvm-python3 python3-cffi-backend python3-cryptography python3-dfdatetime python3-distutils python3-dtfabric python3-idna python3-mock python3-pbr python3-pip python3-pytsk3 python3-pyxattr python3-setuptools python3-six python3-yaml
- name: Install tox
run: |
python3 -m pip install tox
Expand Down
2 changes: 1 addition & 1 deletion config/appveyor/install.ps1
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Script to set up tests on AppVeyor Windows.

$Dependencies = "PyYAML cffi cryptography dfdatetime dtfabric idna libbde libewf libfsapfs libfsext libfshfs libfsntfs libfsxfs libfvde libfwnt libluksde libmodi libphdi libqcow libsigscan libsmdev libsmraw libvhdi libvmdk libvsgpt libvshadow libvslvm mock pbr pytsk3 six xattr"
$Dependencies = "PyYAML cffi cryptography dfdatetime dtfabric idna libbde libewf libfsapfs libfsext libfsfat libfshfs libfsntfs libfsxfs libfvde libfwnt libluksde libmodi libphdi libqcow libsigscan libsmdev libsmraw libvhdi libvmdk libvsgpt libvshadow libvslvm mock pbr pytsk3 six xattr"
$Dependencies = ${Dependencies} -split " "

$Output = Invoke-Expression -Command "git clone https://github.com/log2timeline/l2tdevtools.git ..\l2tdevtools 2>&1"
Expand Down
2 changes: 1 addition & 1 deletion config/dpkg/control
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Homepage: https://github.com/log2timeline/dfvfs

Package: python3-dfvfs
Architecture: all
Depends: libbde-python3 (>= 20220121), libewf-python3 (>= 20131210), libfsapfs-python3 (>= 20220709), libfsext-python3 (>= 20220529), libfshfs-python3 (>= 20220709), libfsntfs-python3 (>= 20211229), libfsxfs-python3 (>= 20220706), libfvde-python3 (>= 20220121), libfwnt-python3 (>= 20210717), libluksde-python3 (>= 20220121), libmodi-python3 (>= 20210405), libphdi-python3 (>= 20220228), libqcow-python3 (>= 20201213), libsigscan-python3 (>= 20191221), libsmdev-python3 (>= 20140529), libsmraw-python3 (>= 20140612), libvhdi-python3 (>= 20201014), libvmdk-python3 (>= 20140421), libvsgpt-python3 (>= 20211115), libvshadow-python3 (>= 20160109), libvslvm-python3 (>= 20160109), python3-cffi-backend (>= 1.9.1), python3-cryptography (>= 2.0.2), python3-dfdatetime (>= 20211113), python3-dtfabric (>= 20220219), python3-idna (>= 2.5), python3-pytsk3 (>= 20210419), python3-pyxattr (>= 0.7.2), python3-yaml (>= 3.10), ${misc:Depends}
Depends: libbde-python3 (>= 20220121), libewf-python3 (>= 20131210), libfsapfs-python3 (>= 20220709), libfsext-python3 (>= 20220529), libfsfat-python3 (>= 20220816), libfshfs-python3 (>= 20220709), libfsntfs-python3 (>= 20211229), libfsxfs-python3 (>= 20220706), libfvde-python3 (>= 20220121), libfwnt-python3 (>= 20210717), libluksde-python3 (>= 20220121), libmodi-python3 (>= 20210405), libphdi-python3 (>= 20220228), libqcow-python3 (>= 20201213), libsigscan-python3 (>= 20191221), libsmdev-python3 (>= 20140529), libsmraw-python3 (>= 20140612), libvhdi-python3 (>= 20201014), libvmdk-python3 (>= 20140421), libvsgpt-python3 (>= 20211115), libvshadow-python3 (>= 20160109), libvslvm-python3 (>= 20160109), python3-cffi-backend (>= 1.9.1), python3-cryptography (>= 2.0.2), python3-dfdatetime (>= 20220816), python3-dtfabric (>= 20220219), python3-idna (>= 2.5), python3-pytsk3 (>= 20210419), python3-pyxattr (>= 0.7.2), python3-yaml (>= 3.10), ${misc:Depends}
Description: Python 3 module of dfVFS
dfVFS, or Digital Forensics Virtual File System, provides read-only access to
file-system objects from various storage media types and file formats. The goal
Expand Down
10 changes: 9 additions & 1 deletion dependencies.ini
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ version_property: __version__

[dfdatetime]
dpkg_name: python3-dfdatetime
minimum_version: 20211113
minimum_version: 20220816
rpm_name: python3-dfdatetime
version_property: __version__

Expand Down Expand Up @@ -63,6 +63,14 @@ pypi_name: libfsext-python
rpm_name: libfsext-python3
version_property: get_version()

[pyfsfat]
dpkg_name: libfsfat-python3
l2tbinaries_name: libfsfat
minimum_version: 20220816
pypi_name: libfsfat-python
rpm_name: libfsfat-python3
version_property: get_version()

[pyfshfs]
dpkg_name: libfshfs-python3
l2tbinaries_name: libfshfs
Expand Down
1 change: 1 addition & 0 deletions dfvfs/analyzer/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from dfvfs.analyzer import cs_analyzer_helper
from dfvfs.analyzer import ewf_analyzer_helper
from dfvfs.analyzer import ext_analyzer_helper
from dfvfs.analyzer import fat_analyzer_helper
from dfvfs.analyzer import gpt_analyzer_helper
from dfvfs.analyzer import gzip_analyzer_helper
from dfvfs.analyzer import hfs_analyzer_helper
Expand Down
52 changes: 52 additions & 0 deletions dfvfs/analyzer/fat_analyzer_helper.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# -*- coding: utf-8 -*-
"""The FAT format analyzer helper implementation."""

from dfvfs.analyzer import analyzer
from dfvfs.analyzer import analyzer_helper
from dfvfs.analyzer import specification
from dfvfs.lib import definitions


class FATAnalyzerHelper(analyzer_helper.AnalyzerHelper):
"""FAT analyzer helper."""

FORMAT_CATEGORIES = frozenset([
definitions.FORMAT_CATEGORY_FILE_SYSTEM])

TYPE_INDICATOR = definitions.TYPE_INDICATOR_FAT

def GetFormatSpecification(self):
"""Retrieves the format specification.
Returns:
FormatSpecification: format specification or None if the format cannot
be defined by a specification object.
"""
format_specification = specification.FormatSpecification(
self.type_indicator)

# Boot sector signature.
format_specification.AddNewSignature(b'\x55\xaa', offset=510)

# FAT-12 and FAT-16 file system hint.
format_specification.AddNewSignature(b'FAT12 ', offset=54)
format_specification.AddNewSignature(b'FAT16 ', offset=54)

# FAT-32 file system hint.
format_specification.AddNewSignature(b'FAT32 ', offset=82)

# exFAT file system signature.
format_specification.AddNewSignature(b'EXFAT ', offset=3)

return format_specification

def IsEnabled(self):
"""Determines if the analyzer helper is enabled.
Returns:
bool: True if the analyzer helper is enabled.
"""
return definitions.PREFERRED_FAT_BACK_END == self.TYPE_INDICATOR


analyzer.Analyzer.RegisterHelper(FATAnalyzerHelper())
21 changes: 16 additions & 5 deletions dfvfs/analyzer/tsk_analyzer_helper.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,28 @@ def GetFormatSpecification(self):
format_specification = specification.FormatSpecification(
self.type_indicator)

# FAT volume header signature.
format_specification.AddNewSignature(b'\x55\xaa', offset=510)
if definitions.PREFERRED_FAT_BACK_END == self.TYPE_INDICATOR:
# Boot sector signature.
format_specification.AddNewSignature(b'\x55\xaa', offset=510)

# FAT-12 and FAT-16 file system hint.
format_specification.AddNewSignature(b'FAT12 ', offset=54)
format_specification.AddNewSignature(b'FAT16 ', offset=54)

# FAT-32 file system hint.
format_specification.AddNewSignature(b'FAT32 ', offset=82)

# exFAT file system signature.
format_specification.AddNewSignature(b'EXFAT ', offset=3)

if definitions.PREFERRED_NTFS_BACK_END == self.TYPE_INDICATOR:
# NTFS file system signature.
format_specification.AddNewSignature(b'NTFS ', offset=3)

# HFS boot block signature.
format_specification.AddNewSignature(b'LK', offset=0)

if definitions.PREFERRED_HFS_BACK_END == self.TYPE_INDICATOR:
# HFS boot block signature.
# format_specification.AddNewSignature(b'LK', offset=0)

# HFS+ file system signature.
format_specification.AddNewSignature(b'H+', offset=1024)

Expand Down
149 changes: 149 additions & 0 deletions dfvfs/file_io/fat_file_io.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# -*- coding: utf-8 -*-
"""The File Allocation Table (FAT) file-like object implementation."""

import os

from dfvfs.file_io import file_io
from dfvfs.resolver import resolver


class FATFile(file_io.FileIO):
"""File input/output (IO) object using pyfsfat.file_entry"""

def __init__(self, resolver_context, path_spec):
"""Initializes a file input/output (IO) object.
Args:
resolver_context (Context): resolver context.
path_spec (PathSpec): a path specification.
"""
super(FATFile, self).__init__(resolver_context, path_spec)
self._file_system = None
self._fsfat_data_stream = None
self._fsfat_file_entry = None

def _Close(self):
"""Closes the file-like object."""
self._fsfat_data_stream = None
self._fsfat_file_entry = None

self._file_system = None

def _Open(self, mode='rb'):
"""Opens the file-like object defined by path specification.
Args:
mode (Optional[str]): file access mode.
Raises:
AccessError: if the access to open the file was denied.
IOError: if the file-like object could not be opened.
NotSupported: if a data stream, like the resource or named fork, is
requested to be opened.
OSError: if the file-like object could not be opened.
PathSpecError: if the path specification is incorrect.
"""
data_stream_name = getattr(self._path_spec, 'data_stream', None)

self._file_system = resolver.Resolver.OpenFileSystem(
self._path_spec, resolver_context=self._resolver_context)

file_entry = self._file_system.GetFileEntryByPathSpec(self._path_spec)
if not file_entry:
raise IOError('Unable to open file entry.')

fsfat_data_stream = None
fsfat_file_entry = file_entry.GetFATFileEntry()
if not fsfat_file_entry:
raise IOError('Unable to open FAT file entry.')

if data_stream_name == 'rsrc':
fsfat_data_stream = fsfat_file_entry.get_resource_fork()
elif data_stream_name:
raise IOError('Unable to open data stream: {0:s}.'.format(
data_stream_name))

self._fsfat_data_stream = fsfat_data_stream
self._fsfat_file_entry = fsfat_file_entry

# Note: that the following functions do not follow the style guide
# because they are part of the file-like object interface.
# pylint: disable=invalid-name

def read(self, size=None):
"""Reads a byte string from the file-like object at the current offset.
The function will read a byte string of the specified size or
all of the remaining data if no size was specified.
Args:
size (Optional[int]): number of bytes to read, where None is all
remaining data.
Returns:
bytes: data read.
Raises:
IOError: if the read failed.
OSError: if the read failed.
"""
if not self._is_open:
raise IOError('Not opened.')

if self._fsfat_data_stream:
return self._fsfat_data_stream.read(size=size)
return self._fsfat_file_entry.read(size=size)

def seek(self, offset, whence=os.SEEK_SET):
"""Seeks to an offset within the file-like object.
Args:
offset (int): offset to seek to.
whence (Optional(int)): value that indicates whether offset is an absolute
or relative position within the file.
Raises:
IOError: if the seek failed.
OSError: if the seek failed.
"""
if not self._is_open:
raise IOError('Not opened.')

if self._fsfat_data_stream:
self._fsfat_data_stream.seek(offset, whence)
else:
self._fsfat_file_entry.seek(offset, whence)

def get_offset(self):
"""Retrieves the current offset into the file-like object.
Return:
int: current offset into the file-like object.
Raises:
IOError: if the file-like object has not been opened.
OSError: if the file-like object has not been opened.
"""
if not self._is_open:
raise IOError('Not opened.')

if self._fsfat_data_stream:
return self._fsfat_data_stream.get_offset()
return self._fsfat_file_entry.get_offset()

def get_size(self):
"""Retrieves the size of the file-like object.
Returns:
int: size of the file-like object data.
Raises:
IOError: if the file-like object has not been opened.
OSError: if the file-like object has not been opened.
"""
if not self._is_open:
raise IOError('Not opened.')

if self._fsfat_data_stream:
return self._fsfat_data_stream.get_size()
return self._fsfat_file_entry.get_size()
3 changes: 2 additions & 1 deletion dfvfs/helpers/source_scanner.py
Original file line number Diff line number Diff line change
Expand Up @@ -718,7 +718,8 @@ def ScanForFileSystem(self, source_path_spec):
'Unsupported source found more than one file system types.')

# TODO: determine root location from file system or path specification.
if type_indicator == definitions.TYPE_INDICATOR_NTFS:
if type_indicator in (
definitions.TYPE_INDICATOR_FAT, definitions.TYPE_INDICATOR_NTFS):
root_location = '\\'
else:
root_location = '/'
Expand Down
Loading

0 comments on commit 6c6ce51

Please sign in to comment.