File systems¶
High-level APIs¶
- PyFilesystem
works with files and directories in archives, in storages, in the cloud, etc.
Integrated file systems:
AppFS for predefined storage locations in operating systems where applications can store data
FTPFS for working with FTP servers
MemoryFS for caches, temporary data storage, unit tests, etc. that exist in the working memory
MountFS for a virtual file system that can mount other file systems
MultiFS for a virtual file system that combines other file systems
OSFS for the OS file system
TarFS reads and writes compressed tar archives
TempFS contains temporary data
ZipFS reads and writes Zip files
File systems of the PyFilesystem organization on GitHub:
File systems from third-party developers:
fs_basespace for read access to the Illumina Basespace
fs.dropboxfs for Dropbox
fs.imapfs for Imap
fs.googledrivefs for Google Drive
fs.onedrivefs for OneDrive
fs.smbfs for Samba
mp-fs-wsgidav for WsgiDAV
- fsspec
Unified Python interface for many local, remote and embedded file systems and byte storages. If you already use pandas, Intake, Dask or DVC in your project, for example,
fsspec
is already available.In addition to the integrated implementations, there are also many extensions, for example:
abfs for the Azure Blob Service
adl for the Azure DataLake storage
alluxiofs for the Alluxio distributed cache
boxfs for access to Box file storage
dropbox for access to Dropbox shares
dvc for accessing a DVC repository as a file system
gcsfs for Google Cloud Storage
gdrive for access to Google Drive and shares
huggingface_hub for access to the Hugging Face Hub file system
lakefs for lakeFS datalakes
ocifs for access to the Oracle Cloud Object Storage
ossfs for the Alibaba Cloud (Aliyun) object storage system (OSS)
p9fs for 9P servers
s3fs for Amazon S3 and other compatible storage
wandbfs for accessing Wandb data
webdav4 for WebDAV
Specialised libraries¶
- PyArrow
Apache Arrow Python bindings for the Hadoop Distributed File System (HDFS) and other fsspec-compatible file systems.
- paramiko
Python implementation of the SSHv2 protocol, which offers both client and server functions. It forms the basis for the high-level SSH library Fabric.
- boto3
AWS SDK for Python facilitates integration with Amazon S3, Amazon EC2, Amazon DynamoDB and others.
- azure-storage-blob
Azure Storage Blobs client library for Python.
- oss2
Python SDK for the Alibaba Cloud Object Storage.
- minio
MinIO Python Client SDK for Amazon S3 compatible cloud storage.
- PyDrive2
Python wrapper library of the google-api-python-client, which simplifies many common Google Drive API tasks.
- Qcloud COSv5 SDK
Python SDK for the Tencent Cloud Object Storage (COS).
- linode_api4
Python bindings for the Linode API v4.
- airfs
brings standard Python I/O to various storages (such as Alibaba Cloud OSS, Amazon Web Services S3, GitHub, Microsoft Azure Blobs Storage and Files Storage, OpenStack Swift/Object Store.
- yandex-s3
Asyncio-compatible SDK for Yandex Object Storage.
Dormant projects¶
- PyDrive
Python wrapper library of the google-api-python-client, which simplifies many common Google Drive API tasks.
- digital-ocean-spaces
Python client for Digital Ocean Spaces with an inbuilt shell.