ADDED: get_file method to AzureStorageShare to generate file list for reports.
ADDED: storagestats.periodicity option validator.
CHANGE: Moved time helper functions to time.py file.
CHANGED: Used flake8 to format the code according to convention.
FIXED: wlcg example and output where the top storage service was an array
and should not.
CHANGED: Keys from the received 'Metadata' dictionary from an object are
converted to lowercase to fix discrepancy in capitalization from
different S3 implementations.
FIXED: Exception name from AssertError to AssertionError.
ADDED: type=str.lower to make sure checksums and checksums types are always
ADDED: Requirement of PyYAML.
ADDED: '--rucio' flag to 'reports filelist' that sets the prefix='rucio' and
delta='1' used for rucio consistency check filedumps.
ADDED: '--wlcg' flag to 'reports storage' that generates a json file according
to WLCG's storage accounting format from specified site schema file.
CHANGED: The reports file list now removes the chosen prefix from the paths.
ADDED: '--version' flag.
FIXED: Some numerical stats were failing to be integer type.
CHANGED: For s3 boto3 clients, rather than creating the client right away,
it now first creates a session and from that the client. This is done
to make the execution multithreding 'safe'. Otherwise the sessoin is
global and was causing the execution to fail sometimes.
ADDED: "-f/--force" flag for "stats" sub-command to skip checks and force
FIXED: Frequency issue that would constantly give wrong stats. Reason was that
the check was checking the connectionstats timestamp rather than the
CHANGED: How storage endpoints are read from the configuration files, moved
this selection process from get_storage_endpoints() to
ADDED: Added checks and exception handling for when no endpoints are found.
ADDED: requirement of module 'prometheus_client'
ADDED: 'minio_prometheus' api for S3 endpoints.
ADDED: '-f, --force' flag to checksums put to force checksum metadata update.
FIXED: Issue when requesting storagestats with the list-objects API only 1000
objects would be returned for S3 and Azure classes.
FIXED: Typo on memcahced missing index log line.
REMOVED: Redundant memcahce empty checks.
ADDED: 'checksums' positional command with 'get' and 'put' sub-commands.
ADDED: base exceptions for checksum error and warnings.
ADDED: self contained get_object_checksum method for S3StorageShare.
ADDED: self contained put_object_checksum method for S3StorageShare.
CHANGE: Renamed set_object_metadata to put_object_metadata.
REMOVED: validate_checksum and deps, does note belong here.
ADDED: get_cached_storage_stats function.
CHANGE: Workflow now tries to obtain storage stats from memcache and update
objects accordingly. This way we still get the available
information even if the frequency time has not been surpassed.
REMOVED: get_object_checksum from s3. This method does not belong here.
ADDED: Added get_object_metadata, get_object_checksum, set_object_metadata
methods to S3StorageShare.
ADDED: Memcached ttl for data which is a multiple of 10 of the 'frequency'
setting, and is set to 1 hour as a minimum.
ADDED: Added storagestats.frequency error code to README.md
CHANGE: Renamed 'periodicity' to 'frequency'.
CHANGE: Moved -v switch to the positional arguments, now it needs to be placed
after them instead of in-between.
ADDED: Exception handling when unable to create report files.
BUGFIX: DAV storage now properly calculates free space when quota is manually
set and RFC4331 is used to get stats.
ADDED: Exception handler and logging when file list report is not supported
by plugin type using "AttributeError" exception.
ADDED: python-dateutil requirement.
ADDED: '-p', '--delta', '-o' switches to reports sub-command.
ADDED: S3's list-objects function to output a file list instead of storage
stats when the 'reports' sub-command is called.
CHANGE: All time module imports have been changed to datetime module.
CHANGE: enabled 'reports' sub-command.
CHANGE: Import full path to exceptions modules. Removed 'DSS' from class names.
ADDED: Adds support to specify more than one endpoint to check instead of just
one or all, using the -e flag.
ADDED: __init__.py files to create pypi package.
CHANGE: Python is now chosen via environmental variable instead of path.
ADDED: Try-except to configloader.parse_conf_files() to deal with bad unicode.
This will skip the file if there are issues decoding and log a warning.
CHANGE: Updated docstrings on all modules (except 'exceptions') to conform with
ADDED: For S3, added cloudwatch() function to obtain the following metrics:
BucketSizeBytes and NumberOfObjects. These need to be setup in AWS
before being able to poll.
CHANGE: Argparse has been moved from runner.py to args.py. Now it works as
different functions for each of the sub-commands.
CHANGE: main() in runner.py has been split. Now it selects which sub-command
function to run depending on the positional argument passed by the
user. Added stats() and reports() functions.
FIX: Azure list-blob can get full list of blobs on containers with > 5k blobs.
CHANGE: Installable package.
CHANGE: Added sub-commands to dyanfed-storage executable instead of
CHANGE: -c flag now accepts several arguments for file paths instead of
having to use it several times. -d flag has been removed.
CHANGE: -o specifies output directory for flags -j, -x and -p.
CHANGE: Arguments for -j, -x, and -p change the default filename.
and suggestions from pylint.
output_dir var to output_path.
specific or multiple configuration files and directories.
v0.12.1 Changed logger handler from FileHandler to TimedRotatingFileHandler.
This allows the script to rotate the logfile at midnight, and keeps 15
if api quota setting, then it is copied from the parent endpoint. Added
INFO and DEBUG log information for these operations.
same URL, making them actually a unique physical endpoint, to avoid
multiple queries. Also added the ability to use the "-e flag" to specify
a single endpoint from the CLI to check.
need it) which tries the standard CA's in case the given ca_path fails.
If it fails again, then an error is raised.
v0.10.3 Renamed endpoints_tuple to endpoints_args_tuple.
to parse for settings. Added conn_timeout to validators in StorageStats
with a default of 10 seconds and this is used as the timeout for
method "starmap" necessary to pass arguments to the function being
called. Removed deprecated "optsparse" in favour for "argsparse".
Changed "super" calls to use the python3 format. Corrected formats and
var names (where makes sense) according to PEP8. Normalized logger
wait until all endpoints have been polled.
bytes. While this might be a legitimate empty endpoint, it could also
be a system that is not configured properly or lacks the ability to
report this information in which case a manual quota should be input.
"OK". Also now the memcached index holds multiple errors in a CSV format.
"mlogger" while keeping a consistent format on error reporting and removing
the necessity to instantiate a second logger. This also facilitates
flexibility for exception handling so these can be removed from levels
where it does not belong (before it was used for logging as well and that
was a bad design.) Removed error message from memcache's status field.
the error codes should give sufficient information. If more is needed
the user should either check the logfile or manually run with verbose.
settings in config files.
finish. Added function to check Dynafed's connection stats from memcache
in order to flag endpoints that are offline and skip them from checks.
v0.9.1 Added verbose option to print on sterr logger events.
before the schema. Added exception for SchemaIssues. Fixed issue with
exceptions by adding keywords to the "supr" calls. Added exception
message to the debug so the --debug output contains the full information.
v0.8.11 Added 'Delimiter': '*' to S3 generic, necessary to iterate AWS buckets.
v0.8.12 Added timeouts to S3 and DAV methods of 5 seconds.
dynafed_storagestats with its corresponding extension. Adds json file
v0.8.9 Adds plain txt file output.
which are not used and caused the program to fail.
v0.8.7 Added filecount for S3 ceph-admin api.
comments and docs.
v0.8.5 Added Azure support. Utilizes azure-storage module's list_blobs function.
Added the necessary exceptions. Stdout shows filecount, when available.
and handler for DAV endpoints.
v0.8.1 Added option to specify ouptut logfile.
v0.7.6 Fixes multiple white-lines in config files.
v0.7.7 Added logging to files, pwd and WARNING and above.
unused parameters in exception classes. Added documentation to all
exception classes. Cleaned code according to pep8 guidelines.
v0.7.4 Single XML StAR file output with all endpoints' stats.
options from quota to storagetats.quota and s3.api to storagestats.api.
v0.7.1 XML StAR files output implemented for S3 and DAV.
v0.6.0 Added quota options and logic to S3 and DAV operations.
v0.6.1 Moved ouptut to object class method.
v0.6.2 Added debug output.
v0.6.3 Renamed storagestats attribute from options to plugin_options.
to StorageStats objects. Status appends the last ERROR. Debug appends
all the ones that occur with more detail if available.
v0.4.8 Improved memcached and status/debug output.
v0.4.9 Added starttime and execbeat output.
v0.4.4 Added exceptions and error handling for DAV storagestats.
v0.4.5 Changed error to use the exception names. Works better and cleaner.
v0.4.6 ssl_check now uses the ca_path if declared and ssl_check is true.
v0.4.1 Added exceptions and error handling for S3 storagestats ceph-admin.
v0.4.2 Added exceptions and error handling for S3 storagestats, generic.
solve issue with json module of python 3.4.
v0.3.3 Fixed exception handling for python3 syntax.
v0.3.0 Added DAV/Http support.
v0.3.1 Added exceptions and logic when ceph-admin option fails.
doesn't have the "NextContinuationToken" directive and thus would only
list the first 1000. This needs to be updated one Ceph has this as
v1 is sort of deprecated.
v0.2.9 Added ability to specify S3 signature version.
v0.2.10 Added options for memcached, stdoutput and some debugging.
v0.2.7 Added files counted to S3 Generic.
v0.2.4 Added URL schema validator function and updated code. Works for dav,davs.
v0.2.5 Moved schema validator from function to class method. Added for S3.
v0.1.0 Changed aws-list to generic and now uses boto3 for generality.
v0.2.0 Added validators key and 'validate_options' function.
v0.2.1 Cleaned up code to PEP8.
v0.2.2 Exception for plugin types not yet implemented.
v0.0.4 Changed from single configuration file to all *.conf files in given directory.
v0.0.5 Added module import checks.
v0.0.6 StorageStats object class chosen dynamically based on configured plugin.