PyUp Safety actively tracks 232,000 Python packages for vulnerabilities and notifies you when to upgrade.
ADDED: get_file method to AzureStorageShare to generate file list for reports. ADDED: storagestats.periodicity option validator. CHANGE: Moved time helper functions to time.py file.
ADDED: Documentation on new names on minio-pormetheus docstring.
CHANGED: Forgot to set the version also on setup.py.
CHANGED: Requirement of azure to azure-storage=0.36.0
ADDED: New prometheus metric names for minIO version >= RELEASE.2019-12-17T23-26-28Z
CHANGED: Location of logid moved to second column.
ADDED: --logid argument. Whichever string is passed will be logged as the first field in a log line.
ADDED: Check for memcache before asking endpoints when generating a storage report.
REMOVED: wglc-min-schema sample file as it was not enough information. ADDED: wlcg-schema sample file according to WCLG SRR. ADDED: Dynafed version and update time for the WLCG space-usage.json file. ADDED: helper get_dynafed_version() function to query rpm from dynafed version.
CHANGED: Used flake8 to format the code according to convention. FIXED: wlcg example and output where the top storage service was an array and should not.
CHANGED: Keys from the received 'Metadata' dictionary from an object are converted to lowercase to fix discrepancy in capitalization from different S3 implementations. FIXED: Exception name from AssertError to AssertionError.
ADDED: type=str.lower to make sure checksums and checksums types are always lowercase.
ADDED: Requirement of PyYAML.
ADDED: '--rucio' flag to 'reports filelist' that sets the prefix='rucio' and delta='1' used for rucio consistency check filedumps. ADDED: '--wlcg' flag to 'reports storage' that generates a json file according to WLCG's storage accounting format from specified site schema file. CHANGED: The reports file list now removes the chosen prefix from the paths.
ADDED: '--version' flag. FIXED: Some numerical stats were failing to be integer type.
CHANGED: For s3 boto3 clients, rather than creating the client right away, it now first creates a session and from that the client. This is done to make the execution multithreding 'safe'. Otherwise the sessoin is global and was causing the execution to fail sometimes. ADDED: "-f/--force" flag for "stats" sub-command to skip checks and force stats check. FIXED: Frequency issue that would constantly give wrong stats. Reason was that the check was checking the connectionstats timestamp rather than the storagestats. CHANGED: How storage endpoints are read from the configuration files, moved this selection process from get_storage_endpoints() to get_storage_shares(). ADDED: Added checks and exception handling for when no endpoints are found.
ADDED: requirement of module 'prometheus_client'
ADDED: 'minio_prometheus' api for S3 endpoints.
ADDED: '-f, --force' flag to checksums put to force checksum metadata update. FIXED: Issue when requesting storagestats with the list-objects API only 1000 objects would be returned for S3 and Azure classes. FIXED: Typo on memcahced missing index log line. REMOVED: Redundant memcahce empty checks.
ADDED: 'checksums' positional command with 'get' and 'put' sub-commands. ADDED: base exceptions for checksum error and warnings. ADDED: self contained get_object_checksum method for S3StorageShare. ADDED: self contained put_object_checksum method for S3StorageShare. CHANGE: Renamed set_object_metadata to put_object_metadata. REMOVED: validate_checksum and deps, does note belong here.
ADDED: get_cached_storage_stats function. ADDED: update_storage_share_storagestats. CHANGE: Workflow now tries to obtain storage stats from memcache and update objects accordingly. This way we still get the available information even if the frequency time has not been surpassed.
REMOVED: get_object_checksum from s3. This method does not belong here.
ADDED: Added get_object_metadata, get_object_checksum, set_object_metadata methods to S3StorageShare.
ADDED: Memcached ttl for data which is a multiple of 10 of the 'frequency' setting, and is set to 1 hour as a minimum.
ADDED: Added storagestats.frequency error code to README.md
CHANGE: Renamed 'periodicity' to 'frequency'. CHANGE: Moved -v switch to the positional arguments, now it needs to be placed after them instead of in-between. ADDED: Exception handling when unable to create report files.
BUGFIX: DAV storage now properly calculates free space when quota is manually set and RFC4331 is used to get stats. ADDED: Exception handler and logging when file list report is not supported by plugin type using "AttributeError" exception.
ADDED: python-dateutil requirement. ADDED: now_in_utc() ADDED: mask_timestamp_by_delta() ADDED: '-p', '--delta', '-o' switches to reports sub-command. ADDED: S3's list-objects function to output a file list instead of storage stats when the 'reports' sub-command is called. CHANGE: All time module imports have been changed to datetime module. CHANGE: enabled 'reports' sub-command.
CHANGE: Import full path to exceptions modules. Removed 'DSS' from class names.
ADDED: Adds support to specify more than one endpoint to check instead of just one or all, using the -e flag.
ADDED: __init__.py files to create pypi package.
CHANGE: Python is now chosen via environmental variable instead of path. ADDED: Try-except to configloader.parse_conf_files() to deal with bad unicode. This will skip the file if there are issues decoding and log a warning.
CHANGE: Updated docstrings on all modules (except 'exceptions') to conform with PEP257.
ADDED: For S3, added cloudwatch() function to obtain the following metrics: BucketSizeBytes and NumberOfObjects. These need to be setup in AWS before being able to poll.
CHANGE: Argparse has been moved from runner.py to args.py. Now it works as different functions for each of the sub-commands. CHANGE: main() in runner.py has been split. Now it selects which sub-command function to run depending on the positional argument passed by the user. Added stats() and reports() functions.
FIX: Azure list-blob can get full list of blobs on containers with > 5k blobs.
CHANGE: Installable package. CHANGE: Added sub-commands to dyanfed-storage executable instead of dyanfed_storagestats. CHANGE: -c flag now accepts several arguments for file paths instead of having to use it several times. -d flag has been removed. CHANGE: -o specifies output directory for flags -j, -x and -p. CHANGE: Arguments for -j, -x, and -p change the default filename.
and suggestions from pylint.
output_dir var to output_path.
specific or multiple configuration files and directories.
README. v0.12.1 Changed logger handler from FileHandler to TimedRotatingFileHandler. This allows the script to rotate the logfile at midnight, and keeps 15 days worth.
if api quota setting, then it is copied from the parent endpoint. Added INFO and DEBUG log information for these operations.
same URL, making them actually a unique physical endpoint, to avoid multiple queries. Also added the ability to use the "-e flag" to specify a single endpoint from the CLI to check.
need it) which tries the standard CA's in case the given ca_path fails. If it fails again, then an error is raised. v0.10.3 Renamed endpoints_tuple to endpoints_args_tuple.
to parse for settings. Added conn_timeout to validators in StorageStats with a default of 10 seconds and this is used as the timeout for connections.
method "starmap" necessary to pass arguments to the function being called. Removed deprecated "optsparse" in favour for "argsparse". Changed "super" calls to use the python3 format. Corrected formats and var names (where makes sense) according to PEP8. Normalized logger format.
wait until all endpoints have been polled.
bytes. While this might be a legitimate empty endpoint, it could also be a system that is not configured properly or lacks the ability to report this information in which case a manual quota should be input.
"OK". Also now the memcached index holds multiple errors in a CSV format.
"mlogger" while keeping a consistent format on error reporting and removing the necessity to instantiate a second logger. This also facilitates flexibility for exception handling so these can be removed from levels where it does not belong (before it was used for logging as well and that was a bad design.) Removed error message from memcache's status field. the error codes should give sufficient information. If more is needed the user should either check the logfile or manually run with verbose.
settings in config files.
finish. Added function to check Dynafed's connection stats from memcache in order to flag endpoints that are offline and skip them from checks. v0.9.1 Added verbose option to print on sterr logger events.
before the schema. Added exception for SchemaIssues. Fixed issue with exceptions by adding keywords to the "supr" calls. Added exception message to the debug so the --debug output contains the full information. v0.8.11 Added 'Delimiter': '*' to S3 generic, necessary to iterate AWS buckets. v0.8.12 Added timeouts to S3 and DAV methods of 5 seconds.
dynafed_storagestats with its corresponding extension. Adds json file output. v0.8.9 Adds plain txt file output.
which are not used and caused the program to fail. v0.8.7 Added filecount for S3 ceph-admin api.
comments and docs. v0.8.5 Added Azure support. Utilizes azure-storage module's list_blobs function. Added the necessary exceptions. Stdout shows filecount, when available.
and handler for DAV endpoints.
memcached status. v0.8.1 Added option to specify ouptut logfile.
v0.7.6 Fixes multiple white-lines in config files. v0.7.7 Added logging to files, pwd and WARNING and above.
unused parameters in exception classes. Added documentation to all exception classes. Cleaned code according to pep8 guidelines. v0.7.4 Single XML StAR file output with all endpoints' stats.
options from quota to storagetats.quota and s3.api to storagestats.api.
interpreter. v0.7.1 XML StAR files output implemented for S3 and DAV.
output. v0.6.0 Added quota options and logic to S3 and DAV operations. v0.6.1 Moved ouptut to object class method. v0.6.2 Added debug output. v0.6.3 Renamed storagestats attribute from options to plugin_options.
to StorageStats objects. Status appends the last ERROR. Debug appends all the ones that occur with more detail if available. v0.4.8 Improved memcached and status/debug output. v0.4.9 Added starttime and execbeat output.
unsupported plugins. v0.4.4 Added exceptions and error handling for DAV storagestats. v0.4.5 Changed error to use the exception names. Works better and cleaner. v0.4.6 ssl_check now uses the ca_path if declared and ssl_check is true.
warnings. v0.4.1 Added exceptions and error handling for S3 storagestats ceph-admin. v0.4.2 Added exceptions and error handling for S3 storagestats, generic.
solve issue with json module of python 3.4.
ceph-admin error. v0.3.3 Fixed exception handling for python3 syntax.
v0.3.0 Added DAV/Http support. v0.3.1 Added exceptions and logic when ceph-admin option fails.
doesn't have the "NextContinuationToken" directive and thus would only list the first 1000. This needs to be updated one Ceph has this as v1 is sort of deprecated. v0.2.9 Added ability to specify S3 signature version. v0.2.10 Added options for memcached, stdoutput and some debugging.
v0.2.7 Added files counted to S3 Generic.
S3 endpoints. v0.2.4 Added URL schema validator function and updated code. Works for dav,davs. v0.2.5 Moved schema validator from function to class method. Added for S3.
v0.1.0 Changed aws-list to generic and now uses boto3 for generality. v0.2.0 Added validators key and 'validate_options' function. v0.2.1 Cleaned up code to PEP8. v0.2.2 Exception for plugin types not yet implemented.
v0.0.4 Changed from single configuration file to all *.conf files in given directory. v0.0.5 Added module import checks. v0.0.6 StorageStats object class chosen dynamically based on configured plugin.