Changelogs » Dynafed-storagestats

PyUp Safety actively tracks 316,954 Python packages for vulnerabilities and notifies you when to upgrade.



ADDED: get_file method to AzureStorageShare to generate file list for reports.
  ADDED: storagestats.periodicity option validator.
  CHANGE: Moved time helper functions to file.


ADDED: Documentation on new names on minio-pormetheus docstring.


CHANGED: Forgot to set the version also on


CHANGED: Requirement of azure to azure-storage=0.36.0


ADDED: New prometheus metric names for minIO version >= RELEASE.2019-12-17T23-26-28Z


CHANGED: Location of logid moved to second column.


ADDED: --logid argument. Whichever string is passed will be logged as the
  first field in a log line.


ADDED: Check for memcache before asking endpoints when generating a storage


REMOVED: wglc-min-schema sample file as it was not enough information.
  ADDED: wlcg-schema sample file according to WCLG SRR.
  ADDED: Dynafed version and update time for the WLCG space-usage.json file.
  ADDED: helper get_dynafed_version() function to query rpm from dynafed version.


CHANGED: Used flake8 to format the code according to convention.
  FIXED: wlcg example and output where the top storage service was an array
  and should not.


CHANGED: Keys from the received 'Metadata' dictionary from an object are
  converted to lowercase to fix discrepancy in capitalization from
  different S3 implementations.
  FIXED: Exception name from AssertError to AssertionError.


ADDED: type=str.lower to make sure checksums and checksums types are always


ADDED: Requirement of PyYAML.


ADDED: '--rucio' flag to 'reports filelist' that sets the prefix='rucio' and
  delta='1' used for rucio consistency check filedumps.
  ADDED: '--wlcg' flag to 'reports storage' that generates a json file according
  to WLCG's storage accounting format from specified site schema file.
  CHANGED: The reports file list now removes the chosen prefix from the paths.


ADDED: '--version' flag.
  FIXED: Some numerical stats were failing to be integer type.


CHANGED: For s3 boto3 clients, rather than creating the client right away,
  it now first creates a session and from that the client. This is done
  to make the execution multithreding 'safe'. Otherwise the sessoin is
  global and was causing the execution to fail sometimes.
  ADDED: "-f/--force" flag for "stats" sub-command to skip checks and force
  stats check.
  FIXED: Frequency issue that would constantly give wrong stats. Reason was that
  the check was checking the connectionstats timestamp rather than the
  CHANGED: How storage endpoints are read from the configuration files, moved
  this selection process from get_storage_endpoints() to
  ADDED: Added checks and exception handling for when no endpoints are found.


ADDED: requirement of module 'prometheus_client'


ADDED: 'minio_prometheus' api for S3 endpoints.


ADDED: '-f, --force' flag to checksums put to force checksum metadata update.
  FIXED: Issue when requesting storagestats with the list-objects API only 1000
  objects would be returned for S3 and Azure classes.
  FIXED: Typo on memcahced missing index log line.
  REMOVED: Redundant memcahce empty checks.


ADDED: 'checksums' positional command with 'get' and 'put' sub-commands.
  ADDED: base exceptions for checksum error and warnings.
  ADDED: self contained get_object_checksum method for S3StorageShare.
  ADDED: self contained put_object_checksum method for S3StorageShare.
  CHANGE: Renamed set_object_metadata to put_object_metadata.
  REMOVED: validate_checksum and deps, does note belong here.


ADDED: get_cached_storage_stats function.
  ADDED: update_storage_share_storagestats.
  CHANGE: Workflow now tries to obtain storage stats from memcache and update
  objects accordingly. This way we still get the available
  information even if the frequency time has not been surpassed.


REMOVED: get_object_checksum from s3. This method does not belong here.


ADDED: Added get_object_metadata, get_object_checksum, set_object_metadata
  methods to S3StorageShare.


ADDED: Memcached ttl for data which is a multiple of 10 of the 'frequency'
  setting, and is set to 1 hour as a minimum.


ADDED: Added storagestats.frequency error code to


CHANGE: Renamed 'periodicity' to 'frequency'.
  CHANGE: Moved -v switch to the positional arguments, now it needs to be placed
  after them instead of in-between.
  ADDED: Exception handling when unable to create report files.


BUGFIX: DAV storage now properly calculates free space when quota is manually
  set and RFC4331 is used to get stats.
  ADDED: Exception handler and logging when file list report is not supported
  by plugin type using "AttributeError" exception.


ADDED: python-dateutil requirement.
  ADDED: now_in_utc()
  ADDED: mask_timestamp_by_delta()
  ADDED: '-p', '--delta', '-o' switches to reports sub-command.
  ADDED: S3's list-objects function to output a file list instead of storage
  stats when the 'reports' sub-command is called.
  CHANGE: All time module imports have been changed to datetime module.
  CHANGE: enabled 'reports' sub-command.


CHANGE: Import full path to exceptions modules. Removed 'DSS' from class names.


ADDED: Adds support to specify more than one endpoint to check instead of just
  one or all, using the -e flag.


ADDED: files to create pypi package.


CHANGE: Python is now chosen via environmental variable instead of path.
  ADDED: Try-except to configloader.parse_conf_files() to deal with bad unicode.
  This will skip the file if there are issues decoding and log a warning.


CHANGE: Updated docstrings on all modules (except 'exceptions') to conform with


ADDED: For S3, added cloudwatch() function to obtain the following metrics:
  BucketSizeBytes and  NumberOfObjects. These need to be setup in AWS
  before being able to poll.


CHANGE: Argparse has been moved from to Now it works as
  different functions for each of the sub-commands.
  CHANGE: main() in has been split. Now it selects which sub-command
  function to run depending on the positional argument passed by the
  user. Added stats() and reports() functions.


FIX: Azure list-blob can get full list of blobs on containers with > 5k blobs.


CHANGE: Installable package.
  CHANGE: Added sub-commands to dyanfed-storage executable instead of
  CHANGE: -c flag now accepts several arguments for file paths instead of
  having to use it several times. -d flag has been removed.
  CHANGE: -o specifies output directory for flags -j, -x and -p.
  CHANGE: Arguments for -j, -x, and -p change the default filename.


and suggestions from pylint.


output_dir var to output_path.


specific or multiple configuration files and directories.


  v0.12.1 Changed logger handler from FileHandler to TimedRotatingFileHandler.
  This allows the script to rotate the logfile at midnight, and keeps 15
  days worth.


if api quota setting, then it is copied from the parent endpoint. Added
  INFO and DEBUG log information for these operations.


same URL, making them actually a unique physical endpoint, to avoid
  multiple queries. Also added the ability to use the "-e flag" to specify
  a single endpoint from the CLI to check.


need it) which tries the standard CA's in case the given ca_path fails.
  If it fails again, then an error is raised.
  v0.10.3 Renamed endpoints_tuple to endpoints_args_tuple.


to parse for settings. Added conn_timeout to validators in StorageStats
  with a default of 10 seconds and this is used as the timeout for


method "starmap" necessary to pass arguments to the function being
  called. Removed deprecated "optsparse" in favour for "argsparse".
  Changed "super" calls to use the python3 format. Corrected formats and
  var names (where makes sense) according to PEP8. Normalized logger


wait until all endpoints have been polled.


bytes. While this might be a legitimate empty endpoint, it could also
  be a system that is not configured properly or lacks the ability to
  report this information in which case a manual quota should be input.


"OK". Also now the memcached index holds multiple errors in a CSV format.


"mlogger" while keeping a consistent format on error reporting and removing
  the necessity to instantiate a second logger. This also facilitates
  flexibility for exception handling so these can be removed from levels
  where it does not belong (before it was used for logging as well and that
  was a bad design.) Removed error message from memcache's status field.
  the error codes should give sufficient information. If more is needed
  the user should either check the logfile or manually run with verbose.


settings in config files.


finish. Added function to check Dynafed's connection stats from memcache
  in order to flag endpoints that are offline and skip them from checks.
  v0.9.1 Added verbose option to print on sterr logger events.


before the schema. Added exception for SchemaIssues. Fixed issue with
  exceptions by adding keywords to the "supr" calls. Added exception
  message to the debug so the --debug output contains the full information.
  v0.8.11 Added 'Delimiter': '*' to S3 generic, necessary to iterate AWS buckets.
  v0.8.12 Added timeouts to S3 and DAV methods of 5 seconds.


dynafed_storagestats with its corresponding extension. Adds json file
  v0.8.9 Adds plain txt file output.


which are not used and caused the program to fail.
  v0.8.7 Added filecount for S3 ceph-admin api.


comments and docs.
  v0.8.5 Added Azure support. Utilizes azure-storage module's list_blobs function.
  Added the necessary exceptions. Stdout shows filecount, when available.


and handler for DAV endpoints.




memcached status.
  v0.8.1 Added option to specify ouptut logfile.


v0.7.6 Fixes multiple white-lines in config files.
  v0.7.7 Added logging to files, pwd and WARNING and above.


unused parameters in exception classes. Added documentation to all
  exception classes. Cleaned code according to pep8 guidelines.
  v0.7.4 Single XML StAR file output with all endpoints' stats.


options from quota to storagetats.quota and s3.api to storagestats.api.


  v0.7.1 XML StAR files output implemented for S3 and DAV.


  v0.6.0 Added quota options and logic to S3 and DAV operations.
  v0.6.1 Moved ouptut to object class method.
  v0.6.2 Added debug output.
  v0.6.3 Renamed storagestats attribute from options to plugin_options.


to StorageStats objects. Status appends the last ERROR. Debug appends
  all the ones that occur with more detail if available.
  v0.4.8 Improved memcached and status/debug output.
  v0.4.9 Added starttime and execbeat output.


unsupported plugins.
  v0.4.4 Added exceptions and error handling for DAV storagestats.
  v0.4.5 Changed error to use the exception names. Works better and cleaner.
  v0.4.6 ssl_check now uses the ca_path if declared and ssl_check is true.


  v0.4.1 Added exceptions and error handling for S3 storagestats ceph-admin.
  v0.4.2 Added exceptions and error handling for S3 storagestats, generic.


solve issue with json module of python 3.4.


ceph-admin error.
  v0.3.3 Fixed exception handling for python3 syntax.


v0.3.0 Added DAV/Http support.
  v0.3.1 Added exceptions and logic when ceph-admin option fails.


doesn't have the "NextContinuationToken" directive and thus would only
  list the first 1000. This needs to be updated one Ceph has this as
  v1 is sort of deprecated.
  v0.2.9 Added ability to specify S3 signature version.
  v0.2.10 Added options for memcached, stdoutput and some debugging.


v0.2.7 Added files counted to S3 Generic.


S3 endpoints.
  v0.2.4 Added URL schema validator function and updated code. Works for dav,davs.
  v0.2.5 Moved schema validator from function to class method. Added for S3.


v0.1.0 Changed aws-list to generic and now uses boto3 for generality.
  v0.2.0 Added validators key and 'validate_options' function.
  v0.2.1 Cleaned up code to PEP8.
  v0.2.2 Exception for plugin types not yet implemented.


v0.0.4 Changed from single configuration file to all *.conf files in given directory.
  v0.0.5 Added module import checks.
  v0.0.6 StorageStats object class chosen dynamically based on configured plugin.