Yamlpath

Enhancements:
* Enable verified support for ruamel.yaml up to version 0.17.10.

Bug Fixes:
* Some peculiar data constructions could cause the ConsolePrinter to stack-dump
when writing debug messages.
* A NotImplementedError stack-dump was generated whenever a YAML Path was
evaluated when written improperly for Collector Math operations. This
specifically occurred when the RHS term was not demarcated with a parenthesis
pair. Now, a YAML Path parsing error is generated, indicating where the
missing character must appear.
* Use of an Array Element index in square-bracket notation ([N]) within a
sub-path -- such as in a Search Expression or Collector -- caused incorrect
YAML Path parsing. This usually manifested as a "not an integer index"
error.
* Byte strings were causing stack-dumps during JSON serialization; they are now
serialized as double-demarcated strings (a ' pair within a " pair) with a b
prefix, like: {"byte_value": "b'BYTEVAL'"}.
* Bare Anchor name references were treated as Hash key names when & was not the
very first non-separator character of a YAML Path or immediately following a
`[`. So, /my_hash/&my_anchor was not working as expected.
* The Merger (and thus, the yaml-merge command-line tool) would only return the
LHS document when neither LHS nor RHS documents were Hashes, no matter what
the merge options were set to. This did not affect content which was
children of Hashes.

Enhancements:
* YAML Path parsing errors are now a little more specific, indicating at which
character index the issue occurs. API users who have been scanning error
messages will need to update their code to accommodate the new text.
* Collector subtraction now handles Hash-of-Hashes and Array-of-Array results,
which were not possible before.
* Array-of-Hash nodes can now be searched for the presence of a given key in
its Hash elements using the . search operand, yielding matching elements (the
entire Hash elements having the given key). The difference can be
illustrated by contrasting these now-equivalent YAML Paths (where "books"
is an Array-of-Hashes; imagine only some Hash elements have an "isbn" key):
1. `/books/*[.=isbn]` or `books.*[.=isbn]`
2. `/books[.=isbn]/isbn` or `books[.=isbn].isbn`
3. `/books/*[has_child(isbn)]/isbn` or `books.*[has_child(isbn)].isbn`
4. `/books[has_child(isbn)]/isbn` or `books[has_child(isbn)].isbn`
All four of those queries yield exactly the same data. Note that example 2
behaves like examples 3 and 4. Examples 2-4 yield the entire matching Hash,
not just the "isbn" value. This enables access to other keys of the Hash
without necessitating use of a `[parent()]` search keyword, which would be
necessary for example 1 if you wanted to access any key other than "isbn"
from the matches.
* YAML Merge Keys can now be accessed directly by Anchor name, yielding the
entire original -- pre-merged -- reference Hash. This has _very limited_
utility. Using this in isolation will only reveal the default values for any
referenced keys, ignoring -- perhaps confusingly -- any local overrides. It
can however be helpful when reverse-engineering very complex merge
arrangements.
* The yaml-merge command-line tool (and the underlying Merger class) now offer
an option -- --preserve-lhs-comments (-l) -- that will attempt to preserve
LHS document comments. USE WITH CAUTION. At present, comment handling
during a merge is unwieldy, so some comments or new-line characters may
appear to become divorced from nodes they should obviously be attached to.
As such, the default behavior of the merge engine will continue to be removal
of all comments. At this time, RHS document comments will still be discarded
during merge operations. This will be revisited when ruamel.yaml refactors
how YAML comments are handled.
* The yaml-merge command-line tool now offers a new option, --multi-doc-mode
(-M), which accepts one of the following modes:
* CONDENSE_ALL: This is the default, which merges all multi-documents up
into single documents during the merge.
* MERGE_ACROSS: Condense no multi-documents; rather, only merge documents
"across" from right to left such that the first document in the RHS multi-
document merges only into the first document in the LHS multi-document, the
second across similarly, and so on.
* MATRIX_MERGE: Condense no multi-documents; rather, merge every RHS
document in a multi-document RHS into every LHS document in a multi-
document LHS.
* The [has_child(NAME)] Search Keyword now accepts an &NAME form of its first
(only) parameter. This switches the function to match against Anchor/Alias
names, including YAML Merge Keys.
* YAML Merge Keys can now be deleted by their Anchor/Alias name via the
yaml-set command-line tool and the underlying Processor class.
* YAML Merge Keys can now be created, offering run-time merging of
same-document Hash data. The yaml-set command-line tool offers a new option,
--mergekey, which applies to --change targets the new YAML Merge Key, as long
as each target is a Hash.
WARNING: As a consequence of adding this capability to the yaml-set command-
line tool, it is no longer possible to implicitly alias scalar nodes by
passing only the --change and --anchor parameters. The operation must now be
explicit by setting --aliasof or --mergekey along with --change and
optionally with --anchor.
* The yaml-diff tool now supports multi-document sources. Only one document of
any multi-document source on each side of the LHS-RHS comparison can be
selected for the operation (diffs are performed only between two documents).
Such selection is made via two new options, --left-document-index|-L and
--right-document-index|-R. An error is emitted whenever a multi-document
source is detected without an appropriate document index selection.
* YAML Unordered Sets -- https://yaml.org/type/set.html -- are now fully
supported in YAML Paths, this project's API, and the reference command-line
tools. Because an Unordered Set is effectively a Hash (map/dict) where the
entries are key-value pairs all having null (None) values, their entries are
accessible only by their exact key. While they look in YAML data like Arrays
(sequences/lists) with a leading `?` rather than a `-`, they are not; their
entries cannot be accessed by a numerical index because they are defined in
the YAML specification as deliberately unordered.

API Changes:
* The common.nodes utility class now has a generally-useful static method which
accepts any String data and safely converts it to its native Python data-type
equivalent with special handling for case-insensitive Booleans via
ast.literal_eval: typed_value.
* The common.searches utility class now requires both terms to be of the same
data-type for comparisons. When they types materially differ -- int and
float are treated as similar enough -- a String comparision is performed.
This is how it has always been excepting that types were lazily coalesced in
older versions; they are now converted before the comparison is considered.
* The NodeCoords wrapper now supports more utility properties and methods:
* .unwrapped_node is the same output as calling
NodeCoords.unwrap_node_coords(data) except it can be called directly upon
instances of NodeCoords rather than as a static method call. The static
method is still available.
* .deepest_node_coord returns whichever NodeCoord instance is most deeply
wrapped by an instance of NodeCoords. Such wrapping comes from nesting
Collectors. This method simplfies getting to the original data element(s).
* .wraps_a(Type) indicates whether the deepest wrapped data element is of a
given data-type.

Bug Fixes:
* Search expressions against Boolean values, [key=True] and [key=False], were
impossible. Now, they are working and are not case-sensitive, so [key=True],
[key=true], [key=TRUE], and such all work as expected.
* When null values were present, Deep Traversal (**) segments would always
return every node with a null value even when they would not match filter
conditions after the ** segment. When mustexist=False, this would also cause
a YAMLPathException.
* Descendent searches were considering only the first child of the search
ancestor. Now, ANY matching descendent node will correctly yield the
ancestor.
* Some Python-generated complex data types were escaping JSONification,
leading to unexpected stack-dumps when writing out JSON data for data types
like date and datetime.

Enhancements:
* An entirely new segment type has been added to YAML Path and is now supported
by the library and reference implementation command-line tools: Keyword
Searches. Similar to programming language keywords, these reserved Keywords
work much like functions, accepting parameters and performing algorythmic
operations or returning data not otherwise accessible to other YAML Path
segment types. These new capabilities -- explored on the project Wiki --
include:
* [has_child(NAME)]
* [name()]
* [max(NAME)]
* [min(NAME)]
* [parent([STEPS])]
* When stringified, YAML Paths with a solitary * wildcard segment were printed
using their internal RegEx variant, [.=~/.*/]. They are now printed as they
are entered, using a solitary *. As a consequence, any deliberate RegEx of
[.=~/.*/] is also printed as its equivalent solitary *.
* The yaml-paths command now allows printing YAML Paths without protective
escape symbols via a new --noescape option. While this makes the output more
human-friendly, the unescaped paths will not be suitable for use as YAML Path
input to other YAML Path processors where special symbols require escaping.
* [API] The NodeCoords class now tracks ancestry and the last YAML Path segment
responsible for triggering its generation. The ancestry stack --
List[AncestryEntry] -- was necessary to support the [parent()] Search
Keyword. The responsible YAML Path segment tracking was necessary to enable
Hash/map/dict key renaming via the [name()] Search Keyword. These optional
attributes may be set when the NodeCoords is generated.
* [API] YAMLPath instances now have a pop() method. This mutates the YAMLPath
by popping off its last segment, returning that segment.

Bug Fixes:
* yaml-set (and the underlying Processor class) were unable to change nodes
having a null (None) value to anything else. This changes how null/None
values are handled by the Processor during node retrieval; they are no longer
discarded, so you will receive None as the data of any retrieved NodeCoords
for appropriate null/None leaf nodes.

Enhancements:
* Python 3.9 is now supported (because common testing tools finally work with
Python 3.9).
* The node deletion capability of the yaml-set command is now part of the
library. See Processor::delete_nodes(...) and
Processor::delete_gathered_nodes(...) for details.
* The node aliasing capability of the yaml-set command is now part of the
library. See Processor::alias_nodes(...) and
Processor::alias_gathered_nodes(...) for details.
* The node tagging capability of the yaml-set command is now part of the
library. See Processor::tag_nodes(...) and
Processor::tag_gathered_nodes(...) for details.
* The library now supports loading YAML from String rather than only from file.
Simply pass a new `literal=True` keyword parameter to
Parsers::get_yaml_data(...) or Parsers::get_yaml_multidoc_data(...) to
indicate that `source` is literal serialized (String) YAML data rather than a
file-spec. This mode is implied when reading from STDIN (source is "-").
* The emitter_write_folded_fix.py patch file for ruamel.yaml has been removed
in favor of an author-supplied solution to the problem --
https://sourceforge.net/p/ruamel-yaml/tickets/383/ -- for which the patch was
originally written.

Known Issues:
* ruamel.yaml version 0.17.x is a major refactoring effort by the project's
owner. As such, only select versions will be marked as compatible with
yamlpath. Such marking occurs in this project's dependencies list via the
setup.py file. This is necessary because I use yamlpath in production
environments where stability is paramount; I need the freedom to update
yamlpath at-will without incurring any unexpected failures due to
incompatible ruamel.yaml changes. I will try to test some -- but not all --
ruamel.yaml releases from time to time and update yamlpath dependency
compatibilities accordingly.
* ruamel.yaml version 0.17.4 somewhat resolves a previously reported issue --
https://sourceforge.net/p/ruamel-yaml/tickets/351/ -- wherein certain
arrangements of comments or new-lines within YAML files near aliased hash
keys would cause a total loss of data when the stream was written to file.
Now, the data is no longer entirely lost. However, the preceding comment or
new-line is deleted when the stream is written to file. This is deemed to be
an acceptable compromise, for now, because the alternative is to either lose
the entire document or lose all attempted changes to the affected document.
Until the issue is properly fixed, an XFAIL test will continue to be in the
yamlpath unit test suite.

Bug Fixes:
* For the yaml-diff command-line tool, custom identity keys for specific
records in AoHs of the RHS document were not being considered for comparison.
Any identity key set for the whole AoH was being applied to every record in
the set.

Enhancements:
* The yaml-get command-line tool -- and the underlying Processor::get_nodes
method -- now retrives nodes with `null` values. Non-JSON null results from
yaml-get are printed using the ASCII NULL control-character (Hexadecimal 00)
wherever encountered. While most terminals and shells won't visibly print
this character, it is there and can be picked up by downstream parsers of
STDOUT. When the output is rendered as JSON, the unquoted "null" identifier
is used.
* The yaml-set command-line tool now enables assigning null values using a new
--null|-N input option.
* The yaml-set command-line tool now supports assigning YAML Aliases to target
--change|-g nodes, referencing any other node via --aliasof|-A whether it is
already Anchored, or not. Should the Anchor not already exist, a unique name
will be automatically assigned. This automatic name can controlled via
--anchor|-H. Setting --change|-g and --aliasof|-A to the same node along
with a new --anchor|-H explicitly renames an existing Anchor and its Aliases.
The same is implicitly possible by specifying --change|-g and --anchor|-H
without --aliasof|-A. Using these against non-YAML files merely duplicates
the target value to the indicated --change|-g nodes.
* With a new --tag|-T auxilliary option, the yaml-set command-line tool now
supports assigning custom YAML tags (data-type specifiers) to nodes created
or updated during the operation.
* The yaml-merge and yaml-diff command-line tools now also handle YAML tags.
* The single-star wildcard segment translation was changed from [.!=""] to
[.=~/.*/] which enables it to match any key or value (including empty-string
and null).

API Changes:
* The yamlpath.func library is too big and some very useful general code has
become attached to special-purpose classes. As such, it's time for
refactoring before this becomes untennable. New, better-organized static
classes were created under a new yamlpath.common namespace. Since this
would be destructive to other Python code which depends on the old
organization, this is a heads-up. Your existing code will continue to work
without any changes, but only until 4.x (which is not yet planned). Until
then, you must update your own code to use the new static classes for the
following functions and methods so as to avoid fatal errors down the road:
- yamlpath.func.append_list_element
-> yamlpath.common.Nodes.append_list_element

- yamlpath.func.build_next_node
-> yamlpath.common.Nodes.build_next_node

- yamlpath.func.clone_node
-> yamlpath.common.Nodes.clone_node

- yamlpath.func.create_searchterms_from_pathattributes
-> yamlpath.common.Searches.create_searchterms_from_pathattributes

- yamlpath.func.ensure_escaped
-> yamlpath.YAMLPath.ensure_escaped

- yamlpath.func.escape_path_section
-> yamlpath.YAMLPath.escape_path_section

- yamlpath.func.get_node_anchor
-> yamlpath.common.Anchors.get_node_anchor

- yamlpath.func.get_yaml_data
-> yamlpath.common.Parsers.get_yaml_data

- yamlpath.func.get_yaml_editor
-> yamlpath.common.Parsers.get_yaml_editor

- yamlpath.func.get_yaml_multidoc_data
-> yamlpath.common.Parsers.get_yaml_multidoc_data

- yamlpath.func.make_float_node
-> yamlpath.common.Nodes.make_float_node

- yamlpath.func.make_new_node
-> yamlpath.common.Nodes.make_new_node

- yamlpath.func.search_anchor
-> yamlpath.common.Searches.search_anchor

- yamlpath.func.search_matches
-> yamlpath.common.Searches.search_matches

- yamlpath.func.stringify_dates
-> yamlpath.common.Parsers.stringify_dates
NOTE that this method is deprecated and will be eliminated in favor of
using its more comprehensive replacement,
yamlpath.common.Parsers.jsonify_yaml_data

- yamlpath.func.unwrap_node_coords
-> yamlpath.wrappers.nodecoords.NodeCoords.unwrap_node_coords

- yamlpath.func.wrap_type
-> yamlpath.common.Nodes.wrap_type

- yamlpath.merger.Merger.combine_merge_anchors
-> yamlpath.common.Anchors.combine_merge_anchors

- yamlpath.merger.Merger.delete_all_comments
-> yamlpath.common.Anchors.delete_all_comments

- yamlpath.merger.Merger.rename_anchor
-> yamlpath.common.Anchors.rename_anchor

- yamlpath.merger.Merger.replace_anchor
-> yamlpath.common.Anchors.replace_anchor

- yamlpath.merger.Merger.scan_for_anchors
-> yamlpath.common.Anchors.scan_for_anchors

- yamlpath.merger.Merger.set_flow_style
-> yamlpath.common.Anchors.set_flow_style

Until you update your code, a deprecation warning will be printed to STDERR
every time the yamlpath.func file is imported and the first time one of the
relocated Merger static methods is called. To be rid of the message, update
your code to use the new sources of the deprecated functions/methods and
remove every import of and from yamlpath.func.

Bug Fixes:
* It was impossible to install yamlpath 3.x without first installing
ruamel.yaml via pip for Python 3.x. Not only has this been fixed but
explicit tests have been created to ensure this never happens again.

Enhancements:
* A new command-line tool, yaml-diff, now compares exactly two
YAML/JSON/Compatible documents, producing a GNU diff-like report of any
differences in the data they present to parsers. Along with diff's "a"
(added), "c" (changed), and "d" (deleted) report entries, affected YAML Paths
are printed in lieu of line numbers. Further, a report entry of "s" (same)
is available and can be enabled via command-line options. This tool also
features optional special handling of Arrays and Arrays-of-Hashes, which can
be configured as CLI options or via an INI file for distinct settings per
YAML Path. See --help or the Wiki for more detail.

API Changes:
* NodeCoords now employ a new `path` attribute. This is an optional parameter
which is assigned during construction to later report the translated origin
YAML Path; this is where the node was found or created within the DOM. Note
that Collector segments work against virtual DOMs, so the YAML Path of an
outer Collector will be virtual, relative to its parent at construction; when
nested, this will be a bare list index. Any NodeCoords in the virtual
container which point to real nodes in the DOM will have their own concrete
YAML Paths.
* YAMLPath instances now support nonmutating addition of individual segments
via the + operator. Whereas the append() method mutates the YAMLPath being
acted upon, + creates a new YAMLPath that is the original plus the new
segment. In both cases, the orignal YAMLPath's separator is retained during
both operations. As with .append(), new segments added via + must also be
properly escaped -- typically via path.escape_path_section -- before being
added.