Beautifulsoup4

* Instead of converting incoming data to Unicode and feeding it to the
lxml tree builder in chunks, Beautiful Soup now makes successive
guesses at the encoding of the incoming data, and tells lxml to
parse the data as that encoding. Giving lxml more control over the
parsing process improves performance and avoids a number of bugs and
issues with the lxml parser which had previously required elaborate
workarounds:

- An issue in which lxml refuses to parse Unicode strings on some
systems. [bug=1180527]

- A returning bug that truncated documents longer than a (very
small) size. [bug=963880]

- A returning bug in which extra spaces were added to a document if
the document defined a charset other than UTF-8. [bug=972466]

This required a major overhaul of the tree builder architecture. If
you wrote your own tree builder and didn't tell me, you'll need to
modify your prepare_markup() method.

* The UnicodeDammit code that makes guesses at encodings has been
split into its own class, EncodingDetector. A lot of apparently
redundant code has been removed from Unicode, Dammit, and some
undocumented features have also been removed.

* Beautiful Soup will issue a warning if instead of markup you pass it
a URL or the name of a file on disk (a common beginner's mistake).

* A number of optimizations improve the performance of the lxml tree
builder by about 33%, the html.parser tree builder by about 20%, and
the html5lib tree builder by about 15%.

* All find_all calls should now return a ResultSet object. Patch by
Aaron DeVore. [bug=1194034]

* The default XML formatter will now replace ampersands even if they
appear to be part of entities. That is, "<" will become
"<". The old code was left over from Beautiful Soup 3, which
didn't always turn entities into Unicode characters.

If you really want the old behavior (maybe because you add new
strings to the tree, those strings include entities, and you want
the formatter to leave them alone on output), it can be found in
EntitySubstitution.substitute_xml_containing_entities(). [bug=1182183]

* Gave new_string() the ability to create subclasses of
NavigableString. [bug=1181986]

* Fixed another bug by which the html5lib tree builder could create a
disconnected tree. [bug=1182089]

* The .previous_element of a BeautifulSoup object is now always None,
not the last element to be parsed. [bug=1182089]

* Fixed test failures when lxml is not installed. [bug=1181589]

* html5lib now supports Python 3. Fixed some Python 2-specific
code in the html5lib test suite. [bug=1181624]

* The html.parser treebuilder can now handle numeric attributes in
text when the hexidecimal name of the attribute starts with a
capital X. Patch by Tim Shirley. [bug=1186242]

* The Tag.select() method now supports a much wider variety of CSS
selectors.

- Added support for the adjacent sibling combinator (+) and the
general sibling combinator (~). Tests by "liquider". [bug=1082144]

- The combinators (>, +, and ~) can now combine with any supported
selector, not just one that selects based on tag name.

- Added limited support for the "nth-of-type" pseudo-class. Code
by Sven Slootweg. [bug=1109952]

* The BeautifulSoup class is now aliased to "_s" and "_soup", making
it quicker to type the import statement in an interactive session:

from bs4 import _s
or
from bs4 import _soup

The alias may change in the future, so don't use this in code you're
going to run more than once.

* Added the 'diagnose' submodule, which includes several useful
functions for reporting problems and doing tech support.

- diagnose(data) tries the given markup on every installed parser,
reporting exceptions and displaying successes. If a parser is not
installed, diagnose() mentions this fact.

- lxml_trace(data, html=True) runs the given markup through lxml's
XML parser or HTML parser, and prints out the parser events as
they happen. This helps you quickly determine whether a given
problem occurs in lxml code or Beautiful Soup code.

- htmlparser_trace(data) is the same thing, but for Python's
built-in HTMLParser class.

* In an HTML document, the contents of a <script> or <style> tag will
no longer undergo entity substitution by default. XML documents work
the same way they did before. [bug=1085953]

* Methods like get_text() and properties like .strings now only give
you strings that are visible in the document--no comments or
processing commands. [bug=1050164]

* The prettify() method now leaves the contents of <pre> tags
alone. [bug=1095654]

* Fix a bug in the html5lib treebuilder which sometimes created
disconnected trees. [bug=1039527]

* Fix a bug in the lxml treebuilder which crashed when a tag included
an attribute from the predefined "xml:" namespace. [bug=1065617]

* Fix a bug by which keyword arguments to find_parent() were not
being passed on. [bug=1126734]

* Stop a crash when unwisely messing with a tag that's been
decomposed. [bug=1097699]

* Now that lxml's segfault on invalid doctype has been fixed, fixed a
corresponding problem on the Beautiful Soup end that was previously
invisible. [bug=984936]

* Fixed an exception when an overspecified CSS selector didn't match
anything. Code by Stefaan Lippens. [bug=1168167]

* Skipped a test under Python 2.6 and Python 3.1 to avoid a spurious
test failure caused by the lousy HTMLParser in those
versions. [bug=1038503]

* Raise a more specific error (FeatureNotFound) when a requested
parser or parser feature is not installed. Raise NotImplementedError
instead of ValueError when the user calls insert_before() or
insert_after() on the BeautifulSoup object itself. Patch by Aaron
Devore. [bug=1038301]

* As per PEP-8, allow searching by CSS class using the 'class_'
keyword argument. [bug=1037624]

* Display namespace prefixes for namespaced attribute names, instead of
the fully-qualified names given by the lxml parser. [bug=1037597]

* Fixed a crash on encoding when an attribute name contained
non-ASCII characters.

* When sniffing encodings, if the cchardet library is installed,
Beautiful Soup uses it instead of chardet. cchardet is much
faster. [bug=1020748]

* Use logging.warning() instead of warning.warn() to notify the user
that characters were replaced with REPLACEMENT
CHARACTER. [bug=1013862]

* Fixed an html5lib tree builder crash which happened when html5lib
moved a tag with a multivalued attribute from one part of the tree
to another. [bug=1019603]

* Correctly display closing tags with an XML namespace declared. Patch
by Andreas Kostyrka. [bug=1019635]

* Fixed a typo that made parsing significantly slower than it should
have been, and also waited too long to close tags with XML
namespaces. [bug=1020268]

* get_text() now returns an empty Unicode string if there is no text,
rather than an empty bytestring. [bug=1020387]