Streamparse

This is the final release of streamparse 3.0.0. The developer preview versions of this release have been used extensively by many people for months, so we are quite confident in this release, but please let us know if you encounter any issues.

You can install this release via pip with `pip install streamparse==3.0.0`.

Highlights
- Topologies are now specified via a [Python Topology DSL](http://streamparse.readthedocs.org/en/master/topologies.html#topology-dsl) instead of the Clojure Topology DSL. This means you can/must now write your topologies in Python! Components can still be written in any language supported by Storm, of course. (Issues 84 and 136, PR 199, 226)
- When `log.path` is not set in your `config.json`, pystorm will no longer issue warning about how you should set it; instead, it will automatically set up a `StormHandler` and log everything directly to your Storm logs. This is really handy as in Storm 1.0 there's support through the UI for searching logs.
- The `--ackers` and `--workers` settings now default to the number of worker nodes in your Storm environment instead of 2.
- Added `sparse slot_usage` command that can show you how balanced your topologies are across nodes. This is something that isn't currently possible with the Storm UI on its own. (PR 218)
- Now fully Python 3 compatible (and tested on up to 3.5), because we rely on [fabric3](https://pypi.python.org/pypi/Fabric3) instead of plain old [fabric](http://www.fabfile.org/) now. (4acfa2f)
- Now rely on [pystorm](https://github.com/pystorm/pystorm) package for handling Multi-Lang IPC between Storm and Python. This library is essentially the same as our old `storm` subpackage with a few enhancements (e.g., the ability to use MessagePack instead of JSON to serialize messages). (Issue #174, Commits aaeb3e9 and 1347ded)

:warning: API Breaking Changes :warning:
- Topologies are now specified via a [Python Topology DSL](http://streamparse.readthedocs.org/en/master/topologies.html#topology-dsl) instead of the Clojure Topology DSL. This means you can/must now write your topologies in Python! Components can still be written in any language supported by Storm, of course. (Issues 84 and 136, PR 199, 226)
- The deprecated `Spout.emit_many` method has been removed. (pystorm/pystorm004dc27)
- As a consequence of using the new Python Topology DSL, all Bolts and Spouts that emit anything are expected to have the `outputs` attribute declared. It must either be a list of `str` or `Stream` objects, as described in the docs.
- We temporarily removed the `sparse run` command, as we've removed all of our Clojure code, and this was the only thing that had to still be done in Clojure. (Watch issue 213 for future developments)
- `ssh_tunnel` has moved from `streamparse.contextmanagers` to `streamparse.util`. The `streamparse.contextmanagers` module has been removed.
- The `ssh_tunnel` context manager now returns the hostname and port that should be used for connecting nimbus (e.g., `('localhost', 1234)` when `use_ssh_for_nimbus` is `True` or unspecified, and `('nimbus.foo.com', 6627)` when `use_ssh_for_nimbus` is `False`).
- `need_task_ids` defaults to `False` instead of `True` in all `emit()` method calls. If you were previously storing the task IDs that your tuples were emitted to (which is pretty rare), then you must pass `need_task_ids=True` in your `emit()` calls. This should provide a little speed boost to most users, because we do not need to wait on a return message from Storm for every emitted tuple.
- Instead of having the `log.level` setting in your `config.json` influence the root logger's level, only your component (and its `StormHandler` if you haven't set `log.path`)'s levels will be set.
- When `log.path` is not set in your `config.json`, pystorm will no longer issue warning about how you should set it; instead, it will automatically set up a `StormHandler` and log everything directly to your Storm logs. This is really handy as in Storm 1.0 there's support through the UI for searching logs.
- The `--par` option to `sparse submit` has been remove. Please use `--ackers` and `--workers` instead.
- The `--ackers` and `--workers` settings now default to the number of worker nodes in your Storm environment instead of 2.

Features
- Added `sparse slot_usage` command that can show you how balanced your topologies are across nodes. This is something that isn't currently possible with the Storm UI on its own. (PR 218)
- Can now specify `ssh_password` in `config.json` if you don't have SSH keys setup. Storing your password in plaintext is not recommended, but nice to have for local VMs. (PR 224, thanks motazreda)
- Now fully Python 3 compatible (and tested on up to 3.5), because we rely on [fabric3](https://pypi.python.org/pypi/Fabric3) instead of plain old [fabric](http://www.fabfile.org/) now. (4acfa2f)
- Now remove `_resources` directory after JAR has been created.
- Added `serializer` setting to `config.json` that can be used to switch between JSON and msgpack pack serializers (PR 238). Note that you cannot use the msgpack serializer unless you also include a Java implementation in your topology's JAR such as the [one provided by Pyleus](https://github.com/Yelp/pyleus/blob/develop/topology_builder/src/main/java/com/yelp/pyleus/serializer/MessagePackSerializer.java), or the one being added to Storm in apache/storm#1136. (PR 238)
- Added support for custom log filenames (PR 234 — thanks kalmanolah)
- Can now set environment-specific `options`, `acker_count`, and `worker_count` settings, to avoid constantly passing all those pesky options to `sparse submit`. (PR 265)
- Added option to disable installation of virtualenv was stilling allowing their use, `install_virtualenv`. (PR 264).
- The Python Topology DSL now allows topology-level config options to be set via the `config` attribute of the `Topology` class. (Issue 276, PRs 284 and 289)
- Can now pass any valid YAML as a value for `sparse submit --option` (Issue 280, PR 285)
- Added `--override_name` option to `kill`, `submit`, and `update_virtualenv` commands so that you can deploy the same topology file multiple times with different overridden names. (Issue 207, PR 286)

Fixes
- `sparse slot_usage`, `sparse stats`, and `sparse worker_uptime` are much faster as we've fixed an issue where they were creating _many_ SSH subprocesses.
- All commands that must connect to the Nimbus server now properly use SSH tunnels again.
- The output from running `pip install` is now displayed when submitting your topology, so you can see if things get stuck.
- `sparse submit` should no longer sporadically raise exceptions about failing to create SSH tunnels (PR 242).
- `sparse submit` will no longer crash when your provide a value for `--ackers` (PR 241).
- pin pystorm version to `>=2.0.1` (PR 230)
- `sparse tail` now looks for `pystorm` named filenames (9339908)
- Fixed typo that caused crash in `sparse worker_uptime` (7085804)
- Added back `sparse run` (PR 244)
- `sparse run` should no longer crash when searching for the version number on some versions of Storm. (Issue 254, PR 255)
- `sparse run` will no longer crash due to PyYAML dumping out `!!python/unicode` garbage into the YAML files. (Issue 256, PR 257)
- A `sparse run` TypeError with Python 3 has been fixed. (e232224)
- `sparse update_virtualenv` will no longer ignore the `virtualenv_flags` setting in `config.json`. (Issue 281, PR 282)
- `sparse run` now supports named streams on Storm 1.0.1+ (PR 260)
- No longer remove non-topology-specific logs with `sparse remove_logs` (45bd005)
- `sparse tail` will now find logs in subdirectories for Storm 1.0+ compatibility (Issue 268, PR 271)

Other Changes
- Now rely on [pystorm](https://github.com/pystorm/pystorm) package for handling Multi-Lang IPC between Storm and Python. This library is essentially the same as our old `storm` subpackage with a few enhancements (e.g., the ability to use MessagePack instead of JSON to serialize messages). (Issue #174, Commits aaeb3e9 and 1347ded)
- All Bolt, Spout, and Topology-related classes are all available directly at the `streamparse` package level (i.e., you can just do `from streamparse import Bolt` now) (Commit b9bf4ae).
- `sparse kill` now will kill inactive topologies. (Issue 156)
- All examples now use the Python DSL
- The Kafka-JVM example has been cleaned up a bit, so now you can click on Storm UI log links and they'll work.
- Docs have been updated to reflect latest Leiningen installation instructions. (PR 261)
- A broken link in our docs was fixed. (PR 273)
- JARs are now uploaded before killing the running topology to reduce downtime during deployments (PR 277)
- Switched from PyYAML to ruamel.yaml (18fd2e9)
- Added docs for handling multiple streams and groupings (Issue 252, 344ce8c)
- Added VPC deployment docs (Issue 134, d2bd1ac)

This is the fourth developer preview release of streamparse 3.0. In addition to having been extensively tested in production, this version also is the first in the 3.0 line that has `sparse run` back in it. However it is only supports on Storm 0.10.0+ and requires you to add `[org.apache.storm/flux-core "0.10.0"]` to your dependencies in your `project.clj`, because it uses Storm's new Flux library to start the local cluster.

You can install this release via pip with `pip install --pre streamparse==3.0.0.dev3`. It will not automatically install because it's a pre-release.

:warning: API Breaking Changes :warning:

In addition to those outlined in the [3.0.0dev0](https://github.com/Parsely/streamparse/releases/tag/v3.0.0.dev0) and [3.0.0dev1](https://github.com/Parsely/streamparse/releases/tag/v3.0.0.dev1) release notes, this release introduces the following backwards incompatible changes from pinning our pystorm version to 3.0+:
- `need_task_ids` defaults to `False` instead of `True` in all `emit()` method calls. If you were previously storing the task IDs that your tuples were emitted to (which is pretty rare), then you must pass `need_task_ids=True` in your `emit()` calls. This should provide a little speed boost to most users, because we do not need to wait on a return message from Storm for every emitted tuple.
- Instead of having the `log.level` setting in your `config.json` influence the root logger's level, only your component (and its `StormHandler` if you haven't set `log.path`)'s levels will be set.
- When `log.path` is not set in your `config.json`, pystorm will no longer issue warning about how you should set it; instead, it will automatically set up a `StormHandler` and log everything directly to your Storm logs. This is really handy as in Storm 1.0 there's support through the UI for searching logs.

Features
- Added back `sparse run` (PR 244)

This is the third developer preview release of streamparse 3.0. Unlike when we released the previous two, this one **has** been tested extensively in production, so users should feel more confident using it. It's still missing `sparse run`, which will try to fix before the final release.

You can install this release via pip with `pip install --pre streamparse==3.0.0.dev2`. It will not automatically install because it's a pre-release.

:warning: API Breaking Changes :warning:

These are outlined in the [3.0.0dev0](https://github.com/Parsely/streamparse/releases/tag/v3.0.0.dev0) and [3.0.0dev1](https://github.com/Parsely/streamparse/releases/tag/v3.0.0.dev1) release notes.

Features
- Added `serializer` setting to `config.json` that can be used to switch between JSON and msgpack pack serializers (PR 238). Note that you cannot use the msgpack serializer unless you also include a Java implementation in your topology's JAR such as the [one provided by Pyleus](https://github.com/Yelp/pyleus/blob/develop/topology_builder/src/main/java/com/yelp/pyleus/serializer/MessagePackSerializer.java), or the one being added to Storm in apache/storm#1136. (PR 238)
- Added support for custom log filenames (PR 234 — thanks kalmanolah)

Fixes
- `sparse submit` should no longer sporadically raise exceptions about failing to create SSH tunnels (PR 242).
- `sparse submit` will no longer crash when your provide a value for `--ackers` (PR 241).
- pin pystorm version to `>=2.0.1` (PR 230)
- `sparse tail` now looks for `pystorm` named filenames (9339908)
- Fixed typo that caused crash in `sparse worker_uptime` (7085804)

This is the second developer preview release of streamparse 3.0. It has **not been tested extensively in production** yet, so we are looking for as much feedback as we can get from users who are willing to test it out.

You can install this release via pip with `pip install --pre streamparse==3.0.0.dev1`. It will not automatically install because it's a pre-release.

:warning: API Breaking Changes :warning:

In additions to those outlined in the [3.0.0dev0 release notes](https://github.com/Parsely/streamparse/releases/tag/v3.0.0.dev0), we've made a few more changes.
- `ssh_tunnel` has moved from `streamparse.contextmanagers` to `streamparse.util`. The `streamparse.contextmanagers` module has been removed.
- The `ssh_tunnel` context manager now returns the hostname and port that should be used for connecting nimbus (e.g., `('localhost', 1234)` when `use_ssh_for_nimbus` is `True` or unspecified, and `('nimbus.foo.com', 6627)` when `use_ssh_for_nimbus` is `False`).

Fixes
- `sparse slot_usage`, `sparse stats`, and `sparse worker_uptime` are much faster as we've fixed an issue where they were creating _many_ SSH subprocesses.
- All commands that must connect to the Nimbus server now properly use SSH tunnels again.
- The output from running `pip install` is now displayed when submitting your topology, so you can see if things get stuck.

This is the first developer preview release of streamparse 3.0. It has **not been tested extensively in production** yet, so we are looking for as much feedback as we can get from users who are willing to test it out.

You can install this release via pip with `pip install --pre streamparse==3.0.0.dev0`. It will not automatically install because it's a pre-release.

:warning: API Breaking Changes :warning:
- Topologies are now specified via a [Python Topology DSL](http://streamparse.readthedocs.org/en/master/topologies.html#topology-dsl) instead of the Clojure Topology DSL. This means you can/must now write your topologies in Python! Components can still be written in any language supported by Storm, of course. (Issues 84 and 136, PR 199, 226)
- The deprecated `Spout.emit_many` method has been removed. (pystorm/pystorm004dc27)
- As a consequence of using the new Python Topology DSL, all Bolts and Spouts that emit anything are expected to have the `outputs` attribute declared. It must either be a list of `str` or `Stream` objects, as described in the docs.
- We temporarily removed the `sparse run` command, as we've removed all of our Clojure code, and this was the only thing that had to still be done in Clojure. (Watch issue 213 for future developments)

Features
- Added `sparse slot_usage` command that can show you how balanced your topologies are across nodes. This is something that isn't currently possible with the Storm UI on its own. (PR 218)
- Can now specify `ssh_password` in `config.json` if you don't have SSH keys setup. Storing your password in plaintext is not recommended, but nice to have for local VMs. (PR 224, thanks motazreda)
- Now fully Python 3 compatible (and tested on up to 3.5), because we rely on [fabric3](https://pypi.python.org/pypi/Fabric3) instead of plain old [fabric](http://www.fabfile.org/) now. (4acfa2f)
- Now remove `_resources` directory after JAR has been created.

Other Changes
- Now rely on [pystorm](https://github.com/pystorm/pystorm) package for handling Multi-Lang IPC between Storm and Python. This library is essentially the same as our old `storm` subpackage with a few enhancements (e.g., the ability to use MessagePack instead of JSON to serialize messages). (Issue #174, Commits aaeb3e9 and 1347ded)
- All Bolt, Spout, and Topology-related classes are all available directly at the `streamparse` package level (i.e., you can just do `from streamparse import Bolt` now) (Commit b9bf4ae).
- `sparse kill` now will kill inactive topologies. (Issue 156)
- All examples now use the Python DSL
- The Kafka-JVM example has been cleaned up a bit, so now you can click on Storm UI log links and they'll work.

This minor release adds support for specifying `ui.port` in `config.json` to make the `sparse stats` and `sparse worker_uptime` commands work when `ui.port` is not set to the default 8080.