Spruned

good news! spruned 0.0.3b5 comes with a lot of bugfixing and improvements.

- bloom filters: the default filter is the genesis address, new txids are no more notified by p2p peers.
- network: disconnect the peers not aligned to the best well known height (from electrum servers)
- estimatefee: a fee group is established gathering data from multiple electrum peers to provide an average median fee and avoid casual changes.
- scoring system: every peer now has a score, lower scores are disconnected, many unwanted behaviors can lower the scores
- caching: smarter cache hooks to reduce network load

- performances improvements on non verbose block downloads
- peers discovery and scoring, generally the p2p connection pool is more reliable
- bugfixes
- improved block caching
- loglevels adjustment
- global application speedup

On a raspberry zero the expected behaviour for an healthy connections pool in spruned (may take a bit to establish) on the bitcoin mainnet blockchain, with non verbose blocks, is the following:


2018-09-14 22:08:15,830 - p2p - INFO - Block 00000000000000000023cf82610d8cc7b929b91917653bd5e1bcaafb21d1df27 served in 1.3266s
2018-09-14 22:08:23,721 - p2p - INFO - Block 00000000000000000014916b46b8f5f3e722a5bcbd738adf550a9d765583ea73 served in 4.6273s
2018-09-14 22:08:29,499 - p2p - INFO - Block 0000000000000000002007fe9d65dab839839430852f9e449fd9f82dd2acd191 served in 3.2248s
2018-09-14 22:08:33,518 - p2p - INFO - Block 0000000000000000000173baf61960c70f87b5fea81ee0eb84759123a0e72df1 served in 3.1161s
2018-09-14 22:08:38,170 - p2p - INFO - Block 00000000000000000008c49c77029c568d1f81d065d5158a7f493f60785c2628 served in 1.8292s
2018-09-14 22:08:42,040 - p2p - INFO - Block 000000000000000000024ccfb7e1f25076e9806f0f024384dad89909b8f9c79a served in 1.2097s
2018-09-14 22:08:46,331 - p2p - INFO - Block 00000000000000000008cba9967e020fe782271e073b274f35d74ab056d7ba9e served in 1.3243s
2018-09-14 22:08:55,389 - p2p - INFO - Block 00000000000000000008b108f2df965b85b280b006e368e638f51e81a26c16fe served in 8.2317s
2018-09-14 22:09:00,315 - p2p - INFO - Block 0000000000000000001ce948fee7fd82629a9879172f36e6e75906a1e13a797f served in 3.3176s
2018-09-14 22:09:14,242 - p2p - INFO - Block 0000000000000000001ab81222886b6350bd38427fa112e45f466f13c2fc0868 served in 11.4722s
2018-09-14 22:09:20,615 - p2p - INFO - Block 000000000000000000275bd8ea62d34a88be77ad9fc87becaf5e42d68cb93f44 served in 3.7804s
2018-09-14 22:09:27,853 - p2p - INFO - Block 0000000000000000002575234fd5d0e64f3136279dedfe56133ee91117f2d558 served in 4.5819s
2018-09-14 22:09:32,679 - p2p - INFO - Block 00000000000000000018bf050790126eb480fd3dc4a972c262ead1efb566ca37 served in 0.4589s
2018-09-14 22:09:35,250 - p2p - INFO - Block 0000000000000000000c4e009379dac3e2ca6cf73b65bf444bbeadcf0ab40ba6 served in 0.8339s
2018-09-14 22:09:46,326 - p2p - INFO - Block 000000000000000000242fb80cb0a2a326615eb70d2b6ef5e414f975e9da860a served in 5.2715s
2018-09-14 22:09:50,992 - p2p - INFO - Block 000000000000000000144c8a7971d481aec0a2a919b86bec7b62a694dece599f served in 0.2485s
2018-09-14 22:09:55,140 - p2p - INFO - Block 000000000000000000129581394f545d6e92d02e60b38596a8aef379071b3ce6 served in 3.7976s
2018-09-14 22:09:57,822 - p2p - INFO - Block 00000000000000000025fb7aa34dec9d3b34d6a69f47d6a167142fe45ca7b0cb served in 0.4071s
2018-09-14 22:10:01,797 - p2p - INFO - Block 00000000000000000010a50966d4e94be3303c9be1ec6abaf4db887fce986b64 served in 3.1227s
2018-09-14 22:10:09,170 - p2p - INFO - Block 0000000000000000002074376e4e970004a743ba77996d731dc76475030b23d7 served in 6.0753s
2018-09-14 22:10:13,941 - p2p - INFO - Block 0000000000000000000950a95793a7df49498223d6740eb0602ee33cc1d5f465 served in 1.0358s
2018-09-14 22:10:16,873 - p2p - INFO - Block 00000000000000000016a28cb550844773f4902cf2341c950e7c097dc8864f75 served in 1.7823s
2018-09-14 22:10:20,663 - p2p - INFO - Block 00000000000000000025725c6f369e79102c22785a31e0bc4f9d23c1482f9f28 served in 3.2550s
2018-09-14 22:10:24,418 - p2p - INFO - Block 000000000000000000219684c0fff4354fc809100661529fe785ef3b308e7646 served in 1.2439s
2018-09-14 22:10:25,494 - p2p - INFO - Block 0000000000000000001acf094496f5a272ea3f63a1e5931a7a656bdca09333f7 served in 0.7833s
2018-09-14 22:10:32,608 - p2p - INFO - Block 00000000000000000007fb360457edf480fb3b013f85dcadb05e5e76250ceeda served in 6.2963s
2018-09-14 22:10:35,351 - p2p - INFO - Block 00000000000000000025a15f04d553d949d399620c6d138072a3bb8cda1c9c28 served in 0.1448s
2018-09-14 22:10:44,016 - p2p - INFO - Block 0000000000000000000b9dd0be851d2d152650a4ebd69437c8759432fa124f17 served in 8.4693s
2018-09-14 22:10:49,692 - p2p - INFO - Block 0000000000000000002083418fecf1a0b18d6181ecf657eaf32cddeb0e2c457f served in 1.9351s
2018-09-14 22:10:55,623 - p2p - INFO - Block 000000000000000000208e9ca72f117fd88db5bc458155eb7e874b0116446185 served in 3.1718s
2018-09-14 22:10:58,927 - p2p - INFO - Block 00000000000000000012efa79ee5de7149564bfc51e7bc805821ae59e555162e served in 0.8526s
2018-09-14 22:11:03,326 - p2p - INFO - Block 0000000000000000000076d945ca3b37d7e4469c39275855882a7c9b048d666f served in 3.4746s
2018-09-14 22:11:11,638 - p2p - INFO - Block 0000000000000000000e4bc5f4d5dc1195687173de35b016b0829221e3a6dc8d served in 4.0430s
2018-09-14 22:11:15,551 - p2p - INFO - Block 00000000000000000027c756c58b7b6f0547b73f87ca0a33c0bafc16aaed6481 served in 0.7130s
2018-09-14 22:11:17,212 - p2p - INFO - Block 0000000000000000000ec77fafb4fdbe426be01acd7b4fcdcf90be6ee9b5c14f served in 0.6503s
2018-09-14 22:11:18,980 - p2p - INFO - Block 000000000000000000248dae091988a7a85f0e96324f1433f072731d2823dbd8 served in 0.6084s
2018-09-14 22:11:20,248 - p2p - INFO - Block 000000000000000000112df0ff2fb9ebfde59cbccd8ad75e3d00050801d03cac served in 0.6265s
2018-09-14 22:11:21,501 - p2p - INFO - Block 0000000000000000001277a2c765f728be3dc2bc851755bb3d7b1e4d2f092618 served in 0.4725s
2018-09-14 22:11:23,259 - p2p - INFO - Block 000000000000000000275fd719bffd6030c62bb797914e88ff3148aa638e94b9 served in 0.9213s
2018-09-14 22:11:26,400 - p2p - INFO - Block 00000000000000000014d23aacb7a3352c140dd0d78be37a542683893a3d3803 served in 1.4890s
2018-09-14 22:11:31,336 - p2p - INFO - Block 0000000000000000000b78f225be0474be732910e0939c6a25cbcbb57c4e1e81 served in 2.8372s
2018-09-14 22:11:34,762 - p2p - INFO - Block 0000000000000000001b4f0c376b279d291b03acc5c8ac6bfc2fe9be88995ae4 served in 2.6523s
2018-09-14 22:11:39,830 - p2p - INFO - Block 0000000000000000001700afee9531eb988f88dc4381b4ddd074b7f0f1a4df94 served in 1.7372s
2018-09-14 22:11:45,138 - p2p - INFO - Block 0000000000000000000895948986bdea42a08804d6aefb5f4970e99a6146ec02 served in 1.7867s
2018-09-14 22:11:59,602 - p2p - INFO - Block 0000000000000000002196a0b8b916e1099497fc14ac610fedd09c6c56af5ec2 served in 8.9184s
2018-09-14 22:12:06,099 - p2p - INFO - Block 00000000000000000020a37db7d78b140cafac1b87cc3fdd85c9833d20a11e97 served in 1.6268s
2018-09-14 22:12:17,475 - p2p - INFO - Block 0000000000000000001ed139a204a70b98f9c9c052c3c6cc7af7deb5659b3be3 served in 1.9709s
2018-09-14 22:12:19,925 - p2p - INFO - Block 0000000000000000000c3b4ee73e5a176c463fa004414df9c5be8f539764dbc9 served in 0.3514s
2018-09-14 22:12:23,122 - p2p - INFO - Block 00000000000000000022869966d1cd97943d51d37d92fc16434953804374ce74 served in 2.1919s
2018-09-14 22:12:26,489 - p2p - INFO - Block 0000000000000000001bdd33d6694124c652397da61ed07e14801235866487fe served in 2.5332s
2018-09-14 22:12:29,549 - p2p - INFO - Block 00000000000000000010b1460409458cbda4bfdba880716a9eef9e47d5842b3d served in 0.2512s
2018-09-14 22:12:32,856 - p2p - INFO - Block 0000000000000000001ab229b37ab88513342f28cfa0c32263552fd99276b969 served in 2.9045s



Optimization of verbose blocks may be addressed in future releases.

P2P Improvements:
- Use getaddr to obtain, persist and connect to new peers
- Ban peers after too many failures

This release comes with:

- a minimal subset of the getpeerinfo API emulation

- improvements on the RPC emulation that should adhere best on the original
- getblock error code bugfix
- contributions (systemd script)

The release comes with a huge refactory on the dependencies side.

- Fix a localstorage bug that caused spruned to fail on the first execution.
- Drop github dependencies
- Bring pip distribution working (pip install spruned) up & running again
- Add third party code & licenses into the main codebase
- Add connectrum UT
- Use pycoin pip distribution, add pycoin patches to dependency module

This release comes with:

- verbose getrawtransaction support
- full merkle proof validation on getrawtransaction & gettxout
- dynamic electrum servers update
- requirements.txt must be installed explicitly
- remove useless requirement
- increased test coverage & code cleaning

http://spruned.readthedocs.io/en/latest/