Scrapydd

Latest version: v0.7.5

Safety actively analyzes 621892 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 1 of 5

0.7.5

* docker image change to debian-python3.8 base image
* add grpc and restapi for node/server communication.
* add client module support both rest and grpc client, remove old asynchttp client
* add openapi doc.
* modify node register / login process.
* admin can delete node on webui.
* add node name field, will auto-populate with random name when creating.
* node.id change to bigint type, use twitter snowflake algorithm to generate new value.
* restructure some manager, receive a session parameter to give better control of caller.
* (Important) enable_authentication default value change to false.
* fix keys generating issue.
* fix jobs are not schedule concurrently across spiders ( a global counter).

0.7.3

* fix: docker runner AttributeError
* admin users page.
* move jobs page into admin pages.
* relate projects to owner user, add related authorization.
* fix: delete project error.
* upgrade dependency tornado v6.0, scrapy v2.1
* a creating empty project page and a separated page to upload package of existing project.
* a project info page.
* share database session instance in request processing context.
* fix: io error when uploading failed job log.

0.7.1

* Fix `scrapy package` command error.
* ProjectWorkspace subprocess cross-platform compatibility, including PIPE length limitation and encoding.
* Server-side Job killing timeout.
* Fix: upload stream file not closed issue.
* Webhook use new post mechanism.
* Add GetJob and GetJobItems rest apis.
* Dockerfile add only necessary files.
* Add new Runner2, support plugin system.

0.7.0

* add storage_version of Project entity
* add default_project_storage_version configuration, version with "2" support data stored by project.
* support project deleting.
* add ProjectPackage entity, store egg related information.
* add "runner_type" config, support "docker" runner type.
* add "runner_docker_image" config, docker image for runner.

0.6.5

* modify tag matching rule: none tag agent cannot run any spider tag against none.
* tag matching use spider's tag, will effect existing job when modify spider's tag after job been created.
* enhance authentication check of web views and apis.
* http stream tool python3 compatible.
* move agent relatived handler into handlers package.
* fix: job complete fail issue.
* Workspace list_spiders return list of str.
* add new unit tests.

0.6.3

* enhance security: add xsrf protection.
* password encrypted with SHA1.
* support both HMAC and Basic Authentication for API endpoints.
* fix error of node key page.
* remove scrapyd dependency.
* runner support SCRAPY_EXTRA_SETTINGS_MODULE env parameter.

Page 1 of 5

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.