Asynchronously Prototyping

Self signed TLS server and client using Twisted Prerequisites openssl twisted & pyOpenSSL modules- pip install twisted[tls] treq module - pip install treq Basic knowledge of Twisted TCP servers/clients Generate self signed certificate Generate the server's private key using a secret, which is SuperSecretPassword in this case. openssl genrsa -aes256 -passout pass:SuperSecretPassword -out server.key 2048 Perform a CSR (certificate signing request). Ensure the FQDN (fully qualified domain name) matches the hostname of the server, otherwise the server won't be properly validated. openssl req -new -key server.key -passin pass:SuperSecretPassword -out server.csr # Common Name (e.g. server FQDN or YOUR name) []:localhost openssl x509 -req -passin pass:SuperSecretPassword -days 1024 -in server.csr -signkey server.key -out server.crt For development purposes, remove the password from the certificate. openssl rsa -in server.key -out server_no_pass.key -passin pas

WSGI and Scrapy A common question on Scrapy Stackoverflow is "How to use Scrapy with Flask, Django, or any other Python web framework?" Most are used to using the Scrapy’s generated projects and cli options, which make crawling a breeze, but are confused when trying to integrate Scrapy into a WSGI web framework. A common traceback encountered is ReactorNotRestartable , which stems from the underlaying Twisted framework. This occurs because, unlike asyncio or Tornado, Twisted’s eventloop/reactor cannot be restarted once stopped (the reason is a bit out of scope). So it becomes apparent that the trick to integrating Scrapy and WSGI frameworks involves being able to tame Twisted. Luckily, integrating async Twisted code with synchronous code has become quite easy and is only getting easier. In this post, the following will be demonstrated: Embed a crawler in a WSGI app and run it using Twisted’s twist web WSGI server. Embed a crawler in a WSGI app and run it any WSGI serve