All posts by Juan Manuel Schillaci

Nginx as an automatic reverse proxy

Nginx is a nice piece of software, an elegant webserver keeping things simple (although it has given me some headaches). On this case I'll show you how to setup a reverse proxy for any hostname on your internal/external network. A practical use case for this, could be the following
[PC] <-VPN-> [ VPN TERMINATION POINT ]     <-->[HOST A.INTRANET.LOCAL]
                                                                                                               <-->[HOST B.INTRANET.LOCAL]
                                                                                                               <-->[HOST C.INTRANET.LOCAL]
Lets say we are working remotely and had a VPN connection that is able to access a single linux box (VPN termination point), but we need to navigate to other hosts on the internal network i.e: A.INTRANET.LOCAL The solution to this problem is simple, but we need to make some assumptions:
  • The intranet has an internal DNS server capable of resolving INTRANET.LOCAL subdomains.
  • The websites we want to access are all accessible via hostname.
All we need to do is install nginx. On Ubuntu/Debian is as simple as:
$ sudo apt-get install nginx
Then put the following inside the /etc/nginx/sites-enabled/default file:  
server {
listen   80;
server_name  localhost;
access_log  /tmp/nginx.access.log;
 
location / {
resolver 10.47.4.109;
proxy_pass $scheme://$host;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
  Lets explain the tricky parts a little bit:
  • resolver 10.0.0.2: This is necessary as nginx does not use the standard dns resolution method (a.k.a resolv.conf) so we need to configure a dns server. On this case 10.0.0.2 is the intranet dns server.
  • proxy_pass $scheme://$host: This is simple, it redirects all incoming requests, to the same hosts it was originally intended to. The special variable $scheme contains the protcol (http, https) and the $host variable the hostname.
  • proxy_set_header Host $host: This sets the Host header on the requests, necessary to process a virtualhost directive on any webserver listening on that hostname. proxy_set_header X-Forwarded-For $remote_addr: This attachs the original remote address to the request.
Note: This configuration, as it is, it will work only for websites listening on port 80, you may have to adjust the listen port to accomdate to other requirements.
WARNING:  One has to be very carefull implementing this solution, as the nginx configuration will act as a proxy for *any* host on the internet. You need to make sure that is not exposed to the outside world and be aware that anyone knowing the ip address inside the intranet will be able to use it, so you are encourage to take securiy measures

Creating a bot for slack

Slack is one of the coolest and most versatile IM platforms available today, we use it all the time here at Devecoop as our primary channel for communication. One of the greatest things it has, is its integration capabilities with 3rd part services (i.e.: twitter, github, bitbucket, circleci, etc), that can be exploited right out of the box, without too much hussle. On this opportunity I will show you how you can create your own bot, using Slack's outgoing webhooks feature.

Lets get to work.

As all we need on our side is a small webserver , listening to requests from slack, we are going to start writing it using Python and its BaseHTTPServer module

  1. #!/usr/bin/python
  2.  
  3. import cgi
  4. import io
  5. from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
  6.  
  7. PORT_NUMBER = 3000
  8.  
  9. class SlackBotHandler(BaseHTTPRequestHandler):
  10. def do_something(self, text):
  11. with io.open('slacklog.txt', 'a') as file:
  12. file.write(text + "\n")
  13.  
  14. def do_POST(self):
  15. content_len = int(self.headers.getheader('content-length', 0))
  16. post_body = self.rfile.read(content_len)
  17. try:
  18. postvars = cgi.parse_qs(post_body, keep_blank_values=1)
  19. user_name = postvars.get('user_name')[0]
  20. text = postvars.get('text')[0]
  21. text = text.strip("!save &lt;").rstrip("&gt;")
  22.  
  23. self.send_response(200)
  24. self.send_header('Content-type','text/html')
  25. self.end_headers()
  26. self.do_something(text)
  27.  
  28. payload = '{"text" : "Your data has been save %s!"}' % user_name;
  29. except:
  30. payload = '{"text" : "Sorry %s Could not save your data!}' % user_name;
  31.  
  32. # Send the html message
  33. self.wfile.write(payload)
  34. return
  35.  
  36.  
  37. try:
  38. #Create a web server and define the handler to manage the
  39. #incoming request
  40. server = HTTPServer(('', PORTNUMBER), SlackBotHandler)
  41. print 'Started httpserver on port ' , PORTNUMBER
  42. server.serve_forever()
  43.  
  44. except KeyboardInterrupt:
  45. print '^C received, shutting down the web server'
  46. server.socket.close()

What this small script does is pretty simple. It fires an httpserver that listens on port 3000, then when it receives a post, the "doPOST" method handles it, writing the text it receives to a text file and returning a simple message to slack. All you need to do is execute this script and it will start serving and listening for slack events. You need a server with  a public IPAddress in order for this to work, alternativately you can use a service like Heroku. Lets say our hostname is example.com so the url we have to put on slack would be "http://example.com:3000·

The second part is far more simple. On the Slack website, go to the upper left corner and click on the dropdown link with the name of your team, then Click on "Configure integrations"-> "Outgoing webhooks" -> "Add Outgoing Webhooks Integration", the fields you need to fill are self explanAtory, the most important ones are what channel/s you want your hook to get called, the word/s that are going to be used to fire the hook, and the url/s it will call when matching those words.

In our case we will use this bot on any channel and configure the hook to be fired when hitting the "!save" keyword, and the url as we previously mention will be "http://example.com:3000.

All we need to do then is press save, go to a channel write a line using the magic "!save" word and enjoy our fresh new bot!

Note: In order to add more security, each hook we create generates a token, we should use this value if we want our server to reject anything that does not have this token on the body.

 

 

 

 

Setup multiple scenarios for e2e testing on django with django-remote-scenario

When we are testing an application (blackbox), either manually or on an automated fashion, often have the need to create different sets of data, for the different scenarios of each feature we want to test (Checkout this link: http://dannorth.net/introducing-bdd/). That is where django-remote-scenario comes to the rescue!.

I wrote this tool, because I needed to do e2e testing for an Angular application with a Django backend. I needed to create different sets of data, so a third party application could retrieve each one of them, via REST services at will. The idea is simple, create a "scenario file" for each scenario you want to test,and django remote scenario will translate it into an URL that can be remotely call to load the data into the database, and be ready to be consume.

Quickstart

Install django-remote-scenario::

pip install django-remote-scenario

Then add it to an exsiting django project::

INSTALLED_APPS = [
...
django_rs

You need add django_rs urls to your project url file like this::

urlpatterns = patterns('',
...
url(r'^drs/', include('django_rs.urls')),
..
)

To create custom scenarios, just create a directory inside your app named "scenarios" , then add as many files as scenarios you want to implement and create a init.py file to import them. Inside each of those files, you need to implement a main() function setting up the models you want to create for the scenario, you could create them by hand or use something like django_dynamic_fixtures.

Note: Your scenario is not limited to creating new models, you may also mock specific parts of the enviroment as well

Once everything is ready, start the server this way, this will enable the dynamic call of scenarios::

python manage.py rune2eserver initial_data.json

Note: You need to pass a initial fixture file with the barebones of your data.

It is also possible to pass a specific settings file, for testing purposes, in case you want to do the tests using a different database for example::

python manage.py rune2eserver initial_data.json --settings=demoproject.test_settings

To start using it, just go to the following url:

http://127.0.0.1:8000/drs/[APPLICATION]/[SCENARIO]

after doing that the database will be populated with the data you provided in your scenario. Take into account that, everytime you call an scenario, all the other data in the database is erased, except for the one in your initial_data fixture files, wich are loaded again, and also the one you pass as a parameter when you call the command.

Inside this repository you will find a demo Django project preconfigured with a simple scenario that sets up four objects. Use it like this:

First run the server:

$ python manage.py rune2eserver initial_data.json --settings=demoproject.test_settings

Then go to your browser and setup a scenario:

http://127.0.0.1:8000/drs/demoapp/scenario_1

You may also pass a parameter to avoid flushing the database on a specific call:

http://127.0.0.1:8000/drs/demoapp/scenario_1/?flush=0

Later you could see the results on the following url::

http://127.0.0.1:8000/demoapp/

E2E tests with django-casper

We often need to test our "javascript rich" Django application and the infamous TestClient provided with Django is not enough on this cases. Here is when django-casper comes to the rescue

First a brief introduction. Javascript has a great package named PhantomJs. PhantomJs is a headless webkit browser (yeap no need to open FF/Chrome for testing ala selenium!). CasperJS is a library on top of that to ease the testing from CasperJS website:

CasperJS is an open source navigation scripting &testing utility written in >Javascript for the PhantomJS WebKit headless browser and SlimerJS (Gecko).It >eases the process of defining a full navigationscenario and provides useful >high-level functions,methods & syntactic sugar for doing common tasks such as:

  • defining & ordering browsing navigation steps
  • filling & submitting forms
  • clicking & following links
  • capturing screenshots of a page (or part of it)
  • testing remote DOM
  • logging events
  • downloading resources, including binary ones
  • writing functional test suites, saving results as JUnit XML
  • scraping Web contents

django-casper is a sort of wrapper for casperjs, allowing us to run javacript/casper tests from Django's built-in test command, this is great, not only to facilitate the development process, but to take advantage of Django's own test runner to create mocks, stubs, fixtures, etc. for our front end testing.

Installation:

We will need a running python and node instance with django and casperjs installed respectively. If you don't have it already on your box here is a quick howto for GNU/Linux

Node

Install nvm

$ curl https://raw.github.com/creationix/nvm/v0.4.0/install.sh | sh

From nvm, install latest node version

$ nvm install latest

then open a new terminal or source ~/.zshrc or ~/.bashrc or whatever rc file for the shell you are using

Create virtualenv

Create a new virtualenv instance if you don't have virtualenv installed please refer to the official documentation http://virtualenv.readthedocs.org/en/latest/virtualenv.html Note: You can skip this step and install Django globally, but is not recommended

$ virtualenv django-casper && source django-casper/bin/activate

Now we are ready to install casperjs and django

Installing django

$ pip install django

Installing casperjs

$ npm install -g casperjs

Installing django-casper

It is possible to install django-casper from pip (pip install django-casper), but we are going to use the following method, in order to get the example code for tests.

$ git clone git@github.com:dobarkod/django-casper.git $ cd django-casper $ python setup.py install

Using it

Django-casper comes with a testproject that implements some test tests.

$ cd testproject $ python manage.py test testapp

This should run all the test included in the application.

Where are my tests?!!

Tests are divided in two parts, the django part where the backend stuff is prepared (fixture, backend mocks, etc) and the casper part where the actual tests are written. Lets see an example:

$ vim testapp/tests.py

from casper.tests import CasperTestCase
import os.path

from django.contrib.auth.models import User


class CasperTestTestCase(CasperTestCase):

    def test_that_casper_integration_works(self):
        self.assertTrue(self.casper(
            os.path.join(os.path.dirname(__file__),
            'casper-tests/test.js')))

On this file we have one test that in turn calls the casper library passing the test casper should run on this case casper-tests/test.js. Inside this django test, we could add new data to the database, and test the result on the casper-tests/test.js test.

Lets see the content of the casper-tests/test.js test file

 casper.test.comment('Casper+Django integration example');
 var helper = require('../../../casper/jslib/djangocasper.js');

 helper.scenario('/',
function() {
   this.test.assertSelectorHasText('em', 'django-casper',
 "There's a mention of django-casper on the page");
},
function() {
  this.click('a');
   this.test.assertSelectorHasText('#messages p', 'Good times!',
 "When the link is clicked, a message is added to the page");
}
 );

helper.run();

We can observe, that the test is opening the root "/" page (helper.scenario('/'.... ), and then asserting the content of the page on the first function. The second function, clicks on a link on the page, and asserts that a message is added to the page.

These are basic tests, I encourage you to give it a try. Also take a look a Django dynamic fixture, a library to create dynamic test data for your django tests.

Happy Testing!

Useful links

django-casper: https://github.com/dobarkod/django-casper

django-dynamic-fixture: https://github.com/paulocheque/django-dynamic-fixture

casperjs: http://casperjs.org/

virtualenv: http://www.virtualenv.org/en/latest/

nvm: https://github.com/creationix/nvm

Unignoring files in bazaar

Bazaar is a great tool for quickly start versioning a python project. For example if you have this one in particular:

my_project 
 - __init__.py 
 - my_module.py 
 - my_module.pyc 
 - main.py 
 - library.so

To start versioning, at the root level execute the following commands:

$ bzr init 
$ bzr commit -m "Initial commit"

Bazaar, by default, will ignore all .pyc files, so we don't have to worry about commit them by mistake. .pyc files are not the only ones being ingored by default. Bazaar will also ignore vim buffer files (.*swp), dynamically linked libraries (.so) and some others too. So, what if we need to "unignore" some of this default patterns?.

Here is what we should do

Just create a .bzrignore file at the top level of the project, and add the pattern you want to unignore, preceded by a ! mark, for example if we want to start versioning all .so files, we just need to add the following pattern

!*.so

If we check our repository status now it will say

unknown
library.so

Now we can add this file and start versioning it.

Happy "bazaaring"!