Menu

linux/

Modify response body with mitmproxy 2

Run python script with mitmproxy:

mitmproxy --ignore ws.sitesupport.net:443 -s script.py

Script example:

from mitmproxy.script import concurrent

OLD = """var totalExpGems=0;"""
NEW = """var totalExpGems=0;debugger;"""


@concurrent
def response(flow):
    if "gem_finder" in flow.request.path:
        flow.response.headers["XX"] = "PATCHED"
        body = flow.response.content.decode("utf-8")
        if OLD in body:
            flow.response.content = body.replace(OLD, NEW).encode("utf-8")
            flow.response.headers["XXX"] = "PATCHED"
· 1 Jul 2017

Use mitmproxy with GNOME Web (Epiphany)

To use mitmproxy with Epiphany you need to install CA certificate from mitmproxy. To do that, start mitmproxy and navigate to mitm.it. Download *.pem certificate (available under Other link). Install mitmproxy certificate with command:

sudo trust anchor mitmproxy-ca-cert.pem 

With this command you can also install any custom certificate to use in Epiphany.

· 1 Jul 2017

Get size of the subfolders

Use this command to print size of the subfolders:

du -sh *

If you want to exclude Permission denied errors, for example:

du -sh * 2>/dev/null
· 1 May 2017

Test context of the rendered template with RequestFactory in Django

When writing tests for the Django’s view functions I prefer to use RequestFactory instead of the django test client. Tests that use RequestFactory are easier to understand and they are faster. For example, you need to test a simple view function:

def incoming_queue(request):
    if request.method == "POST":
        pass
    else:
        incoming_queue_list = UsageHistory.objects.filter(company=request.user.agent.company, queued=True)
        context = {
            "incoming_queue_list": incoming_queue_list,
            "page": "queue",
            "agent_available_url": reverse("agent_available")
        }
        return render(request, 'agents/incoming_queue.html', context)

If you want to test the context of the rendered template, it is easy to do with django test client as it attaches it to the response. If you want to use RequestFactory for that purpose, there is no easy way to do it. It would be comfortable to have something like assertTemplateContextIn(expected_context) method, that checks that expected_context was used during the template rendering. The following code was inspired by assertTemplateUsed method:

from copy import copy

from django.db.models.query import QuerySet
from django.test.signals import template_rendered
from django.test.utils import ContextList


class TemplateContextTestCase(object):
    def assertTemplateContextIn(self, expected_context=None):
        """
        Asserts that the template was rendered with the context that includes
        expected_context. Example:
            with self.assertTemplateContextIn(expected_context):
                vuew_function(self.request)
        """
        return _AssertTemplateContext(self, expected_context)


class _AssertTemplateContext(object):
    def __init__(self, test_case, expected_context):
        self.test_case = test_case
        self.expected_context = expected_context
        self.context = ContextList()

    def on_template_render(self, sender, signal, template, context, **kwargs):
        self.context.append(copy(context))

    def test(self):
        missing = []
        mismatched = []
        for key in self.expected_context.keys():
            if key in self.context:
                if self.context[key] != self.expected_context[key]:
                    match = False
                    # Compare QuerySets
                    if isinstance(self.context[key], QuerySet) and isinstance(self.expected_context[key], QuerySet):
                        if self.context[key].model == self.expected_context[key].model:
                            if list(self.context[key].values_list("id", flat=True)) == \
                               list(self.expected_context[key].values_list("id", flat=True)):
                                match = True
                    if not match:
                        mismatched.append(key)
            else:
                missing.append(key)
        return missing, mismatched

    def __enter__(self):
        template_rendered.connect(self.on_template_render)
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        template_rendered.disconnect(self.on_template_render)
        if exc_type is not None:
            return

        missing, mismatched = self.test()
        message = ""
        if missing:
            message = message + "Missing keys: " + ", ".join(x for x in missing) + "\n"
        if mismatched:
            for item in mismatched:
                message = message + "Context key mismatch '%s': %s != %s \n" % (
                    item, self.expected_context[item], self.context[item]
                )
        if message:
            self.test_case.fail(message)

We often pass querysets instances to the context of the rendered template. The code above checks if 2 QuerySets are the same by comparing their models and ids of the elements. TemplateContextTestCase class can be used to extend TestCase class:

from django.test import TestCase, RequestFactory

from yourapp.tests.helpers import TemplateContextTestCase
from django.core.urlresolvers import reverse


class TestIncomingQueue(TestCase, TemplateContextTestCase):
    def test_context(self):
        agent = AgentFactory()
        self.request.user = agent.user
        expected = {
            "page": "queue",
            "agent_available_url": reverse("agent_available"),
            "incoming_queue_list": UsageHistory.objects.none()
        }
        with self.assertTemplateContextIn(expected):
            incoming_queue(self.request)
· 1 Mar 2017

How to check if package is installed

To check if ca-certificates package is installed run this command:

dpkg -s ca-certificates
· 1 Mar 2017

Print exit code of the last command

To print exit code of the last command type:

echo $?
· 1 Mar 2017

Show raw SQL queries in PostgreSQL

If you want to inspect what queries exactly Django ORM sends to the PostgreSQL, you can do it with postgres logging. The first step is to enable logging. Add

log_statement = 'all'

to the file:

sudo vim /etc/postgresql/9.4/main/postgresql.conf

Then you are able to see raw SQL queries with command:

sudo tail -f /var/log/postgresql/postgresql-9.4-main.log
· 1 Jan 2017

How to fix ERROR: new encoding (UTF8) is incompatible

Sometimes I’m unable to create a database in UTF8 encoding because the template database is in the ASCII encoding. To fix it we need to recreate template database in UTF8. Start psql console:

psql -U postgres

Run the following commands:

UPDATE pg_database SET datistemplate = FALSE WHERE datname = 'template1';

DROP DATABASE template1;

CREATE DATABASE template1 WITH TEMPLATE = template0 ENCODING = 'UNICODE';

UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template1';

\c template1

VACUUM FREEZE;
· 1 Jan 2016

Increase watches limit

Open configuration file:

sudo vim /etc/sysctl.d/90-override.conf

Add the following line to the end of the file to increase watches limit:

fs.inotify.max_user_watches = 524288
· 1 Jan 2016

How to format USB stick as FAT32

First step is to find the name of the USB stick in the system:

sudo fdisk -l

In my case the name of the device is /dev/sdc If the USB stick is mounted, unmount it with command:

sudo umount /dev/sdc

Finally, run the command to format USB stick:

mkdosfs -F 32 -I /dev/sdc
· 1 Jan 2016