Fully Unroot Custom Android ROMs to run Banking Apps

Certain banking apps and e-wallets refuse to start on custom ROMs. Unrooting doesn't seem to help much. The issue seems to arise from "insecure" settings in the ROM's properties file. To reconfirm you can use the Rootbeer app. I suspect many apps use their library to check for signs of root access.

If your ROM has "insecure" property settings, you can edit them to fully unroot the ROM via different properties files. For some ROMs this file is inside the boot image and you will need to extract and repack it. For other ROMs, like LineageOS, the boot image just includes a link to the /system partition and you can directly change that file.

These settings in default.prop can prevent banking apps from working:

ro.secure=0  # should be 1
ro.debuggable=1  # should be 0

If you are using LineageOS (or some related ROM) you can sometimes edit those values directly on /system:

$ mount -o rw,remount /system
$ nano /system/etc/prop.default

So much for the simple cases. If the properties file is located inside the boot image, you can follow those steps to unpack and update the boot image using a command line tool, magiskboot, that comes with Magisk.

1. Using the ADB tool on your computer, become root

$ adb root

2. Download Magisk, find magiskboot, copy to phone and change permissions

$ adb push Magisk-v19.3/arm/magiskboot /data/local/tmp

3. Shell into phone and find boot partition

$ adb shell
$ ls -l /dev/block/platform/soc/*/by-name/

4. Dump boot partition

cd /data/local/tmp
chmod 555 magiskboot
dd if=/dev/block/mmcblk0p21 of=boot.img

5. Unpack boot partition to current dir

mkdir repack; cd repack
../magiskboot unpack ../boot.img

6. Dump default.prop, make necessary edits and re-add to ramdisk

../magiskboot cpio ramdisk.cpio "extract default.prop default.prop"
nano default.prop  # make required edits and save.
../magiskboot cpio ramdisk.cpio "add 750 default.prop default.prop"

7. Repack boot image and write to partition

../magiskboot repack ../boot.img ../new-boot.img
dd if=new-boot.img of=/dev/block/mmcblk0p21

Resources

Update all macOS Apps with a Single Command

Updates are important. While not as great as on Linux, you can use different package managers to manage all installed software and easily keep applications updated. Here the command I use. Requires homebrew, homebrew-cask-upgrade and mas.

function update-all {
    # update cli homebrew
    brew update
    brew upgrade
    brew prune
    brew cleanup

    # Homebrew cask (via https://github.com/buo/homebrew-cask-upgrade)
    brew cu -a -y --cleanup
    brew cleanup --force -s && rm -rf $(brew --cache)

    # Node
    npm update -g

    # Apple App store
    mas upgrade
    softwareupdate --install --all
}

 

Optimize Spamassassin Detection Rate

Email is a terrible way to communicate and should be avoided where possible. Unfortunately it is also the lowest common denominator on the web and will continue to be for the near future.

In the early days of the internet it was easy to run your own mailserver. Due to the absurd quantity of spam this task got increasingly harder and many tech-savvy people gave up and switched to Gmail or other services. This is a pity because a decentralized email infrastructure is harder to surveil, subpoena or shut down. I encourage everyone to run their own mail service if possible.

In this guide I will summarize the steps needed to get an effective spamassassin (SA) setup. Continue reading

Shell Function to Remove all Metadata from PDF

A handy function to remove all metadata from a PDF file. When done it will show all the remaining metadata for inspection. Needs pdftk and exiftool installed.

Combines commands from here and here. Good job, guys.

clean_pdf() {
 pdftk $1 dump_data | \
  sed -e 's/\(InfoValue:\)\s.*/\1\ /g' | \
  pdftk $1 update_info - output clean-$1
 
 exiftool -all:all= clean-$1
 exiftool -all:all clean-$1
 exiftool -extractEmbedded -all:all clean-$1
 qpdf --linearize clean-$1 clean2-$1
 
 pdftk clean2-$1 dump_data
 exiftool clean2-$1
 pdfinfo -meta clean2-$1
}

After adding this snippet in ~/.profile or copy and pasting it in the shell, you can just run

clean_pdf my-unclean.pdf

Incremental FTP backups

If you happen to only have FTP access to a server or account (CPanel) you're looking after, LFTP is an efficient tool to keep incremental backups. This will make hard links of the previous backup and updated it, copying and storing only changed files.

#!/usr/bin/env bash
username='xxx'
password='xxx'
host='ftp.host'
localBackupDir='/backups/host'
remoteDir='/public_html/'
cd $localBackupDir
rm -rf backup.3
mv backup.2 backup.3
mv backup.1 backup.2
mv backup.0 backup.1
cp -al backup.1 backup.0 #-al or -r
lftp -e "set ssl:verify-certificate no; \
         mirror --only-newer --parallel=4 $remoteDir $localBackupDir/backup.0;\
         exit"\
     -u $username,$password $host

Extend Pandas DataFrame with custom functions and attributes

At Quantego.com we love working with Pandas Dataframes. We use them to store and analyze results from simulation runs. On top of our data matrix and a multi-level index we also need to accommodate custom plotting functions and attributes from the previous simulation run.

Subclassing pandas.DataFrame for this task was a no-brainer. The new version 0.16.1 (to be released in the next days) includes some fixes to make working with subclasses of complex data-frames (DF) easier. Here an example of what can be done. First define two new classes for pandas.Series (single col DF) and pandas.DataFrame . You can define new functions or attributes, as needed.

class CustomSeries(pandas.Series):
    @property
    def _constructor(self):
        return CustomSeries

    def custom_series_function(self):
        return 'OK'

class CustomDataFrame(pandas.DataFrame):
    "My custom dataframe"
    def __init__(self, *args, **kw):
        super(CustomDataFrame, self).__init__(*args, **kw)

    @property
    def _constructor(self):
        return CustomDataFrame

    _constructor_sliced = CustomSeries

    def custom_frame_function(self):
        return 'OK'

Notice _constructor  and _constructor_sliced . They make sure you get the correct class back, when slicing the DF.

Via self  you have convenient access to all Pandas functions and can even roll your own.

Regex to find phone numbers in every format

I couldn't find a truly universal regular expression (regex) to match phone numbers, no matter from which country and in which format. They all seemed to be limited in some way. Even named entity extraction APIs require you to set a country to find phone numbers.

In the end I rolled my own regex. It simply looks for a certain amount of numbers and characters generally used to make phone numbers human-readable. If you are looking to match longer or shorter numbers, you can just change the quantifiers. Some examples it will match:

540 297 1860
+886-2-8663-8287
0870882993
0090 530 229 12 04
+66 (0) 28340463
058 218 0600
(2014-2015
062-21-8608888
886-5-2781880
011-81-27372-9341
03- 7722 5012
+886-2-8663-8287
+62 – 21 – 5694 2002
+34 918 380 082
+90 532 643 34 34
+7 495 228 3513
+ 7 702 270 38 13 + 7 777

While not matching:

9:00-17:00
2015(15:00

And here the regex:

(?!.*[a-zA-Z\,:])(?=(\D*\d){7,14})([\+\d\(]{1,2}.{6,23}\d)

To use it in Python

import re

rex = '(?!.*[a-zA-Z\,:])(?=(\D*\d){7,14})([\+\d\(]{1,2}.{6,23}\d)'
numbers = re.findall(rex, str_with_phone_numbers)

 

Let me know, if this is useful for you or if you find space for improvement. Currently the biggest issue I see is that the matching ranges between numbers and total chars are unrelated. Due to many filling chars higher values are needed. Those can lead to false negatives. Best test it for yourself.

Python clipboard access

I was using Python and Jinja2 to generate some tables with 100+ rows for WordPress. This package saved me the extra step to open a file and copy+paste it from there.

There should be many other uses to integrate it into semi-automated workflows.

Check it out here.

Scalable Docker Monitoring with Fluentd, Elasticsearch and Kibana 4

Screen Shot 2014-11-20 at 14.38.27

Docker is a great set of technologies. Once you are comfortable with using it, you are presented with a set of challenges, you didn't have before. To name some:

  • log consolidation: How to retrieve log files from dozens of containers?
  • monitoring: How much RAM and CPU is each container using?

There are a few articles on this topic out there. After reading them none of the solutions really hit me, but they all had some nice features which I chose to combine here. Continue reading

Linksnappy Command Line Downloader (Python)

Simple Python script to download files via Linksnappy.

#! /usr/bin/env python

import requests
import json
import sys

USERNAME = 'my username'
PASSWORD = 'my pass'

params = {'link': sys.argv[1],
'type': '',
'username': USERNAME,
'password': PASSWORD}

resp = requests.post('http://gen.linksnappy.com/genAPI.php',
data={'genLinks': json.dumps(params)})

url = json.loads(resp.text)['links'][0]['generated']

local_filename = url.split('/')[-1]

# http://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py
r = requests.get(url, stream=True)
    with open(local_filename, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)
                f.flush()
print local_filename

Access Docker container attributes in Ansible

Ansible is a great automation solution. I mainly use it to provision servers and launch Docker instances on them. Sometimes I need container attributes, like PID or Port to configure Nginx or monitoring tools.

While the Ansible documentation gives you some hints, I didn't find it 100% obvious on how to solve this. Basically all your newly-created containers will end up in a list called docker_containers. It has the same structure as docker inspect.

For the PID:

docker_containers[0]['State']['Pid']

For the host port:

docker_containers[0]['NetworkSettings']['Ports']['8888/tcp'][0]['HostPort']

So you could add a PID-file for a container like this:

 - copy: dest="/var/run/{{ image_name }}.pid"
   content="{{docker_containers[0]['State']['Pid']}}"

Also read the full docs here.

Advanced monit: Keep track of daemons, websites, RAIDs and partitons

Introduction

Are you already hosting your own mail- or webserver and do you enjoy the flexibility, control and freedom self-hosting gives you? Besides the many advantages like better privacy and the power to customize it gives you personally, you can also offer your services to other people. Even tough there are a large number of budget hosting companies, many customers are willing to pay for better support or the comfort to have you around for questions. Continue reading