Creating a black-list of IP's for iptables. (Python)

Hi there.
Let's create a blacklist of IP's for iptables firewall. In Python.
Last week the homework was to create a black-list 'object' of IPSET for iptables firewall.

Really, don't know if someone of my classmates has done this, so i asked my teacher for if I can post it here. Maybe it will help someone to understand how to do that.

What we need to do, is to parse all IP addresses from sources below:

BAN-LISTS:

http://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1
http://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1
http://www.maxmind.com/en/anonymous_proxies
http://danger.rulez.sk/projects/bruteforceblocker/blist.php
http://rules.emergingthreats.net/blockrules/compromised-ips.txt
http://rules.emergingthreats.net/blockrules/rbn-ips.txt
http://www.spamhaus.org/drop/drop.lasso
http://cinsscore.com/list/ci-badguys.txt
http://www.openbl.org/lists/base.txt
http://www.autoshun.org/files/shunlist.csv
http://lists.blocklist.de/lists/all.txt

It's a simple algorithm:
1. Read URL's in loop.
2. Extract needed data
3. Append it to some list
4. Create new SET in IPSET (if needed)
5. Apply all data to IPSET 
 
When you take a look at the sources one-by-one, you can notice, that there are two variations of data to parse:
1. IP Address
2. IP Network

Fine. I think, the easiest way to extract this data is regular expressions (regex).

First what we need to do: list variable with allllll URL's:
Can be done with opening file, or just create a variable:

# Your links with blacklisted IP's
links = '''
http://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1
http://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1
http://www.maxmind.com/en/anonymous_proxies
http://danger.rulez.sk/projects/bruteforceblocker/blist.php
http://rules.emergingthreats.net/blockrules/compromised-ips.txt
#http://rules.emergingthreats.net/blockrules/rbn-ips.txt
http://www.spamhaus.org/drop/drop.lasso
http://cinsscore.com/list/ci-badguys.txt
http://www.openbl.org/lists/base.txt
http://www.autoshun.org/files/shunlist.csv
http://lists.blocklist.de/lists/all.txt
'''

 
Note, that there is one commented URL, this one is not working. You can delete it, or just parse '#' character. (see below)

Next, will load URL, and parse it.
I like Python Requests module, you can use urllib module:

import requests
from netaddr import *

ip_list = []

# Get & Extract IP addresses function
def geturl(i):
   try:
      global ip_list
      print 'Parsing \'%s\'' % i
      r = requests.get(i)
      
      #if status code 200
      if r.ok:
         ip_list += [IPNetwork(i) for i in re.findall(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\/?[0-9]{1,2}?\b',r.content)]
   except:
      pass

Let me explain some BOLD points:

1. First of all, maybe you don't know how to make a for loop in one line: 

For example, next two pieces of code are identical:

tuple_list = [('abc',123),('def',456),('ghi',789)]

new_list = []

for i in tuple_list:
   new_list.append(i[0])

tuple_list = [('abc',123),('def',456),('ghi',789)]

new_list = [i[0] for i in tuples_list]

2. re.findall() method: It will return LIST type. So we can loop in it.
3. IPNetwork() is method from netaddr module, that will: 
    a) check if the IP Address is valid; 
    b) Append to it netmask /32 if single IP.
4. HTTP Status CODE. If you don't know what HTTP Status code is, you have a huge gap in your knowledge. Read here.
5. requests.get() method of requests module, that will load your URL. r.content = raw data from that URL
6. Last one is first one: global ip_list. This mean that in that specific function 'ip_list' variable will be global and not local variable.

Next:

# Get every URL in loop, IGNORE commented lines, extract every IP including subnet mask e.g. '1.1.1.1/32'

for i in links.split():
   if not i[0] == '#':      #ignore commented line
      geturl(i)

Pretty easy, huh?

And the last piece of code:

ip_list = cidr_merge(ip_list)

It is amazing function that will merge IP addresses to subnets!
For example: 1.1.1.0 and 1.1.1.1 will be merged to 1.1.1.0/31

About applying all the data to IPSET, but replace 'hash:ip' with 'hash:net'

For educational purposes I will not post FULL CODE. You can easily make your's from the examples above.

Good Luck. A.K.

P.S. Hope your list[] of knowledge has been .updated :)