Screen Scraping: A Hands-on Introduction

An IDE, or Integrated Development Environment, is used to facilitate programming. A good IDE does things like code highlighting, error checking, one-click-running, and easy integration across multiple files. An example of a crappy IDE is notepad. I like to use emacs. Most people prefer something else.

For this session, I recommend Wing 101. It's a free version of a more fully-featured IDE, but for beginners, it's perfect. If you don't already have an IDE that you're invested in, or you want your intro to python to be as painless as possible, you should install it. It's cross platform.

• Getting Started in Wing
  Once you have Wing installed, you might want to use the tutorial to learn how to navigate around in it.

  

  Opening the tutorial in Wing 101.

• Of course, this graphing calculator can handle text, too!

mystr = "Hello, World!"
print mystr
print len(mystr)

Hello, World!
13

Note to Stata Users:
Assigning a variable is not the same as adding a "column" to your dataset.

Get the first element of a string.

• Note: Python counts from 0. This is a common convention in most languages constructed by computer scientists.

mystr = "Dogs in outer space"
print mystr[0]

D

Get the last element of a string

mystr = "Dogs in outer space"
print mystr[-1]
print mystr[len(mystr)-1]

e
e

mystr = "Dogs in outer space"
print mystr[1:3]
print mystr[3:]
print mystr[:-3]

og
s in outer space
Dogs in outer sp

• By default, python doesn't include every possible "package"
  • This is similar to R, but unlike Matlab
  • Use the include statement to load a library

import math
print math.sin(math.pi)

1.22464679915e-16

After we import from a package, we have to access sub-elements of that package using the . operator. Notice also that while the value 1.22464679915e-16 is very nearly 0, the math module doesn't know that sin(π) = 0. There are smarter modules for doing math in Python, like scipy and numpy. Some people love using Python for Math. I think it makes more sense to use R.

• If you want to import something into your namespace
  • from math import <myfunction> or
  • from math import *

from math import *
print sin(pi)

1.22464679915e-16

Python makes extensive use of objects. An object has

• Methods: functions that work only on that option
• Fields: data that only that type of object has

For example, let's imagine a fruit object. A fruit might have a field called hasPeel, which tells you whether this fruit is peeled. It could also have a method called peel, which alters the state of the fruit.

str = "THE World is A BIG and BEAUTIFUL place.  "
print str.upper()
name = "Alex Storer"
print name.swapcase()

THE WORLD IS A BIG AND BEAUTIFUL PLACE.  
aLEX sTORER

Here we defined two strings, str and name, and used these to invoke string methods which affect the case of the string.

• You can write your own objects and methods
• Objects can be sub-classes of other objects
  • e.g., a psychologist is a type of researcher, who does everything a researcher does but also some other things only a pyschologist does.

You can write your own functions, pieces of code that can be used to take specific inputs and give outputs. You can create a function by using the def command.

def square(x):
    return x*x
print square(9)

81

Pay close attention to the whitespace that is used in Python! Unlike other languages, it is not ignored. Everything with the same indentation is in the same level. Above, the statement return x*x is part of the square function, but the following line is outside of the function definition.

You can think about this logical process as being in pseudocode.

IF do things right
   ---> code well
OTHERWISE
   ---> do things fast

A lot of programming is figuring out how to fit things into this sort of if=/=else structure. Let's look at an example in Python.

• The method find returns the index of the first location of a string match

mystr = "This is one cool looking string!"
if mystr.find("string")>len(mystr)/2:
    print "The word 'string' is in the second half"
else:
    print "The word 'string is not in the second half"

The word 'string' is in the second half

What happens if the word "string" is not there at all?

• The method find returns -1 if the string isn't found

mystr = "I don't know about you, but I only use velcro."
print mystr.find("string")
if mystr.find("string")>len(mystr)/2:
    print "The word 'string' is in the second half"
elif mystr.find("string")>=0:
    print "The word 'string is not in the second half"
else:
    print "The word 'string' isn't there!"

-1
The word 'string' isn't there!

• Important Note: In Python, most everything evaluates to True. Exceptions include 0 and None. This means that you can say things like if (result) where the result may be a computation, a string search, or anything like that. As long as it evaluates to True, it will work!

• if, elif and else can be used to control the flow of a program
• strings are a type of a object, and have a number of methods that come with them, including find, upper and swapcase
  • methods are called using mystring.method()
  • The list of methods for strings can be found in the Python documentation
• def can be used to define a function
  • The return statement determine what the function returns

str = "Daddy ran to help Ann.  Up and down went the seesaw."      
for word in str.split():
    print word

Daddy
ran
to
help
Ann.
Up
and
down
went
the
seesaw.

Notice the use of str.split(): this is an example of calling a method of a string object. It returns a list of words after splitting the string on whitespace.

• Exercise 1:
  Write a function that takes in a string, and outputs the square of its length.

  Notice that a function can call another function that you wrote.

  def square(x):
      return x*x
  
  def sqlen(x):
      return square(len(x))
  
  print sqlen("Feet")
  

  16
  

• Exercise 2
  Write a function that returns the number of capitalized letters in a string.

  def numcaps(x):
      lowerstr = x.lower()
      ncaps = 0
      for i in range(len(x)):
          if lowerstr[i]!=x[i]:
              ncaps += 1
      return ncaps
  
  teststr = "Dogs and Cats are both Animals"
  print teststr, "has", str(numcaps(teststr)), "capital letters"
  

  Dogs and Cats are both Animals has 3 capital letters
  

• Exercise 3
  

  def findDog(x):
      mylist = x.split("dog")
      if len(mylist) < 2:
          return "not found"
      else:
          return mylist[0]    
      return mylist
  print findDog("i have a dog but not a cat")
  print findDog("i have a fish but not a cat")
  print findDog("i have a dog but not a dogwood")
  
  

  i have a 
  not found
  i have a 
  

mydict = dict()
mydict["dogs"] = 14
mydict["fish"] = "slumberland"
mydict["dogs"]+= 3
print mydict

{'fish': 'slumberland', 'dogs': 17}

len(mydict["fish"])

One of the nice things about python is that even when very condensed, it is still readable. People talk about coding in a pythonic way, meaning to write very tight, readable code.

print dict([(x, x**2) for x in (2, 4, 6)]) 

{2: 4, 4: 16, 6: 36}

Let's use a dictionary to store word counts from a sentence.

str = "Up and down went the seesaw. Up it went.  Down it went.  Up, up, up!"
print str
for i in [",",".","!"]:
    str = str.replace(i," ")
print str
str = str.lower()
print str
print set(str.lower().split())

Up and down went the seesaw. Up it went.  Down it went.  Up, up, up!
Up and down went the seesaw  Up it went   Down it went   Up  up  up 
up and down went the seesaw  up it went   down it went   up  up  up 
set(['and', 'up', 'it', 'down', 'seesaw', 'went', 'the'])

We see that a set contains an unordered collection of the elements of the list returned by split(). Let's make a dictionary with keys that are pulled from this set.

str = "Up and down went the seesaw. Up it went.  Down it went.  Up, up, up!"
for i in [",",".","!"]:
    str = str.replace(i," ")
words = str.lower().split()
d = dict.fromkeys(set(words),0)
print d
for w in words:
    d[w]+=1
print d

{'and': 0, 'down': 0, 'seesaw': 0, 'went': 0, 'the': 0, 'up': 0, 'it': 0}
{'and': 1, 'down': 2, 'seesaw': 1, 'went': 3, 'the': 1, 'up': 5, 'it': 2}

A very useful feature of dictionaries is that there is an easy method to write them out to a CSV (comma-separated variable) file.

import csv
f = open('/tmp/blah.csv','w')
nums = [1,2,3]
c = csv.DictWriter(f,nums)
for i in range(0,10):
    c.writerow(dict([(x, x**i) for x in nums]))
f.close()

This writes out the following csv file:

1,1,1
1,2,3
1,4,9
1,8,27
1,16,81
1,32,243
1,64,729
1,128,2187
1,256,6561
1,512,19683    

• Think about file objects like a book
  • If a file is open, you don't want other people to mess with it
  • Files can be opened for reading or writing
  • There are methods to move around an open file
• Close the book when you're done reading it!
• Python documentation on "File I/O" is here

To play with this, download this file somewhere on your hard drive. I'm putting it on my hard drive as /tmp/gaga.txt. On Windows, it may look more like C:\temp\gaga.txt - just make sure you get the path correct when you tell Python where to look!

f = open('/tmp/gaga.txt','r')
print f
str = f.read()
print "str has length: ", len(str)
str2 = f.read()
print "str2 has length: ", len(str2)
f.seek(0)
str3 = f.readline()
print "str3 has length: ", len(str3)
f.close()

<open file '/tmp/gaga.txt', mode 'r' at 0x10045e8a0>
str has length:  1220
str2 has length:  0
str3 has length:  77

You'll use file objects a lot. As we see them, I'll try to point out what's important about them.

• Exercise 1
  Write a function that counts the number of unique letters in a word.

• Exercise 2
  Write a function that takes in a string, and returns a dict that tells you how many words of each number of letters there are.

  "Dogs and cats are all animals"
   dogs and cats are al  animls
   4    3   4    3   2   6
   {2: 1, 3: 2, 4: 2, 6: 1}
  

• Exercise 3
  Loop over a list of strings, and write a csv that contains a column for each number and a row for each string.

   1,2,3,4,5,6,7,8,9,10,11,12,13
   2,3,2,3,4,5,2,3,2,1 , 0, 0, 0
   5,2,1,0,1,2,0,0,0,0 , 0, 0, 0
   etc.
  

• Exercise 1
  Write a function that counts the number of unique letters in a word.

  def uniqueletters(w):
      d = dict()
      for char in w:
          d[char] = 1
      return len(d.keys())
  print uniqueletters("dog")
  print uniqueletters("dogged")
  
  

  3
  4
  

• Exercise 2
  Write a function that takes in a string, and returns a dict that tells you how many words of each number of letters there are.

  "Dogs and cats are all animals"
   dogs and cats are al  animls
   4    3   4    3   2   6
   {2: 1, 3: 2, 4: 2, 6: 1}
  

  def uniqueletters(w):
      d = dict()
      for char in w:
          d[char] = 1
      return len(d.keys())
  
  def wordcounter(str):
      d = dict()
      for w in str.split():
          u = uniqueletters(w)
          if u in d.keys():           
              d[u]+=1
          else:
              d[u] = 1
      return d
  
  print wordcounter("Dogs and cats are all animals")
  
  

  {2: 1, 3: 2, 4: 2, 6: 1}
  

• Exercise 3
  Loop over a list of strings, and write a csv that contains a column for each number and a row for each string.

   1,2,3,4,5,6,7,8,9,10,11,12,13
   2,3,2,3,4,5,2,3,2,1 , 0, 0, 0
   5,2,1,0,1,2,0,0,0,0 , 0, 0, 0
   etc.
  

  import csv
  def uniqueletters(w):
      d = dict()
      for char in w:
          d[char] = 1
      return len(d.keys())
  
  def wordcounter(str):
      d = dict()
      for w in str.split():
          u = uniqueletters(w)
          if u in d.keys():           
              d[u]+=1
          else:
              d[u] = 1
      return d
  
  def listwriter(l):
      emptydict = dict([(x, 0) for x in range(1,26)])
      f = open('/tmp/blah.csv','w')
      c = csv.DictWriter(f,sorted(emptydict.keys())) 
      c.writeheader()    
      for str in l:
          c.writerow(dict(emptydict.items()+wordcounter(str).items()))
      f.close()
  
  listwriter(["Five score years ago, a great American, in whose symbolic shadow we stand today, signed the Emancipation Proclamation.",
              "We observe today not a victory of party, but a celebration of freedom -- symbolizing an end, as well as a beginning -- signifying renewal, as well as change.", 
              "So, first of all, let me assert my firm belief that the only thing we have to fear is fear itself -- nameless, unreasoning, unjustified terror which paralyzes needed efforts to convert retreat into advance."])
  
  

  Here is the resulting CSV file:

  1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
  1,2,1,2,5,3,0,1,2,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
  5,8,4,1,2,4,3,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
  1,7,6,9,4,2,3,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
  

• import re
  • tells python to use the regular expression library. (Like library(zelig))
• str = ...
  • defines a string
  • Python will figure out that the type is a string based on the fact that it's in quotes
  • There is a difference between

    foo = '333'
    

    and

    foo = 333     
    

• re.findall("\d\d\d",str)
  • From the re library, call the findall function
    • When in doubt, Google it.
      • By the way, googling things effectively is the most important modern research skill there is.
  • Finds all of the matched of the regular expression \d\d\d in str
    • Returns it as a list

import re
str = "Joseph Schmoe (15), Phone:(934) 292-2390, SSN:295-48-2311"  
print re.findall("\((\d\d\d)\)",str)

['934']

"Joseph Schmoe (15), Phone:(934) 292-2390, SSN:295-48-2311"  

• A brittle system is one that is not resistant to change
• For example, between early April and late April of 2012, Egypt Independent transitioned from

  http://www.egyptindependent.com/node/725861
  

  to a new URL naming scheme that involves the title:

  http://www.egyptindependent.com/news/european-union-will-keep-mubarak-assets-ice-illicit-gains-authority-head-says
  

  All scrapers are brittle.

  • The assumptions you're forced to make about how information is organized on a given website will not hold forever.
  • In fact, the legality of scraping is not entirely clear, and some sites may not be interested in you hammering their servers!

Sometimes, metadata is included which tells us important things about our article

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="msvalidate.01" content="F1F61CF0E5EC4EC2940FCA062AB13A53" />
<meta name="google-site-verification" content="Q8FKHdNoQ2EH7SH1MzwH_JNcgVgMYeCgFnzNlXlR4N0" />
<title>European Union will keep Mubarak assets on ice, Illicit Gains Authority head says | Egypt Independent</title>
<!-- tC490Uh18j-7O_rp7nG0_e6U9QY -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<link rel="canonical" href="http://www.egyptindependent.com/node/725861" />
<meta name="keywords" content="Assem al-Gohary, corruption, EU, freezing  Mubarak’s assets, Hosni Mubarak, Illicit Gains Authority (IGA), News, Top stories" />
<meta name="description" content="The European Union will continue to freeze the assets of former President Hosni Mubarak, his family and other former officials although Egypt has thus far been unsuccessful in recovering funds siphoned abroad by the regime." />
<meta name="abstract" content="Al-Masry Al-Youm - Egypt&#039;s leading independent media group المصرى اليوم للصحافة والنشر هى مؤسسة إعلامية مصرية مستقلة تأسست عام  ,2003." />

• Keywords, abstract, description and title are all clear
• Lots of other gunk that isn't relevant to us!
• Pulling information out of this document requires that we know how they organize title their metadata!
  • What if keywords were called terms?

The actual body of the article can be found by right-clicking on the text we're interested in from Chrome or Firefox and selecting "Insepct Element"

<div class="panel-region-separator"></div><div class="panel-pane pane-node-body" >    
  <div class="pane-content">
    <p>The European Union will continue to freeze the assets of former President Hosni Mubarak, his family and other former officials although Egypt has thus far been unsuccessful in recovering funds siphoned abroad by the regime.</p>
<p>The Illicit Gains Authority (IGA), the judicial committee responsible for recovering the money, on Wednesday received an official notification from the European Union, confirming its freeze on the assets would be renewed another year as of 19 March, state-run MENA news service reported on Wednesday.&nbsp;</p>
<p>&ldquo;This was in response to a request by Egypt,&rdquo; the state news agency quoted IGA head Assem al-Gohary as saying.&nbsp;</p>
<p>Egypt formally asked European Union countries earlier this month to continue freezing funds belonging to Mubarak, his two sons and other members of his administration.</p>
<p>Shortly after Mubarak was forced to step down in February 2011, the public prosecutor ordered that the foreign assets of the deposed president and his family be frozen.</p>
<p>Mubarak&#39;s actual worth is still unknown after more than a year of investigations into his foreign and domestic assets. Last year claims that Mubarak, in his nearly 30-year reign as head of state, may have amassed a fortune of up to US$70 billion &mdash; greater than that of Microsoft&#39;s Bill Gates &mdash; helped drive the protests that eventually brought him down.</p>
<p>Last year Swiss authorities also froze Mubarak&rsquo;s assets, acting more speedily than when the EU froze the assets of another deposed North African ruler, former Tunisian President Zine al-Abidine Ben Ali.</p>
<p>On Wednesday, the IGA met with the Swiss ambassador in Cairo to discuss the difficulties it faces in recovering those funds, in light of the obligations of the United Nations Convention Against Corruption on the member states, reported MENA.</p>
<p>Gohary once estimated the frozen assets at 410 million Swiss francs (LE2.7 billion), which Egypt is trying to repatriate in cooperation with the Foreign Ministry.</p>
  </div>

All of the body is included in the panel-pane pane-node-body section of this site, within the sub-section pane-content. Our "algorithm" for getting this information out will require finding the exact section of the site that we require pulling this data out from. If you don't do this, any terms that are on the sidebar will end up being in your analysis!

Every News Feature is on a page in the following scheme:

http://www.egyptindependent.com/subchannel/News%20features?page=5

And this paper goes back 77 pages, to April, 2009.

Investigating the source for a single search page can tell us what we have to do to get at the relevant information:

<div class="views-row views-row-4 views-row-even">
  <div class="views-field-field-published-date-value">
    <span class="field-content"><span class="date-display-single">09 Feb 2012</span></span>
  </div>
  <div class="views-field-title">
    <span class="field-content"><a href="http://www.egyptindependent.com/node/647936">Parliament Review: A week of comedy and disappointment</a></span>
  </div> 
  <div class="views-field-body">
    <span class="field-content">This week&rsquo;s parliamentary sessions had the public joking about airing future sessions on comedy channels instead of news, and those who abstained from the polls telling those who participated, in hope of having a legitimate authority...</span>
  </div>  

Our algorithm to scrape articles from this page will be as follows:

1. Initialize FOO=1
2. Go to http://www.egyptindependent.com/subchannel/News%20features?page=FOO
3. Repeat until complete:
   1. Find the next occurence of views-row...
   2. Find the sub-field called views-field-field-published-date-value and retrieve its value (the date)
   3. Find the sub-field called views-field-title and retrieve its value (the title)
   4. Follow the link from above
   5. Within the link, find the meta-data keywords and retrieve their values (the keywords)
   6. Within the link, find the panel-pane pane-node-body section, and retrieve the test (the article itself)

Not all web sites are designed in the same way. Go to the site of your choice, and figure out how to get the articles you're interested in. Write out pseudocode that will tell you:

1. How to download individual articles
2. How to get the Author of an article
3. How to get the Title of an article
4. How to get the Date of an article
5. How to get the text of the article

If you need a site to practice on that isn't too challenging, check out Robert Ebert's Blog.

External packages can be easily installed in python using the easy_install command. The only challenge is in making sure that if you have multiple version of python installed, you are installing the libraries to the correct location. I'm on a mac, but the Python version on a mac is 2.6, and I prefer using 2.7. Make sure you install the setuptools for 2.7 following these instructions. Then, run

sudo easy_install-2.7 lxml

If you have no idea what I'm talking about, it will probably be fine if you simply use the following:

sudo easy_install lxml

On windows, try doing

easy_install lxml

To verify that this installed for you, open up python, and type

import lxml

If you get an error, sheck your setup and try reinstalling.

lxml will generate an ElementTree for us after parsing the xml. Let's review some of the functions that will be useful for us in this example.

The xpath syntax is described in more detail here. Briefly, we are finding every occurence of spans with the class date-display-single, no matter where they live in the tree. Then we can iterate over them to get the actual dates. Similarly, we can iterate over all links that are within the <span class="field content"> that are within the <div class="views-field-title"> and zip it with the dates to iterate over both simultaneously. Notice that whenever foreign characters are used, Python will be unable to display them unless we encode the string first as unicode. The following code makes this explicit.

==

from lxml import etree
fname = '/tmp/page1.html'
fp = open(fname, 'rb')
parser = etree.HTMLParser()
tree   = etree.parse(fp, parser)
dateelems = tree.xpath('.//span[@class="date-display-single"]')
linkelems = tree.xpath('.//div[@class="views-field-title"]/span[@class="field-content"]/a')     
for (d,l) in zip(dateelems,linkelems):
    print d.text
    print l.get('href')         
    print l.text.encode("utf-8")

13 Apr 2012
http://www.egyptindependent.com/news/muslim-brotherhood-returns-street-politics-fills-square
Muslim Brotherhood returns to street politics, fills square
12 Apr 2012
http://www.egyptindependent.com/news/meet-your-presidential-candidate-omar-suleiman-phantom
Meet your presidential candidate: Omar Suleiman, the phantom
11 Apr 2012
http://www.egyptindependent.com/news/administrative-court-ruling-leaves-transition-timetable-disarray
Administrative court ruling leaves transition timetable in disarray
11 Apr 2012
http://www.egyptindependent.com/news/shater-faces-early-hiccups-campaign-trail
Shater faces early hiccups on campaign trail
11 Apr 2012
http://www.egyptindependent.com/news/new-alternatives-may-bolster-moussa%E2%80%99s-chances
New alternatives may bolster Moussa’s chances
09 Apr 2012
http://www.egyptindependent.com/news/profile-kamal-al-helbawy-defector-conscience
Profile: Kamal al-Helbawy, a defector of conscience
09 Apr 2012
http://www.egyptindependent.com/news/suleiman-president-game-changer-or-set-plan
Suleiman for president: Game changer or set plan?
09 Apr 2012
http://www.egyptindependent.com/news/despite-prison-time-revolutionaries-uniform-continue-struggle
Despite prison time, revolutionaries in uniform continue struggle
06 Apr 2012
http://www.egyptindependent.com/news/parliament-review-constitution-crisis-continues-brothers-spread-their-wings
Parliament Review: Constitution crisis continues as Brothers spread their wings
06 Apr 2012
http://www.egyptindependent.com/news/profile-april-6-genealogy-youth-movement
Profile: April 6, genealogy of a youth movement

• Get all links under <div class="views-field-title">
  

  from lxml import etree
  fname = '/tmp/page1.html'
  fp = open(fname, 'rb')
  parser = etree.HTMLParser()
  tree   = etree.parse(fp, parser)
  elems = tree.xpath('.//div[@class="views-field-title"]//a')
  for e in elems:
      print e.text.encode('utf-8')
  

  Muslim Brotherhood returns to street politics, fills square
  Meet your presidential candidate: Omar Suleiman, the phantom
  Administrative court ruling leaves transition timetable in disarray
  Shater faces early hiccups on campaign trail
  New alternatives may bolster Moussa’s chances
  Profile: Kamal al-Helbawy, a defector of conscience
  Suleiman for president: Game changer or set plan?
  Despite prison time, revolutionaries in uniform continue struggle
  Parliament Review: Constitution crisis continues as Brothers spread their wings
  Profile: April 6, genealogy of a youth movement
  Parliament Review: A week of political exclusion and inclusion
  Friday's protest to unite, or further divide
  On campaign trail, Moussa speaks villagers' language
  Elections commission's disqualifications dubious, say experts
  Lawyers blame flawed justice system for acquittal of protesters’ accused killers
  Halloween!
  An Ode to Love
  Letters to Treze
  علشان مننشاش المغربي
  Wedding dance of "Beja" tribe
  Egyptian protester passes out after harassment
  "Pearly Pink Flower"
  I Cry!
  "The Amazing Bibliotheca Alexandrina!"
  am agree  title
  am agree  title
  Divorce between  Margaret Scobey and Mr. Baradei
  

• Get all clickable images
  

  These will look like:

  <a href="www.webpage.com"><img src="laksjdasldkj.jpg"></a>
  

  from lxml import etree
  fname = '/tmp/page1.html'
  fp = open(fname, 'rb')
  parser = etree.HTMLParser()
  tree   = etree.parse(fp, parser)
  elems = tree.xpath('.//a/img')
  for e in elems:
      print e.get('src')
  

  /sites/default/files/img/english_logo.png
  http://www.egyptindependent.com//sites/default/files/imagecache/video_thumbnail/video/2011/03/28/22597/lshn_mnnshsh_lmgrby_87049_1558797204.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/video_thumbnail/video/2010/05/09/3685/wedding_dance_17135_1577232530.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/video_thumbnail/video/2010/04/14/2252/rabw_14731_451336279.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/photo/2011/11/13/27866/pink_flower.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/photo/2011/11/09/27866/ips.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/photo/2011/10/29/27866/amazing.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/caricature/2010/04/14/120/piioioioioi.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/caricature/2010/04/13/120/wewweweww.jpg
  http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/caricature/2010/04/05/489/separation.jpg
  /sites/default/files/W300.jpg
  

Write a csv file that contains every image along with the location that it links to. If the webpage has:

<a href="www.webpage.com"><img src="laksjdasldkj.jpg"></a>

Your entry in the csv file would look like:

www.webpage.com, laksjdasldkj.jpg

Hint: use the elt.getparent() method to query elements 'above' a given element elt.

import csv
from lxml import etree

fname = '/tmp/page1.html'
fp = open(fname, 'rb')
f = open('/tmp/links.csv','w')
entries = ["Image","Link"]
c = csv.DictWriter(f,entries)

parser = etree.HTMLParser()
tree   = etree.parse(fp, parser)
lnkelems = tree.xpath('.//a/img')
for lnk in lnkelems:
    d = dict()
    d["Image"] = lnk.get('src')
    d["Link"] = lnk.getparent().get('href')
    c.writerow(d)

fp.close()
f.close()

The resulting file is a CSV file.:

/sites/default/files/img/english_logo.png,/
http://www.egyptindependent.com//sites/default/files/imagecache/video_thumbnail/video/2011/03/28/22597/lshn_mnnshsh_lmgrby_87049_1558797204.jpg,http://www.egyptindependent.com/node/377786
http://www.egyptindependent.com//sites/default/files/imagecache/video_thumbnail/video/2010/05/09/3685/wedding_dance_17135_1577232530.jpg,http://www.egyptindependent.com/node/40195
http://www.egyptindependent.com//sites/default/files/imagecache/video_thumbnail/video/2010/04/14/2252/rabw_14731_451336279.jpg,http://www.egyptindependent.com/node/26345
http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/photo/2011/11/13/27866/pink_flower.jpg,http://www.egyptindependent.com/node/514135
http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/photo/2011/11/09/27866/ips.jpg,http://www.egyptindependent.com/node/513040
http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/photo/2011/10/29/27866/amazing.jpg,http://www.egyptindependent.com/node/509791
http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/caricature/2010/04/14/120/piioioioioi.jpg,http://www.egyptindependent.com/node/26275
http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/caricature/2010/04/13/120/wewweweww.jpg,http://www.egyptindependent.com/node/26269
http://www.egyptindependent.com//sites/default/files/imagecache/news-featured/caricature/2010/04/05/489/separation.jpg,http://www.egyptindependent.com/node/24767
/sites/default/files/W300.jpg,http://www.almasryalyoum.com/en/your-guide

Goal: A file with the dates, titles and location of each article. Save each article in html form to the hard drive.

from lxml import etree
import csv     
import urllib
import re

f = open('/tmp/files.csv','w')
entries = ["Day","Month","Year","Title","Remote","Local"]
c = csv.DictWriter(f,entries)


destpath = '/tmp/'
fname = '/tmp/page1.html'
fp = open(fname, 'rb')
parser = etree.HTMLParser()
tree   = etree.parse(fp, parser)
dateelems = tree.xpath('.//div[@class="views-field-field-published-date-value"]/span[@class="field-content"]/span[@class="date-display-single"]')
linkelems = tree.xpath('.//div[@class="panel-pane pane-views pane-subchannel-news subchannel-pane"]//div[@class="views-field-title"]/span[@class="field-content"]/a')

for (d,l) in zip(dateelems,linkelems):
    entry = dict()
    myDate = d.text.split()
    urlname = l.get('href')
    print urlname
    entry["Day"] = myDate[0]
    entry["Month"] = myDate[1]
    entry["Year"] = myDate[2]
    remotename = re.match('.*/(.*)',urlname)
    dest = destpath+remotename.group(1)+".html"
    urllib.urlretrieve (urlname,dest)
    entry["Local"] = dest
    entry["Remote"] = urlname
    entry["Title"] = l.text.encode("utf-8")
    c.writerow(entry)
    print entry

f.close()
fp.close()

http://www.egyptindependent.com/news/muslim-brotherhood-returns-street-politics-fills-square
{'Remote': 'http://www.egyptindependent.com/news/muslim-brotherhood-returns-street-politics-fills-square', 'Title': 'Muslim Brotherhood returns to street politics, fills square', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/muslim-brotherhood-returns-street-politics-fills-square.html', 'Day': '13'}
http://www.egyptindependent.com/news/meet-your-presidential-candidate-omar-suleiman-phantom
{'Remote': 'http://www.egyptindependent.com/news/meet-your-presidential-candidate-omar-suleiman-phantom', 'Title': 'Meet your presidential candidate: Omar Suleiman, the phantom', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/meet-your-presidential-candidate-omar-suleiman-phantom.html', 'Day': '12'}
http://www.egyptindependent.com/news/administrative-court-ruling-leaves-transition-timetable-disarray
{'Remote': 'http://www.egyptindependent.com/news/administrative-court-ruling-leaves-transition-timetable-disarray', 'Title': 'Administrative court ruling leaves transition timetable in disarray', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/administrative-court-ruling-leaves-transition-timetable-disarray.html', 'Day': '11'}
http://www.egyptindependent.com/news/shater-faces-early-hiccups-campaign-trail
{'Remote': 'http://www.egyptindependent.com/news/shater-faces-early-hiccups-campaign-trail', 'Title': 'Shater faces early hiccups on campaign trail', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/shater-faces-early-hiccups-campaign-trail.html', 'Day': '11'}
http://www.egyptindependent.com/news/new-alternatives-may-bolster-moussa%E2%80%99s-chances
{'Remote': 'http://www.egyptindependent.com/news/new-alternatives-may-bolster-moussa%E2%80%99s-chances', 'Title': 'New alternatives may bolster Moussa\xe2\x80\x99s chances', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/new-alternatives-may-bolster-moussa%E2%80%99s-chances.html', 'Day': '11'}
http://www.egyptindependent.com/news/profile-kamal-al-helbawy-defector-conscience
{'Remote': 'http://www.egyptindependent.com/news/profile-kamal-al-helbawy-defector-conscience', 'Title': 'Profile: Kamal al-Helbawy, a defector of conscience', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/profile-kamal-al-helbawy-defector-conscience.html', 'Day': '09'}
http://www.egyptindependent.com/news/suleiman-president-game-changer-or-set-plan
{'Remote': 'http://www.egyptindependent.com/news/suleiman-president-game-changer-or-set-plan', 'Title': 'Suleiman for president: Game changer or set plan?', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/suleiman-president-game-changer-or-set-plan.html', 'Day': '09'}
http://www.egyptindependent.com/news/despite-prison-time-revolutionaries-uniform-continue-struggle
{'Remote': 'http://www.egyptindependent.com/news/despite-prison-time-revolutionaries-uniform-continue-struggle', 'Title': 'Despite prison time, revolutionaries in uniform continue struggle', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/despite-prison-time-revolutionaries-uniform-continue-struggle.html', 'Day': '09'}
http://www.egyptindependent.com/news/parliament-review-constitution-crisis-continues-brothers-spread-their-wings
{'Remote': 'http://www.egyptindependent.com/news/parliament-review-constitution-crisis-continues-brothers-spread-their-wings', 'Title': 'Parliament Review: Constitution crisis continues as Brothers spread their wings', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/parliament-review-constitution-crisis-continues-brothers-spread-their-wings.html', 'Day': '06'}
http://www.egyptindependent.com/news/profile-april-6-genealogy-youth-movement
{'Remote': 'http://www.egyptindependent.com/news/profile-april-6-genealogy-youth-movement', 'Title': 'Profile: April 6, genealogy of a youth movement', 'Month': 'Apr', 'Year': '2012', 'Local': '/tmp/profile-april-6-genealogy-youth-movement.html', 'Day': '06'}

The resulting file is a CSV file.:

13,Apr,2012,"Muslim Brotherhood returns to street politics, fills square",http://www.egyptindependent.com/news/muslim-brotherhood-returns-street-politics-fills-square,/tmp/muslim-brotherhood-returns-street-politics-fills-square.html
12,Apr,2012,"Meet your presidential candidate: Omar Suleiman, the phantom",http://www.egyptindependent.com/news/meet-your-presidential-candidate-omar-suleiman-phantom,/tmp/meet-your-presidential-candidate-omar-suleiman-phantom.html
11,Apr,2012,Administrative court ruling leaves transition timetable in disarray,http://www.egyptindependent.com/news/administrative-court-ruling-leaves-transition-timetable-disarray,/tmp/administrative-court-ruling-leaves-transition-timetable-disarray.html
11,Apr,2012,Shater faces early hiccups on campaign trail,http://www.egyptindependent.com/news/shater-faces-early-hiccups-campaign-trail,/tmp/shater-faces-early-hiccups-campaign-trail.html
11,Apr,2012,New alternatives may bolster Moussa’s chances,http://www.egyptindependent.com/news/new-alternatives-may-bolster-moussa%E2%80%99s-chances,/tmp/new-alternatives-may-bolster-moussa%E2%80%99s-chances.html
09,Apr,2012,"Profile: Kamal al-Helbawy, a defector of conscience",http://www.egyptindependent.com/news/profile-kamal-al-helbawy-defector-conscience,/tmp/profile-kamal-al-helbawy-defector-conscience.html
09,Apr,2012,Suleiman for president: Game changer or set plan?,http://www.egyptindependent.com/news/suleiman-president-game-changer-or-set-plan,/tmp/suleiman-president-game-changer-or-set-plan.html
09,Apr,2012,"Despite prison time, revolutionaries in uniform continue struggle",http://www.egyptindependent.com/news/despite-prison-time-revolutionaries-uniform-continue-struggle,/tmp/despite-prison-time-revolutionaries-uniform-continue-struggle.html
06,Apr,2012,Parliament Review: Constitution crisis continues as Brothers spread their wings,http://www.egyptindependent.com/news/parliament-review-constitution-crisis-continues-brothers-spread-their-wings,/tmp/parliament-review-constitution-crisis-continues-brothers-spread-their-wings.html
06,Apr,2012,"Profile: April 6, genealogy of a youth movement",http://www.egyptindependent.com/news/profile-april-6-genealogy-youth-movement,/tmp/profile-april-6-genealogy-youth-movement.html

Modify the above code so that instead of iterating over only the first page, it iterates over all pages.

• Consider using the glob library to look for all of the html files in a directory.
• Can you do this so you don't save the pages, but parse them directly?
  • Use google and the python documentation to help figure it out!

  Now that we've seen lxml in action, let's figure out how to use it to pull out just the text of the article. Recall that all of the original text is in the following tags:

  <div class="panel-pane pane-node-body" >    
  <div class="pane-content">