In this post, I would share how this extraction was done and how similar extraction scripts or scrapers could be written.

The first step in every scraping project I begin is to understand the HTTP dialog for the website I want to scrape. So I attempt to answer questions like these:

1. Does the application need me to login?
2. Is it sensitive to certain HTTP features like cookies or referrers?
3. What urls do I access to view the content I want to extract?
4. What variables can I set to change the view and specify what I want?

Determining the answers to these questions can be obtained by using tools that enable you view this dialog. I personally like to use Firebug for this task.

Click for a larger version

After you’ve determined the HTTP dialog, you can then write your script to do the extraction. You can write scrapers in any language provided it has support to retrieve HTTP resources and parse HTML. The parsing aspect of a scraper is usually the most interesting part because a lot of parsing libraries choke when they encounter badly formed HTML.

In the code snippet below, I used BeautifulSoup for parsing the HTML and python’s urllib2 for the HTTP communication.

The code is available on Github and although it changes as more functionality is added, you can view the revision log of the gist to see the history of changes.

<?php $unique_identifier = md5(rand(100000, 999999)); ?>

The problem with this is that I have given an allowance for only 899,999 possible values. I didn’t realize my error until I started getting mysql integrity check errors for a unique column that stored that value.

I reverted to using a more elegant solution:

<?php $unique_identifier = md5(uniqid(rand(), true)); ?>

The uniqid statement generates a globally unique identifier with a rand() prefix and using much more entropy (true).

Essentially, hooks allow you to execute custom scripts when you perform certain actions on your repository like committing files, pulling updates and so on. This is a very useful as you can write hook scripts to (say for example) automatically ftp a file to your web server when a change has been made.

A whole lot of really cool hook scripts have been written and if you use any code versioning tools, you should check out the ones that have been written for the tool you use.

In particular, I find that sometimes developers could check in code that has syntactic bugs. This happens in environments where there are no strict code testing rules. It can be really annoying when you or someone else does this and you have to fix that and then commit again… not professional at all. So I came across this post by Travis Swicegood that lists code that does a php lint on your PHP files before committing them to the repository. PHP lint (php -l) basically checks the syntax of your code and either gives an “ok” or prints the offending line.

For one of the projects I’m working on, I had to change line 11 of Travis’ code to read:

$filename_pattern = '/\.(php|engine|theme|install|inc|module|test)$/';

instead of

$filename_pattern = '/\.php$/';

If you’ve done Drupal coding, you’ll quickly recognize that

I’d been having a particular issue with an API I built for a web application I’m managing and I just couldn’t figure out what was wrong until I got to using the API in an application that had HEX output.

So I put the application in debug mode and watched the communication stream. The API was supposed to return an integer value but instead, I noticed a EF BB BF hex sequence being prepended in the output. All attempts to remove this were futile.

Since I used the Symfony framework for the application, I suspected that it might be a bug and googled for a solution. The best answer I could get was to upgrade the framework, which I did but the problem persisted.

My big break came when I attempted to make another API call to a separate method that resulted in the correct output. While tracing the source of the problem, I traced it back to a php file that was incorrectly saved in the UTF-8 character encoding format; which was included when making the buggy API call. Opening and saving this file in ANSI format solved the problem.

So next time you run into a problem like this, make sure to check the encoding of your source files. Better still, you can use a tool like iconv to convert your files to the appropriate character encoding format.

There was really no point in scratching my head and trying to pull my hair out in order to fix this. So I googled a solution. This post gave me a hint but wasn’t helpful enough so I decided to write a blog post on how to go about solving this.

You’ll have to configure your repository in your .ssh/config file. Here’s an example. Simply substitute the #*# placeholders for the actual values

Host #hostname#
  User #username#
  Hostname #hostname#
  Port #non-standard port#

And if you prefer to use private keys to login:

Host #hostname#
  User #username#
  Hostname #hostname#
  Port #non-standard port#
  PreferredAuthentications publickey
  IdentityFile "#path_to_private_key#"

1. Who made what change
2. What change was made
3. When was the change made

If you happen to have a number of developers and you’re working on the same project (even if you’re the only developer, this helps) you like to know when a change to the source code breaks the application (this happens alot). Not only that, having an SCM allows you to easily revert that change. This post is not about SCMs but I thought it necessary to provide a little background information.

I recently started using git and have really loved it (although it’s not perfect, it does the job well). I prefer git over subversion (svn) for a major reason that it is distributed: does not require a connection to a server in order to commit to the repository.

Recently, I thought about migrating my svn repositories to git and I found a wonderful resource for that here. I’m not going to attempt to republish what has been said there here. Instead just point out a couple of things I had to do differently.

First of all, depending on the way your git is installed, these are basically the commands to run:

git svn init svn://server/repo/trunk/ --no-metadata
git config svn.authorsfile /path/to/svn-authors
git svn fetch

The difference here is to use git svn instead of git-svn. Also, if you happen to have blank authors (as I had in mine), then your svn-authors file should contain an entry similar to this:

(no author) = Firstname Lastname <emailaddress>