(Apologies for the awful alliteration…boo-ya!)
A few weeks ago, my child competed in the 2018 Queen City Classic Chess Tournament. Close to 700 students–from Kindergarten to High School age–participated. Despite a long day (nearly twelve hours of travel and tournament play), my boy and his team did well and maintained good spirits.
Since the organizers are courteous enough to publish the match results, I thought this might be a good opportunity (read, nerd moment) to download the data and analyze it a little to see what sort of patterns I might find.
Well, this turned out to be easier said than done. The organizers published the results in 28 PDF files. In my experience, PDF files are no friend to the data analyst. So, my first challenge became downloading the PDF files and transforming them into objects useful for data analysis.
In this post, I will highlight a few of the steps I implemented to acquire the data and transform it into something useful. In a future post, I will discuss some of the analysis I performed on the data. The complete data acquisition code I wrote is available as a Jupyter Notebook on my github page.
Step 1: Getting the data locations
All the PDF files are exposed as download links on the results page, so I decided the simplest approach would be to just scrape the page and use BeautifulSoup to parse out the download links I needed:
1
2
3
4
5
6
7
8
9
10
11
12
| # get the download page
result = requests.get("https://ccpf.proscan.com/programs/queen-city-classic-chess-tournament/final-results/")
soup = BeautifulSoup(result.content, "lxml")
# build a list of the download links
pdf_list = []
results_section = soup.find("section", class_="entry-content cf")
pdf_links = results_section.find_all("a")
for pdf_link in pdf_links:
title = pdf_link.text
link = pdf_link.attrs['href']
pdf_list.append([title, link]) |
Step 2: Download the PDFs
With a list of the download links, I then just iterated through the list to download the files:
1
2
3
4
5
| for p in pdf_list:
# save pdf to disk
r = requests.get(p[1])
pdf_file = p[1].split('/')[-1]
open("./data/" + pdf_file, 'wb').write(r.content) |
Step 3: Read the PDFs
Here’s where I hit my challenge: my go-to solution for PDFs, PyPDF2, just didn’t work on these files. So, I had to find another option. My searching revealed the suite of utilities Xpdf tools. Even then, it took some playing with these tools to find a path forward. In the end, I was able to use the tool pdftotext to at least extract the results from the PDF files to simple text files that would be tremendously easier to work with.
In Jupyter Notebook, I looped through my PDF list and used the shell command to run pdftotext:
1
2
3
4
| for p in (pdf_list + pl_pdf_list):
pdf_path = './data/' + p[1].split('/')[-1]
txt_path = pdf_path.replace('.pdf', '.txt')
! {pdftotext} -table {pdf_path} {txt_path} |
Step 4: Parse the resulting text
With the results safely in easy-to-read text files, the hard part was over, right? Well, unfortunately, no delimiters came with the data extract, so, I had to get creative again. Enter regular expressions (regex).
Even without delimiters, the extracted data seemed to follow some basic patterns, so I devised three different regular expressions–one for each file type as the results in the PDFs followed one of three schemas–to match the data elements. Initially, I tried numpy’s fromregex hoping for a quick, one-liner win. The function worked well for the most part, but it still stumbled on a few lines inexplicably. So, I just resorted to conventional Python regex:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
| re_ind_nr = r'\s+([0-9]+)\s+(.+?(?=\s{2}))\s{2,}(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'
re_ind_r = r'\s+([0-9]+)\s+(.+?(?=\s{2}))\s{2,}(.+?(?=\s{2}))\s+([0-9 ]+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'
re_team = r'([0-9]+)\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'
# the regex still let's some garbage rows come through like headers and footers. use this list to weed the garbage out
elems_of_rows_to_remove = ['Score', 'Rnd1', 'Code', 'TBrk1']
ind_nr_list = []
ind_r_list = []
team_list = []
# iterate through the list of result files I downloaded. The PDFs fall into one of three categories: team results,
# ranked player results, or non-ranked player results. The file names follow a loose convention: if "team" or "tm"
# is in the file name, that file is a list of team results. If a file name starts with "n", that file represents
# results of non-ranked players. All the rest are results of ranked players.
for p in pdf_list:
title = p[0]
txt_file = './data/{0}'.format(p[1].split('/')[-1].replace('.pdf', '.txt'))
with open(txt_file, 'r') as f:
t = f.read()
if 'team' in title.lower() or 'tm' in title.lower():
l = re.findall(re_team, t)
l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]
[team_list.append(r) for r in l]
elif title.lower().startswith('n'):
l = re.findall(re_ind_nr, t)
l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]
[ind_nr_list.append(r) for r in l]
else:
l = re.findall(re_ind_r, t)
l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]
[ind_r_list.append(r) for r in l] |
Step 5: Call it a day
Finally, I had the data in three different lists I could work with, but I’ll save that part for another day. Again, my complete code is at my github page. Hopefully soon, I’ll find some extra time to do the analysis I originally intended.
Recent Comments