DadOverflow.com

Musings of a dad with too much time on his hands and not enough to do. Wait. Reverse that.

Month: March 2018

My cup overfloweth

March 26, 2018 /

I feel like I’m always running at 100 miles-per-hour helping fulfill all the various activities of my family. This week in particular was tough as two of my kids were in a play that had them at school until 9pm each night and three showtimes on Thursday, Friday, and Saturday. All these commitments can easily drain my free time of any personal accomplishment. Certainly, I’m spending time with the kids: but more in the capacity of taxi driver and activity observer than anything else. So, I’m always keen on finding ways to squeeze out a few minutes of productivity here-and-there. Here are a few ways that help accomplish that:

1. Bring the kindle

Wherever I go, I always try to bring my kindle with me. I keep it loaded with hundreds of books: from non-fiction books on the ridiculous number of topics I’m interested in to fiction books to just entertain. The kindle is my number-one, go-to item to try to find some sort of accomplishment when I’m out-and-about.

2. Listen to podcasts

I’m frequently criss-crossing town to and from activities, so I try to keep my phone filled with podcasts from which I can learn and try to make the driving somewhat productive.

3. Keep the car stocked with bars and water

Much of the time, I’m too busy to stop at some fast food restaurant for refreshment, but even if time permitted, it’s cheaper and healthier to just keep my car stocked with water and protein bars or other relatively healthy food that won’t spoil or melt.

4. Maintain and manage a to-do list

In the must-see movie Dead Men Don’t Wear Plaid, the main character is a detective trying to solve the kidnapping of noted cheese scientist, Dr. Forrest. Dr. Forrest has left different clues in the form of lists that the detective has to discover and then deduce the meaning. These lists are everywhere: torn off corners of dollar bills in desk drawers, one is inside a woman’s brooch, another is hidden inside the lid of a jar of coffee. Unfortunately, I find that I’m a lot like Dr. Forrest: I write to-do lists on envelopes or post-it notes only to misplace them and begin anew. Lately, I’ve started using Google Keep in hopes I can be less like the good doctor. Regardless, having a to-do list handy to review and work on during your various excursions can be helpful.

5. Take notes on your bright ideas

Occasionally, I’ll have a half-baked idea on a new blog post or entrepreneurial endeavor: making sure I have tools on hand to write down these brilliant ideas is important. To that end, I always keep pens and small notebooks in the car–that comes in very handy during parent/teacher conferences for note taking. Also, I try to take notes electronically as much as possible. I used to use Evernote quite a bit for that effort, but lately I’ve switched to using Google Docs. Both work great on your smart phone.

6. Read blogs

Like my addiction to podcasts, I subscribe to hundreds of blogs covering my wide variety of interests. Back in the day, I used Google Reader to aggregate the blogs I like to read, but when that was discontinued, I switched to Feedly. Feedly works great on your phone, too!

7. Wear clothes with lots of pockets

How do you effectively carry all the material that aid your on-the-go productivity–kindle, phone, pens, paper, etc.–particularly when you have to march deep into a sports venue, school, or Boy Scout camp? Pockets, I tell you! I own several pairs of cargo pants and shorts that help me haul around the items I need. I’m a big fan of ScottEVest, as well, and own a few of their vests, coats, and jackets that each come with dozens of pockets for storing essentials.

8. Keep a full gym bag and towel in the car

Going to the gym is an important release for me and I’m able to do it less and less as the family’s activities increase. Most days I have to plan out a few days in advance when I can hit the gym, but practices are occasionally canceled, so it’s a good idea to always keep my gym clothes in the car in case I can slip in a workout.

9. Keep portable batteries, charging cables, and related items handy

As a few of these tips rely on electronic devices, I find it helpful to keep portable batteries, cables, and other paraphernalia around and available in case one of your devices gets low on juice. Obviously, keep your devices as charged as possible and don’t forget to charge your portable batteries, as well!

10. Think about other “everyday carry” items

“Preppers” will sometimes discuss the topic of “everyday carry” (EDC) items. Items like flashlights, band-aids, tweezers, multi-tools, and the like. Eyeglass screws come loose, splinters happen, and small items fall into dark spaces. Compiling EDC items and keeping larger kits like first aid kits in the car can help fix an unexpected problem that might otherwise throw off your schedule and undermine your on-the-go work.

Parsing PDFs in Python

March 15, 2018 /

(Apologies for the awful alliteration…boo-ya!)

A few weeks ago, my child competed in the 2018 Queen City Classic Chess Tournament.   Close to 700 students–from Kindergarten to High School age–participated. Despite a long day (nearly twelve hours of travel and tournament play), my boy and his team did well and maintained good spirits.

Since the organizers are courteous enough to publish the match results, I thought this might be a good opportunity (read, nerd moment) to download the data and analyze it a little to see what sort of patterns I might find.

Well, this turned out to be easier said than done. The organizers published the results in 28 PDF files. In my experience, PDF files are no friend to the data analyst. So, my first challenge became downloading the PDF files and transforming them into objects useful for data analysis.

In this post, I will highlight a few of the steps I implemented to acquire the data and transform it into something useful. In a future post, I will discuss some of the analysis I performed on the data. The complete data acquisition code I wrote is available as a Jupyter Notebook on my github page.

Step 1: Getting the data locations

All the PDF files are exposed as download links on the results page, so I decided the simplest approach would be to just scrape the page and use BeautifulSoup to parse out the download links I needed:



1
2
3
4
5
6
7
8
9
10
11
12
# get the download page

result = requests.get("https://ccpf.proscan.com/programs/queen-city-classic-chess-tournament/final-results/")

soup = BeautifulSoup(result.content, "lxml")



# build a list of the download links

pdf_list = []

results_section = soup.find("section", class_="entry-content cf")

pdf_links = results_section.find_all("a")

for pdf_link in pdf_links:

    title = pdf_link.text

    link = pdf_link.attrs['href']

    pdf_list.append([title, link])

Step 2: Download the PDFs

With a list of the download links, I then just iterated through the list to download the files:



1
2
3
4
5
for p in pdf_list:

    # save pdf to disk

    r = requests.get(p[1])

    pdf_file = p[1].split('/')[-1]

    open("./data/" + pdf_file, 'wb').write(r.content)

Step 3: Read the PDFs

Here’s where I hit my challenge: my go-to solution for PDFs, PyPDF2, just didn’t work on these files. So, I had to find another option. My searching revealed the suite of utilities Xpdf tools. Even then, it took some playing with these tools to find a path forward. In the end, I was able to use the tool pdftotext to at least extract the results from the PDF files to simple text files that would be tremendously easier to work with.

In Jupyter Notebook, I looped through my PDF list and used the shell command to run pdftotext:



1
2
3
4
for p in (pdf_list + pl_pdf_list):

    pdf_path = './data/' + p[1].split('/')[-1]

    txt_path = pdf_path.replace('.pdf', '.txt')

    ! {pdftotext} -table {pdf_path} {txt_path}

Step 4: Parse the resulting text

With the results safely in easy-to-read text files, the hard part was over, right? Well, unfortunately, no delimiters came with the data extract, so, I had to get creative again. Enter regular expressions (regex).

Even without delimiters, the extracted data seemed to follow some basic patterns, so I devised three different regular expressions–one for each file type as the results in the PDFs followed one of three schemas–to match the data elements. Initially, I tried numpy’s fromregex hoping for a quick, one-liner win. The function worked well for the most part, but it still stumbled on a few lines inexplicably. So, I just resorted to conventional Python regex:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
re_ind_nr = r'\s+([0-9]+)\s+(.+?(?=\s{2}))\s{2,}(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'

re_ind_r = r'\s+([0-9]+)\s+(.+?(?=\s{2}))\s{2,}(.+?(?=\s{2}))\s+([0-9 ]+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'

re_team = r'([0-9]+)\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'

# the regex still let's some garbage rows come through like headers and footers.  use this list to weed the garbage out

elems_of_rows_to_remove = ['Score', 'Rnd1', 'Code', 'TBrk1']

ind_nr_list = []

ind_r_list = []

team_list = []



# iterate through the list of result files I downloaded.  The PDFs fall into one of three categories: team results, 

# ranked player results, or non-ranked player results.  The file names follow a loose convention: if "team" or "tm"

# is in the file name, that file is a list of team results.  If a file name starts with "n", that file represents

# results of non-ranked players.  All the rest are results of ranked players.

for p in pdf_list:

    title = p[0]

    txt_file = './data/{0}'.format(p[1].split('/')[-1].replace('.pdf', '.txt'))

    with open(txt_file, 'r') as f:

        t = f.read()

        if 'team' in title.lower() or 'tm' in title.lower():

            l = re.findall(re_team, t)

            l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]

            [team_list.append(r) for r in l]

        elif title.lower().startswith('n'):

            l = re.findall(re_ind_nr, t)

            l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]

            [ind_nr_list.append(r) for r in l]

        else:

            l = re.findall(re_ind_r, t)

            l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]

            [ind_r_list.append(r) for r in l]

Step 5: Call it a day

Finally, I had the data in three different lists I could work with, but I’ll save that part for another day. Again, my complete code is at my github page. Hopefully soon, I’ll find some extra time to do the analysis I originally intended.

Tips to improving your vocabulary

March 7, 2018 /

My oldest child has engaged in the college quest: meditating on what profession she might want to pursue then reverse-engineering that to an associated major and ideal college to support that vision, visiting schools, and, most importantly, studying for and taking the standardized tests–ACT and SAT.

On more than one occasion, she’s complained about the English and/or writing portions of the tests, bemoaning the fact that these sections make use of advanced vocabulary than she’s unused to. For many years, I’ve tried to press on her the importance of expanding her vocabulary; yet, she continues to ignore my appeals (as seems to be our standard father/daughter dynamic). If she would ever listen to me, here are ten practical tips I would encourage her to employ to increase her command of the English language.

1. Go looking for great words on the Internet

As you’d expect, the Internet is a great resource for improving your vocabulary. There are word-of-the-day sites that you might visit daily for new material, but there are also plenty of “themed” lists to work your way through, as well. Here are a few that I’ve found educational:

2. Install a word-of-the-day application on your phone

Why go to the words when they can come to you? There are a number of free word-of-the-day mobile applications out there. Currently, I’m using Dictionary.com’s app. One nice feature of this app is its notifications: at 8:00am every day, the app sends me a notification with the new word. If I like the new word (or any other word I might look up in the app), I can add it to my “favorites”–so, I always have a list handy of some of my favorite words.

 

3. Get a word-of-the-day calendar

There are a variety of calendar and planner-type products out there aiming to help grow your vocabulary!  If you prefer more of a traditional interface from which to learn, this just might be your ticket.

 

4. Get a dictionary and/or thesaurus

Maybe this is my pre-Internet brain talking, but a dictionary and thesaurus should definitely be part of your library. Probably your kids’, too!

 

5. Read challenging books

Words only work when they’re uttered in proper context–and reveal your ignorance when used otherwise. What better way to learn a new word than through the pen of the professionals? Read the likes of Umberto Eco, Gore Vidal, and David Stockman, among others, to deepen your communication options.

 

6. Listen to challenging podcasts

There are podcasts, like the Grammar Girl podcast, dedicated to improving your communication skills. After that, merely listening to intelligent people discussing challenging topics can be quite beneficial. For example, just the other day, Tom Woods reintroduced me to the wonderful word, “vicissitudes“. Podcasts, then, can be an excellent way to both learn more about a particular topic and extend your vocabulary.

 

7. Listen to word-of-the-day apps on your Amazon Echo or Google Home devices

To my Amazon daily briefing, I’ve included the Peppercorn Media word of the day skill. Every school morning, just before venturing out to the bus stop with one of my children, we listen to the daily briefing and acquire a new word of the day. Thus, we get our word-of-the-day in a quick and entertaining way.

 

8. Watch challenging movies or television

Personally, I find movies and television predominantly a waste of time, but if you must imbibe, try to make it media that positively augments your intellect. I find science fiction and historical works occasionally useful for this purpose.  Star Trek, The Martian, and Amistad are a few creations that seem to work in this regard. However, I did learn the word “flibbertigibbet” from the highly underrated Joe Versus the Volcano.

 

9. Force yourself to use your new words in conversation

Stephen Covey wrote, “to learn and not to do is really not to learn. To know and not to do is really not to know.” It’s not enough to learn new and interesting words, but to actually incorporate them into your regular dialog. Similar to martial arts where you repeat a punch or kick hundreds of times until it becomes part of your muscle memory, you must also invoke your new words multiple times so they become easy go-to options in your conversations and writings.

 

10. Write more, forcing yourself to use your new words

As with enhancing your conversations, littering your writing with your new words can help ingrain those new options into your writing toolbox. Also, look for additional writing opportunities like the school newspaper and yearbook (and even blogging) to help further hone your craft.

© 2024 DadOverflow.com

Theme by Anders Noren — Up ↑