DadOverflow.com

Musings of a dad with too much time on his hands and not enough to do. Wait. Reverse that.

Tag: tools (Page 34 of 35)

Are you recording your family events?

April 6, 2018 /

This video gets me.  There I am, trying to record every little scene I can in my kids’ lives.  I hope you’re doing the same.  As my parents constantly remind me: they’ll be gone before you know it!  To that end, here are several ideas I try to keep in mind when recording family events:

1. Get a good camera

I am cheap frugal, so I am loath to spend lots of money on nice video cameras; however, you do want to produce quality video, so you should be prepared to drop a few more smackers for it.  I definitely recommend a camera capable of recording video in high definition (HD).  Also, think about the media on which cameras record: I find ones that record on SD cards to be very convenient.  Recording on your smart phone is fine, too, and I often mix video recorded on my smart phone with video I’ve recorded with my regular camera; nevertheless, video cameras tend to offer many more useful features above what you get with your smart phone.

2. Film everything

It’s a given to film birthday parties, graduations, and plays, but you never know when some seemingly insignificant event might become a long-lasting memory.  Think about some of those smaller events: carving pumpkins, a quiet family dinner, your kids playing a game and laughing together, walking the property of a family farm, etc.  These moments in time may never repeat themselves, so think about recording them!

3. Think about your film technique

I did some camera work in college, but for the most part I’m a complete amateur.  [Side note: I should probably take a class or two to improve my technique.]  For my family videos, though, one technique I use quite often is to keep my camera in tight on my subject for the majority of my shots.  For example, when my child is playing basketball or soccer, I’ll keep my camera trained tight on him instead of holding a wide shot of the field or following the ball from player to player.  With wide shots, your viewers–grandparents and other friends and family–may not even be able to pick out your kids from the 10-20 others on the field, let alone see the skills your child exhibits.  When you follow the ball around, you’re usually filming other children the majority of the time: footage you’ll likely discard, anyway.  So, I generally keep my camera trained in tight on my children.  When they make a great play, I’m already capturing the full lifespan of the play and can easily edit that into my final product.  The one drawback of this technique is that I usually miss great plays by my kids’ teammates.  In general, that doesn’t worry me, but every-so-often a parent will ask me if I captured their child’s buzzer-beater and I reluctantly reply in the negative.

4. Take notes

If you’ve captured an important moment–say, your child’s first goal or some great award or honor–make a note of it somewhere.  If possible, record not only the date and time, but even the name of the video file and timestamp within the file.  That way, when you set down to create a final product (see #10 below), it will be easy for you to find the event and edit it into your product.

5. Keep a steady hand

Few things are worse than watching a home video that gives you motion sickness.  On more than one occasion, my wife and children have filmed events with their phones that I wanted to include in one of our family movies but refrained since the shots were so shaky–which is another reason to not rely solely on your smart phone for your video recorder.  A lot of the nicer video cameras include stabilization controls, which can be helpful, especially with the tight shots.  In addition, I recommend bringing along a decent tripod and/or monopod.

6. Backup, backup, backup

Don’t let all your hard work go to waste: backup early, backup often.  I try never to go more than a few weeks without downloading all the video I’ve taken and back it up to my different storage devices.  This goes for the pictures I take, as well.  High definition files take up a lot of space, so be prepared to have ample storage available.

7. Keep spares handy

I bought a second battery for my video camera that I keep charged and nearby whenever my primary battery gets too low.  I also keep spare, formatted SD cards handy in case I fill up my current card.

8. Go-Pro?  Drone?

I own neither a go-pro nor a drone, but I think either device would definitely introduce a unique flavor to your home videos.  I could certainly see recording a fun bike stroll, roller coaster ride, or rafting adventure with a go-pro.  We have two farms in the family that would be fun to document with a drone and even just recording a few flyovers of my suburban house might be fun video to have years from now.

9. Don’t always be the man behind the camera

Even if no one else is lining up to take over as cameraman, try to get in front of the camera every once-in-a-while.

10. Make a polished product

Having raw footage of your various family activities is great, but why not stretch your creative muscles and make a real video out of it?  For more than 10 years, every Christmas, I meticulously parse through a year’s worth of video footage and create a 60+ minute video recapping all our various activities.  In the early years, I used some of the free tools from Microsoft to create my videos and eventually burn them to DVD (complete with disk art drawn by the kids) that I’d hand out to the extended family.  As Microsoft has deprecated those tools, I moved to the CyberLink family of products.

Like my Christmas Letter, I always press multiple copies of my family movies so that I can eventually gift them to the kids; however, I recently decided to stop burning DVDs altogether, instead, simply creating an MP4 of the finished product that I can upload to my media server.  Eventually, whenever I gift our old Christmas Letters to my kids, I’ll also gift a thumb drive of two decades of family movies.  How cool will that be?

My cup overfloweth

March 26, 2018 /

I feel like I’m always running at 100 miles-per-hour helping fulfill all the various activities of my family. This week in particular was tough as two of my kids were in a play that had them at school until 9pm each night and three showtimes on Thursday, Friday, and Saturday. All these commitments can easily drain my free time of any personal accomplishment. Certainly, I’m spending time with the kids: but more in the capacity of taxi driver and activity observer than anything else. So, I’m always keen on finding ways to squeeze out a few minutes of productivity here-and-there. Here are a few ways that help accomplish that:

1. Bring the kindle

Wherever I go, I always try to bring my kindle with me. I keep it loaded with hundreds of books: from non-fiction books on the ridiculous number of topics I’m interested in to fiction books to just entertain. The kindle is my number-one, go-to item to try to find some sort of accomplishment when I’m out-and-about.

2. Listen to podcasts

I’m frequently criss-crossing town to and from activities, so I try to keep my phone filled with podcasts from which I can learn and try to make the driving somewhat productive.

3. Keep the car stocked with bars and water

Much of the time, I’m too busy to stop at some fast food restaurant for refreshment, but even if time permitted, it’s cheaper and healthier to just keep my car stocked with water and protein bars or other relatively healthy food that won’t spoil or melt.

4. Maintain and manage a to-do list

In the must-see movie Dead Men Don’t Wear Plaid, the main character is a detective trying to solve the kidnapping of noted cheese scientist, Dr. Forrest. Dr. Forrest has left different clues in the form of lists that the detective has to discover and then deduce the meaning. These lists are everywhere: torn off corners of dollar bills in desk drawers, one is inside a woman’s brooch, another is hidden inside the lid of a jar of coffee. Unfortunately, I find that I’m a lot like Dr. Forrest: I write to-do lists on envelopes or post-it notes only to misplace them and begin anew. Lately, I’ve started using Google Keep in hopes I can be less like the good doctor. Regardless, having a to-do list handy to review and work on during your various excursions can be helpful.

5. Take notes on your bright ideas

Occasionally, I’ll have a half-baked idea on a new blog post or entrepreneurial endeavor: making sure I have tools on hand to write down these brilliant ideas is important. To that end, I always keep pens and small notebooks in the car–that comes in very handy during parent/teacher conferences for note taking. Also, I try to take notes electronically as much as possible. I used to use Evernote quite a bit for that effort, but lately I’ve switched to using Google Docs. Both work great on your smart phone.

6. Read blogs

Like my addiction to podcasts, I subscribe to hundreds of blogs covering my wide variety of interests. Back in the day, I used Google Reader to aggregate the blogs I like to read, but when that was discontinued, I switched to Feedly. Feedly works great on your phone, too!

7. Wear clothes with lots of pockets

How do you effectively carry all the material that aid your on-the-go productivity–kindle, phone, pens, paper, etc.–particularly when you have to march deep into a sports venue, school, or Boy Scout camp? Pockets, I tell you! I own several pairs of cargo pants and shorts that help me haul around the items I need. I’m a big fan of ScottEVest, as well, and own a few of their vests, coats, and jackets that each come with dozens of pockets for storing essentials.

8. Keep a full gym bag and towel in the car

Going to the gym is an important release for me and I’m able to do it less and less as the family’s activities increase. Most days I have to plan out a few days in advance when I can hit the gym, but practices are occasionally canceled, so it’s a good idea to always keep my gym clothes in the car in case I can slip in a workout.

9. Keep portable batteries, charging cables, and related items handy

As a few of these tips rely on electronic devices, I find it helpful to keep portable batteries, cables, and other paraphernalia around and available in case one of your devices gets low on juice. Obviously, keep your devices as charged as possible and don’t forget to charge your portable batteries, as well!

10. Think about other “everyday carry” items

“Preppers” will sometimes discuss the topic of “everyday carry” (EDC) items. Items like flashlights, band-aids, tweezers, multi-tools, and the like. Eyeglass screws come loose, splinters happen, and small items fall into dark spaces. Compiling EDC items and keeping larger kits like first aid kits in the car can help fix an unexpected problem that might otherwise throw off your schedule and undermine your on-the-go work.

Parsing PDFs in Python

March 15, 2018 /

(Apologies for the awful alliteration…boo-ya!)

A few weeks ago, my child competed in the 2018 Queen City Classic Chess Tournament.   Close to 700 students–from Kindergarten to High School age–participated. Despite a long day (nearly twelve hours of travel and tournament play), my boy and his team did well and maintained good spirits.

Since the organizers are courteous enough to publish the match results, I thought this might be a good opportunity (read, nerd moment) to download the data and analyze it a little to see what sort of patterns I might find.

Well, this turned out to be easier said than done. The organizers published the results in 28 PDF files. In my experience, PDF files are no friend to the data analyst. So, my first challenge became downloading the PDF files and transforming them into objects useful for data analysis.

In this post, I will highlight a few of the steps I implemented to acquire the data and transform it into something useful. In a future post, I will discuss some of the analysis I performed on the data. The complete data acquisition code I wrote is available as a Jupyter Notebook on my github page.

Step 1: Getting the data locations

All the PDF files are exposed as download links on the results page, so I decided the simplest approach would be to just scrape the page and use BeautifulSoup to parse out the download links I needed:



1
2
3
4
5
6
7
8
9
10
11
12
# get the download page

result = requests.get("https://ccpf.proscan.com/programs/queen-city-classic-chess-tournament/final-results/")

soup = BeautifulSoup(result.content, "lxml")



# build a list of the download links

pdf_list = []

results_section = soup.find("section", class_="entry-content cf")

pdf_links = results_section.find_all("a")

for pdf_link in pdf_links:

    title = pdf_link.text

    link = pdf_link.attrs['href']

    pdf_list.append([title, link])

Step 2: Download the PDFs

With a list of the download links, I then just iterated through the list to download the files:



1
2
3
4
5
for p in pdf_list:

    # save pdf to disk

    r = requests.get(p[1])

    pdf_file = p[1].split('/')[-1]

    open("./data/" + pdf_file, 'wb').write(r.content)

Step 3: Read the PDFs

Here’s where I hit my challenge: my go-to solution for PDFs, PyPDF2, just didn’t work on these files. So, I had to find another option. My searching revealed the suite of utilities Xpdf tools. Even then, it took some playing with these tools to find a path forward. In the end, I was able to use the tool pdftotext to at least extract the results from the PDF files to simple text files that would be tremendously easier to work with.

In Jupyter Notebook, I looped through my PDF list and used the shell command to run pdftotext:



1
2
3
4
for p in (pdf_list + pl_pdf_list):

    pdf_path = './data/' + p[1].split('/')[-1]

    txt_path = pdf_path.replace('.pdf', '.txt')

    ! {pdftotext} -table {pdf_path} {txt_path}

Step 4: Parse the resulting text

With the results safely in easy-to-read text files, the hard part was over, right? Well, unfortunately, no delimiters came with the data extract, so, I had to get creative again. Enter regular expressions (regex).

Even without delimiters, the extracted data seemed to follow some basic patterns, so I devised three different regular expressions–one for each file type as the results in the PDFs followed one of three schemas–to match the data elements. Initially, I tried numpy’s fromregex hoping for a quick, one-liner win. The function worked well for the most part, but it still stumbled on a few lines inexplicably. So, I just resorted to conventional Python regex:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
re_ind_nr = r'\s+([0-9]+)\s+(.+?(?=\s{2}))\s{2,}(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'

re_ind_r = r'\s+([0-9]+)\s+(.+?(?=\s{2}))\s{2,}(.+?(?=\s{2}))\s+([0-9 ]+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'

re_team = r'([0-9]+)\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+?(?=\s{2}))\s+(.+)'

# the regex still let's some garbage rows come through like headers and footers.  use this list to weed the garbage out

elems_of_rows_to_remove = ['Score', 'Rnd1', 'Code', 'TBrk1']

ind_nr_list = []

ind_r_list = []

team_list = []



# iterate through the list of result files I downloaded.  The PDFs fall into one of three categories: team results, 

# ranked player results, or non-ranked player results.  The file names follow a loose convention: if "team" or "tm"

# is in the file name, that file is a list of team results.  If a file name starts with "n", that file represents

# results of non-ranked players.  All the rest are results of ranked players.

for p in pdf_list:

    title = p[0]

    txt_file = './data/{0}'.format(p[1].split('/')[-1].replace('.pdf', '.txt'))

    with open(txt_file, 'r') as f:

        t = f.read()

        if 'team' in title.lower() or 'tm' in title.lower():

            l = re.findall(re_team, t)

            l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]

            [team_list.append(r) for r in l]

        elif title.lower().startswith('n'):

            l = re.findall(re_ind_nr, t)

            l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]

            [ind_nr_list.append(r) for r in l]

        else:

            l = re.findall(re_ind_r, t)

            l = [[title] + list(r) for r in l if not any(i in r for i in elems_of_rows_to_remove)]

            [ind_r_list.append(r) for r in l]

Step 5: Call it a day

Finally, I had the data in three different lists I could work with, but I’ll save that part for another day. Again, my complete code is at my github page. Hopefully soon, I’ll find some extra time to do the analysis I originally intended.

« Older posts Newer posts »

© 2024 DadOverflow.com

Theme by Anders Noren — Up ↑