Programming on Neil Grogan

Convert Bank Transactions XLS to CSV in Python

Fri, 25 Nov 2022 00:00:00 +0000

I’ve written previously on importing transactions to hledger/ledger from KBC bank in JavaScript and PTSB bank in Python. I took different approaches to each:

For KBC, you needed to log in and run Javascript which scrape the transaction table and download it formatted as CSV
For PTSB, the script automatted logging in, get the transaction table and save locally as CSV

Both approaches are valid - but suffer from the same issues: any change the bank makes to it website needs to be updated in the code. The KBC/JavaScript approach was a bit more robust in that it would just search for rows on a website and download as CSV.

Scraping Transaction Data from KBC Bank

Mon, 13 May 2019 00:00:00 +0000

In a previous post, I mentioned importing transactions using ledger/hledger and plain text accounting. As my former bank PTSB recently raised their fees, I decided to move to KBC. I was excited to see, as part of the open payments directive, they have a nice developer portal. I reached out to KBC’s dev team, but alas they are only accepting registered companies, who meet stringent criteria. They told me they hope to open it up soon to end users, I live in hope!

Python 3 In-Memory Zip File

Sun, 14 Jan 2018 00:00:00 +0000

In Python, BytesIO is the way to store binary data in memory. Most examples you’ll see using zip files in memory is to store string data and indeed the most common example you’ll find online from the zipfile module is zipfile.writestr(file_name, "Text Data"). But what if you want to store binary data of a PDF or Excel Spreadsheet that’s also in memory? You could use zipfile.write() (designed to take binary data) but then you can’t specify a filename (since our in-memory file was never written to a location on disk). The reason for this is simple: for a web request or for a test case, you shouldn’t need to store any files on disk.

Scraping Data from your Bank in Python

Wed, 12 Jul 2017 00:00:00 +0000

As part of my previous posts, I talked about ledger and plain text accounting. The only part missing is that you need a method to import transactions from your bank. For this I have been doing this by hand, bi-weekly. I would have to do the following:

Log in to online banking
Go to the transactions page
Select the date range for transactions I needed (double check last date of transaction in ledger at this point)
Download the Microsoft Excel format file that wasn’t in the proper format
Convert this Excel file into a CSV file that matched my import format (watch the dates, is it YYYY-MM-DD or DD/MM/YYYY?)
Finally import the CSV file into ledger
Check the balance matches between my online banking and ledger

Sounds like a lot of work right?

Slack Bots for Work

Thu, 16 Feb 2017 00:00:00 +0000

In a previous post, I mentioned how I get notified of the restaurant menu via a Ruby script. Recently I’ve moved to a totally different product area and the main communication channel we use is Slack. Naturally enough, I ported the Ruby code I wrote, and it now posts the menu every day to our Slack channel.

This also got me thinking of what other information would be handy to have. I scouted around for ideas and came up with an obvious one: reminder of the bus times to and from the office. So here’s my bus times notification slack bot:

Automatic Newsletter Cleanup in Gmail

Fri, 19 Aug 2016 00:00:00 +0000

If you haven’t tried Google Apps Script, I found a really nifty use for it: smart filtering for email. Wait, shouldn’t I just use Gmails’ built-in filters? As it turns out you can’t - my filter needs to act on email that matched that filter in the past. So in other words: a filter can only act on email it actually “filters”, which kinda makes sense! I’m a big fan of automation (and email is ripe for automation), as you can see from my post on meetings in Outlook.

Fill hours worked in SAP Netweaver Automatically

Mon, 01 Feb 2016 00:00:00 +0000

Continuing the theme of automation, one of the most repetitive tasks if you work for a big company is timesheets. So I set out to rectify this by scripting it!

Start with you configuration, I named mine hours.ini:

[DEFAULT]
url = FILL_ME_IN
username = FILL_ME_IN
password = FILL_ME_IN

then we need the magic of Selenium to do the heavy lifting, so we install it:

$ pip3 install selenium

I called this script, unsurprisingly hours.py:

Automated 'Push' Restaurant menu

Sun, 10 Jan 2016 00:00:00 +0000

I love trying to automate the world, it just feels like magic some of the time! I also really enjoy information coming to me, instead of having to seek it. As we are still only in 2016, we have no world killing Artificial Intelligence (yet). So we have to start small, ease the first world problems! So I decided to make my workplaces’ restaurant menu come to me!

I decided to write it in Ruby and use push notifications, rather than email or SMS. It uses a web automation framework called Selenium, which is available in many languages, including Javascript, Python, Java and obviously Ruby!

Adventures with React

Thu, 22 Oct 2015 00:00:00 +0000

Recently I’d seen a post on Hacker News about a course for the React JavaScript Framework. I haven’t done much with JavaScript, apart from a course in work, which was focused on the language itself and jQuery. React is a front end framework, which allows you to build reusable components (and generate them with data.)

The only real experience I have is writing an application (using Node.js) for my MSc course which was an AJAX application for interacting with Amazon SimpleDB. It was fun to write and weird that I could use the same language from the front-end to the back-end.

Automatically Decline and Delete or Accept and Delete Outlook 2010 Meetings

Tue, 06 Oct 2015 00:00:00 +0000

You can follow the Microsoft TechNet guide to add VisualBasic code in Outlook rules.

You can just replace the code they give with this:

Sub AutoDeclineMeetings(oRequest As MeetingItem)

' If its not a meeting, we don't process
If oRequest.MessageClass <> "IPM.Schedule.Meeting.Request" Then
 Exit Sub
End If

' Get the appointment in the meeting
Dim oAppt As AppointmentItem
Set oAppt = oRequest.GetAssociatedAppointment(True)

' Send a decline response
Dim oResponse
 Set oResponse = oAppt.Respond(olMeetingDeclined, True)
 oResponse.Send

' Lastly, delete the message
oRequest.Delete
End Sub

Sub AutoAcceptMeetings(oRequest As MeetingItem)

' If its not a meeting, we don't process
If oRequest.MessageClass <> "IPM.Schedule.Meeting.Request" Then
 Exit Sub
End If

' Get the appointment in the meeting
Dim oAppt As AppointmentItem
Set oAppt = oRequest.GetAssociatedAppointment(True)

' Send an accept response
Dim oResponse
 Set oResponse = oAppt.Respond(olMeetingAccepted, True)
 oResponse.Send

' Lastly, (optionally) delete the message
'oRequest.Delete
End Sub

For anything else you may want to do with the meeting, check the Outlook Visual Basic Developer Docs.

Full Stack Web Application with Node.js Tutorial

Mon, 31 Mar 2014 00:00:00 +0000

I’ve followed this tutorial personally, not using Mongo as the database at first. It’s clear and to the point and provides just what you need to know to get started. Highly recommended.

Programming is hard

Mon, 07 May 2012 00:00:00 +0000

A great essay on why programming is hard:

It’s harder than you think. Right now you’re probably underestimating the amount of frustration and discomfort you’re about to experience, without realizing that by doing so you’re creating all sorts of subtle barriers to obtaining a deep understanding of programming.

Language does matter

I find this part extremely true: my first language is Java. Every programming language I’ve tried since has been through comparing it to Java. I’ve had criticism of each different language based on my assumptions. Scripting languages being prone to bad coding because they are not strictly typed, having to do tedious work with languages that require memory management (my time is too valuable!).

Very Simple Java Twitter Bot (Twitter4J)

Wed, 01 Feb 2012 00:00:00 +0000

Below is code for the start of a Twitter bot I am going to build in Java. It’s the most basic way of getting Oauth working (with any account, not just your developer account) and it shows your timeline and can update your status - that’s it for now. The neat thing is it uses Java’s awesome serialisation, so you only should have to authorise your twitter account once!

What you will need:

Twitter4j Libraries
Oauth Consumer and Secret Key off twitter
That is it!

What the code does:

Once only:

Output Percentage of File (Bash code)

Sat, 05 Feb 2011 00:00:00 +0000

I wrote this script recently for a friend in a job who needed to output a certain percentage of a log file, but no more and no less. This was in Linux using bash, so I had a go at writing a solution, which you see below. Posting this up in case it’s useful to anyone else!

 # !/bin/sh # Public Domain, by Neil Grogan 2010 # Script
to output last 30% of file by lines

 OLDFILE=&quot;oldlog.txt&quot;
 NEWFILE="newlog.txt"
 PERCENT=0.7

 #Use wc to count lines
 LINES=$(wc -l $OLDFILE | awk '{ print $1}')

 #Linespercent = 2 decimal places, lines by percent, round to whole
 LINESPERCENT=$(echo "scale=2; $LINES*$PERCENT" | bc | xargs printf "%1.0f" )

 # Use tail to get last 30% and output, can use tail -s with sleep time to have it run on sched.
 tail -n $LINESPERCENT $OLDFILE >> $NEWFILE

Technology New Years Resolutions

Mon, 04 Jan 2010 00:00:00 +0000

Some people make new years resolutions about eating healthier, exercising more etc. I find that boring, who doesn’t want to be healthier? So I am making a list of technology resolutions:

Learn C in XCode on a Mac
Make an awesome CRM program for College degree (Python, Google App Engine)
Learn Vim and Emacs bindings, using them for code
Learn to use Git, possibly mercurial
Learn some Obj-C with Cocoa
Learn some C++ with Qt
Learn Python +Tcl /tk
Learn more about the design of Linux kernel and study some of it’s code
Do Sysadmin stuff on my server on Linode; like setting up Sendmail ,IRC Servers and other stuff
If I have time, investigate CakePHP, Django, maybe Ruby (on Rails?) and it’s ilk

That should keep me busy besides college and working! I didn’t include any security stuff, as we’re doing best practice for my honours degree next year. If anyone has any of their own, I’d love to see, or any resources to help me complete my list ^-^ Happy 2010!

Why Should I Care What Color the Bikeshed Is?

Mon, 07 Dec 2009 00:00:00 +0000

“The really, really short answer is that you should not. The somewhat longer answer is that just because you are capable of building a bikeshed does not mean you should stop others from building one just because you do not like the color they plan to paint it. This is a metaphor indicating that you need not argue about every little feature just because you know enough to do so. Some people have commented that the amount of noise generated by a change is inversely proportional to the complexity of the change.”

Comp Sci - An Interview with Stanford's Undegraduate Professor

Fri, 07 Sep 2007 00:00:00 +0000

This is the guy who runs Stanford University Undergraduate programme. He talks about the future of computer science:

Computer Science will evolve more than most other subjects as it expands into other areas, especially Biology and other areas people don’t normally associate with it
Industry ties with Colleges are going to be very important
Computer Science with Law is the next big area

Thanks to Robert Scoble for doing the interview.