Tag: Data

How to get Sunrise and Sunset times for data analysis

jinsngjung / Pixabay

Today is 2nd January and the days are getting longer again. I was thinking about sunrise and sunset times and wondering if I should do some data analysis and plotting of these to visualize how they change over the course of a year. Of course, the first step is to get a suitable data set.

You can get a data set of sunrise and sunset times from this page on the US Naval Observatory (USNO) website. Sunrise and sunset times depend on your specific location on the globe so you have to specify the location for your data set. Using the “Form B – Locations Worldwide” section on the page it is possible to enter precise coordinates for any location in terms of latitude and longitude. You can also do it for a timezone, but this is very broad and not precise enough for my liking.

Getting your coordinates

You can get your exact coordinates from the website of the National Geospatial-Intelligence Agency (NGA). You could also do the same on OpenStreetMap, by right-clicking on a location and choosing “show address” (or on Google Maps), but the NGA system gives more precise information about the type of coordinate system used. FYI, OpenStreetMap gives the coordinate that corresponds to “Latitude North” and “Longitude East”. But if your Longitude East value is negative, this just corresponds to the Longitude West value if you make it positive. However, the NGA form wants the coordinates in minutes and seconds, not a decimal fraction of minutes, so if you don’t want to convert units I recommend using the NGA site.

Downloading the sunset and sunrise data

Once you know the coordinates, you can enter them into the USNO page and click “Compute table” for any year that you specified in the “year” field. Here are the settings I used to get the data for Greenwich, London:

USNO sunrise webform

These are the settings I used in the USNO “Complete Sun and Moon Data for One Day” form to get data for Greenwich, UK.

You should now see a text based data table with columns for sunrise and sunset for each month for that year, with a row for each day in the month. Go ahead and save as a text file on your local drive and then you can use Pandas or whatever to wrangle the data into whatever format you want. I will make a further post at a later date with some suggestions. Bye for now and good luck!

USNO sunrise data file

This is the USNO sunrise data file for Greenwich, UK.


Configuring KNIME to work with Python 2.7.x on Windows

Apparently it is tricky to get Python integration working in the KNIME Analytics Platform. If you read the official guide too quickly you can miss some critical information at the bottom of the page. I was getting an error complaining that the google.protobuf library was missing even though I thought that I had everything installed correctly:

Library google.protobuf is missing, required minimum version is 2.5.0

Here is the correct sequence of actions to fix this problem and get Python integration working in KNIME. It worked for me, so I hope it works for you too!

  1. If you don’t already have it on your system, download and install Anaconda Python. I use the Python 2.7 version of Anaconda. I recommend installing Anaconda in the folder “C:\Anaconda2”.
  2. Create a new conda environment by using the following code at the windows command prompt (or power shell):
    conda create -y -n py27_knime python=2.7 pandas jedi protobuf
  3. Create a new Windows batch file with the following contents:
    @REM Adapt the directory in the PATH to your system
    @SET PATH=C:\Anaconda2\Scripts;%PATH%
    @CALL activate py27_knime || ECHO Activating py27_knime failed
    @python %*

    Of course, you might have to change the path to your Anaconda2 installation if you installed it to a different path. Save the batch file somewhere, like your user directory or the KNIME workspace folder. I named my file “py27.bat” and placed it in the knime-workspace folder.
    Just in case anyone reading this is confused…. a windows batch file is just a text file that is saved with the file extension “.bat”. You can create it in any text editor by creating a new empty text file, pasting the above four lines of text into it and saving the file as “py27.bat”.

  4. If you haven’t already, download and install KNIME Analytics Platform. At the time of writing, the latest version is 3.4.0, so that’s the one I used. You might as well get the installer with all the free extensions included, presuming you have enough disk space for it and your internet connection is decent enough. This will save having to install further packages later, although to be fair KNIME seems to do a fairly good job at installing packages on the fly (after asking you first) whenever you load a workspace that needs them.

    Downloading KNIME

  5. Start KNIME. Go to File>Preferences, then KNIME > Python scripting. In the “path to the local python executable” paste the full path to your batch file, e.g. “C:\Users\yourname\knime-workspace\py27.bat”. To be future-proofed, also do this for KNIME > Python (Labs).
    KNIME now calls our batch file instead of calling the system-wide Python executable directly. The batch file activates the “py27_knime” environment before calling Python. This makes sure that the additional libraries required by KNIME are available.
    I guess you could also get your usual Python environment working with KNIME by installing additional packages, but let’s just do what the knowledgeable guys at KNIME have suggested this time. 🙂

    Python scripting preferences for KNIME Analytics Platform in Windows

  6. Now restart KNIME. Try opening or creating a workflow with a node that uses some Python code. It should work now!