e · l · n

Need help with 3D CAD project and/or parametric design in Rhino 3D and Grasshopper? Then get in touch with Rolf at RIL Labs!

Debugging inside Jinja templates using pdb/ipdb
Mar 4, 2024: I'm working on a static reporting tool using the Jinja2 templating engine for Python. I was trying to figure out a way to enter into the Jinja templating code with the pdb/ipdb commandline debugger. I tried creating an .ipdbrc file in my local directory with the line: path/to ...

Read more »

SciCommander - track provenance of any shell command
Nov 9, 2023: I haven't written much about a new tool I've been working on in some extra time: SciCommander. I just presented a poster about it at the Swedish Bioinformatics Workshop 2023 , so perhaps let me first present you the poster instead of re-iterating what it is (click to view large ...

Read more »

Troubleshooting Nextflow pipelines
Nov 1, 2023: Background - The number one skill In my new position as a bioinformatician in clinical microbiology at the Karolinska University Hospital, I'm no longer only developing my own pipeline tools :) but am also a user of Nextflow, which is the leading pipeline tool at least in clinic ...

Read more »

Random notes from installing Debian 11 with separate accounts for work and private
Mar 24, 2022: After having run Qubes OS 4.1 for a couple of weeks, mainly to enable a way to separate my work and private content, I realized that I could actually solve the same thing with two accounts on a normal Linux operating system, while avoiding some of the annoyances and limitations I ...

Read more »

Installing Qubes OS
Dec 29, 2021: I just switched to Qubes OS as operating system on my main work laptop (a Dell Latitude). Or in fact, one of the reasons was to be able to combine work and private hobby coding projects, that's increasinbly been happening on the same machine. Anyways, these are my experiences and ...

Read more »

Composability in functional and flow-based programming
Feb 12, 2021: I love a lot of the thinking in functional programming (FP): Side-effect free functions - That they do not do hidden things beside doing work on the input and returning output Referential transparency of functions - A bit of the same as above, that what you put in to a functi ...

Read more »

Crystal: Go-like concurrency with easier syntax
Sep 5, 2020: I have been playing around a lot with concurrency in Go over the years, resulting in libraries such as SciPipe, FlowBase and rdf2smw. My main motivation for looking into Go has been the possibility to use it as a more performant, scaleable and type-safe alternative to Python for ...

Read more »

Viewing Go test coverage in the browser with one command
Aug 20, 2020: Go has some really nice tools for running tests and analyzing code. One of these functionalities is that you can generate coverage information when running tests, that can later be viewed in a browser using the go tool cover command. It turns out though, since doing it requires e ...

Read more »

Creating a static copy of a Drupal, Wordpress or other CMS website
Aug 20, 2020: I recently needed to sunset my old Drupal 6 blog at saml.rilspace.com, as Drupal 6 is since long not getting security updates, and might not even keep working with newer versions of PHP and MySQL. I was thus delighted to find that one can create a full static copy of such a site ...

Read more »

Basic PUB/SUB connection with ZeroMQ in Python
Nov 13, 2019: ZeroMQ is a great way to quickly and simply send messages between multiple programs running on the same or different computers. It is very simple and robust since it doesn't need any central server. Instead it talks directly between the programs through sockets, TCP-connections o ...

Read more »

Table-driven tests in C#
Nov 2, 2019: Folks in the Go community have championed so called table-driven tests (see e.g. this post by Dave Cheney and the Go wiki) as a way to quickly and easily writing up a bunch of complete test cases with inputs and corresponding expected outputs, and looping over them to execute the ...

Read more »

SciPipe paper published in GigaScience
Apr 27, 2019: We just wanted to share that the paper on our Go-based workflow library, SciPipe, was just published in GigaScience: Abstract Background The complex nature of biological data has driven the development of specialized software tools. Scientific workflow management systems s ...

Read more »

Structured Go-routines or framework-less Flow-Based Programming in Go
Mar 2, 2019: I was so happy the other day to find someone else who found the great benefits of a little pattern for how to structure pipeline-heavy programs in Go, which I described in a few posts before. I have been surprised to not find more people using this kind of pattern, which has been ...

Read more »

Setting up a reasonable and light-weight Linux-like (non-WSL) terminal environment on Windows
Nov 29, 2018: I was looking for was a no-fuss, lightweight, robust and as simple as possible solution to running my normal Bash-based workflow inside the main Windows filesystem, interacting with the Windows world. Turns out there are some solutions. Read on for more info on that. Windows Sub ...

Read more »

Linked Data Science - For improved understandability of computer-aided research
Sep 21, 2018: This is an excerpt from the "future outlook" section of my thesis titled "Reproducible Data Analysis in Drug Discovery with Scientific Workflows and the Semantic Web" (click for the open access full text), which aims to provide various putative ways towards improved reproducibili ...

Read more »

Make your commandline tool workflow friendly
May 25, 2018: Update (May 2019): A paper incorporating the below considerations is published: Björn A Grüning, Samuel Lampa, Marc Vaudel, Daniel Blankenberg, "Software engineering for scientific big data analysis" GigaScience, Volume 8, Issue 5, May 2019, giz054, https://doi.org/10.1093/giga ...

Read more »

To make computational lab note-taking happen, make the journal into a todo-list (a "Todournal")
Apr 13, 2018: Good lab note-taking is hard Good note-taking is in my opinion as important for computational research as for wet lab research. For computational research it is much easier though to forget doing it, since you might not have a physical notebook lying on your desk staring at yo ...

Read more »

Parsing DrugBank XML (or any large XML file) in streaming mode in Go
Mar 15, 2018: I had a problem in which I thought I needed to parse the full DrugBank dataset, which comes as a (670MB) XML file (For open access papers describing DrugBank, see: [1], [2], [3] and [4]). It turned out what I needed was available as CSV files under "Structure External Links". ...

Read more »

Equation-centric dataflow programming in Go
Dec 27, 2017: Mathematical notation and dataflow programming Even though computations done on computers are very often based on some type of math, it is striking that the notation used in math to express equations and relations is not always very readily converted into programming code. Out ...

Read more »

What is a scientific (batch) workflow?
Dec 7, 2017: Workflows and DAGs - Confusion about the concepts Jörgen Brandt tweeted a comment that got me thinking again on something I've pondered a lot lately: "A workflow is a DAG." is really a weak definition. That's like saying "A love letter is a sequence of characters." r ...

Read more »

Go is growing in bioinformatics workflow tools
Nov 10, 2017: TL;DR: We wrote a post on gopherdata.io, about the growing ecosystem of Go-based workflow tools in bioinformatics. Go read it here It is interesting to note how Google's Go programming language seems to increase in popularity in bioinformatics. Just to give a sample of ...

Read more »

On Provenance Reports in Scientific Workflows
Oct 19, 2017: Hoping to write up a series of posts that go through some of the design decisions made in the SciPipe, the Go-based scientific workflow library we're working on, to share some of the thinking behind them, and get the opportunity to get feedback and suggestions that might be imple ...

Read more »

(Almost) ranging over multiple Go channels simultaneously
Oct 5, 2017: In my experiments (flowbase, scipipe) in using Flow-based programming (FBP) principles in pure Go, there is a common pattern occuring all the time: The need to synchronously read a set of values from multiple channels at the same time. That is, if I have three in-bound channels, ...

Read more »

First production run with SciPipe - A Go-based scientific workflow tool
Sep 28, 2017: Today marked the day when we ran the very first production workflow with SciPipe, the Go-based scientific workflow tool we've been working on over the last couple of years. Yay! :) This is how it looked (no fancy GUI or such yet, sorry): The first result we got in this ...

Read more »

New paper on RDFIO for interoperable biomedical data management in Semantic MediaWiki
Sep 11, 2017: As my collaborator and M.Sc. supervisor Egon Willighagen already blogged, we just released a paper titled: "RDFIO: extending Semantic MediaWiki for interoperable biomedical data management", with uses cases from Egon and Pekka Kohonen, coding help from Ali King and project superv ...

Read more »

Notes on launching kubernetes jobs from the Go API
Feb 15, 2017: This post is also published on medium My current work at pharmb.io entails adding kubernetes support to my light-weight Go-based scientific workflow engine, scipipe (kubernetes, or k8s for short, is Google’s open source project for orchestrating container based compute clust ...

Read more »

SMWCon Fall 2016 - My talk on large RDF imports
Oct 7, 2016: I was invited to give a talk at Semantic MediaWiki (SMW) conference (SMWCon) in Frankfurt last week, on our work on enabling import of RDF datasets into SMW. I have presented at SMWCon before as well (2011: blog, slides, video, 2013: slides), so it was nice to re-connect with som ...

Read more »

Tutorial: Luigi for Scientific Workflows
Jun 21, 2016: This is a Luigi tutorial I held at the e-Infrastructures for Massively parallel sequencing workshop (Video archive) at SciLifeLab Uppsala in January 2015, moved here for future reference. What is Luigi? Luigi is a batch workflow system written in Python and developed by ...

Read more »

The unexpected convenience of JSON on the commandline
Dec 8, 2015: I was working with a migration from drupal to processwire CMS:es, where I wanted to be able to pipe data, including the body field with HTML formatting and all, through multiple processing steps in a flexible manner. I'd start with an extraction SQL query, through a few component ...

Read more »

Wanted: Dynamic workflow scheduling
Oct 26, 2015: Photo credits: Matthew Smith / Unsplash In our work on automating machine learning computations in cheminformatics with scientific workflow tools, we have came to realize something; Dynamic scheduling in scientific workflow tools is very important and sometimes badly needed. ...

Read more »

FBP inspired data flow syntax: The missing piece for the success of functional programming?
Jul 16, 2015: Often when I suggest people have a look at Flow-based Programming (FBP) or Data Flow for one reason or another, people are often put off by the strong connection between these concepts and graphical programming. That is, the idea that programs will be easier to understan ...

Read more »

A few thoughts on organizing computational (biology) projects
Jun 23, 2015: I read this excellent article with practical recommendations on how to organize a computational project, in terms of directory structure. Directory structure matters The importance of a good directory structure seems to often be overlooked in teaching about computationa ...

Read more »

Flow-based programming and Erlang style message passing - A Biology-inspired idea of how they fit together
Jun 13, 2015: I think Erlang/Elixir fits great as control plane or service-to-service messaging layer for distributing services built with flow-based programming Just back from a one day visit to Erlang User Conference. I find the Erlang virtual machine fascinating. And with the new Elixir ...

Read more »

A cheatsheet for the iRODS rule language
Jun 11, 2015: iRODS, the "integrated rule oriented data system" is a super cool system for managing datasets consisting of files, from smallish ones, to really large ones counted in petabytes, and possibly spanning multiple continents. There's a lot to be said about iRODS (up for another blog ...

Read more »

Workflow tool makers: Allow defining data flow, not just task dependencies
Jun 10, 2015: Upsurge in workflow tools There seem to be a little upsurge in light-weight - often python-based - workflow tools for data pipelines in the last couple of years: Spotify's Luigi, OpenStack's Mistral, Pinterest's Pinball, and recently AirBnb's Airflow, to name a few. These are ...

Read more »

Patterns for composable concurrent pipelines in Go
Jun 1, 2015: I realize I didn't have a link to my blog on Gopher Academy, on patterns for compoasable concurrent pipelines in Go(lang), so here it goes: blog.gopheracademy.com/composable-pipelines-pattern ...

Read more »

The role of simplicity in testing and automation
Mar 23, 2015: Disclaimer: Don't take this too seriously ... this is "thinking-in-progress" :) It just struck me the other minute, how simplicity is the key theme behind two very important areas in software development, that I've been dabbling with quite a bit recently: Testing, and automation ...

Read more »

The problem with make for scientific workflows
Mar 14, 2015: The workflow problem solved once and for all in 1979? As soon as the topic of scientific workflows is brought up, there are always a few make fans fervently insisting that the problem of workflows is solved once and for all with GNU make, written first in the 70's :) Pe ...

Read more »

Dynamic Navigation for Higher Performance
Mar 11, 2015: Improving performance in Delphi Bold MDA applications by replacing navigation code with derived links in the model This post on Model Driven Architecture in Delphi and Bold, by Rolf Lampa, has been previously published on howtodothings.com. Modeling class structures takes some ...

Read more »

Profiling and creating call graphs for Go programs
Aug 8, 2013: In trying to get my head around the code of the very interesting GoFlow library, (for flow-based programming in Go), and the accompanying flow-based bioinformatics library I started hacking on, I needed to get some kind of visualization (like a call graph) ... something like this ...

Read more »

(E)BNF parser for parts of the Galaxy ToolConfig syntax with ANTLR
Jul 28, 2011: As blogged earlier, I'm currently into parsing the syntax of some definitions for the parameters and stuff of command line tools. As said in the linked blog post, I was pondering whether to use the Galaxy Toolconfig format or the DocBook CmdSynopsis format. It turned out though W ...

Read more »

Partial Galaxy ToolConfig to DocBook CmdSynopsis conversion with XSLT RegEx
Jul 21, 2011: As blogged about before, I was interested in knowing the difference between the Galaxy Toolconfig, and the DocBook cmdsynopsis format, for the purpose of automatically generating wizards (see an example that I screencasted here) to fill in the required parameters to command line ...

Read more »

Answering questions without answers - by wrapping simulations in semantics
Feb 17, 2010: (This post is migrated from this post on Samuels old tech blog) Explicit knowledge is too expensive There are lots of things that can't be answered by a computer from data alone. Maybe the majority of what we humans perceive as knowledge is inferred from a combination of data ( ...

Read more »

A "Hello World" program in SWI-Prolog
Sep 22, 2009: This is how to create a "Hello World" kind of program in SWI-Prolog, as installed on Ubuntu (Jaunty). Prolog usage can be divided into two phases, one in which you write the Prolog program, and the other in which you query the program. The first phase is easiest done in a separa ...

Read more »