SciCommander - track provenance of any shell command
Nov 9, 2023: I haven't written much about a new tool I've been working on in some extra time: SciCommander. I just presented a poster about it at the Swedish Bioinformatics Workshop 2023 , so perhaps let me first present you the poster instead of re-iterating what it is (click to view large ...

Troubleshooting Nextflow pipelines
Nov 1, 2023: Background - The number one skill In my new position as a bioinformatician in clinical microbiology at the Karolinska University Hospital, I'm no longer only developing my own pipeline tools :) but am also a user of Nextflow, which is the leading pipeline tool at least in clinic ...

Random notes from installing Debian 11 with separate accounts for work and private
Mar 24, 2022: After having run Qubes OS 4.1 for a couple of weeks, mainly to enable a way to separate my work and private content, I realized that I could actually solve the same thing with two accounts on a normal Linux operating system, while avoiding some of the annoyances and limitations I ...

Installing Qubes OS
Dec 29, 2021: I just switched to Qubes OS as operating system on my main work laptop (a Dell Latitude). Or in fact, one of the reasons was to be able to combine work and private hobby coding projects, that's increasinbly been happening on the same machine. Anyways, these are my experiences and ...

Composability in functional and flow-based programming
Feb 12, 2021: I love a lot of the thinking in functional programming (FP): Side-effect free functions - That they do not do hidden things beside doing work on the input and returning output Referential transparency of functions - A bit of the same as above, that what you put in to a functi ...

Crystal: Go-like concurrency with easier syntax
Sep 5, 2020: I have been playing around a lot with concurrency in Go over the years, resulting in libraries such as SciPipe, FlowBase and rdf2smw. My main motivation for looking into Go has been the possibility to use it as a more performant, scaleable and type-safe alternative to Python for ...

Viewing Go test coverage in the browser with one command
Aug 20, 2020: Go has some really nice tools for running tests and analyzing code. One of these functionalities is that you can generate coverage information when running tests, that can later be viewed in a browser using the go tool cover command. It turns out though, since doing it requires e ...

Creating a static copy of a Drupal, Wordpress or other CMS website
Aug 20, 2020: I recently needed to sunset my old Drupal 6 blog at saml.rilspace.com, as Drupal 6 is since long not getting security updates, and might not even keep working with newer versions of PHP and MySQL. I was thus delighted to find that one can create a full static copy of such a site ...

Basic PUB/SUB connection with ZeroMQ in Python
Nov 13, 2019: ZeroMQ is a great way to quickly and simply send messages between multiple programs running on the same or different computers. It is very simple and robust since it doesn't need any central server. Instead it talks directly between the programs through sockets, TCP-connections o ...

Table-driven tests in C#
Nov 2, 2019: Folks in the Go community have championed so called table-driven tests (see e.g. this post by Dave Cheney and the Go wiki) as a way to quickly and easily writing up a bunch of complete test cases with inputs and corresponding expected outputs, and looping over them to execute the ...

SciPipe paper published in GigaScience
Apr 27, 2019: We just wanted to share that the paper on our Go-based workflow library, SciPipe, was just published in GigaScience: Abstract Background The complex nature of biological data has driven the development of specialized software tools. Scientific workflow management systems s ...

Structured Go-routines or framework-less Flow-Based Programming in Go
Mar 2, 2019: I was so happy the other day to find someone else who found the great benefits of a little pattern for how to structure pipeline-heavy programs in Go, which I described in a few posts before. I have been surprised to not find more people using this kind of pattern, which has been ...

Setting up a reasonable and light-weight Linux-like (non-WSL) terminal environment on Windows
Nov 29, 2018: I was looking for was a no-fuss, lightweight, robust and as simple as possible solution to running my normal Bash-based workflow inside the main Windows filesystem, interacting with the Windows world. Turns out there are some solutions. Read on for more info on that. Windows Sub ...

Linked Data Science - For improved understandability of computer-aided research
Sep 21, 2018: This is an excerpt from the "future outlook" section of my thesis titled "Reproducible Data Analysis in Drug Discovery with Scientific Workflows and the Semantic Web" (click for the open access full text), which aims to provide various putative ways towards improved reproducibili ...

Make your commandline tool workflow friendly
May 25, 2018: Update (May 2019): A paper incorporating the below considerations is published: Björn A Grüning, Samuel Lampa, Marc Vaudel, Daniel Blankenberg, "Software engineering for scientific big data analysis" GigaScience, Volume 8, Issue 5, May 2019, giz054, https://doi.org/10.1093/giga ...

To make computational lab note-taking happen, make the journal into a todo-list (a "Todournal")
Apr 13, 2018: Good lab note-taking is hard Good note-taking is in my opinion as important for computational research as for wet lab research. For computational research it is much easier though to forget doing it, since you might not have a physical notebook lying on your desk staring at yo ...

Parsing DrugBank XML (or any large XML file) in streaming mode in Go
Mar 15, 2018: I had a problem in which I thought I needed to parse the full DrugBank dataset, which comes as a (670MB) XML file (For open access papers describing DrugBank, see: [1], [2], [3] and [4]). It turned out what I needed was available as CSV files under "Structure External Links". ...

Equation-centric dataflow programming in Go
Dec 27, 2017: Mathematical notation and dataflow programming Even though computations done on computers are very often based on some type of math, it is striking that the notation used in math to express equations and relations is not always very readily converted into programming code. Out ...

What is a scientific (batch) workflow?
Dec 7, 2017: Workflows and DAGs - Confusion about the concepts Jörgen Brandt tweeted a comment that got me thinking again on something I've pondered a lot lately: "A workflow is a DAG." is really a weak definition. That's like saying "A love letter is a sequence of characters." r ...

Go is growing in bioinformatics workflow tools
Nov 10, 2017: TL;DR: We wrote a post on gopherdata.io, about the growing ecosystem of Go-based workflow tools in bioinformatics. Go read it here It is interesting to note how Google's Go programming language seems to increase in popularity in bioinformatics. Just to give a sample of ...

On Provenance Reports in Scientific Workflows
Oct 19, 2017: Hoping to write up a series of posts that go through some of the design decisions made in the SciPipe, the Go-based scientific workflow library we're working on, to share some of the thinking behind them, and get the opportunity to get feedback and suggestions that might be imple ...

(Almost) ranging over multiple Go channels simultaneously
Oct 5, 2017: In my experiments (flowbase, scipipe) in using Flow-based programming (FBP) principles in pure Go, there is a common pattern occuring all the time: The need to synchronously read a set of values from multiple channels at the same time. That is, if I have three in-bound channels, ...

First production run with SciPipe - A Go-based scientific workflow tool
Sep 28, 2017: Today marked the day when we ran the very first production workflow with SciPipe, the Go-based scientific workflow tool we've been working on over the last couple of years. Yay! :) This is how it looked (no fancy GUI or such yet, sorry): The first result we got in this ...

New paper on RDFIO for interoperable biomedical data management in Semantic MediaWiki
Sep 11, 2017: As my collaborator and M.Sc. supervisor Egon Willighagen already blogged, we just released a paper titled: "RDFIO: extending Semantic MediaWiki for interoperable biomedical data management", with uses cases from Egon and Pekka Kohonen, coding help from Ali King and project superv ...

Notes on launching kubernetes jobs from the Go API
Feb 15, 2017: This post is also published on medium My current work at pharmb.io entails adding kubernetes support to my light-weight Go-based scientific workflow engine, scipipe (kubernetes, or k8s for short, is Google’s open source project for orchestrating container based compute clust ...

SMWCon Fall 2016 - My talk on large RDF imports
Oct 7, 2016: I was invited to give a talk at Semantic MediaWiki (SMW) conference (SMWCon) in Frankfurt last week, on our work on enabling import of RDF datasets into SMW. I have presented at SMWCon before as well (2011: blog, slides, video, 2013: slides), so it was nice to re-connect with som ...

Tutorial: Luigi for Scientific Workflows
Jun 21, 2016: This is a Luigi tutorial I held at the e-Infrastructures for Massively parallel sequencing workshop (Video archive) at SciLifeLab Uppsala in January 2015, moved here for future reference. What is Luigi? Luigi is a batch workflow system written in Python and developed by ...

The unexpected convenience of JSON on the commandline
Dec 8, 2015: I was working with a migration from drupal to processwire CMS:es, where I wanted to be able to pipe data, including the body field with HTML formatting and all, through multiple processing steps in a flexible manner. I'd start with an extraction SQL query, through a few component ...

Wanted: Dynamic workflow scheduling
Oct 26, 2015: Photo credits: Matthew Smith / Unsplash In our work on automating machine learning computations in cheminformatics with scientific workflow tools, we have came to realize something; Dynamic scheduling in scientific workflow tools is very important and sometimes badly needed. ...

FBP inspired data flow syntax: The missing piece for the success of functional programming?
Jul 16, 2015: Often when I suggest people have a look at Flow-based Programming (FBP) or Data Flow for one reason or another, people are often put off by the strong connection between these concepts and graphical programming. That is, the idea that programs will be easier to understan ...

A few thoughts on organizing computational (biology) projects
Jun 23, 2015: I read this excellent article with practical recommendations on how to organize a computational project, in terms of directory structure. Directory structure matters The importance of a good directory structure seems to often be overlooked in teaching about computationa ...

Flow-based programming and Erlang style message passing - A Biology-inspired idea of how they fit together
Jun 13, 2015: I think Erlang/Elixir fits great as control plane or service-to-service messaging layer for distributing services built with flow-based programming Just back from a one day visit to Erlang User Conference. I find the Erlang virtual machine fascinating. And with the new Elixir ...

A cheatsheet for the iRODS rule language
Jun 11, 2015: iRODS, the "integrated rule oriented data system" is a super cool system for managing datasets consisting of files, from smallish ones, to really large ones counted in petabytes, and possibly spanning multiple continents. There's a lot to be said about iRODS (up for another blog ...

Workflow tool makers: Allow defining data flow, not just task dependencies
Jun 10, 2015: Upsurge in workflow tools There seem to be a little upsurge in light-weight - often python-based - workflow tools for data pipelines in the last couple of years: Spotify's Luigi, OpenStack's Mistral, Pinterest's Pinball, and recently AirBnb's Airflow, to name a few. These are ...

Patterns for composable concurrent pipelines in Go
Jun 1, 2015: I realize I didn't have a link to my blog on Gopher Academy, on patterns for compoasable concurrent pipelines in Go(lang), so here it goes: blog.gopheracademy.com/composable-pipelines-pattern ...

The role of simplicity in testing and automation
Mar 23, 2015: Disclaimer: Don't take this too seriously ... this is "thinking-in-progress" :) It just struck me the other minute, how simplicity is the key theme behind two very important areas in software development, that I've been dabbling with quite a bit recently: Testing, and automation ...

The problem with make for scientific workflows
Mar 14, 2015: The workflow problem solved once and for all in 1979? As soon as the topic of scientific workflows is brought up, there are always a few make fans fervently insisting that the problem of workflows is solved once and for all with GNU make, written first in the 70's :) Pe ...

Dynamic Navigation for Higher Performance
Mar 11, 2015: Improving performance in Delphi Bold MDA applications by replacing navigation code with derived links in the model This post on Model Driven Architecture in Delphi and Bold, by Rolf Lampa, has been previously published on howtodothings.com. Modeling class structures takes some ...

Profiling and creating call graphs for Go programs
Aug 8, 2013: In trying to get my head around the code of the very interesting GoFlow library, (for flow-based programming in Go), and the accompanying flow-based bioinformatics library I started hacking on, I needed to get some kind of visualization (like a call graph) ... something like this ...

(E)BNF parser for parts of the Galaxy ToolConfig syntax with ANTLR
Jul 28, 2011: As blogged earlier, I'm currently into parsing the syntax of some definitions for the parameters and stuff of command line tools. As said in the linked blog post, I was pondering whether to use the Galaxy Toolconfig format or the DocBook CmdSynopsis format. It turned out though W ...

Partial Galaxy ToolConfig to DocBook CmdSynopsis conversion with XSLT RegEx
Jul 21, 2011: As blogged about before, I was interested in knowing the difference between the Galaxy Toolconfig, and the DocBook cmdsynopsis format, for the purpose of automatically generating wizards (see an example that I screencasted here) to fill in the required parameters to command line ...

Answering questions without answers - by wrapping simulations in semantics
Feb 17, 2010: (This post is migrated from this post on Samuels old tech blog) Explicit knowledge is too expensive There are lots of things that can't be answered by a computer from data alone. Maybe the majority of what we humans perceive as knowledge is inferred from a combination of data ( ...

A "Hello World" program in SWI-Prolog
Sep 22, 2009: This is how to create a "Hello World" kind of program in SWI-Prolog, as installed on Ubuntu (Jaunty). Prolog usage can be divided into two phases, one in which you write the Prolog program, and the other in which you query the program. The first phase is easiest done in a separa ...