e ·
l ·
nNeed help with 3D CAD project and/or parametric design in Rhino 3D and Grasshopper? Then get in touch with Rolf at RIL Labs!
SciCommander - track provenance of any shell command
Nov 9, 2023: I haven't written much about a new tool I've been working on in some extra time: SciCommander.
I just presented a poster about it at the Swedish Bioinformatics Workshop 2023 , so perhaps let me first present you the poster instead of re-iterating what it is (click to view large ...
Troubleshooting Nextflow pipelines
Nov 1, 2023: Background - The number one skill
In my new position as a bioinformatician in clinical microbiology at the Karolinska University Hospital, I'm no longer only developing my own pipeline tools :) but am also a user of Nextflow, which is the leading pipeline tool at least in clinic ...
Installing Qubes OS
Dec 29, 2021: I just switched to Qubes OS as operating system on my main work laptop (a Dell Latitude). Or in fact, one of the reasons was to be able to combine work and private hobby coding projects, that's increasinbly been happening on the same machine. Anyways, these are my experiences and ...
Composability in functional and flow-based programming
Feb 12, 2021: I love a lot of the thinking in functional programming (FP):
Side-effect free functions - That they do not do hidden things beside doing work on the input and returning output
Referential transparency of functions - A bit of the same as above, that what you put in to a functi ...
Crystal: Go-like concurrency with easier syntax
Sep 5, 2020: I have been playing around a lot with concurrency in Go over the years, resulting in libraries such as SciPipe, FlowBase and rdf2smw. My main motivation for looking into Go has been the possibility to use it as a more performant, scaleable and type-safe alternative to Python for ...
Viewing Go test coverage in the browser with one command
Aug 20, 2020: Go has some really nice tools for running tests and analyzing code. One of these functionalities is that you can generate coverage information when running tests, that can later be viewed in a browser using the go tool cover command. It turns out though, since doing it requires e ...
Creating a static copy of a Drupal, Wordpress or other CMS website
Aug 20, 2020: I recently needed to sunset my old Drupal 6 blog at saml.rilspace.com, as Drupal 6 is since long not getting security updates, and might not even keep working with newer versions of PHP and MySQL. I was thus delighted to find that one can create a full static copy of such a site ...
Basic PUB/SUB connection with ZeroMQ in Python
Nov 13, 2019: ZeroMQ is a great way to quickly and simply send messages between multiple programs running on the same or different computers. It is very simple and robust since it doesn't need any central server. Instead it talks directly between the programs through sockets, TCP-connections o ...
Table-driven tests in C#
Nov 2, 2019: Folks in the Go community have championed so called table-driven tests (see e.g. this post by Dave Cheney and the Go wiki) as a way to quickly and easily writing up a bunch of complete test cases with inputs and corresponding expected outputs, and looping over them to execute the ...
SciPipe paper published in GigaScience
Apr 27, 2019: We just wanted to share that the paper on our Go-based workflow library, SciPipe, was just published in GigaScience:
Abstract
Background
The complex nature of biological data has driven the development of specialized software tools. Scientific workflow management systems s ...
Structured Go-routines or framework-less Flow-Based Programming in Go
Mar 2, 2019: I was so happy the other day to find someone else who found the great benefits of a little pattern for how to structure pipeline-heavy programs in Go, which I described in a few posts before. I have been surprised to not find more people using this kind of pattern, which has been ...
Make your commandline tool workflow friendly
May 25, 2018: Update (May 2019): A paper incorporating the below considerations is published:
Björn A Grüning, Samuel Lampa, Marc Vaudel, Daniel Blankenberg,
"Software engineering for scientific big data analysis"
GigaScience, Volume 8, Issue 5, May 2019, giz054,
https://doi.org/10.1093/giga ...
Parsing DrugBank XML (or any large XML file) in streaming mode in Go
Mar 15, 2018: I had a problem in which I thought I needed to parse the full DrugBank dataset, which comes as a (670MB) XML file (For open access papers describing DrugBank, see: [1], [2], [3] and [4]).
It turned out what I needed was available as CSV files under "Structure External Links". ...
Equation-centric dataflow programming in Go
Dec 27, 2017: Mathematical notation and dataflow programming
Even though computations done on computers are very often based on some type of math, it is striking that the notation used in math to express equations and relations is not always very readily converted into programming code. Out ...
What is a scientific (batch) workflow?
Dec 7, 2017:
Workflows and DAGs - Confusion about the concepts
Jörgen Brandt tweeted a comment that got me thinking again on something I've pondered a lot lately:
"A workflow is a DAG." is really a weak definition. That's like saying "A love letter is a sequence of characters." r ...
Go is growing in bioinformatics workflow tools
Nov 10, 2017:
TL;DR: We wrote a post on gopherdata.io, about the growing ecosystem of Go-based workflow tools in bioinformatics. Go read it here
It is interesting to note how Google's Go programming language seems to increase in popularity in bioinformatics.
Just to give a sample of ...
On Provenance Reports in Scientific Workflows
Oct 19, 2017: Hoping to write up a series of posts that go through some of the design decisions made in the SciPipe, the Go-based scientific workflow library we're working on, to share some of the thinking behind them, and get the opportunity to get feedback and suggestions that might be imple ...
(Almost) ranging over multiple Go channels simultaneously
Oct 5, 2017: In my experiments (flowbase, scipipe) in using Flow-based programming (FBP) principles in pure Go, there is a common pattern occuring all the time: The need to synchronously read a set of values from multiple channels at the same time. That is, if I have three in-bound channels, ...
First production run with SciPipe - A Go-based scientific workflow tool
Sep 28, 2017: Today marked the day when we ran the very first production workflow with SciPipe, the Go-based scientific workflow tool we've been working on over the last couple of years. Yay! :)
This is how it looked (no fancy GUI or such yet, sorry):
The first result we got in this ...
Notes on launching kubernetes jobs from the Go API
Feb 15, 2017: This post is also published on medium
My current work at pharmb.io entails adding kubernetes support to my light-weight Go-based scientific workflow engine, scipipe (kubernetes, or k8s for short, is Google’s open source project for orchestrating container based compute clust ...
SMWCon Fall 2016 - My talk on large RDF imports
Oct 7, 2016: I was invited to give a talk at Semantic MediaWiki (SMW) conference (SMWCon) in Frankfurt last week, on our work on enabling import of RDF datasets into SMW. I have presented at SMWCon before as well (2011: blog, slides, video, 2013: slides), so it was nice to re-connect with som ...
Tutorial: Luigi for Scientific Workflows
Jun 21, 2016: This is a Luigi tutorial I held at the e-Infrastructures for Massively parallel sequencing workshop (Video archive) at SciLifeLab Uppsala in January 2015, moved here for future reference.
What is Luigi?
Luigi is a batch workflow system written in Python and developed by ...
The unexpected convenience of JSON on the commandline
Dec 8, 2015: I was working with a migration from drupal to processwire CMS:es, where I wanted to be able to pipe data, including the body field with HTML formatting and all, through multiple processing steps in a flexible manner. I'd start with an extraction SQL query, through a few component ...
Wanted: Dynamic workflow scheduling
Oct 26, 2015:
Photo credits: Matthew Smith / Unsplash
In our work on automating machine learning computations in cheminformatics with scientific workflow tools, we have came to realize something; Dynamic scheduling in scientific workflow tools is very important and sometimes badly needed.
...
A few thoughts on organizing computational (biology) projects
Jun 23, 2015:
I read this excellent article with practical recommendations on how to organize a computational project, in terms of directory structure.
Directory structure matters
The importance of a good directory structure seems to often be overlooked in teaching about computationa ...
A cheatsheet for the iRODS rule language
Jun 11, 2015: iRODS, the "integrated rule oriented data system" is a super cool system for managing datasets consisting of files, from smallish ones, to really large ones counted in petabytes, and possibly spanning multiple continents.
There's a lot to be said about iRODS (up for another blog ...
Patterns for composable concurrent pipelines in Go
Jun 1, 2015: I realize I didn't have a link to my blog on Gopher Academy, on patterns for compoasable concurrent pipelines in Go(lang), so here it goes:
blog.gopheracademy.com/composable-pipelines-pattern
...
The role of simplicity in testing and automation
Mar 23, 2015: Disclaimer: Don't take this too seriously ... this is "thinking-in-progress" :)
It just struck me the other minute, how simplicity is the key theme behind two very important areas in software development, that I've been dabbling with quite a bit recently: Testing, and automation ...
The problem with make for scientific workflows
Mar 14, 2015:
The workflow problem solved once and for all in 1979?
As soon as the topic of scientific workflows is brought up, there are always a few make fans fervently insisting that the problem of workflows is solved once and for all with GNU make, written first in the 70's :)
Pe ...
Dynamic Navigation for Higher Performance
Mar 11, 2015: Improving performance in Delphi Bold MDA applications by replacing navigation code with derived links in the model
This post on Model Driven Architecture in Delphi and Bold, by Rolf Lampa, has been previously published on howtodothings.com.
Modeling class structures takes some ...
Profiling and creating call graphs for Go programs
Aug 8, 2013: In trying to get my head around the code of the very interesting GoFlow library, (for flow-based programming in Go), and the accompanying flow-based bioinformatics library I started hacking on, I needed to get some kind of visualization (like a call graph) ... something like this ...
(E)BNF parser for parts of the Galaxy ToolConfig syntax with ANTLR
Jul 28, 2011: As blogged earlier, I'm currently into parsing the syntax of some definitions for the parameters and stuff of command line tools. As said in the linked blog post, I was pondering whether to use the Galaxy Toolconfig format or the DocBook CmdSynopsis format. It turned out though W ...
Answering questions without answers - by wrapping simulations in semantics
Feb 17, 2010: (This post is migrated from this post on Samuels old tech blog)
Explicit knowledge is too expensive
There are lots of things that can't be answered by a computer from data alone. Maybe the majority of what we humans perceive as knowledge is inferred from a combination of data ( ...
A "Hello World" program in SWI-Prolog
Sep 22, 2009: This is how to create a "Hello World" kind of program in SWI-Prolog, as installed on Ubuntu (Jaunty).
Prolog usage can be divided into two phases, one in which you write the Prolog program, and the other in which you query the program. The first phase is easiest done in a separa ...