z - Advanced topic: Handling packages with remote dependencies

Introduction

Packages published on CRAN must have their dependencies on either CRAN or BioConductor, but not on GitHub. However, there are many packages available on GitHub that never get published on CRAN, and some of these packages may even depend on other packages that are also only available on GitHub. rix makes it possible to install packages from GitHub, but in case one of the package’s dependencies has also only been released on GitHub, building the Nix environment will fail. This is because Nix will be looking for these packages on nixpkgs, but only packages released on CRAN and Bioconductor are available through nixpkgs. This vignette explains how to install such a packages that have dependencies only available on GitHub.

The {lookup} package

As an example we are going to use the {lookup} package which has only been released on GitHub. Here is the repository. This package comes with the lookup() function which makes it possible to check the source code of any function from a loaded package, even if the source of that function is in C or Fortran. To create a reproducible development environment that makes {lookup} available to you, you could use the following rix::rix() call:

path_default_nix <- tempdir()

rix(
  r_ver = "latest",
  r_pkgs = NULL,
  system_pkgs = NULL,
  git_pkgs = list(
    package_name = "lookup",
    repo_url = "https://github.com/jimhester/lookup/",
    commit = "eba63db477dd2f20153b75e2949eb333a36cccfc"
  ),
  ide = "other",
  project_path = path_default_nix,
  overwrite = TRUE,
  print = TRUE
)

Trying to build this environment will fail with following error message:

error: attribute 'highlite' missing

Building remote dependencies

{highlite} is a dependency of {lookup} that is only available on GitHub. This can be checked by looking at the DESCRIPTION file of {lookup}:

Remotes:
    jimhester/highlite,
    gaborcsardi/gh,
    hadley/memoise

We see that there are actually three packages that come from GitHub: but gh and memoise have in the meantime been released on CRAN, which means that they are also available through nixpkgs. We have to deal with {highlite} however, because it never got released on CRAN. Doing so is fairly easy: first, create a new expression using rix to install {highlite}:

path_default_nix <- tempdir()

rix(
  r_ver = "latest",
  r_pkgs = NULL,
  system_pkgs = NULL,
  git_pkgs = list(
    package_name = "highlite",
    repo_url = "https://github.com/jimhester/highlite/",
    commit = "767b122ef47a60a01e1707e4093cf3635a99c86b"
  ),
  ide = "other",
  project_path = path_default_nix,
  overwrite = FALSE,
  print = TRUE
)

(you don’t need to overwrite the previous expression, simply printing this one on screen will do). Copy the following lines:

git_archive_pkgs = [(pkgs.rPackages.buildRPackage {
  name = "highlite";
  src = pkgs.fetchgit {
    url = "https://github.com/jimhester/highlite/";
    rev = "767b122ef47a60a01e1707e4093cf3635a99c86b";
    sha256 = "sha256-lkWMlAi75MYxiBUYnLwxLK9ApXkWanA4Mt7g4qtLpxM=";
  };
  propagatedBuildInputs = builtins.attrValues {
    inherit (pkgs.rPackages) Rcpp BH;
  };
}) ];

into the previous expression (and change git_archive_pkgs into highlite). The file should look like this now:

let
 pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/b200e0df08f80c32974a6108ce431d8a8a5e6547.tar.gz") {};

  highlite = [(pkgs.rPackages.buildRPackage {
    name = "highlite";
    src = pkgs.fetchgit {
      url = "https://github.com/jimhester/highlite/";
      rev = "767b122ef47a60a01e1707e4093cf3635a99c86b";
      sha256 = "sha256-lkWMlAi75MYxiBUYnLwxLK9ApXkWanA4Mt7g4qtLpxM=";
    };
    propagatedBuildInputs = builtins.attrValues {
      inherit (pkgs.rPackages) Rcpp BH;
    };
  }) ];
  git_archive_pkgs = [(pkgs.rPackages.buildRPackage {
    name = "lookup";
    src = pkgs.fetchgit {
      url = "https://github.com/jimhester/lookup/";
      rev = "eba63db477dd2f20153b75e2949eb333a36cccfc";
      sha256 = "sha256-arl7LVxL8xGUW3LhuDCSUjcfswX0rdofL/7v8Klw8FM=";
    };
    propagatedBuildInputs = builtins.attrValues {
      inherit (pkgs.rPackages) gh memoise Rcpp codetools crayon rex highlite jsonlite rstudioapi withr httr;
    };
  }) ];
 system_packages = builtins.attrValues {
  inherit (pkgs) R ;
};
  in
  pkgs.mkShell {
    buildInputs = [ git_archive_pkgs  system_packages  ];
      shellHook = ''
R --vanilla
'';
  }

The only thing that we need to change is this line:

propagatedBuildInputs = builtins.attrValues {
      inherit (pkgs.rPackages) gh memoise Rcpp codetools crayon rex highlite jsonlite rstudioapi withr httr;
    };

into:

propagatedBuildInputs = builtins.attrValues {
      inherit (pkgs.rPackages) gh memoise Rcpp codetools crayon rex jsonlite rstudioapi withr httr;
    } ++ [highlite];

and this line:

buildInputs = [ git_archive_pkgs  system_packages  ];

into:

buildInputs = [ git_archive_pkgs  system_packages highlite ];

Building the expression now succeeds.

We know that this is quite tedious, but at the moment there are no plans to make rix handle remote dependencies automatically. This is for mainly three reasons:

• packages with remote dependencies are rare, and never on CRAN on Bioconductor;
• packages may have remote dependencies, but these dependencies may later be available on CRAN (such as with memoise and gh for {lookup});
• rix cannot decide for the user which commit to use for a remote dependency (or whether to use the released version of that dependency from CRAN).

Because of these reasons, we believe that it is safer for users that really need to use such packages to manually edit their Nix expressions. Don’t hesitate to open an issue if you require assistance.