New opensafely command: opensafely exec

There is new opensafely exec command in recent versions of the opensafely tool. It is designed to aid in development of analysis code using the published OpenSAFELY docker images.

The need to add your your work-in-progress code to project.yaml in order to test it is awkward. This leads some users to use their own locally installed R/python/stata tooling to develop their code. However, this means they may end up inadventantly using a library (or version of library) which is not available in the relevant OpenSAFELY image, and then the code fails when running with project.yaml.

We previously added the opensafely jupyter command to help improve this workflow when working with Jupyter notebooks, but with opensafely exec, we’ve extended this support to other tools.

What does it do?

Running opensafely exec IMAGE COMMAND does the following:

• runs the relevant docker IMAGE (r, python, stata-mp, cohortextractor)
• shares the files in your current directory
• executes COMMAND (or the default command for the image if you don’t supply one)

This might sound fairly simple, but it opens up some convenient new workflows.

Examples

To run an interactive R session in your current directory, just do:

opensafely exec r R

This will start the opensafely R image, run R, and make the files in your current directory available to you. So you can load, test, and develop against your current code and data locally, and save your changes. It’ll even save your R workspace if you want it to.

For python, you might like to run an interactive ipython session:

opensafely exec python ipython

And similarly for stata:

opensafely exec stata-mp

You can also run cohortextractor directly, which can help testing study definitions:

opensafely exec cohortextractor --help

You can actually run any command you want by passing it as an argument.

For example, you can open an interactive bash shell in any of our images with, if you want to look around inside the image:

opensafely exec $IMAGE bash

For more information, you can run opensafely exec --help, or see the documentation

Further work

We are always looking for ways to make developing with OpenSAFELY tools easier. opensafely exec is a key feature to help acheive this. Using it as a base, some other improvements we may be able to explore in future:

• An option to confgure Rstudio use the opensafely R image for OpenSAFELY projects
• Possibly running the graphical version of Stata
• VSCode integration to run your code automatically with opensafely exec
• Tooling to make testing your analysis code much easier.

New version of the R image published

We have just published a new build of the R image. No action needed, but more information below!

What has not changed?

This is not a semantic change - the same versions of the same set of libraries are installed. However the way the image is built has been reworked, in order to address a number of shortcomings of the previous image, and provide a stonger base to build on in future.

What has changed?

We built the new image from a more up-to-date base. This means a number of core system libraries have been updated to newer versions. If you want to, you can see the list of system libraries that were updated. These are not R libraries, but rather some of the base C libraries used at a low level. Keeping them up to date is important for security and bug fixes.

In addition, we’ve moved from R 4.0.2 to 4.0.5 to pick up a slew of minor fixes.

What does this mean for me?

Hopefully, nothing! The next time you run opensafely pull, downloading the R image may take a bit longer then normal. But otherwise, all your studies should run as normal, locally and via jobs.opensafely.org.

We have tested the new image with variety of study code without issue, but if you do find something has changed, please contact tech support and let us know.

Who is to blame for this!?

Special thanks go to Tom Palmer from the University of Bristol for helping motivate and then exhaustively test the new build process.

Using compressed files is now the default recommendation in documentation and templates. On the backends, where datasets can be very large, using uncompressed files significantly slows execution and consumes more disk space.

The research-template has been updated to generate csv.gz files from cohortextractor by default, and the examples and Getting Started documentation have been updated to match.

In addition, recommendations for using compressed formats for further data files in python, R and Stata has been updated.

Because of the change in filename, if you have a workspace with a large amount of data in uncompressed CSV files, ask tech-support about moving to compressed CSVs, and we can help do this efficiently.

dm+d codes for Virtual Medicinal Products (VMPs) can change over time, with the result that codelists that contain VMP codes can become stale.

As a consequence, studies that use patients.with_these_medications() with an old codelist may not have captured all the medication events that the author intended.

Until now, the mitigation has been to review and update codelists manually. With this update, cohort-extractor keeps stale medication codelists up-to-date by expanding the original codelist, so that it includes all current and previous codes of any VMPs in the codelist. This happens automatically.

Note that a codelist containing a VMP and all associated AMPs will still become stale with respect to new AMPs for that VMP are added to dm+d; this case is not currently handled automatically.

We will be contacting all study authors to explain the potential impact of using stale codelists more widely.

Added instructions regarding rounding to our disclosure control documentation, including an example.

Study definitions can now be “parameterised” so they accept values passed in by the action. This allows multiple cohorts with different properties to be generated by a single study definition.

Added support for querying by first and last day of the (English) school year.

Users of the OpenSAFELY platform use the opensafely tool to run their code locally prior to running against the real data.

This change adds some options to control the concurrency and memory usage of running actions, which should help users manage their local resource usage.

Added ability to filter a patient’s hospital admissions to only those with at least one day in critical care.

We have rolled out a status page for the platform. Services on this page will be automatically updated if incidents occur.

We have clarified in the documentation that data on referrals is incomplete. Any codelists which include referral codes used with patients.with_these_clinical_events are unlikely to return complete data as it is largely held separately in a difficult-to-analyse format. We hope to offer an alternative data source for referrals in due course.

The name of the Oxford-AstraZeneca vaccine has changed to COVID-19 Vaccine Vaxzevria 0.5ml inj multidose vials (AstraZeneca). To continue to access data for these vaccines, all users will need to update study definitions appropriately.

Added support for querying the UK Renal Registry

Added support for querying COVID-19 Infection Survey data provided by the Office for National Statistics.

Added support for returning total_bed_days_in_period and total_critical_care_days_in_period from admitted_to_hospital.

Added support for querying the ISARIC (International Severe Acute Respiratory and Emerging Infection Consortium) dataset.

Added support for querying the COVID-19 Therapeutics dataset.

Added ability to query a patient’s place_of_death (TPP backend only).

• Extended mean_recorded_value to support querying for mean recorded value across a full date range, as well as the most recent day of measurement.
  • Docs: cohortextractor.patients.mean_recorded_value
• Added min_recorded_value to support querying for minimum recorded value.
  • Docs: cohortextractor.patients.min_recorded_value
• Added max_recorded_value to support querying for maximum recorded value.
  • Docs: cohortextractor.patients.max_recorded_value

Added support for querying by first and last day of the NHS financial (recording) year.

Show how to extract cohortextractor output in compressed gzip (csv.gz) format - recommended to be used in all studies to minimise file storage size. (However, CSVs can still be used where Stata is used for analysis).

This is now shown in all parts of documentation with cohortextractor steps.

Added support for users to provide their own dummy data.

Added a page in the docs explaining (briefly) how permissions work.

Add ability to query a patient’s health care worker status in vaccination records (TPP backend only)

The OpenSAFELY command-line tool can now start a JupyterLab server. This allows you to easily develop code within the OpenSAFELY Python environment.

Added support for case-control studies