We’ve made it easier to pass custom parameters to ehrQL, so that you can reuse your ehrQL code more effectively.

Parameters can be passed from a pipeline action:

version: '4.0'

actions:
  generate_dataset:
    run: ehrql:v1 generate-dataset dataset_definition.py --output output/dataset.arrow
      --
      --start_date 2025-01-01
    ...

And then defined and used in an ehrQL python file:

from ehrql import get_parameter

start_date = get_parameter("start_date")

For more information on how to use custom parameters in your ehrQL, see the reference documentation and the How To guide

We know many users are waiting for news about OpenSAFELY’s expansion to allow research on non-COVID topics. Right now, we are working with NHS England to design a project application system that works swiftly and makes the best use of the platform. We still don’t know when the change will happen - we will share that news as soon as we get it. Thanks for your patience!

OpenCodelists provides an interactive builder for compiling codelists based on searches and user selections. The NHS Dictionary of Medicines and Devices (dm+d) coding system is now available in this builder.

Previous methods of compiling dm+d codelists on OpenCodelists (upload of csv, conversion from Psuedo-BNF codelists) continue to be supported.

The opensafely-matching python package has been upgraded and converted to an OpenSAFELY reusable action.

Case-control studies compare the population with the condition or event of interest with a control population, matched on specific categorical and/or scalar variables.

The matching reusable action can be used to take a dataset of “cases” and a second dataset of potential controls, both extracted using ehrQL dataset definition, and match cases to controls using defined variables.

More information about using case control studies in OpenSAFELY can be found in the OpenSAFELY documentation.

For detailed information on the configuration option and use of the OpenSAFELY matching reusable action, please see the reusable actions registry documentation.

OpenCodelists already supports conversion of medication codelists using the Pseudo-BNF coding system into codelists using the NHS Dictionary of Medicines and Devices (dm+d) system.

Previously, it was not possible to update these converted codelists using the one-click method used to originally create them. We have now altered the behaviour of this button to create a new version of the dm+d codelists if there are any updated codes available.

On OpenCodelists it is now possible to edit the metadata for a codelist during the draft phase.

An important part of the metadata for a codelist is explaining why certain searches were made, and why codes were included or excluded. Previously you could only add or edit metadata when a codelist was ready for review. However, by this point users may have forgotten the decisions they made during the creation process. Now there is a metadata tab to allow key things to be recorded during the creation of a codelist:

This change came about from direct user feedback. If you see anything with any OpenSAFELY product that you think could be improved, please let us know!

ehrQL has always supported having multiple measures defined in a single measure definition file. However the results of these measures were always written to a single, combined output file.

That’s still the default, but you now have the option of having each measure’s output written to a separate file. This might make some downstream processing actions easier. And it makes it possible to selectively submit some measure outputs for release without submitting all of them.

We have exposed two more columns on the outpatient appointments table from the HES data: outcome_of_attendance and referral_request_received_date. These are examples of data that are held in the backend database, but that had not been exposed via ehrQL. These columns were requested in the past week and are now available.

More details about these columns can be found in the HES data dictionary and the NHS data dictionary (see the spec for outcome of attendance).

When you push code to an OpenSAFELY project on Github it will automatically run your full project pipeline to check that it completes successfully. However, for larger projects this can take a long time to complete and sufficiently large projects will never complete due to Github’s six hour time limit.

To work around this problem we now support running just a specified list of project actions in the Github tests. You can do this by editing the .github/workflows/test_runner.yaml file and listing the required actions as below:

 - name: Test that the project is runnable
   uses: opensafely-core/research-action@v2
   with:
     actions: test_action_1 test_action_2 test_action_3

We are pleased to announce the availability of a new version of the R image, which we call r:v2. This includes R 4.4.3, and an updated suite of libraries. This has been in a beta testing period for a while, but is now generally available.

Upgrading

To upgrade, you should just be able to change the relevant run: commands for your actions in your project.yaml file to use r:v2 rather than r:latest. Our testing indicates that most existing R code should work with the v2 image unchanged, however, there may be some breaking changes, and you may need to fix your code to work with the newer version or R or some of the libraries

The old version of the R image is still available under r:v1, so you do not need to change to v2 if you do not want to (for example, if your project is nearly completed). However, we do recommend that ongoing projects switch to using the r:v2 action image, and the documentation and tooling has been changed to default to the v2 image for new projects.

You can read more details and information about the whys and hows behind the new image on our blog post OpenSAFELY, but new-R

As always, any further questions, or any problems with the v2 image, and you can reach out to tech-support.

ehrQL’s measures framework is used to calculate quotients (i.e. a numerator divided by a denominator) and to see how these vary over time and when broken down by different groupings.

We’ve recently improved the efficiency of measures calculations. All measures jobs are expected to see some improvement in execution time - our testing runs saw 10-80% speed-ups, depending on the type of measure. Measures which use simple variables and extract large populations are likely to see the biggest improvements.

We’ve recently spent some time improving Airlock, based on feedback from users after a few months of active use.

You can see the full list of changes we’ve made (over 80 pull requests!) here. Some highlights, in no particular order:

• Output checkers can now return a request early without having to do two independent reviews.
• Co-pilots now have access to see files in all the workspaces that they are helping with.
• We show users’ full names rather than their GitHub handles.
• Researchers can now move files between groups and change file types.
• We’ve added extra metadata to give more context about release requests.
• We’ve added several guide rails for researchers and output checkers to make it easier to know what you’re expected to do next.
• Researchers are now required to provide comments to explain how they have addressed feedback when resubmitting a request.
• All users can use markdown to add formatting to their comments.
• We’ve added row numbers and sticky headers to tables of data.
• We’ve made the infrastructure for releasing files more resilient to things like network failures.

We have added a new way of working with R code in OpenSAFELY, with the addition of the opensafely launch rstudio command.

This will start up an instance of RStudio using our official OpenSAFELY R image, and automatically open a web browser window on your computer pointing at the RStudio interface. You can edit, save, commit, and run your OpenSAFELY R code using a familiar environment, and be sure that you are using the same version of R, and all the libraries, that you will get when you run your code on our secure OpenSAFELY backends.

Previously, the only way to do this had been to use the RStudio provided with our Github Codespaces support, which is our recommended way to work with OpenSAFELY. However, this work extracts that experience, and builds on it, allowing it to be used both inside and outside of Codespaces. It will also enable us to support newer versions of R in the future, with updated libraries - watch this space!

In the process, we have improved the opensafely jupyter command, which, ran a JupyterLab instance in a similar fashion. This is now opensafely launch jupyter and has seen other improvements. We hope to add other interactive tools to opensafely launch in the future.

We have new documentation for using opensafely launch, and as always, you can inspect the help options for the command with:

opensafely launch --help

If you are an OpenSAFELY R user, we encourage you to try out the bundled RStudio support, and let us know if it works well for you or not. We are always keen to get feedback on the changes we make to the system.

And finally, we want to say a big thank you to Tom Palmer for leading a lot of this work, and to the University of Bristol for seconding him to work with the Bennett Institute. Tom’s extensive knowledge of R and its ecosystem has proven immensely valuable.

We have exposed two more columns on the admitted patient care table from the HES data: discharge_destination and discharge_method. These are examples of data that are held in the backend database, but that had not been exposed via ehrQL. We’re always happy to add existing data to ehrQL, but typically we wait for someone to ask so we know there’s a real need. These columns were requested last week and are now available.

More details about these columns can be found in the NHS spec for HES data. Here is the spec for discharge destination [archived here] and here is the spec for discharge method [archived here].

We have made the TPP DecisionSupportValues table available in ehrQL.

This table is used by TPP to store the precalculated results of any decision support algorithms. At the time of writing, the only algorithm available is v1 of the electronic frailty index (EFI), but this could be extended in the future.

Full documentation is here, but here is an example of how to get the latest EFI for each patient.

from ehrql import create_dataset
from ehrql.tables.tpp import decision_support_values

dataset = create_dataset()

latest_efi_record = (
  decision_support_values
    .electronic_frailty_index()
    .sort_by(decision_support_values.calculation_date)
    .last_for_patient()
)

dataset.latest_efi = latest_efi_record.numeric_value
dataset.latest_efi_date = latest_efi_record.calculation_date

We have improved the way in which codelists can be added to an OpenSAFELY project.

Previously, to add a codelist to a project, one had to manually add a codelist and version identifier to the codelists.txt file, then run the opensafely codelists update command to download it. This method is still supported.

We have now added the opensafely codelists add command, which allows you to directly add and download a codelist to your project in a single step using its URL.

For example:

opensafely codelists add https://www.opencodelists.org/codelist/opensafely/covid-identification/2020-06-03/

Many OpenSAFELY studies have made use of the NHS Digital Primary Care Domain Reference Sets [archived here].

These are republished on opencodelists.org under the NHSD Primary Care Domain Refsets organisation to make integration of these reference sets into your study easy. Based on user feedback, we have updated the published versions of these codelists to the 20241205 release.

Anyone wanting to use the updated reference sets should update their codelists.txt file to point at the latest versions.

We have made significant improvements to the autocomplete behaviour when writing ehrQL in VSCode, both locally and in online codespace environments. Previously autocomplete only worked on the first level for each table, so e.g. VSCode would know that the patients table had a column called date_of_birth, but it didn’t know that it was a column of dates. Now it does, so you can see all the date methods like is_after(date) and is_before(date), and the properties like day, month and year:

A larger example is:

Every method and property in the above has autocomplete e.g. where, clinical_events.date, is_on_or_after, numeric_value, and maximum_for_patient(). Also, VSCode knows that the resulting variable highest_recent_hba1c is a numeric column and so provides relevant autocomplete such as as_int() in case you wanted to convert a patient’s recent hba1c to an integer.

We have released a new extension for VS Code and Codespaces to help users test their ehrQL queries against tables of dummy data.

You can read more in the documentation, or try it out by following the updated ehrQL tutorial.

We have recently published a significant update to the ehrQL tutorial. The tutorial now walks you through using ehrQL to identify patients who should be on the QOF register for diabetes, and to categorise patients on the register according to various QOF business rules.

You can work through the tutorial entirely in your browser. After completing the tutorial, you can work through a quiz to test your understanding of ehrQL.

We have added better handling for the all_diagnoses and all_procedures fields in the apcs table. These fields contain every diagnosis (ICD10 codes), and each procedure undertaken (OPCS4 codes), during a hospital spell. The field is a list of clinical codes, joined together with ,s and ||s (see the above links for more details on why). Previously ehrql considered these fields to be strings, and so people would need to take a codelist, split it up, convert each code to a string, and then search the field for each one in turn. Now you can simply do:

all_diagnoses.contains_any_of(icd10_codelist)

all_diagnoses.contains(icd10_code) also works if you just want to check for the presence of a single code. Also, both methods are happy to be passed ICD10Codes, or ICD10 string prefixes eg. “N17” - or a combination of the two.

The same is true for all_procedures, except replace ICD10, with OPCS4 in the above explanation.

Dear OpenSAFELY users,

If you have submitted work for publication approval, we appreciate your patience with our delay in providing a response. We look forward to sending you a response as soon as this is possible.

A rapid review of the publication approval process was recently prompted by a number of factors:

• OpenSAFELY user feedback on the previous approach;
• the need to ensure we continue to meet the requirements of the GP data controllers;
• the need to ensure resources are appropriately allocated to the publication approval process;
• and to review how this publication approval process will work alongside ongoing plans to extend OpenSAFELY’s legal basis to studies beyond COVID-19 (which has strong support of NHS England, the GP profession and many other stakeholders).

As a result of this review, the overall publication approval service will have a new function established - the Clinical DATAPAST.

It is important that you read the information on the Clinical DATAPAST which describes this new function, how it will work, and your responsibilities as researchers when submitting work for publication approval.

Here is the final paper on this topic which was presented to the OpenSAFELY Steering Board and the Joint GP IT Committee in December 2024; the preferred option in section 6 was supported by both groups.

Dear OpenSAFELY Users,

We previously communicated that we had to temporarily remove all deregistered patients from the population available in OpenSAFELY-TPP. This was to ensure that we respected all Type 1 Opt-outs, and allowed us to reopen earlier, but introduced substantial difficulties with the representativeness and usefulness of the data.

We are pleased to let you know that the OpenSAFELY Steering Group (Chaired by NHS England, with membership including the BMA, the RCGP, and medConfidential, and patient representation through the OpenSAFELY Digital Critical Friends) support the deregistered patients being added back into OpenSAFELY-TPP.

The Board concluded that the Type 1 Opt-out be upheld for all currently registered patients in TPP.

We will implement a phased approach to reach this requirement, which will initially result in de-registered patient data (from patients who have moved to a non-TPP practice) also being removed from OpenSAFELY if the patient had a Type 1 Opt-out code in their TPP GP record.

The available TPP primary care population is now:

1. All patients currently registered at a TPP practice
2. All patients registered at any time from 2009-01-01 onwards, but having since de-registered
3. All patients registered at any time from 2009-01-01 onwards, but having since died.
4. EXCEPT all patients who have a type 1 opt-out code in their TPP GP record¹

As you know, OpenSAFELY is now required to respect type 1 opt-outs. This is easily solvable for most people, but people who have deregistered from a TPP practice may have subsequently requested a type 1 opt-out, and this would not be recorded in their TPP data,

In order to restart the OpenSAFELY service without further delay, we have put in place an interim measure to ensure that these type 1 opt-outs are respected. This measure will mean that for the moment, anyone who deregistered from a TPP practice while alive will be unavailable to OpenSAFELY users.

We are acutely aware that this interim measure will cause a range of serious epidemiological issues. This includes making the data less representative of the true population, and artificially inflating the mortality rate the further back a study starts. These issues are explained in detail in this document:
Interim measure to ensure we exclude type 1 opt-outs.

We are actively working on a long-term solution to this issue, which would resolve the epidemiological issues, and will update you on this as soon as possible.

Dear OpenSAFELY Users,

There have been a number of significant changes to the NHS England (NHSE) OpenSAFELY service and governance recently. Unfortunately some of these changes mean that the service is effectively - temporarily - closed at this time. While we are confident that the service will re-open in the near future, we wanted to write to explain, and to give a wider update, including on all the positive things coming shortly to the platform.

Type 1 Opt-Outs

As you know, we completed a significant piece of work earlier this year, to ensure that OpenSAFELY would honour the wishes of everyone who has registered a Type 1 Opt-Out (T1OO).

As a number of projects were underway in the service at that time, and were using that data as permitted by the emergency legislation introduced during the pandemic, these “Transitional” projects were given permission to conclude their work. The majority of these projects completed their work as scheduled, by June, but a small number requested additional time to analyse data, primarily to respond to peer-review comments – NHSE considered all such requested, and have granted some projects permission to continue until the end of the year, but only on the condition that this work excludes data for people registering a T1OO. All project leads who requested an extension will receive details of this by the end of this week.

After implementing the change to honour T1OOs for all future uses of OpenSAFELY, we identified a small potential gap: it is not possible to identify whether patients who have moved between EHR system suppliers have subsequently registered a T1OO (e.g. a patient moves from a TPP surgery to an EMIS surgery and then registers an opt-out in the EMIS surgery. In this scenario, TPP won’t know which of the patients who have now moved to EMIS, have registered a T1OO: the systems do not currently share this information). We are working rapidly on proposing a long-term solution to this, and intend to formally present that solution to the NHSE OpenSAFELY Steering Group and the Joint GP IT Committee in September.

To be cautious and ensure we fully honour all opt-outs, we are introducing an interim solution - we will remove all patients who have moved from TPP to EMIS (or vice-versa) from the accessible data for users. No analyses can be run until this change has been made, which we anticipate being made and tested before the end of this month.

We are happy to discuss this with users, and any possible impact on analyses: however, a benefit of OpenSAFELY is that users can develop and run their code on the current data, then easily re-run it later when this issue is addressed to establish whether their results were affected.

Formal agreements with the GP System Suppliers

NHSE are currently working with the GP System Suppliers (TPP and EMIS) on the formal agreements necessary to govern their continued support for the service. The service requires a commercial agreement between NHSE and each supplier, to cover their costs to support OpenSAFELY, and a Data Processing Agreement (DPA) to document how data should be managed. While this is ongoing the TPP half of the service will remain open, but the EMIS half of the service will remain closed.

NHSE are in the final stages of implementing a new DPA with TPP, which should be in place imminently – at that point, the TPP half of the service will be able to restart, and NHSE will focus on finalising a DPA with EMIS. Until they conclude this work with TPP, unfortunately, no new project analyses can run in OpenSAFELY-TPP.

Analytical Methods Policy

You may have seen that we have recently introduced a new policy, which sets out the analytical methods that can be used in OpenSAFELY. This policy aims to manage two limitations imposed upon us by external factors: we can’t pay for additional compute to support certain machine learning methods; and some machine learning analytic methods present privacy challenges around secure output checking that have not yet been resolved by any teams anywhere, and therefore cannot yet be implemented in a dataset with total population coverage. We anticipate that this policy will be refreshed and updated over the coming months: we would welcome feedback on it, and suggested changes that could simplify work for researchers while maintaining patients’ privacy, and addressing the additional challenges, for example, around interpretability, explainability and safety of these methods. Maintaining the trust of the public and the profession is a key priority for teams running the OpenSAFFELY COVID-19 service .

We have recently written to all study leads with a current project, asking them to briefly confirm that their project will adhere to this policy, and require a response before those projects can restart. We will update the application form and Data Access Agreement to make this requirement clear for future work.

Non-Covid Uses of OpenSAFELY

Much more positively, we are continuing to progress work on access to GP data through OpenSAFELY for analyses beyond COVID-19, as announced by NHSE in November (link to NHSE announcement [archived here]); progress is extremely good, and our primary focus, other than resolving the current barriers to use, is on ensuring the service is ready to support this significant expansion. The GP profession (the BMA and RCGP) also continues to provide strong support for the use of OpenSAFELY beyond COVID-19 purposes. We expect to have news in the coming months.

Recent Outputs

Despite the changes introduced this year, and the current restrictions on access to the service, the NHS England OpenSAFELY Service continues to be highly productive, thanks principally to your outstanding work. Some recently published papers are listed below, which help demonstrate the enduring value of the service, and the potential of what we can collectively achieve once current issues are resolved and new projects addressing important questions beyond COVID-19 can be approved:

Curation of GP ethnicity data
https://www.bennett.ox.ac.uk/papers/10.1186/s12916-024-03499-5/

Fit notes activity for COVID
https://www.bennett.ox.ac.uk/papers/10.1136/bmjopen-2023-080600/

Vaccine Effectiveness
https://www.bennett.ox.ac.uk/papers/10.1097/ede.000000000000174/

Real time reporting for Strep C outbreak
https://www.bennett.ox.ac.uk/papers/10.1136/bmjmed-2023-000791/

Trends and variation opioids during COVID
https://www.bennett.ox.ac.uk/papers/10.1016/s2468-26672400100-2/

Weight gain during COVID
https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1004398

OpenPrompt - incorporating smartphone application data within OpenSAFELY to collect quality of life information
https://www.thelancet.com/journals/lanepe/article/PIIS2666-7762(24)00074-7/fulltext

Risk of emergency hospital admission after antibiotic treatment
https://pubmed.ncbi.nlm.nih.gov/38956603/

Antibiotic Exposure Characteristics for COVID-19 Hospital Admission
https://pubmed.ncbi.nlm.nih.gov/38927232/

OpenSAFELY Symposium

Following our inaugural conference last year, we will be holding an OpenSAFELY symposium on 25-26 November.

The symposium will be two days of talks and workshops on OpenSAFELY, data infrastructure, and open science, and we hope as many of you as possible will be able to join us.

Please register for the conference on November 25th and 26th using our EventBrite link. If you would like to speak or run a workshop we would be delighted and are currently accepting suggestions using our online form.

When can you start work again?

Once we have implemented the interim solution for patients who have moved between EMIS and TPP (see T1OO heading above), and NHSE have finalised the DPA with TPP, the OpenSAFELY service will re-open in TPP for all existing projects (including those approved this year but not yet started, and those Transitional projects given an extension until the end of December). Researchers on those projects will be able to start/resume work as soon as they have confirmed adherence to the Analytical Methods Policy. We are confident that we will re-open in the very near future.

The OpenSAFELY-EMIS service will take longer to restart, as this requires completion of the commercial agreement between NHSE and EMIS.

In the meantime, NHSE are now happy to receive and review new applications to use OpenSAFELY (for Covid-19 related research, in TPP, initially), so please do submit your proposals using the online application or contact us to discuss further.

All research projects can now access OpenSAFELY development environments hosted in GitHub Codespaces. For more information, see our blog post on “Research-ready computers in the cloud” and our how to guide on “How to add GitHub Codespaces to your project”.

What has changed?

Once logged into OpenSAFELY Jobs, the 5 workspaces shown to you on your home page are now the five most recently active (by creation/modification date or most recently run job) that you have access to. This is a change from previously showing you the five most recently created.

What does this mean for me?

For most users with a small number of workspaces, this change in ordering will likely have little effect. For those with access to many workspaces, this will hopefully make it easier to get access to the most relevant workspaces as quickly as possible.

OpenSAFELY Jobs has always shown the Git commit hash for a given job request or job, with a link to view the workspace’s code on GitHub as of that revision. We now also provide GitHub code comparison links (example) so you can see exactly what code has changed between this revision and previous versions.

Job Requests

The code comparison links can be found on a job request’s page in the Code comparison section beneath the job information. If this is the first job request for this workspace on this backend, no code comparison is possible and so no link is available. If there are previous job requests, a link to compare the workspace’s code for this job request to the previous, and previous successful runs (if present) will be present.

Jobs

The code comparison links can be found on a job’s page at the bottom of the Job information section. If this is the first run of this action in this workspace on this backend, no code comparison is possible. If it has been run before, links to compare the workspace’s code for this job to the previous, and previous successful runs (if present) will be present.

N.B. If there lots of changes between versions of a workspace’s codebase, the code comparisons for a job may also include changes that are not directly related to the job in question. Identification of the job by its action name in the project.yaml file and examination of changes to related files listed in its configuration should reveal what the relevant changes are.

Users can now use Markdown formatting to update the Status Description field in the Project page.

Screenshots

Example URLs

• Antipsychotics Prescribing During Covid-19
• Service Restoration Observatory
• Online Consultations

We have released v1 of ehrQL.

It contains a small number of breaking changes from v0.

For existing users of ehrQL, please refer to the release notes for guidance about updating from v0.

Thank you to all users who have tried out ehrQL and given us feedback!

ehrQL’s measures framework is used to calculate quotients (i.e. a numerator divided by a denominator) and to see how these vary over time and when broken down by different groupings.

Previously, numerators and denominators were not subject to disclosure control: a user had to apply disclosure control by writing their own downstream action. Now, ehrQL’s measures framework has disclosure control enabled by default. First, values less than or equal to seven are replaced with zero (suppressed); then, values are rounded to the nearest five.

When the built-in disclosure control method is unhelpful, such as when an alternative method is required or when unsuppressed/unrounded values are needed by a downstream action, the configure_disclosure_control method allows a user to disable disclosure control. For example:

measures = create_measures()

measures.configure_disclosure_control(enabled=False)

measures.define_measure(...)

For more information about disclosure control in OpenSAFELY, please see the “Updated disclosure control guidance” page.

As always, if you need help with ehrQL, then please ask for help on the #ehrql-support Slack channel. (If you’re unsure how to join, then please ask your co-pilot.)

The opensafely tool has been updated to warn you if you have invalid outputs marked as moderately_sensitive in your project.yaml. This is a follow on from the previous change about stricter output file paths and means that you’ll get more accurate feedback when running the code locally.

Specifically, it will check that moderately_sensitive outputs meet the appropriate output file restrictions.

These currently are:

• The file must be of the correct type. You will not be able to run jobs locally at all or on the server if the file is not a valid type, i.e. it must have a valid file extension.

• If it is a .csv file, it must not have a patient_id column. Your code will still run, but the log file and the on-screen summary text will show a warning. If you run it on your own computer using the opensafely command line tool, you will still get an output. If you run it in the live system via jobs.opensafely.org, then it will still run, but the file will not be available in level 4

• If it is too large, it will be handled in the same way as above. This is unlikey to occur when running locally against dummy data, but may happen when run via jobs.opensafely.org.

Fixing these is likely a case of marking the file as highly_sensitive instead. If you do need it to be moderately_sensitive, then you may need to process the data a bit more, e.g. remove the patient_id column or reduce the size.

As a reminder, the policy for moderately_sensitive senstitive outputs is that they must be aggregate data, not patient level. These checks are designed to catch accidental misclassification of outputs with patient level data as moderately_sensitive.

Any questions or problems, please let us know.

We’ve made some changes to slightly restrict output file paths.

Specifically, all paths must now end with file extension, e.g. .csv. Previously, it was possibly to use a trailing * character to match all files (e.g. outputs/data.*). You can still use * to match groups of similar files, but you must end the pattern with a file extension (e.g. outputs/data*.csv).

Nearly all project.yaml files do already have explicit file extensions in all their output paths, and this will not require any changes for most users. There are a few users who may need to add an explicit file extensions in order for jobs to run, both locally and on backends. The opensafely tool that you run locally has been updated, and it will warn you that you need to do this.

There are multiple motivations for this change, but the main one is that we want to be able to inspect the file types when loading a project.yaml. This allows us to warn users if they are trying to mark an unsupported file type as moderately_sensitive.

It also stops accidentally capturing output files that were not intended to be outputs (which incurrs a performance and disk space cost).

Any questions or problems, please let us know.

In order to add a new package to the R Docker image we had to upgrade some of the existing package versions. Hopefully these upgrades will be welcome but there’s a small possibility they might cause issues with existing code so please be on the look out for this.

The most significant upgrade is of the readr package from 1.3.1 to 2.1.4. You can find the changelog here: https://readr.tidyverse.org/news/index.html

You can see the full list of upgraded packages on Github here: https://github.com/opensafely-core/r-docker/commit/30fc019

Previously, when submitting jobs to run the platform would reject requests if any of the requested action’s dependencies had previously failed.

This required users to explicitly request those failed dependencies to be re-run. This was cumbersome, and was not the same behaviour as when running actions locally on user’s machines with the opensafely run command.

We have now changed this behaviour, so that if you submit an action to run, any dependent actions that have previously failed will also be scheduled to run. This should make submitting jobs simpler in many cases.

For context, the original design for this behaviour was to encourage users to ensure their code runs succesfully locally before submitting to run for real. This avoids overloading the system with jobs that may fail anyway due to code errors. This is still a very important best practice what we strongly encourage users to do before submitting jobs.

However, this resulted in an awkward workflow for submitting batches of jobs, hence this change.

We have recently made some updates to our disclosure control guidance. This is important for both researchers requesting release of outputs and those who review them. We have made the following changes, which are summarised in more detail below:

• Requiring rounding of counts in addition to suppression of low counts
• Increasing the small number suppression threshold to 7
• Encouraging release of underlying data earlier in the analysis pipeline
• A new checklist for requesting a release
• Updated recommendations for release of log files
• More detail on allowed file types

OpenSAFELY Jobs has been updated with new Event Log pages for users, projects, and organisations.

These pages provide specific information about current Job Requests, and allow users to see a complete audit log of all previous Job Requests.

Examples of new Event Log pages include:

• User: Louis Fisher
• Organisation: The London School of Hygiene & Tropical Medicine
• Project: Investigating the effectiveness of the COVID-19 vaccination programme in the UK

The global OpenSAFELY Job Server Event Log page has also been updated to match the other Event Log pages, and contains a complete log of all Job Requests ever run by OpenSAFELY across all Backends.

As a user of the Job Server, you can access your own Event Log page from your dashboard.

We have added a command to the opensafely command line tool: opensafely clean

This command will safely remove any leftover OpenSAFELY docker artifacts from your system. Specifically, it removes old images, job containers, and file volumes that may have been created when running jobs locally on a user’s computer.

Users who have Docker runnng on WSL may find this particularly useful, as it looks like removing these leftover artifacts can help reduce memory usage.

The documentation has more information on managing your local docker resources.

OpenSAFELY Jobs is used by researchers from within the secure environments (such as TPP level 4) to support the review and output checking of files.

We’ve added an easier way for users to log in to OpenSAFELY Jobs from the TPP secure environment, by implementing auto-generated single use tokens. Previously, users were required to manually type their Github password and 2FA token, which was too easy to get wrong.

To use this new method, users first generate a single use token by going to their settings page and clicking on the “Generate Single Use Token” button. Then, they log in as normal to TPP level 4 over the VPN and open a browser window pointing at the OpenSAFELY Jobs log in page. They are now able to authenticate to OpenSAFELY Jobs using just their username or email and the single use token. Much quicker while still being highly secure!

OpenSAFELY Jobs home page has been updated to create a personalised dashboard for users.

Once logged in, users will now be able to view:

• The most recent job requests they have created
• Workspaces they have access to
• Most recently updated projects
• In progress and completed application forms

Alongside the new home page, users are now able to access a personalised list of:

• Your projects
• Your workspaces
• Your organisations

For visitors to the website who don’t have an account, a new home page is visible containing information on why they might want to use OpenSAFELY for their next research project.

New opensafely command: opensafely exec

There is new opensafely exec command in recent versions of the opensafely tool. It is designed to aid in development of analysis code using the published OpenSAFELY docker images.

The need to add your your work-in-progress code to project.yaml in order to test it is awkward. This leads some users to use their own locally installed R/python/stata tooling to develop their code. However, this means they may end up inadventantly using a library (or version of library) which is not available in the relevant OpenSAFELY image, and then the code fails when running with project.yaml.

We previously added the opensafely jupyter command to help improve this workflow when working with Jupyter notebooks, but with opensafely exec, we’ve extended this support to other tools.

What does it do?

Running opensafely exec IMAGE COMMAND does the following:

• runs the relevant docker IMAGE (r, python, stata-mp, cohortextractor)
• shares the files in your current directory
• executes COMMAND (or the default command for the image if you don’t supply one)

This might sound fairly simple, but it opens up some convenient new workflows.

Examples

To run an interactive R session in your current directory, just do:

opensafely exec r R

This will start the opensafely R image, run R, and make the files in your current directory available to you. So you can load, test, and develop against your current code and data locally, and save your changes. It’ll even save your R workspace if you want it to.

For python, you might like to run an interactive ipython session:

opensafely exec python ipython

And similarly for stata:

opensafely exec stata-mp

You can also run cohortextractor directly, which can help testing study definitions:

opensafely exec cohortextractor --help

You can actually run any command you want by passing it as an argument.

For example, you can open an interactive bash shell in any of our images with, if you want to look around inside the image:

opensafely exec $IMAGE bash

For more information, you can run opensafely exec --help, or see the documentation

Further work

We are always looking for ways to make developing with OpenSAFELY tools easier. opensafely exec is a key feature to help acheive this. Using it as a base, some other improvements we may be able to explore in future:

• An option to confgure Rstudio use the opensafely R image for OpenSAFELY projects
• Possibly running the graphical version of Stata
• VSCode integration to run your code automatically with opensafely exec
• Tooling to make testing your analysis code much easier.

New version of the R image published

We have just published a new build of the R image. No action needed, but more information below!

What has not changed?

This is not a semantic change - the same versions of the same set of libraries are installed. However the way the image is built has been reworked, in order to address a number of shortcomings of the previous image, and provide a stonger base to build on in future.

What has changed?

We built the new image from a more up-to-date base. This means a number of core system libraries have been updated to newer versions. If you want to, you can see the list of system libraries that were updated. These are not R libraries, but rather some of the base C libraries used at a low level. Keeping them up to date is important for security and bug fixes.

In addition, we’ve moved from R 4.0.2 to 4.0.5 to pick up a slew of minor fixes.

What does this mean for me?

Hopefully, nothing! The next time you run opensafely pull, downloading the R image may take a bit longer then normal. But otherwise, all your studies should run as normal, locally and via jobs.opensafely.org.

We have tested the new image with variety of study code without issue, but if you do find something has changed, please contact tech support and let us know.

Who is to blame for this!?

Special thanks go to Tom Palmer from the University of Bristol for helping motivate and then exhaustively test the new build process.

Using compressed files is now the default recommendation in documentation and templates. On the backends, where datasets can be very large, using uncompressed files significantly slows execution and consumes more disk space.

The research-template has been updated to generate csv.gz files from cohortextractor by default, and the examples and Getting Started documentation have been updated to match.

In addition, recommendations for using compressed formats for further data files in python, R and Stata has been updated.

Because of the change in filename, if you have a workspace with a large amount of data in uncompressed CSV files, ask tech-support about moving to compressed CSVs, and we can help do this efficiently.

dm+d codes for Virtual Medicinal Products (VMPs) can change over time, with the result that codelists that contain VMP codes can become stale.

As a consequence, studies that use patients.with_these_medications() with an old codelist may not have captured all the medication events that the author intended.

Until now, the mitigation has been to review and update codelists manually. With this update, cohort-extractor keeps stale medication codelists up-to-date by expanding the original codelist, so that it includes all current and previous codes of any VMPs in the codelist. This happens automatically.

Note that a codelist containing a VMP and all associated AMPs will still become stale with respect to new AMPs for that VMP are added to dm+d; this case is not currently handled automatically.

We will be contacting all study authors to explain the potential impact of using stale codelists more widely.

Added instructions regarding rounding to our disclosure control documentation, including an example.

Study definitions can now be “parameterised” so they accept values passed in by the action. This allows multiple cohorts with different properties to be generated by a single study definition.

Added support for querying by first and last day of the (English) school year.

Users of the OpenSAFELY platform use the opensafely tool to run their code locally prior to running against the real data.

This change adds some options to control the concurrency and memory usage of running actions, which should help users manage their local resource usage.

Added ability to filter a patient’s hospital admissions to only those with at least one day in critical care.

We have rolled out a status page for the platform. Services on this page will be automatically updated if incidents occur.

We have clarified in the documentation that data on referrals is incomplete. Any codelists which include referral codes used with patients.with_these_clinical_events are unlikely to return complete data as it is largely held separately in a difficult-to-analyse format. We hope to offer an alternative data source for referrals in due course.

The name of the Oxford-AstraZeneca vaccine has changed to COVID-19 Vaccine Vaxzevria 0.5ml inj multidose vials (AstraZeneca). To continue to access data for these vaccines, all users will need to update study definitions appropriately.

Added support for querying the UK Renal Registry

Added support for querying COVID-19 Infection Survey data provided by the Office for National Statistics.

Added support for returning total_bed_days_in_period and total_critical_care_days_in_period from admitted_to_hospital.

Added support for querying the ISARIC (International Severe Acute Respiratory and Emerging Infection Consortium) dataset.

Added support for querying the COVID-19 Therapeutics dataset.

Added ability to query a patient’s place_of_death (TPP backend only).

• Extended mean_recorded_value to support querying for mean recorded value across a full date range, as well as the most recent day of measurement.
  • Docs: cohortextractor.patients.mean_recorded_value
• Added min_recorded_value to support querying for minimum recorded value.
  • Docs: cohortextractor.patients.min_recorded_value
• Added max_recorded_value to support querying for maximum recorded value.
  • Docs: cohortextractor.patients.max_recorded_value

Added support for querying by first and last day of the NHS financial (recording) year.

Show how to extract cohortextractor output in compressed gzip (csv.gz) format - recommended to be used in all studies to minimise file storage size. (However, CSVs can still be used where Stata is used for analysis).

This is now shown in all parts of documentation with cohortextractor steps.

Added support for users to provide their own dummy data.

Added a page in the docs explaining (briefly) how permissions work.

Add ability to query a patient’s health care worker status in vaccination records (TPP backend only)

The OpenSAFELY command-line tool can now start a JupyterLab server. This allows you to easily develop code within the OpenSAFELY Python environment.

Added support for case-control studies