Components and configuration

Overview

A Murfey deployment is comprised of multiple components, illustrated in the diagram below. These consist of:

• A RabbitMQ message broker instance for passing messages between processing services
• A PostgreSQL database instance for keeping track of data collection and processing information
• The Murfey server which handles data collection registration and processing requests
• A Murfey client which watches for data and notifies the Murfey server

In addition there are optional components:

• ISPyB laboratory information management system for integration with user visit information and for processing results recording for display with the PATo web frontend
• A Graylog instance for log recording

Configuration and credentials files

Murfey will require a variety of configuration and credentials files to control the processes it will execute and secure the connections between its components. In this section, we show example configuration files for a Murfey setup where SPA TEM data is transferred from a Windows PC to a Linux server and storage system for subsequent processing.

Client configuration

An example of the .murfey file:

[Murfey]
instrument_name = krios
server = 192.168.0.0:8000
token = antidisestablishmentarianism

Machine configuration

An example of the murfey-machine-config.yaml file that has been set up for a TEM:

# General microscope information
display_name: Krios
instrument_name: krios
machine_override: ""
image_path: /home/User/Pictures/my_microscope.png

# Acquisition hardware and software (TEM only)
camera: K3
superres: true
flip_gain: ""
calibrations:
  magnification:
    10000: 8.000
    20000: 4.000
    40000: 2.000
    80000: 1.000
    160000: 0.500
acquisition_software:
  - epu
software_versions:
  - epu: 1.00
data_required_substrings:
  epu:
    .mrc:
      - fractions
    .tiff:
      - fractions

# Client-side directories
data_directories:
  - "D:/DoseFractions"
create_directories:
  atlas: atlas
analyse_created_directories:
  - atlas
gain_reference_directory: "C:/ProgramData/Gatan/Reference Images"

# Data transfer settings
data_transfer_enabled: true
rsync_basepath: /home/User/my-data
rsync_module: data
rsync_url: "http://192.168.0.2"
allow_removal: true

# Data processing settings
processing_enabled: true
processed_directory_name: processed
processed_extra_directory: ""
recipes:
  em-spa-bfactor: em-spa-bfactor
  em-spa-class2d: em-spa-class2d
  em-spa-class3d: em-spa-class3d
  em-spa-preprocess: em-spa-preprocess
  em-spa-refine: em-spa-refine
  em-tomo-align: em-tomo-align
  em-tomo-preprocess: em-tomo-preprocess
default_model: my_particle_picking_ml_model.h5
model_search_directory: processing
initial_model_search_directory: processing/initial_model
external_executables: {}
external_environment: {}

# Network and security settings
security_configuration_path: /home/User/murfey-security-config.yaml
murfey_url: "http://192.168.0.0:8000"
frontend_url: "http://192.168.0.0:3000"
instrument_server_url: "http://192.168.0.1:8000"
node_creator_queue: "node_creator"

Security configuration

An example of the murfey-security-config.yaml file:

murfey_db_credentials: /home/User/murfey-db-credentials.yaml
crypto_key: supercalifragilisticexpialidocious
sqlalchemy_pooling: false

auth_key: hippopotomonstrosesquippedaliophobia
auth_algorithm: HS256
auth_type: "password"
cookie_key: "cookie_auth"
session_validation: ""
session_token_timeout: 30000

rabbitmq_credentials: /home/User/rabbitmq-credentials.yaml
feedback_queue: murfey_feedback

graylog_host: 192.168.0.5
graylog_port: 9000

ispyb_credentials: ""

PostgreSQL database credentials

An example of the murfey-db-credentials.yaml file:

username: murfey
password: llanfair­pwllgwyngyll­gogery­chwyrn­drobwll­llan­tysilio­gogo­goch
host: 192.168.0.4
port: 5432
database: murfey_db

RabbitMQ credentials

An example of the rabbitmq-credentials.yaml file:

[rabbit]
plugin: pika
host: 192.168.0.6
port: 5672
username: murfey
password: pneumonoultramicroscopicsilicovolcanoconiosis
vhost: zocalo