Data Structure

The Data Tracker is based on a few main components:

  • Order

  • Dataset

  • Project

  • User

  • Log

  • DOI

Terminology

  • Fields:

    • Fields in the documents for the datatype/collection

  • Computed fields:

    • Values that are either calculated or retrieved from documents in other collection(s)

    • Included when the entity is requested via API

Order

  • Added automatically when e.g. order in order portal changes to accepted

    • Import data from order portal

  • Can have any number of associated datasets

Fields

  • _id

    • Uuid for the order

  • Title

    • Name

  • Description

    • Description in markdown

  • Creator (facility name)

    • Creator can be set to e.g. external if a non-facility wants to add a dataset

  • Receiver

    • Email or uuid of the user who made the order

    • Input to add will be an email address, which is mapped to the user collection

    • Uuid saved: user exists

    • Email saved: user does not exist

  • Datasets

    • All datasets generated for the order

  • Extra fields

    • Custom fields in the style {'key': 'key_name', 'value': 'data value'}

Dataset

  • Data generated by e.g. facility

  • One per “data delivery” from facility

  • Can have identifier(s) (e.g. DOIs)

  • Data links can be added by receiver

  • Receiver and creator can edit the entry (inherited from order)

Fields

  • _id

    • Uuid of the dataset

  • Title

    • Name

  • Description

    • Description in markdown

  • Links

    • List of links to where the dataset can be found.

      • List entry:

      {
  'title': 'name',
  'url': 'https://place',
  'hashes': {
    'type': 'sha256',
    'files': [
      {
        'name': 'filename',
        'hash': 'FEDCBA9...'
      }
    ]
  }
}

      • title and url are mandatory for each link, hashes is optional

  • Extra

    • Custom fields in the style {'key': 'key_name', 'value': 'data value'}

Computed fields

  • Related

    • All other datasets from the same order

  • Projects

  • Identifiers

    • Local identifier

    • DOIs

  • Creator

    • Inherited from order

    • Name of e.g. facility that generated the dataset

Project

  • Created by users

  • Can have multiple owners

  • Can have identifiers

  • Intended as a way for a user to have a page to show off their data and be able to get an identifier (DOI)

Fields

  • _id

    • Uuid for the projects

  • Title

    • Name

  • Description

    • Description in markdown

  • Contact

    • Contact information (email) for the project

  • Datasets

    • Datasets connected to the dataset

    • Can be added by receiver/creator of dataset

    • Can be removed by any user listed in owners

  • Publications

    • List of publications related to the project

      • Entry:

      {
  'title': 'name',
  'doi': 'doi-id'
}

      • title+doi mandatory, but maybe include ability to add e.g. journal, year etc

  • DMP

    • Data management plan

    • URL

  • Owners

    • List of uuids/emails

    • Just like with order; email can be used if user not in db yet

      • Allow facilities to prepare project pages

  • Extra fields

    • Custom fields in the style {'key': 'key_name', 'value': 'data value'}

Computed fields:

  • Identifiers

    • Local identifier

    • DOIs

User

  • Everyone using the system is a user

  • Login via Elixir AAI

  • On first login, the user will be added to db

    • Use auth_id to recognize user

    • Read e.g. email from the login info

  • API can also be accessed using an API key

    • may be created by any user

  • “admin” can create user for facility

  • A user can “claim entries”

    • Will check all order receivers/project owners whether the users email is listed

      • Email will be replaced with user uuid

  • Facilities cannot log in via Elixir, but must do so via api_key

Fields

  • _id

    • Uuid for the order

  • Email

    • Email address of the user

  • Auth_id

    • Identifer received from Elixir

    • Is set to --facility-- for facilities to avoid Elixir login

  • Api_key

    • Key that can be used as an alternative to login for authentication

  • Name

    • Name of the user (can be e.g. name of facility for facility accounts)

  • Affiliation

    • University/company etc

  • Country

    • The country of the user

  • Permissions

    • A list of the extra permissions the user has (see Permissions)

Log

  • Whenever an entry (order, dataset, project, or user) is changed, a log should be written

  • All logs are in the same collection

  • A function is required to show changes between different versions of an entry

Fields

  • _id

    • uuid for the log

  • `Action’

    • Type of action (add, edit, or delete)

  • Data_type

    • The collection that was modified (order, dataset, project, or user)

  • Data

    • Add/edit: complete copy of document

    • Delete: empty

  • Timestamp

    • The time the action was performed

  • User

    • Uuid of the user performing the action

DOI

  • Two collections

    • doi_req - Requests for a DOI

    • doi - Accepted DOIs

  • Users can request a DOI for datasets and projects

  • Upon request, data is copied to doi_req

  • A reviewer will need to check the data for the request

    • Required fields

    • File hashes

  • If accepted, the data will be copied to doi

  • Each DOI document is a complete copy of the entire data structure that was accepted for the DOI

Fields (request)

  • _id

    • Uuid for the request

  • Data

    • A complete copy of all relevant data

    • A project with associated datasets will include copies of the datasets in datasets instead of only uuids

  • Status

    • Requested, Accepted, Rejected

  • User

    • User that made the request

  • Updates

    • Mini log system

    {
  'timestamp': <current time>,
  'new_status': 'new_status'
}

  • Type

    • dataset or project

  • Comments

    • Comments from the reviewer

Computed Fields (request)

  • Other_requests

    • Other requests that have been made for the same entry

    • To allow the reviewer to see e.g. earlier comments

Fields (doi entry)

  • _id

    • The DOI identifier

  • timestamp

    • When the entry was created

  • Data

    • The complete entry that has been accepted

Other topics

Identifiers

  • Only uuid initially

  • Can request a “fancier” local identifer for dataset/project

    • Style similar to:

      • scilifelab.facility.orderxyz.dataset1

      • scilifelab.projects.title1

  • All datasets and projects can request DOI

    • The required fields will be checked if empty. If not the request will be sent for evaluation by e.g. admin

Permissions

  • “Permission classes” used to evaluate what a user may do

    • CREATE_ORDERS

    • MANAGE_USERS

    • EDIT_ANY_DATA

    • READ_OWNERS

    • DOI_REVIEWER

  • “Default groups”

    • Template for user, giving a specific set of permissions

    • Admin - “all”

    • Facility - “create orders”+”read ownerships”