¶ Introduction
As the MoH Uganda switches from using DHIS2 as its Facility Registry, a Master Facility List (MFL) was created for the same purpose. Health systems working with public health information are thus required to pick or synchronise health facilities from the MFL.
¶ What is the MFL Integrator?
The MFL integrator is an application that helps you to integrate multiple DHIS2 instances with the Master Facility List Registry. The application ensures addtions and modifications of health facilities in the MFL are synchronized to all the subscribing DHIS2 instances.
 |
|---|
| MFL Integrator Context Diagram |
¶ How the MFL Integrator works
- The application works by referencing a base DHIS2 instance which it uses to pick an organisation unit hierarchy.
- The organisation unit hierarcy picked in step 1 above is added to a database within the integrator and is used for mapping individual organisation units to their corresponding locations in the
MFL(Regions, Districts, and Sub-counties). The MFL API returns health facilities asFHIRlocation resources. - New and modified health facilities added to the integrator DB are publised to the Base DHIS2 instance.
- The subscribing DHIS2 instances receive copies of the base DHIS2 instance hierarchy and the new and modified health facilities from the MFL.
The MFL Integrator has been implemented as a Go application that compiles to a binary and that binary along with the config file mfld.yml is all you need to run it on your server.
¶ Components of the MFL Integrator
The data mfl-integrator is comprised of the following tools and components.
| Tool/Component | Description |
|---|---|
| Database | The core database for the mfl-integrator. It keeps a clone of the MFL and also tracks revisions to health facilities over time. |
| In-built Data Exchange Middleware | The data exchange middleware is used to queue requests for adding and modifying health facilities intended for subscribing DHIS2 instances. Requests can be monitored for success, errors and any failures. The middleware offers the capability of sending a single request to multiple DHIS2 instances while tracking communication with each instance. |
| Scheduler | The scheduler is responsible to scheduling the synchronisation process |
¶ Configuration
mfl-integrator uses a tiered configuration system, each option takes precendence over the ones above it:
- The configuration file. The default path is
/etc/mflintegrator/mfld.yml - Environment variables starting with
MFLINTEGRATOR_ - Command line parameters
We recommend running mfl-integrator with no changes to the configuration and no parameters, using only environment variables to configure it. You can use mfl-integrator --help to see a list of the environment variables and parameters and for more details on each option.
You will need to configure the following settings:
The configuration file is of a yaml format, with the following configurations:
Default path /etc/mflintegrator/mfld.yml
| Tool/Component | Description | Default |
|---|---|---|
| Databse Configurations | ||
| uri | URL describing how to connect to the mfl-integrator database | "postgres://postgres:postgres@localhost/mfldb?sslmode=disable" |
| Server Configurations | ||
| host | The hostname/ IP address for the host where the mfl-integrator is installed | localhost |
| http_port | The port on which to run the mfl-integrator daemon | 9090 |
| max_retries | The number of maximum retries for queued requests | 3 |
| max_concurrent | The maximum number of consumer go routines submitting requests | 3 |
| sync_on | Whether facility synchronisation via the exhange middleware is enabled | true |
| request_process_interval | The sleep interval in seconds for producer routine to add requests to dynamic queue | 4 |
| logdir | The log directory for the application log files | /var/log/mflintegrator |
| API Configurations | ||
| mfl_base_url | The MFL base API URL | |
| mfl_user | The MFL API username | |
| mfl_password | The MFL API user password | |
| mfl_dhis2_base_url | The MFL base DHIS2 instance base API URL | |
| mfl_dhis2_username | The MFL base DHIS2 instance username | |
| mfl_dhis2_password | The MFL base DHIS2 user password | |
| mfl_dhis2_pat | The MFL base DHIS2 instance Personal Access Token | |
| mfl_cc_dhis2_hierarchy_servers | ||
| mfl_cc_dhis2_servers | The MFL CC DHIS2 instances to receive copy of facilities | |
| mfl_cc_dhis2_create_servers | The MFL CC DHIS2 instances to receive copy of OU hierarchy | |
| mfl_cc_dhis2_ougroup_add_servers | The MFL CC DHIS2 instances name of configured API endpoint used to add ous to groups | |
| mfl_dhis2_tree_ids | The MFL base DHIS2 instance top-level organisation unit UIDs | |
| mfl_metadata_batch_size | The MFL Metadata items to chunk in a metadata request | 50 |
| mfl_sync_cron_expression | The MFL health facility syncronisation cron expression | 0 0-23/6 * * * |
| mfl_sync_cron_expression | The retry interval used by the data exchange middleware is re-queuing failed requests | */5 * * * * |
| mfl_dhis2_facility_level | The MFL base DHIS2 instance organisation unit level for health facilities |
Database Releated
MFLINTEGRATOR_DB: URL describing how to connect to the mfl-integrator database(default: "postgres://postgres:postgres@localhost/mfldb?sslmode=disable")
Exchange Middleware Related
MFLINTEGRATOR_HOST: The hostname/ IP address for the host where the mfl-integrator is installed (default localhost)MFLINTEGRATOR_SERVER_PORT: The port on which to run the mfl-integrator daemon (default 9090)MFLINTEGRATOR_MAX_RETRIES: The number of maximum retries for queued requests (default 3)MFLINTEGRATOR_START_SUBMISSION_PERIOD: The global hour marking the start of request submission to destination endpoints (default 0)MFLINTEGRATOR_END_SUBMISSION_PERIOD: The global hour marking the end of request submission to destination endpoints (default 23)MFLINTEGRATOR_MAX_CONCURRENT: The maximum number of consumer go routines submitting requests (default 3)MFLINTEGRATOR_SYNC_ON: Whether facility synchronisation via the exhange middleware is enabled (default true)MFLINTEGRATOR_REQUEST_PROCESS_INTERVAL: The sleep interval in seconds for producer routine to add requests to dynamic queue (default 4)MFLINTEGRATOR_LOGDIR: The log directory for the application log files (default "/var/log/mflintegrator")MFLINTEGRATOR_USE_SSL: Whether the use of SSL is enabled for requests submission (default false)
MFL Integration related
MFLINTEGRATOR_BASE_URL: The MFL base API URLMFLINTEGRATOR_USER: The MFL API usernameMFLINTEGRATOR_PASSWORD: The MFL API user passwordMFLINTEGRATOR_DHIS2_BASE_URL: The MFL base DHIS2 instance base API URLMFLINTEGRATOR_DHIS2_USER: The MFL base DHIS2 instance usernameMFLINTEGRATOR_DHIS2_PASSWORD: The MFL base DHIS2 user passwordMFLINTEGRATOR_DHIS2_PAT: The MFL base DHIS2 instance Personal Access TokenMFLINTEGRATOR_DHIS2_TREE_IDS: The MFL base DHIS2 instance top-level organisation unit UIDsMFLINTEGRATOR_DHIS2_FACILITY_LEVEL: The MFL base DHIS2 instance organisation unit level for health facilities.MFLINTEGRATOR_DHIS2_OU_MFLID_ATTRIBUTE_ID: The DHIS2 OU MFLID Attribute IDMFLINTEGRATOR_CC_DHIS2_SERVERS: The MFL CC DHIS2 instances to receive copy of facilitiesMFLINTEGRATOR_CC_DHIS2_HIERARCHY_SERVERS: The MFL CC DHIS2 instances to receive copy of OU hierarchyMFLINTEGRATOR_CC_DHIS2_CREATE_SERVERS: The MFL CC DHIS2 instances to receive copy of OU creationsMFLINTEGRATOR_CC_DHIS2_OUGROUP_ADD_SERVERS: The MFL CC DHIS2 instances name of configured API endpoint used to add ous to groupsMFLINTEGRATOR_METADATA_BATCH_SIZE: The MFL Metadata items to chunk in a metadata request (default 50)MFLINTEGRATOR_SYNC_CRON_EXPRESSION: The MFL health facility syncronisation cron expressionMFLINTEGRATOR_RETRY_CRON_EXPRESSION: The retry interval used by the data exchange middleware is re-queuing failed requests
database:
uri: "postgres://postgres:postgres@localhost/mfldb?sslmode=disable"
db_host: "localhost"
db_port: "5432"
db_username: "xxx"
db_password: "xxx"
server:
host: "localhost"
http_port: 9090
max_retries: 3
max_concurrent: 5
sync_on: true
request_process_interval: 4
logdir: "/var/log/dispatcher2"
api:
mfl_base_url: ""
mfl_user: ""
mfl_password: ""
mfl_dhis2_base_url: ""
mfl_dhis2_username: ""
mfl_dhis2_password: ""
mfl_dhis2_pat: ""
mfl_cc_dhis2_hierarchy_servers: "test240_OU,test237_OU"
mfl_cc_dhis2_servers: "test237_OU,test240_OU"
mfl_cc_dhis2_create_servers: "test240_OU,test237_OU"
mfl_cc_dhis2_ougroup_add_servers: "test240_OU_GroupAdd,test237_OU_GroupAdd"
mfl_dhis2_tree_ids: "akV6429SUqu"
mfl_metadata_batch_size: 50
mfl_dhis2_facility_level: 5
mfl_sync_cron_expression: "0 0-23/6 * * *"
authtoken: ABC
smsurl: ""
The sample .env file you can source to update your environment variables
- MFLINTEGRATOR_DB="postgres://postgres:postgres@localhost/mfldb?sslmode=disable"
- MFLINTEGRATOR_HOST="localhost"
- MFLINTEGRATOR_SERVER_PORT="9090"
- MFLINTEGRATOR_MAX_RETRIES=3
- MFLINTEGRATOR_START_SUBMISSION_PERIOD=0
- MFLINTEGRATOR_END_SUBMISSION_PERIOD=23
- MFLINTEGRATOR_MAX_CONCURRENT=3
- MFLINTEGRATOR_SYNC_ON=false
- MFLINTEGRATOR_REQUEST_PROCESS_INTERVAL=4
- MFLINTEGRATOR_LOGDIR="/var/log/mflintegrator/"
- MFLINTEGRATOR_USE_SSL=false
- MFLINTEGRATOR_BASE_URL=""
- MFLINTEGRATOR_USER=""
- MFLINTEGRATOR_PASSWORD=""
- MFLINTEGRATOR_DHIS2_BASE_URL=""
- MFLINTEGRATOR_DHIS2_USER=""
- MFLINTEGRATOR_DHIS2_PASSWORD=""
- MFLINTEGRATOR_DHIS2_PAT=""
- MFLINTEGRATOR_DHIS2_TREE_IDS=""
- MFLINTEGRATOR_DHIS2_FACILITY_LEVEL=5
- MFLINTEGRATOR_DHIS2_OU_MFLID_ATTRIBUTE_ID=""
- MFLINTEGRATOR_SYNC_CRON_EXPRESSION="0 0-23/6 * * *"
- MFLINTEGRATOR_RETRY_CRON_EXPRESSION="*/5 * * * *"
- MFLINTEGRATOR_CC_DHIS2_SERVERS="test237_OU,test240_OU"
- MFLINTEGRATOR_CC_DHIS2_HIERARCHY_SERVERS="test237_OU,test240_OU"
- MFLINTEGRATOR_CC_DHIS2_CREATE_SERVERS="test237_OU,test240_OU"
- MFLINTEGRATOR_CC_DHIS2_OUGROUP_ADD_SERVERS="test240_OU_GroupAdd,test237_OU_GroupAdd"
- MFLINTEGRATOR_METADATA_BATCH_SIZE=5
We can add configurations for the sync DHIS2 instance in conf.d directory in the directory where the configuration file resides - that is; /etc/mflintegrator. The server configurations are expected in a JSON format.
Create the configuration directory if it does not exist.
mkdir -p /etc/mflintegrator/conf.d
For each DHIS2 instance we sync with, we require two configurations
- one for createing and updating organisation units
- You can use a style like; <servername_OU.json> for these files. epivac_OU.json for example.
- one for add organisation units to their respective orgainisation unit groups (that is; the health facility levels HC II, HC III, e.t.c)
- You can use a style <servername_OU_GroupAdd.json> for these files. epivac_OU_GroupAdd.json for example.
Here is a sample configuration file for a typical <servername_OU.json>.
{
"name": "test240_OU",
"username":"admin",
"password":"district",
"URL":"https://play.dhis2.org/40.0.1/api/metadata.json",
"IsProxyServer":false,
"systemType":"DHIS2",
"endPointType":"Organisationunit",
"authMethod":"Token",
"authToken":"d2pat_xxxxxxxxxxxxxxxxxxxxxxxxxx",
"HTTPMethod":"POST",
"callbackURL":"",
"allowCallbacks":false,
"allowCopies":false,
"useSSL": true,
"CCURLS": [],
"useAsync": false,
"parseResponses": true,
"startSubmissionPeriod":0,
"endSubmissionPeriod":23,
"URLParams": {
"mergeMode":"MERGE",
"importStrategy":
"CREATE_AND_UPDATE",
"async": true,
"importReportMode": "DEBUG"},
"suspended":false,
"IPAddress": "localhost",
"JSONResponseXPATH": "",
"XMLResponseXPATH": "",
"allowedSources":["localhost", "dhis2"]
}
Here is a sample configuration file for a typical <servername_OU_GROUPADD.json>.
{
"name": "test240_OU_GroupAdd",
"username":"admin",
"password":"district",
"URL":"https://play.dhis2.org/40.0.1/api/organisationUnitGroups",
"IsProxyServer":false,
"systemType":"DHIS2",
"endPointType":"OU_ORGUNIT_GROUP_ADD",
"authMethod":"Token",
"authToken":"d2pat_xxxxxxxxxxxxxxx",
"HTTPMethod":"PATCH",
"callbackURL":"",
"allowCallbacks":false,
"allowCopies":false,
"useSSL": true,
"CCURLS": [],
"useAsync": false,
"parseResponses": true,
"startSubmissionPeriod":0,
"endSubmissionPeriod":23,
"URLParams": {},
"suspended":false,
"IPAddress": "*",
"JSONResponseXPATH": "",
"XMLResponseXPATH": "",
"allowedSources":["localhost", "dhis2"]
}
Only the url configurations change in the two types of configurations.
Changes to any of these configurations will be hot reloaded by the application.
¶ Deployment
Install mfl-integrator source in your workspace with:
go get github.com/HISP-Uganda/mfl-integrator
Build mfl-integrator with:
go install github.com/HISP-Uganda/mfl-integrator
This will create a new executable in $GOPATH/bin called mfl-integrator
¶ Running the MFL-Integrator
If the $GOPATH/bin directory is part of your $PATH environment variable, then the mfl-integrator is all you need to run the application if you finished the configurtation step.
To see how to use the binary, issue the following command
mfl-integrator -h
The output of the help command above is as follows:
Usage of ./mfl-integrator:
--config-file string The path to the configuration file of the application (default "/etc/mflintegrator/mfld.yml")
--force-sync Whether to forcefully sync organisation unit hierarchy
--skip-ousync Whether to skip ou and facility sync. But process requests
Here is a typical example for running the binary by passing a path to a configuration file
mfl-integrator --config-file /etc/mflintegrator/mfl.yml