Skip to content
GitHub

VECTR MongoDB to Postgres Migration Tool

The purpose of this tool is to convert a VECTR 8.x MongoDB to a VECTR 9.0 Postgres DB.

Note

All of these instructions assume you're running a standard community edition install with docker-compose on Ubuntu. If you have deviated from this standard install you will need to adjust accordingly

Directory Configuration

Create a directory for the tool to run in, for this example we'll use /opt/vectr-sql-migration

mkdir /opt/vectr-sql-migration

Download and unzip the migration zip

cd /opt/vectr-sql-migration
wget https://github.com/SecurityRiskAdvisors/vectr-sql-migration/releases/download/ce-1.0.0/sqlmigration-1.0.0.zip
unzip sqlmigration-1.0.0.zip
Create required directories

mkdir -p /opt/vectr-sql-migration/user/mongo
mkdir -p /opt/vectr-sql-migration/sqlmigrations
mkdir -p /opt/vectr-sql-migration/mongodumps

Run Mongodump and Extract

Run the perform_mongodump.sh script with parameters as follows. - Following -d insert the VECTR directory, in this example we will assume /opt/vectr - Following the -o insert the directory created previously, in our example /opt/vectr-sql-migration/mongodumps 

./perform_mongodump.sh -d /opt/vectr -o /opt/vectr-sql-migration/mongodumps

Extract the contents to the user/mongo directory created previously, in our example /opt/vectr-sql-migration/user/mongo Note you will need the tar filename output from the perform_mongodump.sh script. This should be in a time format, for example 2024_02_05__22_39_53.tgz

tar -zxvf /opt/vectr-sql-migration/mongodumps/2024_02_05__22_39_53.tgz -C /opt/vectr-sql-migration/user/mongo/

At this point you should have the dump extracted. To confirm the folder structure is correct look for the following GoldStandard folder, in our example /opt/vectr-sql-migration/user/mongo/GoldStandard

If so you are ready to proceed, if not look for any unnecessary nested folders until it is as above.

Run Migration

Attention

There are new constraints on the size of some fields in the sql data model that previously did not exist.

The default behavior of the migration tool is to truncate large strings to conform to the new schema. If you wish to change this, in the .env file set FORCE_MIGRATE=false This will cause the migration tool to exit with errors instead of modifying data.**

From your vectr-sql-migration directory run the docker-compose. Note we're running this command without the -d. We must observe the migration running in case of any errors. If you do see a violation it will print to the log. Open an issue under this project using the issue template.

docker compose up

This may take some time to process depending on the speed of your VM. A successful migration will show this at the end: 

vectr-migration_1  | DONE MIGRATION
vectrmigration1_vectr-migration_1 exited with code 0

Clean up

In the case you have a successful migration, you will want to Ctrl + C to stop the containers. Then to remove the migration docker envionment Warning: This command destroys the volumes used for migration, which should no longer be needed. If you run this command in the incorrect directory you may delete data inadventently. Ensure you are in the directory with the sql-migration docker-compose.yml 

docker compose down -v

Load into VECTR 9.0

Note

As with any migration we believe it is important to reiterate the importance of backups. Please ensure you have a known good backup before removing any VECTR 8.x volumes or directories.

Stop your existing 8.x install

cd /opt/vectr
docker compose down

We recommend moving your VECTR directory and creating a new one for 9.x

cd ..
mv /opt/vectr /opt/vectr_8x
mkdir /opt/vectr

Download VECTR 9.0 Note this version is at the point in time this was written, there may be a newer version available that requires additional .env configuration. Check Release Notes for the latest version available.

cd /opt/vectr
wget https://github.com/SecurityRiskAdvisors/VECTR/releases/download/ce-9.0.2/sra-vectr-runtime-9.0.2-ce.zip 
unzip sra-vectr-runtime-9.0.2-ce.zip

Configure .env

You will need to edit the new .env file to set it up for 9.x. These should come from your 8.x .env file. For a refresher on .env configuration see the docs site

https://docs.vectr.io/

  • VECTR_HOSTNAME
  • VECTR_PORT
  • VECTR_DATA_KEY

These are optional variables you may have and need

  • VECTR_EXTERNAL_HOSTNAME
  • VECTR_EXTERNAL_PORT
  • CA_PASS
  • VECTR_SSL_CRT
  • VECTR_SSL_KEY

You should set the new database password

  • POSTGRES_PASSWORD

These are only used for the session and do not need to be the same as 8.x but should be set

  • JWS_KEY
  • JWE_KEY

Create restore directory and copy sqldump from migration tool

mkdir -p /opt/vectr/user/postgres
cp /opt/vectr-sql-migration/sqlmigrations/sqldump.sql.gz /opt/vectr/user/postgres
cd /opt/vectr
docker compose up -d

Login and verify your data has been migrated successfully!