Mnesia migration

[vkatsuba]

Proposed changes include:

• CTL for migration from Mnesia to RDMS

How To:
Migrate pubsub_nodes:

$ mongooseimctl migrate mnesia rdbms pubsub_nodes

Migrate pubsub_subscriptions:

$ mongooseimctl migrate mnesia rdbms pubsub_subscriptions

Migrate pubsub_affiliations:

$ mongooseimctl migrate mnesia rdbms pubsub_affiliations

Migrate pubsub_items:

$ mongooseimctl migrate mnesia rdbms pubsub_items

Migrate users:

$ mongooseimctl migrate mnesia rdbms users

Migrate vcard_search:

$ mongooseimctl migrate mnesia rdbms vcard_search

Migrate vcard:

$ mongooseimctl migrate mnesia rdbms vcard

Migrate event_pusher_push_subscription:

$ mongooseimctl migrate mnesia rdbms event_pusher_push_subscription

Migrate rosterusers:

$ mongooseimctl migrate mnesia rdbms rosterusers

Migrate roster_version:

$ mongooseimctl migrate mnesia rdbms roster_version

Migrate rostergroups:

$ mongooseimctl migrate mnesia rdbms rostergroups

Migrate last:

$ mongooseimctl migrate mnesia rdbms last

Migrate private_storage:

$ mongooseimctl migrate mnesia rdbms private_storage

Migrate offline_message:

$ mongooseimctl migrate mnesia rdbms offline_message

Migrate muc_light_rooms:
NOTES: Looks like before migration of muc_light_config and muc_light_occupants tables need to know room ID from muc_light_rooms, by this reason when run migrate mnesia rdbms muc_light_rooms will be migrated tables: muc_light_rooms, muc_light_config, muc_light_occupants

$ mongooseimctl migrate mnesia rdbms muc_light_rooms

Migrate all:

$ mongooseimctl migrate mnesia rdbms all

[michalwski]

That's a huge effort and functionality we've been missing for long time already. Thanks a lot for that.
I managed to do partial review.

[codecov]

Codecov Report

Merging #2561 (6b0903f) into master (53935c5) will increase coverage by 0.96%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2561      +/-   ##
==========================================
+ Coverage   78.02%   78.99%   +0.96%     
==========================================
  Files         374      348      -26     
  Lines       31190    29647    -1543     
==========================================
- Hits        24337    23419     -918     
+ Misses       6853     6228     -625     

Impacted Files	Coverage Δ
src/sasl/cyrsasl_scram.erl	68.11% <0.00%> (-27.44%)	⬇️
src/metrics/mongoose_metrics_mam_hooks.erl	66.66% <0.00%> (-20.00%)	⬇️
src/rdbms/rdbms_queries_mssql.erl	85.71% <0.00%> (-14.29%)	⬇️
src/auth/ejabberd_auth_internal.erl	57.14% <0.00%> (-13.77%)	⬇️
src/ejabberd.erl	45.00% <0.00%> (-10.00%)	⬇️
src/mongoose_lib.erl	73.68% <0.00%> (-8.54%)	⬇️
src/jlib.erl	82.84% <0.00%> (-6.69%)	⬇️
src/auth/ejabberd_auth.erl	61.42% <0.00%> (-6.28%)	⬇️
src/shaper.erl	94.11% <0.00%> (-5.89%)	⬇️
src/config/mongoose_config_parser.erl	74.29% <0.00%> (-5.71%)	⬇️
... and 277 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53935c5...73f4ce0. Read the comment docs.

[michalwski]

Thanks for the update I had 2 more comments. Also we usually don't merge new functionality to master without tests. I don't expect you to work on tests now, you did huge work already. I'll try accommodate some time for writing tests for these migrations in January.

[michalwski]

Hi @vkatsuba, sorry it took me so long to respond.
Regarding the approach to tests, I think the following would the simplest and enough approach:

• It makes sens to run the tests only on these jobs which have a RDBMS backend configured. You can detect that by calling mongoose_helper:is_rdbms_enabled(XMPPHost).
• The test could do sth like this:
  • use the mod_*_mnesia module to generate some data in Mnesia
  • run appropriate migration function
  • use the mod_*_rdbms and verify if the migration succeeded. Basically if functions from mod_*_rdbms returns the same values as corresponding function from mod_*_mnesia
  • remember to call mod_*_mnesia:init in order to create the mnesia table
  • remember to remove mnesia tables after the test

Let me know what do you think about it.

[vkatsuba]

@michalwski: Please look it :)

Still need add CT. I will try to do it soonish.

[vkatsuba]

@vkatsuba: Any news?

Hi @Neustradamus. Current implementation a little bit outdated and should be updated with using latest tables structure. Still in progress. Hopefully in the next few weeks I can take the time to update and complete the current implementation. Thank you for reminding.

[vkatsuba]

Hi @Neustradamus. Sorry but I don't have a lot of time for now for finalize this version for MIM, however if you need include this functionality into your project you can always ping me and I will try to help you will adaptation of this code for your project.

Regards,
-V