Sync using OneDrive

[wizardict]

You are right that a centralized DB will solve problems, but most of MMEX users don't want to have their data on a webserver not under their control, so the only way it's keep in sync the full DB file..

That's a good point. I wasn't thinking of a hosted server, more a database that is local to the user so they still have full control, but still that would not suit everyone.

There could be a solution that would suit everyone though, something like SymmetricDB.  That would allow local sqllite databases which you have now, but with the option to sync to a central server for those who want it. It does proper SQL sync so would allow multiple users to use the application at the same time, and would also still allow the mobile app to be used offline.


If anyone is interested in this kind of solution I would be prepared to help develop the desktop side, but it would need your Android developer to implement the android version (which is documented on their website) as I have no experience in android development unfortunately.

[GabrieleV]

Is this one? https://www.symmetricds.org

Considering case where there isn't a central server for security reasons, how will sync handled?
There will be for example 3 DB files on Dropbox called "master", "android", "desktop" and symmetricds will keep them in sync?

[wizardict]

Is this one? https://www.symmetricds.org

Considering case where there isn't a central server for security reasons, how will sync handled?
There will be for example 3 DB files on Dropbox called "master", "android", "desktop" and symmetricds will keep them in sync?

Yes that is the correct product, as you can see it is open source so fits with your product nicely.


The intention of this would be to allow people the option to include a central server, not exclude it. People who don't want a central server would continue to work as they are, people who do want improved functionality and proper sync functions would use this. So...


They would set up a central server of their own (it could be on their desktop still, or a web server whatever they prefer. It could by MSSQL, MYSQL or even SQLITE still, whatever they prefer and is supported by symmetricds).

All of the installs (desktop, mobile client etc) would store the data locally in their own SQLite database, not on dropbox or OneDrive etc these are not suitable methods to sync databases and aren't intended to be used in this way.

SymmetricDS would be used to sync each client with the "central server" owned by the user.




Synchronisation would be far improved, without the "single client only" model that is al that's possible with the current system. This would also open the possibility for different client development, such as a fully functional web based client. As soon as you have a proper "client > server" model you open up a lot of new possibilities and functionality, but doing it this way would still allow people to have the simple standalone install they currently have if they prefer.

[MisterY]

This is starting to feel like my personal blog! Does anyone involved in development monitor or reply to posts on here?

There's a new release of the android app, but unfortunately the sync issues remain. Entities do not sync to the app, and any changes made on the mobile app uploads the database and delete all of the changes that haven't been synchronised.

There is also the issue of the mobile app causing a new database to be created (i.e. it doesn't update the database it results in a separate database being created). I see this issue is reported elsewhere on the forum with dropbox too, so it isn't OneDrive specific.



Unless there is a response from development that these issue are being looked in to I think I will have to abandon MMEX and look elsewhere. It's shame, as it has promise, but without a working mobile version I don't think this is a viable project, I'm happy to help trying to resolve these issues, but with no response I don't really know who I'm trying to help.

Hi,

As the very first item, I would kindly point you to read the documentation that was written for this purpose. It might not be "in-your-face" type of documentation but there is a link to the User Manual from the app's home page. There the process of setting up the synchronization is explained into some detail. There are also a few more pages referring specifically to the (previous) Dropbox-only synchronization but their value is in explaining the concepts. I mention this because some people open the database file from a Dropbox client and expect the synchronization to work seamlessly.
Secondly, please note that this is an Open Source project and it is nobody's duty to supply support. You are certainly welcome to contribute functionality, documentation, code, translations, and provide support to other users in the forums, just like we all are.

If you experience specific issues, you can report them through several channels but do please keep in mind that we also have a life and full-time jobs that take priority over any hobby project. Do get in touch via email, or post an issue on GitHub.

[MisterY]

OK.....I have located the root of this issue.

The problem occurs when the mobile app can't update the database (i.e. it is locked by the desktop client) and a 2nd database is created. As long as the 2 databases exist OneDrive does not update them correctly, so the 2nd database needs to be deleted then everything syncs as expected.


So the issue is with conflicting updates. Do Dropbox and/or Google drive handle conflicts in the same way? I've seen the same issue reported on dropbox, so presumably that's the same, what about the others?

All of this would be resolved with a proper database and network capabilities

Hi, sorry I replied before reading the whole thread.
Yes! This is one of the key issues for the desktop version. The database file is locked as long as the application is running. This will **always** cause a sync conflict if the file is modified on mobile at the same time.
It has been noted earlier and I'm not sure if there is something that can be done in the application, except perhaps constantly closing the connection when it is not needed.
**Or**, a user could simply close the application when he's done with it.

[wizardict]

OK.....I have located the root of this issue.

The problem occurs when the mobile app can't update the database (i.e. it is locked by the desktop client) and a 2nd database is created. As long as the 2 databases exist OneDrive does not update them correctly, so the 2nd database needs to be deleted then everything syncs as expected.


So the issue is with conflicting updates. Do Dropbox and/or Google drive handle conflicts in the same way? I've seen the same issue reported on dropbox, so presumably that's the same, what about the others?

All of this would be resolved with a proper database and network capabilities

Hi, sorry I replied before reading the whole thread.
Yes! This is one of the key issues for the desktop version. The database file is locked as long as the application is running. This will **always** cause a sync conflict if the file is modified on mobile at the same time.
It has been noted earlier and I'm not sure if there is something that can be done in the application, except perhaps constantly closing the connection when it is not needed.
**Or**, a user could simply close the application when he's done with it.

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested? The reality is dropbox etc were never intended to be used in this way, so to develop an application that requires sync across multiple clients really means moving away from this model.

[MisterY]

Is this one? https://www.symmetricds.org

Considering case where there isn't a central server for security reasons, how will sync handled?
There will be for example 3 DB files on Dropbox called "master", "android", "desktop" and symmetricds will keep them in sync?

Yes that is the correct product, as you can see it is open source so fits with your product nicely.


The intention of this would be to allow people the option to include a central server, not exclude it. People who don't want a central server would continue to work as they are, people who do want improved functionality and proper sync functions would use this. So...


They would set up a central server of their own (it could be on their desktop still, or a web server whatever they prefer. It could by MSSQL, MYSQL or even SQLITE still, whatever they prefer and is supported by symmetricds).

All of the installs (desktop, mobile client etc) would store the data locally in their own SQLite database, not on dropbox or OneDrive etc these are not suitable methods to sync databases and aren't intended to be used in this way.

SymmetricDS would be used to sync each client with the "central server" owned by the user.




Synchronisation would be far improved, without the "single client only" model that is al that's possible with the current system. This would also open the possibility for different client development, such as a fully functional web based client. As soon as you have a proper "client > server" model you open up a lot of new possibilities and functionality, but doing it this way would still allow people to have the simple standalone install they currently have if they prefer.

I like the idea in general but wouldn't this destroy the option of having a standalone application, whether desktop or mobile? There's still plenty of users who do not synchronize the database and use a single platform exclusively.

[MisterY]

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested?

There would probably be an option to initialize this only as a part of synchronization settings?

[wizardict]

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested?

There would probably be an option to initialize this only as a part of synchronization settings?

Yes exactly. The application would work exactly as it does currently, with data saved to a local SQLite database, so people that are happy using it this way can continue to do so.
For people that require greater network capabilities there would be an option in the settings to configure the "master server", which would then sync the data to that central database. If you don't want to use it, simply don't enable it.

[MisterY]

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested?

There would probably be an option to initialize this only as a part of synchronization settings?

Yes exactly. The application would work exactly as it does currently, with data saved to a local SQLite database, so people that are happy using it this way can continue to do so.
For people that require greater network capabilities there would be an option in the settings to configure the "master server", which would then sync the data to that central database. If you don't want to use it, simply don't enable it.

Certainly seems interesting. I'll do some more reading through their documentation. And would certainly love to use it. Especially if it
could do a record-level sync. This would  greatly reduce the size of the network traffic, for example.
However, I'm concerned that most users would have great issues installing their
own server. Even supporting the sync via well-known file-syncing
providers is causing too much trouble already.

Going back to one thing you mentioned, that you don't know much about Android development.
Well, neither did I before joining this project. It was a way to learn
more and get my hands dirty, so to speak. In any case, I wholeheartedly
recommend the following video to everyone. Please watch at least the
first five minutes:

https://www.youtube.com/watch?v=TK1mBqKvIyU

[MisterY]

Actually, if it can do a row-level replication, I'd be happy to implement it in addition to existing file-level sync.

https://github.com/moneymanagerex/andro ... issues/991

[wizardict]

I agree that many users would not know how to set up their own server, but would want the functionality it provides. I think ultimately there would need to be a "server" installation of MMEX. That should be fairly simple, an installed for a SQL DB of choice and the sync component. For the initial phase though I would say implement the sync and let people with can set up their own server try it out.

In terms of support I think you would find this requires less support than the current system, not more. The issues you're having with dropbox sync is that dropbox was never intended to sync database files in this way, trying to use something for a purpose it was not intended for is always going to cause issues. By implementing something that is designed for this purpose, once the bugs are sorted it, it should work fairly smoothly.


I appreciate this is potentially a fair bit of work, but I think it's worth it. I like MMEX but there are numerous other free open source solution out there that do a similar job. The thing that set MMEX apart for me (and why I'm here) is that I need the ability to enter transactions when I'm out of the house. That's when I make most of my smaller transactions, and currently regularly forget to record them when I get home. The android integration you currently have sort of resolves that issue, but there are too many limitations. Not least the fact that if I forget to close the desktop version and enter a transaction on android I'm going to end up with 2 versions of the database that are difficult to reconcile.

Take a look at the SymmetricDS site and see if you're happy with it and what it can do (there are other similar solutions out there but I think this is the best fit for this project). I believe if you get this right it will really set MMEX apart from the crowd.

[trice001]

thanks for that.

[javiermarcon]

Excelent idea on using symmetricds to manage cetralized database. Is there a tutorial on how to configure symmetricds to sync several mmex databases?

Thanks.

[wizardict]

Excelent idea on using symmetricds to manage cetralized database. Is there a tutorial on how to configure symmetricds to sync several mmex databases?

Thanks.

As far as I'm aware no testing or development has been done in this area so far, so there are no tutorials as you would be breaking new ground. Whilst I think there's agreement that it would improve synchronisation, there are a lot of other things to work on and devs only have a certain amount of time.

[MisterY]

Yes, I'd certainly encourage anyone interested to explore and contribute. As I keep saying, it's an Open Source hobby project.
Please keep in mind that, even though in MMEX there are several participating developers, for Android there is only one person (more or less) actively involved so don't expect much in terms of development or documentation at this point.
Without two-three or more actively participating developers, I don't think we'll be progressing much in the short term.

[Renee]

Excelent idea on using symmetricds to manage cetralized database. Is there a tutorial on how to configure symmetricds to sync several mmex databases?

Thanks.

As far as I'm aware no testing or development has been done in this area so far, so there are no tutorials as you would be breaking new ground. Whilst I think there's agreement that it would improve synchronisation, there are a lot of other things to work on and devs only have a certain amount of time.

IMO, this feature is so important, I may be able to volunteer my time to do documentation, as I am not a programmer and I haven't ever made a youtube video (although, I am considering making some English quality videos for this product). I am so grateful for this product, I feel it is the least I can do to contribute back at least 1-2 hrs per week to start, which will also enable them to continue to be there for us.

I too find this product superior for the same reasons as Wizardict. These developers have been very active and attentive to their audience for over 10 years and again.

Wizardict, have you made this feature a reality as of yet? Where do you stand with this feature? Thank you for your time.

Any other volunteers?