Sync using OneDrive

[wizardict]

Hi

As I understand it sync using OneDrive is now possible? Is there any documentation anywhere? Everything I can find seems to be old information.

I've set up a database on my PC which saves to OneDrive. I've now installed the Android app, and in sync settings have configured my OneDrive account. Everything up to that point seems fine.

The app then asks me to browse for the database location, but it doesn't include OneDrive in the locations to browse, just the local device. Am I missing something?

[wizardict]

OK I figured if out.  I didn't notice the option to select the database within the Onedrive settings.

There were a number or errors syncing the database, these seem to be primarily related to the format of the currencies, only my primary currency can be used. This isn't a big problem for me, I'm unlikely to use the app for foreign currency transactions.

Other than that everything is looking promising so far.

[wizardict]

Not so good I'm afraid

Many things do not sync from the desktop client to the android app. These include Categories and Payees and may include other entities I haven't noticed.

When you make a change on the android app and sync it back all of the changes to Categories etc that have been made on the desktop are wiped out by the ones on the android app. So it appears to sync from the app to the desktop but not the other way round.

If you have both open at the same time, or open the desktop client soon after making a change on the android app it results in 2 versions of the database as it can't sync them (probably more of an issue with OneDrive than MMEX but still a major issue).


I've also downloaded the beta android app to see if any of the issues are resolved, but unfortunately that won't even use OneDrive. When selecting it in the Sync options it opens the OneDrive login page. After logging in it returns back to the app but the provider shows as "%s" instead of OneDrive. From that point it's impossible to select OneDrive as a provider without reinstalling the app.



Are there any fixes to these issues? Or am I doing something wrong?

Regrettably if the android app and desktop client can't sync correctly it makes the app useless, and ultimately MMEX useless to me. This would be extremely disappointing as it was looking very promising, but the very basics I require are the ability to manage my money on my desktop at home and at least enter basic transaction information on my phone whilst out. The web app could be helpful, but that means I could only enter transactions on my phone if I have an internet connection, which again would be too limiting.


Any thoughts or ideas on how these issues can be resolved? I really would like to use MMEX if it can at least do the basics.

[wizardict]

This is starting to feel like my personal blog! Does anyone involved in development monitor or reply to posts on here?

There's a new release of the android app, but unfortunately the sync issues remain. Entities do not sync to the app, and any changes made on the mobile app uploads the database and delete all of the changes that haven't been synchronised.

There is also the issue of the mobile app causing a new database to be created (i.e. it doesn't update the database it results in a separate database being created). I see this issue is reported elsewhere on the forum with dropbox too, so it isn't OneDrive specific.



Unless there is a response from development that these issue are being looked in to I think I will have to abandon MMEX and look elsewhere. It's shame, as it has promise, but without a working mobile version I don't think this is a viable project, I'm happy to help trying to resolve these issues, but with no response I don't really know who I'm trying to help.

[GabrieleV]

Hi, I'm one of developers of desktop version, seems that all problems that you refer to are caused by OneDrive sync.

Sync it's made for all database without any exclusion, so it's impossible that you have categories but not transactions, maybe you don't have categories newly added because DB it's not synced.

Unfortunately they are related to OneDrive sync process (at least on desktop side), so there isn't anything that we could do here, maybe something could be done on mobile side.

surely best way it's wait some minutes before open on different device and never open both contemporary.

[wizardict]

Hi, I'm one of developers of desktop version, seems that all problems that you refer to are caused by OneDrive sync.

Sync it's made for all database without any exclusion, so it's impossible that you have categories but not transactions, maybe you don't have categories newly added because DB it's not synced.

Unfortunately they are related to OneDrive sync process (at least on desktop side), so there isn't anything that we could do here, maybe something could be done on mobile side.

surely best way it's wait some minutes before open on different device and never open both contemporary.

Hi

Thank you for the reply. It doesn't matter if I wait 1 minute, 1 hour or 1 week the entities simply do not sync. The database has synced with OneDrive, I can view it on my phone and see the timestamp has updated.

These issues are not specific to Onedrive, look elsewhere on the forum and you will see issues posted with Dropbox also.

Does the android app use a local cache? It appears this is the case and the entities are cached and never updated.

Transactions do sync, so the database is clearly syncing with Onedrive or that would not be the case.

[GabrieleV]

Unfortunately we have to wait for Android developer because I have no idea about it.

[wizardict]

Thanks for you honest response, I will await an update from the android developer.


Have you considered changing the structure of your application to a centralised sql database rather than local sqllite databases? It would make synchronisation between any device much easier and slicker, as well as making it possible to use the application from multiple devices simultaneously. It would also make the development of a full Web based client much easier and open up new avenues as a networked application.

[wizardict]

OK.....I have located the root of this issue.

The problem occurs when the mobile app can't update the database (i.e. it is locked by the desktop client) and a 2nd database is created. As long as the 2 databases exist OneDrive does not update them correctly, so the 2nd database needs to be deleted then everything syncs as expected.


So the issue is with conflicting updates. Do Dropbox and/or Google drive handle conflicts in the same way? I've seen the same issue reported on dropbox, so presumably that's the same, what about the others?

All of this would be resolved with a proper database and network capabilities

[GabrieleV]

You are right that a centralized DB will solve problems, but most of MMEX users don't want to have their data on a webserver not under their control, so the only way it's keep in sync the full DB file..

[wizardict]

You are right that a centralized DB will solve problems, but most of MMEX users don't want to have their data on a webserver not under their control, so the only way it's keep in sync the full DB file..

That's a good point. I wasn't thinking of a hosted server, more a database that is local to the user so they still have full control, but still that would not suit everyone.

There could be a solution that would suit everyone though, something like SymmetricDB.  That would allow local sqllite databases which you have now, but with the option to sync to a central server for those who want it. It does proper SQL sync so would allow multiple users to use the application at the same time, and would also still allow the mobile app to be used offline.


If anyone is interested in this kind of solution I would be prepared to help develop the desktop side, but it would need your Android developer to implement the android version (which is documented on their website) as I have no experience in android development unfortunately.

[GabrieleV]

Is this one? https://www.symmetricds.org

Considering case where there isn't a central server for security reasons, how will sync handled?
There will be for example 3 DB files on Dropbox called "master", "android", "desktop" and symmetricds will keep them in sync?

[wizardict]

Is this one? https://www.symmetricds.org

Considering case where there isn't a central server for security reasons, how will sync handled?
There will be for example 3 DB files on Dropbox called "master", "android", "desktop" and symmetricds will keep them in sync?

Yes that is the correct product, as you can see it is open source so fits with your product nicely.


The intention of this would be to allow people the option to include a central server, not exclude it. People who don't want a central server would continue to work as they are, people who do want improved functionality and proper sync functions would use this. So...


They would set up a central server of their own (it could be on their desktop still, or a web server whatever they prefer. It could by MSSQL, MYSQL or even SQLITE still, whatever they prefer and is supported by symmetricds).

All of the installs (desktop, mobile client etc) would store the data locally in their own SQLite database, not on dropbox or OneDrive etc these are not suitable methods to sync databases and aren't intended to be used in this way.

SymmetricDS would be used to sync each client with the "central server" owned by the user.




Synchronisation would be far improved, without the "single client only" model that is al that's possible with the current system. This would also open the possibility for different client development, such as a fully functional web based client. As soon as you have a proper "client > server" model you open up a lot of new possibilities and functionality, but doing it this way would still allow people to have the simple standalone install they currently have if they prefer.

[MisterY]

This is starting to feel like my personal blog! Does anyone involved in development monitor or reply to posts on here?

There's a new release of the android app, but unfortunately the sync issues remain. Entities do not sync to the app, and any changes made on the mobile app uploads the database and delete all of the changes that haven't been synchronised.

There is also the issue of the mobile app causing a new database to be created (i.e. it doesn't update the database it results in a separate database being created). I see this issue is reported elsewhere on the forum with dropbox too, so it isn't OneDrive specific.



Unless there is a response from development that these issue are being looked in to I think I will have to abandon MMEX and look elsewhere. It's shame, as it has promise, but without a working mobile version I don't think this is a viable project, I'm happy to help trying to resolve these issues, but with no response I don't really know who I'm trying to help.

Hi,

As the very first item, I would kindly point you to read the documentation that was written for this purpose. It might not be "in-your-face" type of documentation but there is a link to the User Manual from the app's home page. There the process of setting up the synchronization is explained into some detail. There are also a few more pages referring specifically to the (previous) Dropbox-only synchronization but their value is in explaining the concepts. I mention this because some people open the database file from a Dropbox client and expect the synchronization to work seamlessly.
Secondly, please note that this is an Open Source project and it is nobody's duty to supply support. You are certainly welcome to contribute functionality, documentation, code, translations, and provide support to other users in the forums, just like we all are.

If you experience specific issues, you can report them through several channels but do please keep in mind that we also have a life and full-time jobs that take priority over any hobby project. Do get in touch via email, or post an issue on GitHub.

[MisterY]

OK.....I have located the root of this issue.

The problem occurs when the mobile app can't update the database (i.e. it is locked by the desktop client) and a 2nd database is created. As long as the 2 databases exist OneDrive does not update them correctly, so the 2nd database needs to be deleted then everything syncs as expected.


So the issue is with conflicting updates. Do Dropbox and/or Google drive handle conflicts in the same way? I've seen the same issue reported on dropbox, so presumably that's the same, what about the others?

All of this would be resolved with a proper database and network capabilities

Hi, sorry I replied before reading the whole thread.
Yes! This is one of the key issues for the desktop version. The database file is locked as long as the application is running. This will **always** cause a sync conflict if the file is modified on mobile at the same time.
It has been noted earlier and I'm not sure if there is something that can be done in the application, except perhaps constantly closing the connection when it is not needed.
**Or**, a user could simply close the application when he's done with it.

[wizardict]

OK.....I have located the root of this issue.

The problem occurs when the mobile app can't update the database (i.e. it is locked by the desktop client) and a 2nd database is created. As long as the 2 databases exist OneDrive does not update them correctly, so the 2nd database needs to be deleted then everything syncs as expected.


So the issue is with conflicting updates. Do Dropbox and/or Google drive handle conflicts in the same way? I've seen the same issue reported on dropbox, so presumably that's the same, what about the others?

All of this would be resolved with a proper database and network capabilities

Hi, sorry I replied before reading the whole thread.
Yes! This is one of the key issues for the desktop version. The database file is locked as long as the application is running. This will **always** cause a sync conflict if the file is modified on mobile at the same time.
It has been noted earlier and I'm not sure if there is something that can be done in the application, except perhaps constantly closing the connection when it is not needed.
**Or**, a user could simply close the application when he's done with it.

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested? The reality is dropbox etc were never intended to be used in this way, so to develop an application that requires sync across multiple clients really means moving away from this model.

[MisterY]

Is this one? https://www.symmetricds.org

Considering case where there isn't a central server for security reasons, how will sync handled?
There will be for example 3 DB files on Dropbox called "master", "android", "desktop" and symmetricds will keep them in sync?

Yes that is the correct product, as you can see it is open source so fits with your product nicely.


The intention of this would be to allow people the option to include a central server, not exclude it. People who don't want a central server would continue to work as they are, people who do want improved functionality and proper sync functions would use this. So...


They would set up a central server of their own (it could be on their desktop still, or a web server whatever they prefer. It could by MSSQL, MYSQL or even SQLITE still, whatever they prefer and is supported by symmetricds).

All of the installs (desktop, mobile client etc) would store the data locally in their own SQLite database, not on dropbox or OneDrive etc these are not suitable methods to sync databases and aren't intended to be used in this way.

SymmetricDS would be used to sync each client with the "central server" owned by the user.




Synchronisation would be far improved, without the "single client only" model that is al that's possible with the current system. This would also open the possibility for different client development, such as a fully functional web based client. As soon as you have a proper "client > server" model you open up a lot of new possibilities and functionality, but doing it this way would still allow people to have the simple standalone install they currently have if they prefer.

I like the idea in general but wouldn't this destroy the option of having a standalone application, whether desktop or mobile? There's still plenty of users who do not synchronize the database and use a single platform exclusively.

[MisterY]

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested?

There would probably be an option to initialize this only as a part of synchronization settings?

[wizardict]

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested?

There would probably be an option to initialize this only as a part of synchronization settings?

Yes exactly. The application would work exactly as it does currently, with data saved to a local SQLite database, so people that are happy using it this way can continue to do so.
For people that require greater network capabilities there would be an option in the settings to configure the "master server", which would then sync the data to that central database. If you don't want to use it, simply don't enable it.

[MisterY]

Hi

Thank you for your reply. If you read a little further you will se I have suggested a solution, which I will happily contribute towards, but isn't something I could achieve on my own. Do you have any thoughts on the Client>Server model with synchronisation suggested?

There would probably be an option to initialize this only as a part of synchronization settings?

Yes exactly. The application would work exactly as it does currently, with data saved to a local SQLite database, so people that are happy using it this way can continue to do so.
For people that require greater network capabilities there would be an option in the settings to configure the "master server", which would then sync the data to that central database. If you don't want to use it, simply don't enable it.

Certainly seems interesting. I'll do some more reading through their documentation. And would certainly love to use it. Especially if it
could do a record-level sync. This would  greatly reduce the size of the network traffic, for example.
However, I'm concerned that most users would have great issues installing their
own server. Even supporting the sync via well-known file-syncing
providers is causing too much trouble already.

Going back to one thing you mentioned, that you don't know much about Android development.
Well, neither did I before joining this project. It was a way to learn
more and get my hands dirty, so to speak. In any case, I wholeheartedly
recommend the following video to everyone. Please watch at least the
first five minutes:

https://www.youtube.com/watch?v=TK1mBqKvIyU

[MisterY]

Actually, if it can do a row-level replication, I'd be happy to implement it in addition to existing file-level sync.

https://github.com/moneymanagerex/andro ... issues/991

[wizardict]

I agree that many users would not know how to set up their own server, but would want the functionality it provides. I think ultimately there would need to be a "server" installation of MMEX. That should be fairly simple, an installed for a SQL DB of choice and the sync component. For the initial phase though I would say implement the sync and let people with can set up their own server try it out.

In terms of support I think you would find this requires less support than the current system, not more. The issues you're having with dropbox sync is that dropbox was never intended to sync database files in this way, trying to use something for a purpose it was not intended for is always going to cause issues. By implementing something that is designed for this purpose, once the bugs are sorted it, it should work fairly smoothly.


I appreciate this is potentially a fair bit of work, but I think it's worth it. I like MMEX but there are numerous other free open source solution out there that do a similar job. The thing that set MMEX apart for me (and why I'm here) is that I need the ability to enter transactions when I'm out of the house. That's when I make most of my smaller transactions, and currently regularly forget to record them when I get home. The android integration you currently have sort of resolves that issue, but there are too many limitations. Not least the fact that if I forget to close the desktop version and enter a transaction on android I'm going to end up with 2 versions of the database that are difficult to reconcile.

Take a look at the SymmetricDS site and see if you're happy with it and what it can do (there are other similar solutions out there but I think this is the best fit for this project). I believe if you get this right it will really set MMEX apart from the crowd.

[trice001]

thanks for that.

[javiermarcon]

Excelent idea on using symmetricds to manage cetralized database. Is there a tutorial on how to configure symmetricds to sync several mmex databases?

Thanks.

[wizardict]

Excelent idea on using symmetricds to manage cetralized database. Is there a tutorial on how to configure symmetricds to sync several mmex databases?

Thanks.

As far as I'm aware no testing or development has been done in this area so far, so there are no tutorials as you would be breaking new ground. Whilst I think there's agreement that it would improve synchronisation, there are a lot of other things to work on and devs only have a certain amount of time.