Average Spending per Category/Subcategory Report

Build and share your cool customized reports built w/ one sql, Chart.js and Lua

Moderator: Renato

dimidimi
Senior MMEX User
Posts: 37
Joined: Tue Mar 04, 2014 10:46 am
Are you a spam bot?: No

Re: Average Spending per Category/Subcategory Report

Post by dimidimi »

guanlisheng wrote:No time to review your SQL carefully but to be honest, your code is too long and complex.
These unions are not needed at all and i have posted one simplified one days ago.

https://gist.github.com/guanlisheng/9905673
Hi guanlisheng!

I don't know how I missed that :( You're absolutely right about the unions, the ifnull() implementation is so much easier to follow! Thanks! However, I believe that the query does not calculate the average correctly. If, for example, you have transactions that repeat on the same month but in different years, they will be counted as one which obviously is not correct. I have implemented a correct version by tweaking the database schema using views. If would be extremely complex to make one single query for getting everything. I will give it a try though and see how it would work ;-)

Thanks again
Dimitrios
guanlisheng
MMEX Developer
Posts: 375
Joined: Wed Dec 21, 2011 5:58 am
Are you a spam bot?: No
Location: China
Contact:

Re: Average Spending per Category/Subcategory Report

Post by guanlisheng »

Hi Dimitrios, thanks for pointing active months count issue and i just post a in-place fix on gist.

Regarding the SQL aspect for general report, it should be less than 50 lines which meant we can read all sql in on screen. and better to write code in gist and just post on link in forum.
Lisheng Guan,
Developer of MoneyManagerEX (http://moneymanagerex.org)
dimidimi
Senior MMEX User
Posts: 37
Joined: Tue Mar 04, 2014 10:46 am
Are you a spam bot?: No

Re: Average Spending per Category/Subcategory Report

Post by dimidimi »

guanlisheng wrote:Hi Dimitrios, thanks for pointing active months count issue and i just post a in-place fix on gist.

Regarding the SQL aspect for general report, it should be less than 50 lines which meant we can read all sql in on screen. and better to write code in gist and just post on link in forum.
guanlisheng hi!

I have managed to make the entire output recordset into a single SQL query. It is somewhat larger that 50 lines, 78 lines to be exact, but what bothers the most is how the database engine would handle a particular subquery; i.e. the one which gets all withdrawals for both normal and splitted transactions. This is actually the one you fine tuned in your version of report:

Code: Select all

(select ifnull(split.categid, tran.categid) as categid,
          ifnull(split.subcategid, tran.subcategid) as subcategid,
             tran.transdate as transdate,
             ifnull(split.splittransamount, tran.transamount) as transamount,
             tran.notes
     from checkingaccount_v1 as tran
     left join splittransactions_v1 as split
     on tran.transid = split.transid
     where tran.payeeid != 15
     and tran.subcategid <> 66
     and tran.transcode = 'Withdrawal'
     order by categid,subcategid,transdate
) categ_subcateg_withdrawals
group by categ_subcateg_withdrawals.categid, categ_subcateg_withdrawals.subcategid
It looks to me that the 3rd occurrence might be redundant, but I haven't figured a way to remove it yet. At any rate, though, this is the idea for computing a correct average which also covers for the case in which you pay expenses not only in a monthly base, but also bi-montly, 4 times/year, twice a year etc.

If you find an easier to way to do this please let me know ;) Thanks!
Dimitrios


EDIT:
OK... I managed to eliminate the 3rd occurrence of the categ_subcateg_withdrawals subuquery, improving the performance and simplicity of the overall statement. Please check the attached SQL.
Attachments
AveragesPerCategSubcategOneSubqueryLess.zip
(1001 Bytes) Downloaded 561 times
guanlisheng
MMEX Developer
Posts: 375
Joined: Wed Dec 21, 2011 5:58 am
Are you a spam bot?: No
Location: China
Contact:

Re: Average Spending per Category/Subcategory Report

Post by guanlisheng »

I don't understand what is your problem here since i think the SQL in gist already resolves requirements to retrieve active month (or other period) count and total amount.

Regarding the SQL line #, if it is over 50 lines, i can 90% guarantee that it is problematic.
Lisheng Guan,
Developer of MoneyManagerEX (http://moneymanagerex.org)
dimidimi
Senior MMEX User
Posts: 37
Joined: Tue Mar 04, 2014 10:46 am
Are you a spam bot?: No

Re: Average Spending per Category/Subcategory Report

Post by dimidimi »

guanlisheng wrote:I don't understand what is your problem here since i think the SQL in gist already resolves requirements to retrieve active month (or other period) count and total amount.

Regarding the SQL line #, if it is over 50 lines, i can 90% guarantee that it is problematic.
guanlisheng, the problem with the query is that it does not correctly calculate the number of months between the first and the last withdrawal for e particular (categid, subcateg) pair. Based of the month portion of the transdate, it counts the number of occurrences. This is incorrect as two payments on the same month but different year, say 11/2/2013 and 21/2/2014, will be averaged on 1 instead of 12 months. Also, half-year payments, say 15/2/2013 and 15/8/2013, will be averaged over 2 instead over 12 months. Hope this makes sense. Given that I was not able to implement two queries and a more appropriate logic in the Lua section, the correct calculation has to be performed in a single query that has to be more that 50 lines long.
guanlisheng
MMEX Developer
Posts: 375
Joined: Wed Dec 21, 2011 5:58 am
Are you a spam bot?: No
Location: China
Contact:

Re: Average Spending per Category/Subcategory Report

Post by guanlisheng »

Hi,
This is incorrect as two payments on the same month but different year, say 11/2/2013 and 21/2/2014, will be averaged on 1 instead of 12 months
this has been fixe by

Code: Select all

count(distinct strftime('%Y%m', tran.TRANSDATE)) as [Months Active]
on https://gist.github.com/guanlisheng/9905673 and i have mentioned it as well.
Also, half-year payments, say 15/2/2013 and 15/8/2013, will be averaged over 2 instead over 12 months.
A little confusion about your definition upon 'Active Months'. Intuitively, the correct answer should be 2 here. therefore, i still don't get your point here. would you please provide more example to illustrate the calculation logic here.


From my understanding, 99% SQL for general report should be less 50 lines.
Lisheng Guan,
Developer of MoneyManagerEX (http://moneymanagerex.org)
guanlisheng
MMEX Developer
Posts: 375
Joined: Wed Dec 21, 2011 5:58 am
Are you a spam bot?: No
Location: China
Contact:

Re: Average Spending per Category/Subcategory Report

Post by guanlisheng »

Regarding the Year/Quarter/Month/Week Actives count, the key here is to align TRANSDATE to the according period level.
Lisheng Guan,
Developer of MoneyManagerEX (http://moneymanagerex.org)
guanlisheng
MMEX Developer
Posts: 375
Joined: Wed Dec 21, 2011 5:58 am
Are you a spam bot?: No
Location: China
Contact:

Re: Average Spending per Category/Subcategory Report

Post by guanlisheng »

Let's talk more on skype (lisheng.guan) if possibly
Lisheng Guan,
Developer of MoneyManagerEX (http://moneymanagerex.org)
dimidimi
Senior MMEX User
Posts: 37
Joined: Tue Mar 04, 2014 10:46 am
Are you a spam bot?: No

Re: Average Spending per Category/Subcategory Report

Post by dimidimi »

guanlisheng hi-

You're right... The term 'active' months does not completely reflect the concept I have in mind ;-) Specifically, what I want to accomplish with this report is to know what I spend per Category/Subcategory on a monthly basis.

For payments made on a monthly basis counting the number of transactions, such as count(distinct strftime('%Y%m', tran.TRANSDATE)) as [Months Active], would work just fine. For other payments that occur yearly, quarterly etc. simply counting the number of transactions would not yield to a correct average. For example, I pay my car insurance (Category: Vehicles, Subcategory: Insurance) in two installments:

2014/02/24 (first installment for the period 2014/02 – 2014/08): 324.15 Euros
2014/08/24 (first installment for the period 2014/02 – 2015/08): 335.24 Euros

So my monthly average here should be 659.39 / 12 = 54.95 whereas by simply counting the number of withdrawals will give me 329.70. Hope this makes sense. I will try to contact you through Skype what timezone are you in?
guanlisheng
MMEX Developer
Posts: 375
Joined: Wed Dec 21, 2011 5:58 am
Are you a spam bot?: No
Location: China
Contact:

Re: Average Spending per Category/Subcategory Report

Post by guanlisheng »

Still take your installments as example, i think there should be 12 transactions in database instead of only 2. so the monthly average IS 659.39 / 12 = 54.95. (there must be some mis-spell in your case).

To this case, it depends how you record the installments. From my understanding it should be repeat transaction which will happen monthly and result in 12 separate transactions in six months.


I am living in China and the timezone is GMT+8.
Lisheng Guan,
Developer of MoneyManagerEX (http://moneymanagerex.org)
Post Reply