Implement itemized invoices for VMs

[knikolla]

Have the scripts output 1 CSV that contains the usage per each VM of a project.

[QuanMPhm]

Just to clarify requirements, we want the VM itemized invoice to be made with the format as the Pod itemized invoice created by @naved001 in openshift-usage-scripts. This means, columns we would want for our invoice is:

• Namespace
• VM Start time
• VM End time
• Duration
• VM Name
• CPU/GPU/RAM request
• GPU Type
• GPU Resource
• Node
• Node Model
• Determining Source
• SU Type
• SU Count

We can remove any columns that are not applicable for Openstack VMs

[KelvinLinBU]

My initial thoughts on this are to add an optional CLI for the path of the itemized invoice. From there, we'd implement a write_vm_itemized_invoice that writes the csv similar to the existing write function. If the itemized invoice path is passed in, we'd run this function and output another csv file. Thoughts?

[QuanMPhm]

@KelvinLinBU I'd say that the script should always export the itemized invoice, and that you can add an cli argument for the name of the itemized invoice, with a default defined. All of our billing code that export files (i.e invoicing) will always produce the same set of files, regardless if a custom name is given through the cli. For this issue, a list of things you'd probably want to do is:

• A a cli argument for the name of the itemized invoice, defaults to /tmp/vm_invoices.csv or something similar
• From generate_billing, you should implement 2 functions, collect_itemized_vm_data_from_openstack, to get the list of VM invoices, and write_vm_itemized_invoice, as you suggested, to write the itemized VM invoice.
• By extension of getting the vm invoices, you should also implement a VMInvoice dataclass, similar to ProjectInvoice
• Make sure the file can be uploaded to S3, given the state of the S3 code, you can probably refactor it a bit by making a upload_to_s3 function, similar to how things are handled in openshift-usage-scripts. @knikolla @joachimweyl Is VM-NERC OpenStack <YYYY-MM>.csv fine for the VM invoice?
• Add test cases for every function you implemented

I can't give you a whole SQL DB dump to test the scripts end-to-end (at least not yet, I have to figure out how to anonymize the data or create a fake dump for you), but the unit tests should be enough for you to understand how VMs are represented (the Instance class) and how we derive billing data from them. Once you submit your PR, I will test your changes.

[knikolla]

• I would put the default to $PWD/vm_invoices.csv as this isn't an intermediate and temporary output of the program.
• That S3 path is fine for now, we can always revise later.

I don't have any further design feedback at this stage.