Plugin Wanted: Collector for vCenter

[byronschaller]

I'd like to get a collector that can play well with VMware vCenter. Given vCenter telemetry metrics can be pulled via an API using govmomi the collector plugin would most likely leverage the library.

An alterrnative would be to use a intermediary like the Statsfeeder Fling to publish the metrics. Ideally this would be some kind of consumable endpoint for Snap, however nativily it publishes to CSV.

[mbbroberg]

I know you already know this, but I love the idea @byronschaller. Plugin Authoring documentation is in flight as we migrate our client library here: https://github.com/intelsdi-x/snap-plugin-lib-go

So if you want to get started, I'm thinking we need:

1. A target example of using the library in Go. Hey @IRCody or @kindermoumoute, is there a plugin out there you can recommend?
2. Clone that example as a skeleton and we'll work through it on Slack

Thanks so much for posting this!

[kindermoumoute]

You can use a mock example from Snap repo: https://github.com/intelsdi-x/snap/tree/master/plugin/collector/snap-plugin-collector-mock2-grpc

[mbbroberg]

Good thought @kindermoumoute, though mocks != examples because they're quite confusing for new users. The PR on the library (intelsdi-x/snap-plugin-lib-go#26) has a better example coming together. I'm more interested in a skeleton that could be a basis for a good new plugin.

[kindermoumoute]

There is also a collector example from the library repo.
https://github.com/intelsdi-x/snap-plugin-lib-go/tree/master/examples/collector

AFAIK this one and the above mock2-grpc are the only collector you can find that use the new library.

[mbbroberg]

Hey @byronschaller! Have you had a chance to design anything out for this one? Keep me posted 🏹

[byronschaller]

Hey @mjbrender, not yet. Caught up in work projects. It is on my docket for after the first of the year though. I will keep you posted.

[jjlakis]

Did a lot of research and tests of govmomi already. I'll be working on this plugin in next few weeks. @andrzej-k please assign

[andrzej-k]

Hey @jjlakis - I cannot assign you (seems that only snap maintainers can be assigned) - but acknowledge that you'll work on this - thank you!

[mbbroberg]

Cool @jjlakis. Heads up that API calls to vCenter can be very expensive, so it will be important for you to recommend usage in the documentation (for example, say you only recommend 1m intervals).

[jjlakis]

@byronschaller We are actively working on vCenter plugin, especially on host-, datastore- and virtualdisk-level. Metric list will be generated from available vSphere perf counters. We don't want to miss anything, so could you tell me is there any specific usecase or feature of this plugin you think it should contain?

cc @andrzej-k @mbbroberg

[exbane]

Hey @jjlakis - I can send you a list of the specific Metrics I collect via vsphere-influxdb-go collector and the one's that i'd like it to collect but currently doesn't. Would that help?

[jjlakis]

@exbane From what I see, vsphere-influx-db-go collects all available perfCounter metrics, so metrics you can collect depends on the version of vSphere cluster and limitations of govmomi.
We are finishing work on our plugin and it will be upstreamed recently. Not all perfCounter metrics are included in it for now, but it would be easily extendable. We also created a mechanism to calculate values that are missing in perfCounters using available perfCounters.
It would be great if I can get this list from you, maybe we can add some missing metrics to cover your use cases :)

[exbane]

Here is what I collect right now with vsphere-influxdb-go. Everything works with the exception of datastore consumption metrics (free/used space) vmop. metrics such as vmotions/poweron/poweroff/svmotion/creation/deletion


   "Metrics": [
	{ 
		"ObjectType": [ "VirtualMachine", "HostSystem"],
		"Definition": [
			{ "Metric": "cpu.usage.average", "Instances": "*" },
			{ "Metric": "cpu.ready.summation", "Instances": "*" },
			{ "Metric": "mem.usage.average", "Instances": "*" },
			{ "Metric": "mem.capacity.usage.average", "Instances": "*" },
			{ "Metric": "mem.consumed.average", "Instances": "*" },
			{ "Metric": "mem.active.average", "Instances": "*" },
			{ "Metric": "mem.vmmemctl.average", "Instances": "*" },
			{ "Metric": "net.packetsRx.summation", "Instances": "*" },
		        { "Metric": "net.packetsTx.summation", "Instances": "*" },
			{ "Metric": "net.throughput.usage.average", "Instances": "*" },
			{ "Metric": "net.received.average", "Instances": "*" },
			{ "Metric": "net.transmitted.average", "Instances": "*" },
			{ "Metric": "net.throughput.usage.nfs.average", "Instances": "*" },
			{ "Metric": "datastore.numberReadAveraged.average", "Instances": "*" },
			{ "Metric": "datastore.numberWriteAveraged.average", "Instances": "*" },
			{ "Metric": "datastore.read.average", "Instances": "*" },
			{ "Metric": "datastore.write.average", "Instances": "*" },
			{ "Metric": "datastore.totalReadLatency.average", "Instances": "*" },
			{ "Metric": "datastore.totalWriteLatency.average", "Instances": "*" }
			
			
		]
	},
	{ 
		"ObjectType": [ "VirtualMachine" ], 
		"Definition": [
	
		{ "Metric": "virtualDisk.read.average", "Instances": "*" },
		{ "Metric": "virtualDisk.write.average", "Instances": "*" },
		{ "Metric": "virtualDisk.readOIO.latest", "Instances": "*" },
		{ "Metric": "virtualDisk.writeOIO.latest", "Instances": "*" },
		{ "Metric": "virtualDisk.totalReadLatency.average", "Instances": "*" },
		{ "Metric": "virtualDisk.totalWriteLatency.average", "Instances": "*" },
		{ "Metric": "virtualDisk.numberReadAveraged.average", "Instances": "*" },
		{ "Metric": "virtualDisk.numberWriteAveraged.average", "Instances": "*" },
		{ "Metric": "cpu.readiness.average", "Instances": "*" },
		{ "Metric": "mem.capacity.provisioned.average", "Instances": "*"},
		{ "Metric": "cpu.corecount.provisioned.average", "Instances": "*" },
		{ "Metric": "disk.capacity.provisioned.average", "Instances": "*" },
		{ "Metric": "disk.capacity.usage.average", "Instances": "*" },
		{ "Metric": "mem.swapped.average", "Instances": "*" }
		]
	},
	{ 
		"ObjectType": [ "ClusterComputeResource"], 
		"Definition": [
			{ "Metric": "mem.capacity.provisioned.average", "Instances": "*"},
			{ "Metric": "cpu.corecount.provisioned.average", "Instances": "*" },
			{ "Metric": "vmop.numCreate.latest", "Instances": "*" },
			{ "Metric": "vmop.numPoweroff.latest", "Instances": "*" },
			{ "Metric": "vmop.numPoweron.latest", "Instances": "*" },
			{ "Metric": "vmop.numDestroy.latest", "Instances": "*" },
			{ "Metric": "vmop.numClone.latest", "Instances": "*" },
			{ "Metric": "vmop.numDeploy.latest", "Instances": "*" },
			{ "Metric": "vmop.numVMotion.latest", "Instances": "*" },
			{ "Metric": "vmop.numSVMotion.latest", "Instances": "*" },
			{ "Metric": "vmop.numXVMotion.latest", "Instances": "*" },
			{ "Metric": "mem.capacity.usable.average", "Instances": "*" }

[exbane]

@jjlakis How's it coming with the integration? curious if you need anybody to test anything.

Thanks!

[jjlakis]

@exbane Thanks! Plugins are ready, I am focused on integrating them with some internal products. I need to do couple more things before opensourcing, hope it won't take a lot of time.

[king-jam]

@jjlakis any update on this plugin?

Edit: asking because I'm about to start writing one and would rather contribute to what you folks have if there are gaps I'm going to fill.

[king-jam]

@jjlakis back from vacation? Any update?

[Frank-G]

@jjlakis @king-jam I am also interested in this plugin and would contribute.

[jjlakis]

@king-jam @Frank-G I'm no longer working on snap plugins. This collector is done and needs to be opensourced, @andrzej-k could you please point a person who is responsible for this now?

[king-jam]

@jjlakis @andrzej-k I have contacts with your internal engineering folks if you want me to sign some NDAs and write/test code for any final work to open source.

[taotod]

@king-jam, please contact @ZhengQingyuan and me for publishing this connector. We have taken over this project from @andrzej-k 's team.
Please follow the guide of https://github.com/intelsdi-x/snap/blob/master/docs/PLUGIN_AUTHORING.md
If you have any question or need any help, please feel free to contact us. Many thanks for your contribution.

[king-jam]

@taotod I believe @jjlakis has the code. We would need a handoff somehow.

[taotod]

@jjlakis do you want to directly publish the plugin by yourself or have us do that?
If you want to do, please follow the guide of https://github.com/intelsdi-x/snap/blob/master/docs/PLUGIN_AUTHORING.md
If you want us to do, please share the code with @ZhengQingyuan and me. Thanks.

@king-jam Many thanks for the info.

[andrzej-k]

Hello, @taotod - I would be great if you could continue working on open sourcing this plugin. The source code is here: https://github.com/intelsdi-x/snap-plugin-collector-vsphere, and you should have access to it now.

[Frank-G]

Hi @andrzej-k, thanks for the effort, but I can´t access https://github.com/intelsdi-x/snap-plugin-collector-vsphere (404).

[andrzej-k]

Hi @Frank-G, sorry about that - we just need to make some final polishing before opening this repo to the public - but that shouldn't take long. Right now it's accessible to @intelsdi-x/snap-maintainers group to let them complete work, once it's done this plugin will be added to our Plugin Catalog - making it available to everyone. Hope that helps.

[Frank-G]

@andrzej-k Thanks a lot!

[Frank-G]

Hi @andrzej-k , do you have a rough schedule for the release?

Many thanks in advance!

[andrzej-k]

Hi @Frank-G, I'm sorry about redirection, but since @taotod is working on this I think it would be the best to get first hand answer.

@taotod Could you share the expected date when vSphere plugin will be open sourced?

[taotod]

@Frank-G Many thanks for asking. We are going through some internal legal review process. The release time is still to be decided. We will try our best to release it asap.

[king-jam]

@taotod any update?

[ZhengQingyuan]

@king-jam @taotod We are working on SWLC and SDL process, because the 'govmomi' library is not on Intel's white list, so we have made a request to add this library to Intel's white list. And now waiting approval.

[king-jam]

Thank you @ZhengQingyuan

[ZhengQingyuan]

@king-jam vCenter plugin already publish, url: https://github.com/intelsdi-x/snap-plugin-collector-vsphere