cmd/ido2icingadb: migrate history from IDO to Icinga DB #253

Al2Klimov · 2021-02-04T15:30:56Z

No description provided.

julianbrost

Just some first comments for now.

tools/ido2icingadb/main.go

tools/ido2icingadb/util.go

tools/ido2icingadb/sync.go

tools/ido2icingadb/util.go

Al2Klimov · 2021-04-07T15:28:06Z

Now it should work as expected. Please have a look.

Al2Klimov · 2021-09-30T13:32:06Z

1 rewrite, 42 quadrillion centuries and one s/Debian/Ubuntu/ later...

Al2Klimov · 2021-09-30T13:33:45Z

Al2Klimov · 2021-09-30T13:37:04Z

Setup

12 CPU
32 GB RAM
75 GB IDO DB
all on one host

Full migration

took 1269m19.229s
no interrupts

julianbrost · 2022-10-12T10:05:43Z

Do I understand correctly that the time range limitation also applies to populating the cache, i.e. it won't take earlier rows into account, even if they would hold information (like the previous state) for a later event within the migration time range?

Not sure yet if that's good or bad. If you restrict the lower limit, you probably have a reason. If we want to fill as much as possible, we'd have to build the cache from the very beginning, so it might scan 5 years to migrate 1 year. That might not be desired, on the other hand, reading doesn't seem to be the bottleneck.

Al2Klimov · 2022-10-12T13:59:57Z

Oh. You're right. 😅

Just to be sure.

julianbrost · 2022-10-17T08:59:15Z

cmd/ido2icingadb/misc.go

+// slowlyIncrementingProgressDecorator is the base decorator
+// for possibly slowly incrementing progresses possibly starting from >0%.
+type slowlyIncrementingProgressDecorator struct {
+	decor.WC
+
+	// startProgress is the first progress >0 seen by Decor.
+	startProgress int64
+	// startTime tells when is startProgress from.
+	startTime time.Time
+	// lastProgress is the last progress >0 seen by Decor.
+	lastProgress int64
+	// lastTime tells when is lastProgress from.
+	lastTime time.Time
+}
+
+// update is to be called by Decor and returns whether sipd is warmed up.
+func (sipd *slowlyIncrementingProgressDecorator) update(s decor.Statistics) bool {
+	if s.Completed || s.Current < 1 {
+		return false
+	}
+
+	if sipd.startProgress < 1 {
+		sipd.startProgress = s.Current
+		sipd.startTime = time.Now()
+		sipd.lastProgress = sipd.startProgress
+		sipd.lastTime = sipd.startTime
+
+		return false
+	}
+
+	if s.Current == sipd.startProgress {
+		return false
+	}
+
+	if s.Current > sipd.lastProgress {
+		sipd.lastProgress = s.Current
+		sipd.lastTime = time.Now()
+	}
+
+	return true
+}
+
+// eta indicates the ETA for possibly slowly incrementing progresses possibly starting from >0%.
+type eta struct {
+	slowlyIncrementingProgressDecorator
+}
+
+// Decor implements the decor.Decorator interface.
+func (e *eta) Decor(s decor.Statistics) string {
+	if e.update(s) {
+		timePerItem := float64(e.lastTime.Sub(e.startTime)) / float64(e.lastProgress-e.startProgress)
+		lastETA := time.Duration(float64(s.Total-s.Current) * timePerItem)
+
+		return e.FormatMsg(((lastETA - time.Since(e.lastTime)) / time.Second * time.Second).String())
+	} else {
+		return ""
+	}
+}
+
+// eta indicates ops/s for possibly slowly incrementing progresses possibly starting from >0%.
+type opsPerSec struct {
+	slowlyIncrementingProgressDecorator
+}
+
+// Decor implements the decor.Decorator interface.
+func (ops *opsPerSec) Decor(s decor.Statistics) string {
+	if ops.update(s) {
+		return ops.FormatMsg(fmt.Sprintf(
+			"%.0f/s",
+			float64(ops.lastProgress-ops.startProgress)/
+				(float64(ops.lastTime.Sub(ops.startTime))/float64(time.Second)),
+		))
+	} else {
+		return ""
+	}
+}


The progress bars use the average from start to show rates and time estimates. Given that there are index updates involved when inserting, insert complexity should be (at least) logarithmic, i.e. how the insert rate progresses over time can be approximated by 1/ln(x). So inserting will likely only get slower over time (unless there's a magic performance improvement on the database server in the meantime), making the values displayed too optimistic.

Thus, I think displaying an average value over the last few minutes or transactions (to avoid having the exact numbers jump around like crazy) would produce more useful numbers here.

Is it worth investing the time?

Shouldn't take that much time I think. Should just need a slice of progress/time pairs where you keep the last like 10 updates and then perform the same calculation just with the oldest and newest values from that slice.

Feel free to continue reviewing while I'm on it.

julianbrost

Apart from the inline comment, I think you can start doing the other things that will be needed to get this ready:

Rename the cmd/ido2icingadb/ to the final name (probably icingadb-migrate as it already says in the README).
Integrate the README into doc/.

cmd/ido2icingadb/convert.go

julianbrost

Please have a look at the paths, that rename went wrong.

julianbrost

Code should be done now, just getting the docs ready left I think. In general, I miss a clear "do this because this will probably be what you want" with some "in some cases you might want this instead" options.

doc/06-Migration.md

julianbrost · 2022-10-26T12:56:04Z

doc/06-Migration.md

+   type: pgsql
+   host: 192.0.2.1
+   port: 5432
+   database: icinga
+   user: icinga
+   password: CHANGEME
+   # Input time range
+   #from: 0
+   #to: 2147483647
+# Icinga DB database
+icingadb:
+   type: mysql


I'd not mix database types here, might lead to confusion. I get that this is supposed to say that both types are supported, but I think that would be easier to understand if both occurrences said type: mysql # or pgsql for PostgreSQL.

This shall also express that cross-RDBMS-type migration is supported.

doc/06-Migration.md

Al2Klimov · 2022-10-26T13:31:09Z

?

julianbrost · 2022-10-26T13:46:36Z

?

?

(If you're wondering about an incomplete sentence you've read in a notification e-mail, please read it again on the website.)

(PS: I hate ISO keyboard layouts where the backspace and return key are directly next to each other.)

Al2Klimov · 2022-10-26T13:51:20Z

Er, no, I've read #253 (review) on the website.

julianbrost · 2022-10-26T14:07:11Z

I think for almost all users, the workflow of setting up Icinga DB in parallel and then migrating everything up to that point in time where they started running Icinga DB is what they want. With the current state of the documentation, they have to figure this out themselves but I think this should be documented as the way to go. This can then have extra sections for things like "If you only want to migrate the history for the last year, you can additionally set a start time" or "If you had trouble setting up Icinga DB in the first place, please double check that the time stamp obtained from the database makes sense".

julianbrost · 2022-10-31T11:09:09Z

I've pushed a suggestion with how to improve the docs to the feature/migrate-history-docs branch.

While writing the documentation, I asked myself why we even need to ask the user to provide an endpoint name. All endpoint_id columns in the history tables are nullable so it should be perfectly fine to just write NULL there (needs checking that web handles this correctly). AFAIK this column isn't really used much anyways and I don't think it would hurt if it could be used to distinguish migrated events from native events.

julianbrost

I'll merge this now as-is as there are no real blockers in here anymore so that we can go on with other changes without introducing merge conflicts with this huge PR all the time. I'll open a new PR for the outstanding details.

Al2Klimov added the needs-feedback We'll only proceed once we hear from you again label Feb 4, 2021

Al2Klimov assigned N-o-X, julianbrost and lippserd Feb 4, 2021

julianbrost reviewed Feb 5, 2021

View reviewed changes

Al2Klimov force-pushed the feature/migrate-history branch from b1a9720 to 70d901a Compare April 6, 2021 11:39

Al2Klimov force-pushed the feature/migrate-history branch from 70d901a to 3b3f6d2 Compare April 23, 2021 09:55

Al2Klimov force-pushed the feature/migrate-history branch 3 times, most recently from def926e to 3eb22d9 Compare August 3, 2021 13:50

lippserd added this to the v1.0.0-rc2 milestone Aug 12, 2021

julianbrost mentioned this pull request Sep 7, 2021

SLA reporting #247

Merged

Al2Klimov force-pushed the feature/migrate-history branch from 3eb22d9 to a2b2286 Compare September 30, 2021 13:32

cla-bot bot added the cla/signed label Sep 30, 2021

Al2Klimov requested a review from julianbrost September 30, 2021 13:34

Al2Klimov unassigned julianbrost Sep 30, 2021

Al2Klimov requested a review from N-o-X September 30, 2021 13:35

Al2Klimov unassigned N-o-X and lippserd Sep 30, 2021

Al2Klimov changed the title ~~tools/ido2icingadb: migrate history from IDO to Icinga DB~~ cmd/ido2icingadb: migrate history from IDO to Icinga DB Sep 30, 2021

Al2Klimov force-pushed the feature/migrate-history branch 6 times, most recently from cd8977e to 7936077 Compare October 1, 2021 10:41

cmd/ido2icingadb: build cache from [0,toId], not just [fromId,toId]

649f7e0

Just to be sure.

julianbrost reviewed Oct 17, 2022

View reviewed changes

Al2Klimov force-pushed the feature/migrate-history branch from d099855 to 649f7e0 Compare October 24, 2022 14:31

cmd/ido2icingadb: use built-in decorators for the progress bar

70a0bfb

Al2Klimov requested a review from julianbrost October 24, 2022 14:46

Al2Klimov added this to the 1.1.0 milestone Oct 25, 2022

julianbrost requested changes Oct 25, 2022

View reviewed changes

cmd/ido2icingadb/convert.go Show resolved Hide resolved

Al2Klimov requested a review from julianbrost October 25, 2022 16:25

cmd/ido2icingadb: move docs to the main tree

0ff1728

Al2Klimov force-pushed the feature/migrate-history branch from d1f1dc1 to ec4750d Compare October 26, 2022 08:25

julianbrost requested changes Oct 26, 2022

View reviewed changes

cmd/: rename ido2icingadb/ to icingadb-migrate/

6a5f3fd

julianbrost mentioned this pull request Oct 26, 2022

Fix output not rendered yet Icinga/icingadb-web#666

Merged

Al2Klimov force-pushed the feature/migrate-history branch from ec4750d to 6a5f3fd Compare October 26, 2022 09:19

Al2Klimov mentioned this pull request Oct 26, 2022

go get -u github.com/goccy/go-yaml #534

Merged

julianbrost requested changes Oct 26, 2022

View reviewed changes

Migration: docs enhancements

c51d767

julianbrost approved these changes Nov 2, 2022

View reviewed changes

julianbrost merged commit 9fb9a10 into master Nov 2, 2022

julianbrost deleted the feature/migrate-history branch November 2, 2022 11:16

julianbrost mentioned this pull request Nov 2, 2022

Improve documentation for migration tool #536

Merged

2 tasks

julianbrost mentioned this pull request Mar 17, 2023

Write checkable create & delete sla history events #566

Open

oxzi added the area/migrate cmd/icingadb-migrate label Jun 4, 2025

cmd/ido2icingadb: migrate history from IDO to Icinga DB #253

cmd/ido2icingadb: migrate history from IDO to Icinga DB #253

Uh oh!

Conversation

Al2Klimov commented Feb 4, 2021

Uh oh!

julianbrost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Al2Klimov commented Apr 7, 2021

Uh oh!

Al2Klimov commented Sep 30, 2021

Uh oh!

Al2Klimov commented Sep 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

Al2Klimov commented Sep 30, 2021

Setup

Full migration

Uh oh!

julianbrost commented Oct 12, 2022

Uh oh!

Al2Klimov commented Oct 12, 2022

Uh oh!

julianbrost Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

Al2Klimov Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

julianbrost Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

Al2Klimov Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

julianbrost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

julianbrost left a comment

Choose a reason for hiding this comment

Uh oh!

julianbrost left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

julianbrost Oct 26, 2022

Choose a reason for hiding this comment

Uh oh!

Al2Klimov Oct 26, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Al2Klimov commented Oct 26, 2022

Uh oh!

julianbrost commented Oct 26, 2022

Uh oh!

Al2Klimov commented Oct 26, 2022

Uh oh!

julianbrost commented Oct 26, 2022

Uh oh!

julianbrost commented Oct 31, 2022

Uh oh!

julianbrost left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Al2Klimov commented Sep 30, 2021 •

edited

Loading

julianbrost left a comment •

edited

Loading